Differentially Private Generalized Linear Models Revisited
Authors: Raman Arora
Raef Bassily
Cristóbal A Guzmán
Michael Menart
Enayat Ullah
Venue: NeurIPS 2022 Accept (2022)
Keywords: differential privacy
supervised learning
generalized linear model
optimization

Abstract: We study the problem of $(\epsilon,\delta)$-differentially private learning of linear predictors with convex losses. We provide results for two subclasses of loss functions. The first case is when the loss is smooth and non-negative but not necessarily Lipschitz (such as the squared loss). For this case, we establish an  upper bound on the excess population risk of $\tilde{O}\left(\frac{\Vert w^*\Vert}{\sqrt{n}} + \min\left\{\frac{\Vert w^* \Vert^2}{(n\epsilon)^{2/3}},\frac{\sqrt{d}\Vert w^*\Vert^2}{n\epsilon}\right\}\right)$, where $n$ is the number of samples, $d$ is the dimension of the problem, and $w^*$ is the minimizer of the population risk. Apart from the dependence on $\Vert w^\ast\Vert$, our bound is essentially tight in all parameters. In particular, we show a lower bound of $\tilde{\Omega}\left(\frac{1}{\sqrt{n}} + {\min\left\{\frac{\Vert w^*\Vert^{4/3}}{(n\epsilon)^{2/3}}, \frac{\sqrt{d}\Vert w^*\Vert}{n\epsilon}\right\}}\right)$. We also revisit the previously studied case of Lipschitz losses \cite{SSTT21}.  For this case, we close the gap in the existing work and show that the optimal rate is (up to log factors) $\Theta\left(\frac{\Vert w^*\Vert}{\sqrt{n}} + \min\left\{\frac{\Vert w^*\Vert}{\sqrt{n\epsilon}},\frac{\sqrt{\text{rank}}\Vert w^*\Vert}{n\epsilon}\right\}\right)$, where $\text{rank}$ is the rank of the design matrix. This improves over existing work in the high privacy regime. Finally, our algorithms involve a private model selection approach that we develop to enable attaining the stated rates without a-priori knowledge of $\Vert w^*\Vert$. 

Differentially Private Generalized Linear Models Revisited
Raman Arora Department of Computer Science The Johns Hopkins University arora@cs. juu
Raef Bassily
Department of Computer Science & Engincering Translational Data Analytics Institute (TDAI)
.du edu
The Ohio State University bassily.. 1uosu.ueeeeeeooooo
Cristóbal Guzmán
Michael Menart
Inst. for Mathematical and Comput. Eng. Fac. Matemáticas and Esc. de Ingenieria Pontificia Universidad Católica de Chile
Department of Computer Science & Engineering
The Ohio State University menart 20osu, edu
c. guwman@utwente.cm ml
Enayat Ullah
Department of Computer Science The Johns Hopkins University enayat@jhu. edu
Abstract
We study the problem of ( differenttilyy private learning of linear predie- tors with convex losses. We provide results for two subclasses of loss functions. The first case is when the loss is smooth and non-negative but not necessar- ily Lipschitz (such as the squared loss). For this case, we establish an upper
bound on the excess population risk of Ō [["'l min {
||="|| Vall"" [mc)=>5 ni
Vn
where is the number of samples, d is the dimension of the problem, and
wr is the minimizer of the population risk. Apart from the dependence on [w= our bound is essentially tight in all parameters. In particular, we show lower bound of Ω [1L
vdlle=II)) )). We also revisit the pre- viously studied case of Lipschitz losses [SSTT20). For this case, we close the gap in the existing work and show that the optimal rate is (up to log factors) + [| VT
+ min [w 14/3
vn
[ne)273',
[|="|| vrank|]lw" ((un)le""")) where rank is the rank of the design ma- Vne
+ min
trix. This improves over existing work the high privacy regime. Finally, our algorithms involve a private model selection approach that we develop to enable attaining the stated rates without a-priori knowledge of ||w=|]
1 Introduction
Ensuring privacy users' data in machine learning algorithms is an important desideratum for multiple domains, such as social and medical sciences. Differential privacy (DP) has become the gold standard of prreecy-p data analysis as it offers formal and quantitative privacy guarantees and enjoys many attractive properties from an algorithmic design perspective [DR + 14]. However, for various basic machine learning tasks, the known risk guarantees are potentially sub-optimal and pessimistic.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

T-SSooth, Non-negative
G Lipschit
VIl lio*|
Lxl > DIIl
[ 11 [ilme
[ I (xller
d (me) (
d (ne)2/a
rank S
rank ne
UB
#ijle (")
iIvi' livint
[sagh
ne Themasi <Pli' [X
(wa)
vhr
Thorens
Taenss 2,3
Thoeesi RA.5
LB
Tight
(
Tight
Tight
UB
Nesptink
[o733]
LB
main yIp TI EIT
Tight
Table I: Summary of Rates. Parameters: d: dimension, n: sample size, H: smoothness parameter, w": minimum norm population risk minimizer, ||A bound on feature vectors, || bound on loss at zero, G: Lipschitzness parameter, rank: expected rank of the design matrix, e: privacy parameter factors omitted). The actual private excess risk bounds are the sum of the expressions shown in the DP rows and their non-private counterparts, Details on nou-prie private lower bounds Appendix
In this work, make progress towards resolving one of the most basic machine learning problems under differential privacy: learning linear predictors with convex losses.
1.1 Related Work
Differentially private machine learning has been thoroughly studied for over a decade. the Lipschitz- convex setting, tight rates known for both the empirical and population risk [BST14, BFTGT19). Specifically, it was shown that in the constrained setting, dependence on the dimension in the form of O vdd is unavoidable even for generalized linear models (GLM) (see Section 2 for formal definition). By contrast, in the unconstrained setting, it has been shown that dimension independent rates possible for GLMs [JT14]. In this setting, assuming prior knowledge of ||w*]], the best llv/rak [SSTT20], where rank is the expected rank of the design matrix. ||
zur
known rate is O
[| v
+
However, without prior knowledge of ||w"|| these methods exhibit quadratic dependence on || I|- Furthermore, these results crucially rely the assumption that the loss is Lipschitz to bound the sensitivity, Although gradient clipping has been proposed to remedy this problem (SSTT20, CWH20), is known that the solution obtained by clipping may not coincide with the one of the original model.
Without Lipschitzness, work on differentially private GLMs has largely been limited to linear regression [Wan18, CWZ21]. Here, dimension independent rates have only been obtained under certain sparsity assumptions.
More generally, smooth non-negative losses have been studied in the non-private setting by [SST10), where was shown such functions can obtain risk guarantees with linear dependence on the minimizer norm (as in the Lipschitz case). This work also established lower bound of 9 (t))) the excess population risk for this class of loss functions. [Sha15] additionally establishes lower bound of the excess population risk by way of linear regression'.
Ω
min
10
ww")
}
vn
1.2 Our Contributions
Smooth nonegative GLMs. Our primary contribution is and nearly optimal rate for the problem of differentially private learning of smooth GLMs. In this setting, we focus on characterizing the excess risk in terms of n, d, and |w" Specifically, show that it is possible to achieve a rate w=|"" 1
the excess population risk. Our new rates exhibit an interesting low/high dimensional transition at d [[]w=]] )2/3/3 First, in the low dimensional regime, we develop novel analysis of noisy gradient decent (GD) inspired by techniques from (SST10). In particular, we show that Noisy GD gives an improved rate for non-negative smooth functions (not necessarily GLMs). This is based on an average stability analysis of Noisy GD. As we claborate in
of Ô I
+ min
||"" [mc)
Vn
ne
'The [SST10) bound assumes |D'|| H. || = 2(1). The bounds of [Sha15] were originally stated for the constrained setting, but can easily be converted. More details in Appendix E.

---

Section 3.1, a straightforward application of uniform stability leads to sub- sub-optimal bounds and hence a new analysis is required. We note passing that this upper bound works for (unconstrained) DP-SCO with smooth (non- lipshhit) losses, which is of independent interest. For the high dimensional regime, we perform random projections of the data (specifically, the Lohnson-Lindennsrauuss transform) for dimensionality reduction, roughly reducing the problem to its low dimensional counterpart. We also
develop a lower bound for the excess risk under DP of 2 min
vāll=" !|
}).
We note that
[me)273
ne
non-privately a lower bound of γ (1
+ min {
d+|l==]1 11_ ||
vn []
is known on the excess population
vn
risk [SST10, Sha15]. We note that these private and non-privee non-private lower bounds imply that our bound is optimal up to factors of ||=|| (see Table I).
Lipschitz GLMs. For the Lipschitz case, we close a subtle but important gap in existing rates, In this setting, has been shown that one can characterize the excess risk in terms of the expected rank of the design matrix, rank, instead of d [SSTT20]. In this setting, the best known rate was
δ [w'l Vn
+ \/rank]w""I)) We show an improved rate of Ō [|w="| + min
[|="||
yrank]] w" me
This
Vnn
improves in the high privacy regime where 6 S rank v
In fact, the upper bound [Iw"1I <xz
for this
rate can be obtained with only minor adjustments to the regularization method of [JT14]. Our second contribution in this setting is extending the lower bound of [SSTT20] to hold for all values and rank E [n]. This in contrast the original lower bound which only holds for problem instances where ||""|| = rank and rank E [ne].
of ||w"|| ||
Model selection. As part of our methods, we develop a differentially private model selection approach which eliminates the need for a-priori knowledge of ||w*||- Although such methods are well established the non-private case, (see e.g. [SSBD14]), in the private case no such methods have been established. Our method, as in the non-private case, performs grid search over estimates of ||w"1| || and picks the best model based on the loss. However, the private setting we must account for the fact the the loss evaluation must be privatized. This is non-trivial in the non-Lipschitz smooth case as the loss at a point w may grow quadratically with [w].
Lower bounds for Non-Euclidean DP-SCO. Our lower bound construction generalizes to Non- Euclidean fp/ lg variants of DP-SCO with Lipschitz convex losses [BGN21]. Herein, we assume that the loss function -ipsschtt with respect to by norm, and radius of the constraint set bounded in Ep norm by Bp. For this setting, we give a lower bound of 52 Bp min (nc) ar excess empiriial/popuulation risk of any (potentially unconstrained) (c, DDP algorithm; Corollary 4 in Appendix B.4 for a formal statement and proof. For do and p 2 2, d S ne, this matches the best known upper bounds in [BGN21].
dir 13/2
on
Non-private settings. As bypouuus,ts, we give the following new results for the non-private setting. For details on the parameters used below we refer to Table 1.
1. We show that gradient descent, when run on convex non non-negative H smooth functions (not
necessarily GLMs), it achieves the optimal rate of O V H|ll IID*)) (see Corollary 1). This is done via an sterage-stability analysis of gradient descent. This result is interesting it also shows GD only needs n iterations, which is known not to work for non-smooth SCO [BFGT20, ACKL21, AKI.21].
vn
2. In Section D, we give procedure to boost the confidence of algorithms for risk minimization with convex non-negative H smooth functions (not necessarily GLMs). The standard boosting analysis based on Hoeffding's 's inequality does not give a bound with a linear dependence on the parameters ([]w*) || [|]e]], [[>][], and hence a tighter analysis is required.
1.3 Techniques
Upper bounds. We give two algorithms for both the smooth and Lispschitz cases. The first method simple and has two main steps. First, optimize the regularized empirical risk over the constraint set (w |||||| < B} for some B > ||w"|]. Then output perturbation of the regularized minimizer
3

---

with Gaussian noise (which is not requried to be in the constraint set). This method is akin to that of [JT14] with the modification that the regularized minimizer is constrained to ball. We elaborate on this key difference shortly.
The second method is based dimensionality reduction. We use smaller dimensional data-oblivious embeddings of the feature vectors. A linear JL transform suffices to give embeddings with the required properties. We then run constrained DP-SCO method (Noisy GD) in the embedded space, and use the transpose of the JL transform to get a d dimensional model. In this method, the embedding dimension required is roughly the threshold on dimension at which the rates switch from dimension dependent to independent bounds. We also remark that [N²UZ20] applied similar technique to provide dimension independent classification error guarantees for privately learning support vector machines under hard margin conditions.
We note that crucial part in all of these methods is the of constrained optimization subroutine, where the constraint set is a ball of radius ||w*] | This is in stark contrast to the Lipschitz case where existing methods such as those presented by [JT14, SSTT20] rely on the fact that projection is not required. In the smooth case however, constrained optimization helps ensure that the norm of the gradient roughly bounded by the diameter of the constraint set. We note that in the high dimensional regime, the property that the final output of the algorithm can have large norm is still crucial to the success of our algorithms.
Lower bounds. For our lower bounds in the smooth case we rely on the connection between stability and privacy. Specifically, we will utilize lemma from [CHI2] which bounds the accuracy of one-dimensional differentially private algorithms. We then combine this with packing arguments to obtain stronger lower bounds for high dimensional problems. For the Lipschitz case, we adapt the method of [SSTT20].
2 Preliminaries
In the following we detail several concepts needed for the presentation of this paper.
Risk minimization. Let X C Rd be the domain of features, and y S E be the domain of responses. A linear predictor is any w IRd. Let f: Rd (X' x)) - R be loss function. Given some unknown distribution ID over (X' ()))), we define the population loss L(w; D)
E
[f(w:; (I. y))]
(x,g)~
Given some dataset S E (X' x y)" drawn i.dd from D, the objective is to obtain @ E Rd which minimizes the excess population risk, L(w; D) min {L(w; D)). Given a population risk wERd
minimization problem, we will denote w* to be a minimum norm solution to this problem. We define the empirical risk as L(w; 1 Σ l(w; (I, v)). We define the following quantities (2,0)ES
for notational convenience: (riss(w) I[w;; D) I(w"; D), (w) (w; S) T(w": S), and Egen (w) = L(w; D) - I(w; S). We define BB to be the Euclidean ball of radius B on Rd.
Generalized linear models. We will more specifically be interested in the problem of learning generalized linear models, where there exists some function & : yx -> TR such that the loss function can be written as E(w; (ar, y)) @((w, 2²) .v). We define parameter bounds || = maxxex ||>||| and ||>|||2 = maxyey 1e(0, y) Note that for many common loss functions, the latter condition is the moral equivalent of assuming labels bounded by ||D']. For ease of notation, we write of @((w, x), y) as d ( ((w, .r)) and denote function dy : 2 = 0(2, y). We say that the loss function is G-Lipschitz GLM all y E y, Φy IR + is G-Lipschitz. We similarly define H-smooth GLM.
Differential privacy [DKM*06]. We restrict our investigation to the class of algorithms which minimize the excess population risk under the constraint of differential privacy. A randomized algorithm A is said to be (c, 6) differentially private (i.e., (c, 8DDD)) if for any pair of datasets S and S' differing in one point and any event & in the range of A it holds that
P[A(S) E 8] < e P(A(SS)) E E] + 6.
For our lower bounds, we will make use of the following Lemma from [CH12).

---

Lemma 1. Let Z be data domain and let S and S' be two datasets each in 2 that differ in at most A entries, and let A t - R be any (c, 6DD algorithm. For all T E RR if log(1/27) and 8 < J/6 (1-E))) then IE [[A(S)) - - 7 T
Finally, we introduce the Johnson-Lindenstaaanl (JL) transform perform random projections. Definition 1 ((a, JJL property). A distribution over matrices Rixd satisfies (a:, B)JJ property if for any u, Rd [1(bu, Uu) (u, e)] > a [/][u|]||l]] s B.
It is well known that several such "data-oblivious" (i.e. independent of u,v) distributions exist with = 1og(1/9) [Nel1]]. We note that JL property is typically described as approximation of norms (or distances), but it is easy to deduce the above dot product preservation property from it; for completeness we give this as Lemma 12. Finally, we use 4D to denote the push-forward measure of the distribution D under the map + (z. v) - (#z y). Similarly, given data set S {(x,, y)),. we define $ = { (1x, }
3 Smooth Non-negative GLMs
For smooth non-negative GLMs, we present new upper and lower bounds on the excess risk. For our upper bounds, we here assume that the algorithm is given access some upper bound ||w*]l, I|- that we denote by B. We later show in Section 5 how to obtain such rate without prior knowledge of [||""] ||. We also emphasize that the privacy of these algorithms holds regardless of whether or not |
B
3.1 Upper Bounds
Before presenting our algorithms, we highlight some key ideas underlying all our methods. A crucial property of non-negative smooth loss functions which allows ensure privacy, is the self-bounding property (e.g. Sec. 12.1.3 in [SSBD14)), which states that for an
bound sensitivity, and thus
H-smooth non-negative function f and dom((), [V)( S v /4Hf(w)
This property implies that the gradient grows most linearly with [|w|] More precisely, we have the following.
Lemma 2. Let be an H H-smooth non-negative GLM. Then for any w E BB and (x, y) E (X' x x) we have [[V,(w, ( (x, y))ll < 2 ||/||| [ + 2H 2H 2|
In order to leverage this property, all our algorithms in this setting utilize constrained optimization as a subroutine, where the constraint set is B3 and the Lipsehitz constant is ||>|| VH + BHB |2 This, in conjunction with the self- bounding property ensures reasonable bounds on sensitivity. turn, this allows us to ensure privacy without excessive levels of noise.
Finally, we note that our upper bounds for smooth GLMs distinguish low and high dimensional
regimes, transitioning at = min (( [BVHIXX)) 1 (ne)3.
3.1.1 Low dimensional regime
We start with the low dimensional setting where we use techniques developed for constrained DP- SCO for Lipschitz losses (not necessarily GLMs). Existing private algorithms for DP-SCO (e.g., [BFTGT19, FKT20J) lead to excess risk bounds that scale with n On the other hand, the optimal vn
non- non-private rate [SST10] scales with VHB which may indicate that the private rate implied by the Va known methods is sub-optimal. We show that this gap can be closed by novel analysis of private GD.
A standard proof of excess risk of (noisy) gradient descent for smooth convex functions is based on uniform stability [HRS15, BFTGT19). However, this still leads to sub-optimal rates. Hence we turn to average stability based analysis of GD, yielding the following result.
5

---

Theorem 1. Let f be non-negative, corvex, H- H-smooth loss function. bounded at zero by ||>1] Ler B, n > 0. no na2 T G = 2 | VH BHB. Then, for any e, > 0, Algorithm I invoked with
(c, ddifffereent private. Further, given a dataset S of no ii.d sumples from an unknown distribution DD the excess risk of ouiput of Algorithm is bounded as,
= = BB, T= x² SG (1)) (18
min
n²c
VT max (Vinyllova) 4H
V HBBD
GBVdlog (1/6)
ElErisk [)) 0o
Vn
n
We note that since H-smooth GLMs satisfy the Theorem condition with parameter H S H|| we obtain results for GLMs as direct corollary.
Non-private risk bound: As corollary (see Corollary 1 in Appendix A.3.I), with no privacy constraint, the above result (setting c <- oc and = 1) shows that gradient descent achieves the optimal excess risk bound, previously shown to be achievable by regularized ERM and one-pass SGD [SST10]. The lower bound, nu, simply means that the trivial solution "zero" has larger excess risk.
Proof Proofsketh sketch of Theorem 1. The privacy proof simply follows from [BST14] since the loss function G-Lipschitz in the constraint set. For utility, we first introduce two concepts used in the proof. Let S be dataset of i.id samples {(r,, y:) s((i) be the dataset where the i-th data point is replaced by an i..d. point (x', y'). Let A be an algorithm which takes dataset as input and outputs A(S). The average argument stability of A, denoted as cav-stab (A) is defined
(av-stab(AA) Es.i,("' wn) (|A(S)) - A
The average regret of gradient descent (Algorithm 1) with iterates {wi } is (r; = Σ [[[(w,; L(w;; 5)].
The key arguments are follows: we first bound the generalization error, or on-average stability, in terms of average argument stability and excess empirical risk (Lemma 4). We then bound average argument stability in terms of average regret (Lemma 5). Finally, in Lemma 6, we provide bounds on excess empirical risk and average regret of noisy gradient descent. Substituting these in the excess risk decomposition gives the claimed bound. The full proof with the above lemmas is deferred to Appendix A.3.1.
Algorithm 1 Noisy GD
Input: Dataset S, loss function f, constraint set W, steps T, learning rate noise scale o²2
wh
2: for = 1 to T - do
3: { E~ 2=00 II)
4: w++1 ITw w0 1 17 (f(,;; + ++))
where 11w is the Euclidean projection on to W
5: end for
Output: T = 1
3.1.2 High dimensional regime
In the high dimensional setting, we present two techniques.
JL method. In the JL method, Algorithm 2, we use data-oblivious JL map & to embed all feature vectors in dataset S to l dimensions. Let dataset ~ {( 21 H-) We then run projected Noisy GD method (Algorithm 1) on the loss with dataset and the diameter of the constraint set as 2B. Finally, we map the returned output w back to d dimensions using & to get @ & W. We note that no projection is performed on and thus the output may have large norm due to re-scaling induced by 4 T.

---

Algorithm 2 JL Method
Input: Dataset (111,11,,.. (In, Wn)}, loss function k. B, o²
Sample JL matrix 4 E Rexd
2: S {(021, 21) (222, v2...... (Px,, yn)}
3: to Noisy (D e, B 0²)
Output: +
Algorithm 3 Regularized Output Perturbation
Input: Dataset { 3/1; ..., (I', yn)) loss function X, B, 02
w- arg min { (w:; 5) [|wl(?) wEBg
2: { (0, o La)
Output: ã á +
Theorem 2. Let k O BVTT [X]] log (2n. 28)8//m
2/3
W
BB, T
o²
SG Ton(T////// (/8) ntte*
|DIII| |
1
? min max (V [
and no
Algorithm satisfies (c, 6)-
differential privacy. Given a dataset S of n 2 no i.d samples, of the output @ is bounded by
VHB P + (VHBIE*)
1DT) + (VHBBIZ1)
E Fiss 5Ō
(nE)
Proof sketch of Theorem From the JL property, with k: = O(log (n/6) /a2). w.h.p. norms of all feature vectors and w", well inner products between preserved upto tolerance (see Definition 1). The preservation of norms of feature vectors implies the gradient norms are preserved, and thus privacy guarantee of sub-routine, Algorithm 1, suffices establish DP. For the utility proof, from our analysis of Noisy GD (Theorem 1) and using the JL property, the excess risk w under 'D w.r.t the risk of &w" under 4D bounded as,
+
B B log (1/6)
VH X BJDI
( D) L( (.D) So
nc
From smoothness and JL property, the, "JL error" is:
ELL(Ow 4D) L(w"; ;D] <
The above is optimized for the value of k prescribed in Theorem 2, substituting which gives the claimed bound.
Constrained regularized ERM output perturbation. Our second technique is constrained regularized ERM with output perturbation (Algorithm 3). A similar technique for the Lipschitz case seen in [JT14), however note that the addition of the constraint set B is crucial in bounding the sensitivity in the smooth case.
HB x11
Then Algorithm 3 run with o² =0 (112)) (1? B3 (1(1)))))))))))))))))))))))))) log( (1/8)
Theorem 3. Let no =
DPP
X9n2e2
and X (IDI+HBIRRT)))
2/3
(log (13 satisfies (c, 8) diiffferrent privacy. Given
Biee
dataset S of 2 no i.i.d samples, the excess risk of its output @ is bounded as
n + (VTIBII1I) 4/3 ||2/|1// + [
VHB XII |II 1
E Erikk (0) (ō
(ne)2/3

---

We note that we can use the same technique in the low dimensional setting too, yielding rate of V +1VE
GBVd However, contrast to Theorem and 2, these results have an
uc
additional term (in both regimes). Thus, in the regime when || S VHB | the two upper bounds are of the same order.
12'
va
Proof sketch of Theorem 3. The privacy proof follows the Gaussian mechanism guarantee together with the fact that the εz-sensitivity of constrained regularized ERM is O ( [BE02]. For utility, we use the Rademacher complexity based result of [SSTI0] to bound the generalization error of w. The other term, error from noise, E[ (; D) L(w; D)] O(Ho² [|]] from smoothness of GLM. Combining these two and optimizing for gives the claimed result.
3.2 Lower Bounds
The proof technique for our lower bound relies on the connection between differential privacy and sensitivity shown in [CH12]. The underlying mechanisms behind the proof is also similar in nature to the method seen in [SU17]. We note that although we present the following theorem for empirical risk, reduction found in [BFTGT19] implies the following result holds for population risk as well (up to log factors).
Theorem 4. Let e C [1/n.1,, dataset S with empirical minimizer of norm such that the excess empirical risk of is lower bounded by 52 { min {1DDIP (B112(1)(/$(m)) pi van y
For any (DP algorithm, A, there exists
(me)a73
12
The problem instance used in the proof is the squared loss function. and thus this result holds additionally for the more specific case of linear regression. We also note this lower bound implies upper bound is optimal whenever B|| s ||']] which is commonly studied regime in linear regression [SST10, KL15J. We here provide proof sketch and defer the full proof to Appendix A.7.
Proof sketch of Theorem 4 Define F F(w; S) = = Eoewne x) - - Let d' < min {n, d}
and 6, [0, 1] be parameters be chosen later. For any consists of the union d' subdatasets, Si.... Sa given as follows. Set pm of the feature vectors in Sj as ||A' (the rescaled j j'th standard basis vector) and the rest as the the labels as |||| and pin (1 b) labels as , Let w" = arg min {F(w; {o)} be the 2d ERM minimizer of F-; S.). Following from Lemma 2 of [Sha15] we have that for any w e IRd that
==1]" define the dataset So which
d'
vector. Set pn (1 b) of
24
2
d'
F F(w; - F(w ;S)
Σ
10g
3
(1)
2d
We will now show lower bounds the per-coordinate error. Consider any and o' which differ only
at index j for some j E [d']. Note that the datasets Sg and So differ in A = 22 pm [[1 + 6) (1-b)) b)] =
pbn 4f
y and r' u | points. Let w 지사ा
[| (i.e. the j components of the empirical
minimizers for S and Sy respectively). Note that |uj - w
2|D ||
Setting d' = || 2/3
13l
2/3 B (
and
ensures < and thus Lemma I can be used to obtain
14/1/1/122/3
IV
[((())) - 10 (A(r))) - uz z 32 |
One can now show via a packing argument that
(122111))43110011333114
(E(F(A(So)) - F(w0))}
sup (=(1)
128(nc)2/3
The result then follows from setting p = min {1,
22/21211
8

---

4
Lipschitz GLMs
In the Lipschitz case, close subtle gap in existing rates. We recall that in this setting a more precise characterization in terms of the expected rank of the design matrix is possible (as opposed to using d). The best known upper bound Ö
[| || viakk assuming knowledge of || ||- This bound ne
was shown to be optimal when 2 rank/n and || ||= vrank [SSTT20].
We first show that in the high privacy regime where S rank/n, improved rate is possible. Specifically, we show that in this regime constrained regularized ERM with output perturbation and achieves the optimal rate. In fact, we note that the method of [JT14] (i.e. unconstrained regularized ERM with output perturbation), can obtain this rate when e = ((1) if the regularization parameter is set differently. We present the constrained version in order to leverage Rademacher complexity arguments and provide a slightly cleaner bound that holds for all 6 > 0.
Theorem 5. Algorithm 3 run with parameters o² || log(1/8) and X G(]x"]((lo((//8)) Byne
satisfies (c, S)-P.. Given a dataset of n. ii..d. samples DD its output has excess risk [Srisk (1)s))))] =
GB [ δ
GBI
V/
Vie
We also state and prove similar bound using the JL technique in Appendix B.
Next, we generalize the lower bound of [SSTT20] to show this new bound is optimal for all settings of B, rank, and e. We now show that modification of the lower bound present in [SSTT20] shows our upper bound tight. We note that their lower bound only held for problem instances where /rank]|
rank- ||"*|| 1| and e S rank/n. By contrast, the upper bound O rank and ||
holds for any values of
Theorem 6. Let G. |12'1 1I 0, e 1.2 and 5 S e. For any (e, DDD algorithm A, there exists a G-lipschitz GLM loss bounded at zero by ||a|| and a distribution DD with ||w*|| S B such that the output of A on S ~ Dn satisfies (Eleris
2 ( GBI min (1, - s/unkt
Vne. ne ))
All proofs for this section are deferred to Appendix B.
5 Adapting to ||=0||
Our method for privately adapting to ||w*|| || is given in Algorithm 4. We start by giving a high level overview and defining some necessary preliminaries. The algorithm works in the following manner. First we define number of "guesses" K for ||w"|], Bi.... Bx where B₁ = 2 V E [K]. Then given black box access to a DP optimization algorithm, A, Algorithm generates K candidate vectors wi...., WK using A, training set S1 E (X' x ///m/2, and the guesses BI.... Bk. We assume A satisfies the following accuracy assumption for some confidence parameter B > 0.
Assumption 1. There exists a function ERR g g+ such that for any IR+,, when- ever B > |w= |. w.p. least 1 B TK under the randomness of S₁ D and A it holds that
Erick (A(S,, B); D) S ERR(B).
After generating the candidate vectors, the goal is to pick guess with the smallest excess population risk in differentially private manner using a validation set S2. The following assumption on A allows us both to ensure the privacy of the model selection algorithm and verify that L(w,; S2) provides a tight estimate of L(w;; D).
Assumption 2. There exist a function A RR ET such that for any dataset S2 (X' 2/2n/2 and 0 PB(2,)) E S2 1I(ASS) (), A(D) < min {5, 13)
4K
Specifically, our strategy will be to use the Generalized Exponential Mechanism, GenExpMech, of (RS15] in conjunction with a penalized score function. Roughly, this score function penalty ensures the looser guarantees on the population loss estimate when B is large do not interfere with the loss estimates at smaller values of B. We provide the relevant details for Gen GepExphech in Appendix C.I. We now state result.

---

Algorithm 4 Private Grid Search
Input: Dataset S E (X' x))", grid parameter K E R, optimization algorithm: A: (XxJ))" Rd privacy parameters (c
1: Partition S into two disjoint sets, Si and S2, of size
2: who = 0
3: for j E [K] do
4: B3 2
5: wj = A(Si, BB)
6: Lj Llwyj S2) + A(B,) logg))
4 yl 2 (K/8)
a
"
7: end for
8: Set j" as the output of Menh run with privacy parameter 2. confidence parameter &
and sensitivity/score pairs (0, [] (A(B)), (A(Bk),
9: Output wg. wy.
Theorem 7. LetE: : Rd (X Y) be smooth non-negative loss function such that f(0, (I, v)) S
|D'| for any z.y E (X x V). Let e, 8,8 E [0, 1]. Let K > 0 satisfy ERR(2*) ) 2 || Let A be an (2 2K )DP algorithm satisfying Assumption 2. Then Algorithm is (c, DDD. Further, if A satisfies Assumption and S ~ D=/2 then Algorithm 4 outputs w s.t. with probability at least 1 B.
/aclaa log (4K/3) 52 max (1l"") 11, 13)
Erisk (w;; D) S min { (1D'1P ERR(2 max [1]w") "I, +
7
TLE
We note that develop generic confidence boosting approach obtain high probability guarantees from our previously described algorithms in Section 5, and thus obtaining algorithms which satisfy 1 is straightforward. We provide more details on how algorithms satisfy Assumption 2 in Appendix C.4. The following Theorem details the guarantees implied by this method for output perturbation with boosting (see Theorems 13,15). Full details are in Appendix C.3.
Theorem Let K,e, 5, 3 > and A be the algorithm formed by running 3 with boost- ing and privacy parameters e' K. 8' K Then there exists setting of K such that K + (1og maxx !!'I (1)])( | VH VH| 21
}) and Algorithm 4 with A and K is (e, 8)P and when given S ~ D", satisfies the following w.p. at least 1 - 3 (letting B = 2 max (1ww") 11,1)
(VHB* (211) 1/3 1|012/33 - (VHB** (1x1))
Erisk (w) = Ō min {INIP
(nc)2/3
VHB" || max (121) 11 + 1
| + H (1
YLE
Confidence Boosting: We give algorithm to boost the confidence of unconstrained, smooth DP-SCO (with possibly non-Lipschitz losses). We split the dataset S into m + chunks and run an (e, 6D-DP algorithm over the m chunks get m models, and then use Report Noisy Max mechanism to select model with approximately the least empirical risk. We show that this achieves the optimal
rate Ö [>BBIXX]])))] whereas the previous high probability result [SSTIO] had an additional
//
δ [V' Vn
term, which was also limited to only GLMs. The key idea is that non-ncgativity, convexity, smoothness and loss bounded at zero, all together enable strong bounds the variance of the loss, and consequently give stronger concentration bounds. The details are deferred to Appendix D.
Acknowledgments and Disclosure of Funding
RA and EU are supported, in part, by NSF BIGDATA award IIS- 1838183812 139 and NSF CAREER award IIS- 1943251. RB's and MM's work on this research is supported by NSF Award AF-1908281 and Google Faculty Research Award. CG's research is partially supported by INRIA Associate Teams project, FONDECYT 1210362 grant, and National Center for Artificial Intelligence CENIA 10017 Basal ANID. Part of this work was done while CG was at the University of Twente.
10

---

References
[ACKL21] Idan Amir, Yair Carmon, Tomer Koren, and Roi Livni. Never go full batch (in stochastic convex optimization). Advances in Neural Information Processing Systems, 34, 2021.
[AKL21]] Idan Amir, Tomer Koren, and Roi Livni. Sgd generalizes better than gd (and regulariza- tion doesn't help). In Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 Proceedings of Machine Learning Research, pages 63-92. PMLR, 15-19 Aug 2021.
[BBL03] Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning theory. In Summer school on machine learning, pages 169-207 Springer, 2003.
[BE02] Olivier Bousquet and André Elisseeff. Stability and generalization. J. Mach. Learn. Res., 2:499-526, March 2002.
[BFGT20] Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochas- tic gradient descent nonsmooth convex losses. Advances in Neural Information Processing Systems, 33, 2020.
[BFTGT19] Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[BGN21] Raef Bassily, Cristóbal Guzmán, and Anupama Nandi. Non-euclidean differentially private stochastic convex optimization. In Conference on Learning Theory, pages 474-499, PMLR, 2021.
[BLM13] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: nonasymptotic theory of independence. Oxford university press, 2013.
[BST14] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimiza- tion: Efficient algorithms and tight error bounds. In 20/4 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 464-473. IEEE, 2014.
[CH12] Kamalika Chaudhuri and Daniel J. Hsu. Convergence rates for differentially private statistical estimation. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 July 1, 2012. icml.cc / Omnipress, 2012.
(CWH20) Xiangyi Chen, Steven Z. Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspective. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 13773-13782. Curran Associates, Inc., 2020.
[CWZ21] T. Tony Cai, Yichen Wang. and Linjun Zhang. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. The Annals of Statistics, 49(5):2825 2850, 2021.
[DKM+06] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. EUROCRYPT, 2006.
[DR+ 14] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211-40)) 2014.
[FKT20] Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimiza- tion: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium Theory of Computing, pages 439 449, 2020.
[HRS15) Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. CoRR, 00s/1500.01240,2000

---

[JT14] Prateek Jain and Abhradeep Guha Thakurta. (near) dimension independent risk bounds for differentially private learning. In Eric Xing and Tony Jebara, editors, Proceedings of the s1 International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 476-484, Bejing, China, 22-24 Jun 2014. PMIR.
[KL15] Tomer Koren and Kfir Y Levy. Fast rates for exp-concave empirical risk minimization. In NIPS, pages 1477-1485, 2015.
[Nel1]] Jelani Nelson. Sketching and streaming high-dimensional vectors. PhD thesis, Mas- sachusetts Institute of Technology, 2011.
[NeUZ20] Huy Lå Nguyên, Jonathan Ullman, and Lydia Zakynthinou. Efficient Private Algorithms for Learning Proceedings of the 31st International Conference on Algorithmic Learning Theory, volume 117 of Proceedings of Machine Learning Research, pages 704-724. PMLR, 08 Feb-11 Feb 2020.
Halfspaces. In Aryeh Kontorovich and Gergely Neu, editors,
[NY83] A.S. Nemirovsky and E.R. Yudin. Problem Complexity and Method Efficiency in Optimization. A Inter--ntersccience publication. Wiley, 1983.
[RS15] Sofya Raskhodnikova and Adam D. Smith. Efficient lipschitz extensions for high-dimensional graph statisties and node private degree distributions. CoRR, abs/1504.07912,
2015. 2015.
[Sha15] Ohad Shamir. The sample complexity of learning linear predictors with the squared loss. J. Mach. Learn. Res., 16:3475-3486, 2015.
[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[SSTI0] Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. In J. Lafferty, C. Williams, J. Shawe-Taylor R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23, Curran Associates. Inc., 2010
[SSTT20] Shuang Song, Thomas Steinke, Om Thakkar, and Abhradeep Thakurta. Characteriz- ing private clipped gradient descent on convex generalized linear problems. CoRR, abs/2006.06783,, 2020.
[SU17] Thomas Steinke and Jonathan Ullman. Tight lower bounds for differentially private selection. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), pages 552-563. IEEE, 2017.
[Ver18] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[VGNA20] Mariia Vladimirova, Stéphane Girard, Hien Nguyen, and Julyan Arbel. Sub-weibull distributions: Generalizing sub- -aabsaa and sub-exponential properties heavier tailed distributions. Stat, 91))e331, 2020.
[Wan18] Yu-Xiang Wang. Revisiting differentially private linear regression: optimal and adap- tive prediction & estimation in unbounded domain. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Mhirty-Fourth Conference Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 93-103. AUAI Press, 2018
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work
largely theoretical. Moreover, there are no foreseeable negative societal impacts for our proposed algorithms.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "y4IhkKh6iBj", "reviewer": "Reviewer_VnwD", "summary": "This paper revisits DP GLM and it provides a thorough theoretical analysis about different cases. Both upper and lower bounds are provided.\n\nBased on their DP definition, it is a global DP model with $(\\epsilon, \\delta)$ as privacy parameters i.e., approximated DP . All the key results seem to have high probability guarantees. I am wondering if it is possible to extend the result to pure-DP, i.e., the $(\\epsilon, 0)$ one?\n", "questions": "See Summary", "limitations": "See Summary", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: revisit the DP GLMs; thorough analysis are provided. \nWeakness: n/a", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "9mmhF_P3uj", "reviewer": "Reviewer_pgk1", "summary": "The paper studies the problem of differentially private generalized linear models (GLM) with convex losses. The main contribution of the paper are new upper bounds on the rate for smooth and non-negative functions that improve over existing bounds. The authors also prove lower bounds which show that these bounds are nearly-optimal up to factors that depend on the norm of the optimal solution. The authors also consider the setting of Lipschitz functions and show that their algorithms achieve the optimal rate up to log factors.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe paper studies an interesting and important problem and improves over the previously known rates for this problem under various settings.\n\nWeaknesses\n1.\tThe rates for the smooth case depend on the dimension d (not the rank as in the Lipschitz case). While the authors show tight lower bounds up to factors that depend on |w*|, this lower bound is probably obtained by taking rank=d and therefore the upper bound may not have tight dependence on the rank. It is important to clarify this in the paper.\n2.\tOne minor weakness is that all the algorithms in the paper are somewhat standard as similar algorithms have been used in the literature. However, the obtained results and some of the techniques are interesting (such as using average stability instead of uniform stability to improve the rates), therefore I don’t consider this to be a real limitation.\n3.\tThe rates obtained for the smooth case are very similar to the rates for DP-SCO in ell_1 geometry [AFKT21, BGN21] which also have a phase transition. I’m not sure if there is any fundamental connection here but it may be useful to explore this a little and comment on this.\n4.\tWhy didn’t the authors use the algorithms from “private selection from private candidates” [LT18] for adaptivity to |w*|? This will only require to privatize the score functions (as [LT18] assumes a non-private one) but may be simpler than redoing everything from scratch.\nMore minor comments:\n1.\tText in table 1 is too small and hard to read\n2.\tAlgorithm 1: gradient symbol is missing in line 4\n\nReferences:\n[AFKT21] Private Stochastic Convex Optimization: Optimal Rates in ℓ1 Geometry\n[BGN21] Non-Euclidean Differentially Private Stochastic Convex Optimization\n[LT18] Private selection from private candidates\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "807ukJGATOk", "reviewer": "Reviewer_qYvP", "summary": "This paper studies the sample complexity of learning differentially private generalized linear models and proposes upper and lower bounds for this problem. For smooth non-negative GLMs, their rates exhibit an interesting learning phenomenon in terms of the privacy budget and dimension. For high privacy budget (low-dimension), a novel analysis of noisy GD is shown to achieve the upper bound. For  low-privacy regime, the upper bound is achieved by an algorithm that uses random projection to derive low-dimensional projections of the data. When the GLM loss function is Lipschitz, they close an open gap in literature in terms of lower and upper bound. ", "questions": "--\nAre projection steps necessary for proof? Alternatively, did the authors consider a Frank-Wolfe version of Algorithm 1/2? If so, are the guarantees very weak? \n\n-- \nHave the authors tried to consider an adaptive grid search algorithm for Algorithm~4? ", "limitations": "--\nComputational implementation seems limited, thereby limiting applicability.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "--\nStrength:\nThe paper proposes rate-optimal differentially-private algorithms for the learning GLM loss functions. To my knowledge, this is much harder than regression because of the curvature of the loss function. The results presented use low-magnitude noise and applicable to a wide variety of loss-function curvature that occur in practice. From a purely technical perspective, some of the intermediate results are of independent interest. Results for several applications of interest are also presented. \n\n--\nWeakness: \nA major limitation of Algorithm 1 and 2 is its reliance on projection-based techniques for restricting the iterate to the constraint set. As is well-established this is very difficult to implement in practice and is computationally burdensome. Similarly, Algorithm 4 is a static gird search, which could be made adaptive to the dataset. \n\n Very strong theoretical results, limited computational applicability.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept