Automatic Clipping: Differentially Private Deep Learning Made Easy and Stronger
Authors: Zhiqi Bu
Yu-Xiang Wang
Sheng Zha
George Karypis
Venue: NeurIPS 2022 Submitted (2022)
Keywords: deep learning
differential privacy
per-sample gradient clipping
convergence

Abstract: Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping norm $R$, however, is shown to be vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called AutoClipping, that eliminates the need to tune $R$ for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB and many others.
The automatic variants are as private and computationally efficient as existing DP optimizers, but require no DP-specific hyperparameters and thus make DP training as amenable as the standard non-private training. We give a rigorous convergence analysis of automatic DP-SGD in the non-convex setting, which shows that it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric noise assumption of the per-sample gradients. We also demonstrate on various language and vision tasks that automatic clipping outperforms or matches the state-of-the-art, and can be easily employed with minimal changes to existing codebases.

Automatic Clipping: Differentially Private Deep
Learning Made Easier and Stronger
Anonymous Author(s)
Affiliation
Address
email
Abstract
Per-example gradient clipping is key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping threshold R. however, is shown be vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called AutoClipping, that eliminates the need tune R for any DP optimizers, including DP-SGD, DP-Adam, DP- LAMB and many others. The automatic variants are as private and computationally efficient as existing DP optimizers, but require no DP-specific hyperparameters and thus make DP training as amenable as the standard non-priv non-private training. We give rigorous convergence analysis of automatic DP-SGD in the non-convex setting, which shows that it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric noise assumption of the per-sample gradi- ents. We also demonstrate on various language and vision tasks that automatic clipping outperforms matches the state-of-the-art, and can be easily employed with minimal changes to existing codebases.
2
10
11
12
13
14
Introduction
15
16 Deep learning has achieved impressive progress a wide range of computer vision and natural 17 language processing tasks. These successes are made available, in part, by the collection of large datasets, sometimes containing sensitive private information of individual data points (e.g.. chest sean images, DNA sequences). Prior works have illustrated that deep learning models pose severe privacy
18
20 risks to individual subjects the training data and are susceptible to various practical attacks. For example, machine learning services such as Google Prediction API and Amazon Machine Learning 22 can leak membership information from the purchase records [58]; if one feeds the GPT2 language model with some specific prefix, the model will autocomplete texts that contain the full name, phone number, email address, etc., from the training data that memorizes [11].
21
24
25 Differential privacy (DP) [21, 23, 22] is formal definition of privacy that has been shown to prevent
26 the aforementioned privacy risks in deep learning [1]. On high level, the key difference between the
27 DP deep learning and the regular one is whether the gradient is privately released. In other words,
28 while the standard optimizers update on the summed gradient E, Ji. and DP optimizers update on
the private gradient:
private gradient
DP Optimizer( (g: [=1) = Optimizer
9i Clip(1lg: R1) +R
(1.1)
Standard Optimizer( (g: ( Optimizer( Σ gi)
(1.2)
30 Here 9: E dd is the per-sample gradient of loss 4, /V the standard normal random variable, a is the noise multiplier, and R is the clipping threshold. The clipping function Clip Rd - R is defined 32 such that l|g, Clip(gn; R) < R. For instance, the DP-SGD in [1] on batch B, is
Öl; min (R/
dw, 1.1) + (oR--N((II) (1.3)
Ø1,
DP-SGD Abadi
W
wi
1
Dw.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

In comparison to the regular training (1.2), two additional Dp-specffie hyperparameters R and a need to be determined in DP learning (I.1). On the one hand, setting the noise multiplier or is easy and can be derived analytically prior to the training. Whenever the privacy budget (e, 6) is determined, one can apply off-the-shelf privacy accounting tools in Section 2.1 to determine o, based on the subsampling probability (e.g. expected batch size over sample size) and the number of iterations T:
privacy_accountante p, T; =
33
On the other hand, the choice of clipping threshold R is crucial to the performance of DP models, yet the hyperparameter tuning is much labor-intensive. Recent advances of DP deep learning on ImageNet [34] and on E2E datasets [37], using & and GPT2 respectively, illustrate that the performance is very sensitive to R. We have reproduced their results in Figure 1. Observe that on ImageNet, 8 8 can drop from the highest 45% accuracy to 31% if R is chosen 2 times larger, and to 0.1% if R is chosen 4 times larger. Similar drastic drop can also be observed in [34, Figure 3] even if the noise multiplier o 0. Unlike the noise multiplier a,, the clipping threshold I cannot be inferred from the privacy budget (e, 6) and have to be tuned. Consequently, DP training necessarily requires 2D grid search for (R, n). like the lower plot of Figure 1, whereas the regular training only requires an easy ID grid search for 17. Even worse, the difficulty of tuning a per-layer clipping threshold vector [43], i.c. one clipping threshold for one layer, may increase exponentially as the number of layers increases.
34
35
36
37
38
39
40
41
42
43
44
To save the effort of tuning R. previous researches have proposed different approaches. In [3, 51, 25]. researchers advocate to use data-adaptive information to select R, such as a specified quantile of the gradient norm distribution. These adaptive clipping methods can be little ad-hoc: they often replace the the need to tune R by the need to tune one or more new hyperparameters, e.g. the quantile use and the ratio split the privacy budget between the quantile decision and the gradient perturbation. Another approach used by the practitioners is to replace an expensive 2D grid search 51 by multiple cheaper ID grid searches. For example, the researchers propose, in [34, Section 3.3] to fine-tune with non-DP SGD, fix and sweep over various values of the clipping threshold R with SGDSGD, then further fix R and do one more grid search on 7). However, tuning R formally in a
46
48
49
50
52
58
54 data-dependen way (e.g. through cross- pross-validation) introduces additional privacy loss [48], and most
55
existing empirical work does not privately conduct hyperparameter tuning.
56 We take a completely different route by proposing a new clipping principle that removes R, instead of coming up with methods to find the appropriate R. We term our method automatic clipping (AutoClipping) and we term the versions of DP optimizers using it as automatic DP optimizers.
57
58
59
We summarize contributions follows.
60 1. We propose the automatic clipping in (4.1) that expunge the clipping threshold from general DP optimizers, allowing DP learning to be as amenable as regular learning.
61
62
2. We show that automatic DP optimizers are as private and efficient as existing DP optimizers.
63
3. We show that automatic DP-SGD converges in the non-convex setting, at the same asymptotic convergence rate the standard SGD. Our theoretical analysis successfully explains the training behaviors in previous empirical works.
65
86
4. We demonstrate the superiority of automatic clipping on a variety of vision and language tasks, especially with large models including ResNet, RoBERTa and GPT2.
87
88
5. In Appendix K, we include simple code snippets that demonstrate how easy it is to switch from Abadi's clipping our automatic clipping in popular codebases, that implement DP optimizers for deep learning, e.g. Opacus and OblAX.
69
70
2 Preliminaries
71
72
2.1 Differential Privacy
73 We consider the (c, 8)-DP in Definition 2.1, where smaller (c, means stronger privacy guarantee.
74 Definition 2.1 ([22]). A randomized algorithm AM is (e, -ddfferrrenttalllyy private (DP) if for any two neighboring' datasets S, S', and for any event E,
(SM(S) E E] < (M (S") E E) 5
(2.1)
's' is neighbor of S if one obtain S' by adding or removing one data point from S.
2

---

78 In words, DP restricts the influence of an arbitrary sample, so that the information contributed by such sample is limited and less vulnerable to privacy attacks. In deep learning, DP is generally achieved by applying the subsampled Gaussian mechanism to privatize the minibatch gradients during training.
77
78
As illustrated in Equation (1.1), the subsampled Gaussian mechanism involves (1) Sampling minibatch by including each data point iid with probability (2) per-sample gradient clipping to bound the l2 norm sensitivity at R and (3) adding independent Gaussian noise proportional to the sensitivity R and o, which is derived from the privacy loss e. This can be realized by leveraging a variety of modern privacy accounting tools, such as those based on Renyi DP (or moments accountant)
79
80
81
82
83
84
[1, 45, 63], Privacy Loss distribution (Fourier accountants) [33, 27, 70], or Gaussian DP [16, 7].
85
2.2
Differentially Private optimizers with general clipping operations
86
Privately released stochastic gradients (through the Gaussian mechanism) can be used to instantiate various off-the-shelff optimizers, which gives rise to DP-SGD in (1.3), DP-HeavyBall, Dd-AdaGrad, Ad-m,,m, De-FedAvg, DP-FedSGD [43], etc. To improve the performance of DP optimizers, previous researches can be classified into two categories.
88
90
The first category, where the majority of researches lie in, works with Abadi's clipping and focuses on better design of R. To name a few examples, one can adaptively design Rg for each iteration t [3, 51, 25], or design the per-layer clipping threshold vector R. E Re for L layers [1, 43] so as to apply a different clipping threshold for each layer.
91
92
93
Much fewer works fall into the second category that proposes new clipping method. In fact, any
85 function Clip: : Rd - R satisfying [[Clip(g) -gil < R can serve as a valid clipping function besides [[[]g]] < 2) R//Z Z correct the bias in the private gradient and alleviate the mis-cali mis-calibration issue of DP classifiers. Our automatic clipping also belongs to this category. We note that different clipping methods work orthogonally to optimizers, network architectures and gradient norm computation (see Section 7).
96 Abadi's. For instance, the global clipping [9] proposes Clip (g)
98
99
100
3
Motivation
101
3.1 Small clipping threshold works best
13-1-60.26
59.58
60.01
57.57
35.44
60
102
One intriguing observation that we can make about the re- cent studies on DP learning with large models is that the -tate-of-the-art (SOTA) results are often achieved with very
50
Leaminn rate
20 922
50.48
50.31
50.48
49.25
34.47
104
105
small clipping threshold R. This observation is consistent in both vision and language tasks. In [37], GPT2 (over 800 million parameters) and RoBERTa models (over 400 mil- lions parameters) achieve the best results under DP on QNLI, MNLI SST-2, QQP, E2E, and DART datasets, with each per- sample gradient clipped to length It = 0.1. In [34, 14, 44], ResNets and Vision Transformers achieve the best DP re-
30-921-
53.17
35.38
35.13
32.90
29.50
106
10-1p*
28.74
29.55
29.61
29.62
27.07
107
108
10
92*
9.69
9.69
9.68
9.05
5.79
109
0.
0.112
110
Clipping norm
111
112
sults on ImageNet with R = 1; in [60]. the best DP results
0110!
43.0
2.0
0.1
0.1
0.1
113
on CIFARIO use R = 0.1 with ResNeXt-29 and SimCLRv2 [12]. The effectiveness of small clipping threshold together with proper learning rate is depicted in Figure
Learning rate
8-10"
45.0
0.1
0.1
114
115
0
2.1
22.0
45.0
11.0
0.09
20
116
Intuitively, smaller R implies that the Abadi s clipping 13.1)
0.29
2.2
22.0
24.0
0.11
15
117
happens, which means min (R/ [R/llg:ll. 1) = R/ /| |l. Given that the clipping threshold It is so small compared to the number of parameters in large neural networks, and that strong DP is guaranteed when the number of training iter- ations is small (i.e. ||9.|| has not converged to small values yet). we expect and empirically observe that the clipping happens on a large proportion of per-sample gradients at all iterations. For instance, we find in the GPT2 genera- tion experiments in [37] that 100% of per-sample gradi- ents are clipped at all iterations; in classification tasks such
30
118
0.25
0.44
119
19
19
19
120
Clipping norm
121
Figure 1: Ablation study of clip- ping threshold and learning rate that achieves SOTA results. Upper: BLEU score of GPT2 on E2E dataset, adapted from [37], trained with DP-Adam W. Lower: Test accuracy of ResNet18 on ImageNet dataset, adapted from [34], trained with DP-SGD with momen- tum.
122
123
124
125
126
as QOP/ONLI/MNLI the percentage of clipping is about
127
20 ~ 60% on average (more details in Appendix H.
3

---

129 3.2
Per-sample gradient normalization as new clipping: similarity viewpoint In the small clipping threshold regime, we can approximately view
130
C (g:: R) = min (R/ = R/ =: CliPAUTO (g R)
(3.1)
131
and thus derive a novel private gradient Σ, R 2 + oR- (O, Here AUTO-V stands for the g:ll vanilla automatic clipping, which essentially performs the gradient normalization on each per-sample gradient. As a specific example, can write the R-dependent automatic DP-SGD
132
133
04
Ol, dw.
R-dependent DP-SGDAUTO.V 01+11 = =w;-- 77
R
/1l Sw.
|| o ((O,I) (3.2)
NEB.
134
We may view our AUTO-V clipping as to maximize the dot-product similarity between the clipped gradient and the regular gradient, commonly used similarity measure, e.g. in the attention block in transformers [61]. Suppose we want
135
136
max C,
(E
s.t. 0 0 S R/
137
Note that the constraint is a sufficient condition for clipping, as discussed in Section 2.2. It is not hard that the optimal clipping factor is
138
R R
if (g: E, 9j) > if (f. 2, 91) 0
C; 1
139 If the per-sample gradients are indeed concentrated the sense Vi, (g., Σ, g2) 2 0, then AUTO-V the optimal per-sample gradient clipping. We compare with Abadi's clipping in Figure 2, where the dot-product similarity is significantly magnified by our AUTO-V clipping.
140
Waen
AUTO-V cloptng
dipping Abadi cippiig
Abadi capping
Count
5080
213
15
1300 a05o 1 www.eutdaaaaaooooo wot.001110000.. Jutiiv
itoration
Figure 2: RoBERTa-baas base with DP-Adam (c 3) on SST2 dataset, as in Section 6.2.
182
3.3
Stability constant breaks scale-invariance and remains stationary
One potential drawback of AUTO-V clipping is that all gradients lose their magnitudes information completely, since l1g:
144
RI R, Vi. This scale- invariance in AUTO-V and partially in Abadi's clipping (when | || > Ri leads to the "lazy region" issue: the parameters will not be updated by DP-GD even if the true gradients are non-zero. In Figure 3, illustrate in logistic regression' that AUTO-V and Abadi's is clip-
6.02-
146
-0.002
Handand -heet gnal (1ua))
148
Hoper 20mm
149
150
151
ping have zero clipped gradient for the trainable parameter
Figure Gradient (scalar) each 0.
152
0 E 2- 2], as the per-sample gradients from two classes cancel each other.
Another benefit of y is to remain stationary as g: 0. i.e making the clipped gradient C.g. 9:/7 small rather than having a magnitude R AUTO-V. We elaborate this point in Section 4.3.
154
155
To preserve the magnitude information and thus escape the lazy region, we propose the AUTO-S clipping, with a positive stability constant 7:
156
Cl AiiPAUTOS(s: (g;; R) = R/(1g::1 +
(3.3)
157 We visualize in Figure 4 that AUTO-S allows larger per-sample gradients to have larger magnitudes after the clipping, while still allowing smaller gradients to vanish after "clipping". This is critical in 159 our convergence analysis and allows OP-SGDAUTOS (but not IP-SGDAUTO-V to converge to zero gradient norms in Section 5.
'The settings are in Appendix F, where the lazy region issues also emerge in the mean estimation problem. We note that the lazy region is also discussed [13, Example 2].

---

4
Automatic DP Training
162
One may wonder why our clipping (3.3) automatic at all, if the hyperparameter R is still present and there is an additional parameter to choose. It turns out that any constant choice of R > 0 equivalent choosing
163
184
1, and common deep learning optimizers are insensitive the choice of y (e.g. for any > O, we show that the gradient norm converges to zero at the same asymptotic
185
186
rate Theorem 4; see also the ablation study in Figure 14). Consequently, we set 7 = 0.01 as the default. Specifically, let us redefine the R-independent clipping function:
CLiPAUTO-S(9.) 1/( +?).
(4.1)
168
With this clipping, we can design automatic DP optimizers similar to (1.1):
Automatic DP Optimizer( (g. |nn)
Optimizer(g:)
9t.i
where ÿr
+ a 10)
(4.2)
g +y
iEB,
189
Clearly, the new private gradient 9e from our automatic clipping is R-independent, in contrast to the one used in (1.1). A concrete example (in the case of y =0) that comparable to (3.2) will be
170
R-independent DP-SGDAUTO.Y : W1+1 = wi
Σ
04
04.
to N(. (4.3)
Sw.
dw
iEB
171
Leveraging the private gradient 9: in (4.2), we can train DP neural networks without tuning DP- specific hyperparamters I and o,, demonstrated in Algorithm 1.
Algorithm 1 Automatic Deep Learning with DP
Parameters: initial weights wo, learning rate y)t, sampling probability p. number iterations T.
1: Find a such that Accoountami (6, o, p. T) < from any privacy accountant.
173 2: for iteration t 1,... do
3: Sample a batch B, by including each data point iid with probability p [BacchSiezz = pn).
4: Apply automatic clipping per-sample gradients {gi B, : 9: gg/((l9g) 1lllll12221 + 0.01).
5: Add Gaussian noise to the sum of clipped gradients: 9 = >, 9: + (O,
6: Update w, by any optimizer the private gradient 9 with learning rate vjc-
We will elaborate two distinct reasons in each section for the following statement:
DP Optimizer Abadi Ro R-dependent DP Optimizer AUTO = R-independent DP Optimizer
174
which reduces the hyperparameter tuning of DP training to that of the regular training, i.c. only on learning rate, weight decay, etc. The significant save in the tuning effort illustrated Figure 15.
175
Non-adaptive optimizer couples clipping threshold with learning rate With R-dependent automatic clipping, DP-SGD becomes
4.1
R 1 +y
Σ 9 BEB.
OR -RR .R..))) =w; Rg...
w++11 we
177
We can view Neffective = R as whole: increasing R has the same effect as increasing 7, which explains the diagonal pattern in Figure 1(lower plot) where DP-SGD Abadi is applied with small clipping threshold We extend to general non-adaptive optimizers in Theorem with proof in Appendix B.
178
179
Theorem 1. Non-adaptive R-dependent automatic DP optimizers (including SGD, Heavyball 52] and NAG[47J), with learning rate 1) and weight decay X. is equivalent to R-independent automatic DP optimizers, with learning rate T/ nR and weight decay X X/R.
182
When we further consider weight decay automatic clipping (included in Theorem 1), increasing R is no longer equivalent to increasing 7), as y also couples with the weight decay constant X.
"This coupling of y and It is also partially observed in [14] through reparameterization trick of Abadi's clipping. Unlike AUTO-S/V, their coupling is not strict (c.g. doubling not equivalent to doubling n in their Figure 8, thus necessitating tuning both (n, R)). and the relationship weight decay not discussed.
5

---

184 4.2 Adaptive optimizer can be insensitive to clipping threshold
Adaptive automatic DP optimizers are different than the non-adaptive ones, as the clipping threshold cancels out instead of being coupled with learning rate. To see this, we scrutinize DP-Adam (which is similar to AP-AAamAuTo-v) in Figure (uppee plot), where columns to the left are almost identical. Further evidence is observed in [44, Table 5] that shrinking R. has zero effect on LAMB. We now give a simple explanation using AdaGrad [19]:
9t VG.
W1++1 wt "1
where g: - E; It. the gradient sum and G. - - g is sum of gradient square by Hadamard product over the past iterations. In R-dependent DP- DP-AdaGradduuoo the private gradient is Rý. in place of the standard gradient sum gt, and G. R²² Er 92:
Rg:
g:
W w. 11 VG.
WT -7
V Er (0-)²
185
We generalize to the general adaptive optimizers in Theorem 2, with proof in Appendix B.2. Theorem 2. Adaptive R-dependent automatic DP optimizers (including AdaGrad[19] AdaDelta[69), AdaMax/Addam311 J. [Adam[17], RAdam[39] LARS]65], LAMB[66]), with learning rate and weight decay X is equivalent to (B-independent automatic DP optimizers with learning rate n and weight decay = A/R. With decoupled weight decay[42], R-dependent automatic DP-Adam W is equivalent to R-independent automatic DP-AdamW with the same and X.
187
189
190
191
Similarly. we demonstrate the automatic DP optimizers with per-layer clipping style in Appendix B.3.
182
4.3
Automatic clipping guarantees the same level of privacy while maximizes utility
193
In Theorem 3 (proved in Appendix A), we show that the new private gradient 9: in (4.2) has the same level of privacy guarantee as the existing one in (1.1), since the global sensitivity remains the same (see Figure 4).
Norm of clipped gradient
194
196
We note that as long as y > O, the magnitude informa- tion of per-sample gradients preserved by AUTO- || + |Cag.ll |C |l. whereas this can be violated in both the AUTO-V and Abadi's clipping (as depicted by the
197
AUTO-S, y 0.001
196
AUTO-S, y- 2.01
S. in the sense that ||
AUTO.S. y=
y-1
200
0.00
025
0.58
3000
125
2.30
175
2.00
||lll
202
flat curve in Figure 4 when ||9.|| > 1).
Figure 4: Gradient norms before and after being clipped by different methods R 1.
203
Additionally, note that when is small, almost all data
204
Points "max out" the signal relative to the amount of noise we add. To say differently, for the same amount of noise, AUTO-S with small y allows more signal to be pushed through a differentially private channel. Towards the end of the training, i.e., the limit when ||g: -> 0 for all i, then we have E: Tg. + Σ, 9i- In words, the clipped gradients become closer to the standard SGD, thus do not suffer from the instability of AUTO-V.
206
207
209
Theorem 3. Under the noise multiplier o, number of iterations T, subsampling probability B/n, DP optimizers using AUTO-V AUTO-S clipping satisfy (EAccomntan (6, T), where Accommmnnt is any valid privacy accountant for DP-SGD under Abadi's 's clipping.
211
Convergence analysis of DP-SGD with automatic clipping
212
213
5.1
Convergence theory of DP-SGD to stationary points
214
We highlight that automatic clipping can be more amenable to analysis than Abadi's clipping in [13], since no longer need to decide whether each per-sample gradient is clipped.
216
To analyze the convergence of automatic DP-SGD (4.2) in the non-convex setting, we follow the standard assumptions in the SGD literature [24, 2, 6], with one additional symmetry assumption on the gradient noise.
217
218
219
Assumption 5.1 (Lower bound of loss). For all w and some constant L., we have L(w) 2 L.. Assumption 5.2 (Smoothness). Let g(w) denote the gradient of the objective L(w). Then Vw, there is an non-negative constant L. such that
221
L w v 1/
L(v) - [[(w) g(w) T (v-
w)] s 2
(5.1)

---

Assumption 5.3 (Gradient noise). The per-sample gradient noise gr.; 9: is i.idd from some ditribution such that
Rigr g:) = 0, Rlj.,, 9:11² s
and Jr.; is centrally symmetric' about Jr in distribution:
g. gt 9: gt.i-
222 We show in Theorem 4 that DP-SGD with AUTO-S clipping allows the true gradient norm to converge
223 to zero, but not so with AUTO-V V clipping. We leave the proof in Appendix C.I.
Theorem 4. Under Assumption 5.1, 5.2, 5.3, running DP-SGD with automatic clipping for T
iterations and setting the learning rate 1i x 1/VT give
min B(1lool)) 5 ( L.)), L. (1 (1+
o²d :5.77
min E + F ( ;,, (5.2) r20
OSIST
B
226
Here represents the first argument of and increasing and positive. As T - co, we have O0T- 1/1) for AUTO-S, the same rate as the standard SGD given in Theorem 9,
min; R( O
227
Upppee bf oppoulilt mppctjiitt
Upper Gouine of mpperttalll
Unyor www. of POneA1NNN
Rxco
0R555 AUTO-5
5GD
Ranations?
Figure 5: Leftt DP-SGD with AUTO-V clipping. Middle: DP-SGD with AUTO-S clipping. Right: Log-log plot of convergence rate in comparison to standard SGD, Here & = 25, 7 0.01, and the O(1/VT) term is set to 10 for DP-SGD and for standard SGD.
Remark 5.4. In Theorem 4, the upper bound takes implicit form of E(: because it is
lower envelope of functions & + F(:;r, 7) over all possible r O, whose forms are detailed in Theorem 6. Notice that & results only from the clipping operation, not from the noise addition.
229
230
231
Remark 5.5. We show in Theorem and demonstrate Figure 5 that the upper bound (5.2) is always larger than & with AUTO-V (ry = 0), and can only be reduced to zero with AUTO-S (y > 0). We provide real data evidence Figure 13 that strictly positive y reduces the gradient norm significantly.
232
233
234
5.2
Analysis of factors affecting the convergence
235
We now analyze the many factors that affect the convergence in Theorem 4, from a unified viewpoint of both the convergence and the privacy.
236
We start with the stability constant y and the learning rate The both only affect the convergence not the privacy. We empirically observe in Figure that small y benefits the convergence at initial iterations (when the privacy guarantee is strong) but larger y converges faster asymptotically. For ne, the optimal is in fact the miminizer of the hyperbola in (C.4), that is unique and tunable.
238
240
Next, we focus on the hyperparameters that affect both convergence and privacy: the batch size B, the noise multiplier a, and the number iterations T. These hyperparameters have to be considered along the privacy-accuracy tradeoff, not just from a convergence perspective.
242
244
Recall that given fixed privacy budget (e, 8), we rely on modern privacy accountant for computing the appropriate combinations of parameter a, T, B. The exact expression of the bound a function of 5) is somewhat messy. For this reason, we illustrate our analysis in terms of the surrogate parameter
245
246
for GDGDP [16]. [7] showed that DP-SGD privacy guarantee asymptotically converges to μ-GDP
as T + 20) with = a /T(e 1). GDGP implies (c, DD with 41² + Hv 2 log(1/))
248
249
We can alternatively leverage PtCCP [10] for similar conclusions, using in place of 11² in (5.3).
'The symmetry assumption has been empirically verified in [13, Figure 3J. For theoretical analysis, can be
extended to mirror symmetry about the hyperplane normal to gr. that is {v : g v = 0). "More precisely, H-GDP is equivalent to an entire family of (c,8)-DP for any > 0 and 6 = 2(u/2 c/u) - e @(-u/2 n/2 c/μ) where & is the standard Gaussian CDF.
7

---

250
Theorem 5. Under Assumption 5.1, 5.2, 5.3, fixing the asymptotic ulc, 6DDD parameter, running DP-SGD with automatic clipping for T iterations and setting the learning rate L L/vT give
(* (2//co--0) (C-(oo-C)) C.) (
d
B B(1.:: 5 OgiST
min
+0 12n2
:5,7
(5.3)
B2T
252
To show that our analysis matches the training behaviors observed in SOTA empirical work [37, 34, 14, 60, 44, 68), minimize the first argument of g in (5.3), denoted X X(B, T, p,, d, I,, fo)-
253
254
1. [Train longer with larger noise] Fixing the expected batch size B, we see that X is decreasing in T. Hence larger T and consequently larger are preferred.
256
2. [Larger batch size helps] Fixing number of iterations T or epochs E = BT/ /m, we see that X is decreasing in B. Hence larger B and consequently larger are preferred.
257
258
3. [Pretraining is critical] Pretraining can boost the accuracy of DP learning through a much smaller initial loss Co and from a smooth (small L) and flat (small &, c.f. Figure 7(eff)) initialization.
259
4. [Learning rate needs tuning] The optimal learning rate by minimizing (C.4) v "(ann+dTT)" This indicates that one should use larger learning rate for smaller model d, weaker privacy (larger I or small e), or smaller iteration budget T. Interestingly. the optimal choice of learning rate independent to (expected) batch-size B.
280
261
262
Experiments
265
We evaluate our automatic DP training on image classification, sentence classification, and table-to- text generation tasks. Detailed settings including hyperparameters can be found in Appendix G.
266
267
6.1
Image classification
use the same setup in [49, 60, 57] with simple CNN. For 10-class sub-task of ImageNet [15],
For MNIIS/FashionMNIIST,
269
CIFARIO, we use the same setup as in [60] with pretrained SimCLRv2 [12]. For ImageNette, a the same setup in [32] without the learning rate decay. For CelebA [41], the real human face dataset, we train ResNet9 [28] with group normalization to replace the batch normalization. Notice that CelebA contains high-resolution (178x218) images, each with 40 labels. We consider CelebA for either multi-class classification on one label, e.g. 'Smiling' and 'Male', or for multi-label/multi-taee problem to learn all labels simultaneously.
271
273
274
Accuracy %
Task
Model
(c,
non-DP (c = xc)
Abadi's clipping
AUTO-S clipping
MNIST
4-yayer CNN
(3, le-5)
98.04 + 0.09
98.15 0.07
99.11 + 0.07
FaShionMNIST
4-yyer CNN
(3, Ie-5)
86.04 + 0.26
86.36 + 0.18
89.57 + 0.13
CIFARTO pretrained
SimCLRv2
(2, le-5)
92.44 0.13
92.70 0.02
94.42 0.01
ImageNette
ResNet9
(8, lc-4)
60.29 + 0.53
60.71 + 0.48
71.11 + 0.37
CelebA [Smiling]
ResNet9
(8, 56-6)
90.75 + 0.11
91.08 + 0.08
92.61 1 0.20
CelebA [Male]
ResNer9
[8, 5e-6)
95.54 0.14
95.70 0.07
97.90 0.04
CelebA Multi-label
ResNet9
(3, 5c-6)
86.81 + 0.03
87.05 = 0.01
90.30 + 0.02
CelebA Multi-label
ResNet9
(8, 5e-6)
87.52 + 0.15
87.58 + 0.04
90.30 + 0.02
Table 1:
Average test accuracy and 95% confidence interval on image tasks over 5 runs.
In Table 1, we observe that AUTO-S clipping outperforms existing clipping in all datasets with statis- tical significance. Interestingly, the standard deviation from different runs smaller for automatic DP optimizers, indicating better reproducibility and stability. We additionally experiment 40 binary classification problems on CclebA with respect to each label, and observe that the mcan accuracy further improves to 91.63% at 8 for AUTO-S (see Appendix J).
276
278
6.2
Sentence classification
On five benchmark language datasets (MNL[(m/mm]664) QQP[30], QNLI[55), S5T2[59]), we 281
282
compare our automatic DP training with reparameterized gradient perturbation (RGP, [68]) and full parameter finetuning (full, [37]) using RoBERTa models [40]. These methods use the same experimental setup. For language models, our automatic training is based on the codebase of [37]'.
283
Seee /ttps://get //gi::/iii thub. com conm/xuechen//aee private- transformers and the detailed modification in Ap- pendix K.3.
8

---

Method
6=3
6=8
D0 (non-DP)
MNLI
QQP QNLI SST2
MNLI
QQP
QNLI
SST2
MNLT
QQP QNLI SST2
RGP [68]
80.5779.6
85.5
87.2
91.6
83.6/83.2
89.3 91.3 92.9
Tull [37]
82.45/82.99 85.56 87.42 91.86 83,20783.46
86.08
87.94
92.09
full AUTO-V 81 81.21/82.03 84.72 86.56 91.86 82. 18/82.62
86.23
87.24
92.09
85.91/86.14 14 87.34 91.40 94.49
full AUTO-S 83.22/83.21 85.76 86.91 92.32 83.82/83.55
86.58
87.85
92.43
Table 2: Test accuracy on language tasks with RoBERTa-base (12 blocks, 163 million parameters).
Method
C=8
Do (non- DP)
MNLI
QQP QNLI SST2
MNLI
QQP
QNLI
SST2
MNLI
QQP QNLI SST2
RGP [68]
86. 1786.0
86.7 90.0 93.0
Tull [37]
86.43/86.46 86.43 90.76 93.04
87.02/87.26
87.47 91.10
93.81
full AUTO-V
85.33/85.61 86.61 89.99 93.12
85.91/86.10
86.86
90.55
93.35
90.33/90.03 87.90 93.61 96.21
full AUTO-S 86.27/86.67 86.76 91.01 93.92 87.07/87.16
87.47
91.45
94.61
Table 3: Test accuracy language tasks with RoBERTa-large (24 blocks, 407 million parameters).
In Table 2 and Table 3, we note that full parameter finetuning with AUTO-S outperforms or least matches SOTA on all tasks. We use exactly the same hyperparameters as in [37].
286
287
6.3
Table-to-text generation
288
We compare our automatic DP training with a variety of fine-tuning methods, for table-to-text generation task on E2E dataset [20], where the goal is generate texts about different aspects of
289
290
restaurant's data. We measure the success on this task by BLEU, ROUGE-L (in Table 4), METEOR,
291
NIST, CIDEr (extended in Table 7), with higher value meaning better model quality.
DP
GPT2
GPT2
GPT2
Metric
guarantee
large
medium full
full
full
full
full
LoRA
RGP
prefix
top2
retrain
AUTO-S
AUTO-S
AUTO-S
AUTO-V
[37]
[29]
[68]
[36]
[37]
[37]
3
64.180
63.850
61.340
61.519
61.519
58.153 58.482
47.772 25.920
15,457
BLEU
8
64.640
64.220
63.600
63.189
63.189
63.389 58.455 49.263 26.885 24.247
non-DP
66.840
68.500
69.463
69.463
69.463
69.682 68.328 68 68.845 65.752 65.731
3
67.857
67.071
65.872
65.670
65.670
65.773 65.560 58.964 44.536 35.240
ROGUE-L
c
68.968
67.533
67.073
66.429
66.429
67.525 65.030 60.730 46.421 39,951
Don-DP
70.384
71.458
71.359
71.359
71.359 71.709 68 68.844 70.805 68. 704 68.751
Table 4: Test performance on E2F dataset with GPT2. Additional performance measures are included
in Table 7. The best two GPT2 models for each row are marked in bold.
Competitive methods include low-rank adaption (LoRA), prefix-tuning (prefix), RGP, only fine-tuning the top 2 Transformer blocks (top2), and training from scratch (retrain), as were recorded [37]. the exactly the same hyperparameters GPT2 medium (406 million), and GPT2 large (838 million), Table 4 shows that AUTO-S is scalable with stronger performance larger models. Our automatic uall-parameter finetuning has the best overall performance. Additionally, we highlight that AUTO-S and methods like LoRA are not mutually exclusive and can be combined to yield strong performance, since AUTO-S modifies the optimizers and LoRA modifies the architecture
293
Again,
in [37]. For GPT2 (163 million parameters),
295
297
299
Discussion
300
In this work, we proposed AutoClipping as a drop-in replacement to the standard per-example clipping differentially private training. This is the first technique that eliminate the need to tune the clipping threshold R, thus making DP deep learning as easy regular learning. Our AUTO-S method enjoys both thcoretical guarantee of convergence in non-convex problems (under various conditions), and strong empirical performance that advances the state-of-the-art (SOTA) of DP learning on both computer vision and language tasks.
302
304
306
307
We are excited about the future of automatic DP training, especially along with other working techniques. Notably, our automatic clipping applies compatibly with general optimizers (e.g. [8, 18]), clipping styles (all-layer or per-layer), architecture modifications (e.g. LoRA, RGP, prefix), and data augmentation (e.g. adversarial training [26] and multiple augmentation [14]). Thus, we expect to achieve comparable results to all SOTA in a lightweight fashion.
308
310

---

References
312
313
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 008-318, 2016.
314
315
316
[2] Zeyuan Allen-Zhu. Natasha 2: Faster non-convex optimization than sgd. Advances in neural information processing systems, 31, 2018.
317
318
[3] Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. Advances in Neural Information Processing Systems, 34, 2021.
319
320
[4] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan, June 2005, Association for Computational Linguistics.
322
323
324
325
[5] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Arizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560-569, PMLR, 2018.
328
327
328
[6] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-11, 2018.
329
330
Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard data science review, 2020(23), 2020.
331
332
[8]
Zhiqi Bu, Sivakanth Gopi, Janardhan Kulkarni, Yin Tat Lee, Hanwen Shen, and Uthaipon Tantipongpipat. Fast and memory efficient differentially private-sgd via jl projections. Advances in Neural Information Processing Systems, 34, 2021.
333
334
335
99] Zhiqi Bu, Hua Wang, and Qi Long. On the convergence and calibration of deep learning with differential privacy. arXiv preprini arXiv:2106.000 07830, 2021.
336
[10] Mark Bun, Cynthia Dwork, Guy N Rothblum, and Thomas Steinke. Composable and versatile privacy via truncated cdp. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pages 74-86, 2018
338
339
[11]
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Frlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 2/), pages 2633-26511 2021.
341
342
343
344
[12 Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
345
346
347
[13] Xiangyi Chen, Steven 2 Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspeetive. Advances in Neural Information Processing Systems, 13:13773-13782, 2020.
349
350
[14] Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlock- ing high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.1360 13650, 2022.
352
353
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
355
356
[16] Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. Journal of the Royal Statistical Society Series B, 84(1):3-37, 2022.
[17]
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
[18] Jian Du and Haitao Mi. Dp-fp: Differentially private forward propagation for large models. arXiv preprint arXiv:2112. 14430, 2021.
360
361 [19] John Duchi, Flad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning
and stochastic optimization. Journal of machine learning research, 12(7), 2011.
10

---

363
[20]
Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. Evaluating the ofate-of-the-Ai of End-to-End Natural Language Generation: The E2E NLG Challenge. Computer Speech & Language, 59:123-111110 123-156, January 2020.
384
365
[21] Cynthia Dwork. Differential privacy:
A survey of results. In International conference on theory and applications of models of computation, pages 1-19. Springer, 2008
367
388
[22] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
389
370
[23] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy, Found. Trends Theor. Comput. Sci., 9(3-4):211-40) 2014.
372
373
[24]
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2368, 2013.
374
375
[25] Aditya Golatkar, Alessandro Achille, Yu-Xiang Wang. Aaron Roth, Michael Kearns, and Stefano Soatto. Mixed differential privacy in computer vision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
376
377
2022.
378
[26] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial
379
examples. In International Conference on Learning Representations, 2015.
380
[27] Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems, 34, 2021.
381
[28)
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
383
385
[29]
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
386
387
388
[30]
Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs, 2017.
[31] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
381
382
[32] Helena Klause, Alexander Ziller, Daniel Rueckert, Kerstin Hammernik, and Georgios Kaissis. Differentially private training of residual networks with scale normalisation. arXiv preprint arXiv:2203.0mm 003224, 2022.
393
394
[33] Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using fft. In International Conference on Artificial Intelligence and Statistics, pages 2560-2569.
396
PMLR, 2020.
398
[34] Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at imagenet scale with differential privacy. arXiv preprint arXiv:2201.12325 2022.
389
400
401
[35] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac. Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
403
405
407
408
409
410
411
[36]
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lnguage Processing (Volume 1: Long Papers), pages 4582-4597, 2021.
412
413
11

---

415
[37] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language mod- els can be strong differentially private learners. In International Conference on Learning Representations, 2021.
416
417
418
(38) Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
419
420
[39] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference Learning Representations, 2019.
422
[40] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 2019.
425
[41] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
428
429
[42] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018.
[43] H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, 2018.
432
438
[44] Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint wxXiv22055.02977 2022.
435
436
[45] Iya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263-275 IEEÉ, 2017.
437
438
[46]
Diganta Misra. Mish: A self regularized non-monotonic activation function. BMVC 2020, 2019.
439
[47]
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o (1/K" 2). In Dokl. akad. nauk Sssr, volume 269, pages 543-547, 1983,
440
441
[48]
Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy. In International Conference on Learning Representations, 2021.
442
[49] Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Úlfar Erlingsson. Tempered sigmoid activations for deep learning with differential privacy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9312-9321, 2021.
443
445
446
[50] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311-318, USA, 2002. Association for Computational Linguistics.
448
419
450
[51] Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank J Reddi, and Sanjiv Kumar. Adaclip: Adaptive clipping for private sgd. arXiv preprint arXiv:1908. 07643, 2019.
451
452
[52]
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5):1-17, 1964.
453
454
[53]
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838 855, 1992.
455
456
[54]
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint 19Xiv:1903.1052
457
458
2019.
459
[55] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016.
460
461
[56] Seyed Omid Sadjadi, Timothee Kheyrkhah, Audrey Tong, Craig S Greenberg, Douglas A Reynolds, Elliot Singer, Lisa P Mason, Jaime Hermandez-Cordero, et al. The 2017 nist language recognition evaluation. Odyssey, pages 82-89, 2018
462
463
464
[57] Ali Shahin Shamsabadi and Nicolas Papernot. Losing less: A loss for differentially private deep learning. 2021.
12

---

486
[58]
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE sympostum on security and privacy (SP), pages 3-18. IEEE, 2017.
467
469
[59] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1631 1631-1642, 2013.
470
472
473
[60] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In International Conference Learning Representations, 2020.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
476
[62]
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-bast based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566 2015.
479
480
633
Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differ- ential privacy and analytical moments accountant. In International Conference on Artificial Intelligence and Statistics, pages 1226-1235. PMLR, 2019.
482
483
484
[64]
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics, 2018.
485
486
487
488
489
[65] Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. Xixiv preprint arXiv: 708.0088,,,,888,, 6(12):6, 2017.
490
491
[66]
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020.
432
494
495
[67]
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv:2109.122 12298, 2021.
497
[68]
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie Yan Liu. Large scale private learning via low-rank reparametrization. In International Conference on Machine Learning, pages PMLR, 2021.
500
1
502
[69]
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 1212.5701 2012.
503
504
[70] Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics, pages 4782-4817. PMLR, 2022.
505
506
Checklist
1. For all authors...
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
510
(b) Did you describe the limitations of your work? [N/A]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
513
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
514
[Yes]
2. If you are including theoretical results...
13

---

518
(a)
Did you state the full set of assumptions of all theoretical results? [Yes] See Assumptions 5.1-5.3.
517
518
(b)
Did you include complete proofs of all theoretical results? [Yes] See Appendix A.
519
3.
If you ran experiments...
Did you include the code, data, and instructions needed reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We include our code in Appendix K, and cite all datasets.
521
522
525
(b)
Did you specify the training details (e-g.. data splits, hyperparameters, how they were chosen)? [Yes] See all details in Appendix G.
(c) Did you report error bars (e.g., with respect the random seed after running experiments multiple times)? [Yes] See Table 1.
626
527
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] This work is unrelated computational efficiency.
528
529
530
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
531
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite all codebases, datasets, models clearly.
(b) Did you mention the license of the assets? [N/A] Experiments conducted on public data. Did you include any new assets either in the supplemental material as a URL? [Yes] We include code snippet in Appendix K and will include a URL to release the full codebase. dd) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
534
535
537
538
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
539
5.
If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
542
[N/A]
543
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
545
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
546
14


Reviews:
[{"review_id": "wiTmAwe8S8", "reviewer": "Reviewer_PkCd", "summary": "DPSGD requires gradient clipping to bound the sensitivity of the information output function, however, this operation results in a heavy workload on tuning the clipping bound, as it turns out the performance of the network is sensitive to this hyperparameter. This work proposes to replace gradient clipping $min(R/||g||, 1)g$ with gradient normalization $Rg/(||g|| + \\gamma)$ based on two aspects. First, it is observed in practice that gradient normalization results in higher cosine similarity between the clipped gradient and the original gradient. Second, the additive term $\\gamma$ helps to retain some information about magnitude as otherwise all gradients are pushed onto the boundary of a ball with radius $R$. Applying gradient normalization, $R$ can either couple with the learning rate or cancel out, therefore needing no further tuning.", "questions": "A disadvantage of the proposed automatic clipping is that it always distorts the distribution of gradients even if all gradients locates within the clipping bound. According to the experiments in this work, the clipping rate dramatically reduces after some iterations indicating that many gradients are not clipped with Abadi's clipping. Therefore automatic clipping theoretically should carefully tune the hyperparameters which determine the distortion, e.g. $\\gamma$. Although the authors argue that the network is insensitive to the choice of $\\gamma$, I note that the ablation study of $\\gamma$ is conducted with one epoch training, however, this could not represent the whole training procedure, complete the whole training epochs would be more convincing. I'd be happy to increase the score if my question is answered, overall I still think this is a good work.", "limitations": "There is no potential negative societal impact of this work, for limitations refer to Questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This work is well-written and easy to follow.\n2. The proposed automatic tuning method is interesting, also simple, and easy to implement.\n3. The experiments are intensive and convincing.\n\nWeakness:\n\n1. I see some layout problems, for example, Figure 6 has a broken link, and in line 151, I think the ablation study is presented in Figure 14 instead of 13. There are some others, this paper should be carefully polished before it is published.\n2. The study of $\\gamma$ should be more thorough, it is very important to know that tuning work is not transferred from $R$ to $\\gamma$, please refer to Questions for detail.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XyszsmCRAxq", "reviewer": "Reviewer_1dkx", "summary": "Differentially private SGD has two additional components over SGD: gradient clipping and noise addition. The clipping parameter and scale of added noise determine the privacy of the algorithm. For a target privacy budget, if we allow the scale to vary accordingly, then the clipping parameter becomes a hyperparameter that must be tuned. This work proposes a method to expunge the tuning of the clipping hyperparameter.", "questions": "I have included my questions in the above.", "limitations": "I do not see any obvious negative societal consequences from this work.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea behind this work is to remove the need to tune the clipping norm parameter C. Instead of a practitioner choosing C and then applying $g\\cdot \\min({\\frac{C}{||g||}, 1})$, we set $C=1$ and apply $\\frac{1}{||g|| + \\gamma}$. The private gradient becomes independent of the clipping norm, and we only need to choose $\\gamma$. The core idea can be seen in Figure 4, where the idea is to create a smoothed version of clipping operation, where the rate of increase of norm depends of $\\gamma$. Ablations show that the experimental results are insensitive to $\\gamma$ and so can be set to 0.01 for all experiments.\n\nOverall, the paper is well-written and contains sound justification for why this procedure works. My main concerns are two-fold: contribution and significance.\n\nOn contribution, De et al. [1] also introduce a method to decouple learning rate tuning from clipping norm tuning (making the claim in line 280 incorrect). They do this by reparameterizing DP-SGD such that the learning rate absorbs a factor of $C$, and so the clipping norm has no influence on the scale of the update. Similar to yours, they then set $C=1$ in all main experiments. E.g. See App B.1 in [1] where the choice of clipping norm has practically no effect on validation accuracy across different learning rates. Could the authors comment on the comparison between the idea in [1] and this work, and what they see as the contributions made here?\n\nIn terms of motivation, can the authors comment on how much of a problem tuning $C$ is in practice? For the NLP tasks, it seems like the choice of $C$ is insensitive to learning rate (if the learning rate is sufficiently large), while for ImageNet it seems like we only need to choose $C$ to be of the same order of magnitude as the learning rate. Is it possible to use these as a rule-of-thumb over different vision and NLP tasks? I'd be curious to know how sensitive these tasks are to the learning rate *without* privacy. Could the authors provide these numbers? Additionally, does the choice of architecture make a difference? I.e. what if I were to choose VGG-19 instead of ResNet in Fig 1, would the plot look the same? Why are the ablation studies in Fig 14 (not Fig 13 as stated in line 151) for different datasets than in Fig 1? I was expecting to see an identical plot to Fig 1 for a fixed choice of $\\gamma$ showing the problems observed are no longer present.\n\n[1] S. De, L. Berrada, J. Hayes, S. L. Smith, and B. Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bo1lU_vr4DY", "reviewer": "Reviewer_4CCG", "summary": "This paper proposes a novel way to automatically clip which eschews the need for an extra hyperparameter. This saves from a combinatorial increases in number of trials in sweep to tune DP hyperparameters. Both convergence analysis of the proposed DP optimizer and experiments on practical setup is provided.", "questions": "As mentioned as weakness above, I would request the authors to conduct a study on at least one dataset where AUTO-S/V is superior than vanilla DP-SGD clipping, where we track what happens when 1) clipping norm R is made smaller in the baseline and 2) change \\gamma in AUTO-S/V", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Great overall direction in reducing total cost of training with differential privacy.\n- Paper is well-written, with a mix of intuition and illustrative experiments.\n- Experiments on practical models and datasets suggest strong performance with proposed AUTO-S/V clipping scheme.\n\nWeaknesses\n- Normalizing per-example gradients is equivalent to setting a very small clipping norm R (smaller than minimum of gradient norm of any example), which conflates the effect of a small clipping norm and \\gamma. I encourage the authors to make it more explicit.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0_VxTqSZETH", "reviewer": "Reviewer_pC48", "summary": "Motivated by the observation that clipping happens very frequently on real-world dataset. The author(s) proposed a new clipping method (clipping all gradients regardless of its magnitude). The new clipping method is “almost” hyper-parameter free and can be implemented as efficient as DP-SGD with gradient clipping. On the theoretical side, the author(s) analyzed the convergence of DP-SGD with the proposed clipping method for nonconvex problems. On the empirical side, experiments on real-world datasets demonstrate the effectiveness of the proposed clipping method.", "questions": "- Can we relax Assumption 5.3?\n- Does AUTO-S require a more careful tunning of $eta$? Does Abadi's clipping and AUTO-S use different learning rate in the experiments?", "limitations": "Yes, limitation and societal impact are well-addressed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The contribution on the theoretical side is a little bit exaggerated. The main problem is that the author(s) use a much stronger assumption (Assumption 5.3) when analyzing the convergence. This assumption comes from [1], it was shown that this assumption can simplify the convergence when gradient clipping is adopted. However, this assumption can be false for simple linear regression problem. Therefore the claim in the abstract “the proposed clipping enjoys an convergence rate that matches the standard SGD” is kind of over-claimed since the convergence analysis of the proposed method and standard SGD rely on a different set of assumptions.\n- The AUTO-S method fused $R$ into $\\eta$ and therefore does not require to tune the clipping threshold $R$. However, this may end up requiring a more careful tunning of the learning rate $\\eta$ since $R$ is implicitly involved in $\\eta$ now.\n- Overall, I appreciate the empirical contribution of this paper but I think the claimed theoretical contribution should be weakened. With the theoretical contribution being discounted, I am not sure if this paper has enough contribution to be accepted at NeurIPS.\n\nMinor:\n- What does the number of the heat map in Figure 1 stand for?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject