Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger
Authors: Zhiqi Bu
Yu-Xiang Wang
Sheng Zha
George Karypis
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
differential privacy
per-sample gradient clipping
convergence

Abstract: Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping threshold $R$, however, is shown to be vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called automatic clipping, that eliminates the need to tune $R$ for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB and many others.
The automatic variants are as private and computationally efficient as existing DP optimizers, but require no DP-specific hyperparameters and thus make DP training as amenable as the standard non-private training. We give a rigorous convergence analysis of automatic DP-SGD in the non-convex setting, which shows that it {can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric gradient noise assumption of the per-sample gradients.} We also demonstrate on various language and vision tasks that automatic clipping outperforms or matches the state-of-the-art, and can be easily employed with minimal changes to existing codebases.

Under review as a conference paper at ICLR 2023
AUTOMATIC CLIPPING: DIFFERENTIALLY PRIVATE DEEP LEARNING MADE EASIER AND STRONGER
Anonymous authors Paper under double-blind review
ABSTRACT
Per-example gradient clipping is key algorithmic step that enables practical dif- ferential private (DP) training for deep learning models. The choice of clipping threshold R. however, is shown to be vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called automatic clipping, that elim- inates the need to tune R for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB and many others. The automatic variants as private and compu- tationally efficient as existing DP optimizers, but require no DP-specific hyper- parameters and thus make DP training amenable the standard non-private training. We give a rigorous convergence analysis of automatic DP-SGD the non-convex setting, which shows that it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric gradient noise assumption of the per-sample gradients. We also demonstrate on various language and vision tasks that automatic clipping outperforms or matches the state-of-the and can be easily employed with minimal changes to existing codebases.
INTRODUCTION
Deep learning has achieved impressive progress in a wide range of tasks. These successes are made available, in part, by the collection of large datasets, sometimes containing sensitive private information of individual data points (e.g., chest sean images, DNA sequences). Prior works have illustrated that deep learning models pose severe privacy risks to individual subjects in the training data and are susceptible to various practical attacks. For example, machine learning services such Google Prediction API and Amazon Machine Learning leak membership information from the purchase records (Shokri et al. 2017); if one feeds the GPT2 language model with some specific prefix, the model will autocomplete texts that contain someone's full name, phone number, email address, etc., from the training data that it memorizes (Carlini et al. 2021).
Differential privacy (DP) (Dwork 2008, Dwork al. 2014
formal definition of privacy that has been shown to prevent the aforementioned privacy risks in deep learning (Abadi et al. 2016). On a high level, the key difference between the DP deep learning and the regular one whether the gradient is privately released. In other words, while the standard optimizers update on the summed gradient E, gi, and DP optimizers update on the private gradient: privatc gradient
2006
DP Optimizer((g.) Optimizer( Σ 9 clip(1lg.ll R) ao- ((.,))
(1.I)
Standard Optimizer((g:))3 B Optimizer( Σ, gi)
(1.2)
Here g: E RRd the per-sample gradient of loss I,, / is the standard normal, is the noise multiplier,
and R is the clipping threshold. The clipping function Clip IRd ->
IR is defined such that
lg.. Clip(g; R|ll < R. For instance, the DP-SGD in Abadi et al. 2016 on batch B,
DP-SGDAMa Ahadi
W0++1 = W: - 1/ Σ
0l ow:
min (R/
Ow. 1) + oR -N)
Ø4
(1.3)
B.
In comparison to the regular training (1.2 two additional DP-specific hyperparameters It and o need to be determined in DP learning (I.I) On the one hand, setting the noise multiplier a is easy and can be derived analytically prior to the training. Whenever the privacy budget (c.6) is determined, one can apply off-the- fff-the-hhell shelf privacy accounting tools in Section 2.1 to determine o, based on the subsampling probability p and the number of iterations T:
privacy accountant( (a, 8)

---

Under review as a conference paper at ICLR 2023
On the other hand, the choice of clipping threshold R is crucial to the performance of DP models, yet the hyperparameter tuning is much labor-intensive. Recent advances of DP deep learning on ImageNet (Kurakin et al., 2022) and on E2E datasets (Li et al. 202I), using ResNet18 and GPT2 respectively, illustrate that the performance is very sensitive to R. We have reproduced their results in Figure [ Observe that ImageNet, & can drop from the highest 45% accuracy to 31 % if R is chosen 2 times larger, and to 0.1 I % if R is chosen 4 times larger. Similar drastic drop can also be observed in (Kurakin et al. 2022 Figure 3) even if the noise multiplier o O. Unlike the noise multiplier the clipping threshold R cannot be inferred from the privacy budget (c. 6) and have to be tuned. Consequently. DP training necessarily requires a 2D grid search for (R. n), like the lower plot of Figure[] whereas the regular training only requires an easy ID grid search for 1y. Even worse, the difficulty of tuning a per-layer clipping threshold vector (McMahan et al. 2018), i.e. one clipping threshold for one layer, may increase exponentially as the number of layers increases.
To save the effort of tuning R. previous researches have proposed different approaches. In (Andrew ett al. 2021: Pichapati et al. 2019 Golatkar et al. 2022), researchers advocate to use data-adaptive information to select It such as a specified quantile of the gradient norm distribution. These adaptive clipping methods can be little ad-hoc: they often replace the the need to tune R by the need to tune one or more new hyperparameters, e.g. the quantile to use and the ratio to split the privacy budget between the quantile decision and the gradient perturbation. Another approach used by the practitioners is to replace an expensive 2D grid search by multiple cheaper 1D grid searches. For example, the researchers propose, in (Kurakin et al. 2022] Section 3.3) to fine-tune v) with non-DP SGD, fix 1) and sweep over various values of the clipping threshold At with DP-SGD, then further fix R and do one more grid search on 1). However, tuning R formally in data-dependent way (e.g. through cross-validation) introduces additional privacy loss (Papernot & Steinke, 2021). and most existing empirical work does not privately conduct hyperparameter tuning.
We take completely different route by proposing new clipping principle that removes R, instead of coming up with methods to find the appropriate R. We term our method as automatic clipping and we term the versions of DP optimizers using it as automatic D)P optimizers.
We summarize our contributions as follows.
1. We propose the automatic clipping in (4.1) that expunges the clipping threshold from general DP optimizers, allowing DP learning to be as amenable as regular learning.
2. We show that automatic DP optimizers are as private and efficient as existing DP optimizers.
3. We show in Theorem 4 that automatic DP-SGD converges in the non-convex setting, at the same asymptotic convergence rate as the standard SGD. Our theoretical analysis successfully explains the training behaviors in previous empirical works.
4. We demonstrate the superiority of automatic clipping on variety of vision and language tasks, especially with large models including ResNet, RoBERTa and GPT2.
5. In Appendix K we include simple code snippets that demonstrate how easy it is to switch from Abadi's clipping to our automatic clipping in popular codebases, e.g. Opacus and ObJAX.
2 PRELIMINARIES
2.1 DIFFERENTAAL PRIVACY
2.1 where smaller (E, means stronger privacy guarantee. Definition 2.1 Dworrk et al. 2006)). randomized algorithm M is (e, didffferrrrl private (DP) if for any two neighboring datasets S, S", and for any event E,
We consider the (e, 8DD in Definition
P(M(S) E E] < o (M E E +
(2.1)
In words, DP restricts the influence of an arbitrary sample, so that the information contributed by such sample is limited and less vulnerable to privacy attacks. In deep learning, DP is achieved by applying the subsampled Gaussian mechanism to privatize the minibatch gradients during training.
As illustrated in Equation [. the subsampled Gaussian mechanism involves (1) Sampling a mini- batch by including each data point iid with probability p (2) per-sample gradient clipping to bound
S' is neighbor of S if can obtain S" by adding or removing one data point from
2

---

Under review as a conference paper at ICLR 2023
the 12 norm sensitivity at R and (3) adding independent Gaussian noise proportional to the sensitiv- ity R and σ, which is derived from the privacy loss c. This can be realized by leveraging a variety of modern privacy accounting tools, such Renyi DP (or moments accountant) (Abadi et al. 2016
Mironov 2017 Wang et al. 2019). Privacy Loss distribution (Fourier accountants) (Koskela et al,
2020
Copi et at. 2021: Zhu et al. 2022), or Gaussian DP (Dong et al. 2022 Bu al. 2020).
2.2
DIFFERENTIALLY PRIVATE OPTIMIZERS WITH GENERAL CLIPPING OPERATIONS
Privately released stochastic gradients (through the Gaussian mechanism) can be used to instantiate various off-the-shelf optimizers, which gives rise to DP-SGD in (.3). De-Heavy Dd-AdaGrad, DP-Adam. DP-FedAvg, DP-FedSGD (McMahan et al. 2018), etc. To improve the performance of DP optimizers, previous researches can be classified into two categories.
The first category, where the majority of researches lie in, works with Abadi's clipping and focuses on better design of R. To name few examples, one can adaptively design R, for each iteration t (Andrew et al. 2021 Pichapati et al. 2019. Golatkar et al. 2022). or design the per-layer clipping
threshold vector R E R- for L layers (Abadi et al. 2016 McMahan et al. 2018) so as to apply different clipping threshold for each layer.
Fewer works fall into the second category that proposes new clipping methods. In fact, any function Clip: : IR + IR satisfying [[Clip(g) gll <. R can serve as a valid clipping function besides
Abadi's. For instance, the global clipping (Bu et al. 2021b) proposes C1 Pglobal (g) == I( [|g R)
to mitigate the bias of the private gradient and alleviate the mis-calibration issue of DP classifiers Our automatic clipping also belongs this category. We note that different clipping methods work orthogonally to optimizers, network architectures and gradient norm computation (see Section 7.
3 MOTIVATION
3.1 SMALL CLIPPING THRESHOLD WORKS BEST
10-1
60
26
59.58
63.51
$7.87
15.05
One intriguing observation that we can make about the recent studies on DP learning with large models is that the state-of-the- art (SOTA) results are often achieved with very small clipping threshold R. This observation
Learning rate
50
08 50.31
50.48
49:25 36.47
33.38
33.03 32.05
29.50
consistent in both vision and
29.5
language tasks. In Li et al. (2021), GPT2 (about 800 million pa- rameters) and RoBERTa models (over 300 millions parameters) achieve the best results under DP on QNLI, MNLI, SST-2, QQP, E2E, and DART datasets, with each sar-s gradient clipped to length R = 0.1. In (Kurakin et al. 2022 De et al. 2022 Mchta let al. 2022). ResNets and Vision Transformers achieve the best DPP results on ImageNet with R = 1: in (Tramer& & Boneh 2020).
santon
0,51
61"
93-2"
Clipping norm
43.0
2.5
0.1
0.1
Learning rate
**tot 22.0
45.2
01
011
0.1
38
the best DP results on CIFARIO 10 use R = 0.1 with ResNeXt-29 and SimCLRv2 (Chen et al. 2020a). The effectiveness of small clipping threshold together with proper learning rate is depicted in Figure !
2.1
22.0
85.
11.0
0.29
22.5
24.0
0.11
13
y
Intuitively, smaller R implies that the Abadi's s clipping [.J] hap- pens, which means min (R/llg:ll, = R/ Rt/lg:ll Given that the clipping threshold R is so small compared to the number of pa- rameters in large neural networks, and that strong DP is guaran- teed when the number of training iterations is small (i.e. 11g. || has not converged to small values yet), we expect and empiri- cally observe that the clipping happens par-sa gradients at all iterations. For instance, we find in the GPT2 generation experiments in [i al. (2021) that 100% of per-sample gradients are clipped at all iterations; in classifica- tion tasks such as QOP/ONLI/MNL, the percentage of clipping is about 20 ~ 60% on average (more details in Appendix H.l
Clipping norm
Figure 1: Ablation study of clipping threshold and leam- ing rate. Upper: BLEU score of GPT2 on E2E dataset (Li et al., (021). with DP-
large proportion of
AdamW. Lower: Test accu- racy of ResNet18 on ImageNet dataset (Kurakin et al., (022). with DP-SGD and momentum.
3.2 PER-SAMPLE GRADIENT NORMALIZATION AS NEW CLIPPING
In the small clipping threshold regime, we can approximately view Cl Clip P Abadi (g:; R) min (R/ /1g 1) R R/ //llg: CI clipAUTOv(9g; R)
(3.1)
3

---

Under review as a conference paper at ICLR 2023
and thus derive novel private gradient Rigll [igill + oR .V(O.)). Here AUTO-V stands for the vanilla automatic clipping, which essentially performs the gradient normalization on each per- sample gradient. As a specific example, we can write the Rt-dependent automatic DP-SGD as
04,
01,
R-dependent D : #+++1 w0
Σ
R
/11 ow.
+ oR. No.,NO0
(3.2)
Oe.
S.B.
We may view our AUTO-V clipping as to maximize the dot-product similarity commonly used similarity measure, e.g. in the attention block in transformers (Vaswani et al. 2017)) between the clipped gradient and the regular gradient. Suppose we want
max C.... E,9;>
s.t. 0 S C, S R/llall
Note that the constraint is a sufficient condition for clipping, as discussed in Section 2.2 It is not hard to see that the optimal clipping factor is
C = R/ |1:11| I( E, 93) >0)
If the per-sample gradients are indeed concentrated in the sense Vi, (g:: Σ, 93) 2 0, then AUTO-V the optimal per-sample gradient clipping. We compare with Abadi's clipping Figure 2 where the dot-product similarity is significantly magnified by our AUTO-V clipping.
AUTO-V clipping
clpping Abadi clipiing
Abodi clpping
5060
15
111
176 150
1500 vebto 1abs sututatd un Autivve product
Figure 2: RoBERTa-base with DP-Adam (e 3) SST2 dataset, as in Section 6.2
3.3 STABILITY CONSTANT BREAKS SCALE-INVARIANCE AND REMAINS STATIONARY
One potential drawback of AUTO-V clipping is that all gradients lose their magnitudes information completely, since ||g: Cli clippM MLLPAUTO-ViE (g;; R)]| R, Vi. This scale- invariance in AUTO-V and partially in Abadi's clipping (when g > H) leads to the "lazy region" issue: the parameters will not be updated by DP-GD even if the true gradients are non-zero. In Figure 3. we illustrate in logistic regression that AUTO-V and Abadi's clip- ping have zero clipped gradient for the trainable param- eter 0 2- 2]. as the per-sample gradients from two classes cancel each other.
0.002-
0.002
doped groc 9ted)
dipeed AUTO-G1
-23
Figure Gradient (scalar) each 0.
Another benefit of y is to remain stationary as g: - 0, i.e. making the clipped gradient C.g: 9:/7 small rather than having a magnitude R in AUTO-V. We elaborate this point in Section 4.3
To preserve the magnitude information and thus escape the lazy region, we propose the AUTO-S clipping, with a positive stability constant 7:
Clip PLiPAUTOOSSG (g:: R) R/( + +7)
(3.3)
We visualize in Figure that AUTO-S allows larger per-sample gradients to have larger magnitudes after the clipping, while still allowing smaller gradients vanish after "clipping". This is critical in our convergence analysis and allows DP-SGDAUTO-S (but not DP-SGDAVTO-v) to converge to zero
gradient norms in Section 5
AUTOMATIC DP TRAINING
One may wonder why our clipping 3.1 3.3 is automatic at all, if the hyperparameter R is still present and there is an additional parameter 7 choose. It turns out that any constant choice of R > 0 is equivalent to choosing R = 1, and common deep learning optimizers are insensitive
The settings are in Appendix F where the lazy region issues also emerge in the mean estimation problem. We that the lazy region also discussed in (Chen al. 2020bb Example 2).
4

---

Under review as a conference paper at ICLR 2023
to the choice of y (e.g. for any Y O, show that the gradient converges to zero at the same asymptotic rate in Theorem 4 see also the ablation study in Figure|14] [4). Consequently, we set y 0.01 the default. Specifically, let us redefine the R-independent clipping function: C1 Clip
1/(1l:: + 7)).
(4.1)
With this clipping, we can design automatic DP optimizers similar to Automatic DP (enimmizer((g))) Optimizer
Ilg +7 + ((.
gt,s
(4.2)
iEB
denoted as 9r
Clearly, the new private gradient G: from our automatic clipping is R-independent, in contrast to the one used in [1. A concrete example (in the case of 7 0) that is comparable (3.2) will be
01
04:
R-independent DP-SGDAUTO-V
Wr+1 we
Σ
+o +o (4.3)
Sw.
8w.
BEBB
Leveraging the private gradient 9: in specific hyperparamters It and a, as demonstrated in Algorithm
train DP neural networks without tuning DP-
Algorithm Automatic Deep Learning with DP
Parameters: initial weights wo, learning rate n7. sampling probability p. number of iterations T..
1: Find o such that EAcommntnt (6, o. T) S from any privacy accountant.
2: for iteration / = 1,.... ..... T do
3: Sample a batch B, by including each data point i.i.d. with probability p
4: Apply automatic clipping to per-sample gradients {g. he B, 9. (.991 0.01).
5: Add Gaussian noise to the sum of clipped gradients: 9 = >; 9. + a NN(0),
6: Update we by any optimizer the private gradient with learning rate n)--
We will elaborate two distinct reasons in the next sub-sections for the following statement:
DP OptimizerAtatt Abadi R R-dependent DP Optimizer AUTO =\t R--indppendent DP Optimizer AUTO
which reduces the hyperparameter tuning of DP training to that of the regular training, i.e. only learning rate, weight decay, etc. The significant save in the tuning effort is illustrated Figure 15
4.1 NON-ADAPTIVE OPTIMIZER COUPLES CLIPPING THRESHOLD WITH LEARNING RATE With Rtdependent automatic clipping, DP-SGD becomes
R
W++1 wi Σ 9t.
oR ((..I) w, nRgi
19g.i +
B,
We can view Nettective = nr as a whole: increasing has the same effect as increasing 7, which ex- plains the diagonal pattern in Figure]]lowrr plot) where DP-SGDAtsal Absai is applied with small clipping threshold We extend to general non-adaptive optimizers Theorem
Theorem 1. Non-adaptive R-dependent automatic DP optimizers (including SGD, Heavy- Polyk 1964) and NAG(Nesterov) 1983)), with learning rate and weight decay X. is equivalent R-independent automatic DP optimizers, with learning rate nR and weight decay X/R.
4.2 ADAPTIVE OPTIMIZER CAN BE INSENSITIVE TO CLIPPING THRESHOLDD
Adaptive automatic DP optimizers different than the non-adaptive ones, the clipping threshold cancels out instead of being coupled with learning rate. To see this, we scrutinize DP-Adam (which is similar Ad-AAMAAUTO->) in Figure upper plot), where columns to the left are almost identical. Further evidence is observed in (Mehta et al. 2022 Table 5) that shrinking R has zero effect on LAMB. We now give simple explanation using AdaGrad (Duchi al. 2011):
9t
Wi+1 = W.
G
Whee we further consider weight decay in automatic clipping (included in Theorem 1. increasing H is no longer equivalent increasing 7), as 7 also couples with the weight decay constant X.
'This coupling of y and R is also partially observed in (De et al. 2022 through reparameterization trick of Abadi's s clipping. Unlike AUTO-S/V, their coupling not strict (e.g. doubling is not equivalent to doubling in their Figure 8, thus necessitating tuning both (17, R)), and the relationship weight decay not discussed.
5

---

the gradient noise, which empirically verified in Under review as a conference paper at ICLR 2023
where 9: = =E it. is the gradient sum and G2 Σ < of is sum of gradient square by Hadamard product over the past iterations. In R-dependent DP-A place of the standard gradient sum gt. and G. = R² Σ P.:
AdaGradduTooo. the private gradient is Rg. in
W++1 = Wi - Rg: we
9:
VG.
Erec (g-)²
We generalize the general adaptive optimizers in Theorem
Theorem 2. Adaptive R-dependent automatic DP optimizers (e.g. AdaGrad Duchi et al. 2011
AdaDelta 2012), aadaMax/AdamiKinnnng & Ba 2014), NAdam 2016). RAdam et al. 2019a), LARS You al. 2017). LAMBOYou al. 20200). with learning rate 1) and weight decay X is equivalent to R-independent automatic DP optimizers with learning rate nj and weight decay X/R. With decoupled weight decay Loshchilov & Hutter 2018), R-dependent automatic
DP-Adam is equivalent to R-independent automatic DP-Adam with the same r) and X.
In Appendix B.3 we also analyze the automatic DP optimizers with per-layer clipping style.
4.3 AUTOMATIC CLIPPING IS EQUALLY PRIVATE AND MAXIMIZES UTILITY
In Theorem (proved in Appendix A). we show that the new private gradient Dr in (4.2) has the same level of privacy guarantee the existing one in (I.I). since the global sensitivity remains the same (see Fig- ure 4. We note that long as y > O, the magni- tude information of per-sample gradients is preserved by AUTO-S, in the sense that [[g:1] > ||9 < ||Cggil > |[C,gill whereas this can be violated in both the AUTO-V and Abadi's ' clipping (as depicted by the flat curve in Figure when [|g. 1).
Norm plop gradient
AUTO-S y 0.001
y
AUTO-S, y-1
Abadi
035
275
1|l1|
Figure 4: Gradient norms before and after clipping by different methods at R=
Additionally, note that when y is small, almost all data points "max out" the signal relative to the amount noise we add. To say it differently, for the same amount of noise, AUTO-S with small y allows more signal to be pushed through differentially private channel. Towards the end of the training, i.e., at the limit when ||g: || + 0 for all i, then we have >; Toll + + >. 9i. In words, the clipped gradients become closer to the standard SGD, thus do not suffer from the instability of AUTO-V.
g.
Theorem 3. Under the noise multiplier a, number iterations T, subsampling probability B/n, DP optimizers using AUTO-V or AUTO-S clipping satisfy (EAccomminn (6, B B/n, B/n,T), T", 8)-DP, where Accommttant is any valid privacy accountant for DP-SGD under Abadi's clipping.
5 CONVERGENCE ANALYSIS OF DP-SGD WITH AUTOMATIC CLIPPING
5.1
CONVERGENCE THEORY OF DP- TO STATIONARY POINTS
We highlight that automatic clipping be more amenable analysis than Abadi's clipping in Chen et al. 20206). since we no longer need to decide whether each per-sample gradient is clipped.
To analyze the convergence of automatic DP-SGD 4.2 in the non-convex setting, follow the standard assumptions in the SGD literature (Ghadimi & Lan 2013) Allen-Zhu 2018 Bottou et al.) 2018). including symmetry assumption
(Chen et al. 2020b Figure 3
Assumption 5.1 (Lower bound of loss). For all w and some constant C., we have L(w) 2 L.. Assumption 5.2 (Smoothness). Let g denote the gradient of the objective L(ur). Then vw,er there is an non-negative constant T, such that
((v) - [[(w)) g(w) (v-ww w)] < w - 0122
(5.1)
'This symmetry assumption is relaxed from the Gaussian noise assumption (since zero-ment Gaussian is symmetric) in the SGD literature (Mandt et al. [017 [Smith et al.] 2018 Chaudhari & Soatto 2018 2020 By setting minibatch size to 1, reduce the noise assumption per-sample gradient case.
2018) Xie al.

---

from some Under review as a conference paper at ICLR 2023
Assumption 5.3 (Gradient noise). The per-sample gradient noise It.: ditribution such that
9t. is ii.d.
E(gr. 9t) = 0, E B - 9ell² < E²
and gt.s is centrally symmetric about 9: in distribution: Ör.: 9: 9t- gr
We show in Theorem 4 that DP-SGD with AUTO-S clipping allows the true gradient norm to con- verge to zero, though the clipped gradient may still be biased, but not so with AUTO-V clipping. We leave the proof in Appendix C.I
Theorem 4. Under Assumption
running DP-SGD with automatic clipping for T iter- ations and setting the learning rate 1/ V give
5.7
S.2
5.3
(VTV (60 L.)L (1+ LL,)
a2d
$ r>0 r
min B(Ilg:1) II) و OSEST
B² :5,7
min
F (... ;r,E, (5.2)
Here ... represents the first argument of 9, and G is increasing and positive. As + co, we have
min; E R([l 114) for AUTO-S, the same rate the standard SGD given in Theorem
Upper Sound ip
Upaer Souuno of mnge Uilil
Upger Douund m/ 10
20-
DN450 AUTO-A
Nimber of Rerat ons
Number of Rerorore
Number of teaatins
Figure 5: Left: DP-SGD with AUTO-V clipping. Middle: DP-SGD with AUTO-S clipping. Right: Log-log plot of convergence rate in comparison to standard SGD. Here { = 25, = 0.01, and the O(1/VT) term is set to 10 for DP-SGD and to 2 for standard SGD,
Remark 5.4. We show in Theorem 6 and in Figure 5 that the upper bound (.2) has G > & for
AUTO-V (7- 0), and G only reduces to zero for AUTO-S (7 > 0). We provide real data evidence in Figure 13 that strietly positive y reduces the gradient norm significantly.
5.2 ANALYSIS OF FACTORS AFFECTING THE CONVERGENCE
We now analyze the many factors that affect the convergence Theorem from a unified viewpoint of both the convergence and the privacy.
We start with the stability constant 7 and the learning rate T/t both only affect the convergence not the privacy. We empirically observe in Figure 7tha that small y benefits the convergence initial iterations (when the privacy guarantee is strong) but larger y converges faster asymptotically. For Ne. the optimal is in fact the miminizer of the hyperbola in (C.4). that is unique and tunable.
Next, we focus on the hyperparameters that affect both convergence and privacy: the batch size B, the noise multiplier a, and the number of iterations T. These hyperparameters have to be considered along the crivacy-accuracy tradeoff, not just from convergence perspective.
Recall that given fixed privacy budget (e, 6), we rely on modern privacy accountant for computing the appropriate combinations of parameter a,, T, B. The exact expression of the bound as a function of (c,5) is somewhat messy. For this reason, we illustrate our analysis in terms of the surrogate parameter y for j-GDP (Dong et al. 2022). Bu et (2020) showed that DP-SGD's privacy guaran-
tee asymptotically converges to μ GDP (as T - oo) with μ /T(el/a" - 1). μ-GDP implies (e, 5D-DP with = μ + p for similar conclusions, using p in place of 1 (.3).
2 log (1 We can alternatively leverage p-ICDP (Bun et al. 2018
'The upper bound takes implicit form of ((: because is lower envelope of functions $ r. E, 7) over possible r > 0, whose forms are detailed Theorem Notice that results only from the clipping operation, not from the noise addition.
"oore precisely, μ-GDP is equivalent to entire family of (c, 8D-DP for any > 0 and 5 o(n/2 e/p) - 0-p122 e/p² where & is the standard Gaussian CDF.

---

Under review as a conference paper at ICLR 2023
Theorem 5. Under Assumption 5.1 5.2 3.3
fixing the asymptotic pue, 6DP parameter, running DP-SGD with automatic clipping for T iterations and setting the learning rate & 1/ 1/VT give
( (1,1000-.)1
(
min OSYST
B(119: 1) S & ( C,)L
+ +0 u²n²
:5.7
(5.3)
B2T
To show that our analysis matches the training behaviors observed in SOTA empirical work (Li et al.
2021
Kurakin al. 2022, De et al., 2022 Tramer & Boneh 2020; Mehta et al. 2022 Yu et al.
2021 we minimize the first argument of G in (5.3), denoted as (B, μr d, L, Lo).
1.
[Train longer with larger noise] Fixing the expected batch size B, we see that X is decreasing in T. Hence larger T and consequently larger are preferred.
2. [Larger batch size helps] Fixing number of iterations T or epochs E BT/n, we see that X is decreasing in B. Hence larger B and consequently larger a are preferred.
3. [Pretraining is critical] Pretraining can boost the DP accuracy through much smaller initial loss Co and from smooth (small L) and flat (small &. c.f. Figure 7[eft) initialization.
4. [Learning rate needs tuning] The optimal learning rate by minimizing C..4 is
(Co C.)
This indicates that one should use larger learning rate for smaller model d, weaker privacy (larger I or small c), or smaller iteration budget T. Interestingly, the optimal choice of learning rate is independent to (expected) batch-size B.
EXPERIMENTS
We evaluate our automatic DP training on image classification, sentence classification, and table-to- text generation tasks. Detailed settings including hyperparameters can be found in Appendix G
6.1
IMAGE CLASSIFICATION
For MNIST/FashionMNIISTT we use the same setup as in (Papernot et al. 2021 Tramer & Boneh,
2020
Shamsabadi & Papernot 202I) with simple CNN. For CIFARTO, IO, use the same setup
as in
Tramer& & Boneh| (2020) with pretrained SimCLRv2 (Chen et al. 2020a). For ImtteNette, a the same setup in Klause et al.) (2022) without the learning rate decay. For CelebA (Liu et al. 2015). the real human face dataset, we train ResNet9 (He et al. (016)) with group normalization replace the batch normalization. Notice that
10-class sub-task of ImageNet (Deng al. 2009),
CelebA contains high-resolution (178x218) images, each with 40 labels. We consider CclebA for either multi-class classification on one label. e.g. 'Smiling' and 'Male', or for multi-label/multi problem to learn all labels simultancously.
Accuracy %
Task
Model
(c,
Abadi's clipping
AUTO-S clipping
oc)
MNIST
4-layer CNN
(3, le-5)
98.04 + 0.09
98.15 = 0.07
99.11 + 0.07
FushionMNIST
layer CNN
(3, le-5)
86.04 1 0.26
86.36 1 0.18
89.57 1 0.13
CIFARIO pretrained
SimCLRv2
(2, le-5)
92.44 + 0.13
92.70 = 0.02
94.42 +: 0.01
ImageNette
ResNet9
(8, 1e-4)
60.29 + 0.53
60.71 + 0.48
71.11 1: 0.37
CelebA [Smiling]
ResNe19
(8, 5e-6)
90.75 + 0.11
91.08 + 0.08
92.61
+ 0.20
CelebA [Malc]
ResNe19
(8,6e-6)
95.54 = 0.14
95.70
= 0.07
97.90
0.04
CelebA Multi-label
ResNe19
(3, 50-6)
86.81 1 0.03
87.05
0.01
90.30
0.02
CelchA Mubel
ResNe19
5e=66
87.52 + 0.15
87.58
0.04
90.30 0.02
Table 1: Average test accuracy and 95% confidence interval on image tasks over 5 runs.
In Table we observe that AUTO-S clipping outperforms existing clipping in all datasets with sta- tistical significance. Interestingly, the standard deviation from different runs is smaller for automatic DP optimizers, indicating better reproducibility and stability. We additionally experiment 40 binary classification problems on CelebA with respect to each label, and observe that the mean accuracy further improves to 91.63% at for AUTO-S (see Appendix[].
6.2 SENTENCE CLASSIFICATION
On five benchmark language datasets (MNLI(m/mm)) Williams et al. 2018), QQP et al. 2017
QNii(Rajrrrkar et al. 20166), SST2 et al.] 2013). compare automatic DP training with reparameterized gradient perturbation (RGP, (Yu et al. (02T)) and full full-parameter finetuning (full, (Li et al. 202Th using RoBERTa models (Liu al." 2019b). These methods the same
8

---

Under review as a conference paper at ICLR 2023
experimental setup. For language models, our automatic training is based on the codebase of Li
ct al. 2021
Method
3
6
Do (non- DP)
MNLI
QQP
QNLI SST2
MNLI
QQP QNLI
SST2
MNLI
QQP QNLI SST2
RGP Yuu al. 2021
80,5/79.6
85.5 87.2 91.6
83,6/83.2
89.3 91.3 92.9
full (La (Lretal 2021
82.45/82.99
85.56 87.42 91.86
83.20/83.46
86.08 87.94 92.09
AUTOVV
81.21/82.03
84.72 86.56 91.86 82. 18/2..44
86,23
87.24 92.09
85.91/86.14 87.34 91.40 94.49
full AUTO-S
83.22/83.21
85.76 86.91 92.32
83.82/83.55
86.58
87.85 92.43
Table 2: Test accuracy on language tasks with RoBERTa-bass (12 blocks, 125 million parameters).
(non-DP) QQP QNLI SST2
Method
MNLI QQP ONLI SST2
MNLI
QQP
QNLI SST2
MNLI
RGP (Yu al
2021
86, 1/860
86.7 90.0 93.0
full (L al.
2021
86,43/86.46
86.43 90.76 93.04
87.02/87.24
87.47 91.10 93.81
füll AUTO-V
15.33/85.61
86.61 89.99 93.12
85.91/86.1 10 86,86 90.55 93.35
90.33/90.03 87,90 93.61 96.21
full AUTO-S
86.27/86.67 86.76 91.01 93.92
87.07/87.16 87.47 91.45 94.61
Table 3: Test accuracy on language tasks with RoBERTa-large (24 blocks, 355 million parameters).
In Table and Table we note that full parameter finetuning with AUTO-S outperforms or at least matches SOTA on all tasks. We use exactly the same hyperparameters as in Li et al. (202T)
6.3 TABLE-TO-TEXT GENERATION
We compare our automatic DP training with a variety of fine-tuning methods, for table-to-ter toxtet gen- eration task on E2E dataset (Dusek et al.) 2020). where the goal is generate texts about different aspects of restaurant's 's data. We measure the success on this task by BLEU, ROUGE-L (in Table METEOR, NIST, CIDEr (extended in Table with higher value meaning better model quality.
DF
GPT2
GPT2
GPT2
Metric
guarantce
large
medium
Tall
Till
TGll
Fall
Tall
LoRA
RGP
prefi
top2
Retrai
AUTO-S
AUTO-S
AUTO-S
AUTO-V
Li 2021
Hu 2021
Yu al.
2021
Li & Likin 2021
64.180
61.340
61.519
61.319
42:772
25,920 15.457
BLEU
== 8
64.640
64,220
63.600
63.189
63.189
63,389
58.455
49,263
26.885 24.247
Don: DP
66.340
68.500
69.463
69.463
69.463
69.682
68.328
68.345
65.752 65.731
67.857
67.071
65,872
65.670
65.670
65.773
65.560
58.964
44,536 35.240
ROGUE-
68.968
67.533
67.073
66.429
66.429
67.525
65.080
60.730
46,421
39:951
Don-DP
70.384
71:458
71.359
71.359
71.359
71.709
68.844
70,805
68.704 68.751
Table 4: Test performance on E2E dataset with GPT2. Additional performance measures are
cluded in T8 The best two GPT2 models for each row are marked in bold.
Competitive methods include low-rank adaption (LoRA), prefix-tuning (prefix), RGP, only fine- tuning the top 2 Transformer blocks (top2), and training from scratch (retrain), as were recorded in [i et al. (202T). Again, we use the exactly the same hyperparameters as in Li et al. (202T). For GP22 million parameters), GPT2 medium (355 million), and GPT2 large (774 million). Table shows that AUTO-S is scalable with stronger performance on larger models. Our automatic full- parameter finetuning has the best overall performance. Additionally, we highlight that AUTO-S and methods like LoRA are not mutually exclusive and can be combined to yield strong performance, since AUTO-S modifies the optimizers and LoRA modifies the architecture.
7 DISCUSSION
In this work, we proposed the automatic clipping drop-in replacement to the standard per- example clipping differentially private training. This is the first technique that climinate the need tune the clipping threshold R, thus making DP deep learning as easy regular learning. Our AUTO-S method enjoys both theoretical guarantee of convergence in non-convex problems (under various conditions), and strong empirical performance that advances the state-of-the-art (SOTA) of DP learning on both computer vision and language tasks.
We excited about the future of automatic DP training, especially along with other working tech- niques. Notably, our automatic clipping applies compatibly with general optimizers (e.g. (Bu et al. 2021a Du & Mil 202I). ) clipping styles (all-layer or per-layer), architecture modifications (e.g. LoRA, RGP, prefix), and data augmentation (c.g. adversarial training (Goodfellow et al.) 2015) and multiple augmentation De et al. (202)). Thus, we expect to achieve comparable results all SOTA in lightweight fashion.
$See httptsss//ggttheee com/ Appendix K.
http://///..e..oooooooooooo/o/te orapre/rnm/pttttttttrrrrrrrrrme and the detailed modification

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Zeyuan Allen-Zhu. Natasha 2: Faster non-convex optimization than sgd. Advances in neural infor- mation processing systems, 31, 2018.
Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. Advances in Neural Information Processing Systems, 34, 2021.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metrie for MT evaluation with improved corrllation with human judgments. In Proceedings of the ACL Workshop In- trinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL
https: wwww www.wb..ww.b.................... org/ http://ww.ooo/ottooooooooo//0000
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp. 560-569. PMLR, 2018.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-3) --12,, 2018.
Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard data science review, 2020(23), 2020.
Zhiqi Bu, Sivakanth Gopi, Janardhan Kulkarni, Yin Tat Lee, Hanwen Shen, and Uthaipon Tan- tipongpipat. Fast and memory efficient differentially private-sgd via jl projections. Advances in Neural Information Processing Systems, 34, 2021a.
Zhiqi Bu, Hua Wang, and Qi Long. On the convergence and calibration of deep learning with differential privacy. arXiv preprint arXiv:2106.07830, 2021b.
Mark Bun, Cynthia Dwork, Guy Rothblum, and Thomas Steinke. Composable and versatile privacy via truncated cdp. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pp. 74 86, 2018
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. (633-2650,, 2021.
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Work- shop (ITA), pp. 1-10. IEEE, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspective. Advances in Neural Information Processing Systems, 33:13773-13782, 2020b.
Soham De. Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlock- ing high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204. 13650, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
10

---

Under review as a conference paper at ICLR 2023
Jinshuo Dong, Aaron Roth, and Weijie Su. Gaussian differential privacy. Journal of the Royal Statistical Society Series B, 84(1):3-37, 2022.
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
Jian Du and Haitao Mi. Dp-fp: Differentially private forward propagation for large models. arXiv preprint 12.::2112.114300 2021.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal machine learning research, 12(7), 2011.
Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. Evaluating the State-of-the-Art of End-to- End Natural Language Generation: The E2E NLG Challenge. Computer Speech & Language, 59:123-156, January 2020. doi:
Cynthia Dwork. Differential privacy: A survey of results. In International conference on theory and applications of models of compulation, pp. 1-19. Springer, 2008
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284. Springer, 2006.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211 407, 2014.
Saced Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochas- tic programming. SIAM Journal Optimization, 23(4):2341-236 2013.
Aditya Golatkar, Alessandro Achille, Yu-Xiang Wang, Aaron Roth, Michael Kearns, and Stefano Soatto. Mixed differential privacy in computer vision. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8376-8386, 2022.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://
arxiv. www/abs////ooooooooro....o.....r //t.//111..555
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems, 34, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021.
Shankar Iyer, Nikhil Dandekar, and Kornel Csernai.
First quora dataset
release: Question pairs, 2017.
URL https://data. quora. com/
Fir Past-Quora-Dataaae
-sllease-ouestion---
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Helena Klause, Alexander Ziller, Daniel Rueckert, Kerstin Hammernik, and Georgios Kaissis. Differentially private training of residual networks with scale normalisation. arXiv preprint arXiv:2203.00324. 2022.
Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using fft. In International Conference on Artificial Intelligence and Statistics, pp. 2560-2569. PMLR, 2020.
Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at imagenet scale with differential privacy. arXiv preprint 2tti:22011113220 2022.
11

---

Under review as a conference paper at ICLR 2023
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Can- wen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https: http::::/aaalattio Lantho wgy.org......................... wttp://2221.mmm.eeeeeeeeeee .wnnl--deeee 21
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
pp. 4582 4597, 2021.
Xuechen Li. Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations, 2021.
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries, In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: www. acwweb..wb......... org/ http//oohhhlllcc-coooo -0113
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 197.11907..1692, 2019b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) December 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Stephan Mandt, Matthew D Hoffmmnn, and David M Blei. Stochastic gradient descent approximate bayesian inference. Journal of Machine Learning Research, 18:1-35, 2017.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Leamiing differentially private recurrent language models. In International Conference on Learning Representatiinns 2018.
Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205.02973, 2022.
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer foundations sympo- sium (CSF), pp. 263-275. IEEE, 2017.
Diganta Misra. Mish: A self regularized non-monotonic activation function. BMVC 2020, 2019.
Yurii E Nesterov. method for solving the convex programming problem with convergence rate (1/k²2) 2). In Dokl. akad. nauk Sssr, volume 269, pp. 543 343-547,,1 1983.
Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy. In International Conference Learning Representations, 2021.
Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Úlfar Erlingsson. Tem- pered sigmoid activations for deep learning with differential privacy. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9312-9321, 2021.
12

---

Under review as a conference paper at ICLR 2023
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02. pp. 311-318, USA, 2002. Association for Computa- tional Linguistics. doi: 00.3115//073388311733333 URI. https::/ org or..11.3115//
1073083 10731351
Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank Reddi, and Sanjiv Kumar. Adaclip: Adaptive clipping for private sgd. arXiv preprint arXiv: 1908.07488 2019.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr compu- tational mathematics and mathematical physics, 4(5):1-17, 1964.
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838 855, 1992.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-
channel normalization and weight standardization. arXiv preprint arXiv: 10520, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016.
Seyed Omid Sadjadi, Timothee Kheyrkhah, Audrey Tong, Craig S Greenberg, Douglas A Reynolds, Elliot Singer, Lisa Mason, Jaime Bernandez-Cordent et al. The 2017 nist language recognition evaluation. In Odyssey, pp. 82 02-89,20 2018.
Ali Shahin Shamsabadi and Nicolas Papernot. Losing less: A loss for differentially private deep learning. 2021.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiricul methods in natural language pro- cessing, pp. 1631-1642, 2013.
Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. In International Conference on Artificial Intelligence and Statistics, pp. 1226-1233 PMLR, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Languuge Technologies, Volume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL http: http://ccwwbbb.om.. org/ anthology /N18 1101
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations, 2020.
13

---

Under review as a conference paper at ICLR 2023
Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint 1008.0170..03000 6(12):6, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020. /RLLattts///ppeeeviieeee net/ http:///ooooo.....eyyess........
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library PyTorch. arXiv preprint
arXiv:2109.122 12298, 2021.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie- Yan Liu. Large scale private learning via low- rank reparametrization. In International Conference on Machine Learning, pp. 12208-12218. PMLR, 2021.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5700 2012.
Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics, pp- PMLR, 2022.
14


Reviews:
[{"review_id": "PNCmG1om9-O", "reviewer": "Reviewer_6ygh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper suggests two new methods for doing clipping in DP optimization automatically, with the goal of avoiding the need to tune the clipping norm while minimally reducing (or perhaps even improving) model accuracy. The authors propose two methods. AUTO-V and AUTO-S. AUTO-V effectively rescales every gradient norm to 1, but runs into a lazy-region issue where the mean clipped gradient can be very close to even if the mean unclipped gradient are not. AUTO-S remedies this by instead multiplying each gradient g by 1/(||g|| + gamma), where gamma is a small positive constant, which gives large gradients slightly more magnitude than smaller gradients and avoids the lazy region issue. The authors give a survey of clipping thresholds in past work which motivates these clipping methods: Most past work does well when the clipping threshold is sufficiently small such that most gradients are clipped. If all gradients were clipped, then the gradient update using standard clipping can combine the learning rate and clipping norm hyperparameters, and reduces to what AUTO-V does anyway.\n\nThe authors show a theoretical convergence result for AUTO-S that applies to non-convex smooth functions. This analysis also shows the necessity of using AUTO-S over AUTO-V. Noticeably, the convergence proof has the same asymptotic dependence on T as that of non-private SGD.\n\nThe authors also run experiments using AUTO-S clipping to train image classification models, sentence classification models, and text generation models with DP. For image classification models, using the same setups as past work but replacing standard clipping with AUTO-S, the mean accuracy increases across a multitude of tasks. For sentence classification, AUTO-S improves upon the previous work of Li et al. for almost all settings, using the same hyperparameters and setup. For text generation, AUTO-S has the highest performance compared to past methods. ", "summary_of_the_review": "I recommend accepting the paper. Hyperparameter reduction is an important problem in practice, and the authors have done a very good job motivating the method they use to address this problem and making their results as well-studied and suitable for use in practice as existing methods. So I believe the paper can have high impact in practice. The empirical results in the paper are also fairly impressive, improving over the SOTA in performance despite that not being the main focus of the work necessarily. The aforementioned weakness in the paper I believe is easily fixed before the camera-ready version as well.", "strength_and_weaknesses": "I think the main strength of the paper is that it suggests a simple method for hyperparameter reduction backed by theoretical and empirical exploration that also has both impressive theoretical and empirical performance. The authors do a good job using the explorations in the first few pages of the paper to motivate the eventual automatic clipping method they arrive at in the paper. The results in the paper feel very complete and comprehensive; there are theoretical converge results, a wide range of empirical evaluations on benchmarks, and discussions on ease of implementation into existing codebases including code snippets, as well as lengthy discussions on each. Despite the main goal of the paper being to reduce the number of hyperparameters, the authors are also able to achieve empirical increases in accuracy using their method.\n\nI think the main weakness of the paper is that it is unclear from reading the paper just how much of an improvement in terms of hyperparameter reduction this paper gives over past approaches; in particular, I felt that Figure 14 is one of the most important parts of the paper (giving examples that quantify the effect on performance of varying gamma), and should have been discussed in more detail especially in the body of the paper. In particular, the authors state that \"deep learning optimizers are insensitive to the choice of gamma\", but in Figure 14, the effect of gamma on accuracy and the effect of clipping norm on accuracy seem to be within a small constant factor of each other (e.g., the maximum difference b/t two clipping norms that differ by 10 seems to be < 1% absolute accuracy, and for some settings two gammas that differ by 10 can differ in accuracy by half a percent). It seems then arguable in terms of hyperparameter reduction if AUTO-S with gamma = 0.01 is substantially improving over e.g., just fixing the clip norm to be 0.01 in DP-SGD. One could reasonably argue that this constant factor is substantial in practice, but this topic is somewhat swept over by the quoted statement which to me implies the difference is much more drastic.\n\nThat being said, I think since (i) as mentioned above, gamma has less of an effect than the clipping norm and one can argue the decrease of this effect is substantial (ii) the authors also include comparisons to AUTO-V (which truly has strictly fewer hyperparameters to tune) and still see comparable performance from AUTO-V, this is a minor weakness. Furthermore, I think it's easily fixable by just adding a few sentences to the main body of the paper.\n\nNitpick: In various places references probably should go in parentheses, i.e. (X et al. 2022) instead of X et al. (2022) for readability, please fix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IMHp-PvvM7L", "reviewer": "Reviewer_LbVy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of choosing the clipping threshold in DP-SGD routines and proposes two related modifications to the original fixed clipping technique of Abadi et al. They then show that the DP-SGD method with the proposed clipping has desirable theoretical properties in terms of convergence to stationary points. Importantly, they conduct extensive experiments to show that their method compares favourably to other schemes in practice.", "summary_of_the_review": "The paper tackles a practically important problem and present promising empirical results. It is also interesting that it is not a heurisitc but comes with some theoretical guarantees. However, the key ideas behind it are rather simple and a comparison with competing methods is currently missing.", "strength_and_weaknesses": "Strengths\n1. Clipping of gradients is an integral constituent to the practice of modern differentially private machine learning. Moreover, as the authors have pointed out, the downstream performance is usually very sensitive to the choice of the clipping threshold. This has resulted in many works on how to *automatically* set the clipping threshold without tuning it explicitly. The authors of this paper contribute to this practically important area of research. \n\n2. Extensive experiments:\nThe main goal of the paper is to obtain empirical improvement over prior works.\nThe authors propose two variants of their clipping technique, AUTO-V and AUTO-S, and perform extensive experiments on various benchmark tasks. The experimental results are promising and demonstrate better performance than compared techniques.\n\nWeaknesses\n\nThe key idea of the paper is very simple but in my view, its presentation feels rather obfuscated.\nBasically, the trick is normalize all gradients  (AUTO-V) or normalize it with an additive scalar in the denominator (which is standard in implementations of adaptive gradient methods like ADAM). This normalization explicitly controls the sensitivity, and so privacy follows. If the per-sample gradients have different norms, then this method scales all of them to have the same norm, so intuitively it seems that we would lose some information, however the authors show that it still works well in practice.\n\n\n**Theoretical results**:\nThe presentation of the theoretical results is convoluted for some reason -- I would have expected that the right hand side in Thm 4 and 5 are stated as an explicit function of problem parameters; instead they are written as a function F which is not defined in the main text. The authors say that has the same rate as SGD (which is true with respect to its dependence on $T$), but I am not sure if the dependence on other terms is same as SGD. This should be clarified.\nAlso, I am not sure about the remarks following Thm5.\nAren't all these observations could be made from any existing convergence result of this form (say DP-SGD)? So, why are they interesting in the context of the methods proposed? What am I missing here?\n\n**Connections to optimisation methods with normalized updates**: (Variants of) Normalized updates have been used in optimization; for instance, in parameter-free methods [CM20], and variance-reduced methods [FJLZ18] as well as in empirical works (see references in [CM20]).\nHowever, I did not find any discussion on this in the paper; I hope the authors add some to give context to the proposed approach.\nAlso, is the theoretical result and analysis significantly different from prior works? \n\n**Empirical Comparison with other adaptive clipping methods**: \nThe authors do not do any empirical comparison with prior adaptive clipping methods such as Andrew et al, Pichapati et al. The authors rightfully pointed out these introduce new (but arguably robust) hyper-parameters. However, they often come with default settings of these hyperparameters (such as the quanitile). So, why not compare using these default values of hyperparameters of all the methods while keeping the other hyperpameters (like learning rate) fixed?\n\n[CM2020]Momentum Improves Normalized SGD\n[FJLZ18]:SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FyQXLFa5y9M", "reviewer": "Reviewer_adTx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to replace the standard clipping function in DP-SGD, $Clip_R(g) = min(R/|g|, 1)$ with the alternative clipping function $Clip_{\\gamma}(g) = 1/(|g| + \\gamma)$. They argue that this alternative clipping scheme is easier to tune, and matches or exceeds the performance of the standard scheme on a range of datasets.", "summary_of_the_review": "The paper relies on the claim that the clipping threshold in DP-SGD is difficult to tune, however recent works have shown that this is easy in practice if one simply re-scales the clipped gradient by the clipping threshold. Unfortunately I therefore feel the contribution here is simply too small for a top-tier conference.", "strength_and_weaknesses": "Strengths:\n\n1) The proposed scheme is a natural alternative clipping scheme. \n2) The authors provide extensive experiments to confirm that it achieves similar performance to standard clipping\n\nWeaknesses:\n\n1) The core claim of the paper is that the clipping parameter R is difficult to tune, however a number of prior works have shown that this arises only because the choice of clipping norm changes the scale of the update. One can easily fix this by redefining $Clip_R(g) = min(1/|g|, 1/R)$. As De et al. show, after this transformation a wide range of small but finite values of R perform similarly well, and it is standard practice in recent works to simply set R=1.\n\n2) After this transformation, it is clear that the two clipping schemes are almost identical. Essentially the authors propose a smoothed version of the standard clipping function.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject