Title: Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep learning

differential privacy

optimization

hyper-parameter tuning
Abstract: Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping threshold $R$, however, is vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called automatic clipping, that eliminates the need to tune $R$ for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB and many others.
The automatic variants are as private and computationally efficient as existing DP optimizers, but require no DP-specific hyperparameters and thus make DP training as amenable as the standard non-private training. We give a rigorous convergence analysis of automatic DP-SGD in the non-convex setting, showing that it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric gradient noise assumption of the per-sample gradients (commonly used in the non-DP literature). We demonstrate on various language and vision tasks that automatic clipping outperforms or matches the state-of-the-art, and can be easily employed with minimal changes to existing codebases.

Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger

Abstract

Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping threshold R , however, is vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called automatic clipping, that eliminates the need to tune R for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB and many others. The automatic variants are as private and computationally efficient as existing DP optimizers, but require no DP-specific hyperparameters and thus make DP training as amenable as the standard non-private training. We give a rigorous convergence analysis of automatic DP-SGD in the non-convex setting, showing that it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric gradient noise assumption of the per-sample gradients (commonly used in the non-DP literature). We demonstrate on various language and vision tasks that automatic clipping outperforms or matches the state-of-the-art, and can be easily employed with minimal changes to existing codebases 1 .

1 Introduction

Deep learning has achieved impressive progress in a wide range of tasks. These successes are made available, in part, by the collection of large datasets, sometimes containing sensitive private information of individual data points. Prior works have illustrated that deep learning models pose severe privacy risks to individual subjects in the training data and are susceptible to various practical attacks. For example, machine learning services such as Google Prediction API and Amazon Machine Learning can leak membership information from the purchase records [65]; the GPT2 language models auto-complete texts that contain someone's full name, phone number, email address, etc., from the training data that it memorizes, if invoked by specific prefixes [11].

Differential privacy (DP) [24, 26, 25] is a formal definition of privacy that has been shown to prevent the aforementioned privacy risks in deep learning [1]. At a high level, the key difference between the DP deep learning and the standard one is whether the gradient is privately released. In other words, while the standard optimizers update on ∑ i g i , the DP optimizers update on the private gradient :

Here g i ∈ R d is the per-sample gradient of loss l i , N is the standard normal, σ is the noise multiplier, and R is the clipping threshold. The clipping function Clip : R d → R is defined such that ∥ g i · Clip ( g i ; R ) ∥ ≤ R . For instance, the DP-SGD in [1] is

In comparison to the regular training (1.2), two additional DP-specific hyperparameters R and σ need to be determined in DP learning (1.1). On the one hand, setting the noise multiplier σ is easy and can be derived analytically prior to the training. Whenever the privacy budget ( ϵ, δ ) is determined, one can apply off-the-shelf privacy accounting tools in Section 2.1 to determine σ , based on the subsampling probability p and the number of iterations T :

On the other hand, the choice of clipping threshold R is crucial to the performance of DP models, yet the hyperparameter tuning is much labor-intensive. Recent advances of DP deep learning on ImageNet [38] and on E2E datasets [41], using ResNet18 and GPT2 respectively, illustrate that the performance is very sensitive to R . We have reproduced their results in Figure 1. Observe that on ImageNet, ResNet18 can drop from the highest 45% accuracy to 31% if R is chosen 2 times larger, and to 0.1% if R is chosen 4 times larger. Similar drastic drop can also be observed in [38, Figure 3] even if the noise multiplier σ = 0 . Unlike the noise multiplier σ , the clipping threshold R cannot be inferred from the privacy budget ( ϵ, δ ) and have to be tuned. Consequently, DP training necessarily requires an expensive 2D grid search for ( R,η ) , like Figure 1, whereas the regular training only requires an easy 1D grid search for η . Even worse, the difficulty of tuning a per-layer clipping threshold vector [49], i.e. one clipping threshold for one layer, may increase exponentially as the number of layers increases.

To save the effort of tuning R , previous researches have proposed different approaches. In [3, 58, 28, 31], researchers advocate to use data-adaptive information to select R , such as a specified quantile of the gradient norm distribution. These adaptive clipping methods can be a little ad-hoc: they often replace the need to tune R by the need to tune one or more new hyperparameters, e.g. the quantile to use and the ratio to split the privacy budget between the quantile decision and the gradient perturbation. Another approach used by the practitioners is to replace the single 2D grid search by multiple cheaper 1D grid searches. For example, the researchers propose, in [38, Section 3.3] to fine-tune η with non-DP SGD, fix η and sweep over various values of the clipping threshold R with DP-SGD, then further fix R and do one more grid search on η . However, tuning R formally in a data-dependent way (e.g. through cross-validation) introduces additional privacy loss [55], and most existing empirical work does not privately conduct hyperparameter tuning.

We take a completely different route by proposing a new clipping principle that removes R , instead of coming up with methods to find the appropriate R . We term our method as automatic clipping and the DP optimizers using it as automatic DP optimizers . Our contributions are:

We propose the automatic clipping in (4.1) that expunges the clipping threshold from general DP optimizers, making DP training as amenable as regular training. In large-scale tasks (GPT-level) like Figure 1, our automatic clipping can reduce the cost of ablation study by 5 × 2 .

We show that automatic DP optimizers are as private and efficient as existing DP optimizers.

We show in Theorem 4 that automatic DP-SGD converges in the non-convex setting, at the same asymptotic convergence rate as the standard SGD. Our theoretical analysis successfully explains the training behaviors of deep learning in previous empirical works.

We demonstrate the superiority of automatic clipping on a variety of vision and language tasks, especially with large models including ResNet, RoBERTa and GPT2.

In Appendix K, we include simple code snippets that demonstrate how easy it is to switch from Abadi's clipping to our automatic clipping in popular codebases, e.g. Opacus and ObJAX.

2 Preliminaries

2.1 Differential Privacy

We consider the ( ϵ, δ ) -DP in Definition 2.1, where smaller ( ϵ, δ ) means stronger privacy guarantee.

),

Definition 2.1 ([25]) . A randomized algorithm M is ( ε, δ ) -differentially private (DP) if for any two neighboring datasets S, S ′ (i.e. if one can obtain S ′ by adding or removing one data point from S and for any event E ,

In words, DP restricts the influence of an arbitrary sample, so that the information contributed by such sample is limited and less vulnerable to privacy attacks. In deep learning, DP is achieved by applying the subsampled Gaussian mechanism to privatize the minibatch gradients during training.

As illustrated in Equation (1.1), the subsampled Gaussian mechanism involves (I) sampling a minibatch by including each data point iid with probability p (II) per-sample gradient clipping to bound the l 2 norm sensitivity at R and (III) adding independent Gaussian noise proportional to R and σ , where σ is derived from the privacy budget ( ϵ, δ ) . This can be realized by leveraging a variety of modern privacy accounting tools, such as Renyi DP (or moments accountant) [1, 51, 71], Privacy Loss distribution (Fourier accountants) [37, 30, 82], or Gaussian DP [19, 7].

2.2 Differentially Private optimizers with general clipping operations

Privately released stochastic gradients (through the Gaussian mechanism) can be used by various off-the-shelf optimizers, including DP-SGD in (1.3), DP-HeavyBall, DP-AdaGrad, DP-Adam, DPFedAvg/FedSGD [49], etc. To improve the performance of DP optimizers, previous researches on the per-sample clipping can be classified into two categories.

The first category, where the majority of researches lie in, works with Abadi's clipping and focuses on better design of R . To name a few examples, one can adaptively design R t for each iteration t [3, 58, 28], or design the per-layer clipping threshold vector R ∈ R L for L layers [1, 49] so as to apply a different clipping threshold for each layer.

Fewer works fall into the second category that proposes new clipping methods. In fact, any function Clip : R d → R satisfying ∥ Clip ( g ) · g ∥ ≤ R can serve as a valid clipping function besides Abadi's. For example, the global clipping [9] proposes Clip global ( g i ) := I ( ∥ g i ∥ < R ) to mitigate the bias of the private gradient and alleviate the mis-calibration issue of DP classifiers. Another example is the re-parameterized clipping [17], Clip re-param ( g i ) := min(1 / ∥ g i ∥ , 1 /R ) , which is equivalent to Abadi's clipping under a re-scaled learning rate. Our automatic clipping belongs to this category. We note that different clipping methods work orthogonally to optimizers, network architectures and gradient norm computation (see Section 8).

3 Motivation

3.1 Small clipping threshold often works best

0.1

0.1 * 2

2

0.1 * 2

4

0.1 * 2

6

0.1 * 2

8

Clipping norm

10

3

10

3

/2 2

10

3

/2 4

10

3

/2 6

10

3

/2 8

Learning rate

60.26

59.58

60.01

57.87

35.44

50.48

50.31

50.48

49.25

34.47

33.17

33.18

33.13

32.96

29.50

29.74

29.55

29.61

29.62

27.07

9.69

9.69

9.68

9.65

5.79

10

20

30

40

50

60

10

1

10 0

10 1

10 2

10 3

Clipping norm

8*10 1

8*10 0

8*10

1

8*10

2

8*10

3

Learning rate

43.0

2.0

0.1

0.1

0.1

22.0

45.0

0.1

0.1

0.1

2.1

22.0

45.0

11.0

0.09

0.29

2.2

22.0

24.0

0.11

0.1

0.25

2.1

13.0

0.44

5

10

15

20

25

30

35

40

45

One intriguing observation that we can make about the recent studies on DP learning with large models is that the state-of-the-art (SOTA) results are often achieved with very small clipping threshold R . This observation is consistent in both vision and language tasks. In [41], GPT2 (about 800 million parameters) and RoBERTa models (over 300 millions parameters) achieve the best results under DP on QNLI, MNLI, SST-2, QQP, E2E, and DART datasets, with each per-sample gradient clipped to length R = 0 . 1 . In [38, 17, 50], ResNets and Vision Transformers achieve the best DP results on ImageNet with R = 1 ; in [68], the best DP results on CIFAR10 use R = 0 . 1 with ResNeXt-29 and SimCLRv2 [13]. The effectiveness of small clipping threshold together with proper learning rate is depicted in Figure 1.

Intuitively, smaller R implies that the Abadi's clipping (3.1) is effective, which means min ( R/ ∥ g i ∥ , 1 ) = R/ ∥ g i ∥ . Given that the clipping threshold R is so small compared to the number of parameters in large models, and that strong DP is guaranteed when the number of training iterations is small (i.e. ∥ g i ∥ has not converged to small values yet), we expect and empirically observe that the clipping happens on a large proportion of per-sample gradients at all iterations. For instance, we find in the GPT2 generation experiments in [41] that 100% of per-sample gradients are clipped at all iterations; in classification tasks such as QQP, QNLI, and MNLI, the percentage of clipping is about 20 ∼ 60% on average (more details in Appendix H.1).

3.2 Per-sample gradient normalization as new clipping

In the small clipping threshold regime, we can approximately view

and thus derive a novel private gradient ∑ i R g i ∥ g i ∥ + σR · N (0 , I ) . Here AUTO-V stands for the vanilla automatic clipping, which essentially performs the normalization on each per-sample gradient. As a specific example, we can write the R -dependent automatic DP-SGD as

We may view our AUTO-V clipping as to maximize the dot-product similarity (a commonly used similarity measure, e.g. in the attention block in transformers [69]) between the clipped gradient and the regular gradient. Suppose we want to

Note that the constraint is a sufficient condition for clipping, as discussed in Section 2.2. It is not hard to see that the optimal clipping factor (though violating DP guarantee 3 ) regarding (3.3) is

If the per-sample gradients are indeed concentrated in the sense ∀ i, ⟨ g i , ∑ j g j ⟩ ≥ 0 , then AUTO-V is the optimal per-sample gradient clipping. We compare with Abadi's clipping in Figure 2, where this similarity is significantly magnified by our AUTO-V clipping. In fact, the dot-product similarity in (3.3) closely resembles the convergence of DP optimization for Theorem 4 in (C.2).

0

25

50

75

100

125

150

175

200

Iteration

0

50000

100000

150000

200000

250000

300000

350000

Dot product

AUTO-V clipping

Abadi clipping

0

25000

50000

75000 100000125000150000175000200000

Dot product

0

10

20

30

40

50

60

70

Count

AUTO-V clipping

Abadi clipping

3.3 Stability constant breaks scale-invariance and remains stationary

One potential drawback of AUTO-V clipping is that all gradients lose their magnitudes information completely, since ∥ g i · Clip AUTO-V ( g i ; R ) ∥ = R, ∀ i . This scaleinvariance in AUTO-V and partially in Abadi's clipping (when ∥ g i ∥ > R ) leads to the "lazy region" issue: the parameters will not be updated by DP-GD even if the true gradients are non-zero. In Figure 3, we illustrate such issue in a logistic regression 4 for AUTO-V and Abadi's clipping, when the trainable parameter θ ∈ [ -2 , 2] , as the gradients from two classes cancel each other.

10

5

0

5

10

0.004

0.002

0.000

0.002

0.004

standard grad

To preserve the magnitude information and thus escape the lazy region, we propose the AUTO-S clipping, with a positive stability constant γ :

We visualize in Figure 5 that AUTO-S allows larger per-sample gradients to have larger magnitudes after the clipping, while still allowing smaller gradients to vanish after 'clipping'. That is, as g i → 0 , the existence of γ allows the clipped gradient C i g i → g i /γ rather than having a magnitude R as in AUTO-V. We elaborate this point in Section 4.3. This is critical in our convergence analysis and allows DP-SGDAUTO-S (but not DP-SGDAUTO-V) to converge to zero gradient norms in Section 5.

4 Automatic DP Training

One may wonder why our clipping (3.1)(3.5) is automatic at all, if the hyperparameter R is still present and there is an additional parameter γ to choose. It turns out that any constant choice of R > 0 is equivalent to choosing R = 1 , and common deep learning optimizers are insensitive to the choice of γ (e.g. for any γ > 0 , we show that the gradient norm converges to zero at the same asymptotic rate in Theorem 4; see also the ablation study in Figure 15). Consequently, we set γ = 0 . 01 as the default. Specifically, let us redefine the R -independent clipping function:

With this clipping, we can design automatic DP optimizers similar to (1.1):

Clearly, the new private gradient ˆ g t from our automatic clipping is R -independent, in contrast to the one used in (1.1). A concrete example (in the case of γ = 0 ) that is comparable to (3.2) will be

Leveraging the private gradient ˆ g t in (4.2), we can train DP neural networks without tuning DPspecific hyperparamters R and σ , as demonstrated in Algorithm 1.

Algorithm 1 Automatic Deep Learning with DP

Parameters: initial weights w 0 , learning rate η t , sampling probability p , number of iterations T .

1: Compute σ such that ϵ Accountant ( δ, σ, p, T ) ≤ ϵ from any privacy accountant.

2: for iteration t = 1 , · · · , T do

3: Sample a batch B t by including each data point i.i.d. with probability p

4: Apply automatic clipping to per-sample gradients { g i } i ∈ B t : ˆ g i = g i / ( ∥ g i ∥ 2 +0 . 01) .

6: Update w t by any optimizer on the private gradient ˆ g with learning rate η t .

5: Add Gaussian noise to the sum of clipped gradients: ˆ g = ∑ i ˆ g i + σ · N (0 , I ) .

We will elaborate two distinct reasons in the next sub-sections for the following statement:

which expunges the DP hyperparameters, only leaving us the regular hyperparameters such as learning rate, weight decay, etc. The significant save in the tuning effort is illustrated in Figure 4.

0.001

0.01

0.1

1

10

AUTO-S

Clipping norm

0.002

0.001

0.0005

0.0002

0.0001

Learning rate

84.3 85.0 84.5 84.7 81.9 85.0

90.6 90.8 90.6 90.8 91.3 90.7

92.0 92.3 92.0 92.3 92.7 92.7

91.3 91.4 91.4 91.9 92.3 91.4

90.9 90.9 91.2 91.6 92.1 91.1

0.82

0.84

0.86

0.88

0.90

0.92

0.001

0.01

0.1

1

10

AUTO-S

Clipping norm

0.002

0.001

0.0005

0.0002

0.0001

Learning rate

67.0 60.8 75.8 75.9 50.5 64.4

84.3 84.9 85.2 84.0 80.0 85.1

84.3 84.3 84.4 84.6 84.2 84.4

82.3 82.3 82.3 82.4 82.5 82.2

79.2 79.4 79.4 79.3 79.2 79.5

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.001

0.01

0.1

1

10

AUTO-S

Clipping norm

0.002

0.001

0.0005

0.0002

0.0001

Learning rate

47.8 47.8 47.8 47.8 47.8 47.8

57.0 47.8 47.8 47.8 47.8 47.8

77.7 77.8 78.0 78.4 78.2 78.0

76.7 77.2 77.1 77.2 77.2 77.1

74.1 74.5 74.7 74.9 75.0 74.6

0.50

0.55

0.60

0.65

0.70

0.75

4.1 Non-adaptive optimizer couples clipping threshold with learning rate

With R -dependent automatic clipping, DP-SGD becomes

We can view η effective ≡ ηR as a whole: increasing R has the same effect as increasing η , which explains the diagonal pattern in Figure 1(lower plot) where DP-SGDAbadi is applied with small clipping threshold. We extend to general non-adaptive optimizers in Theorem 1 5 .

Theorem 1. Non-adaptive R -dependent automatic DP optimizers (including SGD, Heavyball[59] and NAG[54]), with learning rate η and weight decay λ , is equivalent to R -independent automatic DP optimizers, with learning rate ηR and weight decay λ/R .

4.2 Adaptive optimizer can be insensitive to clipping threshold

Adaptive automatic DP optimizers are different than the non-adaptive ones, as the clipping threshold cancels out instead of being coupled with learning rate. To see this, we scrutinize DP-AdamAbadi (which is similar to DP-AdamAUTO-V) in Figure 1(upper plot), where columns to the left are almost identical. Further evidence is observed in [50, Table 5] that shrinking R has zero effect on LAMB. We now give a simple explanation using AdaGrad [22]:

where g t = ∑ i g t,i is the gradient sum. In R -dependent DP-AdaGradAUTO-V, the private gradient is R ˆ g t in place of the standard gradient sum g t :

We generalize to other adaptive optimizers in Theorem 2 and to the per-layer clipping style in Appendix B.3.

Theorem 2. Adaptive R -dependent automatic DP optimizers (e.g. AdaGrad[22], AdaDelta[79], AdaMax/Adam[35], NAdam[20], RAdam[43], LARS[75], LAMB[76]), with learning rate η and weight decay λ is equivalent to R -independent automatic DP optimizers with learning rate η and weight decay λ/R . With decoupled weight decay[46], R -dependent automatic DP-AdamW is equivalent to R -independent automatic DP-AdamW with the same η and λ .

4.3 Automatic clipping is equally private and maximizes utility

In Theorem 3 (proved in Appendix A), we show that the new private gradient ˆ g t in (4.2) has the same level of privacy guarantee as the existing one in (1.1), since the global sensitivity remains the same (see Figure 5). We note that as long as γ > 0 , the magnitude information of per-sample gradients is preserved by AUTO-S, in the sense that ∥ g i ∥ > ∥ g j ∥ ⇐⇒ ∥ C i g i ∥ > ∥ C j g j ∥ , whereas this can be violated in both the AUTO-V and Abadi's clipping (as depicted by the flat curve in Figure 5 when ∥ g i ∥ > 1 ).

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

||

gi

||

0.0

0.2

0.4

0.6

0.8

1.0

Norm of clipped gradient

AUTO-S,

=0.001

AUTO-S,

=0.01

AUTO-S,

=0.1

AUTO-S,  =1

Abadi

Additionally, note that when γ is small, almost all data points 'max out' the signal relative to the amount of noise we add. To say it differently, for the same amount of noise, AUTO-S with small γ allows more signal to be pushed through a differentially private channel. Towards the end of the training, i.e., at the limit when ∥ g i ∥ → 0 for all i , then we have ∑ i g i ∥ g i ∥ + γ → 1 γ ∑ i g i . In words, the clipped gradients become closer to the standard SGD, thus do not suffer from the instability of AUTO-V.

Theorem 3. Under the noise multiplier σ , number of iterations T , subsampling probability B/n , DP optimizers using AUTO-V or AUTO-S clipping satisfy ( ϵ Accountant ( δ, σ, B/n, T ) , δ ) -DP, where ϵ Accountant is any valid privacy accountant for DP-SGD under Abadi's clipping.

5 Convergence analysis of DP-SGD with automatic clipping

5.1 Convergence theory of DP-SGD to stationary points

We highlight that automatic clipping can be more amenable to analysis than Abadi's clipping in [14], since we no longer need to decide whether each per-sample gradient is clipped.

To analyze the convergence of automatic DP-SGD (4.2) in the non-convex setting, we follow the standard assumptions in the SGD literature [27, 2, 6], including a symmetry assumption on the gradient noise, which is empirically verified in [14, Figure 3] and commonly used in the standard non-DP literature [48, 66, 12, 73]. We refer the curious readers to Appendix E.5 for details.

Assumption 5.1 (Lower bound of loss) . For all w and some constant L ∗ , we have L ( w ) ≥ L ∗ .

Assumption 5.2 (Smoothness) . Let g ( w ) denote the gradient of the objective L ( w ) . Then ∀ w , v , there is an non-negative constant L such that

Assumption 5.3 (Gradient noise) . The per-sample gradient noise ˜ g t,i -g t is i.i.d. from some ditribution such that

and ˜ g t,i is centrally symmetric about g t in distribution: ˜ g t,i -g t D = g t -˜ g t,i .

We show in Theorem 4 that DP-SGD with AUTO-S clipping allows the true gradient norm to converge to zero, though the clipped gradient may still be biased, but not so with AUTO-V clipping.

Theorem 4. Under Assumption 5.1, 5.2, 5.3, running DP-SGD with automatic clipping for T iterations and setting the learning rate η ∝ 1 / √ T give 6

Here · · · represents the first argument of G , and G is increasing and positive. As T →∞ , we have min t E ( ∥ g t ∥ ) = O ( T -1 / 4 ) for AUTO-S, the same rate as the standard SGD given in Theorem 9.

10 1

10 3

10 5

10 7

10 9

10 11

Number of iterations

T

3×10 1

4×10 1

6×10 1

Upper bound of min

t

(||

gt

||)

r

+

, r=0.5

r

+

, r=0.6

r

+

, r=0.7

r

+

, r=0.8

r

+

, r=0.9

r

+

, r=0.99

10 1

10 3

10 5

10 7

10 9

10 11

Number of iterations

T

10 1

10 2

Upper bound of min

t

(||

gt

||)

r

+

, r=0.2

r

+

, r=0.5

r

+

, r=0.9

r

+

, r=1.1

r

+

, r=2.0

r

+

, r=3.0

10 1

10 3

10 5

10 7

10 9

10 11

Number of iterations

T

10

2

10

1

10 0

10 1

Upper bound of min

t

(||

gt

||)

DP-SGD AUTO-S

DP-SGD AUTO-V

standard SGD

Remark 5.4. We show in Theorem 6 and in Figure 6 that the upper bound (5.2) has G ≥ ξ for AUTO-V ( γ = 0 ), and G only reduces to zero for AUTO-S ( γ > 0 ). We provide real data evidence in Figure 14 that strictly positive γ reduces the gradient norm significantly.

5.2 Analysis of factors affecting the convergence

We now analyze the many factors that affect the convergence in Theorem 4, from a unified viewpoint of both the convergence and the privacy.

We start with the stability constant γ and the learning rate η t , both only affect the convergence not the privacy. We empirically observe in Figure 8 that small γ benefits the convergence at initial iterations (when the privacy guarantee is strong) but larger γ converges faster asymptotically. For η t , the optimal is in fact the miminizer of the hyperbola in (C.5), that is unique and tunable.

Next, we focus on the hyperparameters that affect both convergence and privacy: the batch size B , the noise multiplier σ , and the number of iterations T . These hyperparameters have to be considered along the privacy-accuracy tradeoff, not just from a convergence perspective.

Recall that given a fixed privacy budget ( ϵ, δ ) , we rely on modern privacy accountant for computing the appropriate combinations of parameter σ, T, B . The exact expression of the bound as a function of ( ϵ, δ ) is somewhat messy. For this reason, we illustrate our analysis in terms of the surrogate parameter µ for µ -GDP [19], which implies ( ϵ, δ ) -DP with ϵ = µ 2 + µ √ 2 log(1 /δ )) . [7] showed that DP-SGD's privacy guarantee asymptotically converges to µ -GDP (as T →∞ ) with µ = B n √ T ( e 1 /σ 2 -1) . We can alternatively leverage ρ -tCDP [10] for similar conclusions, using ρ in place of µ 2 in (5.3).

Theorem 5. Under Assumption 5.1, 5.2, 5.3, fixing the asymptotic µ ( ϵ, δ ) -GDP parameter, running DP-SGD with automatic clipping for T iterations and setting the learning rate η ∝ 1 / √ T give

To show that our analysis matches the training behaviors observed in SOTA empirical work [41, 38, 17, 68, 50, 78], we minimize the first argument of G in (5.3), denoted as X ( B,T,µ,d,L, L 0 ) .

[Train longer with larger noise] Fixing the expected batch size B , we see that X is decreasing in T . Hence larger T and consequently larger σ are preferred.

[Larger batch size helps] Fixing number of iterations T or epochs E = BT/n , we see that X is decreasing in B . Hence larger B and consequently larger σ are preferred.

[Pretraining is critical] Pretraining can boost the DP accuracy through a much smaller initial loss L 0 and from a smooth (small L ) and flat (small ξ , c.f. Figure 8(left)) initialization.

[Learning rate needs tuning] The optimal learning rate by minimizing (C.5) is √ ( L 0 -L ∗ ) µ 2 n 2 L ( µ 2 n 2 + dT ) . This indicates that one should use larger learning rate for smaller model d , weaker privacy (larger µ or small ϵ ), or smaller iteration budget T .

6 Experiments

We evaluate our automatic DP training on image classification, sentence classification, and table-totext generation tasks. Detailed settings including hyperparameters can be found in Appendix G.

6.1 Image classification

For MNIST/FashionMNIST, we use the same setup as in [56, 68, 64] with a simple CNN. For CIFAR10, we use the same setup as in [68] with pretrained SimCLRv2 [13]. For ImageNette, a 10-class sub-task of ImageNet [18], we use the same setup as in [36] without the learning rate decay. For CelebA [45], the real human face dataset, we train ResNet9 [32] with group normalization to replace the batch normalization. Notice that CelebA contains high-resolution (178x218) images, each with 40 labels. We consider CelebA for either multi-class classification on one label, e.g. 'Smiling' and 'Male', or for multi-label/multi-task problem to learn all labels simultaneously.

In Table 1, we observe that AUTO-S clipping outperforms existing clipping in all datasets with statistical significance. Interestingly, the standard deviation from different runs is smaller for automatic DP optimizers, indicating better reproducibility and stability. We additionally experiment 40 binary classification problems on CelebA with respect to each label, and observe that the mean accuracy further improves to 91.63% at ϵ = 8 for AUTO-S (see Appendix J).

6.2 Sentence classification

On five benchmark language datasets (MNLI(m/mm)[72], QQP[34], QNLI[62], SST2[67]), we compare our automatic DP training with re-parameterized gradient perturbation (RGP, [78]) and full-parameter finetuning (full, [41]) using RoBERTa models [44]. These methods use the same experimental setup. For language models, our automatic training is based on the codebase of [41].

In Table 2 and Table 3, we note that full parameter finetuning with AUTO-S outperforms or at least matches SOTA on all tasks. We use exactly the same hyperparameters as in [41].

6.3 Table-to-text generation

We compare our automatic DP training with a variety of fine-tuning methods, for table-to-text generation task on E2E dataset [23], where the goal is to generate texts about different aspects of a restaurant's data. We measure the success on this task by BLEU, ROUGE-L (in Table 4), METEOR, NIST, CIDEr (extended in Table 8), with higher value meaning better model quality.

Competitive methods include low-rank adaption (LoRA), prefix-tuning (prefix), RGP, only fine-tuning the top 2 Transformer blocks (top2), and training from scratch (retrain), as were recorded in [41]. Again, we use the exactly the same hyperparameters as in [41]. For GPT2 (124 million parameters), GPT2 medium (355 million), and GPT2 large (774 million), Table 4 shows that AUTO-S is scalable with stronger performance on larger models. Our automatic full-parameter finetuning has the best overall performance. Additionally, we highlight that AUTO-S and methods like LoRA are not mutually exclusive and can be combined to yield strong performance, since AUTO-S modifies the optimizers and LoRA modifies the architecture.

7 Related works

While other DP works also normalize the per-sample gradients (instead of clipping them) or use small clipping threshold (making the clipping similar to normalization), our work is very different in terms of theoretical analysis, algorithm design and experiments. In fact, the concurrent work [74] gives the same algorithm as AUTO-S, although its theoretical analysis and experiment design is fundamentally different from ours. [16] proposes to normalize the per-user (not per-sample) gradient in the federated learning setting, and analyzes the convergence in a convex, non-deep-learning setting.

On the other hand, many works apply the per-sample gradient clipping with small R for good utility [1, 41, 50, 38, 17]. These works have led to valuable insights, but also some false or incomplete conclusions, due to the lack of rigorous theoretical analysis. For instance, since R is present in the (re-parameterized) per-sample clipping, it cannot avoid the hyperparameter tuning as the choice of R is not robust; even if a sufficiently small R is used, the clipping does not reveal the stability constant in AUTO-S, which enjoys theoretical and empirical advantages in Remark 5.4 and Section 6. We devote Appendix L to more instances (e.g. Footnote 5) and a thorough comparison.

8 Discussion

In this work, we propose the automatic clipping as a drop-in replacement to the standard per-example clipping for differentially private training. This is the first technique that eliminates the need to tune the clipping threshold R , thus making DP deep learning as easy as regular learning. Our AUTO-S method enjoys both theoretical guarantee of convergence in non-convex problems (under various conditions), and strong empirical performance that advances DP learning on computer vision and language tasks.

We are excited about the future of automatic DP training, especially along with other working techniques, such as general optimizers (e.g. [8, 21]), clipping styles (all-layer or per-layer or adaptive clipping), architecture modifications (e.g. LoRA, RGP, prefix), and data augmentation (e.g. adversarial training [29] and multiple augmentation [17]). Thus, we expect to achieve comparable results to all SOTA in a lightweight fashion.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

Zeyuan Allen-Zhu. Natasha 2: Faster non-convex optimization than sgd. Advances in neural information processing systems , 31, 2018.

Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. Advances in Neural Information Processing Systems , 34, 2021.

Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.

Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning , pages 560-569. PMLR, 2018.

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review , 60(2):223-311, 2018.

Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard data science review , 2020(23), 2020.

Zhiqi Bu, Sivakanth Gopi, Janardhan Kulkarni, Yin Tat Lee, Hanwen Shen, and Uthaipon Tantipongpipat. Fast and memory efficient differentially private-sgd via jl projections. Advances in Neural Information Processing Systems , 34, 2021.

Zhiqi Bu, Hua Wang, and Qi Long. On the convergence and calibration of deep learning with differential privacy. arXiv preprint arXiv:2106.07830 , 2021.

Mark Bun, Cynthia Dwork, Guy N Rothblum, and Thomas Steinke. Composable and versatile privacy via truncated cdp. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 74-86, 2018.

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pages 2633-2650, 2021.

Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA) , pages 1-10. IEEE, 2018.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: Ageometric perspective. Advances in Neural Information Processing Systems , 33:13773-13782, 2020.

Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning , pages 2260-2268. PMLR, 2020.

Rudrajit Das, Abolfazl Hashemi, Sujay Sanghavi, and Inderjit S Dhillon. On the convergence of differentially private federated learning on non-lipschitz objectives, and with normalized client updates. arXiv preprint arXiv:2106.07094 , 2021.

Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650 , 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. Journal of the Royal Statistical Society Series B , 84(1):3-37, 2022.

Timothy Dozat. Incorporating nesterov momentum into adam. 2016.

Jian Du and Haitao Mi. Dp-fp: Differentially private forward propagation for large models. arXiv preprint arXiv:2112.14430 , 2021.

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7), 2011.

Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge. Computer Speech & Language , 59:123-156, January 2020.

Cynthia Dwork. Differential privacy: A survey of results. In International conference on theory and applications of models of computation , pages 1-19. Springer, 2008.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. , 9(3-4):211-407, 2014.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Aditya Golatkar, Alessandro Achille, Yu-Xiang Wang, Aaron Roth, Michael Kearns, and Stefano Soatto. Mixed differential privacy in computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8376-8386, 2022.

Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations , 2015.

Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems , 34, 2021.

Jiyan He, Xuechen Li, Da Yu, Huishuai Zhang, Janardhan Kulkarni, Yin Tat Lee, Arturs Backurs, Nenghai Yu, and Jiang Bian. Exploring the limits of differentially private deep learning with group-wise clipping. arXiv preprint arXiv:2212.01539 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2021.

Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs, 2017.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 12 2014.

Helena Klause, Alexander Ziller, Daniel Rueckert, Kerstin Hammernik, and Georgios Kaissis. Differentially private training of residual networks with scale normalisation. arXiv preprint arXiv:2203.00324 , 2022.

Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using fft. In International Conference on Artificial Intelligence and Statistics , pages 2560-2569. PMLR, 2020.

Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at imagenet scale with differential privacy. arXiv preprint arXiv:2201.12328 , 2022.

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, 2021.

Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations , 2021.

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations , 2019.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

Danilo P Mandic. A generalized normalized gradient descent algorithm. IEEE signal processing letters , 11(2):115-118, 2004.

Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research , 18:1-35, 2017.

H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations , 2018.

Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205.02973 , 2022.

Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF) , pages 263-275. IEEE, 2017.

Diganta Misra. Mish: A self regularized non-monotonic activation function. BMVC 2020 , 2019.

Ryan Murray, Brian Swenson, and Soummya Kar. Revisiting normalized gradient descent: Fast evasion of saddle points. IEEE Transactions on Automatic Control , 64(11):4818-4824, 2019.

Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o (1/kˆ 2). In Dokl. akad. nauk Sssr , volume 269, pages 543-547, 1983.

Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy. In International Conference on Learning Representations , 2021.

Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Úlfar Erlingsson. Tempered sigmoid activations for deep learning with differential privacy. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9312-9321, 2021.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , ACL '02, page 311-318, USA, 2002. Association for Computational Linguistics.

Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank J Reddi, and Sanjiv Kumar. Adaclip: Adaptive clipping for private sgd. arXiv preprint arXiv:1908.07643 , 2019.

Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics , 4(5):1-17, 1964.

Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization , 30(4):838-855, 1992.

Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint arXiv:1903.10520 , 2019.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP , 2016.

Seyed Omid Sadjadi, Timothee Kheyrkhah, Audrey Tong, Craig S Greenberg, Douglas A Reynolds, Elliot Singer, Lisa P Mason, Jaime Hernandez-Cordero, et al. The 2017 nist language recognition evaluation. In Odyssey , pages 82-89, 2018.

Ali Shahin Shamsabadi and Nicolas Papernot. Losing less: A loss for differentially private deep learning. 2021.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations , 2018.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1631-1642, 2013.

Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566-4575, 2015.

Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. In International Conference on Artificial Intelligence and Statistics , pages 1226-1235. PMLR, 2019.

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1112-1122. Association for Computational Linguistics, 2018.

Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations , 2020.

Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Normalized/clipped sgd with perturbation for differentially private non-convex optimization. arXiv preprint arXiv:2206.13033 , 2022.

Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888 , 6(12):6, 2017.

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations , 2020.

Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv:2109.12298 , 2021.

Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large scale private learning via low-rank reparametrization. In International Conference on Machine Learning , pages 12208-12218. PMLR, 2021.

Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 , 2012.

Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. Stochastic normalized gradient descent with momentum for large batch training. arXiv preprint arXiv:2007.13985 , 2020.

Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences , 64:1-13, 2021.

Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics , pages 4782-4817. PMLR, 2022.

A Proof of differential privacy

Proof of Theorem 3. Define the ℓ 2 sensitivity of any function g to be ∆ g = sup S,S ′ ∥ g ( S ) -g ( S ′ ) ∥ 2 where the supreme is over all neighboring ( S, S ′ ) . Then the Gaussian mechanism ˆ g ( S ) = g ( S ) + σ ∆ g · N (0 , I ) .

σ denotes the 'Noise multiplier', which corresponds to the noise-level when a Gaussian mechanism is applied to a query with sensitivity 1 .

Observe that automatic clipping (AUTO-V and AUTO-S (4.1)) ensures the bounded global-sensitivity of the stochastic gradient as in Abadi's clipping. Aligning the noise-multiplier (rather than the noise-level itself) ensures that the the noise-to-sensitivity ratio σ ∆ g ∆ g = σ is fixed regardless of ∆ g . The Gaussian mechanism's privacy guarantees are equivalent. Thus from the privacy accountant perspective, DP-SGD with both Abadi's clipping and our autoclipping method can be equivalently represented as the adaptive composition of T Poisson sampled Gaussian Mechanism with sampling probability B/n and noise multiplier σ .

B Proof of automaticity

B.1 Non-adaptive DP optimizers

Proof of Theorem 1. We prove Theorem 1 by showing that, DP-SGD using R -dependent AUTO-S with learning rate η and weight decay λ is equivalent to R -independent AUTO-S with learning rate ηR and weight decay λ/R . We claim other non-adaptive optimizers such as HeavyBall and NAG can be easily shown in a similar manner.

Recall the standard SGD with weight decay is

Replacing the standard gradient ∑ i ∂l i ∂ w t with the private gradient, we write the R -dependent case as

which is clearly equivalent to the R -independent case:

if we use η ′ = ηR and λ ′ = λ/R .

B.2 Adaptive DP optimizers

Proof of Theorem 2. We prove Theorem 2 by showing that, DP-AdamW using R -dependent AUTOS with learning rate η and weight decay λ is equivalent to R -independent AUTO-S with the same learning rate η and weight decay λ/R . This is the most complicated case. We claim other adaptive optimizers such as AdaDelta, Adam with weight decay (not AdamW), and NAdam can be easily shown in a similar manner.

Recall the standard AdamW is

where β 1 , β 2 are constants, g t := ∑ i ∂l i ∂ w t is the standard gradient,

Replacing the standard gradient with the private gradient R ˜ g t := R ( ∑ i ∂l i ∂ w t / ∥ ∂l i ∂ w t ∥ 2 + σ · N (0 , I )) , we write the R -dependent DP-AdamW as

Clearly, the R factor in the numerator and denominator of ˜ m t / (1 -β 1 ) √ ˜ v t / (1 -β 2 ) cancel each other. Therefore we claim that the R -dependent DP-AdamW is in fact completely independent of R .

B.3 Automatic per-layer clipping

In some cases, the per-layer clipping is desired, where we use a clipping threshold vector R = [ R 1 , · · · , R L ] and each layer uses a different clipping threshold. We claim that DP optimizers under automatic clipping works with the per-layer clipping when R is tuned proportionally, e.g. R = R · [ a 1 , · · · , a L ] , but not entry-wise (see counter-example in Fact B.1). One special case is the uniform per-layer clipping when R 1 = · · · = R L = R/ √ L . This is widely applied as only one norm R requires tuning, instead of L norms in R , particularly in the case of deep models with hundreds of layers. The corresponding DP-SGD with AUTO-S in (3.5) gives

Here the superscript ( l ) is the layer index. Clearly R couples with the learning rate η and the same analysis as in Theorem 1 follows. The adaptive optimizers can be similarly analyzed from Theorem 2.

Fact B.1. Changing one clipping threshold in the clipping threshold vector R (i.e. not proportionally) can break the coupling with learning rate.

Proof of Fact B.1. We prove by a counter-example of R in R 2 . Consider DP-SGD with per-layer clipping thresholds ( R 1 , R 2 ) = (9 , 12) :

Increasing R 1 from 9 to 16 changes the update for the first layer

The noise-to-signal ratio decreases from 5/3 to 5/4 for this layer, and increases from 5/4 to 5/3 for the second layer. This breaks the coupling with learning rate, since the coupling does not change the noise-to-signal ratio.

where

C Main results of convergence for DP-SGD with automatic clipping

C.1 Main proof of convergence for DP-SGD (the envelope version)

Proof of Theorem 4. In this section, we prove two parts of Theorem 4.

The first part of Theorem 4 is the upper bound on min t E ( ∥ g t ∥ ) , which is a direct result following from Theorem 6, and we prove it in Appendix C.2.

Theorem 6. Under Assumption 5.1, 5.2, 5.3, running DP-SGD with automatic clipping for T iterations gives

where

for r < 1 , γ = 0 and η ∝ 1 / √ T , F ( x ) = x min 0 <c< 1 f ( c,r ) and f ( c, r ) := (1+ rc ) √ r 2 +2 rc +1 + (1 -rc ) √ r 2 -2 rc +1 ; for r ≥ 1 , γ = 0 and η ∝ 1 / √ T , F ( x ) = ∞ ;

for r ≥ 1 , γ > 0 and η ∝ 1 / √ T , F is the convex envelope of (C.9) , and is strictly increasing.

0

5

10

15

20

25

30

35

40

O

(1/

T

)

0

5

10

15

20

25

30

35

40

Upper bound of min

t

(||

gt

||)

r=0.5, =0

r=0.9, =0

r=0.99, =0

r=1.01,

=0.1

0

2

4

6

8

10

O

(1/

T

)

0

2

4

6

8

10

Upper bound of min

t

(||

gt

||)

r=0.5, =0

r=0.9, =0

r=0.99, =0

r=1.01,

=0.1

r=1.5,

=0.1

r=5,

=0.1

Notice that, (C.1) holds for any r > 0 . However, we have to consider an envelope curve over r in (C.1) to reduce the upper bound: with AUTO-V clipping ( γ = 0 ), the upper bound in (C.1) is always larger than ξ as r < 1 ; we must use AUTO-S clipping ( γ > 0 ) to reduce the upper bound to zero, as can be seen from Figure 7. In fact, larger T needs larger r to reduce the upper bound.

All in all, we specifically focus on r ≥ 1 and γ > 0 , which is the only scenario that (C.1) can converge to zero. This scenario is also where we prove the second part of Theorem 4.

The second part of Theorem 4 is the asymptotic convergence rate O ( T -1 / 4 ) of DP-SGD, only possible under r ≥ 1 and γ > 0 .

By (C.1) in Theorem 6, our upper bound G from Theorem 4 can be simplified to

where the function M -1 is explicitly defined in (C.9) and the subscript ccv means the upper concave envelope. Clearly, as T → ∞ , M -1 ( 1 √ T ) → 0 . We will next show that the convergence rate of M -1 is indeed O ( 1 √ T ) and the minimization over r makes the overall convergence rate O ( T -1 / 4 ) .

Since M -1 is asymptotically linear as x → 0 , we instead study

That is, ignoring the higher order term for the asymptotic analysis, the M -1 part converges as O ( x ) = O (1 / √ T ) , and we visualize this in Figure 9.

Although DP-SGD converges faster than SGD, the former converges to ξ/r and the latter converges to 0. Thus, taking ξ/r into consideration, the objective reduces to a hyperbola

whose minimum over r is obviously 2 √ ξ (1 -x 2 γ ) x ( ξ + γ ) 2 2 γξ = O ( √ x ) = O ( T -1 / 4 ) .

To give more details about the upper bound in (5.2), we demonstrate its dependence on ξ and γ in Figure 8.

C.2 Main proof of convergence for DP-SGD (the non-envelope version)

Proof of Theorem 6. Consider DP-SGD with AUTO-S clipping

where ˜ g t,i is i.i.d. samples of ˜ g t , an unbiased estimate of g t , with a bounded variance as described in Assumption 5.3.

10 1

10 3

10 5

10 7

10 9

10 11

Number of iterations

T

10

2

10

1

10 0

10 1

10 2

Upper bound of min

t

(||

gt

||)

DP-SGD AUTO-S,  =1

DP-SGD AUTO-V,  =1

DP-SGD AUTO-S,  =10

DP-SGD AUTO-V,  =10

DP-SGD AUTO-S,

=100

DP-SGD AUTO-V,

=100

standard SGD

10 1

10 3

10 5

10 7

10 9

10 11

Number of iterations

T

10

2

10

1

10 0

10 1

Upper bound of min

t

(||

gt

||)

DP-SGD AUTO-V,  =0

DP-SGD AUTO-S,

=0.01

DP-SGD AUTO-S,

=0.1

DP-SGD AUTO-S,  =1

standard SGD

10 1

10 3

10 5

10 7

10 9

10 11

Number of iterations

T

10

2

10

1

10 0

10 1

1

,

r

=1.1

SGD grad norm

By Lipschitz smoothness in Assumption 5.2, and denoting Z = N (0 , I ) , we have

where the second last inequality follows from Cauchy Schwartz, and the last inequality follows from the fact that ∥ C i ˜ g t,i ∥ ≤ 1 , e.g. C i is ∥ ˜ g t,i / ( ∥ ˜ g t,i ∥ + γ ) ∥ or the re-parameterized clipping in [17].

Notice that in the last equality, the first term (ignoring g ⊤ t Z for its zero expectation) can be written in the same form as (3.3), which supports our motivation in Section 3.2; the second term is independent of clipping functions. Note that the last inequality is tight if and only if C i = 1 . This empirically holds in Appendix H.1, especially for GPT2.

Given the fact that ∥ ˜ g t,i / ( ∥ ˜ g t,i ∥ + γ ) ∥ ≤ 1 , the expected improvement at one iteration is

Now we want to lower bound g ⊤ t E ( ˜ g t ∥ ˜ g t ∥ + γ ) in (C.3).

Write ˜ g t = g t +∆ t where the gradient noise ∆ t follows E ∆ t = 0 , E ∥ ∆ t ∥ < ξ by Assumption 5.3. Then

where we use the hyperplane perpendicular to g t to divide the support of ∆ t into two half-spaces:

We use the symmetry assumption in Assumption 5.3 to get

and notice that ∆ t D = -∆ t , i.e., if ∆ t ∈ H + , then -∆ t ∈ H -with the same distribution.

The next result further gives a lower bound for g ⊤ t E ( ˜ g t ∥ ˜ g t ∥ + γ ) using ∥ g t ∥ .

Lemma C.1.

For the simplicity of notation, we denote the distance measure

and leave the fine-grained analysis (e.g. its explicit form in some scenarios) at the end of this section. Using the lower bound from Lemma C.1, the expected improvement (C.3) becomes

Now extend the expectation over randomness in the trajectory, and perform a telescoping sum over the iterations

Substituting ηB = η 0 / √ T where η 0 is a base learning rate, we have

and finally

With η 0 chosen properly at η 0 = √ L 0 -L ∗ L ( 1+ σ 2 d B 2 ) , the hyperbola on the right hand side in (C.5) is minimized to 4 √ ( L 0 -L ∗ ) L ( 1 + σ 2 d B 2 ) , and we obtain

Since the minimum of a sequence is smaller than the average, we have

We claim that M may not be concave or convex. Therefore we use M cvx to denote its lower convex envelope, i.e. the largest convex function that is smaller than M . Then by Jensen's inequality (C.6) becomes

It is obvious that M cvx is increasing as M is increasing by Theorem 8. Hence, ( M cvx ) -1 is also increasing, as the inverse of M cvx . We write (C.7) as

and equivalently

Finally, we derive the explicit properties of M ( ∥ g t ∥ -ξ/r ) in Theorem 8. These properties allow us to further analyze on the convergence of M ( ∥ g t ∥ -ξ/r ) , based on AUTO-V and AUTO-S, respectively.

1. DP-SGD with AUTO-V clipping. By Theorem 8, we write

This is a linear function and thus M cvx = M = 1 / M -1 cvx . As a result, we have

Wenote here r plays an important role under AUTO-V clipping: when r < 1 , we spend more iterations to converge to better and smaller gradient norm ξ/r ; when r ≥ 1 , min c f ( c, r ; 0) = f (1 , r ; 0) = 0 and it takes forever to converge. This is demonstrated in the left plot of Figure 6.

2. DP-SGD with AUTO-S clipping. By Theorem 8 and for r > 1 , we write

Notice that the inverse of a lower convex envelope is equivalent to the upper concave envelope (denoted by the subscript ccv ) of an inverse. Therefore we can derive ( M cvx ) -1 = ( M -1 ) ccv with the explicit form

we can derive it based on r, ξ, γ and substitute back to (C.8).

Note that the domain of M -1 (or the image of M ) is [0 , γ r -1 -γ r +1 ) .

In comparison to the AUTO-V clipping, M -1 takes a much more complicated form, as depicted in the middle plot of Figure 6, where r > 1 plays an important role for the gradient norm to converge to zero.

C.3 Proof of Lemma C.1

Proof of Lemma C.1. We want to lower bound

To simplify the notation, we denote noise-to-signal ratio S := ∥ ∆ t ∥ ∥ g t ∥ and c := cos θ = g ⊤ t ∆ t ∥ g t ∥∥ ∆ t ∥ , with θ be the random angle between g t and ∆ t . Note that 0 < c ≤ 1 when ∆ t ∈ H + .

The term inside the conditional expectation in (C.10) can be written as

Defining Γ = γ/ ∥ g t ∥ and

we turn the conditional expectation in (C.10) into

for which we want to lower bound f ( c, S ; Γ) over 0 < c ≤ 1 , S > 0 , Γ > 0 . We use the next theorem to prepare some helpful properties. The proof can be found in Appendix E.1.

Theorem 7. For f defined in (C.11) , we have

f ( c, S ; Γ) is strictly decreasing in S for all 0 < c < 1 and Γ > 0 .

Consequently, min c ∈ (0 , 1) f ( c, S ; Γ) is strictly decreasing in S .

f ( c, S ; Γ) is strictly decreasing in c for all S > 1 and Γ > 0 .

We consider a thresholding ratio r > 0 and we will focus on the regime that S < r . This r will turn out to measure the minimum gradient norm at convergence: informally speaking, ∥ g t ∥ converges to ξ/r .

By the law of total expectation, (C.12) can be relaxed as follows.

where in the first inequality, the ignoring of last term is justified by f ( c, S ; Γ) ≥ min c ∈ (0 , 1] f ( c, S ; Γ) ≥ min c ∈ (0 , 1] f ( c, ∞ ; Γ) = 0 , from the monotonicity (second statement) in Theorem 7.

We first lower bound ⋆ ⃝ by applying the Markov's inequality:

and hence by Assumption 5.3,

Finally, the conditional expectation of interest in (C.10) gives

C.4 Proof of Theorem 8

To derive some properties of min c f ( c, r ; Γ) , we need to compute separately for AUTO-V (without the stability constant, Γ = 0 ) and for AUTO-S (with the stability constant, Γ > 0 ), as shown in Theorem 8. As we will show, as the number of training iterations T →∞ , DP-SGD with AUTO-V clipping can only compress ∥ g t ∥ to ξ/r for r < 1 . However, DP-SGD with AUTO-S clipping can compress ∥ g t ∥ to ξ/r to any r > 1 .

Theorem 8.

For 0 < r < 1 and Γ = 0 , we have min c ∈ (0 , 1] f ( c, r ; 0) > 0 . Then Equation (C.12) is lower bounded by

which is increasing in ∥ g ∥ -ξ/r .

For r ≥ 1 and Γ = 0 , we have min c ∈ (0 , 1] f ( c, r ; Γ) = f (1 , r ; 0) = 0 . In words, (C.10) has a trivial lower bound and Theorem 6 cannot compress ∥ g t ∥ to ξ/r .

For r ≥ 1 and Γ > 0 , we have min c ∈ (0 , 1] f ( c, r ; Γ) = f (1 , r ; Γ) = ( Γ r +Γ -1 -Γ r +Γ+1 ) . Then Equation (C.12) is lower bounded by

which is increasing in ∥ g t ∥ -ξ/r .

Proof. To prove statement 1, we use the second statement from Theorem 7 and show that min c f ( c, r ; 0) > min c f ( c, ∞ ; 0) = 0 . To prove statement 2 and 3, we use the third statement from Theorem 7 and see that min c f ( c, r ; Γ) = f (1 , r ; Γ) with an explicit formula.

D Convergence rate of standard SGD

Theorem 9. Under Assumption 5.1, 5.2, 5.3 (without the symmetry assumption), running the standard non-DP SGD for T iterations gives, for η ∝ 1 / √ T ,

Proof of Theorem 9. Consider the standard SGD

where ˜ g t,i is i.i.d. unbiased estimate of g t , with a bounded variance as described in Assumption 5.3. By Lipschitz smoothness assumption in Assumption 5.2,

The expected improvement at one iteration is

Now we extend the expectation over randomness in the trajectory, and perform a telescoping sum over the iterations

Notice that we do not need the symmetry assumption in Assumption 5.3 in the non-DP SGD analysis. We apply the same learning rate as in [5], η = 1 L √ T ,

and finally

Using the Jensen's inequality, we can have

E Auxiliary proofs

E.1 Proof of Theorem 7

Proof. We first show df ( c,S ;Γ) dS < 0 for all 0 < c < 1 , Γ > 0 and S > 0 , as visualized in the left plot of Figure 10. We can explicitly write down the derivative, by WolframAlpha

with

It is obvious that, since c < 1 ,

From (E.2), the denominator in (E.1) is positive and it suffices to show A Γ 2 + B Γ + C > 0 for all 0 < c < 1 and S > 0 , in order to show df dS < 0 .

Also from (E.2), we can easily see B ( c, S ) > 0 and C ( c, S ) > 0 . We will show that A ( c, S ) > 0 in Lemma E.1, after very heavy algebraic computation.

Now we can claim that A Γ 2 + B Γ+ C > 0 by Fact E.3, and complete the proof of the first statement.

To further see that min c f ( c, S ; Γ) is decreasing in S , let us denote c ∗ ( x ; Γ) := arg min c ∈ [0 , 1] f ( c, x ; Γ) . Then considering S < S ′ , we prove the second statement by observing

This statement is also visualized in the right plot of Figure 10.

We next show df ( c,S ;Γ) dc < 0 for all 0 < c < 1 , Γ > 0 and S > 1 . We can explicitly write down the derivative, by WolframAlpha

with

Clearly B ′ ( c, S ) > 0 and C ′ ( c, S ) > 0 , since S 2 +2 cS +1 > S 2 -2 cS + c 2 = ( S -c ) 2 ≥ 0 . And we will show A ′ ( c, S ) > 0 in Lemma E.2, after some algebra.

We again claim that A ′ Γ 2 + B ′ Γ+ C ′ > 0 by Fact E.3, which guarantees that the numerator in (E.3) is negative and that df dc < 0 . This is visualized in Figure 11.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

S

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

f

(0.5,

S

,

)

=0

=0.1

=0.5

=1

0.0

0.5

1.0

1.5

2.0

2.5

3.0

S

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

min

c

f

(

c

,

S

,

)

=0

=0.1

=0.5

=1

0.0

0.2

0.4

0.6

0.8

1.0

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

f

(

c

, 0.8,

)

=0

=0.1

=0.5

=1

=2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

f

(

c

, 2,

)

=0

=0.1

=0.5

=1

=2

c

c

E.2 Proof of Lemma E.1

Lemma E.1. For all 0 < c < 1 and S > 0 ,

Proof. We prove by contradiction. Suppose

Then

where the first inequality comes from S 2 -2 cS +1 > S 2 -2 cS + c 2 = ( S -c ) 2 ≥ 0 .

Squaring everything gives

Taking the difference gives

Given that c > 0 , S > 0 , we have

Denoting X := S 2 and viewing the above as a quadratic polynomial of X , we have

Using the closed-form minimizer of quadratic polynomial 1 ⃝ , after some heavy algebra, one can check the minimum of 1 ⃝ is

which is clearly positive. Contradiction!

E.3 Proof of Lemma E.2

Lemma E.2. For all 0 < c < 1 and S > 1 ,

Proof. Notice that ( S 2 +3 cS +2) > S 2 +2 > 0 and √ S 2 ± 2 cS +1 > 0 . Therefore if S 2 -3 cS + 2 ≤ 0 , we are done.

Otherwise, we prove by contradiction and suppose

under the condition that S 2 -3 cS +2 > 0 .

Squaring everything gives

Taking the difference gives

Given that c > 0 , S > 0 , we have

Denoting X := S 2 and viewing the above as a quadratic polynomial of X , we have, for X > 1 ,

The closed-form minimizer of quadratic polynomial 2 ⃝ is (9 c 2 -5) 4 . Given that 0 < c < 1 , we must have -5 4 < 9 c 2 -5 4 < 1 . Hence the minimizer is not within the feasible domain (1 , ∞ ) of X . Thus the minimum of 2 ⃝ is achieved with X = 1 at 9(1 -c 2 ) . This is positive. Contradiction!

E.4 Proof of Fact E.3

Fact E.3. For a quadratic polynomial Ax 2 + Bx + C with A,B,C > 0 , the minimum value on the domain x ≥ 0 is C , at x = 0 . Therefore Ax 2 + Bx + C > 0 .

Proof. Since A > 0 , the quadratic polynomial is convex and increasing on the domain x > -B 2 A . Since B > 0 as well, we know -B 2 A < 0 and hence the quadratic polynomial is strictly increasing on x > 0 . Therefore the minimum value is achieved when x = 0 , and we obtain Ax 2 + Bx + C ≥ C > 0 for all x ≥ 0 .

E.5 Assumption of symmetric gradient noise

We show that Assumption 5.3 is actually relaxed from and less strict than the assumptions used in the non-DP literature. In words, Assumption 5.3 allows our DP convergence to be comparable to the standard convergence (as in Theorem 9), because our assumption does not enforce extra constraint.

In standard non-DP analysis [48, 66, 12, 73], the mini-batch gradient is assumed to be an unbiased estimate of the oracle gradient g t = ∂ L ∂ w :

and ξ is the random gradient noise with ξ ∼ N ( 0 , Σ( w ) /B ) . Since this assumption holds for any batch size B , we can set B = 1 to recover the per-sample gradient noise: ξ = ˜ g t,i -g t is i.i.d. and symmetric because a zero-mean Gaussian is symmetric.

In fact, we can further relax our Assumption 5.3: besides assuming the central symmetry, the same proof of convergence will follow if we instead assume the mirror symmetry about the hyperplane normal to g t , that is { v : g ⊤ t v = 0 } .

F Examples of lazy regions

F.1 Balanced binary classification

We describe the data generation in Section 3.3. The label is uniformly ± 1 , that is P ( y i = +1) = P ( y i = -1) = 0 . 5 . We have 10000 positive and negative samples x i ∼ N ( y i , 1) . We consider a logistic regression model P ( Y = y | x ) = I ( y = 1) · Sigmoid ( x + θ ) + I ( y = -1) · (1 -Sigmoid ( x + θ )) = 1 1+ e -y ( θ + x ) , where θ ∈ R is the intercept. The gradient with respect to this only trainable parameter is ∂ L i ∂θ = -y ( 1 -1 1+ e -y ( θ + x ) ) . We set the clipping threshold R = 0 . 01 and the stability constant γ = 0 . 01 .

F.2 Mean estimation on Gaussian mixture data

We also observe the lazy region issue in the mean estimation problem min θ 1 2 ∥ θ -x i ∥ 2 . Here P ( x i ∼ N (4 , 1)) = P ( x i ∼ N (4 , 1)) = 0 . 5 . We have 10000 samples from each Gaussian distribution. The regular minimum is clearly ∑ i x i → 0 , where the regular gradient and AUTO-S clipped gradient vanish. Yet both AUTO-V and Abadi's clipping lose motivation to update the mean estimator on the interval ( -1 , 1) . We set the clipping threshold R = 0 . 01 and the stability constant γ = 0 . 1 .

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

10 5

10 4

10 3

10 2

10 1

10 0

0

10 0

10 1

10 2

10 3

10 4

10 5

standard grad

G Experiments settings

G.1 Image classification settings

We give the experiments settings for computer vision tasks in Table 1.

MNIST : We use the network architecture from [56, 68, 64], with 40 epochs, 512 batch size, 0.5 learning rate (or 0.005 non-DP learning rate), 0.1 clipping threshold, DP-SGD with 0.9 momentum, and without pretraining. This setting is the same as [68].

FashionMNIST : We use the same network architecture as MNIST, with 40 epochs, 2048 batch size, 4 learning rate (or 0.04 non-DP learning rate), DP-SGD with 0.9 momentum, and without pretraining. This setting is the same as [68].

CIFAR10 pretrained : We use the SimCLR model from [13] 7 , with 50 epochs, 1024 batch size, 4 learning rate (or 0.04 non-DP learning rate), 0.1 clipping threshold, and DP-SGD with 0.9 momentum. The SimCLR model is pretrained on unlabelled ImageNet dataset. After pretraining, we obtain a feature of dimension 4096 on which a linear classifier is trained privately. This setting is the same as [68].

ImageNette : We use the ResNet9 (2.5 million parameters) with Mish activation function [52]. We set 50 epochs, 1000 batch size, 0.0005 learning rate (or 0.000005 non-DP learning rate), 1.5 clipping threshold, and use DP-NAdam, without pretraining. This setting is the same as [36] except we did not apply the learning rate decaying scheduler.

CelebA (Smiling and Male and Multi-label) We use the same ResNet9 as above, with 10 epochs, 500 batch size, 0.001 DP learning rate (or 0.00001 non-DP learning rate), 0.1 clipping threshold, and use DP-Adam, without pretraining. We use the labels 'Smiling' and 'Male' for two binary classification tasks, with cross-entropy loss. For the multi-label task uses a scalar loss by summing up the 40 binary cross-entropy losses from each label.

We refer the code for MNIST, FashionMNIST, CIFAR10, CIFAR10 pretrained to https:// github.com/ftramer/Handcrafted-DP by [68]. ResNet9 can be found in https://github. com/cbenitez81/Resnet9 .

Throughout all experiments, we do not apply tricks such as random data augmentation (single or multiple times [17]), weight standardization [61], or parameter averaging [60].

G.2 Sentence classification settings

We experiment on five datasets in Table 2 and Table 3.

MNLI(m) MNLI-matched, the matched validation and test splits from Multi-Genre Natural Language Inference Corpus.

MNLI(mm) MNLI-mismatched, the matched validation and test splits from Multi-Genre Natural Language Inference Corpus.

QQP The Quora Question Pairs2 dataset.

QNLI The Stanford Question Answering dataset.

SST2 The Stanford Sentiment Treebank dataset.

The datasets are processed and loaded from Huggingface [39], as described in https:// huggingface.co/datasets/glue . We follow the same setup as [78] and [41]. We refer the interested readers to Appendix G,H,I,K,N of [41] for more details.

We emphasize that our automatic clipping uses exactly the same hyperparameters as the Abadi's clipping in [41], which is released in their Private-Transformers library 8 .

Notice that we use DP learning rate 5e-4 across tasks for the R -dependent automatic DP-Adam, which is equivalent to R -independent automatic DP-Adam with the same learning rate. We demonstrate that the results are not sensitive to learning rates around the optimal choice. That is, the automatic clipping does not eliminate R at the cost of more difficult tuning of learning rate.

G.3 Table-to-text generation settings

We experiment multiple GPT2 models on E2E dataset from Huggingface [39] in Table 4. We follow the same setup as [41], and our automatic clipping uses exactly the same hyperparameters as the Abadi's clipping in [41], which is released in their Private-Transformer library 9 .

H Figure zoo

H.1 Frequency of clipping

We show that in all sentence classification tasks, Abadi's clipping happens on a large proportion of per-sample gradients. This supports the similarity between Abadi's clipping and AUTO-V in (3.1).

0

25

50

75

100

125

150

175

200

Iteration

t

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Percentage of

gt

,

i

clipped

0

50

100

150

200

250

300

350

Iteration

t

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of

gt

,

i

clipped

0

20

40

60

80

100

Iteration

t

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Percentage of

gt

,

i

clipped

0

25

50

75

100

125

150

Iteration

t

0.0

0.2

0.4

0.6

0.8

1.0

Percentage of

gt

,

i

clipped

0

25

50

75

100

125

150

175

0.2

0.4

0.6

0.8

1.0

Percentage of

gt

,

i

clipped

0

50

100

150

200

250

300

0.2

0.4

0.6

0.8

1.0

Percentage of

gt

,

i

clipped

Iteration

t

Iteration

t

We note that for GPT2, GPT2 medium and GPT2 large, empirically in all iterations 100% of the per-sample gradients are clipped by the Abadi's clipping, making the performance of Abadi's clipping equivalent to AUTO-V clipping, as shown in Table 4.

H.2 Stability constant helps AUTO clipping reduce gradient norm

To corroborate our claim in Theorem 6, that the stability γ reduces the gradient norm, we plot the actual gradient norm by iteration.

0

50

100

150

200

250

300

Iteration t

50000

100000

150000

200000

250000

300000

||

gt

||

AUTO-V

AUTO-S

0

50

100

150

200

250

300

Iteration t

200000

400000

600000

800000

||

gt

||

AUTO-V

AUTO-S

H.3 Choice of stability constant is robust

We claim in Theorem 6 that, as long as γ > 0 in our automatic clipping, the asymptotic convergence rate of gradient norm is the same as that by standard non-private SGD. We plot the ablation study of learning rate and the stability constant γ to show that it is easy to set γ : in Table 2 and Table 3, we adopt learning rate 0.0005, under which a wide range of 0 . 0001 < γ < 1 gives similar accuracy. Note that the largest good γ is 1000 times bigger than the smallest good γ .

0.0001

0.001

0.01

0.1

1

10

100

1000

Stability constant

0.001

0.0005

0.0002

0.0001

Learning rate

90.5

90.8

90.7

90.8

91.4

92.0

91.6

89.7

91.1

92.1

92.7

92.4

92.4

92.7

93.0

92.2

91.3

91.4

91.4

92.0

92.2

92.8

93.1

91.4

90.7

90.8

91.1

91.4

91.6

92.3

92.2

90.9

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

0.0001

0.001

0.01

0.1

1

10

100

1000

Stability constant

0.001

0.0005

0.0002

0.0001

Learning rate

91.9

92.3

92.3

92.0

92.3

92.2

91.7

89.7

93.0

93.5

93.6

93.5

93.6

93.8

94.2

93.0

93.7

94.0

94.5

94.5

94.5

95.0

95.1

93.2

93.9

94.6

95.1

95.0

95.2

95.2

94.3

92.8

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

0.0001

0.001

0.01

0.1

1

10

100

1000

Stability constant

0.001

0.0005

0.0002

0.0001

Learning rate

85.9

86.2

86.5

86.2

85.3

84.7

66.8

72.8

86.0

86.2

86.1

86.2

86.3

86.1

84.0

78.7

84.5

84.9

85.0

85.0

85.1

85.2

84.0

77.4

83.8

83.6

83.6

83.5

83.7

83.5

82.2

70.3

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.0001

0.001

0.01

0.1

1

10

100

1000

Stability constant

0.001

0.0005

0.0002

0.0001

Learning rate

88.2

88.7

89.1

89.3

87.6

82.7

66.2

79.5

89.7

90.3

90.6

90.2

90.4

89.8

88.1

85.5

88.9

89.2

89.3

89.3

89.4

89.5

88.5

84.1

87.8

87.9

88.1

88.0

88.1

87.8

86.9

77.9

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.0001

0.001

0.01

0.1

1

10

100

1000

Stability constant

0.001

0.0005

0.0002

0.0001

Learning rate

84.9

84.9

85.1

84.5

82.7

81.0

58.4

61.8

84.5

84.3

84.4

84.5

84.3

83.3

78.0

65.8

82.2

82.4

82.2

82.2

82.4

81.8

79.2

65.6

79.0

79.6

79.5

79.2

79.1

78.8

73.8

61.1

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.0001

0.001

0.01

0.1

1

10

100

1000

Stability constant

0.001

0.0005

0.0002

0.0001

Learning rate

88.0

88.4

88.6

88.5

85.5

51.6

50.8

66.6

88.3

89.0

89.1

88.9

88.9

87.4

79.5

76.9

87.0

87.1

87.1

87.1

87.3

86.9

84.8

73.0

83.9

83.9

84.6

84.8

85.0

84.7

81.1

67.7

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

I Full table of GPT2 generation task on E2E dataset

This is the extended version of Table 4 on E2E dataset. The performance measures are BLEU [57], ROGUE-L [42], NIST [63], METEOR [4], and CIDEr [70] scores. Here ϵ is accounted by RDP [51], where ϵ = 3 corresponds to 2.68 if accounted by Gaussian DP [19, 7] or to 2.75 if accounted by numerical composition [30], and ϵ = 8 corresponds to 6.77 if accounted by Gaussian DP or to 7.27 if accounted by numerical composition.

We observe that GPT2 (163 million parameters), GPT2-medium (406 million), and GPT2-large (838 million), Table 4 trained with our automatic clipping consistently perform better in comparison to other methods. In some cases, LoRA trained with Abadi's clipping also demonstrates strong performance and it would be interesting to see how LoRA trained with the automatic clipping will behave.

J Further experiments on CelebA dataset

In this section, we present a complete summary of accuracy results, with DP constraint or not, for the CelebA dataset. We do not apply any data-preprocessing. In the first experiment, we apply a single ResNet on the 40 labels as the multi-task/multi-label learning. In the second experiment, we apply one ResNet on one label. As expected, our automatic DP optimizers have comparable test accuracy to the Abadi's DP optimizers, but we do not need to tune the clipping threshold for each individual task/label. We also notice that, learning different labels separately gives better accuracy than learning all labels together, though at the cost of heavier computational burden.

J.1 Multi-label classification

We apply ResNet9 as in Appendix G.1 on the multi-label classification task. I.e. the output layer has 40 neurons, each corresponding to one sigmoid cross-entropy loss, that are summed to a single loss and all labels are learnt jointly.

J.2 Multiple binary classification

For the second experiment, we apply ResNet9 on each label as a binary classification task. I.e. the output layer has 1 neuron and we run 40 different models for all labels separately.

K Code implementation of automatic clipping

Changing Abadi's clipping to automatic clipping is easy in available codebases. One can set the clipping R = 1 or any other constant, as explained in Theorem 1 and Theorem 2.

K.1 Opacus

For Opacus [77] version 1.1.2 (latest), we can implement the all-layer automatic clipping by changing Line 399-401 in https://github.com/pytorch/opacus/blob/main/opacus/optimizers/ optimizer.py to

The per-layer automatic clipping requires changing Line 61-63 in https://github.com/pytorch/ opacus/blob/main/opacus/optimizers/perlayeroptimizer.py to

For older version ( < 1 . 0 , e.g. 0.15) of Opacus, we can implement the all-layer automatic clipping by changing Line 223-225 in https://github.com/pytorch/opacus/blob/v0.15.0/opacus/ utils/clipping.py to

or implement the per-layer automatic clipping by changing Line 301-302 in https://github.com/ pytorch/opacus/blob/main/opacus/optimizers/perlayeroptimizer.py to per_sample_clip_factor = threshold / (norm + 0.01) clipping_factor.append(per_sample_clip_factor)

K.2 ObJAX

For ObJAX version 1.6.0 (latest), we can implement the automatic clipping in https://github. com/google/objax/blob/master/objax/privacy/dpsgd/gradient.py by changing Line 92 to

K.3 Private-transformers

To reproduce our experiments for sentence classification and table-to-text generation, we modify the 'private-transformers' (version 0.1.0) codebase of [41]. The modification is in https://github.com/lxuechen/private-transformers/blob/main/private_ transformers/privacy_utils/privacy_engine.py , by changing Line 349 to

L More on related works of per-sample clipping

We discuss the difference between our work and the related (see the table below).

i

Our work is very different to most works which do not analyze the convergence of DP deep learning in a non-convex setting, but it is very similar to [74] 10 . However, [74] assumes a relaxed Lipschitz smootheness in place of our Assumption 5.3, where we instead assume the symmetric gradient noise. In addition, our experiments are more comprehensive, covering over 10 tasks including DP-GPT2, while [74] only experimented with 2 smaller models - ResNet20 and Roberta-base.

L.1 Clarifications

We now clarify some false or incomplete conclusion in previous literatures that apply the per-sample gradient clipping (re-parameterized or not).

Per-sample clipping is not robust to R , even with re-parameterization.

In [17, Figure 8] and our Figure 4, the accuracy of DP optimizer with Abadi's clipping is insensitive to R only if one has found a small enough region (e.g. R ≤ 1 ), which takes effort to find or the accuracy will be unacceptably low out of the region. In particular, choosing R = 1 as in [17] is not universally proper, e.g. [41] uses R = 0 . 1 for language models. This dependence on tasks, datasets and optimizers means per-sample clipping still requires the expensive hyperparameter tuning.

In other words, per-sample gradient clipping is at best an approximation of per-sample gradient normalization (i.e. our AUTO-V) and should be considered as semi-automatic, whereas AUTO-V/S is fully automatic in terms of tuning R . Although technically we introduce a new hyperparameter γ in the place of R , we claim that automatic clipping is not sensitive to γ (our only hyperparameter) for a large range, e.g. one can multiply γ by 10000 times, going from γ = 0.001 to 10 with learning rate 0.0005 in Figure 15, and the accuracy is similar.

Per-sample clipping does not decouple R , especially for DP-Adam.

In general, R is not completely decoupled from the re-parameterized per-sample clipping in [17]:

Given that R appears in both terms on the right hand side, one can at most say "... when the clipping norm is decreased k times, the learning rate should be increased k times to maintain similar accuracy." by [38] and "... Using this update, performance becomes less sensitive to the choice of clipping norm." by [17]. In contrast, we can state that adjusting the learning rate proportionally, our AUTO-V/S maintains exactly the same accuracy and is completely insensitive to the choice of R .

Additionally and importantly, the understanding in [38, 17] is limited to DP-SGD (as they only experiment with the computer vision tasks), where "... the learning rate η absorbs a factor of R ." by [17]. As rigorously proved in Theorem 1 and Theorem 2, adaptive optimizers like Adam and AdaGrad do not absorb R but rather cancel it. This is visualized in Figure 1, where the performance landscape is row-wise for DP-Adam and diagonal for DP-SGD.

Re-parameterized per-sample clipping unintentionally changes the weight decay.

Weight decay is a common technique used in any work that uses AdamW and in the re-parameterized trick by [17]. We can see that

Therefore, when we move along R like in [17, Figure 8], from R = 1 to 2 -6 , the weight decay increases from λ to 2 6 · λ by 64 times, which may worsen the accuracy as seen in the blue curve of [17, Figure 8]! Again, this is due to the incomplete decoupling by per-sample clipping, which is only avoided in AUTO-V/S thanks to theoretical analysis in Theorem 1 and Theorem 2.

L.2 Connections to normalized optimisation

Variants of normalized gradient have been used in optimization [47, 53, 81, 80, 15]. These normalized optimizers are fundamentally different to our automatic optimizers, because the normalization is on mini-batch not on each sample and noise is not involved:

The main difference lies in the challenge of analyzing per-sample normalization (which is biased) and the batch-gradient normalization (which is unbiased in the direction). That is, 1 B ∑ i g i ∥ 1 B ∑ i g i ∥ is parallel to the mini-batch gradient 1 B ∑ i g i but 1 B ∑ i g i ∥ g i ∥ is generally not parallel to it (this conclusion also holds if the normalizaiton is replaced by the clipping). On a side note, it is interesting that Theorem 4 indeed shows although a bias is introduced by the per-sample clipping, it is not fatal to the asymptotic convergence and hence may not be a concerning matter.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new alternative version of the DP-SGD algorithm where the gradients are normalized. This new method allows for a proof of convergence of the gradient when we add a small stability constant to the rescaling factor. This work eliminates the need for a costly 2D gridsearch between learning rate and clipping constant when training DP models. The authors then proceed to run experiments on to show their method equals or outperforms Abadi’s DP-SGD algorithm.\n\nSTRENGTHS:\nThe article eliminates a costly operation in DP training and provides strong evidence regarding the convergence of their method. The exhaustive list of datasets on which the method was tested testifies as to how only minimal changes to existing implementations are needed to apply this method.\n\n### Originality\n\nThe method is original.\n\n### Quality\n\nThe experiments presented are very complete and yield high quality results. Also, various supporting githubs and ressources are mentionned, making these results reproducible.\n\n### Clarity\n\nThe article is clear and well written.\n\n### Significance\n\nThe original motivation of the article is clear.\n\nWEAKNESSES:\n## Sensitivity of the hyperparameter optimization\n\nThe results are not clear enough to indicate whether the proposed DP training method is superior to the rescaled DP-SGD method which impedes the significance of the article. If the authors of the paper manage to prove that their method is less expensive than the rescaled DP-SGD trick then the contribution is significant. The article lacks a conclusive experiment or statement proving that the sensitivity of the hyperparameter optimization process is easier with this method than with the re-scaled version of Abadi’s DP-SGD. \n\nThe paper shows in its appendix H. that the training process of the AUTO-S method is relatively robust to the choice of $\\gamma$ on NLP tasks. However the paper doesn’t compare the sensitivity of the **AUTO-S method** to the $\\gamma$ factor to the sensitivity of the **rescaled Abadi clipping** to the $R$ factor. In my opinion this is an important experiment to run, since it would justify using this method instead of the widely known rescaled DP-SGD version introduced in De et al. [15]. This could improve the impact and the significance of the contribution.\n\n### Actionable feedback\n\n> However, R remains and is expensive to tune (l223)\n\nThis sentence in particular is a bit of an overclaim in my opinion, see my comments below.\n\n**In your experiments you specifiy regarding the range of $\\gamma$**\n> ”under which a wide range of 0.0001 < < 1 gives similar accuracy”\n\nand\n>“Note that the largest good is 1000 times bigger than the smallest good ”\n\n\n**In the comparison to related works on clipping**\nYou state “Similarly (De et al.) re-parametrizes the per-sample gradient clipping to mimic normalization. However R remains and is expensive to tune (see figure 8 of De et al.)”. However this fig 8. shows that the re-parametrized per-sample gradient clipping seems to stay stable by a similar factor of approximately 1024 on the clipping norm, on a even harder dataset. Even though their experiment are not extensive enough to prove a superior efficiency of the rescaled DP-SGD method to yours, it is not conclusive enough for you to assume that your method is easier to tune.\n\nIf you add this missing experiment, or if you modify this claim, I would consider improving my rating.\n\nQUESTIONS:\n### Can you provide tangible proof that your method is significantly less expensive to tune than the rescaled DP-SGD algorithm ?\n\nThe figure you cite in the concurrent work of De et al. in section 4.4 does not seem complete enough and gives no indication of the strain in ressources the rescaled DP-SGD might still induce.\n\nLIMITATIONS:\nThe method is original. However some similarities can be drawn with existing concurrent approaches like Yang et al. 2022 [2206.13033]  that did not go through peer-reviewing to the best of my knowledge, however this work might be worth citing.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author(s) proposed a new gradient clipping technique for differential private training algorithms. It was shown that the new clipping technique is more robust to hyper-parameters and can save the time for hyper-parameter tunning. Convergence of the proposed method under a gradient symmetric assumption is developed and experiments on both vision and language tasks are conducted to evaluate the proposed method.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow in general.\n- The experiments are thorough and the algorithm seems to be useful on the practical side.\n\nWEAKNESSES:\n- The convergence analysis is based on a gradient symmetric assumption, therefore the utility bound in Theorem 4 is not really comparable to prior works. Please also see my quetions below. Overall, the theoretical contribution seems limited. However, given the strong empirical performance of the proposed method, this might be acceptable.\n- Some related work should be added [1,2,3].\n\n[1] Clip21: Error Feedback for Gradient Clipping, https://arxiv.org/abs/2305.18929\n\n[2] Improved Convergence of Differential Private SGD with Gradient Clipping, https://openreview.net/forum?id=FRLswckPXQ5\n\n[3] Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization, https://arxiv.org/pdf/2206.13033.pdf\n\nQUESTIONS:\n- Consider the example with 3 data points, $f(x) = f_1(x) + f_2 (x) + f_3(x)$, where $f_1( x ) = 2 x^2, f_2(x) = (x-1)^2, f_3( x ) = f_2(x) = (x-1)^2$. Then the solution is at $x^* = 0.5$, but this point is not stationary for Algorithm 1, because $g_1( x^* ) = 2, g_2(x^*)=g_3(x^*)=-1$, which implies $g_1( x^* ) / \\| g_1( x^* ) \\| = 1, g_2(x^*) / g_2( x^* )=g_3(x^*)/g_3( x^* )=1$. Algorithm 1 willi fail on this example, does the author(s) agree?\n\nLIMITATIONS:\nI do not find any negative societal impact of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper discusses a way to control the sensitivity of DP-SGD by a method that does not require clipping.

STRENGTHS:
- The motivation for the paper is clear
- The authors make a compeling argument that suggest why their method should work
- They have sound theoretical guarantees and experiments
- The idea is simple yet fully developed
- The impact this could have on private learning is huge
- Experimental results are compeling

WEAKNESSES:
N/A

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces 'automatic clipping' to replace the usual clipping operation in DP-SGD. The challenge of usual clipping is to choose a good threshold 'R' especially for deep learning models. The authors claim that automatic clipping maintains the same level of privacy and computational efficiency as existing DP optimizers such as DP-SGD, DP-Adam, and DP-LAMB, but without the need for any DP-specific hyperparameters. In fact, automatic clipping uses a kind of normalization to bound the sensitivity of individual gradient. Additionally, they provide a thorough convergence analysis for DP-SGD with automatic clipping in the non-convex setting, demonstrating that it can match the convergence rate of standard SGD under certain conditions. The authors validate their proposal by showing that automatic clipping either matches or surpasses the state-of-the-art performance on a variety of language and vision tasks.\n\nSTRENGTHS:\n1. **Simplicity**: The proposed method of automatic clipping significantly simplifies the process of DP training by eliminating the need for tuning DP-specific hyperparameters, thereby making DP as user-friendly as standard non-private training.\n\n2. **Performance**: The paper demonstrates that the automatic clipping method is either on par with, or better than, existing state-of-the-art techniques in a variety of tasks. This suggests that the method does not compromise performance for ease-of-use.\n\n3. **Rigorous Analysis**: The authors provide a detailed convergence analysis of automatic DP-SGD in a non-convex setting, showing that it can match the convergence rate of standard SGD under the symmetric gradient noise assumption of per-sample gradients.\n\nWEAKNESSES:\n**Misleading justification**: In figure 2, the paper present the dot products of $\\langle g_i, \\sum_i g_i \\rangle$. First it does not make sense if the dot product is properly normalized.  Moreover, the distribution of  $\\langle g_i, \\sum_i g_i \\rangle$ does not necessarily align with good performance. The illustration in Figure 2 is rather misleading.\n\n**Too strong Assumptions**: The proposed automatic clipping method's performance is based on a symmetric gradient noise assumption of the per-sample gradients. This assumption is too constrained and does not satisfy in practice.\n\n**Overclaim**: Section title 4.3 \"Automatic clipping is equally private and maximizes utility\" is somewhat overclaiming. It is not carefully argued why automatic clipping can \"maximize utility given a privacy budget.\n\n**Missing reference**: The paper missed a very related reference Yang et al. \"Normalized/clipped sgd with perturbation for differentially private non-convex optimization\" which also studied the DP-SGD with normalization. The paper should have a careful comparison with it.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}

comments: SUMMARY:
The script proposed an automatic clipping methods for various DP algorithms. The problem is important because the performance of DP models are sensitive to the choice clipping threshold, yet there is no theorical guidance for tuning it.

STRENGTHS:
The writing is clear and the paper is easy to follow. The idea is simple and effective.  Theory and experiments are provided to show the efficacy of the proposed automatic clipping method. I think this result is worth sharing with the DP community.

WEAKNESSES:
See below

QUESTIONS:
**Only one question:**

According to to line 140, Auto-V maximizes the dot product similartiy **only when** <g_i, \sum_j g_j> >0 for any i.  However, in practice, <g_i, \sum_j g_j>  could be negative, but Auto-V still perform gradient normalization on these samples. I wonder: what if we only normalize g_i with <g_i, \sum_j g_j> >0 and ignore the rest, (i.e., do exactly the optimal normlization in line 140). Will it bring better performance? Similar question also goes to Auto-S.



**Suggestions:**

I suggest add index to the equations in line 138 and 140.

LIMITATIONS:
See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25