SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiqi Bu
AWS AI
zhiqibu@amazon.com
AWS AI, UC Santa Barbara
Sheng Zha
zhasheng@amazon.com
George Karypis
gkarypis@amazon.com
Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping threshold  $R$ , however, is vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called automatic clipping, that eliminates the need to tune  $R$  for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB and many others. The automatic variants are as private and computationally efficient as existing DP optimizers, but require no DP-specific hyperparameters and thus make DP training as amenable as the standard non-private training. We give a rigorous convergence analysis of automatic DP-SGD in the non-convex setting, showing that it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric gradient noise assumption of the per-sample gradients (commonly used in the non-DP literature). We demonstrate on various language and vision tasks that automatic clipping outperforms or matches the state-of-the-art, and can be easily employed with minimal changes to existing codebases<sup>1</sup>.
Deep learning has achieved impressive progress in a wide range of tasks. These successes are made available, in part, by the collection of large datasets, sometimes containing sensitive private information of individual data points. Prior works have illustrated that deep learning models pose severe privacy risks to individual subjects in the training data and are susceptible to various practical attacks. For example, machine learning services such as Google Prediction API and Amazon Machine Learning can leak membership information from the purchase records [65]; the GPT2 language models auto-complete texts that contain someone's full name, phone number, email address, etc., from the training data that it memorizes, if invoked by specific prefixes [11].
Differential privacy (DP) [24, 26, 25] is a formal definition of privacy that has been shown to prevent the aforementioned privacy risks in deep learning [1]. At a high level, the key difference between the DP deep learning and the standard one is whether the gradient is privately released. In other words, while the standard optimizers update on  $\sum_{i} g_{i}$ , the DP optimizers update on the private gradient:
$^{1}$ Code for our experiments is available at FastDP library https://github.com/awslabs/fast-differential-privacy.
$$
\operatorname {D P} \operatorname {O p t i m i z e r} \left(\left\{\boldsymbol {g} _ {i} \right\} _ {i = 1} ^ {B}\right) = \operatorname {O p t i m i z e r} \left(\overbrace {\sum_ {i} \boldsymbol {g} _ {i} \cdot \operatorname {C l i p} \left(\left\| \boldsymbol {g} _ {i} \right\| ; R\right) + \sigma R \cdot \mathcal {N} (0 , \mathbf {I})}\right) \tag {1.1}
$$
$$
\text {S t a n d a r d O p e r i z e r} \left(\left\{\boldsymbol {g} _ {i} \right\} _ {i = 1} ^ {B}\right) = \text {O p t i m i z e r} \left(\sum_ {i} \boldsymbol {g} _ {i}\right) \tag {1.2}
$$
Here  $\pmb{g}_i\in \mathbb{R}^d$  is the per-sample gradient of loss  $l_{i},\mathcal{N}$  is the standard normal,  $\sigma$  is the noise multiplier, and  $R$  is the clipping threshold. The clipping function  $\mathrm{Clip}:\mathbb{R}^d\to \mathbb{R}$  is defined such that  $\| \pmb {g}_i\cdot \mathrm{Clip}(\pmb {g}_i;R)\| \leq R$ . For instance, the DP-SGD in [1] is
$$
\mathrm {D P - S G D} _ {\text {A b a d i}}: \quad \boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \left(\sum_ {i} \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} \min  \left(R / \left\| \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} \right\|, 1\right) + \sigma R \cdot \mathcal {N} (0, \mathbf {I})\right) \tag {1.3}
$$
In comparison to the regular training (1.2), two additional DP-specific hyperparameters  $R$  and  $\sigma$  need to be determined in DP learning (1.1). On the one hand, setting the noise multiplier  $\sigma$  is easy and can be derived analytically prior to the training. Whenever the privacy budget  $(\epsilon, \delta)$  is determined, one can apply off-the-shelf privacy accounting tools in Section 2.1 to determine  $\sigma$ , based on the subsampling probability  $p$  and the number of iterations  $T$ :
$$
\text {p r i v a c y} (\sigma , p, T; \delta) = \epsilon
$$
On the other hand, the choice of clipping threshold  $R$  is crucial to the performance of DP models, yet the hyperparameter tuning is much labor-intensive. Recent advances of DP deep learning on ImageNet [38] and on E2E datasets [41], using ResNet18 and GPT2 respectively, illustrate that the performance is very sensitive to  $R$ . We have reproduced their results in Figure 1. Observe that on ImageNet, ResNet18 can drop from the highest  $45\%$  accuracy to  $31\%$  if  $R$  is chosen 2 times larger, and to  $0.1\%$  if  $R$  is chosen 4 times larger. Similar drastic drop can also be observed in [38, Figure 3] even if the noise multiplier  $\sigma = 0$ . Unlike the noise multiplier  $\sigma$ , the clipping threshold  $R$  cannot be inferred from the privacy budget  $(\epsilon, \delta)$  and have to be tuned. Consequently, DP training necessarily requires an expensive 2D grid search for  $(R, \eta)$ , like Figure 1, whereas the regular training only requires an easy 1D grid search for  $\eta$ . Even worse, the difficulty of tuning a per-layer clipping threshold vector [49], i.e. one clipping threshold for one layer, may increase exponentially as the number of layers increases.
To save the effort of tuning  $R$ , previous researches have proposed different approaches. In [3, 58, 28, 31], researchers advocate to use data-adaptive information to select  $R$ , such as a specified quantile of the gradient norm distribution. These adaptive clipping methods can be a little ad-hoc: they often replace the need to tune  $R$  by the need to tune one or more new hyperparameters, e.g. the quantile to use and the ratio to split the privacy budget between the quantile decision and the gradient perturbation. Another approach used by the practitioners is to replace the single 2D grid search by multiple cheaper 1D grid searches. For example, the researchers propose, in [38, Section 3.3] to fine-tune  $\eta$  with non-DP SGD, fix  $\eta$  and sweep over various values of the clipping threshold  $R$  with DP-SGD, then further fix  $R$  and do one more grid search on  $\eta$ . However, tuning  $R$  formally in a data-dependent way (e.g. through cross-validation) introduces additional privacy loss [55], and most existing empirical work does not privately conduct hyperparameter tuning.
We take a completely different route by proposing a new clipping principle that removes  $R$ , instead of coming up with methods to find the appropriate  $R$ . We term our method as automatic clipping and the DP optimizers using it as automatic DP optimizers. Our contributions are:
1. We propose the automatic clipping in (4.1) that expunges the clipping threshold from general DP optimizers, making DP training as amenable as regular training. In large-scale tasks (GPT-level) like Figure 1, our automatic clipping can reduce the cost of ablation study by  $5 \times 2$ .
2. We show that automatic DP optimizers are as private and efficient as existing DP optimizers.
3. We show in Theorem 4 that automatic DP-SGD converges in the non-convex setting, at the same asymptotic convergence rate as the standard SGD. Our theoretical analysis successfully explains the training behaviors of deep learning in previous empirical works.
4. We demonstrate the superiority of automatic clipping on a variety of vision and language tasks, especially with large models including ResNet, RoBERTa and GPT2.
5. In Appendix K, we include simple code snippets that demonstrate how easy it is to switch from Abadi's clipping to our automatic clipping in popular codebases, e.g. Opacus and ObJAX.
2The hyperparameter tuning of  $(R,\eta)$  takes days (e.g. GPT2 [41]) to months (e.g. GPT3-175B) on large foundation models, highlighting the significance of our method to expunge the additional  $R$ .
We consider the  $(\epsilon, \delta)$ -DP in Definition 2.1, where smaller  $(\epsilon, \delta)$  means stronger privacy guarantee.
Definition 2.1 ([25]). A randomized algorithm  $M$  is  $(\varepsilon, \delta)$ -differentially private (DP) if for any two neighboring datasets  $S, S'$  (i.e. if one can obtain  $S'$  by adding or removing one data point from  $S$ ), and for any event  $E$ ,
$$
\mathbb {P} [ M (S) \in E ] \leqslant \mathrm {e} ^ {\varepsilon} \mathbb {P} [ M \left(S ^ {\prime}\right) \in E ] + \delta . \tag {2.1}
$$
In words, DP restricts the influence of an arbitrary sample, so that the information contributed by such sample is limited and less vulnerable to privacy attacks. In deep learning, DP is achieved by applying the subsampled Gaussian mechanism to privatize the minibatch gradients during training.
As illustrated in Equation (1.1), the subsampled Gaussian mechanism involves (I) sampling a minibatch by including each data point iid with probability  $p$  (II) per-sample gradient clipping to bound the  $l_{2}$  norm sensitivity at  $R$  and (III) adding independent Gaussian noise proportional to  $R$  and  $\sigma$ , where  $\sigma$  is derived from the privacy budget  $(\epsilon, \delta)$ . This can be realized by leveraging a variety of modern privacy accounting tools, such as Renyi DP (or moments accountant) [1, 51, 71], Privacy Loss distribution (Fourier accountants) [37, 30, 82], or Gaussian DP [19, 7].
2.2 Differentially Private optimizers with general clipping operations
Privately released stochastic gradients (through the Gaussian mechanism) can be used by various off-the-shelf optimizers, including DP-SGD in (1.3), DP-HeavyBall, DP-AdaGrad, DP-Adam, DP-FedAvg/FedSGD [49], etc. To improve the performance of DP optimizers, previous researches on the per-sample clipping can be classified into two categories.
The first category, where the majority of researches lie in, works with Abadi's clipping and focuses on better design of  $R$ . To name a few examples, one can adaptively design  $R_{t}$  for each iteration  $t$  [3, 58, 28], or design the per-layer clipping threshold vector  $\mathbf{R} \in \mathbb{R}^{L}$  for  $L$  layers [1, 49] so as to apply a different clipping threshold for each layer.
Fewer works fall into the second category that proposes new clipping methods. In fact, any function  $\mathrm{Clip}:\mathbb{R}^d\to \mathbb{R}$  satisfying  $\| \mathrm{Clip}(\pmb {g})\cdot \pmb {g}\| \leq R$  can serve as a valid clipping function besides Abadi's. For example, the global clipping [9] proposes  $\mathrm{Clip}_{\mathrm{global}}(\pmb {g}_i)\coloneqq \mathbb{I}(\| \pmb {g}_i\| <  R)$  to mitigate the bias of the private gradient and alleviate the mis-calibration issue of DP classifiers. Another example is the re-parameterized clipping [17],  $\mathrm{Clip}_{\mathrm{re - param}}(\pmb {g}_i)\coloneqq \min (1 / \| \pmb {g}_i\| ,1 / R)$ , which is equivalent to Abadi's clipping under a re-scaled learning rate. Our automatic clipping belongs to this category. We note that different clipping methods work orthogonally to optimizers, network architectures and gradient norm computation (see Section 8).
3.1 Small clipping threshold often works best
Figure 1: Ablation study of clipping threshold and learning rate. Left: BLEU score of GPT2 on E2E dataset [41], with DP-AdamW. Right: Test accuracy of ResNet18 on ImageNet [38], with DP-SGD.
One intriguing observation that we can make about the recent studies on DP learning with large models is that the state-of-the-art (SOTA) results are often achieved with very small clipping threshold  $R$ . This observation is consistent in both vision and language tasks. In [41], GPT2 (about 800 million parameters) and RoBERTa models (over 300 millions parameters) achieve the best results under DP on QNLI, MNLI, SST-2, QQP, E2E, and DART datasets, with each per-sample gradient clipped to length  $R = 0.1$ . In [38, 17, 50], ResNets and Vision Transformers achieve the best DP results on ImageNet with  $R = 1$ ; in [68], the best DP results on CIFAR10 use  $R = 0.1$  with ResNeXt-29 and SimCLRv2 [13]. The effectiveness of small clipping threshold together with proper learning rate is depicted in Figure 1.
Intuitively, smaller  $R$  implies that the Abadi's clipping (3.1) is effective, which means  $\min \left(R / \| g_i\| ,1\right) = R / \| g_i\|$ . Given that the clipping threshold  $R$  is so small compared to the number of parameters in large models, and that strong DP is guaranteed when the number of training iterations is small (i.e.  $\| g_{i}\|$  has not converged to small values yet), we expect and empirically observe that the clipping happens on a large proportion of per-sample gradients at all iterations. For instance, we find in the GPT2 generation experiments in [41] that  $100\%$  of per-sample gradients are clipped at all iterations; in classification tasks such as QQP, QNLI, and MNLI, the percentage of clipping is about  $20\sim 60\%$  on average (more details in Appendix H.1).
3.2 Per-sample gradient normalization as new clipping
In the small clipping threshold regime, we can approximately view
$$
\operatorname {C l i p} _ {\text {A b a d i}} \left(\boldsymbol {g} _ {i}; R\right) = \min  \left(R / \left\| \boldsymbol {g} _ {i} \right\|, 1\right) \approx R / \left\| \boldsymbol {g} _ {i} \right\| =: \operatorname {C l i p} _ {\text {A U T O - V}} \left(\boldsymbol {g} _ {i}; R\right) \tag {3.1}
$$
and thus derive a novel private gradient  $\sum_{i}R\frac{g_{i}}{\|g_{i}\|} +\sigma R\cdot \mathcal{N}(0,\mathbf{I})$ . Here AUTO-V stands for the vanilla automatic clipping, which essentially performs the normalization on each per-sample gradient. As a specific example, we can write the  $R$ -dependent automatic DP-SGD as
$$
R \text { - d e p e n d e n t} \mathrm {D P - S G D} _ {\mathrm {A U T O - V}}: \boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \left(\sum_ {i} R \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} / \| \frac {\partial l _ {i}}{\partial \boldsymbol {w} _ {t}} \| + \sigma R \cdot \mathcal {N} (0, \mathbf {I})\right) \tag {3.2}
$$
We may view our AUTO-V clipping as to maximize the dot-product similarity (a commonly used similarity measure, e.g. in the attention block in transformers [69]) between the clipped gradient and the regular gradient. Suppose we want to
$$
\max  _ {C _ {i}} \left\langle \sum_ {i} C _ {i} \boldsymbol {g} _ {i}, \sum_ {j} \boldsymbol {g} _ {j} \right\rangle \quad \text {s . t .} 0 \leq C _ {i} \leq R / \| \boldsymbol {g} _ {i} \| \tag {3.3}
$$
Note that the constraint is a sufficient condition for clipping, as discussed in Section 2.2. It is not hard to see that the optimal clipping factor (though violating DP guarantee<sup>3</sup>) regarding (3.3) is
$$
C _ {i} = R / \left\| \boldsymbol {g} _ {i} \right\| \cdot \mathbb {I} \left(\left\langle \boldsymbol {g} _ {i}, \sum_ {j} \boldsymbol {g} _ {j} \right\rangle > 0\right), \tag {3.4}
$$
If the per-sample gradients are indeed concentrated in the sense  $\forall i, \langle g_i, \sum_j g_j \rangle \geq 0$ , then AUTO-V is the optimal per-sample gradient clipping. We compare with Abadi's clipping in Figure 2, where this similarity is significantly magnified by our AUTO-V clipping. In fact, the dot-product similarity in (3.3) closely resembles the convergence of DP optimization for Theorem 4 in (C.2).
Figure 2: RoBERTa-base with DP-Adam  $(\epsilon = 3)$  on SST2 dataset, as in Section 6.2.
<sup>3</sup>In DP literature, per-sample clipping depend only on individual gradient  $\pmb{g}_i$  separately, hence does not allow the use of  $\sum_{j} \pmb{g}_{j}$ , which changes the sensitivity when adding or removing one data point from the mini-batch.
3.3 Stability constant breaks scale-invariance and remains stationary
One potential drawback of AUTO-V clipping is that all gradients lose their magnitudes information completely, since  $\| g_i\cdot \mathsf{Clip}_{\mathrm{AUTO - V}}(g_i;R)\| = R,\forall i$  . This scaleinvariance in AUTO-V and partially in Abadi's clipping (when  $\| g_i\| >R$  ) leads to the "lazy region" issue: the parameters will not be updated by DP-GD even if the true gradients are non-zero. In Figure 3, we illustrate such issue in a logistic regression for AUTO-V and Abadi's clipping, when the trainable parameter  $\theta \in [-2,2]$  , as the gradients from two classes cancel each other.
Figure 3: Gradient (scalar) at each  $\theta$ .
To preserve the magnitude information and thus escape the lazy region, we propose the AUTO-S clipping, with a positive stability constant  $\gamma$ :
$$
\operatorname {C l i p} _ {\text {A U T O - S}} \left(\boldsymbol {g} _ {i}; R\right) := R / \left(\left\| \boldsymbol {g} _ {i} \right\| + \gamma\right) \tag {3.5}
$$
We visualize in Figure 5 that AUTO-S allows larger per-sample gradients to have larger magnitudes after the clipping, while still allowing smaller gradients to vanish after "clipping". That is, as  $g_{i} \rightarrow 0$ , the existence of  $\gamma$  allows the clipped gradient  $C_i g_i \to g_i / \gamma$  rather than having a magnitude  $R$  as in AUTO-V. We elaborate this point in Section 4.3. This is critical in our convergence analysis and allows DP-SGD<sub>AUTO-S</sub> (but not DP-SGD<sub>AUTO-V</sub>) to converge to zero gradient norms in Section 5.
4 Automatic DP Training
One may wonder why our clipping (3.1)(3.5) is automatic at all, if the hyperparameter  $R$  is still present and there is an additional parameter  $\gamma$  to choose. It turns out that any constant choice of  $R > 0$  is equivalent to choosing  $R = 1$ , and common deep learning optimizers are insensitive to the choice of  $\gamma$  (e.g. for any  $\gamma > 0$ , we show that the gradient norm converges to zero at the same asymptotic rate in Theorem 4; see also the ablation study in Figure 15). Consequently, we set  $\gamma = 0.01$  as the default. Specifically, let us redefine the  $R$ -independent clipping function:
$$
\operatorname {C l i p} _ {\text {A U T O - S}} \left(\boldsymbol {g} _ {i}\right) := 1 / \left(\left\| \boldsymbol {g} _ {i} \right\| + \gamma\right). \tag {4.1}
$$
With this clipping, we can design automatic DP optimizers similar to (1.1):
$$
\text {A t m o m a t i c D P O p e r i z e r} \left(\left\{\boldsymbol {g} _ {i} \right\} _ {i = 1} ^ {B}\right) = \text {O p e r i m i z e r} \underbrace {\left(\sum_ {i} \frac {\boldsymbol {g} _ {t , i}}{| | \boldsymbol {g} _ {t , i} | | + \gamma} + \sigma \cdot \mathcal {N} (0 , \mathbf {I})\right)} _ {\text {d e n o t e d a s} \hat {\boldsymbol {g}} _ {t}} \tag {4.2}
$$
Clearly, the new private gradient  $\hat{g}_t$  from our automatic clipping is  $R$ -independent, in contrast to the one used in (1.1). A concrete example (in the case of  $\gamma = 0$ ) that is comparable to (3.2) will be
$$
R \text { - i n d e p e n d e n t} \quad \mathbf {w} _ {t + 1} = \mathbf {w} _ {t} - \eta \left(\sum_ {i} \frac {\partial l _ {i}}{\partial \mathbf {w} _ {t}} / \left\| \frac {\partial l _ {i}}{\partial \mathbf {w} _ {t}} \right\| + \sigma \cdot \mathcal {N} (0, \mathbf {I})\right) \tag {4.3}
$$
Leveraging the private gradient  $\hat{g}_t$  in (4.2), we can train DP neural networks without tuning DP-specific hyperparameters  $R$  and  $\sigma$ , as demonstrated in Algorithm 1.
Algorithm 1 Automatic Deep Learning with DP
Parameters: initial weights  $w_0$ , learning rate  $\eta_t$ , sampling probability  $p$ , number of iterations  $T$ .
1: Compute  $\sigma$  such that  $\epsilon_{\mathrm{Accountant}}(\delta, \sigma, p, T) \leq \epsilon$  from any privacy accountant.
2: for iteration  $t = 1,\dots ,T$  do
3: Sample a batch  $B_{t}$  by including each data point i.i.d. with probability  $p$
4: Apply automatic clipping to per-sample gradients  $\{\pmb{g}_i\}_{i\in B_t}$ :  $\hat{\pmb{g}}_i = \pmb {g}_i / (\| \pmb {g}_i\| _2 + 0.01)$
5: Add Gaussian noise to the sum of clipped gradients:  $\hat{\pmb{g}} = \sum_{i}\hat{\pmb{g}}_{i} + \sigma \cdot \mathcal{N}(0,\mathbf{I})$
6: Update  $\boldsymbol{w}_t$  by any optimizer on the private gradient  $\hat{\boldsymbol{g}}$  with learning rate  $\eta_t$ .
4The settings are in Appendix F, where the lazy region issues also emerge in the mean estimation problem. We note that the lazy region is also discussed in [14, Example 2].
We will elaborate two distinct reasons in the next sub-sections for the following statement:
$$
\boxed {\mathrm {D P O p e r i z e r} _ {\mathrm {A b a d i}} \approx R \text {- d e p e n d e n t D P O p e r i z e r} _ {\mathrm {A U T O}} \equiv R \text {- i n d e p e n d e n t D P O p e r i z e r} _ {\mathrm {A U T O}}}
$$
which expunges the DP hyperparameters, only leaving us the regular hyperparameters such as learning rate, weight decay, etc. The significant save in the tuning effort is illustrated in Figure 4.
Figure 4: Test accuracy of RoBERTa-base by different clipping thresholds  $R$  and learning rates  $\eta$ . This is trained with DP-Adam (Abadi and AUTO-S) on SST2 (left, 3 epochs), QNLI (middle, 1 epoch), and MNLI (right, 1 epoch), under  $\epsilon = 3$ . Notice by only searching along  $\eta$ , instead of over  $(R, \eta)$ , we can save the cost of hyperparameter tuning by  $5 \times$ .
4.1 Non-adaptive optimizer couples clipping threshold with learning rate
With  $R$ -dependent automatic clipping, DP-SGD becomes
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \left(\sum_ {i} \boldsymbol {g} _ {t, i} \cdot \frac {R}{\| \boldsymbol {g} _ {t , i} \| + \gamma} + \sigma R \cdot \mathcal {N} (0, \mathbf {I})\right) = \boldsymbol {w} _ {t} - \eta R \hat {\boldsymbol {g}} _ {t}.
$$
We can view  $\eta_{\mathrm{effective}} \equiv \eta R$  as a whole: increasing  $R$  has the same effect as increasing  $\eta$ , which explains the diagonal pattern in Figure 1(lower plot) where DP-  $\mathrm{SGD}_{\mathrm{Abadi}}$  is applied with small clipping threshold. We extend to general non-adaptive optimizers in Theorem  $1^{5}$ .
Theorem 1. Non-adaptive  $R$ -dependent automatic DP optimizers (including SGD, Heavyball[59] and NAG[54]), with learning rate  $\eta$  and weight decay  $\lambda$ , is equivalent to  $R$ -independent automatic DP optimizers, with learning rate  $\eta R$  and weight decay  $\lambda / R$ .
4.2 Adaptive optimizer can be insensitive to clipping threshold
Adaptive automatic DP optimizers are different than the non-adaptive ones, as the clipping threshold cancels out instead of being coupled with learning rate. To see this, we scrutinize DP-AdamAbadi (which is similar to DP-AdamAUTO-V) in Figure 1(upper plot), where columns to the left are almost identical. Further evidence is observed in [50, Table 5] that shrinking  $R$  has zero effect on LAMB. We now give a simple explanation using AdaGrad [22]:
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \frac {\boldsymbol {g} _ {t}}{\sqrt {\sum_ {\tau <   t} \boldsymbol {g} _ {\tau} ^ {2}}}
$$
where  $\pmb{g}_t = \sum_i\pmb{g}_{t,i}$  is the gradient sum. In  $R$  -dependent DP-AdaGradAUTO-V, the private gradient is  $R\hat{\pmb{g}}_t$  in place of the standard gradient sum  $\pmb{g}_t$ :
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \eta \frac {R \hat {\boldsymbol {g}} _ {t}}{\sqrt {R ^ {2} \sum_ {\tau \leq t} \hat {\boldsymbol {g}} _ {\tau} ^ {2}}} = \boldsymbol {w} _ {t} - \eta \frac {\hat {\boldsymbol {g}} _ {t}}{\sqrt {\sum_ {\tau <   t} (\hat {\boldsymbol {g}} _ {\tau}) ^ {2}}}.
$$
We generalize to other adaptive optimizers in Theorem 2 and to the per-layer clipping style in Appendix B.3.
Theorem 2. Adaptive  $R$ -dependent automatic DP optimizers (e.g. AdaGrad[22], AdaDelta[79], AdaMax/Adam[35], NAdam[20], RAdam[43], LARS[75], LAMB[76]), with learning rate  $\eta$  and weight decay  $\lambda$  is equivalent to  $R$ -independent automatic DP optimizers with learning rate  $\eta$  and weight decay  $\lambda / R$ . With decoupled weight decay[46],  $R$ -dependent automatic DP-AdamW is equivalent to  $R$ -independent automatic DP-AdamW with the same  $\eta$  and  $\lambda$ .