Automatic Clipping: Differentially Private Deep Learning Made Easy and Stronger

Abstract: Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping norm $R$, however, is shown to be vital for achieving high accuracy under DP. We propose an easy-to-use replacement, called AutoClipping, that eliminates the need to tune $R$ for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB and many others.
The automatic variants are as private and computationally efficient as existing DP optimizers, but require no DP-specific hyperparameters and thus make DP training as amenable as the standard non-private training. We give a rigorous convergence analysis of automatic DP-SGD in the non-convex setting, which shows that it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric noise assumption of the per-sample gradients. We also demonstrate on various language and vision tasks that automatic clipping outperforms or matches the state-of-the-art, and can be easily employed with minimal changes to existing codebases.

Reviews and author responses:

 We thank again the reviewer for participating in the discussion! We totally agree that the distortion of automatic clipping (and Abadi's clipping) needs more study. We would like to mention two facts that may alleviate your concern about the bias:

1. Bias is not always fatal for the convergence. As shown in our proof (Equation (C.2) Line 618), the convergence is largely related to $g_t^\top \mathbb{E}(\text{clipped grad})$, which motivates our maximization of similarity in Line 136, regardless of whether the true gradient or many per-sample gradients are small. That is, we admit that automatic clipping may introduce bias to true gradient, but that does not necessarily hurt the convergence (it slows down the convergence but DP-SGD still converges under our assumptions; an unrealistic but intuitional example: suppose per-sample grad $\tilde g_t=g_t+v$, where $v$ is the distortion bias, if $v$ is orthogonal to $g_t$, it does not hurt as $g_t^\top \mathbb{E}(\text{clipped grad})=\|g_t\|^2$).

2. The scenario that automatic clipping introduces more bias than Abadi's clipping is not universal. As the reviewer correctly points out,"if the true gradient is within this shell and many gradients are within the clipping bound I think the proposed method leads to more bias." In fact, we claimed in Line 124 "For instance, we find in the GPT2 generation experiments in [37] that 100% of per-sample gradients are clipped at all iterations." In this specific example, if we plot a similar plot to Figure 12, it would be a horizontal line (which is why we didn't plot it). And actually AUTO-V is the same as Abadi's clipping so no more bias is introduced.

In summary, automatic clipping may not introduce more bias than Abadi's (e.g. in GPT2 experiments), and even if so, the bias may not necessarily hurt the convergence. 

We admit there are cases that the disadvantage indeed makes AUTO clipping slightly under-performs Abadi's (see the the red arrows in Appendix J for the CelebA). Nevertheless, automatic clipping even AUTO-V at least serves as a useful baseline for practitioners, since we don't need to tune clipping threshold and the choice of $\gamma$ is empirically robust (hopefully the reviewer is satisfied with the additional experiments).

We are more than happy to extend the discussion and polish the paper accordingly. Given that the rebuttal period ends in 15 min, we would appreciate it if you can reflect your new evaluation in the rating.

 Thank the authors for the detailed response, however, I do not quite agree with the response to the second point. Just to leave a quick comment in case the author would like to discuss more. 

When $\gamma$ is much larger than the norm of the gradient, distortion will not be the major negativ effect as all gradients will be largely shrunk due to the large $\gamma$. So this case is not appropriate to alleviate the concern about the gradient distortion caused by the proposed method. The proposed method in this work squeeze all gradients into a shell with radius 1, for auto-v the shell has no volume, while for auto-s the shell has a thickness controlled by $\gamma$. I can agree that when all gradient points are concentrated at one direction this method is effective by considering maximising cosine similarity, but if the true gradient is within this shell and many gradients are within the clipping bound I think the proposed method leads to more bias. Furthermore, by seeing the Figure 12 in the appendix, most gradients (~70% in average) are not clipped after a dozens of iterations, i.e. more than 70% of training time, which actually implies that the distortion problem should be technically non-negligible problem of the proposed method.

 Thank you for the discussion! We hope we can alleviate your concern about Assumption 5.3 and earn us more credits, especially given that our rating is on the borderline.

Firstly, there is empirical evidence to support Assumption 5.3 in [Understanding Gradient Clipping in Private SGD: A Geometric Perspective](https://proceedings.neurips.cc/paper/2020/file/9ecff5455677b38d19f49ce658ef0608-Paper.pdf), which makes the same assumption to analyze DP-SGD. Our Assumption 5.3 has been empirically verified by Figure 1/2/3, where per-sample gradients are projected to 2D (via random Gaussian matrix, repeatedly, see Figure 2). The symmetry is visually presented around the origin, thus this central symmetry is recognized in multiple datasets.

Secondly, from a theoretical viewpoint, this central symmetry is also widely used in non-DP SGD analysis [1,2,3,4]. For instance, in Equation (1) of [1], the mini-batched gradient is $g+N(0,C)$, where $g$ is true gradient and $C$ is the gradient noise covariance matrix. It is also standard to use Brownian motion to characterize the gradient flow of SGD, which means the gradient noise is Gaussian. Setting batch size to 1, this well-recognized assumption is equivalent to that per-sample gradient is centrally symmetric around the true gradient. In fact, this non-DP assumption is more restrict than our Assumption 5.3: we don't assume the Gaussian distribution but only assume the symmetry.

[1] Xie, Zeke, Issei Sato, and Masashi Sugiyama. "A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima." International Conference on Learning Representations. 2020.

[2] Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate
bayesian inference. The Journal of Machine Learning Research, 18(1):4873–4907, 2017.

[3] Samuel L Smith and Quoc V Le. A bayesian perspective on generalization and stochastic gradient
descent. In International Conference on Learning Representations, 2018.

[4] Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference,
converges to limit cycles for deep networks. In 2018 Information Theory and Applications
Workshop (ITA), pp. 1–10. IEEE, 2018.

Lastly, we would like to point out that our central symmetry is non-trivial, in contrast to simply assuming per-sample gradients are symmetric about the direction of true gradient (i.e. axial symmetry). Notice that clipped gradient sum can still be biased and makes the analysis already challenging, while axial symmetry makes the clipped gradient sum to be unbiased and unrealistic.

In short, central symmetry could be appropriate, as it captures the bias from per-sample clipping and has empirical/theoretical support in previous literature. We are happy to add another appendix to clarify this assumption in the camera-ready version, so that subsequent works can be more aware of its applicability.

 I would like to thank the author(s) for the detailed response, some of my questions are resolved. However, I am still a little bit concerned about Assumption 5.3. I understand that analyzing the convergence with clipping is challenging, but I am not sure if simply assume 5.3 is the correct way to circumvent this technical issue. My feeling is that this kind of theoretical analysis would encourage subsequent works in the DP community to simply use Assumption 5.3 as a well-recognized assumption, which I think is inapproperiate. 

Again, I appreciate the empirical contribution of this paper. But for the theoretical analysis, I suggest the author(s) to remove the symmetric noise assumption. Doing so may end up with a non-vanishing bias term in the convergence analysis due to clipping, but I think the comparison to existing result will be more fair. 

 We thank the reviewer for the questions, all of which have been carefully addressed by our response. If there are no more concerns, we would really appreciate it if the reviewer can raise the score. We are happy to discuss had further questions come up.

 We thank the reviewer for the questions, all of which have been carefully addressed by our response. If there are no more concerns, we would really appreciate it if the reviewer can raise the score. We are happy to discuss had further questions come up.

 We thank the reviewer for the questions, all of which have been carefully addressed by our response. If there are no more concerns, we would really appreciate it if the reviewer can raise the score. We are happy to discuss had further questions come up.

 We thank the reviewer for the questions, all of which have been carefully addressed by our response. If there are no more concerns, we would really appreciate it if the reviewer can raise the score. We are happy to discuss had further questions come up.

 We thank all reviewers and area chair for handling our paper. We have addressed each comment and highlighted the modification in the main context (in yellow). In summary,
* We change the formatting of DP optimizers, e.g. Equation (1.3), to match the correct DP mechanism we implemented --- Poisson sampled Gaussian mechanism.  We assure the reviewers this change does not affect our claims and proofs.
 * As per Reviewer PkCD's request, we conduct new ablation study on SST2 and QNLI, with RoBERTa-base and RoBERTa-large, in Figure 14 and Figure 15. The new experiments train the same number of epochs as in Section 6.2 (instead of training for a single epoch) to enhance the consistency. The new ablation study provides even stronger support to our observation that the accuracy is insensitive to the choice of $\gamma$ (over a large range).
* We add a new Theorem 5 (taking both privacy and optimization viewpoints), which complements Theorem 4 (only taking the optimization viewpoint). Theorem 5 calibrates the hyperparameter $\sigma$ to achieve a prescribed privacy budget $\mu$ ($\mu$-GDP, analytically equivalent to $(\epsilon,\delta)$-DP; we also discussed tCDP). This provides a direct insight on how privacy trades-off with convergence.
* We include a short discussion to mark the difference and similarity between De et al. "Unlocking High-Accuracy Differentially Private Image Classification through Scale" and our work in footnote 4.

 **Original Comment** *The contribution on the theoretical side is a little bit exaggerated. The main problem is that the author(s) use a much stronger assumption (Assumption 5.3) when analyzing the convergence. This assumption comes from [1], it was shown that this assumption can simplify the convergence when gradient clipping is adopted. However, this assumption can be false for simple linear regression problem. Therefore the claim in the abstract “the proposed clipping enjoys an convergence rate that matches the standard SGD” is kind of over-claimed since the convergence analysis of the proposed method and standard SGD rely on a different set of assumptions. Can we relax Assumption 5.3?*

**Response:** We appreciate reviewer's constructive criticism on our theoretical analysis and claim. To make our claim more accurate, we reworded the part ''it enjoys an asymptotic convergence rate that matches the standard SGD.'' to ''it can enjoy an asymptotic convergence rate that matches the standard SGD, under a symmetric noise assumption of the per-sample gradients.''

At high level, Assumption 5.3 may be relaxed but cannot be removed. Examples in Reference [1,2,3] have shown that only with Assumption 5.1 and 5.2, DP-SGD may not converge to stationary points (with zero true gradient) even in the convex cases. To drive the point home, we provide the following simple example (two data points, linear loss, $1$-strongly convex due to regularization).  Let $w\in \mathbb{R}$, and the optimization problem is 
$$\min_{w} \ell_1(w) + \ell_2(w) + w^2 \text{ where } \ell_1(w) = -w, \ell_2(w)= 2w.$$  Without per-example gradient clipping, GD converges to an optimal solution $w^* = -0.5$. On the other hand, GD with per-example clipped gradient $R\leq 1$ will instead converge to the wrong solution at $0$.


It is common that strong conclusion usually comes with strong assumption. Therefore, we expect Assumption 5.3 may not be relaxed much, given that our current conclusion leads to the vanishing gradient (i.e. convergence to a stationary point). If we relax Assumption 5.3 to an $\epsilon$-approximate version, it is not hard to show that it converges to a $c\epsilon + o(1)$-stationary point, though this makes the presentation of the theory messier.

We would also like to emphasize that even though Assumption 5.3 is a bit strong, our analysis aligns amazingly well with the empirical success.
For example, our Theorem 4 based on Assumption 5.3 clearly proves the advantage of using any positive $\gamma$, as we achieves higher utility on over 10 datasets in comparison to AUTO-V and Abadi's clipping. The current convergence analysis in Theorem 4 also nicely matches various observed behaviors of DP-SGD (see [original submission's line 233-241]), and justifies various empirical observations made by us and by prior work.  Nothing of its kind exists before. We believe it complements our empirical results well.

[1] Chen, Xiangyi, Steven Z. Wu, and Mingyi Hong. "Understanding gradient clipping in private SGD: A geometric perspective." Advances in Neural Information Processing Systems 33 (2020): 13773-13782.

[2] Song, Shuang, Om Thakkar, and Abhradeep Thakurta. "Characterizing private clipped gradient descent on convex generalized linear problems." arXiv preprint arXiv:2006.06783 (2020).

[3] Bu, Zhiqi, et al. "On the convergence and calibration of deep learning with differential privacy." arXiv preprint arXiv:2106.07830 (2021).

 **Original Comment** *The AUTO-S method fused \mathbb{R} into $\eta$ and therefore does not require to tune the clipping threshold R. However, this may end up requiring a more careful tuning of the learning rate since R is implicitly involved in $\eta$ now. Does AUTO-S require a more careful tunning of $\eta$? Does Abadi's clipping and AUTO-S use different learning rate in the experiments?*

**Response:** We thank the reviewer for this question. We validated that AUTO-S does not require a more careful tuning of learning rate. In fact, we directly used the learning rate from Li et al. [36] and other papers which is indeed optimally tuned for DP-SGD with Abadi's clipping. Therefore, it is slightly surprising that AUTO-S worked out of the box, even though it may enjoy a better utility if we specifically tune the learning rate for our clipping. This choice is explicitly stated in [original submission's line 246/247/248/261/264/272] and listed in Table 5 and Table 6.

We have also tried to more carefully tune learning rates for both Abadi's clipping and AUTO-S by searching through a finer exponential grid, but the numbers remain very similar to simply using  Li et al. [36]'s learning rates, thus this is not reported in the paper.  Please also see our ablation study on learning rates (Figure 14 - 15) to convince yourself that it is not a particularly sensitive choice (at least not more sensitive than that of Abadi's clipping).

**Original Comment** *Overall, I appreciate the empirical contribution of this paper but I think the claimed theoretical contribution should be weakened. With the theoretical contribution being discounted, I am not sure if this paper has enough contribution to be accepted at NeurIPS.*

**Response:** We would like to highlight the significance of our contributions. On the theoretical part, we establish the first equivalence between small-R Abadi's clipping and automatic clipping (AUTO-V), which rigorously explain the two completely different performance landscapes with respect to learning rate and R, as empirically observed in previous works and summarized in Figure 1. We further propose and analyze AUTO-S clipping to solve the non-convergence issue in existing clipping. This is supported both by Theorem 4 and the empirical evidence in Figure 13. On the empirical part, our automatic clipping is the only clipping method that rigorously removes the need to tune the clipping threshold. Given that large models are difficult to train in DP regime, taking days for a single pair of hyperparameter, our method saves the tuning effort by an order of magnitude, e.g. 5 times in Figure 1. It is also empirically encouraging that AUTO-S gives outperforming utility in multiple domains.

In short, we provide new method (AUTO-S/V), new analysis (convergence and automaticity), new state-of-the-art accuracy (on image/text classification and generation), detailed implementation in multiple codebases, and substantial saving in DP hyperparameter tuning at a methodology level. We hope the reviewer would agree that our contribution is effectively pushing the frontier of DP researches.

**Original Comment** *What does the number of the heat map in Figure 1 stand for?*

**Response:** The left plot shows BLEU score and the right plot shows test accuracy. This is added to the caption of Figure 1 in the revision.

 **Original Comment** *On contribution, De et al. [1] also introduce a method to decouple learning rate tuning from clipping norm tuning (making the claim in line 280 incorrect). They do this by reparameterizing DP-SGD such that the learning rate absorbs a factor of R, and so the clipping norm has no influence on the scale of the update. Similar to yours, they then set  in all main experiments. E.g. See App B.1 in [1] where the choice of clipping norm has practically no effect on validation accuracy across different learning rates. Could the authors comment on the comparison between the idea in [1] and this work, and what they see as the contributions made here?*

**Response:** We thank the reviewer for drawing our attention to the recent work of De et al. [1]. We re-state their reparameterization (which is equivalent to original Abadi's clipping mathematically) and our AUTO-S/V here for clearer comparison. 
\begin{align}
w_{t+1}&=w_t-\frac{\eta}{B}\left(\sum_i g_i\cdot\text{min}(\frac{1}{R},\frac{1}{\|g_i\|}\)+\sigma \mathcal{N}(0,I)\right)
\end{align}
\begin{align}
w_{t+1}&=w_t-\frac{\eta}{B}\left(\sum_i g_i\cdot\frac{1}{\|g_i\|+\gamma}+\sigma \mathcal{N}(0,I)\right),
\end{align}
where $g_i$ is per-sample gradient.

While the reparameterization trick by De et al. is a useful empirical technique, it provides no guarantee that ``the clipping norm has no influence on the scale of the update'' in all cases. To illustrate this, we refer to Figure 8, App B.1 in [1], where the dependence on clipping norm is obvious unless $R<1$. As this criterion $R<1$ is entirely empirical, it may not hold on other datasets and would again need expensive hyperparameter tuning to find. In contrast, our automatic clipping rigorously expunges the effect of $R$, therefore if we draw a similar plot to their Figure 8, we would see a flat line including $R>1$ (up to randomness in the Gaussian noise).

There are several benefits to make the analysis rigorous besides pure empirical observations. Firstly, even though De et al. observes that the learning rate couples with R in
DP-SGD$_\text{Abadi}$ when R is small, we are the first to establish the different pattern of all adaptive optimizers (e.g. Adam, which is insensitive to R, rather than coupling R with learning rate). Secondly, only in our Theorem 1 and Theorem 2, we find the relationship between tuning R and weight decay. For example, even in the small R regime, running DP-SGD with $(\eta,R)$ is still different from running $(\eta/2,2R)$, because the weight decay is also halved! Thirdly, De et al. use small clipping norm for DP-SGD, which is only equivalent to AUTO-V but we show, both empirically and theoretically, that our new AUTO-S can enjoy better convergence.

In summary, our contributions additional to [1] are: 

(1) proposing the strong AUTO-S clipping (with a by-product AUTO-V), which is not equivalent to Abadi's clipping even if R is set small; only automatic clipping guarantees independence of R under cases (thus removing tuning), whereas De et al. require tuning to find small R, otherwise the performance still depends on R;

(2) providing the convergence analysis, whereas [1] is completely empirical; examples have shown Abadi's DP-SGD and AUTO-V may not converge;

(3) introducing additional baselines in language models, while [1] only works on computer vision;

(4) studying the adaptive optimizers like AdaGrad, AdaDelta, Adam, etc., which has completely different dependence on R than DP-SGD analyzed by De et al.; we also reveal the relationship between R and weight decay for the first time;

 **Original Comment** *In terms of motivation, can the authors comment on how much of a problem tuning  is in practice? For the NLP tasks, it seems like the choice of  is insensitive to learning rate (if the learning rate is sufficiently large), while for ImageNet it seems like we only need to choose  to be of the same order of magnitude as the learning rate. Is it possible to use these as a rule-of-thumb over different vision and NLP tasks? I'd be curious to know how sensitive these tasks are to the learning rate without privacy. Could the authors provide these numbers?*

**Response:** The tuning of R is indeed a big challenge in practice, as privately training large models (transformer level) takes days for one pair of hyperparameters. For NLP and image tasks (see our Figure 15 and also Figure 1), the effort can be saved by 5 times if we only tune learning rate. As for using the tuning trick as a rule-of-thumb, it still requires heavy tuning for Abadi's (or equivalently De et al.'s) clipping. We believe it makes sense to use that as a rule-of-thumb only because we formally make AUTO-S/V mathematically equivalent in Theorem 1 and 2, say the insensitivity of R in adaptive optimizers and the multiplicative pattern in non-adaptive SGD. As a concrete example, we look at left plot in Figure 1 and suppose one starts with $(0.1*2^8,10^{-3}/2^4)$, because one doesn't know what is a 'small R' or a 'sufficiently large learning rate' for this specific task. If one uses the existing clipping, then using the rule that `DP-AdamW is not sensitive to R', one may want to try different learning rate, where the best learning rate gives 35.44 BLEU and one needs to violate the rule to tune R a few times to get the best 60.26 BLEU (total tuning 6 times). In stark contrast, if one uses the automatic DP-AdamW, one only needs to move along the column (total tuning 2 times).

For the non-private accuracy, we use the setting as in Li et al. (https://arxiv.org/abs/2110.05679) without further tuning. We are happy to provide the non-private numbers for SST2 with RoBERTa-base and RoBERTa-large. These numbers show that the results are not sensitive to learning rates around the optimal choice.

| lr            | 1e-5   | 3e-5   | 5e-5   | 8e-5   | 1e-4  |
|---------------|--------|--------|--------|--------|-------|
| RoBERTa-base  | 93.92  | 94.38  | 94.49  | 94.72  | 93.35 |
| RoBERTa-large | 95.76  | 96.21  | 96.21  | 96.33  | 95.99 |

We would love to run more experiments if you think it helps with your evaluation of our work.



**Original Comment** *Additionally, does the choice of architecture make a difference? I.e. what if I were to choose VGG-19 instead of ResNet in Fig 1, would the plot look the same? Why are the ablation studies in Fig 14 (not Fig 13 as stated in line 151) for different datasets than in Fig 1? I was expecting to see an identical plot to Fig 1 for a fixed choice of  showing the problems observed are no longer present.*

**Response:** By our analysis, stated in [original submission's line 107-108 and 116], the plots will look similar in terms of the diagonal pattern for non-adaptive optimizers (right) and column-insensitive pattern for adaptive optimizers (left), when the clipping norm is small. This is always the case regardless of the architecture and only depends on the choice of optimizers, adaptive or not. In words, small clipping threshold means the clipping is close to AUTO-V, which is analyzed in Theorem 1 and Theorem 2.

The ablation study was conducted on different datasets to share insight on a larger range $\gamma$ and learning rate, as ImageNet and GPT2 are taking too long to fill in such a large table. We add the ablation study on fully trained SST2 and QNLI in the revision (Figure 14). There is no need to experiment on the setting in Fig 1 with automatic clipping, because we have theoretically shown tuning R has no effect (up to randomness in the Gaussian noise addition; please see our Figure 15).

It is a matter of opinions but we believe AUTO-S is one of those ideas that is simple, effective and comes with a theoretical guarantee. We hope our response addresses your concern properly and would appreciate if you are willing to increase the score.

 **Original Comment:** *I see some layout problems, for example, Figure 6 has a broken link, and in line 151, I think the ablation study is presented in Figure 14 instead of 13. There are some others, this paper should be carefully polished before it is published.*

**Response:** Many thanks for the careful review and for spotting the typo and broken links. We have polished the paper and fixed typos in the revision that is available now. For Figure 6, the missing setting is that $\xi=1$.

**Original Comment:** *The study of $\gamma$ should be more thorough, it is very important to know that tuning work is not transferred from R to $\gamma$. A disadvantage of the proposed automatic clipping is that it always distorts the distribution of gradients even if all gradients locates within the clipping bound. According to the experiments in this work, the clipping rate dramatically reduces after some iterations indicating that many gradients are not clipped with Abadi's clipping. Therefore automatic clipping theoretically should carefully tune the hyperparameters which determine the distortion, e.g. $\gamma$.*

**Response:** We would like to share two observations to address the reviewer's concern on the distortion of the gradients. 

First, in an idealistic situation (e.g., approaching 0 training loss), all individual gradients are small, we agree that Abadi's clipping causes no distortion; but thanks to the stability constant $\gamma$, discussed in [original submission's line 139], the distortion in our approach is also very small. To be a little specific, when $g_i$ are all sufficiently smaller than $\gamma$, then $\sum_i g_i\cdot\frac{1}{\|g_i\|+\gamma}\to\sum_i g_i\cdot\frac{1}{\gamma}$, which is approximately unbiased.

Second, from the experiments in our work, the clipping rate may or may not reduce substantially enough to make Abadi's clipping unbiased during the training. For example, in [original submission's line 112], we observed that for GPT2 on E2E dataset, 100\% of the per-sample gradients are clipped at all iterations, and on SST2/QNLI/MNLI/QQP datasets, a large fraction of per-sample gradients remain clipped throughout training [Figure 12 in our appendix]. For this reason, we find that in practice, the distortion to the gradients remains an issue for both Abadi's clipping and for our automatic clipping.

In addition, we have an interesting explanation to why AUTO-S often outperforms rather than merely matches Abadi's clipping in experiments.  We believe AUTO-S could actually benefit from the distortion to the gradient in the initial phases of training. Gradients from most data points would mostly agree with each other in terms of the general direction to move towards a low-error "ball park".  In Abadi's clipping, some data points are clipped, others remain small; but in AUTO-S/AUTO-V, gradients from those not clipped are effectively normalized to have the largest possible magnitude, so it provides the maximum signal allowable in the face of the same amount of noise due to DP. This helps with reducing the variance while introducing a small bias (distortion) and it allows automatic DP-SGD to take larger strides at the beginning than Abadi's clipping. Our derivation of AUTO-V in Section 3.2 provides a more rigorous treatment to this effect, and we could add this further discussion to the revised paper if the reviewer find this explanation insightful.

**Original Comment:** *Although the authors argue that the network is insensitive to the choice of $\gamma$, I note that the ablation study of $\gamma$ is conducted with one epoch training, however, this could not represent the whole training procedure, complete the whole training epochs would be more convincing. I'd be happy to increase the score if my question is answered, overall I still think this is a good work.*

**Response:** We have added new ablation study to train the whole epochs (3 for SST2, 6 for QNLI) on both RoBERTa-base and RoBERTa-large. We thank the reviewer for the insightful discussion and kindly refer to Figure 14 in the revision, of which one plot (RoBERTa-large, SST2) is reproduced below. Reassuringly, the patterns match with the our initial single-epoch training, and the accuracy is insensitive to a wide range of $\gamma$. We hope this addressed your concerns and we would appreciate if you could consider raising the score. 

| learning rate \ clipping threshold | 1e-4  | 1e-3  | 1e-2  | 1e-1  | 1     | 10    | 100   | 1000 |
|------------------------------------|-------|-------|-------|-------|-------|-------|-------|------|
| 0.001                              | 91.9  | 92.3  | 92.3  | 92.0  | 92.3  | 92.2  | 91.7  | 89.7 |
| 0.0005                             | 93.0  | 93.5  | 93.6  | 93.5  | 93.6  | 93.8  | 94.2  | 93.0 |
| 0.0002                             | 93.7  | 94.0  | 94.5  | 94.5  | 94.5  | 95.0  | 95.1  | 93.2 |
| 0.0001                             | 93.9  | 94.6  | 95.1  | 95.0  | 95.2  | 95.2  | 94.3  | 92.8 |


 **Original Comment:** *Normalizing per-example gradients is equivalent to setting a very small clipping norm R (smaller than minimum of gradient norm of any example), which conflates the effect of a small clipping norm and $\gamma$. I encourage the authors to make it more explicit. I would request the authors to conduct a study on at least one dataset where AUTO-S/V is superior than vanilla DP-SGD clipping, where we track what happens when 1) clipping norm R is made smaller in the baseline and 2) change $\gamma$ in AUTO-S/V*

**Response:** We thank the reviewer for the insightful comments. We share the concern on conflating the effects and happen to have conducted the ablation study. The results can be found in the supplementary, e.g. Figure 14 (for SST2 and QNLI datasets) and Figure 15 (for SST2/QNLI/MNLI datasets). In short, making R smaller than a certain constant (say 1) in Abadi's clipping or changing $\gamma$ in the range from 0.0001 to 0.1 for AUTO-S achieves comparable accuracy. We additionally conduct experiments on full training on QNLI (6 epochs) and SST2 (3 epochs) to show the same pattern. We kindly refer the reviewer to the revision Figure 14, top two panels.

We'd like to highlight that even in cases when AUTO-S is just on par with the vanilla Abadi's clipping, the savings from avoiding the hyperparameter tuning for R can be significant. For example, the training of a large transformer model on difficult tasks like QQP and MNLI takes 1 day on Ampere A100 GPU per set of hyperparameters $(\eta,R)$. With automatic clipping, we can save substantial effort on the hyperparameter tuning by one order of magnitude, e.g. by 5 times in Figure 15.

We hope the above satisfactorily addressed your concerns and we would appreciate if you could consider raising the score.

 This paper proposes a novel way to automatically clip which eschews the need for an extra hyperparameter. This saves from a combinatorial increases in number of trials in sweep to tune DP hyperparameters. Both convergence analysis of the proposed DP optimizer and experiments on practical setup is provided. Strengths
- Great overall direction in reducing total cost of training with differential privacy.
- Paper is well-written, with a mix of intuition and illustrative experiments.
- Experiments on practical models and datasets suggest strong performance with proposed AUTO-S/V clipping scheme.

Weaknesses
- Normalizing per-example gradients is equivalent to setting a very small clipping norm R (smaller than minimum of gradient norm of any example), which conflates the effect of a small clipping norm and \gamma. I encourage the authors to make it more explicit. As mentioned as weakness above, I would request the authors to conduct a study on at least one dataset where AUTO-S/V is superior than vanilla DP-SGD clipping, where we track what happens when 1) clipping norm R is made smaller in the baseline and 2) change \gamma in AUTO-S/V Yes

 Differentially private SGD has two additional components over SGD: gradient clipping and noise addition. The clipping parameter and scale of added noise determine the privacy of the algorithm. For a target privacy budget, if we allow the scale to vary accordingly, then the clipping parameter becomes a hyperparameter that must be tuned. This work proposes a method to expunge the tuning of the clipping hyperparameter. The idea behind this work is to remove the need to tune the clipping norm parameter C. Instead of a practitioner choosing C and then applying $g\cdot \min({\frac{C}{||g||}, 1})$, we set $C=1$ and apply $\frac{1}{||g|| + \gamma}$. The private gradient becomes independent of the clipping norm, and we only need to choose $\gamma$. The core idea can be seen in Figure 4, where the idea is to create a smoothed version of clipping operation, where the rate of increase of norm depends of $\gamma$. Ablations show that the experimental results are insensitive to $\gamma$ and so can be set to 0.01 for all experiments.

Overall, the paper is well-written and contains sound justification for why this procedure works. My main concerns are two-fold: contribution and significance.

On contribution, De et al. [1] also introduce a method to decouple learning rate tuning from clipping norm tuning (making the claim in line 280 incorrect). They do this by reparameterizing DP-SGD such that the learning rate absorbs a factor of $C$, and so the clipping norm has no influence on the scale of the update. Similar to yours, they then set $C=1$ in all main experiments. E.g. See App B.1 in [1] where the choice of clipping norm has practically no effect on validation accuracy across different learning rates. Could the authors comment on the comparison between the idea in [1] and this work, and what they see as the contributions made here?

In terms of motivation, can the authors comment on how much of a problem tuning $C$ is in practice? For the NLP tasks, it seems like the choice of $C$ is insensitive to learning rate (if the learning rate is sufficiently large), while for ImageNet it seems like we only need to choose $C$ to be of the same order of magnitude as the learning rate. Is it possible to use these as a rule-of-thumb over different vision and NLP tasks? I'd be curious to know how sensitive these tasks are to the learning rate *without* privacy. Could the authors provide these numbers? Additionally, does the choice of architecture make a difference? I.e. what if I were to choose VGG-19 instead of ResNet in Fig 1, would the plot look the same? Why are the ablation studies in Fig 14 (not Fig 13 as stated in line 151) for different datasets than in Fig 1? I was expecting to see an identical plot to Fig 1 for a fixed choice of $\gamma$ showing the problems observed are no longer present.

[1] S. De, L. Berrada, J. Hayes, S. L. Smith, and B. Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650, 2022. I have included my questions in the above. I do not see any obvious negative societal consequences from this work.

 DPSGD requires gradient clipping to bound the sensitivity of the information output function, however, this operation results in a heavy workload on tuning the clipping bound, as it turns out the performance of the network is sensitive to this hyperparameter. This work proposes to replace gradient clipping $min(R/||g||, 1)g$ with gradient normalization $Rg/(||g|| + \gamma)$ based on two aspects. First, it is observed in practice that gradient normalization results in higher cosine similarity between the clipped gradient and the original gradient. Second, the additive term $\gamma$ helps to retain some information about magnitude as otherwise all gradients are pushed onto the boundary of a ball with radius $R$. Applying gradient normalization, $R$ can either couple with the learning rate or cancel out, therefore needing no further tuning. Strengths:
1. This work is well-written and easy to follow.
2. The proposed automatic tuning method is interesting, also simple, and easy to implement.
3. The experiments are intensive and convincing.

Weakness:

1. I see some layout problems, for example, Figure 6 has a broken link, and in line 151, I think the ablation study is presented in Figure 14 instead of 13. There are some others, this paper should be carefully polished before it is published.
2. The study of $\gamma$ should be more thorough, it is very important to know that tuning work is not transferred from $R$ to $\gamma$, please refer to Questions for detail. A disadvantage of the proposed automatic clipping is that it always distorts the distribution of gradients even if all gradients locates within the clipping bound. According to the experiments in this work, the clipping rate dramatically reduces after some iterations indicating that many gradients are not clipped with Abadi's clipping. Therefore automatic clipping theoretically should carefully tune the hyperparameters which determine the distortion, e.g. $\gamma$. Although the authors argue that the network is insensitive to the choice of $\gamma$, I note that the ablation study of $\gamma$ is conducted with one epoch training, however, this could not represent the whole training procedure, complete the whole training epochs would be more convincing. I'd be happy to increase the score if my question is answered, overall I still think this is a good work. There is no potential negative societal impact of this work, for limitations refer to Questions.

 Motivated by the observation that clipping happens very frequently on real-world dataset. The author(s) proposed a new clipping method (clipping all gradients regardless of its magnitude). The new clipping method is “almost” hyper-parameter free and can be implemented as efficient as DP-SGD with gradient clipping. On the theoretical side, the author(s) analyzed the convergence of DP-SGD with the proposed clipping method for nonconvex problems. On the empirical side, experiments on real-world datasets demonstrate the effectiveness of the proposed clipping method. - The contribution on the theoretical side is a little bit exaggerated. The main problem is that the author(s) use a much stronger assumption (Assumption 5.3) when analyzing the convergence. This assumption comes from [1], it was shown that this assumption can simplify the convergence when gradient clipping is adopted. However, this assumption can be false for simple linear regression problem. Therefore the claim in the abstract “the proposed clipping enjoys an convergence rate that matches the standard SGD” is kind of over-claimed since the convergence analysis of the proposed method and standard SGD rely on a different set of assumptions.
- The AUTO-S method fused $R$ into $\eta$ and therefore does not require to tune the clipping threshold $R$. However, this may end up requiring a more careful tunning of the learning rate $\eta$ since $R$ is implicitly involved in $\eta$ now.
- Overall, I appreciate the empirical contribution of this paper but I think the claimed theoretical contribution should be weakened. With the theoretical contribution being discounted, I am not sure if this paper has enough contribution to be accepted at NeurIPS.

Minor:
- What does the number of the heat map in Figure 1 stand for? - Can we relax Assumption 5.3?
- Does AUTO-S require a more careful tunning of $eta$? Does Abadi's clipping and AUTO-S use different learning rate in the experiments? Yes, limitation and societal impact are well-addressed.

Meta-review: This paper introduces an alternative to clipping for preprocessing gradients to limit sensitivity prior to adding noise for differentially private optimization, with the objective of improving the amount of tuning required by not requiring tuning of a clipping threshold. A key theoretical assumption in the paper is that of symmetric noise, which engendered a significant discussion among reviewers. It seems unclear how valid this assumption is, even in relatively simple linear regression settings. 

Decision: Reject