SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This coupling of  $\eta$  and  $R$  is also partially observed in [17, Appendix B.1] through a re-parameterization trick of Abadi's clipping. Unlike AUTO-S/V, the coupling is not strict (e.g. doubling  $R$  is not equivalent to doubling  $\eta$ , thus still necessitating tuning both  $(\eta, R)$ ), and the relationship to weight decay was not discussed.
4.3 Automatic clipping is equally private and maximizes utility
In Theorem 3 (proved in Appendix A), we show that the new private gradient  $\hat{\pmb{g}}_t$  in (4.2) has the same level of privacy guarantee as the existing one in (1.1), since the global sensitivity remains the same (see Figure 5). We note that as long as  $\gamma >0$ , the magnitude information of per-sample gradients is preserved by AUTO-S, in the sense that  $\| \pmb {g}_i\| >\| \pmb {g}_j\| \Longleftrightarrow \| C_i\pmb {g}_i\| >\| C_j\pmb {g}_j\|$ , whereas this can be violated in both the AUTO-V and Abadi's clipping (as depicted by the flat curve in Figure 5 when  $\| \pmb {g}_i\| >1$ ).
Figure 5: Per-sample gradient norms before and after different clippings at  $R = 1$ .
Additionally, note that when  $\gamma$  is small, almost all data points "max out" the signal relative to the amount of noise we add. To say it differently, for the same amount of noise, AUTO-S with small  $\gamma$  allows more signal to be pushed through a differentially private channel. Towards the end of the training, i.e., at the limit when  $\| g_i\| \to 0$  for all  $i$ , then we have  $\sum_{i}\frac{g_i}{\|g_i\| + \gamma} \to \frac{1}{\gamma}\sum_{i}g_{i}$ . In words, the clipped gradients become closer to the standard SGD, thus do not suffer from the instability of AUTO-V.
Theorem 3. Under the noise multiplier  $\sigma$ , number of iterations  $T$ , subsampling probability  $B / n$ , DP optimizers using AUTO-V or AUTO-S clipping satisfy  $(\epsilon_{Accountant}(\delta, \sigma, B / n, T), \delta)$ -DP, where  $\epsilon_{Accountant}$  is any valid privacy accountant for DP-SGD under Abadi's clipping.
5 Convergence analysis of DP-SGD with automatic clipping
5.1 Convergence theory of DP-SGD to stationary points
We highlight that automatic clipping can be more amenable to analysis than Abadi's clipping in [14], since we no longer need to decide whether each per-sample gradient is clipped.
To analyze the convergence of automatic DP-SGD (4.2) in the non-convex setting, we follow the standard assumptions in the SGD literature [27, 2, 6], including a symmetry assumption on the gradient noise, which is empirically verified in [14, Figure 3] and commonly used in the standard non-DP literature [48, 66, 12, 73]. We refer the curious readers to Appendix E.5 for details.
Assumption 5.1 (Lower bound of loss). For all  $\pmb{w}$  and some constant  $\mathcal{L}_{*}$ , we have  $\mathcal{L}(\pmb{w}) \geq \mathcal{L}_{*}$ .
Assumption 5.2 (Smoothness). Let  $g(w)$  denote the gradient of the objective  $\mathcal{L}(w)$ . Then  $\forall w, v$ , there is an non-negative constant  $L$  such that
$$
\mathcal {L} (\boldsymbol {v}) - \left[ \mathcal {L} (\boldsymbol {w}) + \boldsymbol {g} (\boldsymbol {w}) ^ {\top} (\boldsymbol {v} - \boldsymbol {w}) \right] \leq \frac {L}{2} \| \boldsymbol {w} - \boldsymbol {v} \| ^ {2}. \tag {5.1}
$$
Assumption 5.3 (Gradient noise). The per-sample gradient noise  $\tilde{g}_{t,i} - g_t$  is i.i.d. from some distribution such that
$$
\mathbb {E} \left(\tilde {\boldsymbol {g}} _ {t, i} - \boldsymbol {g} _ {t}\right) = 0, \mathbb {E} \| \tilde {\boldsymbol {g}} _ {t, i} - \boldsymbol {g} _ {t} \| ^ {2} \leq \xi^ {2},
$$
and  $\tilde{\pmb{g}}_{t,i}$  is centrally symmetric about  $\pmb{g}_t$  in distribution:  $\tilde{\pmb{g}}_{t,i} - \pmb{g}_t\stackrel {\mathcal{D}}{=}\pmb {g}_t - \tilde{\pmb{g}}_{t,i}$
We show in Theorem 4 that DP-SGD with AUTO-S clipping allows the true gradient norm to converge to zero, though the clipped gradient may still be biased, but not so with AUTO-V clipping.
Theorem 4. Under Assumption 5.1, 5.2, 5.3, running DP-SGD with automatic clipping for  $T$  iterations and setting the learning rate  $\eta \propto 1 / \sqrt{T}$  give<sup>6</sup>
$$
\min  _ {0 \leq t \leq T} \mathbb {E} (\| \boldsymbol {g} _ {t} \|) \leq \mathcal {G} \left(\frac {4}{\sqrt {T}} \sqrt {\left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L \left(1 + \frac {\sigma^ {2} d}{B ^ {2}}\right)}; \xi , \gamma\right) := \min  _ {r > 0} \frac {\xi}{r} + \mathcal {F} (\dots ; r, \xi , \gamma). \tag {5.2}
$$
The upper bound takes an implicit form of  $\mathcal{G}(\cdot ;\xi ,\gamma)$  because it is a lower envelope of functions  $\frac{\xi}{r} +$ $\mathcal{F}(\cdot ;r,\xi ,\gamma)$  over all possible  $r > 0$  , whose forms are detailed in Theorem 6. Notice that  $\mathcal{G}$  results only from the clipping operation, not from the noise addition.
Here  $\cdots$  represents the first argument of  $\mathcal{G}$ , and  $\mathcal{G}$  is increasing and positive. As  $T \to \infty$ , we have  $\min_t \mathbb{E}(\| \pmb{g}_t \|) = O(T^{-1/4})$  for AUTO-S, the same rate as the standard SGD given in Theorem 9.
Figure 6: Left: DP-SGD with AUTO-V clipping. Middle: DP-SGD with AUTO-S clipping. Right: Log-log plot of convergence rate in comparison to standard SGD. Here  $\xi = 25, \gamma = 0.01$ , and the  $O(1 / \sqrt{T})$  term is set to 10 for DP-SGD and to 2 for standard SGD.
Remark 5.4. We show in Theorem 6 and in Figure 6 that the upper bound (5.2) has  $\mathcal{G} \geq \xi$  for AUTO-V ( $\gamma = 0$ ), and  $\mathcal{G}$  only reduces to zero for AUTO-S ( $\gamma > 0$ ). We provide real data evidence in Figure 14 that strictly positive  $\gamma$  reduces the gradient norm significantly.
5.2 Analysis of factors affecting the convergence
We now analyze the many factors that affect the convergence in Theorem 4, from a unified viewpoint of both the convergence and the privacy.
We start with the stability constant  $\gamma$  and the learning rate  $\eta_t$ , both only affect the convergence not the privacy. We empirically observe in Figure 8 that small  $\gamma$  benefits the convergence at initial iterations (when the privacy guarantee is strong) but larger  $\gamma$  converges faster asymptotically. For  $\eta_t$ , the optimal is in fact the minimizer of the hyperbola in (C.5), that is unique and tunable.
Next, we focus on the hyperparameters that affect both convergence and privacy: the batch size  $B$ , the noise multiplier  $\sigma$ , and the number of iterations  $T$ . These hyperparameters have to be considered along the privacy-accuracy tradeoff, not just from a convergence perspective.
Recall that given a fixed privacy budget  $(\epsilon, \delta)$ , we rely on modern privacy accountant for computing the appropriate combinations of parameter  $\sigma, T, B$ . The exact expression of the bound as a function of  $(\epsilon, \delta)$  is somewhat messy. For this reason, we illustrate our analysis in terms of the surrogate parameter  $\mu$  for  $\mu$ -GDP [19], which implies  $(\epsilon, \delta)$ -DP with  $\epsilon = \mu^2 + \mu \sqrt{2 \log(1 / \delta)}$ . [7] showed that DP-SGD's privacy guarantee asymptotically converges to  $\mu$ -GDP (as  $T \to \infty$ ) with  $\mu = \frac{B}{n} \sqrt{T(e^{1 / \sigma^2} - 1)}$ . We can alternatively leverage  $\rho$ -tCDP [10] for similar conclusions, using  $\rho$  in place of  $\mu^2$  in (5.3).
Theorem 5. Under Assumption 5.1, 5.2, 5.3, fixing the asymptotic  $\mu (\epsilon ,\delta)$ -GDP parameter, running DP-SGD with automatic clipping for  $T$  iterations and setting the learning rate  $\eta \propto 1 / \sqrt{T}$  give
$$
\min  _ {0 \leq t \leq T} \mathbb {E} (\| \boldsymbol {g} _ {t} \|) \leq \mathcal {G} \left(4 \sqrt {\left(\mathcal {L} _ {0} - \mathcal {L} _ {*}\right) L \left(\frac {1}{T} + \frac {d}{\mu^ {2} n ^ {2}} + O \left(\frac {1}{B ^ {2} T}\right)\right)}; \xi , \gamma\right) \tag {5.3}
$$
To show that our analysis matches the training behaviors observed in SOTA empirical work [41, 38, 17, 68, 50, 78], we minimize the first argument of  $\mathcal{G}$  in (5.3), denoted as  $X(B,T,\mu ,d,L,\mathcal{L}_0)$ .
1. [Train longer with larger noise] Fixing the expected batch size  $B$ , we see that  $X$  is decreasing in  $T$ . Hence larger  $T$  and consequently larger  $\sigma$  are preferred.
2. [Larger batch size helps] Fixing number of iterations  $T$  or epochs  $E = BT / n$ , we see that  $X$  is decreasing in  $B$ . Hence larger  $B$  and consequently larger  $\sigma$  are preferred.
3. [Pretraining is critical] Pretraining can boost the DP accuracy through a much smaller initial loss  $\mathcal{L}_0$  and from a smooth (small  $L$ ) and flat (small  $\xi$ , c.f. Figure 8(left)) initialization.
4. [Learning rate needs tuning] The optimal learning rate by minimizing (C.5) is  $\sqrt{\frac{(\mathcal{L}_0 - \mathcal{L}_*)\mu^2n^2}{L(\mu^2n^2 + dT)}}$ . This indicates that one should use larger learning rate for smaller model  $d$ , weaker privacy (larger  $\mu$  or small  $\epsilon$ ), or smaller iteration budget  $T$ .
We evaluate our automatic DP training on image classification, sentence classification, and table-to-text generation tasks. Detailed settings including hyperparameters can be found in Appendix G.
6.1 Image classification
For MNIST/FashionMNIST, we use the same setup as in [56, 68, 64] with a simple CNN. For CIFAR10, we use the same setup as in [68] with pretrained SimCLRv2 [13]. For ImageNette, a 10-class sub-task of ImageNet [18], we use the same setup as in [36] without the learning rate decay. For CelebA [45], the real human face dataset, we train ResNet9 [32] with group normalization to replace the batch normalization. Notice that CelebA contains high-resolution (178x218) images, each with 40 labels. We consider CelebA for either multi-class classification on one label, e.g. 'Smiling' and 'Male', or for multi-label/multi-task problem to learn all labels simultaneously.
Table 1: Average test accuracy and  ${95}\%$  confidence interval on image tasks over 5 runs.
Task	Model	(ε,δ)	Accuracy %
Abadi's clipping	AUTO-S clipping	non-DP (ε = ∞)
MNIST	4-layer CNN	(3,1e-5)	98.04 ± 0.09	98.15 ± 0.07	99.11 ± 0.07
FashionMNIST	4-layer CNN	(3,1e-5)	86.04 ± 0.26	86.36 ± 0.18	89.57 ± 0.13
CIFAR10 pretrained	SimCLRv2	(2,1e-5)	92.44 ± 0.13	92.70 ± 0.02	94.42 ± 0.01
ImageNette	ResNet9	(8,1e-4)	60.29 ± 0.53	60.71 ± 0.48	71.11 ± 0.37
CelebA [Smiling]	ResNet9	(8,5e-6)	90.75 ± 0.11	91.08 ± 0.08	92.61 ± 0.20
CelebA [Male]	ResNet9	(8,5e-6)	95.54 ± 0.14	95.70 ± 0.07	97.90 ± 0.04
CelebA Multi-label	ResNet9	(3,5e-6)	86.81 ± 0.03	87.05 ± 0.01	90.30 ± 0.02
CelebA Multi-label	ResNet9	(8,5e-6)	87.52 ± 0.15	87.58 ± 0.04	90.30 ± 0.02
In Table 1, we observe that AUTO-S clipping outperforms existing clipping in all datasets with statistical significance. Interestingly, the standard deviation from different runs is smaller for automatic DP optimizers, indicating better reproducibility and stability. We additionally experiment 40 binary classification problems on CelebA with respect to each label, and observe that the mean accuracy further improves to  $91.63\%$  at  $\epsilon = 8$  for AUTO-S (see Appendix J).
6.2 Sentence classification
On five benchmark language datasets (MNLI(m/mm)[72], QQP[34], QNLI[62], SST2[67]), we compare our automatic DP training with re-parameterized gradient perturbation (RGP, [78]) and full-parameter finetuning (full, [41]) using RoBERTa models [44]. These methods use the same experimental setup. For language models, our automatic training is based on the codebase of [41].
Table 2: Test accuracy on language tasks with RoBERTa-base (12 blocks, 125 million parameters).
Method	ε = 3	ε = 8	ε = ∞ (non-DP)
MNLI	QQP	QNLI	SST2	MNLI	QQP	QNLI	SST2	MNLI	QQP	QNLI	SST2
RGP [78]	-	-	-	-	80.5/79.6	85.5	87.2	91.6	83.6/83.2	89.3	91.3	92.9
full [41]	82.45/82.99	85.56	87.42	91.86	83.20/83.46	86.08	87.94	92.09
full AUTO-V	81.21/82.03	84.72	86.56	91.86	82.18/82.64	86.23	87.24	92.09	85.91/86.14	87.34	91.40	94.49
full AUTO-S	83.22/83.21	85.76	86.91	92.32	83.82/83.55	86.58	87.85	92.43
Table 3: Test accuracy on language tasks with RoBERTa-large (24 blocks, 355 million parameters).
Method	ε = 3	ε = 8	ε = ∞ (non-DP)
MNLI	QQP	QNLI	SST2	MNLI	QQP	QNLI	SST2	MNLI	QQP	QNLI	SST2
RGP [78]	-	-	-	-	86.1/86.0	86.7	90.0	93.0	-	-	-	-
full [41]	86.43/86.46	86.43	90.76	93.04	87.02/87.26	87.47	91.10	93.81
full AUTO-V	85.33/85.61	86.61	89.99	93.12	85.91/86.10	86.86	90.55	93.35	90.33/90.03	87.90	93.61	96.21
full AUTO-S	86.27/86.67	86.76	91.01	93.92	87.07/87.16	87.47	91.45	94.61
In Table 2 and Table 3, we note that full parameter finetuning with AUTO-S outperforms or at least matches SOTA on all tasks. We use exactly the same hyperparameters as in [41].
6.3 Table-to-text generation
We compare our automatic DP training with a variety of fine-tuning methods, for table-to-text generation task on E2E dataset [23], where the goal is to generate texts about different aspects of a restaurant's data. We measure the success on this task by BLEU, ROUGE-L (in Table 4), METEOR, NIST, CIDEr (extended in Table 8), with higher value meaning better model quality.
Table 4: Test performance on E2E dataset with GPT2. Additional performance measures are included in Table 8. The best two GPT2 models for each row are marked in bold.
Metric	DP guarantee	GPT2 large	GPT2 medium	GPT2
full AUTO-S	full AUTO-S	full AUTO-S	full AUTO-V	full [41]	LoRA [33]	RGP [78]	prefix [40]	top2	retrain
BLEU	ε = 3	64.180	63.850	61.340	61.519	61.519	58.153	58.482	47.772	25.920	15.457
ε = 8	64.640	64.220	63.600	63.189	63.189	63.389	58.455	49.263	26.885	24.247
non-DP	66.840	68.500	69.463	69.463	69.463	69.682	68.328	68.845	65.752	65.731
ROGUE-L	ε = 3	67.857	67.071	65.872	65.670	65.670	65.773	65.560	58.964	44.536	35.240
ε = 8	68.968	67.533	67.073	66.429	66.429	67.525	65.030	60.730	46.421	39.951
non-DP	70.384	71.458	71.359	71.359	71.359	71.709	68.844	70.805	68.704	68.751
Competitive methods include low-rank adaption (LoRA), prefix-tuning (prefix), RGP, only fine-tuning the top 2 Transformer blocks (top2), and training from scratch (retrain), as were recorded in [41]. Again, we use the exactly the same hyperparameters as in [41]. For GPT2 (124 million parameters), GPT2 medium (355 million), and GPT2 large (774 million), Table 4 shows that AUTO-S is scalable with stronger performance on larger models. Our automatic full-parameter finetuning has the best overall performance. Additionally, we highlight that AUTO-S and methods like LoRA are not mutually exclusive and can be combined to yield strong performance, since AUTO-S modifies the optimizers and LoRA modifies the architecture.
7 Related works
While other DP works also normalize the per-sample gradients (instead of clipping them) or use small clipping threshold (making the clipping similar to normalization), our work is very different in terms of theoretical analysis, algorithm design and experiments. In fact, the concurrent work [74] gives the same algorithm as AUTO-S, although its theoretical analysis and experiment design is fundamentally different from ours. [16] proposes to normalize the per-user (not per-sample) gradient in the federated learning setting, and analyzes the convergence in a convex, non-deep-learning setting.
On the other hand, many works apply the per-sample gradient clipping with small  $R$  for good utility [1, 41, 50, 38, 17]. These works have led to valuable insights, but also some false or incomplete conclusions, due to the lack of rigorous theoretical analysis. For instance, since  $R$  is present in the (re-parameterized) per-sample clipping, it cannot avoid the hyperparameter tuning as the choice of  $R$  is not robust; even if a sufficiently small  $R$  is used, the clipping does not reveal the stability constant in AUTO-S, which enjoys theoretical and empirical advantages in Remark 5.4 and Section 6. We devote Appendix L to more instances (e.g. Footnote 5) and a thorough comparison.
In this work, we propose the automatic clipping as a drop-in replacement to the standard per-example clipping for differentially private training. This is the first technique that eliminates the need to tune the clipping threshold  $R$ , thus making DP deep learning as easy as regular learning. Our AUTO-S method enjoys both theoretical guarantee of convergence in non-convex problems (under various conditions), and strong empirical performance that advances DP learning on computer vision and language tasks.
We are excited about the future of automatic DP training, especially along with other working techniques, such as general optimizers (e.g. [8, 21]), clipping styles (all-layer or per-layer or adaptive clipping), architecture modifications (e.g. LoRA, RGP, prefix), and data augmentation (e.g. adversarial training [29] and multiple augmentation [17]). Thus, we expect to achieve comparable results to all SOTA in a lightweight fashion.
We would like to thank Xuechen Li for updating his codebase and for his quick response in technical details to reproduce the results, which was crucial for benchmarking our experiments.
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] Zeyuan Allen-Zhu. Natasha 2: Faster non-convex optimization than sgd. Advances in neural information processing systems, 31, 2018.
[3] Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive clipping. Advances in Neural Information Processing Systems, 34, 2021.
[4] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.
[5] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. *signsgd: Compressed optimisation for non-convex problems*. In International Conference on Machine Learning, pages 560–569. PMLR, 2018.
[6] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311, 2018.
[7] Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard data science review, 2020(23), 2020.
[8] Zhiqi Bu, Sivakanth Gopi, Janardhan Kulkarni, Yin Tat Lee, Hanwen Shen, and Uthaipon Tantipongpipat. Fast and memory efficient differentially private-sgd via jl projections. Advances in Neural Information Processing Systems, 34, 2021.
[9] Zhiqi Bu, Hua Wang, and Qi Long. On the convergence and calibration of deep learning with differential privacy. arXiv preprint arXiv:2106.07830, 2021.
[10] Mark Bun, Cynthia Dwork, Guy N Rothblum, and Thomas Steinke. Composable and versatile privacy via truncated cdp. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pages 74-86, 2018.
[11] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021.
[12] Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pages 1-10. IEEE, 2018.
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[14] Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspective. Advances in Neural Information Processing Systems, 33:13773-13782, 2020.
[15] Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning, pages 2260-2268. PMLR, 2020.
[69] Ashish Vavwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[70] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[71] Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. In International Conference on Artificial Intelligence and Statistics, pages 1226–1235. PMLR, 2019.
[72] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics, 2018.
[73] Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations, 2020.
[74] Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Normalized/clipped sgd with perturbation for differentially private non-convex optimization. arXiv preprint arXiv:2206.13033, 2022.
[75] Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888, 6(12):6, 2017.
[76] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020.
[77] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv:2109.12298, 2021.
[78] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large scale private learning via low-rank reparametrization. In International Conference on Machine Learning, pages 12208-12218. PMLR, 2021.
[79] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
[80] Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. Stochastic normalized gradient descent with momentum for large batch training. arXiv preprint arXiv:2007.13985, 2020.
[81] Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences, 64:1-13, 2021.
[82] Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics, pages 4782-4817. PMLR, 2022.
A Proof of differential privacy
Proof of Theorem 3. Define the  $\ell_2$  sensitivity of any function  $g$  to be  $\Delta g = \sup_{S,S'}\| g(S) - g(S')\| _2$  where the suprem is over all neighboring  $(S,S^{\prime})$ . Then the Gaussian mechanism  $\hat{g} (S) = g(S) + \sigma \Delta g\cdot \mathcal{N}(0,\mathbf{I})$ .
$\sigma$  denotes the "Noise multiplier", which corresponds to the noise-level when a Gaussian mechanism is applied to a query with sensitivity 1.
Observe that automatic clipping (AUTO-V and AUTO-S (4.1)) ensures the bounded global-sensitivity of the stochastic gradient as in Abadi's clipping. Aligning the noise-multiplier (rather than the noise-level itself) ensures that the noise-to-sensitivity ratio  $\frac{\sigma\Delta g}{\Delta g} = \sigma$  is fixed regardless of  $\Delta g$ . The Gaussian mechanism's privacy guarantees are equivalent. Thus from the privacy accountant perspective, DP-SGD with both Abadi's clipping and our autoclipping method can be equivalently represented as the adaptive composition of  $T$  Poisson sampled Gaussian Mechanism with sampling probability  $B / n$  and noise multiplier  $\sigma$ .
B Proof of automaticity