SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset	MNLI(m/mm)	QQP	QNLI	SST2
Epoch	18	18	6	3
Batch size	6000	6000	2000	1000
clipping threshold R	0.1	0.1	0.1	0.1
DP learning rate	5e-4	5e-4	5e-4	5e-4
non-DP learning rate	5e-5	5e-5	5e-5	5e-5
learning rate decay	Yes	Yes	Yes	Yes
AdamW weight decay	0	0	0	0
Max sequence length	256	256	256	256
Notice that we use DP learning rate 5e-4 across tasks for the  $R$ -dependent automatic DP-Adam, which is equivalent to  $R$ -independent automatic DP-Adam with the same learning rate. We demonstrate that the results are not sensitive to learning rates around the optimal choice. That is, the automatic clipping does not eliminate  $R$  at the cost of more difficult tuning of learning rate.
Table 5: Hyperparameters of automatic clipping and Abadi's clipping, for sentence classification in Table 2 and Table 3, using either RoBERTa base or large.
learning rate	1e-4	3e-4	5e-4	8e-4	1e-3
RoBERTa-base	93.92	94.38	94.49	94.72	93.35
RoBERTa-large	95.76	96.21	96.21	96.33	95.99
Table 6: SST2 accuracy with respect to learning rate.
G.3 Table-to-text generation settings
We experiment multiple GPT2 models on E2E dataset from Huggingface [39] in Table 4. We follow the same setup as [41], and our automatic clipping uses exactly the same hyperparameters as the Abadi's clipping in [41], which is released in their Private-Transformer library  $^{9}$ .
See https://github.com/lxuechen/private-transformers/blob/main/examples/classification/runwrapper.py
See https://github.com/lxuechen/private-transformers/blob/main/examples/table2text/run.sh
Model	GPT2	GPT2 medium	GPT2 large
Epoch	10	10	10
Batch size	1024	1024	1024
clipping threshold R	0.1	0.1	0.1
DP learning rate	2e-3	2e-3	2e-3
non-DP learning rate	2e-4	1e-4	1e-4
learning rate decay	No	No	No
AdamW weight decay	0.01	0.01	0.01
Max sequence length	100	100	100
Table 7: Hyperparameters of automatic clipping and Abadi's clipping, for the E2E generation task in Table 4.
H Figure zoo
H.1 Frequency of clipping
We show that in all sentence classification tasks, Abadi's clipping happens on a large proportion of per-sample gradients. This supports the similarity between Abadi's clipping and AUTO-V in (3.1).
Figure 13: Percentage of clipped per-sample gradients when training with DP-AdamAbadi  $(\epsilon = 3)$  as in Section 6.2. Left panel is RoBERTa-base and right panel is RoBERTa-large. Top row: MNLI. Middle row: QNLI. Bottom row: QQP.
We note that for GPT2, GPT2 medium and GPT2 large, empirically in all iterations  $100\%$  of the per-sample gradients are clipped by the Abadi's clipping, making the performance of Abadi's clipping equivalent to AUTO-V clipping, as shown in Table 4.
H.2 Stability constant helps AUTO clipping reduce gradient norm
To corroborate our claim in Theorem 6, that the stability  $\gamma$  reduces the gradient norm, we plot the actual gradient norm by iteration.
Figure 14: Gradient norm by different automatic clipping methods, on SST2 (left) and MNLI (right), trained with RoBERTa-base.
H.3 Choice of stability constant is robust
We claim in Theorem 6 that, as long as  $\gamma > 0$  in our automatic clipping, the asymptotic convergence rate of gradient norm is the same as that by standard non-private SGD. We plot the ablation study of learning rate and the stability constant  $\gamma$  to show that it is easy to set  $\gamma$ : in Table 2 and Table 3, we adopt learning rate 0.0005, under which a wide range of  $0.0001 < \gamma < 1$  gives similar accuracy. Note that the largest good  $\gamma$  is 1000 times bigger than the smallest good  $\gamma$ .
Figure 15: Test accuracy by different stability constant  $\gamma$  and learning rate  $\eta$  in automatic clipping  $(\epsilon = 3)$ . Upper row: SST2 for full 3 epochs. Middle row: QNLI for full 6 epochs. Lower row: QNLI for one epoch. Trained with RoBERTa-base (left) and RoBERTa-large (right).
I Full table of GPT2 generation task on E2E dataset
This is the extended version of Table 4 on E2E dataset. The performance measures are BLEU [57], ROGUE-L [42], NIST [63], METEOR [4], and CIDEr [70] scores. Here  $\epsilon$  is accounted by RDP [51], where  $\epsilon = 3$  corresponds to 2.68 if accounted by Gaussian DP [19, 7] or to 2.75 if accounted by numerical composition [30], and  $\epsilon = 8$  corresponds to 6.77 if accounted by Gaussian DP or to 7.27 if accounted by numerical composition.
Metric	DP guarantee	GPT2 large	GPT2 medium	GPT2
full AUTO-S	full AUTO-S	full AUTO-S	full AUTO-V	full [41]	LoRA [33]	RGP [78]	prefix [40]	top2 [41]	retrain [41]
BLEU	ε = 3	64.180	63.850	61.340	61.519	61.519	58.153	58.482	47.772	25.920	15.457
ε = 8	64.640	64.220	63.600	63.189	63.189	63.389	58.455	49.263	26.885	24.247
non-DP	66.840	68.500	69.463	69.463	69.463	69.682	68.328	68.845	65.752	65.731
ROGUE-L	ε = 3	67.857	67.071	65.872	65.670	65.670	65.773	65.560	58.964	44.536	35.240
ε = 8	68.968	67.533	67.073	66.429	66.429	67.525	65.030	60.730	46.421	39.951
non-DP	70.384	71.458	71.359	71.359	71.359	71.709	68.844	70.805	68.704	68.751
NIST	ε = 3	7.937	7.106	7.071	6.697	6.697	5.463	5.775	5.249	1.510	0.376
ε = 8	8.301	8.172	7.714	7.444	7.444	7.449	6.276	5.525	1.547	1.01
non-DP	8.730	8.628	8.780	8.780	8.780	8.822	8.722	8.722	8.418	8.286
METEOR	ε = 3	0.403	0.387	0.387	0.384	0.384	0.370	0.331	0.363	0.197	0.113
ε = 8	0.420	0.418	0.404	0.400	0.400	0.407	0.349	0.364	0.207	0.145
non-DP	0.460	0.449	0.461	0.461	0.461	0.463	0.456	0.445	0.443	0.429
CIDEr	ε = 3	2.008	1.754	1.801	1.761	1.761	1.581	1.300	1.507	0.452	0.116
ε = 8	2.163	2.081	1.938	1.919	1.919	1.948	1.496	1.569	0.499	0.281
non-DP	2.356	2.137	2.422	2.422	2.422	2.491	2.418	2.345	2.180	2.004
Table 8: Test performance on E2E dataset with GPT2. The best two GPT2 models for each row are marked in bold.
We observe that GPT2 (163 million parameters), GPT2-medium (406 million), and GPT2-large (838 million), Table 4 trained with our automatic clipping consistently perform better in comparison to other methods. In some cases, LoRA trained with Abadi's clipping also demonstrates strong performance and it would be interesting to see how LoRA trained with the automatic clipping will behave.
J Further experiments on CelebA dataset
In this section, we present a complete summary of accuracy results, with DP constraint or not, for the CelebA dataset. We do not apply any data-preprocessing. In the first experiment, we apply a single ResNet on the 40 labels as the multi-task/multi-label learning. In the second experiment, we apply one ResNet on one label. As expected, our automatic DP optimizers have comparable test accuracy to the Abadi's DP optimizers, but we do not need to tune the clipping threshold for each individual task/label. We also notice that, learning different labels separately gives better accuracy than learning all labels together, though at the cost of heavier computational burden.
J.1 Multi-label classification
We apply ResNet9 as in Appendix G.1 on the multi-label classification task. I.e. the output layer has 40 neurons, each corresponding to one sigmoid cross-entropy loss, that are summed to a single loss and all labels are learnt jointly.
Index	Attributes	Abadi's \( \epsilon = 3 \)	AUTO-S\( \epsilon = 3 \)	Abadi's\( \epsilon = 8 \)	AUTO-S\( \epsilon = 8 \)	non-DP\( \epsilon = \infty \)
0	5 o Clock Shadow	90.64	90.99↑	90.81	91.28↑	93.33
1	Arched Eyebrows	75.15	76.31↑	76.84	77.11↑	81.52
2	Attractive	75.85	76.10↑	77.50	77.74↑	81.15
3	Bags Under Eyes	80.75	81.12↑	82.15	82.13↓	84.81
4	Bald	97.84	97.87↑	98.04	97.98↓	98.58
5	Bangs	92.71	92.68↓	93.46	93.55↑	95.50
6	Big Lips	67.51	67.78↑	68.34	68.44↑	71.33
7	Big Nose	78.01	80.23↑	76.69	80.59↑	83.54
8	Black Hair	81.92	80.95↓	83.33	83.28↓	88.55
9	Blond Hair	92.25	92.38↑	93.52	93.09↓	95.49
10	Blurry	94.91	94.82↓	95.08	94.90↓	95.78
11	Brown Hair	80.13	82.50↑	83.74	83.89↑	87.79
12	Bushy Eyebrows	88.06	88.23↑	89.72	88.80↓	92.19
13	Chubby	94.72	94.54↓	94.54	94.50↓	95.56
14	Double Chin	95.19	95.49↑	95.50	95.51↑	96.09
15	Eyeglasses	97.06	97.64↑	98.32	98.06↓	99.39
16	Goatee	95.68	95.45↓	95.84	95.87↑	97.06
17	Gray Hair	96.77	96.79↑	97.02	97.03↑	98.06
18	Heavy Makeup	84.96	85.70↑	87.58	87.29↓	90.76
19	High Cheekbones	81.46	81.42↓	82.62	82.72↑	86.62
20	Male	92.05	92.17↑	93.32	93.17↓	97.46
21	Mouth Slightly Open	86.20	86.32↑	87.84	88.48↑	93.07
22	Mustache	96.05	95.96↓	96.08	95.99↓	96.74
23	Narrow Eyes	84.90	84.78↓	85.14	85.18↑	86.98
24	No Beard	91.55	91.67↑	92.29	92.45↑	95.18
25	Oval Face	71.26	71.42↑	71.98	71.25↓	74.62
26	Pale Skin	96.09	96.04↓	96.15	96.17↑	96.93
27	Pointy Nose	70.34	72.11↑	72.23	73.01↑	75.68
28	Receding Hairline	91.53	91.37↓	91.75	91.74↓	92.87
29	Rosy Cheeks	93.26	93.02↓	93.56	93.35↓	94.86
30	Sideburns	96.16	96.09↓	96.27	96.46↑	97.44
31	Smiling	86.39	87.08↑	88.87	88.63↓	92.25
32	Straight Hair	76.20	77.95↑	78.78	78.52↓	80.66
33	Wavy Hair	70.30	71.79↑	73.58	73.19↓	79.15
34	Wearing Earrings	80.53	81.52↑	82.29	82.20↓	87.56
35	Wearing Hat	96.99	96.83↓	97.46	97.31↓	98.68
36	Wearing Lipstick	88.95	88.04↓	89.87	90.72↑	93.49
37	Wearing Necklace	84.59	85.83↑	85.93	85.42↓	86.61
38	Wearing Necktie	93.91	93.91-	94.43	94.08↓	96.30
39	Young	81.35	81.21↓	82.18	82.52↑	87.18
Table 9: Accuracy on CelebA dataset with settings in Appendix G.1 from one run. The green arrow indicates AUTO-S is better than Abadi's clipping under the same  $\epsilon$ ; the red arrow indicates otherwise; the black bar indicates the same accuracy.
J.2 Multiple binary classification
For the second experiment, we apply ResNet9 on each label as a binary classification task. I.e. the output layer has 1 neuron and we run 40 different models for all labels separately.
Index	Attributes	Abadi's Single \( \epsilon = 8 \)	AUTO-S Single \( \epsilon = 8 \)	Abadi's Multi \( \epsilon = 8 \)	AUTO-S Multi \( \epsilon = 8 \)	non-DP Multi \( \epsilon = \infty \)
0	5 o Clock Shadow	92.15	92.29↑	90.81	91.28↑	93.33
1	Arched Eyebrows	81.18	80.19↓	76.84	77.11↑	81.52
2	Attractive	79.31	79.79↑	77.50	77.74↑	81.15
3	Bags Under Eyes	83.52	83.48↓	82.15	82.13↓	84.81
4	Bald	97.89	97.88↓	98.04	97.98↓	98.58
5	Bangs	94.52	94.83↑	93.46	93.55↑	95.50
6	Big Lips	67.32	67.53↑	68.34	68.44↑	71.33
7	Big Nose	82.31	82.36↑	76.69	80.59↑	83.54
8	Black Hair	87.08	86.93↓	83.33	83.28↓	88.55
9	Blond Hair	94.29	94.73↑	93.52	93.09↓	95.49
10	Blurry	94.95	95.20↑	95.08	94.90↓	95.78
11	Brown Hair	87.41	87.19↓	83.74	83.89↑	87.79
12	Bushy Eyebrows	91.23	91.43↑	89.72	88.80↓	92.19
13	Chubby	94.70	94.70-	94.54	94.50↓	95.56
14	Double Chin	95.43	95.43-	95.50	95.51↑	96.09
15	Eyeglasses	98.88	99.14↑	98.32	98.06↓	99.39
16	Goatee	96.12	96.07↓	95.84	95.87↑	97.06
17	Gray Hair	97.48	97.34↓	97.02	97.03↑	98.06
18	Heavy Makeup	88.85	88.72↓	87.58	87.29↓	90.76
19	High Cheekbones	85.66	85.45↓	82.62	82.72↑	86.62
20	Male	95.42	95.70↑	95.53	93.17↓	97.46
21	Mouth Slightly Open	92.67	92.74↑	87.84	88.48↑	93.07
22	Mustache	96.13	96.13-	96.08	95.99↓	96.74
23	Narrow Eyes	85.13	85.13-	85.14	85.18↑	86.98
24	No Beard	94.26	94.58↑	92.29	92.45↑	95.18
25	Oval Face	70.77	73.05↑	71.98	71.25↓	74.62
26	Pale Skin	96.38	96.34↓	96.15	96.17↑	96.93
27	Pointy Nose	71.48	73.37↑	72.23	73.01↑	75.68
28	Receding Hairline	91.51	91.51-	91.75	91.74↓	92.87
29	Rosy Cheeks	93.26	93.35↑	93.56	93.35↓	94.86
30	Sideburns	96.46	96.34↓	96.27	96.46↑	97.44
31	Smiling	90.82	90.87↑	88.87	88.63↓	92.25
32	Straight Hair	79.01	79.01-	78.78	78.52↓	80.66
33	Wavy Hair	77.55	78.83↑	73.58	73.19↓	79.15
34	Wearing Earrings	87.33	87.50↑	82.29	82.20↓	87.56
35	Wearing Hat	98.04	98.11↑	97.46	97.31↓	98.68
36	Wearing Lipstick	92.05	90.46↓	89.87	90.72↑	93.49
37	Wearing Necklace	86.21	86.21-	85.93	85.42↓	86.61
38	Wearing Necktie	95.85	95.94↑	94.43	94.08↓	96.30
39	Young	85.19	84.12↓	82.18	82.52↑	87.18
Table 10: Accuracy on CelebA dataset with settings in Appendix G.1 from one run. 'Single' means each attribute is learned separately as a binary classification task. 'Multi' means all attributes are learned jointly as a multi-label classification task. The green arrow indicates AUTO-S is better than Abadi's clipping under the same  $\epsilon$  and the same task; the red arrow indicates otherwise; the black bar indicates the same accuracy.
K Code implementation of automatic clipping
Changing Abadi's clipping to automatic clipping is easy in available codebases. One can set the clipping  $R = 1$  or any other constant, as explained in Theorem 1 and Theorem 2.
K.1 Opacus
For Opacus [77] version 1.1.2 (latest), we can implement the all-layer automatic clipping by changing Line 399-401 in https://github.com/pytorch/opacus/blob/main/opacus/optimizers/optimizer.py to
per_sample Clip_factor = self.max_grad_norm / (per_sample_norms + 0.01)
The per-layer automatic clipping requires changing Line 61-63 in https://github.com/pytorch/opacus/blob/main/opacus/optimizers/perlayeroptimizer.py to
per_sample Clip_factor =max_grad_norm / (per_sample(norms + 0.01)
For older version (< 1.0, e.g. 0.15) of Opacus, we can implement the all-layer automatic clipping by changing Line 223-225 in https://github.com/pytorch/opacus/blob/v0.15.0/opacus/utilities/clipping.py to
per_sample Clip_factor = self.float_value / (norms[0] + 0.01)
or implement the per-layer automatic clipping by changing Line 301-302 in https://github.com/pytorch/opacus/blob/main/opacus/optimizers/perlayeroptimizer.py to
per_sample Clip_factor = threshold / (norm + 0.01) clipping_factor.append(per_sample Clip_factor)
K.2 ObjAX
For ObJAX version 1.6.0 (latest), we can implement the automatic clipping in https://github.com/google/objax/blob/master/objax/privacy/dpsgd/gradient.py by changing Line 92 to
idivisor  $=$  self.l2_norm Clip / (total_grad_norm+0.01)
and changing Line 145 to
idivisor  $=$  self.l2_norm_clip/(grad_norms+0.01)
K.3 Private-transformers
To reproduce our experiments for sentence classification and table-to-text generation, we modify the 'private-transformers' (version 0.1.0) codebase of [41]. The modification is in https://github.com/lxuechen/private-transformers/blob/main/private_transformers/privacy_utils/privacy_engine.py, by changing Line 349 to
return self.max_grad_norm / (norm_sample + 0.01)
and Line 510-512 to
coef_sample = self.max_grad_norm * scale / (norm_sample + 0.01)
L More on related works of per-sample clipping
We discuss the difference between our work and the related (see the table below).
Ci	reference	clipping or not	convergence analysis	experiments
min(1, R/||gi||)	[1, 41]	clipping	None	CV and NLP
min(1/R, i/||gi||)	[17]	clipping	None	CV only
R/||gi||	[16]	normalization	convex and federated setting (not per-sample)	CV only
1/||gi||+γ	[74]	normalization	non-convex, relaxed Lipschitz smoothness	CV and NLP
1/||gi||+γ	this work	normalization	non-convex, same smoothness as non-DP	CV and NLP
Table 11: Comparison between clipping functions. CV means computer vision and NLP means natural language processing. Notice that any clipping function with  $R$  is not automatic and requires tuning, and that the stability constant  $\gamma$  enjoys theoretical and empirical benefits.
Our work is very different to most works which do not analyze the convergence of DP deep learning in a non-convex setting, but it is very similar to [74] $^{10}$ . However, [74] assumes a relaxed Lipschitz smootheness in place of our Assumption 5.3, where we instead assume the symmetric gradient noise. In addition, our experiments are more comprehensive, covering over 10 tasks including DP-GPT2, while [74] only experimented with 2 smaller models — ResNet20 and Roberta-base.
L.1 Clarifications
We now clarify some false or incomplete conclusion in previous literatures that apply the per-sample gradient clipping (re-parameterized or not).
1. Per-sample clipping is not robust to  $R$ , even with re-parameterization.
In [17, Figure 8] and our Figure 4, the accuracy of DP optimizer with Abadi's clipping is insensitive to  $R$  only if one has found a small enough region (e.g.  $R \leq 1$ ), which takes effort to find or the accuracy will be unacceptably low out of the region. In particular, choosing  $R = 1$  as in [17] is not universally proper, e.g. [41] uses  $R = 0.1$  for language models. This dependence on tasks, datasets and optimizers means per-sample clipping still requires the expensive hyperparameter tuning.
In other words, per-sample gradient clipping is at best an approximation of per-sample gradient normalization (i.e. our AUTO-V) and should be considered as semi-automatic, whereas AUTO-V/S is fully automatic in terms of tuning  $R$ . Although technically we introduce a new hyperparameter  $\gamma$  in the place of  $R$ , we claim that automatic clipping is not sensitive to  $\gamma$  (our only hyperparameter) for a large range, e.g. one can multiply  $\gamma$  by 10000 times, going from  $\gamma = 0.001$  to 10 with learning rate 0.0005 in Figure 15, and the accuracy is similar.
2. Per-sample clipping does not decouple  $R$ , especially for DP-Adam.
In general,  $R$  is not completely decoupled from the re-parameterized per-sample clipping in [17]:
$$
C _ {\text {A b a d i}} = \min  \left(\frac {R}{\| \boldsymbol {g} _ {i} \|}, 1\right) = R \cdot C _ {\text {r e - p a r a m}} = R \cdot \min  \left(\frac {1}{\| \boldsymbol {g} _ {i} \|}, \frac {1}{R}\right)
$$
Given that  $R$  appears in both terms on the right hand side, one can at most say "... when the clipping norm is decreased k times, the learning rate should be increased k times to maintain similar accuracy." by [38] and "... Using this update, performance becomes less sensitive to the choice of clipping norm." by [17]. In contrast, we can state that adjusting the learning rate proportionally, our AUTO-V/S maintains exactly the same accuracy and is completely insensitive to the choice of  $R$ .
Additionally and importantly, the understanding in [38, 17] is limited to DP-SGD (as they only experiment with the computer vision tasks), where "... the learning rate  $\eta$  absorbs a factor of  $R$ ." by [17]. As rigorously proved in Theorem 1 and Theorem 2, adaptive optimizers like Adam and AdaGrad do not absorb  $R$  but rather cancel it. This is visualized in Figure 1, where the performance landscape is row-wise for DP-Adam and diagonal for DP-SGD.
3. Re-parameterized per-sample clipping unintentionally changes the weight decay.
<sup>10</sup>We emphasize that [74] is a concurrent work with no known dependency either way, which goes public (to arXiv, on 27 Jun 2022) after ours (on 14 Jun 2022).
Weight decay is a common technique used in any work that uses AdamW and in the re-parameterized trick by [17]. We can see that
Before re-parameterization:  $w_{t + 1} = w_t - \eta \left( \frac{1}{B} \sum_i \min(1, \frac{R}{||g_i||}) g_i + \lambda w_t + \frac{\sigma R}{B} N(0, I) \right)$
After re-parameterization:  $w_{t + 1} = w_t - \eta \left( \frac{1}{B} \sum_i \min \left( \frac{1}{R}, \frac{1}{||g_i||} \right) g_i + \frac{\lambda}{R} w_t + \frac{\sigma}{B} N(0, I) \right)$
Therefore, when we move along  $R$  like in [17, Figure 8], from  $R = 1$  to  $2^{-6}$ , the weight decay increases from  $\lambda$  to  $2^{6} \cdot \lambda$  by 64 times, which may worsen the accuracy as seen in the blue curve of [17, Figure 8]! Again, this is due to the incomplete decoupling by per-sample clipping, which is only avoided in AUTO-V/S thanks to theoretical analysis in Theorem 1 and Theorem 2.
AUTO-V/S with weight decay:  $w_{t + 1} = w_t - \eta \left(\frac{1}{B}\sum_i\frac{1}{\|g_i\| + \gamma} g_i + \lambda w_t + \frac{\sigma}{B}\mathcal{N}(0,I)\right)$ .
L.2 Connections to normalized optimisation
Variants of normalized gradient have been used in optimization [47, 53, 81, 80, 15]. These normalized optimizers are fundamentally different to our automatic optimizers, because the normalization is on mini-batch not on each sample and noise is not involved:
$$
\text {N S G D :} w _ {t + 1} = w _ {t} - \eta \left(\frac {\frac {1}{B} \sum_ {i} g _ {i}}{\| \frac {1}{B} \sum_ {i} g _ {i} \|}\right)
$$
$$
\text {A U T O - V :} w _ {t + 1} = w _ {t} - \eta \left(\frac {1}{B} \sum_ {i} \frac {g _ {i}}{\| g _ {i} \|} + \frac {\sigma}{B} \mathcal {N} (0, I)\right).
$$
The main difference lies in the challenge of analyzing per-sample normalization (which is biased) and the batch-gradient normalization (which is unbiased in the direction). That is,  $\frac{\frac{1}{B}\sum_{i}g_{i}}{\|\frac{1}{B}\sum_{i}g_{i}\|}$  is parallel to the mini-batch gradient  $\frac{1}{B}\sum_{i}g_{i}$  but  $\frac{1}{B}\sum_{i}\frac{g_i}{\|g_i\|}$  is generally not parallel to it (this conclusion also holds if the normalizaiton is replaced by the clipping). On a side note, it is interesting that Theorem 4 indeed shows although a bias is introduced by the per-sample clipping, it is not fatal to the asymptotic convergence and hence may not be a concerning matter.