Pre-training Differentially Private Models with Limited Public Data
Authors: Zhiqi Bu
Xinwei Zhang
Sheng Zha
Mingyi Hong
George Karypis
Venue: NeurIPS 2024 poster (2024)
Keywords: Differential Privacy
Foundation Models
Pre-training
Primary area: privacy
TL;DR: We analyze the convergence of differentially private training through the Hessian information and propose a continual pretraining recipe.

Abstract: The superior performance of large foundation models can be attributed to the use of massive amounts of high-quality data. However, such datasets often contain sensitive, private and copyrighted material that requires formal protection. While differential privacy (DP) is a prominent method used to gauge the degree of security provided to large foundation models, its application in large foundation models has been met with limited success because there are often significant performance compromises when applying DP during the pre-training phase. Consequently, DP is more commonly implemented during the model fine-tuning stage, hence not capable of protecting a substantial portion of the data used during the initial pre-training process. In this work, we first provide a theoretical understanding of the efficacy of DP training by analyzing the per-iteration improvement of loss through the lens of the Hessian. We observe that DP optimizers' deceleration can be significantly mitigated by the use of limited public data, and thus propose the DP continual pre-training strategy. Our DP continual pre-training on vision models, using only 10% of public data, have achieved DP accuracy of 41.5% on ImageNet-21k (with epsilon=8) and non-DP accuracy of 55.7% on Places365 and 60.0% on iNaturalist-2021, which are on par with state-of-the-art standard pre-training and outperform existing DP pertained models. Our DP pre-trained models are released in *fastDP* library (https://github.com/awslabs/fast-differential-privacy/releases/tag/v2.1)

Pre-training Differentially Private Models with Limited Public Data
Zhiqi Bu"
Xinwei Zhang University of Southern California
Sheng Zha
Mingyi Hong University of Minnesota
Amazon
Amazon
George Karypis
Amazon
Abstract
The superior performance of large foundation models relies on the use of mas- sive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection. While differential pri- vacy (DP) is a prominent method to gauge the degree of security provided the models, its application is commonly limited to the model fine-tuning stage, due to the performance degradation when DP is applied during the pre- training stage. Consequently, DP is yet not capable of protecting a substantial portion of the data used during the initial pre-training process. In this work, we provide a theoretical understanding of the efficacy DP training by ana- lyzing the per-iteration loss improvement, through the lens of Hessian matrix for large neural networks. We make a key observation that DP optimizers' performance degradation
be significantly mitigated by the use of limited public data, which leads to a novel DP continual pre-training strategy. Em- pirically, using only 10% of public data and 90% of private data, egy can achieve DP accuracy of 41.5% on ImageNet-21K (with = 8), as well as non-DP accuracy of 55.7% and 60.0% downstream tasks Places365 and iNaturalist-2021, respectively, on par with state-of-the-art standard pre- training and substantially outperforming existing DP pre-trained models. Our DP pre-trained models are released in fastDP library /https://gitthu: com/ hullbb///ssrrddffeeeerrrrrrrrrrp
strat-
Introduction
Large pre-trained models have been the backbone of computer vision and natural language process- ing. Finetuning or zero/few-shot learning (including in-context learning) based on these models achieve superior performance. In particular, differentially private (DP) fine-tuning, such full-
parameter training, LoRA, Adapter, BiTFiT, and linear probing [89 49 14 12
58 24), has shown
to be almost accurate the standard non-DP fine-tuning GPT
68
6
VIT [27). and ResNet
39] models, while protecting the privacy of fine-tuning data. To be more specific, DP language and vision models are highly effective in defending against canary insertion attacks [66 40 42]
and membership inference attacks (20 69 32 [6]; DP fine-tuned GPT2 also reduces the personally identifiable information leakage by 5 10 times compared with its non-DP counterpart (53)
These DP munet models all follow two-stage training procedure, which the first stage trains model large public datasets (e.g. ImageNet) from scratch without privacy protection, and the second stage fine-tunes on relatively small private datasets (e.g. CIFAR10). However, a growing concern has been raised against the pre-training on the vast amount of web-collected data [8] 17 79
41 62
The pre-trained models could memorize and re- re-generate the sensitive information in the
'qqual contribution. Email: zhigibu@amazon. com. !Done at Amazon. 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

training data, which include copyright content in books and codes, images of faces and nudity, and other personally identifiable information such as address and phone numbers, even the data have been pre-processed by some content filters. While it is common to use close-source or proprietary datasets (e.g- JFT 75 24 58) instead and not release the models trained on these datasets, this approach renders the models not reproducible and may still violate the data privacy implicitly.
Consequently, because of the uncertainty in seemingly safe-to-use public datasets, it is important to apply DP to the pre-training phase, which is computationally feasible, since DP optimization can be as efficient as the standard non-DP training [3 38 However, DP optimization without any public data suffers from slow convergence, and suboptimal performance: even on CIFAR 10, the non-DP accuracy drops from > 95% < 70% at e 8 663; on GPT2, the non-DP BLEU score degrades from 65.73 (quality often better than human) to 15.457 (hard to get the gist) at e = 3 [49]. In short, pre-training and fine-tuning differ significantly the amount of data and computation resources used, as well as in optimization setups. We summarize the difference in Table [
Table I: Comparing DP pre-training and DP fine-tuning.
pre-training
fine-tuning
Dataset size
large
small
Training iterations
large
small
(amount of compute)
Trainable model parameters
100%
0.1
100%
Major cause of
DP noising
DP clipping
performance degradation
1.1 Contributions
Our main contributions are listed below. We emphasize that we have taken the privacy accounting and the convergence (i.e. the dependence of loss on a (B,T, and B) jointly into consideration.
(1) We provide a new perspective of analyzing the loss improvement in DP training with different optimizers. Specifically, in Section 2 we propose a Hessian- based framework to analyze the per- iteration improvement on the generalization loss and the effects of per-sample gradient clipping. noising, and hyperparameter choices (e.g., batch size, learning rate).
analyze the effect that DP mechanisms, especially the DP noise but less so the DP clipping, have on pre-training and fine-tuning stages. This leads to some theoretical justifications about why pre-training is some empirical comparisons, and Section 3 for some in-depth discussion.
(2) Based the framework,
vulnerable to DP noise; Figure T for
(3) Our analysis suggests that the deceleration due to DP mechanisms can be mitigated by using certain amount of public data, We then propose a DP continual pre-training strategy that continues the pre-training privately from non-DP initialization. Our strategy is easily implementable, auto- matic and effective, obtaining accuracy on upstream and downstream tasks that is on par with its non-DP variants, while significantly outperforming state-of-the-an th-tth-artttttt art DP pretraining algorithms (see Figure
26
WALE
-5LD
SUEPTARG
SLEP
0.5
SUD+
Adarte
150
Rynds
(a) Pre-training ViT-Base (b) Fine-tuning ViT-Large (CIFARIO,
(c) Pre-trainiing GPT2- (d) Fine-tuning GPT2- Large (CodcParrot, e 8) Large (E2E, 1) Figure 1: Comparison among the convergence of standard SGD, clipped SGD without noise, noisy SGD without clipping, and DP-SGD in different tasks and training stages.
(CIFAR100,
1.2 Related works
This work is closely related to other works in DP convergence analysis, DP fine-tuning of large models, the continual training and the Hessian-based analysis. We refer to Appendix C for an extended discussion.

---

Disnee POOEE MNTURN en CSTABII
2B 18
026
IINO XXi [Sra]KIP T
s-Tecal
Duers and
DENO
ViPeel
SyniViP
MIIL
Oanee 10lli
Placeted
CHARII
10-bhtt
CHAARIOD
Figure 2: Summary of results in Section First three figures compare the downstream and few-shot performance and the data efficiency (circle's radius proportional to pre-training data size) of the DP pre-trained models; the last figure shows the performance of DP pre-trained models defending against privacy attacks (closer to 0.5 is stronger in defense).
1.3 Notions & settings
We denote the mini-bated size B with index set B, dataset size as number of training iterations as T, iteration index as f I T, learning rate as 17, model parameters as w E Rd and training samples We use (K,M, B) for (thousand, million, billion].
We study DP optimization under fixed computation budget (with a fixed number of samples), fixed privacy budget (with fixed DP guarantee), and limited availability of public data.
1.4 Computation budget
We consider a fixed computation budget, that the training ends when fixed number of samples := BT have been processed. As B increases, the number of iterations T decreases linearly, while the per-iteration training time increases almost linearly. This is because foundation models generally trained with large batch size, which requires the use of distributed learning and gradient accumulation() For example, Vision Transformers (VIT, (27 74 is trained with B 4k, GPT-3 (6] and LLaMA (77) with B R 2k, DP-ResNet/ViT with B = 4k in (24] and B = 1M in (8).
DP-RoBERTa with B 2k in 89], and DP-GPT2 with B 1k in 49
As a consequence, the batch size has nearly zero influence on the total training time, leaving its effect only the convergence speed.
1.5 Privacy budget
Definition 1.1 29 26 ) A randomized algorithm M is (c, if, for any two neighboring datasets S,S' that differ by one sample and for any event E,
P(M(S) E T
T (M') 8) where 5 < 1/n.
We consider a fixed privacy budget (e, 8). to be realized through the DP optimizers in (2). Essentially, a noise I) is injected to the gradient at each iteration, whereas the noise magnitude a (B) is determined by privacy accountants such as RDP (de- fault) [59]. PRV (34) and GDP 26 (see Figure 3
010
eetivi by RDF
AUTO
werit CDP
AUTO
parum (h=0.11
011.2)
resparam (R-1)
resparam (R:0.2)
cloping
0.2
0.0
0.00 0.25
0.5
0.75 1.02
per-sample grad nomm [|sil
Figure 3: Noise levels by Figure 4: Per-sample gradi-
privacy accountants.
ent clipping in (3).
"The mini-batch of size (a.ka. logical batch size) is divided into B/b micro-batches, where micro-batch size is b < B (a.ka. the physical or per-GPU batch size, which determines the training speed and memory cost). When à is fixed subject to GPU memory, the per-iteration training time is proportional to the number of micro-batee batches and thus to B.

---

2 Understanding DP training through the lens of Hessian
In this section, derive the impact of different components in DP training, including the per-sample clipping. the noising, and hyper-paramette choices. To do so, we characterize the per-iteration per- sample loss improvement through the lens of Hessian.
We aim to optimize the generalization loss (equivalently the expected risk) for any differentiable loss function, e.g. the cross entropy, mean squared error, or adversarial loss 33
min L(w) E2 [x,,
We denote the per-sample gradient g:(w) = (L,,. 8E, [L(w.)))
dw E Rd. the oracle gradient G(w) = Zhn E Rdd and the oracle Hessian matrix H(w) == ('wweL( E Rdxd In Assump-
Ow
tion samples follow identically and independently (i.id.) from some data distribution, with no
restriction on the covariance structure Eww).
Assumption 2.1. Per-sample gradients g: (w) are i.i.d with
E (w)] G(w), Cov(g:(w) [(w).
Consider the general DP optimizers, such as SGD and Adam [46, which update the parameters with the privatized gradient,
Eies Cig + oo, La)
EieB gi +o,(O,
(2)
B
B
Here C; == Illg; Rt) the per-sample clipping factor that restricts the sensitivity of E, C,g, to some constant R, i.e. ||C.gill s R. We set |[Ggg:] (thus omitting R throughout the paper) following the re-parameterized gradient clipping 24 and the automatic (AUTO) clipping
14] whose C,'s s are listed below:
1
1
Cr.e-param min
or CI,AUTO
(3)
||g
lg:
4 Tillustrates the values of C2 under different clipping functions. Note that in 2 we employ crucial approximation e zs [E[C,] so as to unify the formula of DP and non-DP SGD Remark 2.2 This approximation only holds when the directions of vectors E; Cig, and Σ, gi very close, ie., there is little per-sample clipping bias. Such approximation empirically validated Fig- [ where we observe from the 'SGD' and 'SGD+clipping" curves that the convergence (without noising) is not much influenced by the bias of per-sample gradient clipping. Remark 2.2. Setting 1 and α O, the gradient (2) reduces to the standard mini-batch gradient. Hence, the difference between SGD and DP-SGD characterized by (a, c).
Figure
2.1 Per-iteration improvement of DP-SGD
Next, characterize and analyze the per-iteration improvement of DP-SGD through the lens of Hessian, under different choices of hyperparameters and clipping functions: Wt+1 = w, ng. The extension of the analysis to more general optimizers (e.g., DP-Adam) is given in Section 3.4 We
interested in minimizing the second-order Taylor approximation of L(w ng). which is sufficiently accurate since parameter updates are often quite small [57) The loss improvement in iteration can be approximated as:
1 2 Hg
[(w) L(w n78) R G
Taking the expectation of the right-hand side, we obtain the expected per-iteration loss improvement (derived in Appendix A.I):
n (tr (HCov(g) + E(g] HEgg))
AL = G g
(4)
2
By applying Assumption 2.1 and have Elg] cG, Cov(g) E 2 B Substitute to we obtain a quadratic function of 1y:
7 HG
GG HG + c'tr(H
o'tr(H)
A pppi (1, B) nc G
B +
(5)
2
B

---

We denote the batch size used for SGD and DP-SGD Bonn-D and Bpp, respectively. Then by optimizing the learning rate n7 the per-sample and per-iteration improvement simplifies
1
G
max Alyiiv/ BDP Alinno (BDP) == 2 BDP HG - tr tr( + or (Br 2²)
Given that the total number of processed samples is fixed at S = BT, (6) can be used metric to evaluate the data efficiency of DP and non-DP training with different T, B and (c, 8).
2.2 Per-iteration improvement of vanilla SGD
We can analyze the loss improvement of standard SGD as a sub-case of DP-SGD by substituting 1,0 into 5 according to Remark 2.2
1p² tn (r(H2) GTH AT (B)
1
G
A T puh
r/G GGGTG
(7)
DP
B DP G HG (H)
We visualize 6 7 and their individual terms in Figure
De 123
SP (2)-00
(2-
GIRE
ILRE Dieheaae 121 (1
191 2000 28a1 zice aoss acc
1991 ane 2981 ABE cDR
230
622
851
<b0
2000
(a) DP pre-training
(b) DP fine-tuning
Figure 5: Illustration of different terms in (6) and 7 Left sub-plots depict the denominators and 77.. Right sub-plots depict the whole terms and optimal batch sizes.
(6)
Implication 2.3 (Better DP mechanism helps). From (6). is clear that smaller and larger (hence larger C.) can help DP training. To reduce a, we refer to Section 3.1 for a discussion of methods. For the clipping, under the same sensitivity bound, AUTO clipping 14] gives the largest C2 among all clipping functions (see Figure and therefore is more preferred to use practice.
Implication 2.4 (Batch size should be large, but not too large). As visualized by the red solid curves Figure there exists an optimal batch size (marked in red dashed vertical lines)
( 2GT
Bp o (B) (B)=
Compared to previous DP literature, which encourages the batch size be as large as possible, our derivation of Bвr indicates a sweet pot: while we also support the use of large batch size, we highlight the data inefficiency if Bpp is too large, case that is often overlooked.
3 Impact of per-sample clipping and noising
In this section, we examine the effects of per-sample gradient clipping and noising on the DP train- ing, leveraging the per-iteration per-sample loss improvement derived in Section 2 Specifically, define and analyze a "decelerator" term that characterizes the slowdown by DP optimizers.
Comparing DP-SGD to SGD, we can attribute the slow convergence of DP optimization to the term (Bu in 66, which is not present in the standard training 77) We refer to such term as
decelerator:
(8)
BeB
B
which couples the effects of per-sample gradient clipping and noise through the trace of Hessian. We note that tr(H) in (8) characterizes the curvature (i.e. sharpness flatness) of the loss landscape, which strongly correlates with the downstream performance [5] 45 96 31).
Next, we discuss how the decelerator impacts the non-DP DP training, DP pre-training and fine-tuning.
'The optimal learning rate DP-SGD cannot be used in practice because (i) the oracle G and are unknown; (ii) it is data-dependent and hence violates the DP guarantee. Such an optimal learning rate is only used to help understand the best possible per-iteration performance.

---

3.1 No noise, (almost) no deceleration
When d (i.e., no DP noise), the decelerator vanishes and hence 6 reduces to 7 even the per-sample gradient clipping is used. We empirically verified this in Figure (see blue and black curves), where we see that the difference in convergence with or without clipping is negligible.
Given that DP noise critical to the convergence speed, we highlight some techniques to reduce a under the same budget of (c, 8): the advances in privacy accounting theory can justify smaller noise; algorithms such as LoRA, low-pass and Kalman filters 93 92 can reduce the effective noise.
3.2 DP pre-training can be vulnerable to noise
When A O, the decelerator is non-zero. Therefore, DP training is slowed down by the noise; in Figure II, SGD with noise (yellow and red curves) has worse performance than SGD without noise (black and blue curves). Furthermore, in pre-training, the decelerator is relatively large in the denominator of (6). i.e. BG HG + tr(HE) < z'u(Hi when B I BP (see left sub-plot of Be
Figure a))) and therefore the slowdown can be significant.
Note that the deceleration issue cannot be resolved by increasing B. Although increasing B im- proves the relative speed of DP convergence in comparison to non-DP, i.e., the decelerator decreases, hurts the absolute speed since BG HG increases, and thus the loss improvement (6) also wors- ens (see right sub-plot of Figure 5 ())). Therefore, to design efficient DP pre-training strategy, must keep B moderate and reduce the decelerator simultaneously.
3.3 DP fine-tuning robust noise
Empirical evidence has shown that DP fine-tuning is comparable to (though slightly worse than) the standard non-DP fine-tuning 89
14 12,, despite that a * O. Such a phenomenon implies that comparing public and DP finetuning, we have A (B) = A (B)). That is, the de- celerator becomes small after the public pre-training. This is conceptually illustrated in Figure b)b), where the DP curve close to the non-DP curve at moderate B during fine-tuning. but not so during pre-traine training.
49
24
58
10
(B).
To understand the stark contrast between DP fine-tuning and DP pre-training, plug in the optimal
(H)(H) 2G HG HG from Implication 2.4 to AL Aprv (B). Then, we have the optimal improvement of DP-SGD as
BPP
G
AL (Bop) = 2 2 HG (H) tr(HE)
Implication 3.1. Notice that by choosing Bnon-DP BB in 7 we have that DP-SGD with the optimal batch size is as fast as the standard SGD with twice the batch size,
ALjub (2Bp)) A (L
(9)
Moreover, if Bip is moderate, then DP-SGD converge at similar speed fast converging SGD that uses a moderate batch size.
Remark 3.2. From 7. we observe that non-DP training is data-efficient only if BR HG < tr(HE). Otherwise, we can decrease BBon.DP to effectively improve Alj Therefore, DP training ((H
is data-efficien only if Brp ~ B < 2G HG which holds in fine-tuning but not in early stage of pre-training" A We illustrate the magnitude of the three terms in (6) for pre-training and fine-tuning stages in Figure
In the fine-tuning phase of Figure 6 (r( quickly decreases and so does the decelerator. Hence a moderate Bpp 100 can allow fast convergence. However, the pre-training phase, tr(H) increases to large value within 5 epochs and remains for long time (say epoch to 40) before decreases again. Consequently, DP convergence is initially fast but only for a short period and overall DP optimization is much slower than non-DP optimization, shown in T()(c).
An intuitive explanation is that (HH HG
Elig: G: Hig.
TT. HKC.) resembles the variance
Fig Hg.)
of par-sample g: in the space of H. which decreases the model learns the common representation.

---

407
20
100
Figure 6: Evolution of terms in 6 and 7 that explains the deceleration of DP optimization, during pre-training (left two) and fine-tuning (right two) ViT-Base CIFAR100.
3.4 Extension to general optimizers
The analysis in the previous two sections can be easily extended to arbitrary optimizers as well as techniques such as weight decay, gradient clipping, and parameter-efficient fine-tuning (PEFT). Let p be an optimizer's post-processor of gradient and consider W4+1 = Wt 7p((g))
For examples, following the notation in Pytorch library 65 we can write Adam and SGD with momentum (fi) and weight decay (A),
3
Adam: p(g: m, v)
1-p
SGD(H, A): p(g: w) jub + +g Aw.
Bav 8o)g
10-8
Similarly, we write any PEFT for any optimizer, e.g. PEFT (SGD): p(g: M) M @ g where M E {0, is an element-wise mask that makes a parameter non-trainable or frozen when M, = 0.
Specially, consider optimizers such that p(-) is scale- inver-invariant (i.e. p(cG) pG)), such as SignSGD/Adam [4] or normalized SGD/LAMB [61 55 87]. Applying Assumption 2.1 and the op- timal learning rate, derive the expected per-sample per-iteration improvement in Implication 3.3 leaving the details Appendix A.5]
Implication 3.3. Suppose the post-processor p(-) ) is scale-invariant. Denote = p'=())"'==='C the per-sample per-iteration improvement max, A /B simplifies
1
1p G
2 Bp Hp tr(p' Hp )) o2tr(p' (B²)
Interestingly, similar to the decelerator of DP-SGD 8 the decelerator o²tr(p" Ptr(p'THp') Hpp)/((e²))) of these DP optimizers also couples the per-sample gradient clipping, the noise and the Hessian, ren- dering the theoretical implications from DP-SGD extendable to general DP optimizers.
Continual pre-training with DP
4.1 Necessity of public data in DP pre-training
In this section, we propose the DP continual pre-training strategy and demonstrate that the decel- eration by DP can be effectively mitigated by using a certain amount of public data. We consider the mixed data training that uses both public data (with subscript for related hyperparameters) and private data (with subscript 1). Then SGD becomes
at
Σ 8i, EB,++
(1- at)
Σ Ca.ggir + (N
go,t
Bo
B1
JEBO
tEB:
Here O2 E [0, 1] controls the ratio of privatized and non-privatized gradients, taking different forms
by public training (OnlyPublic, ar 1), private training (OnlyPrivate, a; = 0), DPMD 2.
tunable constant 332 52
and Sample [30 44.
Table 2: Summary of an by mixed data training methods.
Ours
DPMD
Sample OnlyPublic
OnlyPrivate
1(t < sT)
सt I-cos
2K
Using the mixed gradient ga following ( we can show that expected loss improvement is
bivariate quadratic function of (1), (see Appendix A.3) After minimizing with respect to both variables, we obtain:
tr(HE) Bi/Bo + +1)
a' = ctr(HH) - (BB)()
(10)
7

---

Remark 4.1. We see that the optimal choice in (T)) gives a' * O, which indicates that Pn mixed i.. the public data helps. On the other hand, the fact that optimal a" * also indicates that PniiyPablii mixed i.c. the private data helps.
4.2 DP continual pre-training strategy overview
Motivated by Section and Remark 4.1 we propose two-phase DP continual pre-training strategy in Appendix D) public pre-training for s'' steps followed by private continual pre-training for (1 - s)T steps, which equivalent to setting or, II(t sT') and the constant 0 < controls the portion of steps of public training.
Remark 4.2. Although (1O) suggests that the optimal a E (0, 1), we only set binary at {0. 1} so as to not implement two data loaders and two back- propagation mechanisms simultaneously (one for DP and one for standard). Note the methods in Table 2 are difficult to scale and not yet openly implemented on distributed systems due to memory and synchronization issues.
We highlight that DP continual pre-training is distinct from DP fine-tuning their non-DP counter-parts, despite both methods extending the learning from a pre-trained phase: the contin- ual pre-training learns common knowledge without adapting to a specific task and serves richer foundation for many downstream tasks (see more discussion in Appendix (C). In fact, we show in Ap- pendix A.7 that the loss improvement of DP continual pre-training can be almost as fast as non-DP pre-training on the full data (FullyPublic), thus closing the utility gap.
24
FullyPublic
FullyPrivate
Table 3: Optimization by different training strategies.
12
OnlyPublic
Test Loss
Mixed (PubRatio=1%)
public data
private data
privacy
Mixed PrubRatio=10%
SwitchPoint
OnlyPublic OnlyPrivate
SGD
not used
Yes
not used
DP-SGD
Yes
PullyPublic
SGD
SGD
No
FullyPrivate
DP-SGD
DP-SGD
Yes
1.5
Mixed (our B.)
SGD
DP-SGD
Yes
Epochs
Figure 7: Pre-training GPT2-small on CodeParrot with different pre-training strategies = 8 Mixed or FullyPrivate).
The switching (i.e. selecting s) can be automatically determined by public statistics without using any DP budget. For example, we can use early stopping based on the loss or accuracy, switching after the metrics stop improving (see Figure Alternatively, we can monitor Bop and switch when it drops to a moderate value so that the DP training converges fast by (9.
DP vision foundation models on ImageNet
We leverage the DP continual pre-training strategy discussed in Section to train vision transformers
27 on ImageNet datasets. We use ImageNet-Ik (1.3M images, 1K classes; (25)) for public pre- training, then ImageNet-11k (formally known ImageNet-21k-112 11M images, 1 classes; (70) for private pre-training. Notice that ImageNet- k/21k is significantly harder to learn, with SOTA accuracy x 47% 770 (4] compared to 85% for ImageNet- kk. We apply data augmentation including random flipping, contract shift, rotation, and resizing to 224 x 224.
We evaluate DP ViT upstream and downstream tasks (including few-shot), achieving high accuracy under low computation budget (compared to existing DP pre-trained models in Table 4 In short, show that unlocking more data (90% of full ImageNet), which cannot be used by DINO due to privacy concern, is significantly beneficial.
'ImageNet- lk improves the dataset quality and thus the resulting models (see Table in (70) ). by removing the infrequent classes from the full ImageNet-2Ikk, thus remaining 78% of its 14M images and 48% of 21.8k classes.
8

---

Table Pre-training strategies of models. Standard non-DP training marked in black: DP training is in green. i indicates self-supervisee without using the labels. "Images x is the total number of images used (dataset size xepochs). "Non-privacy" means no DP guarantee on a subset of training data due to the non-DP pre-training phase.
reference
model
pre-training
continual training
non-privacy
images X
TAN
71
ImageNet-IK
1.28
(Syn)ViP
90
VīT-Base
Shaders21k
Shaders21k
1.3B
ViP
90
ViT-Base
Sh
LAION400M
Shaders2
1.9B
DINO
19
ViT-Base
ImageNet- Ik"
ImageNet-Ik lk
0.3B
Ours
This work
ViT-Base
ImageNet-1k
ImageNet- 11k
ImageNet-
0.7B
MIIL
70
VTT Base
ImageNet-Ik
ImageNet- 11k
ImageNet- lk
1.4B
Original
27
VTT Base
ImageNet-21k
ImageNet-IK Ik
ImageNet-22 21k
1.2B
AugReg
74
ViT-Base
ImageNet-1k
ImageNet-21k
ImageNet-21k
4.3B
NF
24
ResNet-50
JFT300M
JFT300M
4.0B
5.1 Training strategy
For the public pre-training, we follow the self-supervised learning by [9] (self-distillation with no labels, or DINO), whereas the private continual pre-training is a supervised learning following
We employ AdamW optimizer with batch size B 4096 and learning rate n 0.0002 set by the line search. Our training strategy similar to a concurrent work 190], with critical differences highlighted in Appendix
770
When automatically switching from public to private pre-training, the classification head (the last layer) is re-initialized because the number of classes is different in the two pre-training phases: we switch from Ik classes to Ik classes. This switching is triggered by early stopping. In the continual pre-training phase, we train with DP linear probing for 10 epochs then with DP full- call-parameter train- ing for 20 epochs, with a total x 100k training steps. This strategy achieves an upstream accuracy 41.5% on ImageNet- 1k by our ViT-Base under 8, and 39.8% under = 2.
5.2 Algorithm implementation
We employ fastDP library to apply the DP-AdamW with automatic per-sample clipping function [14] and layer-wise clipping style [9]. Specifically, the DP optimization is under the multi GPU distributed system, using DP-ZeRO (7] and mrixed-precision training, so as to enjoy the same training speed and memory efficiency as the non-DP training. We calibrate the DP noise using the improved Renyi accountant.
5.3 Downstream performance
Our DP pre-training learns highly transferable representations, demonstrated through the strong performance on list of downstream datasets. We summarize in Table a number of models from previous literature for comparison. We mark DP pre-trained models in green and leave non-DP models in black. The most informative baselines are DINO and MIIL, since our DP models continue the pre-training from DINO, following a strategy similar to MIIL.
In Table compare different pre-training strategies (all non-DP except ours) leveraging the same dataset and same model architecture ViT-Base (86M param). Our evaluation shows that DP pre- trained models achieve high downstream accuracy under standard and non-DP fine-tuning: 98.4% on CIFAR10, 90.2% on CIFAR 100, 86.5% on Food101 and 96.8% on SVHN. Our DP continual pre-training clearly improves upon DINO, with +0.3 2.0% on accuracy, and comparable the non-DP pre-trained MIIL that uses twice the data size (1.4B v.s. our 0.7B).
In Table when the downstream tasks (non-DP) are more challenging with only a few data samples to learn from, our DP model substantially outperforms previous DP pre- pre-trained models across all settings, for example, by +19 ~ 38% CIFAR100 when compared to ViP and TAN. We attribute the success to the high quality of pre-training data, i.c. ImageNet- MiENNeet-//I// 1k. in contrast to Shaders (by comparing DINO to ViP) and LAION (by comparing the improvement from DINO ours and from (Syn) ViP to ViP).
'We observe that, using AugReg (a supervised pre- pre-training strategy) for public pre- pre-training on ImageNet 1k and/or for private pre- training on ImageNet-21k gives very similar result.

---

Table 5: Standard/DP fine-tuning accuracy with the same architecture (ViT-Base) and pre-training dataset (ImageNet-21) 21kk up to subsampling and preprocessing. Number of processed images by each model is indicated in the parenthesis.
Fine-tuning tuning
CIBÁR10
CIFAR100
Food101
SVHN
Pre-truiniffg
non-DP
non-DP
non-DP
non-DP
DINO (0.3B)
98.1
97.2
97.0
88.2
84.7
82.7
85.2
77.2
73.5
96.2
91.7
90.3
Ours,-2 (0.78)
97.8
96.6
96.1
88.8
83.1
81.1
84.8
75.5
72.5
96.3
91.3
90.1
Ours (0.7B)
98.4
97.2
96.9
90.2
85.0
82.8
86.5
78.4
75.3
96.8
92.5
91.3
MILL
98.8
98.2
91.4
89.2
872
83.0
96.8
93.3
92.0
WT base (1.28)
98.9
98.3
98.1
92.6
89.9
88.2
89.4
85.5
83.1
96.9
93.5
92.5
AugReg (4.38)
98.9
98.5
93.1
90.4
90.2
87.6
85.7
96.9
93.8
92.5
Table 7: Linear Sinear-probing accuracy (non-DP) of pre-trained models, except "full" indicating full-parameter.
Table 6: Few-shot accuracy of DP pre- trained models (TAN, ViP and ours) and their non-DP
initialization.
ImugeNet Ik Places365
NN
line
Airrratt (10-shot)
Auteratt (20-shot)
CIFAR100
(10-shot)
CIFARIOO
(30-shot)
line tune
cpochs
# #Images
TAN..s.
49.0
IM
L8M
40.5
2.7M
31.7
90790790
epochs
TAN_
(Syn)ViP
ViP
21.79
22.84
31.62
46.85
53.05
37.33
27.78
38.96
40.95
57.52
42.35
55.84
200
200
200
(Sy))iP
Ours
ViP
DINO
764
76-2
55.7
49.8
52.5
52.1
46.1
43.2
43.5
46.5
38.1
32.4
90 00/90/900
90/90/
8/5/10
8/5/
DINO
32.04
45.61
47.31
66.92
100
OurS
77.9
53.0
49.1
8/5/ 10
Ours
Our
42.57
36.42
48.27
57.15
64.74
65.26
76.38
74.62
100
100
Ourl
(ful))
78.5
55.7
60.0
57.2
8/5/ 10
8/5/
NFnet-JFT JFT
In Table 7 extend the evaluation of full fine-tuning and linear probing SOTA non-DP base-
lines and to million-image scale Our DP model achieves 55.7% (49.6% over ViP) on Places365 with 1.8M images and 60.0% (+21.99 9% over ViP) iNat2021 with 2.7M images. The current non-DP
SOTA is 57-60% on Places365 24 27 and about 64% on iNat2021 [80, [6) after pre-training on
2.7 4B images. This showcases the effectiveness of DP pre-training our models only leverage
0.7B images.
5.4 Privacy protection
We employ a white-box membership inference attack (MIA) to evaluate the data protection by our DP pre-training: 1) for each image in ImageNet-1 1k, we compute output logits and loss, which serves as the feature of the MIA dataset; 2) we randomly select 50% of the testing images and the same number of training images (522, 496 samples) as the MIA test set, and the rest data as MIA train set; 3) we label the training images class "1" and testing images as class "0", This creates the MIA dataset with 1 features and binary labels.
We fit a logistic regression with the MIA training set classify whether an image belongs to the training set of ImageNet- 11k (class "I") or not. We report the re- sults on the MIA testing set in Table X showing the effectiveness of DP protection when e < 8.
Table 8: Membership inference attack results. Values
closer to 0.5 indicate better privacy protection.
Accuracy
Precision
Recal
F1
AUC
O O
50.1%
50.1%
47.5%
0.49
0.50
Ours
51.3%
51.6%
64.4%
0.57
0.51
Ours,
53.8%
54.1%
71.2%
0.62
0.54
MIII.
54.0%
58.5%
77.3%
0.67
0.62
6
Discussion
In this paper, we conduct insightful and unified convergence analysis on DP optimization. Specif- ically, we identify the decelerator (8) of DP training as result of the per-sample gradient clipping, the noise and the Hessian, which can be significantly mitigated by small amount (< 10%) of public training. Consequently, we propose DP continual pre-training that is almost as accurate and implementable as the fully public pre-training.
For ImageNet-Iee 1k, we strike through some results because this dataset was used in the pre- pre-training. It is only meaningful to compare our models to DINO and observe the benefit of continual pre-traie traininingg but not to others.
10

---

Acknowledgement
The work of Xinwei Zhang was partially done while interning at Amazon. Mingyi Hong holds concurrent appointments as an Amazon Scholar and as a faculty at the University of Minnesota. This paper describes their work performed at Amazon.
References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pages 308-318, 2016.
[2] Ehsan Amid, Arun Ganesh, Rajiv Mathews, Swaroop Ramaswamy, Shuang Song, Thomas Steinke, Vinith M Suriyakumar, Om Thakkar, and Abhradeep Thakurta. Public data-assisted mirror descent for private model training. In International Conference on Machine Learning, pages 517-535, PMLR, 2022.
[3] Raef Bassily. Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Elficient algorithms and tight error bounds. In Proceedings of the 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 464-473, 2014.
[4] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560-569. PMLR, 2018.
[5] Leonard Berrada, Soham De, Judy Hanwen Shen, Jamie Hayes, Robert Stanforth, David Stutz, Pushmeet Kohli, Samuel L Smith, and Borja Balle. Unlocking accuracy and fairness in differ- entially private image classification. arXiv preprinl arXiv:2308. 10888, 2023.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language mod- els are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] Zhiqi Bu, Justin Chiu, Ruixuan Liu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Zero redundancy distributed learning with differential privacy. In ICLR 2023 Workshop Pitfalls of limited data and computation for Trustworthy ML.
[8] Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard data science review, 2020(23), 2020.
[9] Zhiqi Bu, Ruixuan Liu, Yu-Xiang Wang, Sheng Zha, and George Karypis. On the accuracy and efficiency of group-wise clipping in differentially private optimization. arXiv preprint arXiv:23100 10. 19215, 2023.
[10] Zhiqi Bu, Jialin Mao, and Shiyun Xu. Scalable and efficient training of large convolutional neural networks with differential privacy. Advances in Neural Information Processing Systems, 15:38305-38315 2022.
[11] Zhiqi Bu, Hua Wang, Zongyu Dai, and Qi Long. On the convergence and calibration of deep learning with differential privacy. Transactions on Machine Learning Research, 2023.
[12] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Differentially private bias-term fine-tuning of foundation models. In Forty-first International Conference on Machine Learn- ing.
[13) Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Differentially private optimization on large model at small cost. In International Conference on Machine Learning, pages 3192- 3218. PMLR, 2023.
[14] Zhiqi Bu, Yu- Yu-Xiang Wang. Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. Advances Neural Information Processing Systems, 36, 2024.
11

---

[15] Zhiqi Bu and Yuan Zhang. Differentially private optimizers can learn adversarially robust models. Transactions on Machine Learning Research, 2023.
[16] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP). pages 1914-1914. IEEE, 2022.
[17] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022.
[18] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Frlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
pages
[19] Mathilde Caron, Hugo Touvron, Ishan Mísra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceed- ings of the IEEF/CVF international conference computer vision, pages 9650-9660, 2021.
[20] Junjie Chen, Wendy Hui Wang, and Xinghua Shi. Differential privacy protection against mem- bership inference attack on machine learning for genomic data. In BIOCOMPUTING 2021: Proceedings of the Pacific Symposium, pages 26-37. World Scientific, 2020.
[21] Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspective. Advances in Neural Information Processing Systems, 33:13773- 13782, 2020.
[22] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned 11m, 2023.
[23] Rudrajit Das, Abolfazl Hashemi, Sujay Sanghavi, and Inderjit S Dhillon. On the convergence of differentially private federated learning on non-lipschitz objectives, and with normalized client updates. arXiv preprint arXiv:2106.07094, 2021.
[24] Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlock- ing high-accuracy differentially private image classification through scale. arXiv preprint 2rXiv:2204.13650 2022.
[25] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[26] Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. Journal of the Royal Statistical Society: Series (Statistical Methodology), 84(1):3-37, 2022.
[27] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Inter- national Conference on Learning Representations, 2020.
[28] Jingfei Du, Édouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. Self-training improves pre-training for natural lan- guage understanding.
Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5408-5418, 2021.
[29] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensi- tivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
[30] Cecilia Ferrando, Jennifer Gillenwater, and Alex Kulesza. Combining public and private data. arXiv preprint arXiv:211 (tt::2//1.001.,, 15, 2021.
12

---

starcoder-memor [31] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- imization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
aharpness-aware aware min-
[32] Aditya Golatkar, Alessandro Achille, Yu-Xiang Wang, Aaron Roth, Michael Kearns, and Ste- fano Soatto. Mixed differential privacy in computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8376-8386 8386, 2022.
[33] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. arXiv preprint
2014.
[34] Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems, 34, 2021.
[35] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[36] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy. Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, 2020.
[37] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
[38] Jiyan He, Xuechen Li, Da Yu, Huishuai Zhang. Janardhan Kulkarni, Yin Tat Lee, Arturs Back- urs. Nenghai Yu, and Jiang Bian. Exploring the limits of differentially private deep learning with groep-wise clipping. In The Eleventh International Conference on Learning Representa- tions.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 770-778, 2016.
[40] Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell, Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, et al. Learning and evaluating differentially private pre-trained language model. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1178-1189, 2021.
[41] Daniel Huynh. of fine-tuning on code.
Starcoder memorization experiment highlights privacy risks httpeiinfaaccccooogggg//o co/blog//////oooooooo.cc blog/dhuynh95/
2023.
[42] Huseyin A Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James With- ers, and Robert Sim. Training data leakage analysis in language models. arXiv preprint arXiv:2101.05400 2021.
[43] Shaoxiong Ji, Tianlin Zhang, Kailai Yang. Sophia Ananiadou, Erik Cambria, and Jörg Tiede- mann. Domain- -specffic continued pretraining of language models for capturing long context in mental health. arXiv preprint arXiv:2304. 2023.
[44] Zach Jorgensen, Ting Yu, and Graham Cormode. Conservative or liberal? personalized dif- ferential privacy. In 2015 IEEE 31St international conference on data engineering, pages 1023-1034. IEÉE, 2015.
[45] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On batch training for deep learning: Generalization gap and sharp min- ima. In International Conference on Learning Representations, 2016.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6900,6 2014.
[47] Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at imagenet scale with differential privacy. arXiv preprint arXiv:2201. /2328, 2022.
13

---

forerp--pppo-oooooe----sseeee [48] Xuechen Li, Daogao Liu, Tatsunori B Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin- Tat Lee, and Abhradeep Guha Thakurta. When does differentially private learning not suffer in high dimensions? Advances in Neural Information Processing Systems, 2022.
[49] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Repre- sentations, 2021.
[50] Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- tion branches out, pages 74-81, 2004.
[51] Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Ma- PMLR, 2023.
chine Learning, pages
[52] Ruixuan Liu, Zhiqi Bu, Yu-xiang Wang, Sheng Zha, and George Karypis. Coupling public and private gradient provably helps optimization. arXiv preprint arXiv:2310.01304 2023.
[53] Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella- Béguelin. Analyzing leakage of personally identifiable information language models. In 2023 IEEE Symposium Security and Privacy (SP), pages 346-363. IEEE Computer Society, 2023.
[54] Yi-An Ma. Teodor Vanislavov Marinov, and Tong Zhang. Dimension independent gen- eralization of dp-sgd for overparameterized smooth convex optimization. arXiv preprint arXiv:2206.01836, 2022.
[55] Danilo P Mandic. A generalized normalized gradient descent algorithm. IEEE signal process- ing letters, 11(2):115-113 2004.
[56] Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as ap- proximate bayesian inference. Journal of Machine Learning Research, 18:1-35, 2017.
[57] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv: 1812.0616, 2018.
[58] Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky, Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205.0299 02973,, 2022.
[59] Ilya Mironov, Rényi differential privacy, In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263-275. IEEE, 2017.
[60] Andre Nakkab, Benjamin Feuer, and Chinmay Hegde. Lit tuned models for efficient species detection. In 2nd AAAI Workshop on Al for Agriculture and Food Systems, 2023.
[61] Yurii Nesterov. Introductory lectures on convex optimization: basic course, volume 87.
Springer Science & Business Media, 2003.
[62] Lily Hay Newman and Andy Greenberg. when told to repeat 'poem' forever.
Chatgpt spit out sensitive data httpps://wwt wired. com/story/
2023.
[63] Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Úlfar Erlingsson. Tem- pered sigmoid activations for deep learning with differential privacy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9312-9321, 2021.
[64] Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. Bleu: method for automatic evaluation of machine translation. pages 311-318, 2002.
[65] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
14

---

[66] Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta. How dp- fy ml: A practical guide to machine learning with differential privacy. Journal of Artificial Intelligence Research, 17:1113-1201, 2023.
[67] Katyanna Quach. Inside the 1tb imagenet data set used to train the world's ai: Naked kids, drunken frat parties, porno stars, and more, Oct 2019.
[68] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.
[69] Md Atiqur Rahman, Tanzila Rahman, Robert Laganière, Noman Mohammed, and Yang Wang. Membership inference attack against differentially private deep learning model. Trans. Data Priv. (1(1):61-7, 2018
[70] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Selnik-Manon Imagenet- -2k pretraining for the masses. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[71] Tom Sander, Pierre Stock, and Alexandre Sablayrolles. Tan without burn: Scaling laws of dp-sgd. In International Conference on Machine Learning, pages 29937-29949. PMLR, 2023.
[72] Dezhao Song, Andrew Vold, Kanika Madan, and Frank Schilder. Multi-label legal document classification: A deep learning-based approach with label-attention and domain-specific pre- training. Information Systems, 106:101715, 2022.
[73] Shuang Song, Thomas Steinke, Om Thakkar, and Abhradeep Thakurta. Evading the curse of dimensionality in unconstrained private glms. In International Conference on Artificial Intelligence and Statistics, pages 1636-2646. PMLR, 2021.
[74] Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022.
[75] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017.
[76] Rohan Taori, Ishaan Gulrajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Stanford alpaca: An -nstuuction-oolloving llama model. https://g/ thub. wato://ttts-laabttaatt
_alp_caaaaaa 2023.
[77] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2300.. 130.7 2023.
[78] Florian Tramer and Dan Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations, 2020.
[79] Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Considerations for differentially pri- vate learning with large-scale public pretraining. arXiv preprint arXiv:2212.064770, 2022.
[80] Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12884-12893, 2021.
[81] Di Wang, Changyou Chen, and Jinhui Xu. Differentially private empirical risk minimization with non-convex loss functions. International Conference on Machine Learning, pages 6526 6535. PMLR, 2019.
[82] Di Wang and Jinhui Xu. Differentially private empirical risk minimization with smooth non- convex loss functions: A non-stationary view. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1 1182-1189, 2019.
15

---

[83] Zihan Wang. K Karthikeyan, Stephen Mayhew, and Dan Roth. Extending multilingual bert low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649-2656, 2020.
[84] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and per- formance analysis. IEEE Transactions Information Forensics and Security, 15:3454-3450 3469, 2020.
[85] Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Reprexentations, 2020.
[86] Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie- Yan Liu. Normalized//cipped sgd with perturbation for differentially private non-convex optimization. arXiv preprint arXiv:2206. 13033, 2022.
[87] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xi- aodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference Learning Repre- sentations, 2019.
[88] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, et al. Opacus: User-friendly differential privacy library in pytorch.
[89] Da Yu. Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In International Conference on Learning Representations, 2021.
[90] Yaodong Yu, Maziar Sanjabi, Yi Ma, Kamalika Chaudhuri, and Chuan Guo. Vip: A differen- tially private foundation model for computer vision. In Forty-first International Conference on Machine Learning, 2023.
[91] Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32, 2019.
[92] Xinwei Zhang, Zhiqi Bu, Borja Balle, Mingyi Hong, Meisam Razaviyayn, and Vahab Mir- rokni. Disk: Differentially private optimizer with simplified kalman filter for noise reduction. arXiv preprint vrXiv:2410.0388: 2024.
[93] Xinwei Zhang, Zhiqi Bu, Mingyi Hong, and Meisam Razaviyayn. Doppler: Differentially private optimizers with low-pass filter for privacy noise reduction. Advances in neural infor- mation processing systems, 2024.
[94] Xinwei Zhang, Xiangyi Chen, Mingyi Hong, Zhiwei Steven Wu, and Jinfeng Yi. Under- standing clipping for federated learning: Convergence and client-level differential privacy. In International Conference on Machine Learning, ICMI. 2022, 2022.
[95] Yingxue Zhou, Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. In International Conference on Learning Represen- tations, 2020.
[96] Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference on Machine Learning, pages 7654-7663. PMLR, 2019.
[97] Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference Machine Learning, pages 7654-7663. PMLR, 2019.
16

---

A Derivation and proofs
A.1 Derivation of Equation
AL
n/
Eg
Egg
Hg]
2
112
7/
tr [HE gg
2
172
7/G
EEg
tr (HCovvgg) + tr (HE
[g] E[g]
2
1/²
7/
E
g
(tr (r(Coo((g)) E
HE HEgg
where (a) uses equations 13 and then 12) the second term; (b) separates the expectation of gg
into its mean squared and covariance and uses (11 (c) uses equations 112 and then (13) to the last term.
Property of the trace:
tr(A + B) = (r) + +B)),, tr(cA) ctr(A),
Linearity,
(11)
tr ( tr(BA),
Trace of product,
(12)
tr(a) 0,
Trace of a number.
(13)
A.2 Explanation for Implication 2.4
We will leverage GDP to validate that (B) = O(1/B), given that most of other privacy accountants are numerical and hard to interpret. We note that 4-GDP has an one-to-one mapping with the (e, 8).D:
8 2 -- M
1 +
2- 1
20
c H
where is the normal cumulative distribution function.
Lemma A.1. Given an iterative algorithm with & sensitivity 1 at each iteration, which uniformly samples the data in dataset of size n with ratio B. by injecting Gaussian noise N o²I) to the output of the algorithm at each iteration, it satisfies μ-GDP with
V -1) BS(c2/o(B)) 1)/n,
where S denotes the fixed computation budget.
The proof is equivalent to that in Section 2.4 in (26).
By Taylor expansion over B,
BS
1 + + 0
(B) B
S
1
+ of 1
a 2 log
22 BS
+ 1)
12n²
2 B
u2n
2B
B
In the pre-training regime, // is the number of epochs (usually between and 300) and the sample size n is huge, i.c. > 10" for ImageNet and n > 2 for large language model training. Hence the first term is negligible and aBB)²/1 O.5/B.
A.3 Proof of results Section 3.3
Proof. Leveraging we have
Σ
2
E(g) = aG + (1 - a)GG, Cov(g)
o²
+
a)²²
Bu
B
B
17

---

Substituting in the mixed gradient ga and following we can write down the expected improve- ment of mixed data training:
o2 tr(H)
mixed
n/a (1-a)c)
2/3
1p2
a²
(1 a)²
+
Bu
B1
r (a
a)c
which is bivariate quadratic function A to be minimized over (77, a). To make the presenta- tion easier, we denote 77 1ja, 1/2 = 77(1 a), and the above equation can be rewritten as:
Almixed = (nic G G
17f 02
Hr(H)
2 B1
q
e²
n6
)r) - - (mc + G HG.
2 B
Bo
Note that if f) 0 1oo data, respectively.
0, essentially train the model with only public data private
can be written bivariate quadratic function over (7o: m)):
AL mixxed = -Ann)) + Bni + Cnu + Dm + Enon)
which
1 (ir(HE)
A HG
2
2 Bo
B G HG + 1 tr(HH)
1
1 1
GT
B₁
B
C G.
D = GG
E cG HG.
The maximizer is
2BC DE
2AD CE
170 4AB E 111
4AB E
Hence
1-a" a
2AD -CE
11/(2//
2AD CE
-0=1//
2BC DE +1
o*
2BC DE.
Finally
( G G/B, HHH G/B. +1)
ctr Gr(H))
tr(H)) BB/Bo
a"
-
(ctr(H + o +
A.4 Explanation Remark 3.2
From 6 the optimal batch size for DP-SGD is
G
arg max
2 BG HG tr(HE) (
B
H²t
arg min BG HG + +tr(H)) + + /
2 HG
which minimizes
6
to
G
DP-SGD Bip)
2 / G HG o'r /2 (HE
18

---

where Bpp V
dAu(H) GGTH HG"
Note this is equivalent to applying Bm = 2/
d
HG = Bp on 7
SGD(BBon-Dr) = SGD((BBp)) DP-SGD (Brp).
We emphasize that generally
SGD(2Bpp) 전 DP-SGD(Bnp))
A.5
Loss improvement of general DP optimizers Implication
From Wx41
mpp(g)), the expected per-iteration loss improvement becomes
"1² (tr (HCov(p(g)) + Blp(g) HK|p(g)
AT = n Blp(g)]
(14)
2
Applying Assumption 2.1 and delta method, with G
ØI. ōw
we have
VB(((g) p(cG)) = (cG) (g - cG) o1( N
where p'()))
Bp(cG)
E dxd Hence
dcG
B(p(g)) = p(eG), Cov(p(g) = (cG) (c'E/B+ (cG)
Next, the expected improvement contributed by one sample is quadratic function of ry:
of" T Hy))
AL/B ~ TplcG) G
2
B
112 ttrpp' (cG) ( Hp'(cG)))
n?
2 p(cG) HHp(c)))) /B
B
Applying the optimal learning rate, the optimal per-sample loss improvement AL/B B at each itera- tion simplifies
pp(cG) GP
2 B + (tr("((cG)
+
In the special that p is scale-invariant, e.g. in adaptive optimizers like Adam in SignSGD, we get (cG) Op(cG) @p(G)
deG (G) and thus
BeG DG
1
Lp( G
2 Bp( (TGG))Hp())) (G)
TG Hp'(G)E) (2tr(y H/)))))BBBBB
A.6 Improvement of performance measures other than the optimization loss
We now consider two extended cases when the optimization loss is different the performance measures. For example, the model may be trained via the cross-entropy loss but measured on 0-1 accuracy.
optimization performance
example
vanilla
L is cross-entropy
L is cross-entropy; Lother is BLEU or accuracy Lmod adversarial loss; I, is cross-entropy Table 9: DP/hon-DP optimization when the optimization loss is different to the performance mea- sures.
Lathe
vanilla adversarial training
Temad
L
For the first case, we analyze many performance measures DP models (denoted as Lother) beyond the optimization loss L. For instance, foundation models trained on cross-entropy loss can be eval- uated on classification accuracy, F1 score, BLEU (44, ROGUE [50) fairness [7], calibration 355 adversarial robustness, etc.
19

---

We demonstrate that our analysis in previous sections indeed generalizes: equivalent to (5) we have
othher (1) = 1/G other Llgg
Elg Hotherg]
2
where Gtheer other are the oracle gradient and Hessian of Lother
The per-sample per-iteration
improvement, max, A Lother/ B, simplifies
other G|2
BG Hother G + oiterr Σ) + o (
we expect the public pre-training also mitigates this decelerator, that well-trained DP models and non-DP models should be equally performant. Empirically speaking, DP finettnnig has shown to be as accurate 49 24], adversarially robust [I5], calibrated [TT], and fair [5] as the standard fine-tuning.
with the decelerator being trr (f in place of (8). As implied by Section 3.3
For the second case, e.g. adversarial, sharpness-aware, aairness-aware or calibration-awan training, the optimization on the modified loss Hemn but the performance measured on the vanilla loss dw
aL For instance, we want an udversarially trained model to be sufficiently accurate. To specific, Sw
FGSM and PGD (I2/ To perturbation with norm p) lead the modified loss
0Lmmd
Linod = max [,[L(w, x + E) Gmodd 1S1SPP
8w
DP adversarial training applies on
p
B I
Cigmod, + oN((O, 2
Σ
gmod + (0,
B
maxi]e <p L(w,)++E)
where per-sample gradient is Emod,
Note that the modified gradient may &
be post-processed by optimizers like AdamW. To clear, only in this section, we write p and omit the post-processing of the optimizer (i.e. we only show for SGD).
8w
Next, the expected per-iteration loss improvement becomes
n²2 (tr (HCoo(()) Epp HE(p))
AL G Eppl
(15)
Applying Assumption 2.1 we have
E [p] eGmodd Cov(p) = - BanutBBe 0
Hence, the expected per-iteration improvement contributed by sample is quadratic function of y::
172 o 172 (et(((Emma)))
1/2 G GT mGmod)/ B mod
AL/B B. := T modd
Ba
2
B
Applying the optimal learning rate, the optimal per-sample per-iteration improvement, max, AL/B, simplifies to
G G 2 BG mod HG + tr( tr(H)mmo)) 2 oo (H)(())
1
(16)
Again, the decelerator the same as (8) Therefore, DP training will be as good as the standard training when the decelcrator is small under the modified loss. This supports the observation that DP adversarial training can be accurate in [1]. DP natural training is comparable to non-DP natural training.
A.7 Explaining the effectiveness of DP continual pre-training
Taking a step further than Remark 4.1 we claim that DP continual pre-training can converge as fast non-DP pre-training (FullyPublic), conditioning that the non-DP initialization is sufficiently
20

---

strong: running s7' iterations of public training followed by the DP training is only marginally weaker than FullyPublic in
TT
G
T
G
H.G. +
H.E
H.G, +
E.)
B
B
T
Ge
G
G H,G. + WHE
H, + HH,
t=
given that the decelerator
is small for fort? ST.
B
Experiment settings and additional details
In this section, provide additional descriptions and experiment details for the numerical results the main paper.
B.1 Datasets
We the following datasets throughout this paper.
ImageNet: ImageNet-21kk 25 is the full dataset with various releases which contain 14.2M images from 21,841 classes. ImageNet- 1k is subset of ImageNet-2Ik using Ik high-level classes. We use the ILSVRR2012 version, which contains 1.2 million training images and 100000 test images. ImageNet-1 1 (70) is another subset of ImageNet-216 k that removes invalid/infrequuen classes, retaining 11.1M images (Mrain :0.52M and 10,450 classes in the Winter21 version.
CIFAR-1O/CIFRR-100 50,000 training and 10,000 test images, with 10 or 100 classes, respectively.
Food101: it contains 101 classes of food, with 101,000 images in total, 1000 images per class (training 750 and test 250).
contains 10 classes of digits in natural scene images, with 73257 training and 26032 test images.
SVHN:
Aireraft: FGVCAireraft dataset contains 3334 training and 3333 test images with 100 classes.
Places365: Places365-Standare dataset contains 1.8 million training and 36000 test images from 365 scene classes.
iNar2021: iNaturalist 2021 dataset contains 10,000 classes of species, with 2.7 million training and 0.1 million test images.
CodeParrot: a GitHub dataset of about 180 GB containing roughly 20 million Python files. We use 3% of these files, that is 606,720 rows of training and 3322 rows of test data. We take sequence length 128.
E2E: a dataset restaurant reviews, containing 42,061 training and 4693 test instances.
We take sequence length 100.
B.2
Experiment settings
For images, we use 224 224 resolution and patch 16 for vision transformers. After pre-training, an additional classifier head is inserted for each downstream task.
B.2.1 Figure
We use batch size 1000 and search over learning rate E [5e - 5, 1e - 4, 2e 4,5e 4].
CIFAR10, ViT-Base, random initialization, 7j = 5e without noise, = 5e with noise, 10 epochs.
21

---

2. CIFARI00, Vir-Large, pretrained by timm library, 5e without noise, 5e with noise, 5 epochs.
3. CodeParrot, GPT2-Large, random initialization, = 1e - 4 without clipping, v/ 5e with clipping. 'hii follows closely from https: stttp::/huggingfacce http://nnaaee...aaaeaaaeeaeeeeee
nip course/chapeer7/6
4. E2E, GPT2-Large, pretrained by transformers library, f le 4 without noise, 1)
4 with noise, 10 epochs.
B.2.2
Figure 3
10
1.0
1/n.S= 10t (1 epoch). We use the privacy accountants in Opacus [8] RDP
59
GDP 26
PRV 34. The dashed lines depict (1)) /B for these privacy accountants.
B.2.3
Figure 5
For pre-training, x² 0.25, GH = le2, tr(H)/c² = 2e8,tr tr(H)) 2e4; for fine-tuning. a² 0.25, G HG (e2) 2e6, tr HH 2e4.
B.2.4 Figure
We train ViT-Base with 1j = 5e - 4, B = 500, c 2. We use the Hutchinson method to compute Hessian-related trace, c.g. tr(H) EveRa (v Hv] + 1/2 Σ V Hv, where v ~ (0, Ia) and & 100. Note the Hessian-vector product Hv can be computed by one pack-propagation of G G(w) v E I on w.
B.2.5 Figure 7
We train GPT2-small with 7 2 10-4, B 7680, epochs Epub (if not early stopped by patience = 2). pevv for mixed training, = for other cases. In the figure, 'PubRatio' the percentage of public data among all data.
B.2.6 Table
We empirically observe that ViT-Base models with/without fine-tuning on ImageNetIk.
E.g. bise paaseec 224 miil in21k ft in1k v.s. base patcc16. 224 miil.in211
vit base patch16 224. augreg in21k ft in1k v.s. vit base patch16 224 augreg in21k are very similar final accuracy, though those with fine-tuning converge faster at early epochs.
We train for 10 epochs, B = 1000.
non-DP fine-tuning: full-parameter, = 5e - 5 (except for SVHN and CIFARIO under 2, 7) 2e instead).
DP fine-tuning: BiTFIT 12], = (except for SVHN and CIFARIO under 2, use 1 = 5c 4 instead). We observe that for our models, the accuracy may benefit from smaller learning rate (Ie-3) when >
B.2.7
Table
We train with full-parameter fine-tuning and B = 1000. Our learning rate 1) 5e - (c = 8) and 2e- 5 (c 2). s-shot means samples per class, e.g. 30-shot CIFAR 100 means 3000 samples
1
in total.
B.2.8 Table 7
We train with 5e 5 (e 8) and 2e - (e 2) for full-parameter fine-tuning and parameter-efficeett fine-tuning. note that with much heavier (public) pre-training, e.g. on JFT4B, DP NFnet-F3 and JFT4B can achieve even better accuracy [24]
22

---

B.3 Details in training stage switching
When switching from public pre-training to private continual pre vre-training, the DP gradient may have a different scale than the historical optimizer states and make the continual pre- training unstable. Therefore, we investigate different strategies to re-initiallize the optimizer states during the switching. Throughout the training, we use AdamW, which have three states: the iteration index (l), the first- order momentum (m), and the second-order momentum (v).
In the experiment, we train a VīT-Base model from scratch on the CIFAR100 dataset. In the first three epochs, we use vanilla AdamW, then we switch DP-Adam and continue training for one epoch. During the switching, we fixed the learning rate and reset different states (t, V, m) to zeros. The ablation study result is shown in Figure
From the figure, we observe that re- re-initializing the first-order momentum m (R1) results in the best performance when switching from public to private training. For the other cases, we observe performance drop when switching.
-RS
R
-RRS
R2
-2R
Train Accuracy
R12
-RR12
Ref
400
800
1200 1600
Sceps
Figure 8: Ablation study of switching from non-DP to DP training with Adam on CIFAR100 dataset. When switching (T 1200). we re-initialize different states the Adam optimizer in different linestyles. "RI", "R2", and "RS" indicate m, and are re-initialized, respectively. "N" indicates no re-initialization, and "Ref" is the reference behavior of continual training with non-DP AdamW.
B.4 Details for MIA
To conduct the membership inference attack (MIA) in Table we employ a white-box attack with full access to model parameters and data, to evaluate the data protection by our DP pre-training. The attack has two major steps. Step I: construct MIA dataset. To construct the MIA dataset, we use the following procedures: 1) for each image in ImageNet-1 1k, we compute its output logits and loss, which serves as the feature of the MIA dataset; 2) we randomly select 50% of the testing images and the same number of training images (522, 496 samples) as the MIA test set, and the rest 50% of the testing images and 10% of the training images as MIA train set; 3) we label the training images as class "1" and testing images class "0" This creates the MIA dataset with 1 11k features and binary labels. Step II: evaluate MIA performance. After obtaining the MIA dataset, we fit a binary logistic regression with the MIA training set to classify whether image belongs to the training set of ImageNet-11 1k (class "1"). We use the L-BFGS optimizer and class re-weighting to train the model for 50 epochs. After training,
evaluate the performance of the training classification model on the MIA test dataset and compute the classification accuracy, precision, recall, Fl Fcore,,, and AUROC of the classification task. The MIA attack procedures are illustrated in Figure 9
C Related works
Convergence analysis of DP training (the goals) Recent works have attributed the slow DP
convergence to the large number of model parameters
995
48 the noise level added to gradients
84 14 86), and the per-sample gradient clipping [II
21
94
Our analysis in (6) and (8) covers
these factors as well as the choice of hyperparameters. We note that key quantity of DP conver- gence is tr(H), which implicitly covers the model dimension d and is also analyzed [54] to give a dimension-free generalization bound. Moreover, existing works mostly study the empirical conver- gence on the training set, in terms of the gradient norm |g - O, the parameter space w, - w.. or the training loss T, - Ta as f. - co. Our work focuses on the generalization performance,
23

---

Label:
10% Train
Outout Train 50%
ImageNet Train Data
Train Data Output
Classification
ViT Model
Output
Optimize the classification
model with MIA train datasel. Evaluate the model on MIA test dataset
ImageNet Test Data
Label: 0
2.4% Train Output MIA Test 50% Test
Test Data Output
Output
Figure 9: The process of membership inference attack (MIA).
and pays particular attention on the data efficiency (as well computation efficiency) that is rarely captured in the literature.
More on DP convergence analysis (the assumptions) The convergence analysis of non-convex
DP deep learning is very challenging and under-studied. There are roughly 3 routes to walk around the challenge: (1) Working on convex models instead (see 733 23 and more in See 1.2 (66)) ) e-g- assuming some form of convexity or only optimizing the last layer (essentially linear model). This route offers deep theoretical insights thanks to the simplified models, but generally fails to match the training dynamics nor the performance of deep learning. For instance, last-layer training may work reasonably well in computer vision but not so in language tasks, obtaining only 26 BLEU score for DP-GPT2 on E2E dataset compared to 63 BLEU score via DP full-parameter training. (2) Working gradient flow, rather than the gradient descent in practice, in order to get
on the continuous-ting rid of the per-sample gradient clipping the Gaussian noise. This route essentially works with infinitely small learning rate, which falls short in the performance as SOTA DP models are trained with large learning
[49]. (3) Assuming that the loss I. is Lipschitz continuous (i.e. the gradient is While both assumptions lead to some insights of the DP training behaviors, the Lipschitz constant is hardly calculable and time-dependent (maybe even diverging) in practice. Particularly, in the Lipschitz continuous setting, the per-sample gradient clipping is not used in DP deep learning, reducing the technical difficulty significantly but missing the gist of DP-SGD from a practitioner's viewpoint.
bounded) (3 [8 82 or Lipsethiz smooth [1 14 86
In contrast, work does not rely these and many other assumptions in the literature, because we do not study the loss convergence along all iterations. We instead scrutinize the per-iteration loss improvement in (5.. Put differently, we focus the local behavior of DP optimizer rather than the global behavior.
Differentially private training The literature of DP deep learning has predominantly focused on fine-tuning the pre-trained models (e.g. full-parameter [ 49 [4] and PEFT [78 89 12
58) )) Im-
portantly, DP fine-tuning (1) can be as performant the standard non-DP fine-tuning consistently, (2) achieves better empirical performance with larger models [4] 9
and (3) necessarily relies on the public data pre-training Our work is distinct from previous works since we focus on DP pre-training with large sample size, large number of iterations, and very limited public data. Interestingly, we no- tice that DP pre-training also enjoy these desirable characteristics of DP fine-tuning, e.g. requiring public data (only small amount) in 4.
benefiting from scaling up the models (see Fig- ure 4, Figure 5 and Table in (90). This empirical evidence of similar training dynamics between DP pre-training and DP fine-tuning, despite their difference in learning goals, is well-supported by our decelerator analysis in Section
Continual pre-training Continual pre-training is training strategy that accumulates knowledge from large amount of data, in order to improve its generalization ability to unseen domains and datasets B6
4 72]. It is different from fine-tuning which focuses task-specific and often smaller dataset. Therefore, the (continually) pre-trained models possess strong few-shot (in- cludign zero-shot) ability but may not be competent in specific functionality such as conversation, and vice versa for fine-tuned models like ChatGPT, Alpaca (76), and Dolly [22].
28
83
Multiple DP pre-trained models have been developed 47 24 90
but these models either (1) suffer
from low accuracy, e.g. 6.9% on ImageNet- 1k without additional data by [47 and 32.4% by (24]; or (2) demand significantly more data and compute to match the non-DP pre-training, e.g. 100x in
24

---

[90] when comparing DP ViP model to non-DP SimCLR; or (3) rely on uncommon tricks such as huge batch size from 16-98K, that are not adopted in the standard deep learning community. Related to (3), we believe DP continual pre-training can further improve using the existing techniques from the non-DP continual training, including the replay mechanism and curriculum learning, and avoid catastrophic forgetting.
Hessian-based analysis in deep learning Applying the second-order Taylor expansion of loss motivates the famous Newton's method and is commonly adopted in deep learning [57 97 85 56 [T], where the Hessian matrix is useful to analyzing the convergence and data efficiency (e.g- selecting the critical batch size (57] ) even though it is infeasible to derive H E Rdxd explicitly for large models. Our work follows the same path with a specific focus on DP related operations (i.c. the clipping and the noise). We use the Hutchinson method and deessian-vector product to compute tr(H), HG and so on.
System design of DP training To make DP deep learning broadly applicable, we believe it is nec- essary to not only evaluate DP algorithms on the utility, but also from a system design perspective. The design of DP system, such our DP continual pre-trame traininniin should resemble that of the stan- dard non-DP system (see our extensive discussion in Remark (42). Such a design will be compatible to and benefit from advances in the much larger non-DP DP literature, unifying DP and non-DP communities, instead of crafting techniques that are limited to DP learning only.
Data distribution shift There may be some distribution shift between the public pre-training data and the private fine-tuning data or continual pre-training data. Empirically speaking, DP training can be robust such distribution shift. For instance, DP fine-tuning has successfully transferred from ImageNet to CIFAR1O 10, CIFAR100 100, SVHN, Food101, CelebA, FMNIST, GSTRB [4], from JFT to ImageNet 24 58 and from Places365 ImageNet (47). Although this work does not address the distribution shift due to similarity between ImageNet-Ik and ImageNet-1 Ik, our anal- ysis in Section 3 especially the formula of decelerator, still holds in this scenario. Therefore, we expect our DP continual pre-trainin to work withstanding the data distribution shift, as empirically demonstrated from Shaders to LAION 90
Concurrent work The concurrent work - ViP 90 also studies the DP continual pre-training, thus is the closest strategy ours. We elaborate the similarity and differences between two strate- gies.
Similarity
1. Both use ViT-Base as backbone and AdamW as optimizer.
2. Both use self-supervised learning (SSL) for public pre-training.
3. Both use e for DP continual pre-training.
4. Both public datasets (Shaders v.s. ImageNet-1k) have Ri 1M images.
Differences
1. ViP uses SSL on DP continual pre-training. We use supervised learning.
2. ViP adds decoder to ViT-Bast (in total 99.0M parameters) during the pre- pre-training. We replaces the classifier head of ViT-Base (94.4M parameters) for DP continual pre-training Hence the model architecture is different.
3. ViP trains on Shaders (for 1.6B images) and LAION (for 0.6B images). We train on ImageNet-1k (for 0.3B images) and ImageNet- 1k (for 0.4B images). Hence data dis- tribution and quality is different, and our training requires about 1/3 the computation.
4. ViP's private dataset (LAION) has 233M images. Ours (ImageNet- 1k) has 12M images. Hence our training requires much smaller dataset (see Figure ((a) (90).
5. ViP has batch size 8K). We use 4K.
huge batch size 98K (see Figure 4(b) 90 where the loss diverges with
6. ViP experiments with various model sizes, from ViT-Nano (18.6M) to Large (233M). We only use ViT-Base.
25

---

D Algorithm of DP continual pre-training
Algorithm DP continual pre-training
1: switch.to Da-Faase
2:
for 1- 1,., do
3:
Compute the loss L2 by forward pass
4:
if DPttchhto then
5:
Compute public gradient g
6:
else
7:
Compute private gradient g
8:
end if
9: Update Wi+1 W: 7g
10: if L, > Lt-1 then
11:
Set switch DP=True Tru
12: end if
end for
26

---

The abstract and introduction accurately reflect the paper's contributions. NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We recognize the limitations of the second-order Taylor expansion and that we cannot explain the trace of Hessian's evolution.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
This paper has no theorem.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple-
mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification:
We give all the necessary information and will open-source codebase.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
Open access to code will be available after paper decision.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.
oolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips.c///////
ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification:
We give all details in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We believe the pattern is clear.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
29

---

is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproducc the experiments?
Question:
Answer: [No]
Justification: This work focuses the accuracy, independent of the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http::::///uurpss... http:/ps.c..pbbbtttttccccccodude
Answer: [Yes]
Justification: does.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
30

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
31

---

Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the d as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "i6Yqixteh7", "reviewer": "Reviewer_KeLK", "summary": "This paper provides a theoretical framework to analyze impact of various aspects (parameters) of DP training on the performance of resulting models. The framework uses hessian of per-sample gradient to compute per-iteration loss improvement. Using the framework, the paper shows how DP impacts performance of model in pre-training more than during fine-tuning, and suggests using public data pre-training as a remedy. Finally, based on the observations, the paper proposes a DP continual learning and shows how it can help improve the performance of upstream and downstream tasks.", "questions": "- It looks like the conclusions made in the paper rely heavily on the assumption about clip norm multiplier being the same for all per-sample grads. The assumption is because in Figure 1 performances of models with and without clipping are similar. But, why would this be always true? If clipping is very aggressive, it will introduce bias and performance will reduce much more than what we see in Figure 1. Maybe it is good to clarify the scope of the assumption.\n\n- In the “Per-iteration improvement of DP-SGD” it says that parameter updates are generally small. When is this true in practice?  \n\n- In Table 3, why do performances of many fine-tuning cases reduce from DINO to Ours(eps=2)? I think the performance should always improve? At the same time, few-shot performance of proposed method is significantly higher than DINO in all cases. Can you clarify this difference?", "limitations": "- Assumptions made in the work to set up the theoretical framework need better clarification in terms of the scope of theory.\n- It would be good to formally write an algorithm of DP continual pre-training, as I found section 4.2 a bit confusing.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Approach of the proposed framework can be useful to analyze many DP settings.\n- Interesting conclusions especially Implication 2.4 about batch size\n- Proposed continual learning approach seems practically useful in privacy sensitive settings  \n- Few shot accuracy of the proposed approach is impressive", "weaknesses": "- Some assumptions, especially about clip norm, need more clarification \n- Some of the theoretical claims could be paired with empirical evidence\n- Proposed continual pre-training approach needs better explanation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and comments. Given that the weaknesses are mostly about clarification and explanation, we are happy to address them if the reviewer can be slightly more specific, besides the questions which we address below.\n\n\n> It looks like the conclusions made in the paper rely heavily on the assumption about clip norm multiplier being the same for all per-sample grads. The assumption is because in Figure 1 performances of models with and without clipping are similar. But, why would this be always true? If clipping is very aggressive, it will introduce bias and performance will reduce much more than what we see in Figure 1. Maybe it is good to clarify the scope of the assumption.\n\nWe clarify that the clipping norm multiplier is NOT the same for all per-sample grads and this would NOT be always true. We explicitly state in Line 115-116 that \"This approximation **only holds when** the directions of vectors ... are very close, i.e., there is **little** per-sample clipping bias.\" Note little bias must be distinguished from no bias. In fact, we are using the most aggressive clipping, i.e. the normalization which means clipping threshold is infinitely close to 0, but Figure 1 shows the impact of clipping is much less significant than the impact of noising. In short, clipping does has a bias, but this bias is sufficiently small so that the approximation is sufficiently accurate and informative.\n\n> In the “Per-iteration improvement of DP-SGD” it says that parameter updates are generally small. When is this true in practice?\n\nWe thank the reviewer for this question. Here are a few cases where the parameter updates are generally small in deep learning: (1) the learning rate is small because the update is a multiplication of learning rate and gradient; or (2) the model size is large so the training experiences a phenomenon known as lazy training (especially in the neural tangent kernel regime); or (3) weight decay is applied so the parameters are within a ball around the initialization and the updates are bounded.\n\n> In Table 3, why do performances of many fine-tuning cases reduce from DINO to Ours(eps=2)? I think the performance should always improve? At the same time, few-shot performance of proposed method is significantly higher than DINO in all cases. Can you clarify this difference?\n\nWe confirm that on the pretraining dataset ImageNet, the performance indeed always improve with DP continual training, including Ours(eps=2). Therefore the inconsistency may arise from the dataset distribution shift, i.e. some datasets are less similar to ImageNet so the improvement does not transfer well. We will add this discussion as a future direction!\n\n> It would be good to formally write an algorithm of DP continual pre-training, as I found section 4.2 a bit confusing.\n\nWe haved added it in the rebuttal Appendix D.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J5LVO8jOcJ", "reviewer": "Reviewer_JCrd", "summary": "This paper proposed a Hessian-based analysis of the per-iteration loss for DP-SGD and non-private training, and provide a theoretical explanation for slower convergence of differentially private training. Authors identified the *decelerator* compontent in per-iteraton loss improvement, associated with the gradient clipping and noising, providing a theoretical backing for differences between DP impact on pre-training vs. fine-tuning. Using the same framework, authors also suggest the derivation for the optimal batch size for private training under the fixed computation budget (i.e. fixed number of samples processed), which strikes the balance between lower per-sample noise and training slowdown associated with growing batch size.\n\nBased on this analysis above, authors suggest the impact of DP on convergence can be mitigated by public pre-training data. While using public pre-training data to complement DP fine-tuning has been explored before, authors emphasize the difference between fine-tuning and continual pre-training, provide a theoretical analysis for the optimal mixing proportion for private/public data, and evaluate their models after private continual pre-training on a downstream fine-tuning tasks.", "questions": "* I think the formatting for Fig.5 is wrong - in the text you refer to \"upper left\" plot, while in the submitted version it's rendered as a single line of 4 plots\n* I would be interested in reading how did you compute the data for Fig.6 - did you explicitly compute full matrices `G` and `H` or used some approximations?\n* I don't fully understand why in Fig.5 (pre-training) the blue dashed line is linear, suggesting that $G^THG$ is constant.\n* In Table 2, a) does number of images include continual pre-training? and b) what does \"non-privacy\" column refer to?\n* What is the criteria for distinguishing continual pre-training from fine-tuning? For example, in your experiments you perform continual pre-training with a different objective than earlier pre-training (supervised vs non-supervised). Does it not justify the \"fine-tuning\" term?", "limitations": "Authors do not discuss limitations explicitly, but are very upfront about assumptions they make (e.g. fixed compute budget, assuming no bias from clipping, etc). Authors also briefly mention limitations in the checklist (but not the main body of the paper).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I believe this is a strong and well-written paper with valuable contributions to the field.\n\n* The paper is very clearly written, with all major claims supported by arguments, derivations, and graphs. It's easy to follow and it gets the point across.\n* The derivation of the decelerator component of the per-iteration loss is a valuable contribution. It provides a good explanation for the DP-SGD training dynamics, and provides a theoretical backing for the previously observed empirical results. Specifically Fig. 6 provides very interesting insights intro DP-SGD training which I haven't seen analyzed before.\n* The downstream findings (e.g. optimal batch size or optimal ratio of private/non-private data) could be impactful for the practical applications of DP-SGD.", "weaknesses": "The major weakness of the paper (at least its theoretical part) is the assumption that per-sample gradient clipping does not introduce bias into the gradient approximation (line 116). This is a very strong assumption, and something that, to the best of my knowledge, is not generally accepted in the field (e.g. see [this paper](https://proceedings.neurips.cc/paper_files/paper/2020/file/9ecff5455677b38d19f49ce658ef0608-Paper.pdf)). To justify the assumption authors point to Fig. 1, which to me doesn't look like it supports their claim - there's a significant difference between vanilla optimizer and optimizer + clipping in pre-training.\n\nAdditionally, the analysis in this paper assumes the oracle knowledge of matrices G (gradient expectaition) and H (Hessian). For practical applications where this is infeasible, it would be useful to look at, e.g. optimal batch size not only from an optimal loss perspective, but also take into account how well the batch gradient approximates the actual gradient G. \n\nOh the evaluation side, I see a slight disconnect with the theoretical results. For instance, after the results in Sec. 4.1 it would be natural to explore different mixing ratios of public/private data in training - however authors only focus on a setup with fixed pre-training and continual training datasets. \n\nThe results presented in Tables 3,4 and 5 do make a good case that the proposed approach is valid, but lack proper baselines. Comparisons are made either with non-private models, models trained on different dataset, or model trained for different number of epochs - contradicting the scenario with fixed compute budget.\n\nAuthors do not report compute resources used for the experiments, which is especially relevant for reproducibility of the paper's results, as it works with Hessian matrices which can be very computationally expensive to compute.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and comments. We will address them point-to-point. The answers to some questions are merged in the response to the weaknesses.\n\n> The major weakness of the paper (at least its theoretical part) is the assumption that per-sample gradient clipping does not introduce bias into the gradient approximation (line 116). This is a very strong assumption, and something that, to the best of my knowledge, is not generally accepted in the field (e.g. see this paper). To justify the assumption authors point to Fig. 1, which to me doesn't look like it supports their claim - there's a significant difference between vanilla optimizer and optimizer + clipping in pre-training.\n\nWe agree that gradient clipping definitely introduces bias to the gradient approximation, but the extent of the bias is less significant and hence ignorable, compared to the effect of DP noising (see the closeness of blue and black curves, and the remarkable distance between yellow and black curves, especially for GPT2). We are using this assumption to prioritize the analysis of noising (which leads to trace of Hessian and decelerator), which is necessary to simplify our analysis.\n\n> Additionally, the analysis in this paper assumes the oracle knowledge of matrices G (gradient expectaition) and H (Hessian). For practical applications where this is infeasible, it would be useful to look at, e.g. optimal batch size not only from an optimal loss perspective, but also take into account how well the batch gradient approximates the actual gradient G.\n\nWe agree that oracle G and H is not available (as we marked in footnote 2) and we will extend the discussion on the selection of batch size.\n\n> On the evaluation side, I see a slight disconnect with the theoretical results. For instance, after the results in Sec. 4.1 it would be natural to explore different mixing ratios of public/private data in training - however authors only focus on a setup with fixed pre-training and continual training datasets.\n\nThanks for pointing out this precious suggestion. To clarify, the ratio is about the re-weighting of gradients. The reason we didn't experiment with other mixing ratios is in Remark 4.2, as these methods are hard to implement and not scalable, and most do not have open-source code.\n\n> The results presented in Tables 3,4 and 5 do make a good case that the proposed approach is valid, but lack proper baselines. Comparisons are made either with non-private models, models trained on different dataset, or model trained for different number of epochs - contradicting the scenario with fixed compute budget.\n\nWe agree the comparisons are not perfectly comprehensive and it is computationally expensive to explore them. E.g. VIP is trained on Shaders21k (hence we will double our computation budget at least) and NFnet is trained on JFT (which is not publicly available). We hope the reviewer would agree that this is acceptable as we follow the same experiment setup as in VIP, and the utility of our method is clear even though we use much less compute.\n\n> Authors do not report compute resources used for the experiments, which is especially relevant for reproducibility of the paper's results, as it works with Hessian matrices which can be very computationally expensive to compute.\n\nWe are using 1 A100 GPU and we will release the trained models for reproducibility. We actually never computed the Hessian matrix because we only need tr(H), which can be computed via Hutchinson method (briefly mentioned in Line 820): we sample 50 random vectors $z$ and compute $zHz$ as a scalar and then average over 50. Here $zHz$ is computed from finite difference of losses. Please let us know if more details are desired.\n\n> I don't fully understand why in Fig.5 (pre-training) the blue dashed line is linear, suggesting that GHG is constant.\n\nIn Figure 5 the illustration is for one iteration so GHG is a constant and B is the variable. GHG does change over iterations as we show in Figure 6.\n\n> In Table 2, a) does number of images include continual pre-training? and b) what does \"non-privacy\" column refer to?\n\na) Yes. We stated in caption that it is \"the total number of images\". b) DP is defined specifically on dataset. If we train on data A non-privately and then on data B privately, then only B has privacy guarantee and A is indicated in \"non-privacy\". We will clarify this in the camera-ready.\n\n> What is the criteria for distinguishing continual pre-training from fine-tuning? For example, in your experiments you perform continual pre-training with a different objective than earlier pre-training (supervised vs non-supervised). Does it not justify the \"fine-tuning\" term?\n\nIn this work, we consider fine-tuning to be a) on a specific (and usually much smaller) dataset so that some techniques like parameter efficient finetuning (e.g. LoRA) is applicable because the change in parameters are small. However continual pre-training is on a large amount of data for a series of downstream tasks and LoRA won't work. b) the second dataset has a distribution shift, e.g. many DP papers publicly pretrain on ImageNet and finetune on CIFAR10. However, the continual pre-training mostly uses a similar dataset, e.g. we pretrain on 10\\% of ImageNet and continue the training on the other 90\\%.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Uwm4iJaUP", "reviewer": "Reviewer_9edf", "summary": "This paper addresses the challenge of DP pretraining, which has been limited due to performance degradation. The authors first analyze per-iteration improvements of the optimization process and then propose a novel DP continual pre-training strategy that uses a small amount of public data to mitigate the negative effects of DP noise. They then empirically demonstrate the effectiveness of their approach on large-scale vision models.", "questions": "Overall, this paper is interesting. I am happy to raise my score if some of the questions are addressed. \n\n\n- Can the authors provide some comments on the first weakness?\n- Section 3.3 is somewhat confusing. First, Remark 3.1 investigates the differences between pre-training and fine-tuning by comparing $B_{non-DP} G^{\\top} H G$ and $tr(H\\Sigma)$, while in fact, public per-iteration improvement is monotonic with respect to $B$. Why use a comparison between $B_{DP}$ and $B_{non-DP}$ for deriving the explanation? Also, can the authors more rigorously explain what is meant by \"data-efficient\"?\n- Moreover, the difference between public improvement (7) and private improvement (6) lies in the decelerator $\\sigma^2 tr(H) / (Bc^2)$, which seems smaller for pre-training where the loss landscape is flatter and thus the curvature $tr(H)$ is smaller. What is wrong with this intuition?\n- It is observed that prediction accuracy is linear in $\\log B$, for instance in Figure 4 in [1], while the statement here is that $B$ should be moderately chosen. Though these are not the same quantity, could the authors comment on the impact of $B$ on DP-SGD?\n- Based on analysis in Section 3, is there any off-the-shelf rule for choosing $B$? \n- Can the authors provide some ablation studies (can be on toy datasets) on the impact of $s$?\n\n\n\n[1] Tom Sander, Pierre Stock, and Alexandre Sablayrolles. Tan without a burn: Scaling laws of dp-sgd. In International Conference on Machine Learning, pages 29937–29949. PMLR, 2023.", "limitations": "The limitations should appear in the main text.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Conclusions are well supported by figures and illustrations.\n- The framework of analyzing improvements appears novel to me.\n- The experimental evaluations are extensive, and the improvements are significant.", "weaknesses": "- The authors claim that *a certain amount of public data can mitigate deceleration*. However, Section 4 lacks discussion about deceleration. The given claim is that per-iteration improvement is increased, which is obvious given public data. Also, it's unclear why *limited* public data is emphasized, as the analysis does not involve public sample sizes. This makes the article somewhat incoherent, as the interesting theoretical analysis in Section 3 does not provide strong suggestions for the methodology.\n- The experiments lack some ablation studies to support the key methodological proposal.\n- Citations should be updated to published versions rather than arXiv preprints where possible.\n- In line 14, there's an extra \"and\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and the comments. We will address them point-to-point and hope the reviewer can raise the score if satisfied.\n\n> The authors claim that a certain amount of public data can mitigate deceleration. However, Section 4 lacks discussion about deceleration. The given claim is that per-iteration improvement is increased, which is obvious given public data. Also, it's unclear why limited public data is emphasized, as the analysis does not involve public sample sizes. This makes the article somewhat incoherent, as the interesting theoretical analysis in Section 3 does not provide strong suggestions for the methodology.\n\nIndeed, we agree with the reviewer that we do not provide a theoretical characterization of \"limited public data\", whereas the \"limited\" is only emphasized from a practical standpoint. In Section 4, we have leveraged our knowledge about the deceleration in Equation (9) to suggest the mixing ratio of public and private gradient, and to motivate the DP continual pretraining that empirically mitigates the deceleration by achieving higher utility. We emphasize the \"limited public data\" because our mitigation of deceleration is so effective that we use much fewer public data than other DP training strategies (see right-most column in Table 2).\n\n> The experiments lack some ablation studies to support the key methodological proposal.\n\nWe have ablation studies over privacy budgets (multiple epsilon), few-shot or full settings, task difficulties (from CIFAR10 to Places365). We are happy to discuss more ablation studies if the reviewer can be more specific.\n\n> Citations should be updated to published versions rather than arXiv preprints where possible.\n\nWe complete agree and will update in the camera-ready (NeurIPS does not allow revision this year).\n\n> Section 3.3 is somewhat confusing. First, Remark 3.1 investigates the differences between pre-training and fine-tuning ..., while in fact, public per-iteration improvement is monotonic with respect to B. Why use a comparison ... for deriving the explanation? Also, can the authors more rigorously explain what is meant by \"data-efficient\"?\n\nHere \"data-efficient\" means the per-sample improvement is high: the same 0.1 improvement obtained by 10 data samples is much more data-efficient than obtained by 1000 samples. While public per-iteration improvement is monotonic, the rate of improvement can be different. For example, if we have the (batchsize, loss improvement) pairs: (B=1, 0.1), (B=2, 0.08), (B=10, 0.07), (B=1000, 0.0699). Hence, in Remark 3.1, we stated that non-DP batch size should be small to be data-efficient.\n\n> Moreover, the difference between public improvement (7) and private improvement (6) lies in the decelerator, which seems smaller for pre-training where the loss landscape is flatter and thus the curvature is smaller. What is wrong with this intuition?\n\nWe are glad the reviewer points this out! We agree that tr(H) is small initially as we also demonstrate in Figure 6. But it quickly increases to a large value (within 5 epochs) for a long time (say epoch 5 to 40) before it decreases again. Therefore, even though DP training is fast initially, it is only fast for a short period and overall DP is much slower than non-DP. This is also observed in Figure 1 (a)(c). We will add this discussion to the camera-ready paper.\n\n> It is observed that prediction accuracy is linear in log(B), for instance in Figure 4 in [1], while the statement here is that B should be moderately chosen. Though these are not the same quantity, could the authors comment on the impact of B on DP-SGD?\n\nWe believe B should be moderately chosen. However, the optimal B may be very large and hence, for a range of batch sizes smaller than the optimal B, it is not wrong empirically that increasing B could improve convergence. Nevertheless, we notice [1] does not explicitly consider adjusting the learning rate for different batch sizes. Therefore, the conclusion there does not hold in our setting as we also use optimal learning rate (see Equation (5)) in this work. Notice that in Equation (5), if eta learning rate is independent of B, then it is monotonically decreasing in B. However, in theory, and in practice, the learning rate depends on B, and thus Equation (5) is NOT monotonically decreasing in B, as we demonstrate in Figure 5.\n\n> Based on analysis in Section 3, is there any off-the-shelf rule for choosing B?\n\nUnfortunately, choosing B is not easy, even for non-DP training.\n\n> Can the authors provide some ablation studies (can be on toy datasets) on the impact of s?\n\nIn Figure 7, we have three values of s: s=0 is fully private training (red curve); s=1 is fully non-private training (black curve); automatic s (switch point and blue curves) around 25\\%. We also use automatic s (around 10\\%) in all experiments in Section 5. In short, if we plot utility against s, we will observe a sharp trade-off that small s (not too small) suffices to pick up most utility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a6xGby0qAD", "reviewer": "Reviewer_wr8p", "summary": "This work theoretically justifies the loss of utility under DP pre-training under the lens of a Hessian. This theoretical result can then be leveraged to perform efficient pre-training of models on public data (small amount) to significantly improve DP-trained model's performance.", "questions": "Can the framework be used to explain the reason behind DP being 'more difficult' for untrained models through the prism of memorization?\n\nIs there a more theoretically sound way to define the threshold of public data that is required for appropriate performance?", "limitations": "As outlined above: while the theoretical justification of the results is indeed interesting, there are no novel conclusions our takeaways from this work that we have not previously had from similar works in the area of DP training.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem authors attempt to solve in this paper is very important and the work is timely (given the concerns re:usage of copyrighted material during pre-training of large-scale models). To the best of my knowledge, the results are theoretically justified and sound. There are many experiments supporting authors' claims in this work, outperforming the existing approaches.", "weaknesses": "My fundamental issue with the positioning of this work is that we already know that public data improves DP - [1,2,3,4,6]. We also know that DP training during pre-training is harming utility more than it would have harmed the pre-trained model [2,3,4]. Therefore my question is what is the main scientific output of this paper? The conclusions are already widely established and I do not see any new outcomes that can be derived from the conclusions of this work. It would have been interesting to see if this result it linked to model's generalization/memorization capacities as described in [5], showing some novel contextualized understanding of the training dynamics, but there was no discussion on the interplay of the training time and memorization capacity under DP.\n\nThe theoretical justification of why this phenomenon arises is indeed interesting and can be useful, but on its own does not provide any novel insights into the training process that are 'actionable'. This is, again, because it was already previously established that a) DP pre-training harms utility [1,6,7], b) public data aids utility of DP models [2] and c) when models are untrained, they are more susceptible to various phenomena that can affect the training dynamics (e.g. loss of information via clipping [5]). So I am struggling to understand what can one gain from the results of this work? The 10% data threshold is a purely empirical result, which may or may not scale to other settings and datasets.\n\n[1] - Bu, Zhiqi, Jialin Mao, and Shiyun Xu. \"Scalable and efficient training of large convolutional neural networks with differential privacy.\" Advances in Neural Information Processing Systems 35 (2022): 38305-38318.\n[2] - Kurakin, Alexey, et al. \"Toward training at imagenet scale with differential privacy.\" arXiv preprint arXiv:2201.12328 (2022).\n[3] - Nasr, Milad, et al. \"Effectively using public data in privacy preserving machine learning.\" International Conference on Machine Learning. PMLR, 2023.\n[4] - Ganesh, Arun, et al. \"Why is public pretraining necessary for private model training?.\" International Conference on Machine Learning. PMLR, 2023.\n[5] - Feldman, Vitaly. \"Does learning require memorization? a short tale about a long tail.\" Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing. 2020.\n[6] - Mehta, Harsh, et al. \"Large scale transfer learning for differentially private image classification.\" arXiv preprint arXiv:2205.02973 (2022).\n[7] - Mireshghallah, Fatemehsadat, et al. \"Differentially private model compression.\" Advances in Neural Information Processing Systems 35 (2022): 29468-29483.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We will address them point-to-point and hope the reviewer can raise the score if satisfied.\n\n> My fundamental issue with the positioning of this work is that we already know that public data improves DP - [1,2,3,4,6]. We also know that DP training during pre-training is harming utility more than it would have harmed the pre-trained model [2,3,4]. Therefore my question is what is the main scientific output of this paper? The conclusions are already widely established and I do not see any new outcomes that can be derived from the conclusions of this work. \n\nWe would like to highlight that the main scientific output is to answer \"why DP training suffers from slow convergence?\", whereas existing works mostly only observe that \"DP convergence is slower than non-DP\" without a fine-grained explanation. That is, knowing something happens is not the same as knowing why something happens, and scientific insights beyond empirical can only roots from the why. Our explanation via the trace of Hessian works for both pre-training and fine-tuning (our contribution 1; see Figure 1). We specifically separate the clipping and noising and emphasize that clipping is not troublesome but noising is (our contribution 2; again see Figure 1)! We then use our analysis to provide an actionable training strategy that only requires a small amount of pre-training data (which is very different from most works where most data are non-DP and only a fraction of finetuning data requires DP) and automatically switches to DP training (our contribution 3).\n\n> The theoretical justification of why this phenomenon arises is indeed interesting and can be useful, but on its own does not provide any novel insights into the training process that are 'actionable'. This is, again, because it was already previously established that a) DP pre-training harms utility [1,6,7], b) public data aids utility of DP models [2] and c) when models are untrained, they are more susceptible to various phenomena that can affect the training dynamics (e.g. loss of information via clipping [5]). So I am struggling to understand what can one gain from the results of this work?\n\nWe are glad the reviewer finds our theoretical justification interesting. We would like to emphasize some actionable insights in our work: (1) The analysis in Section 3 shows that noising is the main cause of slow convergence. Therefore, we recommend a future direction to improve DP training by reducing the noise vector, instead of improving on the per-sample clipping. Some efforts may include parameter-efficient training (like LoRA), pruning and noise reduction (e.g. via tighter privacy accounting in Implication 2.3). (2) The Section 4 and all experiments rely on our continual pre-training strategy, where we monitor the loss to switch from non-DP to DP training (which implicitly uses the tr(H) information). (3) Given that tr(H) is worsening DP convergence, we may adopt sharpness-aware minimization and regularization to encourage the model to move in a flat region, so that DP convergence can accelerate.\n\n\n> Can the framework be used to explain the reason behind DP being 'more difficult' for untrained models through the prism of memorization?\n\nWe focus on the training dynamics, which the memorization may not directly link to.\n\n> Is there a more theoretically sound way to define the threshold of public data that is required for appropriate performance?\n\nWe show DP convergence is always going to be slower than non-DP, despite the slowdown can be insignificant after initial training, so the threshold will be subjective to \"what level of slowdown is acceptable?\". Generally speaking, if DP batch size is chosen properly (see Line 184), we should expect at most 2 times slowdown, and more non-DP pre-training brings down this factor of 2 to close to 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for the comments and put every effort to address them. Please let us know if there are further questions (though revision is not allowed this year). We provide the algorithm of DP continual pretraining in PDF here (Appendix D).

Meta-review: While most of the reviewers were positive about the paper, there were a few concerns regarding the importance of the result, as pre-training DP models with public data is a well-studied problem. There were also concerns regarding some of the theoretical claims and the practical implementation of the DP continual pre-training approach. 

We would recommend the authors to include rebuttal discussion (mainly with reviewer wr8p) in a future version of the paper.
Decision: Accept (poster)