Towards Large Scale Transfer Learning for Differentially Private Image Classification
Authors: Harsh Mehta
Abhradeep Guha Thakurta
Alexey Kurakin
Ashok Cutkosky
Venue: Accepted by TMLR ()

Abstract: Differentially Private Stochastic Gradient Descent (DP-SGD) has emerged as a popular private training algorithm. Unfortunately, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training. This is further exacerbated by the fact that increasing the number of parameters leads to larger degradation in utility with DP.
In this work, we zoom in on the ImageNet dataset and demonstrate that, similar to the non-private case, pre-training over-parameterized models on a large public dataset can lead to substantial gains when the models are finetuned privately. Moreover, by systematically comparing private and non-private models across a range of large batch sizes, we find that similar to the non-private setting, the choice of optimizer can further improve performance substantially with DP. By using the LAMB optimizer, we saw improvement of up to 20$\%$ points (absolute). We also show that finetuning just the last layer for a \emph{single step} in the full batch setting, combined with extremely small-scale (near-zero) initialization leads to both SOTA results of 81.7 $\%$ under a wide privacy budget range of $\epsilon \in [4, 10]$ and $\delta$ = $10^{-6}$  while minimizing the computational overhead substantially. Finally, we present additional results on CIFAR-10 and CIFAR-100, surpassing previous state of the art by leveraging transfer learning with our recommendations.

Published in Transactions on Machine Learning Research (01/2023)
Towards Large Scale Transfer Learning for Differentially Pri- vate Image Classification
Harsh Mehta
harshlm@goggle.ooo
Google Research
Abhradeep Thakurta
athakurtaakgoooge.ccom
Google Research
Alexey Kurakin
kurakin @iaoiinnggggg...oo com com
Google Research
Ashok Cutkosky Boston University
ashok@cutkosky.com
Reviewed on OpenReview: https: // openreview. net/ forum? id- UuSWuCFpQu
Abstract
Diffcrentially Private Stochastic Gradient Descent (DP-SGD) has emerged as a popular private training algorithm. Unfortunately, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training. This is further exacerbated by the fact that increasing the number of parameters leads to larger degradation in utility with DP. In this work, we zoom in on the ImageNet dataset and demonstrate that, similar to the non-private case, pre-training over-parameterized models on a large public dataset can lead to substantial gains when the models are finetuned privately. Morcover, by systematically comparing private and nou-private models across a range of large batch sizes, we find that similar the non-private setting, the choice of optimizer can further improve performance substantially with DP. By using the LAMB optimizer, we saw improvement of up to 20% points (absolute). We also show that finetuning just the last layer for single slep in the full batch setling, combined with extremely small-scale (near-zero) initialization leads to both SOTA results of 81.7 % under wide privacy budget range of s E [4, 10] and 10-6 while minimizing the computational overhead substantially. Finally, we present additional results on CIFAR-10 and CIFAR-100, surpassing previous state of the art by leveraging transfer learning with our recommendations.
1 Introduction
Despite significant rescarch investment in privacy in machine learning, training private models remains challenging in practice. The standard metric for privacy is Differential Privacy (DP), which bounds the change in model weights (and predictions) if single training example is added or removed. Informally, Differential Privacy implies that an adversary learns almost the same thing about an individual data point independent of their presence or absence in the data set, which intuitively provides protection against membership attacks (Nasr et al. 2021). Prior work illustrates that without DP's guarantees, it is quite plausible attack a variety of models, across modalitics, to reveal individual example information (Shokri et al. [017 Carlini al., 2019: 2021 Choquette- Choo al. 2020 Liu al. 2021 Balle et. al. 2022). Revealing such information can be very dangerous if the model was trained using sensitive data such as demographics personal photos, making it critical to establish practical DP training methods.
Code: http::/:h...cccooooo/ee///eee//a

---

Published in Transactions on Machine Learning Research (01/2023)
Formally we can define DP follows:
Definition 1.1 (Disferential Privacy (Dwork et al. 2006bla)) A randomized algorithem is (e, didffferrennl private if, for any paár of datasets D and D' differing in at most ezample (called neighboring datasets). ) and for all events S in the output range of A, we have
Pr(A(D) E S -P(A(()) E ES] 5,
where the probability is over the randomness of A.
The most popular method for DP training in deep learning is Differentially Private Stochastic Gradient.
Descent (DP-SGD) (Song et al., 2013 Bassily et al. 2014 Abadi et al., 2016). The core recipe implements a common theme in DP: "fuzzing" " an algorithm's 's outputs with noise to obscure the contributions of any individual imput. Specifically, DP-SGD is a preprocessing step prepended to ordinary SGD: for each example in minibatch, compute the gradient, clip to a pre-selected norm, and add Gaussian noise before averaging over the minibatch and taking an SGD step.
In practice, DP training can be very expensive even ineffective for very large models. A naive implementation atin might significantly increase the cost of bounding the sensitivity of each example as parameter count is increased. In addition, the norm of the noise added also increases, which is believed to lead to worse model quality. Our goal is to develop simple and practical procedures for producing high-quality large-scale private models.
To do this, we focus on transfer learning for improving the privacy-utility tradcoff in this large model regime. That is, we first pre-train a model on a large dataset with no privacy concerns, and only then privately finetune the model on the sensitive dataset. This strategy has emerged a promising technique improve the accuracy of private models. We specifically examine the effect of pre-training on JFT-300M, JFT-4B and ImageNet-21k datasets (Sun et al. 2017 Deng et al. 2009), and private finetuning on the popular ImageNet classification ion task with additional results on CIFAR-10 and CIFAR-100 datasets. Here, following Kurakin et al. 2002) ; De et al.] (2022), we are simulating private training on a sensitive dataset by using the publicly available Imagenet data in place of the "sensitive" datasct. We show that using transfer lcarning, utility improves as the size of the model is increased, despite requiring more Gaussian noise. In addition, scaling both model and pre-training dataset size reduces the gap between private and non-private performance.
In search of further improvements, (McMahan et al.) 2018 Anil et al. 2021a Kurakin et al. 2022 De et al., 2022). Thus, we consider very large batch sizes (within a small multiple of the full batch). With these large batch sizes, similar to the non-private setting, we find the choice of optimizer leads to substantial improvements when training the model with privacy. For instance, ResNet50x4 (BiT) obtains 68.8 % test accuracy when trained using DP-SGD with LAMB optimizer (denoted as DP-LAMB below) compared 47.1% using DP-SGD with Momentum over a privacy guarantec of (c (10, 10-6).
recall that increasing batch size improvee performance under DP
Furthermore, we carefully explore other hyperparameters affecting DP performance. Surprisingly, we found the initialization of the last layer to be crucial: initializing the last layer to zero yields significant improvements. In summary, we investigate three distinct recommendations:
1. Increase pretraining data and model size.
2. Use large batch sizes in combination with optimizers that can work well with large batch sizes (e.g. LAMB).
3. Initialize the last layer to zero (or very small) when doing finctuning with DP.
Combining these recommendations, we achieve statc-of-the-art DP results through finetuning just the last layer of ViT-H 14-4b. Remarkably, we only need to train for a single slep in the full batch setting. Recently,
Kurakin et al.
2022) obtain 47.9 % accuracy on a privacy budget of (10, 10 10-6) with 70 epochs of finetuning
and De et al.
(2022) obtain an impressive result of 81.1% on privacy budget of (8, 8 10-") and 77.1% at
(1, 1.8% * 10 10-1), ),
with 2500 epochs. Our results outperform the previous state of the art of Kurakin al (2022)
and De et al.
2022 for all values of considered in both studies. In addition, since our best results were
2

---

Published in Transactions on Machine Learning Research (01/2023)
obtained by finetuning the last layer for single epoch on ImageNet, recommendations lead to at least 70x reduction in private training time than prior work and thus significantly improve the cost-utility ratio of training a high-quality image classification model with DP.
84
81.5
81.7
81.7
85
Top-1 Accuracy
82
81.1
Top-1 Accuracy
79.4
80
80
78
75.6
76
DP
74
Without-DP
70
0.25
0.5
2
4
6
8
10
B/16
L/16
H/14
H/14-4b
(a) Scaling model and pretraining dataset
(b) Varying privacy parameter
Figure 1: (a) Compares best models from our study with and without privacy on ImageNet across model and pre-training dataset sizes. Scaling the model (from VIT-B/16 - VIT-L/16 - ViT-H/14) and using a larger pro-training dataset (JFT-4b instoad of JFT-300M) docreases the gap in accuracy coming from addition of the privacy guarantee of (e 10, 8 10-6) (b) We finetune only the last layer of ViT-H/14- H/1444b model for single step in the full batch setting and vary E [0.25, 10). The previous state of the art of Kurakin et al. (2022) obtains 47.9 % accuracy on privacy budget of (10, 10 with 70 epochs of finetuning. More recently, concurrent work of De el al. (2022) obtain an impressive result of 81.1% on privacy budget of (8, 8 * 10-3) and 77.1% at (1, 8 10-"), with 2500 epochs. We present significant. improvements on both studies at all values we tried with only epoch of finetuning. Note that these results are average over 5 independent runs with different seods. For reproducabiiity purposes, we also make the exact hyperparameters available in the A Appendix.
2 Related Work
Theoretical Understanding. In the convex case, there is a rich literature on DP learning and optimization
CChaudhuri et al.) 2011 Kifer et al. 2012 Song et al.) 2013 Bassily ct al. 2014 Talwar et al. 2015 Wu et al, 2016 Feldman et al. 2020a Song et al. 2021 Asi el al. 2021), and the optimal privacy-utility tradeofls are well-known. Such matching upper and lower bounds are not available in the non-convex context. of large deep learning models. In practice, DP-SGD (Song et al. 2013 Bassily et al. 2014 Abadi et al. 2016) the most popular learning algorithm. Although its optimality is not known, there improving the theoretical properties of DP-SGD, including adaptive clipping (Andrew el. al. 2021
active line of work
Pichapali
et al. 2019 and private aggregation of teacher ensembles (Papernot et al. 2018).
Transfer Learning. Several recent works demonstrate improved performance the setting where we have access to a large public or non-sensitive dataset of the same modality as the private data (Tramèr Boneh al.) 2022 Kurakin et al.) 2022: Hoory al.] (021). Perhaps the dircction most similar to ours the concurrent work of De et al. (2022) which also employs JFT-300M as public dataset. to obtain high utility privately trained ImageNet models, corroborating some of the trends observe well.
2021 Yu al., 2021
While the overall direction is similar, there are many notable differences including model familios explored (ResNet (BiT) (Kolesnikov et al. 2020) and Vision Transformer (Dosovitskiy et al. 2021) vs NeResNet Brock et al. (021b)), batch size (full batch vs 16k) and choice optimizers for best. results (LAMB/ADAA vs Momentum).
Large batch sizes. In the context of differential privacy, several recent paper advocate the use of large batch sizes (McMahan et al. 2018 Anil et. al. 2021b Dormann al.) 2021 Hoory et al. 2021 Liu al.
3

---

Published in Transactions on Machine Learning Research (01/2023)
2021 Kurakin et al. 2022 in order to improve the privacy-utility tradeoff. We confirm this finding in our setting as well, except for few cases where the degradation in utility due to optimization difficulty with batch size increase trumped any potential increase in utility.
3 Background
We start our discussion with optimization details in the non-private sctting. Given a data set D ((11,81),. (Ine yn} with (xt, V1) E D, we are interested in optimizing the function C R denoted as follows
x D + R
((0) -
Σ (a.y)ED
(10, (I,
D
The standard (non-private) approach to this problem is mini-batch Stochastic Gradient Descent (SGD). In single iteration of SGD, minibatch B, C "D of size B, = N examples are chosen randomly from the dataset. Then, given a current iterate I,, SGD performs the update:
9t + BB
Ve(Oe (z. v))
8111 = 0 -779t
(2)
(2,8)EB.
where V/ denotes the learning rate used for for iteration I.
DP-SGD is a private variant of this algorithm. In order to bound the sensitivity of each training example, Abadi et al. (2016) suggest computeing a gradient for each example separately and clip each a maximum norm of C (a user-specified hyperparameter):
clip (Vi(0;; (x, gt f- 9t CN+ (0,00,
Je +
Σ (B
B1l
where clip(u) =
[Tllla] J. This is notably different from non-private training where the forward pass can be vectorized and only a single pre-accumulated gradient need be calculated and used per mini-batch. If implemented naively, this step alone increases the computational cost of DP training proportional to the batch size for a single step. We do however note that, for several deep learning architcctures, it is indoed possible to bound the sensitivity of each example without calculating the gradient for every example separately, leading dramatic cost-reduction both terms of memory and compute (Goodfellow) 2015). One promising recent technique is Ghost Clipping (Li et al.) 2022; Bu al., 2022) where its possible to match the memory requirements of non-private training with marginal additional compute cost. In this paper, however, we stick to the exact scheme proposed in Abadi et al. (2016). Our recommendation for fine-tuning just the layer also reduces computational cost and can be used in conjunction with other cost-saving techniques like Ghost Clipping.
-- e- min {
C
After summing the clipped example gradients, a noise vector sampled from a Gaussian distribution with standard deviation oC is added, where paramcter that will specify the privacy guarantce via the Gaussian mechanism. Formally, any algorithm that uses this noisy ge (not just SGD) will be private by the standard post- pest-procsssing property of differential privacy.
4 Experiments
Datasets. We use the ILSVRC-2012 ImageNet dataset (Deng al. 2009 with 1k classes and 1.3M images (we refer to it as ImageNet in what follows) our final evaluation dataset. However, we provide supplementary results in Section I where evaluate on 2 additional dalasets, namely CIFAR-10 and CIFAR- 100. We also refer to these as the private dataset for which we want a privacy guarantee. We use variants of JFT datasets for our pro-training as our public (in DP senso) dataset. JFT-300M (Sun al. 2017 consists of 18k classes and 303M high-resolution images and JFT-4B with 29.5k classes and 4B images. Additionally, we also explore ImageNet-21k as our pretraining dataset in Section F
4

---

Published in Transactions on Machine Learning Research (01/2023)
Public and private data. We define "public data" and "private data" only in the context of DP. Public data is simply dataset that we don't 't require privacy guarantee over and Private data is a sensitive dataset. which we do want privacy guarantee over. "Public data" should not be confused with a dataset which is freely available. With this definition mind, it is completely plansible that a model trained with public data (in DP sense) still can not be shared widely without scrutiny due to either ethical, societal or legal concerns. For instance, the JFT datasets we use as "public data" are not technically publicly available but have been used extensively as a pre-training dataset in the non-private setting to obtain state-of-the-art results (Dosovitskiy
et al. 2021 Brock et al. 2021a Tolstikhin et al. 2021: Zhai et al., 2021
Finally, note that none of our finctuning datasets (e.g. ImageNet-1KK) in reality are sensitive datascts: we are only simulating public/private private dataset split only for demonstration purposes (Kurakin et al. 2022 De iet al.
2022). To make sure that our simulated "public" and "private" datasets capture a practical scenario,
we carefully de-duplicate our pro-training datasets w.r.t. all splits of our finetuning datasets (Kolesnikov
ct al.
2020 Dosovitskiy ct al., 2021). More details about this process can be found in the appendix.
Model variants. We evaluate the transfer learning capabilities of ResNet (He et al. 2016) and Vision Transformer (ViT) (Dosovitskiy et al. 2021) in our study. Note that in typical non-private training, ResNet employs batch normalization, which has a difficultto--nnalyze effect on the sensitivity of the gradients and so therefore makes computing the privacy parameters & and & difficult. Somewhat fortuitously, to improve transfer, Kolesnikov et al. (2020) replace Batch Normalization with Group Normalization and used standardized convolutions, which has a much simpler privacy analysis. We therefore follow the same practice and denote our modified model "Resnet (BiT)'. For Vision Transformer, we follow the standard notation to indicate the model size and the input patch size, for example, VIT-B/32 means the "Base" variant with 32x32 input patch size. Note that for VIT, compute requirements scales up we reduce the patch size.
Training details. At the pre- teeetraiiing stage, we stick with the common practice of employing Adam optimizer (even for ResNet) (Kingma & Ba 2014) with B1 = 0.9 and 82 = 0.999, with a batch size of 4096 and high weight decay of 0.1 unless mentioned otherwise. We train with sigmoid cross-entropy loss and use linear learning rate warmup until 10k steps, followed by linear decay until the end of training. For our private finetuning experiments, we stick with a reasonably stringent privacy guarantee of = 10 and d 10-6, unless specified otherwisc. We use DP-SGD privacy analysis to compute the noise multiplier. To limit other confounding factors we set the clipping norm C to 1. Also, since training with DP-SGD is computationally expensive, we finetune on ImageNet for at most 10 epochs. Finally, when training the last layer with DP we found it to be crucial to initialize the last layer weights to zero (or a small value). We expand on this more in Section b
4.1 DP-SGD with Large Batch Sizes
There is an inherent tradeoff between privacy enforced by DP and utility obtained from the model when trained privately. The decrease in the quality of the model stems from the additional randomness added to the gradients as part of DP-SGD, Even though there is precedence for gradient noise leading to superior generalization ability (Neelakantan et al. 2015: Smith et al.) 2020), more often than not, in the context of
differential privacy it leads to substantial degradation in model quality (Song et al. 2013 Bassily al. 2014 Abadi et al. 2016).
DP-SGD first adds per-example gradients in mini-batch, then adds noise vector to the accumulated gradicnt, and finally divides by the batch size. Thus, increasing the batch size might seem to decrease the [B.] decreases. The picture is actually more complicated: increasing batch size also decreases the effect of amplification by sampling, which means that the increase in [Ba] must be partially offset by an increase in a in order to maintain the same privacy guarantee. Regardless, even for large models, others McMahan et al., 2018 Anil et al.) 2021b Dormann et al., 2021 Hoory et al.) 2021 Liu et al. 2021 Kurakin ct al. 2022) have observed that employing larger batch size improves performance of DP-SGD. In our case, ImageNet Net contains roughly 1.3M examples; so we perform finetuning in the range of 128k (2?7) - 1M (220)
noise added as a
batch size across our exploration and use the full batch setting for the final results.
5

---

Published in Transactions on Machine Learning Research (01/2023)
Without privacy
With privacy
Batch Size
Model
Tuning type
2
218
219
220
217
218
219
220
Full
75.8
69.5
57.9
57.0
66.5
57.7
57.8
57.9
ViT-B/16
Last layer
81.1
80.2
75.1
69.2
61.3
65.1
66.6
65.3
RosNet50x4 (BiT)
Last layer
74.1
69.4
61.9
46.1
39.6
43.6
47.1
49.2
Comparison of Top-1 test accuracies on Imagenet when trained using DP-SGD with Momentum when using various models with and without privacy. All models are pretrained on JFT-300M datasct. Note that we exclude ResNet50x4 full finetuning results since we did not see accuracies greater than 20% in in our hyperparameter range even without privacy.
Table 1:
We finetune on ImageNet using 2 different schemes: 1) Full transfer learning, where we finetune all the parameters of the model and 2) Last layer tuning, where we only train a classifier produced from the features obtained from the pre-trained model. Despite all the recent advances made in engineering efficient. implementations of DP-SGD for large models, last layer tuning is significantly more cost-effective in the context of DP-SGD since per-example gradients are trivial to compute and don't lead to much additional compute cost in comparison to non-prive non-private training. In fact, last layer tuning can also be cast as a linear regression problem and solved exactly, but in this study, we stick with gradient-based optimization methods which let rely on DP-SGD's privacy accounting for all our experiments. Finally, in both types of finetuning methods, we reinitialize the last layer and train it from scratch.
As the first step in this section, we start with pre-training on 2 medium-sized models, namely 1) ViT-B/16 B/16 and 2) ResNet50x4. We conduct finctuning in almost the same fashion as reported in non-private cases except our batch sizes are much larger and we replace the traditional Momentum Optimizer with DP-SGD with Momentum.
As shown in Table 1 as the batch size increases, performance in the non-private case decreases significantly as optimization qualily degrades. However, with DP, sometimes observe the reverse trend, even to the point where it is better than non-private training (ResNet50x4, batch size 220). Perhaps surprisingly, our results also suggest that training the full model is not necessarily better than training just the last layer. This is true even without privacy. We conjecture that this phenomenon due to the optimization difficulty with large batch sizes.
4.2 Optimization Difficulty with Large Batch Sizes
Our results in Table show that increasing the batch size improves the privacy-utility tradeoff (except for VYT-B/16-Full] However, without privacy, the performance scems to suffer as the batch size is increased in all cases. This phenomenon, outside the context of differential privacy, has been studied by others already (Krizevskyy 2014 Li et al. 2014 Goyal et al. 2017: You et al. 2017 Hoffer et al. 2018). Most notably, You et al. 2019 introduced LAMB optimizer to address this issue, which modifies Adamm (oshchhlov & Hutter 2017] with a layer-wise adaptive learning rate correction. With LAMB, they trained both ResNet-50 and BERT with batch sizes up to 32k. In this work, we would like to train our models with even larger batch sizes i.c. in the range of 128k-1M. Tuning only the learning rate, in our results Table 2 we extend the use LAMB optimizer for both models in this batch size range, with and without privacy on ImageNet.
Switching from Momentum to LAMB provides improvements in several dimensions. First, without privacy, performance at the lower end of our batch size range comes quite close to that when trained with much lower batch size of 512 (as proposed by Dosovitskiy et al. (2021)). Most notably, we see a big jump in performance for ResNet-50x4 when training the full model. Test accuracy on ResNet-50x4 with momentum did not surpass 20% (and hence was excluded from Table I]. In contrast, test accuracy with LAMB was comparable to VIT-B/16 16 across batch sizes. Second, the batch size most attractive in our context is 220 (around 1M). Note this batch size is close to the ImageNet training dataset size of around 1.28M. We observe
6

---

Published in Transactions on Machine Learning Research (01/2023)
Without privacy
With privacy
Batch Size
Model
Tuning type
Best
217
2
2
220
2
2
2
220
Full
84.8
81.7
77.1
76.3
73.1
68.2
70.5
70.5
71.1
V1T-B/16
Last layer
81.2
80.7
76.4
75.9
73.3
66.4
69.1
70.1
70.6
Full
85.3
83.7
79.5
74.5
72.6
55.6
64.5
68.6
70.1
ResNet5004
Last layer
82.9
82.7
79.1
73.5
72.7
52.5
61.7
66.1
68.8
Table Comparison of Top-1 test accuracics on Imagenet when trained using DP-LAMB when using various models with and without privacy. Similar to Table [ all models pretrained on JFT-300M dataset. The
numbers in Best column are obtained by using Momentum Optimizer in non non-private setting with batch size 512 and rest of the hyperparams reproduced from Dosovitskiy et al. (021
Without privacy
With privacy
Batch Size
Model
Tuning type
Best
220
220
Full
87.1
77.0
72.9
ResNec152x4 (BiT)
Last layer
84.6
76.9
72.9
Full
87.2
79.6
77.4
VIT-L/16
Last layer
84.8
79.4
77.3
Full
88.2
81.5
80.1
ViT-H/14
Last layer
86.2
81.4
80.0
Full
88.7
82.5
81.0
ViT-H/14 4B
Last layer
86.9
82.5
81.0
Table 3: Comparison of Top-1 test accuracies Imagenet when the model and dataset size is increased. We observe consistent improvement with scaling model from ViT-B/16 to VIT-L/16 and even ViT-H/14. Similarly, scaling the pretraining dataset. from JFT-300M to JFT-4B also improves both private and non- private performance. Similar to before, all models are trained using DP-LAMB with and without privacy. The numbers Best column are obtained by using Momentum Optimizer in non-private sctting with batch size 512 and rest of the hyperparams reproduced from Dosovitskiy el. al. (2021
significant increase in accuracy when switching to LAMB for batch size 2 Lastly, the increase in accuracies in the non-private setting helps when training the models privately across the board, the extent that we obtain the best private finetuning numbers the largest batch size of 220
4.3 Scaling Analysis
In this section, we systematically study the effect of scaling both the model size and the pre-training dataset. We follow Dosovitskiy et al. (2021) and experiment with 3 additional models, namely ResNet152x4, ViT-L/16 and ViT-H/14. Similar to earlier experiments, we pretrain these models with JFT-300M. To study the effect of incrcasing pre-training data, we consider one more variant "VIT-H/14 4b" where we pro-train ViT-H/14 model JFT-4B dataset for single epoch. Due to the excessive computational cost of running these experiments, only consider batch size of 2 for all our results in this section.
As shown in Table 3. for both the model families, namely ResNet (BiT) and Vision Transformers, incrcasing the model size improves accuracy in both non-private and private settings. We find this to be the case regardless of whether we train the full model or just the last layer. Perhaps even more encouraging the fact that the gap between the performance with privacy and best numbers without privacy also docrcases as the model size increases. For instance, the gap for VYT-L/16 (last layer tuning) between the best non con-private and private is 7.5 percentage points and for VIT-H/14 in the same setting is 6.2 percentage points.
7

---

Published in Transactions on Machine Learning Research (01/2023)
We also present results when the pre-training dataset size is increased when train using much larger JFT-4B dataset instead of JFT-300M. Note that in this case we only train for single epoch with JFT-4BB which results in a similar number of pre-training steps if we were to train with JFT-300M for 14 epochs. This means that changes in performance between ViT-H/14 and ViT-H/14-4D can be largely attributed to the diversity of examples seen and not necessarily "more" pre-training. As shown the best column in Table 3 we notice moderate improvement (about half a percentage point) between ViT-H/14 and ViT-H/14-4b but when training with privacy at a much larger batch size we see an even larger improvement (a full percentage point).
Lastly, when comparing full finetuning and just last layer tuning on ViT-H/14-4b sctting, we still see a noticeable gap in the best column (trained with 512 batch size). However, when trained at a much larger batch size of 2 the difference shrinks significantly regardless of whether was trained with privacy or not. This is quite fortuitous since the computational cost of training with privacy is much larger when we finetune the full model vs finctuning just the last layer. In the former case, vanilla DP-SGD finctuning as proposed in Abadi et al. (2016) needs access to the per example gradient of the full model whereas when tuning only the last layer, the per example gradient of just the last layer would suffice. The difference can be even more striking when the models are made larger (Appendix Table 10). Note that when tuning only the last layer we still perform forward pass again every time an example is visited largely due to data augmentation. An alternative is to perform the forward pass for the whole dataset once and cache to features to be trained with DP. This further simplified when data augmentation is turned off.
5 Influence of other hyperparameters
Across our exploration so far, the only hyperparameter we tuned was the learning rate. We did this to limit the scope of our exploration since each added variable would result a multiplicative increase compute.
Outside the context of differential privacy though, good performance on image classification task can depend a lot of other factors, such data augmentation (Perez & Wang 2017 Cubuk et al., 2018), initialization Zhang et al. 2019 Mchta et al. 2021 Brock et al. 2021c), feature/ data normalization [offe & Szegedy 2015 Wu & He 2018), resolution of the input image, training iterations etc. It is possible that when trained with privacy, the optimal choices of these set selting are different. Thus, in this section, we systematically unpack a potentially mysterious bag of choices when training the model with a privacy guarantoc. To do this, we take the best performing model so far i.c. ViT-H/14 4b, and change one hyperparameter al time as shown in Table | Note that since there was no performance difference between training the full model vs last layer tuning on ViT-H/14 4b (Table 3 we only tune the last layer for this study.
(Hanin& & Rolnick
2018
VIT-H/14- 4b
Accuracy
A
No change
81.0
DP-LAMB - DP-Adam
81.0
DP-LAMB + DP-SGD with Momentum
77.2
-3.8
Zero Init - Lecun Normal Init Inception Crop - Center Crop
Random Chance
-81.0
81.2
+0.2
10 epochs + epochs
81.1
+0.1
Clip at + 10
80.1
-0.9
Clip at 1 + 0.1
81.0
Resolution 384 + 512
79.0
-2.0
Resolution 384 + 256
81.4
+0.4
Table 4: Taking the best performing model from the previous sections (VIT-H/14-4b), we systematically study the effect of changing single hyperparameter at time. Combining the positive deltas from this study led to cumulative improvement and used the resulting model to present the numbers in Figure lb
8

---

Published in Transactions on Machine Learning Research (01/2023)
Choice of optimizer. As shown in Table switching from DP-LAMB to DP-Adam did not change at all in this setting, however, switching to DP-SGD with Momentum leads to a noticeable decrease in accuracy. This suggests that the layer-wise adaptation of the learning rate, as suggested by LAMB, is much less crucial than the contribution made by the Adam update. This is perhaps not surprising since we are only learning the last layer and the layerwise adaptivity provided by LAMB can be subsumed in the learning rate, which we tune.
Initialization. From our results, the choice of initialization seems be a crucial hyperparameter. Switching the last layer from Zero Initialization to Lecun Normal (default in Flax for dense layers) resulted in a complete lack of progress and random chance performance the test set when trained privately. In the non non- non-private case, Zero initialization has a special property that the learning rate used in the first step can be decrcased arbitrarily (up to precision limits) since it only affects how much the loss decreases and not the accuracy (since we use sigmoid cross-entropy loss). This is useful when training privately since the update is artificially made noisy for the privacy guarantee. this case, the magnitude of the update can be controlled using the learning rate without changing the potential accuracy obtained by the original gradient before the noise was added. This argument very similar in spirit to the one made by Bu et. al. (2021) who show that noise multiplier a vanishes and doesn't affect the convergence in the gradient flow regime. Note that there is nothing special about initializing at absolute zero. As long the initialization is small enough to allow setting small learning in the first step, we find, also leads to similar results as shown in Figure 2c
Number of epochs. Since every visitation of the dataset affects the privacy budget, training for less number of epochs requires a lower noise multiplier a for the samne values of (e, Typically, the advantage with less noise is trumped by the gain in performance by training longer (Kurakin et al. 2022 De et al. 2022p p However, observe that training for single epoch performs slightly better than training for 10, as shown in Table
82.0
80
01.5
Top-1 Accuracy
Top-1 Accuracy
81.0
Top-1 Accuracy
60
40
80.0
65
20
Adam / LAMB Momentum
79.0
10
10-4
10
10"
10
6/16
U16
H14
H/14-4b
224
256
364
512
Init stddev
(a) Adam/
LAMB vs SGD
(b) Input Resolution
(c) Last layer initialization siddev
Figure We consider ViT-H/14-4b model in the single step, full batch setting where only the last layer is finetuned and plot more fine-grained relationship between a few important hyperparameters and test accuracy on ImageNet. (a) LAMB outperforms SGD with Momentum in non-private setting when only the last layer is finetuned. Note that in the first step since the norm of the weights is zero, LAMB is equivalent to Adam. (b) We find non-trivial relationship between input resolution and large batch training where sweet spot of 256x256 image resolution performs best. (c) Increase in scale of initialization of the last layer degrades lest accuracy when trained with DP. With big enough initialization, Lest accuracy degenerates to random chance performance. Note that the phenomenon observed in (a) and (b) can be attributed to the use of large batch sizes and not necessarily DP but (c) more tiod the use of DP, as larger initializations are actually standard without DP.
5.1 Extreme case of full batch and single step update
Since we observed best results when the model was trained for a single epoch, we consider an extreme case where we finctune the model for a single step the full batch setting. We also combine all the positive changes from Table g namely 1) replacing Inception Crop with Central Crop and 2) changing the resolution of the input image from 384 to 256. As shown in Figure 1b as we hoped, these changes provide improvements
9

---

Published in Transactions on Machine Learning Research (01/2023)
Epsilon
Finetuning Dataset
Non-Private
0.5
1.0
2.0
4.0
8.0
CIFAR-10
95.2
95.1
95.1
95.1
95.1
95.2
CIFAR-100
79.9
78.3
79.5
79.7
79.7
79.7
ImageNet-Ik 73.8
19.0
39.1
54.2
63.3
68.3
Table 5: Comparison of Top-1 test set accuracics on when pre-training ViT-B/16 model on ImageNet-21k
dataset. All other details are present in the Appendix Section F
almost orthogonal to each other, through which we obtain SOTA results on ImageNet across all values of tried. Finally, we also observe consistent improvements from just the scale of pre-training in Figure [a
5.2 Pretraining with ImageNet-224 21k
In this section, to test our own recommendations on other tasks, we explore using ImageNet-22k as our pre-training dataset. As shown in Table 5 even when pretrained on much smaller ImageNet-21k dataset, our recommendations lead to beating previous state of the art of 47.9% accuracy on ImageNet- Kurakin et al, (2022) with only epoch of finetuning on last layer features. For CIFAR-10, (Klause et al. 2022) oblain an impressive 82.5% top-1 accuracy for = 8 without leveraging transfer learning. With private finetuning, we improve this result to 95.2% at == 8. Finally, at stringent epsilon values like 1, our results outperform best results of 94.7% on CIFAR-10 and 70.3% on CIFAR-100 from concurrent. work of De el al. 2022 even when additional data is used (i.e. employing transfer learning).
6 Conclusion
We demonstrate that large-scale pretraining on public dataset is an effective strategy for obtaining good results when finetuned privately. Moreover, scaling both model size and pre-training dataset improves performance of the private model and closes the gap between the accuracy obtained in the non-private setting. We also find that gains from transfer learning can be further amplified by switching to an optimizer that works well with large batch sizes, LAMB in our case. In addition, our exploration allowed us to obtain a significant improvement
existing results training ImageNet privately wide range of by finetuning just the last. layer for only 1 epoch (1 step in full batch setting shown in Figure lbp. This significantly reduces the computational cost of training with privacy. Additionally, recent low-rank update techniques other than last layer finetuning can also be leveraged for effective DP training (Yu et al. 2021). We leave this interesting direction to future work. Finally, similar to the baselines, we note that our privacy accounting do not include any cost of hyperparameter tuning, which is another direction of interesting subsequent exploration (Papernot) & Steinke 2022)
Bibliography
Martín Abadi, Andy Chu, Tan J. Goodfellow, H. Brendan McMahan, Tly Mironov, Kunal Talwar, and Zhang. Deep learning with differential privacy. In Proc. of the 2016 ACM SIGSAC Conf. on Computer and Communications Security (C))'16), pp. 308-318, 2016.
Galen Andrew, Om Thakkar, Hugh Brendan McMahan, and Swaroop Ramaswamy. Differentially private learning with adaptive elipping. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, 2021. URL
RUQ12WZRB
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Mamurangsi. Large-scale differentially private BERT. CoRR, abs/2108.01624, 2021a. URL httptstt//aatii.....// www.aa/wwww.ggg/a///...... 01624
10

---

Published in Transactions on Machine Learning Research (01/2023)
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private BERT. CoRR, abs/2108.01624, 2021b.
Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar. Private adaptive gradient methods for convex optimization. Tn International Conference on Machine Learning, pp. 383 392. PMLR, 2021.
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversarics, 2022.
Racf Bassily, Adam Smith, and Abhradeop Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proc. of the 2014 IEEE 55th Annual Symp. on Foundations of Compuler Science (FOCS), pp. 464 143, 2014.
Lucas Beyer, Olivier J. Il'enaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? ArXiv, abs/2006.07/ 07159, 2020.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne. JAX: composable transformations of Python+Numlly programs, 2018. URL
com com/google/jax
Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal propagation to close the performance gap in unnormalized resnets. In International Conference on Learning Representations, 2021a. URL https://openrevieet wtt.://oooommmddd/333iii---
Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization, 2021b.
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High peeghperffrmacce large-scale image recognition without normalization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1059-1071. PMLR, 18 24 Jul 2021c. URL https://proceeddngs mml. https//v3//bbbockk20 html
Zhiqi Bu, Hua Wang, and Qi Long. On the convergence and calibration of deep learning with differential privacy, 2021.
Zhiqi Bu, Jialin Mao, and Shiyun Xu. Scalable and efficient training of large convolutional neural networks with differential privacy, 2022.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In Proceedings of the 28th USENIX Conference on Security Symposéum, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Merbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security, 2021.
Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 10(MMr) 2011.
Christopher A. Choquette-Choo, Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only
membership inference attacks, 2020.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data, 2018.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated
data augmentation with a reduced search space. 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jun 2020. doi: 111.0////opr/3000022200000000 URL
http://dx. doi. http:///a.......r////0//00//0/00 020.:/00000000222200000...000
11

---

Published in Transactions on Machine Learning Research (01/2023)
Soham De, Leonard Berrada, Jamie Hayes, Samuel L. Smith, and Borja Balle. Unlocking high-aceuracy differentially private image classification through scale, 2022. URL https:t/asxxxi....... .rg. ha///bbs/2220 13650
Mostafa Dehghani, Alexey Gritsenko, Amurag Arnab, Matthias Minderer, and Yi Tay. Scenic: A JAX library for computer vision rescarch and boyond. arXic preprint arXu:2110.11400; 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pottern recognition, pp. 248-25-- Icce, 2009.
Friedrich Dormann, Osvald Frisk, Lars Norvang Andersen, and Christian Fischer Pedersen. Not all noise is accounted equally: How differcntially private learning benefits from large sampling rates. 2021 IEEE 31st International Workshop on Machine Leurning for Signal Processing (MLSP), Oct 2021. doi: 10.1109/ mlsp52302. 9596307. URL http:://x..deee www.ddi..gg..................... 10. 1ttp://0000000m1m5552222222 9599599
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Hcigold, Sylvain Golly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL https:// wtt.:/tto.oooo//ddiooooaaaa...
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology EUROCRYPT, pp. 486 503, 2006a.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proc. of the Third Conf. on Theory of Cryptography (TCC), pp. 265 225, 2006b. URL http://áx: /oo..xxxoiioo..///ooo org/10. 0ttg/10110111 1007/ 10000771111188..00 14
Ülfar Erlingsson, Vitaly Feldman, Nly Mironov, Ananth Raghunathan, Kunal Talwar, and Abhradeep Thakurta. Amplification by shuffling: From local to central differential privacy via anonymity. In Timothy M. Chan (ed.), Proceedings of the Thirlieth Annual A CM-STAM Symposium on Discrete Algorithoms, SODA 2019, San Diego, California, USA, January 6-9, 2019, pp. 2468 2479. SIAM, 2019. doi: 10.1137/1. 9781611975482.151. URL httts:://ooll btttp::/ddoi.rggg/ org/10. 1137/1 978111191111111111 117 151
Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in linear time. In Proc. of the Fifty-Second ACM Symp. on Theory of Computing (STOC'20), 2020a.
Vitaly Feldman, Audra McMillan, and Kunal Talwar. Hiding among the clones: A simple and nearly optimal analysis of privacy amplification by shuffling, arXiv preprint arXiv:2012.1280 2020b.
Roy Frostig, Matthew Johnson, and Chris Leary. Compiling machine lcarning programs via high-level tracing. 2018. URL
http::///o......oeorraao/a////// pdd]
lan J. Goodfellow. Efficient per-example gradient computations. ArXiv, abs/ Abs/1510.00199, 2015.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming Hc. Accurate, large minibatch sgd: Training imagenet in hour, 2017.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Compuler Vision - ECCV 2016, 2016.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, rend Rondepierre, Andreas Steiner, and Marc van Zeo. Flax: A neural network library and ecosystem for JAX, 2020. URL http://github.ccom
google/flax
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks, 2018.
12

---

Published in Transactions on Machine Learning Research (01/2023)
Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell, Alon Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, and Yossi Matias. Learning and evaluating a differentially private pre-trained language model. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1178-1189, Punta Cana, Dominican Republic, 2021. URL https://acllntthllogg wwwpogg........o.... org/202/ 2021 Findings-emmlp.002
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.
Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam D. Smith. What can we learn privately? In 49th Annual IEEE Symp. on Foundations of Computer Science (FOCS), pp. 531-540, 2008.
Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression. In Conference on Leorning Theory, pp. 25-1, 2012.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Helena Klause, Alexander Ziller, Daniel Rueckert, Kerstin Hammernik, and Georgios Kaissis. Differentially private training of residual networks with scale normalisation, 2022.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. Lecture Notes in Computer Science, pp. 491 507, 2020. ISSN 1611-3349. doi: 111007/978-3-000-5585 29. URL htttp:://dx..oi.org///
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks, 2014.
Alexey Kurakin, Shuang Song, Steve Chien, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at imagenct scale with differential privacy, 2022.
Mu Li, Tong Zhang, Yuqiang Chen, and Alexander J. Smola. Efficient batc- training for stochastic optimization. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, pp. 661 061-670, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781 9784450329569. doi: 10.1145/222 0044445563333.2200 2623612.
URL /tctps://do..ooo 11wp/////...15//223333393 2623612
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations, 2022. URL
Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. MI-doctor: Holistic risk assessment of inference attacks against machine learning models, 2021.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2017.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint xrxiv:1710 170.00663,, 2017.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, 2018. URL https
//ooenreviee.nee//oorr www.:neeeoooumiiddB/Oooo 120b
Harsh Mehta, Ashok Cutkosky, and Behnam Neyshabur. Extreme memorization via scale of initialization. Tn International Conference on Learning Representations, 2021. URL
Z4R1 bRLLO
Iyy Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Synposium (CSF), pp. 263 275. IEFE, 2017.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Ila Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism, 2019.
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. Adversary instanti- ation: Lower bounds for differentially private machine learning. 2021 IEEE Symposium on Security and Privecy (SP), May 2021. doi: 1.11099/p40001.0022 22240000..221000663 URL http://dx..di..0rr/10110000
2021. 00069
Arvind Neelakantan, Luke Vilnis, Quoe V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks, 2015.
Nicolas Papernot and Thomas Steinke. Hyperparameter tuning with renyi differential privacy. In International Conference on Learning Representations, 2022. URL httpp:://pperreiiee.nnttt http://wwwweeeorrrrrroooooaaaaa.
Nicolas Papernot, Shuang Song, Nya Mironov, Ananth Raghunathan, Kunal Talwar, and Ülfrr Erlingsson. Scalable private learning with pate. ar preprint arXiv:1802.0890: 2018.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning, 2017.
Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank J Reddi, and Sanjiv Kumar. Adaclip: Adaptive clipping for private sgd. arXie preprint orXiu: NKKi:100.0000 998.00643,, 2019.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization, 2019.
M. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation learning: the rprop algorithm. In IEFE International Conference on Neural Networks, pp. 586-591 vol.1, 1993. doi: 10.1109/
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3 18, 2017.
Samuel L. Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent, 2020.
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 245 248. IEEE, 2013.
Shuang Song, Thomas Steinke, Om Thakkar, and Abhradeep Thakurta. Evading the curse of dimensionality in unconstrained private glms. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 2638 2646. PMLR, 13 15 Apr 2021. URL v130/song21a.htmm html
wwapeeediinnmm....d....m.mmeem.e mlr. press/
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer How to train your vit? data, augmentation, and regularization in vision transformers, 2021.
Pranav Subramani, Nicholas Vadivelu, and Gautam Kamath. Enabling fast differentially private sgd via just-in-lime compilation and vectorization. ar preprind arXiv:2010. 09066, 2020.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness
of data in deep learning era. 2017 IEEF International Conference on Computer Vision (ICCV), Oct 2017. doi: 10.1109//ccv.2017..9 URL http://dx... http:/xxxxioo.... org/10. 1109/1ccv.2017 2017.97
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanct, Scott E. Rood, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, 2015 IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 1-9, 2015.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Kunal Talwar, Abhradeep Guha Thakurta, and Li Zhang. Nearly optimal private lasso. In C. Cortes,
N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advonces in Newral Information Processing Systems, volume 28, Curran Associates, Inc., 2015. URL httpe::/prooeeddngs.nnne .df
Nla Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-mixer: An all-MLP architecture for vision. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (ods.), Advances in Neural Information Processing Systems, 2021. URL https://openreviev net/forum? htt://orumiidEEIK22e0 P
Florian Tramèr and Dan Boneh. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations, 2021. URL https://openrvvioo.coo hettooouiiiieeto//ooiiiiiii YTWGvPFOQD-
Yu-Xiang Wang, Borja Balle, and Shiva Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. Journal of Privacy and Confidentiaaiity, 10(2), Jun 2020. ISSN 2575-8527, doi:
1129002/jjpc722. URL htthtttp:///////t /ttp://xxx.oo.... doi 2mgg11..9012/jppp 723
Xi Wu, Fengan Li, Arum Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey F. Naughton. Bolt-on differential privacy for scalable stochastic gradient descent-based analytics, 2016.
Yuxin Wu and Kaiming He. Group normalization. Lecture Notes in Computer Science pp. 3-19, 2018. ISSN 1611-3349. doi: 0010077/97--3-33--02200-- 1. URL. http:://aa.doi..rr/11..... htt.:////.....00//0------0000000
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes, 2019.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huscyin A. Inan, Gantam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2017.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normaliza- tion, 2019.
Yuqing Zhu and Yu-Xiang Wang. Poission subsampled rényi differential privacy. In International Conference on Machine Learning, pp. 7634-7642. PMLR, 2019.
15

---

Published in Transactions on Machine Learning Research (01/2023)
A Algorithmic details
We present below a generalized version of DP-SGD where the gradients get processed in the traditional DP-SGD fashion and are then passed to a first order optimizer as an input. This lets us instantiate DP versions of well known optimizers like SGD, Momentum, Adam and LAMB. We prepend the optimizer's name with DP denote that the gradients were first processed as shown in Algorithm [ and then passed to the said optimizer.
Algorithm Generalized First Order Differentially Private Algorithm
Require: Data set = ((11, (Fn, yn} with (ri, vi) E DD loss function: é TR x D -> IR, first order optimizer Opt, clipping norm: C, number of iterations: T, noise multiplier: o
Randomly initialize 0g-
for t O.....
do
Select randomly without replacement a mini-batch of examples B, S
4: 9t + (B.))BB clip (Ve((); (2, where clip(v) = min C cl Tolla
5: Je 9t +N (0, (oC)²)
0t-1 f -iigle step of first order optimization with gradient Opt(ğ)
end for
return
T Σ
0: or 0r.
B Privacy Analysis
The privacy parameters (c, 5) are functions of C, 0, [Be], [D), and the total number of iterations T. DP-SGD algorithm involves setting the right clipping norm C and the noise multiplier a given a privacy budget, batch and dataset size. The (c, guarantee is computed by analysis of the Gaussian Mechanism with privacy
amplification by subsampling and composition across across iterations (asiviswanathan et al. 2008 Bassily
ct al.
2014 Abadi et al. 2016 Mironov
2017 McMahan et al.] 2017 Mironov et al.) 2019 Erlingsson et al.
2019
Zhu & Wang 2019 Feldman et. al., 2020b Wang et al.) 2020 Our implementat ion relies on Tensorflow
Privacy
codebase for conversion (s, 6) and clipping norm C' to/from noise multiplier a.
To put the epsilon-delta values in context, our best model obtains 81.7% accuracy for E E [4, 10). Privacy guarantee for x 4 (Figure [a] in fact satisfy much stronger property of xCDP <=1 (0.154 for E 4)
which is by now an industry standard.
C Training details
The core bottleneck of DP-SGD is the computational cost of computing per-example gradients. A naive implementation of per-example gradient calculation can lead to a dramatic reduction of throughput and an increase memory usage (proportional to the batch size) compared to non-private training. Inspired by Subramani et al. (2020), we conduct all our experiments in Jax (Bradbury et al. 2018 Frostig al. (0188) is framework that leverages just-in-time compilation using XLAP and does nnto-voctorization of the backward pass. We leverage this functionality Uhroughout our experiments. Finally, we conduct our experiments on TPUv4 architecture.
We conduct all our experiment using Scenic library (Dehghani et al.) 2021 for high quality reproducible implementations of both ResNet (BiT) and Vision Transformers. Scenie, in turn, uses Flax (Heek et al. 2020 for may of the layer definitions. We will also open-source our implementation of DP-SGD and DP-LAMB which includes vectorized per-example gradient clipping in a distributed training setup for further auditing by the research community. For the privacy accounting, rely on the default Rényi accountant implementation already open-sourced as part of Tensorflow Privacy library.
https://gitthbb.. http://sttt.ccco///oooo/t//////t http:://www.tennolllooooooo
16

---

Published in Transactions on Machine Learning Research (01/2023)
C.1 Model details
We use 2 compelitive model families for our experiments. 1) ResNet (BiT) and 2) Vision Translormer.
ResNet (BiT). Originally proposed in (Kolesnikov et al. 2020), it modifies the original ResNet architecture
(He et al. 2016 by replacing Batch Normalization with Group Normalization (Wu & Hc 2018 and additionally using Weight. Standardization for the convolutional layers (Qiao et al. 2019
Vision Transformers. We use the exact architecture and notation proposed by (Dosovitskiy et al. 2021 Vision Transformer creates 2d imput patches of images and applies the Transformer backbone typically used NLP tasks. In addition to the models trained in the original paper, we make one addition of ViT-H/14-AA denoting ViT-H/14 size model pretrained on the larger JFT-4B dataset.
D Pretraining details with JFT
Datasets. We use 2 variants JFT datasets for our pre-training. JFT-300M (Sun et al. 2017) consists of 18k classes and 303M high-rcsolutinn images, while JFT-4B consists of 29.5k classes with 4B images. Note that really do mean JFT-4B, which is larger version of JFT-3B used in Zhhi et al. 20021)
Deduplication. In order to both not inflate our results and break privacy guarantee offered by finetuning privately on ImageNet, we extend the deduplication process proposed by Kolesnikov et al. (2020) and deduplicate both JFT-300M and JFT-4B with respect to all splits of ImageNet. We use a model based deduplication system which removes both exact and near-duplicates across common image transformation like crop, shift, resize etc.
Hyperparameters. At the pre-training stage, we stick with the common practice of employing Adam optimizer (even for ResNet) with $1 0.9 and 32 0.999, with batch size of 4096 and high weight docay of 0.1. We also use linear learning rate warmuu until 10k steps and linearly decay it until the end. All our models are pre-trainn trained with 224x224-sized JFT images.
Model
Epochs
Base LR
TPU v4 hours
Model size
VIT-B/16
7
10-4
0.6k
86M
VIT-L/16
14
4 4-10-
4k
307M
VIT-H/14
14
3 10-1
15k
632M
VIT-H/14 4B
1
3 1-10
15k
632M
ResNet-50x4 (BiT)
7
1-10-3
8k
384M
1eaNet-15224 (BiT)
7
6-
11.4k
937M
Table 6: Pre-training hyperparams. All models are trained on deduped JFT-300M with the exception of VIT-H/14-44B, which was trained on much larger JFT-4B dataset but for roughly the same number of steps VIT-H/14. We used batch size of 4096, learning rate warmup of 10k steps and then linear decay. Additionally, set dropout rate to 0.0, clip global norm to and weight decay to 0.1. We use images of resolution 224x224.
Training details. All our models were pre-trained using TPUv4 hardware with exact amounts depending the model. For reference, smallest model VTT-B/16 took around 600 TPUv4 hours to pre-train and our biggest model ViT-H/14-4b took 25k TPUv4 hours.
E Finetuning details
We finetune on ImageNet train split and present the Top-1 accuracies we obtain from the official test split. Unless specified otherwisc, we used images of input resolution 384x384 which inception cropped (Szegedy et al. 2015) from a resolution of 448x448. In addition we perform horizontal flipping as data augmentation.
These pre-processing steps exactly follow Kolesnikov et al. (2020) Dosovitskiy et al. (2011)
17

---

Published in Transactions on Machine Learning Research (01/2023)
E.1 SGD with Momentum hyperparameters
Hyper parameters
Range
Learning rate
0.03, 0.06, 0.1, 0.4, 1.6, 6.4, 25.6, 102.4
Table 7: Finetuning hyperparams. Models trained for 10 epochs, with 0.25 epochs for warmup and cosine decay, no weight decay, no dropout and grad clipping global norm 1. Similar to previous art, we fine-tune at higher resolution of 384. When training the models with DP, we replace the global clipping with per example clipping norm of 1.0
E.2 LAMB hyperparameters
Hyper parameters
Range
Learning rate
0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0
Pre-trained layer LR multiplier a
1.0, 10-1, 10-2, 10 10 10-4, 10-5
Table 8: Finetuning hyperparams. Models are trained for 10 epochs, with 0.25 epochs for warmup and linear decay, no weight decay, no dropout and grad clipping at global norm 1. Similar to previous art, we fine-tune at a higher resolution of 384. When training the models with DP, we replace the global clipping with per example clipping norm of 1.0. In our earlier results we found that sometimes finetuning the full model performed much worse than finetuning just the last layer with privacy. We added an additional hyperparameter which multiply the global learning rate with for all layers except the last. The intuition is that since all layers except last have been pre-trained, the optimal choice of learning rate may be different. for them compared to the last layer, which we start from scratch.
E.3 Hyperparameters to Reproduce Figure 1b
In this setting, we only train the last layer for single epoch in full batch setting. This amounts to single update of the optimizer. We emphasize that we use the full dataset of around 1.28M training set as a single batch. This is different from using batch size of 220 for 1 step, which leaves a subset of the images out. We initialize the last layer with zero and biases with -10 (default in Scenic). Additionally, we also changed the input resolution to 256x256 which is central cropped from an image of resolution 384x384. Finally, since we only make one update, we sweep over constant learning rate values [10-4, 10-3) and make an update with DP-LAMB optimizer. Note that, for 1 step updates and zero initialization, LAMB is equivalent to Adam, thus expect AP-Adam to produce similar results.
Adam in single step full batch setting. It is important to note that Adam in single step, full batch setting in fact reduces to signSGD where adam update V0+s ~ sign(g)- Sign based methods are indeed 9
shown be quite effective (Riedmiller Braun, 1993) and in our case even surpass vanilla SGD. Since LAMB is equivalent to Adam when finetuning just one layer (with learning rate retuned), LAMB also reduces to signSGD in this restricted setting.
Privacy accounting of hyperparameter tuning We additionally note that we do not include any cost. related to hyperparmeter tuning our privacy accounting and budget. This in line with the setup of the basclines we consider. Finally, since learning rate is the only hyperparamter we tune, least in the last layer single step setting, we have found that results are quite robust to the exact value of the learning rate as long it is low enough to not diverge.
18

---

Published in Transactions on Machine Learning Research (01/2023)
F Pretraining with ImageNet21k
Dataset. ImageNet-21k is a superset of ImageNet-k with 21k classes and 14M images (Deng et al. 2009 Similar to before, in order both not inflate our results and break privacy guarantee, we extend the deduplication process proposed by Kolesnikov al. 2020)) and deduplicate ImageNet-2lk with respect to all splits of ImageNet-1k, CIFAR-10 and CIFAR-100.
Pretraining Hyperparameters. At the pre-training stage, stick with the common practice of employing Adam optimizer (even for ResNet) with 31 = 0.9 and 32 = 0.999, with a batch size of 4096. Unlike pre- training with JFT datasct, we follow recommondations from Steiner et al. (2021) to use AugReg stratogy where we lower the weight decay 0.0001 and don't 't use dropout but instead use data augmentation strategy called medium1 which combines Mixup with = 0.2 (Zhang al.) 2017) and RandAugment with ( (= 15 and 2 (Cubuk et al. 2020). We also linear learning rate warmup until 10k steps and linearly decay it until the end. Our model is pre-trained with 224x224-sizer sized images.
Model
Epochs
Base LR
TPU v4 hours
ViT-B/16
300
10-3
2.7k
Table Pre-training hyperparams. We used batch size of 4096, learning rate warmup of 10k steps and then linear decay, Additionally, we set dropout rate 0.0, clip global norm to and weight decay to 0.0001. We use images of resolution 224x224.
F.1 Hyperparameters to Reproduce Table
Similar to Figure lb we only train the last layer for single epoch in full batch setting. This amounts to just a single step of optimization. Similar to before, we initialize the last layer with zero and biases with -10 (default in Scenic). We also changed the input resolution to 256x256 which is central cropped from an image of resolution 384x384. This may look a little unusual first for CIFAR-10 and CIFAR-100 since the original resolution of the images is 32x32. But we first upsample them 384x384 and then central crop them. We found that using higher resolution images made big difference in performance (even in non-private setting), especially when using features from pre-trained model. Finally, since we only make one update, we sweep over constant learning rate values E [10-6, 10 10-5, 4,, 10-3] and make an update with DP-ADAM optimizer as before.
Privacy accounting hyperparameter tuning We additionally note that we do not include any cost. related to hyperparmeter tuning our privacy accounting and budget. This in line with the setup of the baselines we consider. Finally, since learning rate is the only hyperparamter tune, at least in the last layer single step setting, have found that results quite robust to the exact value of the learning rate long as it is low enough to not diverge (10-3 in this case).
G Regarding Reproducibility
In order to aid in reproducability of our work, we provide results with large open pre-training dataset called ImageNet-21k. In addition, we provide exact hyperparameters and other details in the Appendix. We will also release checkpoints and features from models pre-trained on ImageNet-21k, Finally, in order help reproduce our results even further, will also release our code.
19

---

Published in Transactions on Machine Learning Research (01/2023)
H Supplementary Results
H.1 Throughput comparison
Tuning type
Model
Full
Last layer
Speed Up
ViT-B/16
170
900
5.3x
ViT-L/16
25
235
9.4x
ViT-H/14
12
100
8.3x
ResNet50x4 (BiT)
25
225
9x
ResNet152x4 (BiT)
3
90
30x
Table 10:
Throughput [ang/sec/core) comparison of DP-LAMB on various models trained on Imagenet. All models were trained using 64 TPUv4 cores.
H.2 More stringent delta
We conducted all our experiments with 8 = 1e-6 in order for fair comparison with Kurakin et al. (2022) but ImageNet dataset size is close to 1.28M. Here we re-did sweep over on H/14-4b using the same
hyperparameters used to obtain results Figure Ib but with set to 8c-7. Note that De et al. (2022) also set to 8e-7. As shown Figure 3 we find that the change effective noise multiplier is small enough that. our results don't change all across both values of 6.
84
81.5
81.7
81.7
81.7
81.7
82
81.1
Top-1 Accuracy
79.4
80
78
75.6
76
74
0.25
0.5
1
4
6
8
10
E
Figure 3: Comparison of Top-1 accuracies on ImageNet when varying privacy parameter when å set to Se-7.
H.3 ReaL labels
We also provide results when evaluated ImageNet-Rent Real.
Beyer al. 2020 labels in Figure 4
20

---

Published in Transactions on Machine Learning Research (01/2023)
89
87.2
87.3
87.3
87.3
87.4
88
86.8
Top-1 Accuracy
87
85.5
86
85
84
83
81.9
82
0.25
10
Figure 4: Comparison Top-1 accuracies on ImageNet-Real. labels [Beyer al. 2020 when varying privacy
parameter
With privacy
Model
Fineluning lype
215
216
ViT-B/16
Full
45.23
58.6
Table 11: We complement results from Table with full finetuning of ViT-B/16 on smaller batch sizes well. As shown in Table I
we obtain best results with batch size 21?
H.4 Smaller batch sizes
Since get the highest accuracy with full finetuning of VIT-B/16 in Table Dat the lower end of our batch size range, we also try batch sizes even lower than that to verify if that is indeed the maximum we could achieve. We verify this as shown in Table 11
21


Reviews:
[{"review_id": "EvzBn0Y9LS", "reviewer": "Reviewer_UVNH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper has a very good experimental setup and set of results that are very convincing.\n- A thorough description of the DP-SGD\n\n\nWeaknesses:\n- More discussion on private vs public (see \"requested changes\" section)\n- Emphasize other domains, e.g. text, as possible other domains where this method might be relevant\n- Also mention other settings, e.g. federated learning, as possible benefits for training large models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper describes large-scale experiments with transfer learning and DP-SGD. The results are very important for our community as they demonstrate that at large-scale the model performance can reach non-private baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think that the paper might get stronger with a discussion on the difference between private and public data. \n\nMy biggest concern is the experimental setup: a public ImageNet dataset is considered **private** and the private JFT dataset is considered **public**. The described approach sounds to me practical, however I wonder what is the meaning of the \"public data\" that is not publicly available. Is it reasonable to expect that datasets like JFT will become publicly available or due to sensitivity of large sets of data it might never happen? I also might suspect that other domains, e.g. text, might have more data publicly available.\n\nI hope this request does not sound too abstract, however as you describe benefits of using public data for private training, I feel like it's important to dedicate a section on \"defining\" the public and private data. For example, using privacy frameworks like contextual integrity to emphasize that even publicly data might still not allow to train models for certain cases or be released in an aggregated form. ", "broader_impact_concerns": "I would try to still say that public data does not always mean that the model can be created or that data can shared. I think your examples with JFT emphasizes this distinction, which is a very good argument for me. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6XXzhg9Whi", "reviewer": "Reviewer_GzUe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. I really liked how the authors made sure to deduplicate the pre-training data from the fine-tuning data, that is a very important step which is missing in many similar related work, and makes the conclusions much more reliable.\n\n2. The paper's narrative is very easy to follow, and I enjoyed how it answered every question that came to my mind in the next paragraph. The results are thoroughly explained and justified.\n\n3. Studying the effect of the optimizer on performance, and also analyzing the computational overheads (run-time) and achieving faster run time are aspects I had not seen studied much before in similar related work. \n\n\nWeakness:\n\n1. Although there are aspects to this paper that differentiate it from prior work, the main idea is still using pre-training which has been explored before multiple times. However, I don't think this is really that much of a weakness as the findings and insights do have novelty. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the application of transfer learning through pre-training for image classification, in a private setup. The paper shows that privately  fine-tuning pre-trained models helps close the gap between private and non-private setups. Based on their empirical analysis, the authors make recommendations on what best practices are when using DP-SGD for image classification. They show that using the right optimizer and initialization for the classification layer are both very important in achieving desirable privacy-utility trade-off. They also show that through their recommendations, the training time of models can be severely decreased. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main question/potential request for more experiments is regarding the initialization of the last layer, for fine-tuning. right now zero initialization and Lacuna initialization are studied, and there seems to be a huge gap between their performances. I wonder if the authors tried other initializations? Or even maybe a way to initialize with transfer learning and using parameters from either other layers or other models?\n\nAnother question I had was regarding Tables 3 and 4: The performance of the best setup (the final row in table 3) which is the highest scaled one seems to be invariant of optimizer choice (according to table 4). Do the authors think that this might not be the case if full-fine tuning is performed? Or could it be that the large scale of the data/training kind of overshadows the effect of the optimizer? (I mainly mean between Adam and LAMB). \n\n\nSome minor comments/suggestions:\n\n1. Page 1: Introduction, end of line 3, differentially privacy should probably be differential privacy\n2. Page 2: Introduction, line 5, might significant increase -> might significantly increase\n3. Page 8: end of last line \"that that\" -> that\n\n\n", "broader_impact_concerns": "The paper facilitates wider deployment of private training algorithms which in itself can have many positive impacts. However, my only ethical concern is with the training time and hardware, as it seems training takes in the order of thousands of hours, using TPU v4s (based on Appendix D). Given how pre-training is what actually causes the improvements shown in the paper, I think it would be beneficial to the community if the pre-trained models are made publicly available to help accessibility, and alleviate the need for re-training them over and over by different parties. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CkgbaB8wd4", "reviewer": "Reviewer_paed", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The empirical results shown in the paper are very strong and improve over the state-of-the-art significantly. While the work does not present theoretical reasoning for the proposed improvements for the learning procedure, I find this type of empirical work also valuable for building actually functioning DP algorithms. Also, I think the topic is relevant and of high interest to the TMLR community. However, I do have some serious concerns about the method that need to be addressed before the empirical performance can actually be qualified.\n\nIf I do understand the paper correctly, the results are based on fine-tuning the model for only a very iterations. The ablation study in Table 4 shows that the performance is actually the best if we train only for a single epoch. Moreover, the authors suggest that the extreme case of training only a _single step_ using all the data as the \"batch\" provides the best performance. Now, if we use the Adam optimizer from Kingma et al. (I assume that the DP-Adam is just a DP variant of this where the noise is added to the clipped gradients prior to learning rate adaptation), then the first gradient update step should actually correspond to a constant update. This is easy to see from the description of the Adam in the original paper. The moment parameters of Adam are initialized as $m_0=0$, $v_0=0$ and the first update becomes:\n\\begin{align}\n   &m_1 = (1-\\beta_1) g \\\\\\\\\n   &v_1 = (1-\\beta_2) g^2 \\\\\\\\\n   &\\hat{m}_1 = m_1 / (1-\\beta_1) = g \\\\\\\\\n   &\\hat{v}_1 = v_1 / (1-\\beta_2) = g^2 \\\\\\\\\n   &\\theta_1 = \\theta_0 - \\alpha \\hat{m}_1 / (\\sqrt{\\hat{v}_1} + \\epsilon) = \\theta_0  - \\alpha g/(g + \\epsilon)\n\\end{align}\nSo if the $\\epsilon$ (note that this is not the privacy parameter) in the denominator is chosen significantly smaller than the gradient (which typically is the case), then the first step in Adam becomes essentially $\\theta_1 =  \\theta_0  - \\alpha$ and the update is independent of the gradient. Therefore the model update happens independently of the data. The results for the CIFAR-10 and CIFAR-100 seem to be almost non-affected by the epsilon, which could support the hypothesis that the gradient step actually does not use the data at all. However, the results for the ImageNet-1k of Table 5 seem to be affected by the privacy parameter $\\epsilon$, which would suggest that the method is actually dependent on the data. Can authors clarify the use of Adam in the single iteration setting? If the DP-Adam is somehow fundamentally different to the Adam, please specify it in the appendix. \n\nBesides the possible problem with single step learning pointed above, I want to point out some other weaknesses of the paper as well. \n- Simply using more data for pre-training seems somewhat unattainable suggestion for many cases. If there is a large public data set available that could be use to pre-train the model, then in what scenario the analyst _wouldn't_ use it? I think this is crucial point to consider for better motivating the suggestion 1. \n\n- Authors should better specify the model architectures, and which models are more complex than other. I didn't find much discussion about this in the paper. For example I don't quite follow the discussion after Table 3. You state that the accuracy gap decreases with larger models. And then you state \"For instance, the gap for ViT-L/16 (last layer tuning) between the best non-private and private case is 7.5 percentage points and for ViT-H/14 in the same setting is 6.2 percentage points.\". So is the ViT-L/16 model smaller than the ViT-H/14?\n\nSome minor comments:\n- Captions of Table 2 and 3 are equivalent. While the Tables might contain overlapping settings, the captions should at least describe the main result of the table and the particular effect studied.\n- Section 5.1: What do you mean by \"orthogonal\" here? Orthogonal compared to what?\n- Figure 1a is not referred in the main text.\n- Caption of Figure 1a says \"across model and pre-training dataset sizes.\", but it remained bit unclear to me what are the data sets and models corresponding to the labels in the x-axis. \n- The appendix says that the code is included but I didn't find it from the openreview system. Maybe it was there, but just didn't show for me.\n\nTypos:\n- \"computeing\"\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an experimental study of large scale transform learning for differentially private image classification. Authors study the effects of model-architecture and optimization methods in DP fine-tuning of a large image classification deep learning model. Based on their experimental results, authors suggest three practices for the DP transfer learning for this particular set of models:\n1. \"Increase pretraining data and model size.\". Authors show in their experiments (Table 3), that using a larger pre-training data set improves the performance of both non-private and private classification. Authors also show that using a more complex model improves the fine-tuning performance for both private and non-private cases.\n2. \"Use large batch sizes in combination with optimizers that can work well with large batch sizes (e.g.\nLAMB).\". Motivated by prior work on the topic, authors experiment with large batch sizes in the DP fine-tuning. The experiments suggest that using a large batch size and training for only few iterations improves the DP performance. Towards the end of the paper authors suggest using only a single gradient update step with the full data set used as the \"batch\".\n3. \"Initialize the last layer to zero (or very small) when doing finetuning with DP.\". In their experiments, authors demonstrate that fine-tuning only the last layer of the classification model provides on par performance to the full fine-tuning when trained under DP and using a large batch size. In their ablation study authors show that initializing the fine-tuning with zeros (the weights of the last layer) provides significant performance increase compared to the random initialization.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I require following changes:\n- The Adam should be better discussed to understand the results of the single step training.\n- The models should be better described in the appendix, including for example a clear description of the model size.\n\nMore minor changes that will improve the paper\n- The captions for the Tables should be more descriptive to convey the main message of the Table.", "broader_impact_concerns": "I have no concerns relating to the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all thought the work was great, and were generally satisfied by the revisions by the authors. I too agree: I think this is a really nice work, which pushes the limits of public pre-training, showing that differentially private fine-tuning can preserve most of the utility. Congratulations to the authors.

For the paper itself, I would "Accept as is." However, as the authors have promised, the code is in the process of being publicly released. The only remaining concern of the reviewers (and myself) is that it has not yet been released. To add some impetus to this process, I am marking the paper as "Accept with minor revision." To satisfy the minor revision, I would like either a) the code to be publicly released and added to the submission, or b) the authors to publicly commit to a timeline for code release (in the case that administrative overhead precludes releasing within this limited timeframe).
Decision: Accept (poster)