SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: The fraction of total privacy budget allotted to Phase II for  $\varepsilon = 1$ . The performance is stable across a wide range of value [0.1, 0.4]. However, skipping either Phase-II or Phase-III leads to suboptimal test accuracy in DP training.
<sup>3</sup>Due to computational constraints we did not calculate error bars for Fig. 6, resulting in some nonconvexity.
Allocating the entire privacy budget to Phase II is competitive for small privacy budgets but provides diminishing returns. This corresponds to the right extreme in Fig 6 ( $\varepsilon_1 / \varepsilon = 1$ ) and is equivalent to doing linear probing on top of extracted features. We follow the training recipe in Panda et al. [54] and report the result of private linear probing in Tab. 4 for CIFAR10 and Tab. 5 for ImageNet [16] respectively ( $\delta = 7.8 \times 10^{-7}$  is by  $\delta = 1 / |D_{train}|$  for ImageNet). We provide the detailed experimental set-up for CIFAR10 in Appendix F and ImageNet in Appendix G.
CIFAR10 results. We present the result of private linear probing on CIFAR10 by including more conservative privacy constraints like  $\varepsilon = 0.03$ . Our private linear probing can achieve  $57.10\%$  at  $\varepsilon = 0.1$ , that is comparable to the result of  $\varepsilon = 1$  in De et al. [15] that fully trains a WRN-16-4. Notably, previous work [61] also trains a neural network on top of the handcrafted features using an untrained ScatterNet [53]. DP-RandP is slightly better than their result at  $\varepsilon = 3$ . We can see that our non-private baseline  $(74.05\%)$  is slightly better than the non-private baseline  $(73.8\%)$  in Tramér and Boneh [61], that shows that our feature extractor is better than the ScatterNet and therefore improves the performance under DP-SGD. We include a detailed comparison to Tramér and Boneh [61] in Sec. 4. However, this private linear probing variant of DP-RandP can only achieve  $74.05\%$  with no noise added, that is lower than the result at  $\varepsilon = 4$  in De et al. [15], that shows that allocating all privacy cost to Phase II is a sub-optimal design choice.
Table 4: Test accuracy  $(\%)$  of our private linear probing and comparison to previous SOTA for private learning on top of extracted features [61] on CIFAR10. Note that, our result is training a linear layer on top of the extracted features. The result of previous SOTA is by training a CNN on top of extracted features. Tramér and Boneh [61] also report the private linear probing result  $67.0\%$  at  $\varepsilon = 3$ .
ε	0.03	0.1	0.2	0.5	1	2	3
SOTA	-	-	-	-	60.3	67.2	69.3
Ours (Std.)	40.64	57.10	60.89	65.10	67.78	69.92	71.08
2.59	0.42	0.26	0.21	0.17	0.09	0.14
Table 5: Test accuracy  $(\%)$  of our private linear probing with additional designs on ImageNet.
ε	1	8
De et al. [15]	-	32.4
Sander et al. [57]	-	39.2
Ours (ViT)	26.54	39.39
(Std.)	0.11	0.03
ImageNet results. We achieve a new SOTA results on ImageNet. We achieve  $39.39\%$  accuracy at  $\varepsilon = 8$  and the previous SOTA [57] is  $39.2\%$  at  $\varepsilon = 8$ . We use a ViT-base [19] model pretrained on Shaders-21k (the model checkpoint is provided by Yu et al. [71]). If we directly do Phase II with extracted features, we can achieve around  $33\%$  accuracy at  $\varepsilon = 8$ , that is comparable to De et al. [15]. The direct linear probing result shows that linear probing (Phase II only after pretrained on synthetic data) is not enough for difficult tasks like ImageNet, and therefore updating full parameters in Phase III is necessary. While training the full ViT model on large datasets like ImageNet needs much computation resources (for example, Sander et al. [57] used 32 A100 GPUs), we could leverage the core concept of DP-RandP and adapt it to a computationally efficient version, i.e., we train a linear layer with additional modifications. We then obtain a new SOTA result of  $39.39\%$  on the ImageNet-1k validation dataset. We now briefly explain the two main modifications of DP-RandP to achieve  $39.39\%$ . We provide a detailed explanation in Appendix G. We emphasize that these modifications are more for computational efficiency; if we had enough compute to do full fine-tuning of the ViT on ImageNet with sufficiently large batch size, our original DP-RandP would still work.
Our first modification is to approximate full fine-tuning by linear probing on larger feature representations that we create by aggregating intermediate representations from the network. This is because each block of vision transformers learns a different representation. However, linear probing only takes the representation from the penultimate layer as input and therefore the final representation may not be sufficient to learn the task. The representation of the input image after each block in the ViT has both a temporal and feature dimension, so we pool over the temporal dimension to gather a feature map of size (4, feature size). We concatenate the feature map of different blocks into a one-dimensional vector. By doing linear probing on these much larger features, we can also take advantage of intermediate representations.
Our second modification is to approximate the work of a LayerNorm or other normalization layer that we would update during full fine-tuning of the entire ViT, by manually normalizing the features.
To do this we first normalize each feature vector to a fixed norm. We next privately estimate the mean over the entire ImageNet feature vector dataset, using the Gaussian mechanism with a small privacy cost, and subtract the private mean from all feature vectors. This is equivalent to doing non-private centering and then adding the same Gaussian noise to the entire dataset. This procedure can be thought of as a one-time approximation to the normalization layer, which is known to speed up training by centering the data.
Within the two modifications of direct linear probing, we improve upon previous SOTA [57] and achieve  $39.39\%$  at  $\varepsilon = 8$ . This method is computationally efficient and one run can be done on a single A100 GPU in a few hours. We also provide the result for a stronger privacy guarantee, i.e.,  $26.54\%$  at  $\varepsilon = 1$ .
Allocating the entire privacy budget to Phase III struggles for small privacy budgets. We report DP-RandP without Phase II on CIFAR10 in Tab. 6 (equals to  $\varepsilon_1 / \varepsilon = 0$  in Fig. 6). The result in Tab. 6 is slightly worse than DP-RandP in Tab. 1 and the utility gap between Tab. 6 and Tab. 1 decreases as  $\varepsilon$  increases, that justifies the importance of Phase II in DP-RandP. Moreover, the result of DP-RandP without Phase II is significantly better than previous SOTA [15, 35, 61], that shows that the feature extractor pretrained on images from random process can capture the image prior.
Table 6: Fully privately training a WRN-16-4 with warm initialization. Test accuracy on CIFAR10.
ε	ε=1	ε=2	ε=3	ε=4	ε=6	ε=8	ε=∞
Accuracy(%)	69.030.23	75.310.28	78.440.19	80.560.12	82.900.10	84.450.09	91.69
A general privacy budget allocation strategy. We have observed that allocating the entire privacy budget to linear probing or full training is suboptimal. We now propose a simple yet effective general strategy to allocate the privacy budget. For small  $\varepsilon$  ( $\varepsilon \ll 1$ ), we set  $\varepsilon_1 = \varepsilon$  to allocate the entire privacy budget to Phase II. This is because allocating the entire privacy budget to linear probing is competitive for small privacy budgets. As  $\varepsilon$  increases, we decrease  $\varepsilon_1 / \varepsilon$ . This is because as  $\varepsilon$  increases, the noise multiplier will decrease. Therefore, it is easier to train the linear probing layer as  $\varepsilon$  increases, and the percentage of total steps allocated to Phase II can be reduced to train a good linear probing layer and we can use the remaining steps for Phase III. Because the closed-form computation of  $\varepsilon$  with numerical privacy loss distribution accounting by Gopi et al. [27] is challenging, we implement this strategy by using a fixed number of steps  $n$  for linear probing in CIFAR10, CIFAR100, DermaMNIST, and increasing the number of steps in Phase III as  $\varepsilon$  increases. We present the privacy allocation  $\varepsilon_1 / \varepsilon$  on CIFAR10 in Appendix E for results in Tab. 1, which validates this strategy.
3.4 In comparison to DP with public data
A major direction in improving the privacy utility trade-off for DP-SGD is by incorporating priors that are learned on real-world public data [68, 2, 45, 52, 50, 49, 54, 56]. These priors are either from small in-distribution data [68, 2, 45, 52] or large scale public data [50, 49, 54, 56].
Table 7: Comparison of DP-RandP with methods using public data. Test accuracy (%) on CIFAR10.
Method	Model	Public Data	ε = 1	ε = 2	ε = 4	ε = 6	ε = 8
Nasr et al. [52]	WRN-16-4	4% ID	72.10	75.10	77.9	80.0	-
Mehta et al. [50]	ViT-B/16	ImageNet	95.10	95.10	95.10	-	95.20
Mehta et al. [49]	ViT-G/16	JFT	98.80	98.80	98.83	-	98.84
Panda et al. [54]	beitv2	ImageNet	99.00	-	-	-	-
DP-RandP	WRN-16-4	Synthetic	72.32	77.25	81.88	84.01	85.26
We present a quantitative comparison of DP-RandP and DP with public data works [54, 50, 49, 52] in Tab. 7. Our DP-RandP is comparable with Nasr et al. [52], which in fact utilizes a limited amount of in-distribution data as public data for pre-training. This indicates that the prior learned from images
generated from random processes can help as much as the prior learned from limited in-distribution public data. Compared to works [50, 49, 54] with access to large public data, there is still a gap between our DP-RandP and these works. Note that we consider a different threat model to this line of work where we do not have access to public data [62] and must instead make the best possible use of images drawn from random processes. Closing the gap between leveraging synthetic data and leveraging large-scale real public data is an interesting direction for future work.
3.5 Computational cost
DP-RandP consists of three phases. Some phases can be done in a single-run per dataset. For example, for feature extractor in Phase I, we only need to train a single feature extractor once for each evaluated dataset as the synthetic dataset and model architectures are the same. Also, for the private linear probing (LP). We report the computation cost in Tab. 8 for single-run per dataset. For Phase II and Phase III, we need to go through these processes at each run of our training process. Also, after we get the extracted features for LP, we need to train a linear layer each time we run the experiments. We report the computation cost in Tab. 9 for these procedures.
De et al. [15] also needs to fully train a WRN-16-4 and the major additional computation cost for DP-RandP is Phase I compared to De et al. [15]. However, the training in Phase I is done once for CIFAR10. Moreover, we can use the same feature extractor from Phase I for CIFAR10 and CIFAR100.
Table 8: One-time per dataset computational Table 9: Computational cost on CIFAR10 for hypercost on CIFAR10. These procedures only need parameters given in Appendix C. These procedures to be done once for each evaluated dataset. are comparable to the standard training procedure Phase I can be shared for CIFAR10/CIFAR100. such as De et al. [15].
Phase I	feature extraction in LP	linear probing in LP	Phase II	Phase III
Time	16 h	1 min	Time	1 min	12min	5.5 h
For the private linear probing experiment, each run of training a linear layer can be finished in 1 minute for CIFAR10 and 320 minutes for ImageNet while fully privately training a model costs much more time. A single run to privately train a WRN-16-4 for CIFAR10 takes around 5.5 hours for 875 steps with 1 A100 GPU in our evaluation. Also, as reported in previous work [57], a single run for ImageNet experiments needs to take four days using 32 A100 GPUs.
4 Discussion and related work
In this section we first provide a detailed comparison to previous work [61] that also uses priors to improve DP-SGD image classification. We then give an overview of the broader body of work on improving the privacy utility tradeoffs in DP-SGD. Finally we discuss the previous work [43] that also uses the two-stage training with domain-specific data for a different reasoning.
Discussion on DP-RandP and Tramér and Boneh [61]. Tramér and Boneh [61] find that training a neural network (linear layer or CNN) on top of 'handcrafted' ScatterNet [53] features outperforms private 'deep' learning. While this method performs well for smaller values of  $\varepsilon$ , the non-private accuracy is limited because the features cannot be adapted to the private data. There are two key differences between DP-RandP and Tramér and Boneh [61]. In Phase I we use representation learning to train the feature extractor on images sampled from random processes, to learn the prior that extract transformations-invariant features. Our feature extraction process is therefore much more general, and while we explore the use of different synthetic data, model architectures, and representation learning methods, there are many more methods in each of these categories that we have not explored. The second difference is between the training on top of extracted features, that is used by Tramér and Boneh [61] and the private linear probing that we consider as a baseline in Fig. 2, and the combination of linear probing and full training that we use in DP-RandP. Comparing the improvements between DP-RandP and Tramér and Boneh [61] confirms that our empirical improvements are mostly due to the advantage of DP-RandP over DP linear probing. In particular, for  $\varepsilon = 3$ , exchanging the handcrafted features of Tramér and Boneh [61] for the pretrained feature extractor we use only improves performance by a modest  $\sim 2\%$ . However, our full DP-RandP exhibits more than  $10\%$
improvement. Our innovation over Tramèr and Boneh [61] is therefore twofold: we introduce the potential of pretraining on synthetic data for the DP community, and also provide guidance on how to better transfer features learned from synthetic data to private training.
DP with public data. A major direction in improving the privacy utility tradeoff in DP-SGD for image classification is the principled use of public data. Several works [68, 2, 45, 52] make use of public data under a different threat model by treating a small fraction of the private training dataset as public. There is also another line of work that leverages a large real-world public dataset to pretrain models [50, 49, 54, 56].
Besides directly training image classification by DP-SGD, another direction is DP-trained generative models. The generated images can be used for classification tasks without additional privacy costs. Recent work [26] show that DP diffusion models can achieve high-quality images when pretrained on large public data like ImageNet and achieve  $88.8\%$  classification accuracy for CIFAR10 at  $\varepsilon = 10$ .
Another line of work has shown the success of DP-SGD fine-tuning of pretrained large language models (LLMs) [28, 47, 70]. LLM pretraining can be framed as a way to learn structural priors from unstructured data [7].
DP-SGD training from scratch. In addition to the directly related works discussed above, the baselines for training from scratch that we compare to in this work are De et al. [15] and Sander et al. [57]. De et al. [15] make use of multiple techniques that are now a mainstay of DP-SGD training from scratch such as multiple data augmentations [32] that we also use. Sander et al. [57] propose a method for estimating the best hyperparameters for DP training at scale using smaller-scale runs. Recent works [34, 35] propose the use of inductive bias via architectural prior. They use DP-SGD to train the equivariant CNN architecture [14] and achieve  $81.6\%$  at  $\varepsilon = 8$  on CIFAR-10. We note that the design space of novel architectures that are especially compatible with DP is rich and mostly unexplored, and our approach is compatible with any advancements in this domain. In particular, using the architecture of Holzl et al. [34, 35] in DP-RandP could potentially enjoy the improvements by combining the feature priors. We leave this exploration as future work.
Two-stage training with domain-specific data. The two-stage training of first training the classifier head and then tuning all hyperparameters has also shown to be effective for out-of-distribution (OOD) tasks [43]. Their reasoning is that if the full parameters are directly updated using the in-distribution training data, this may lead to the loss of some general features learned in the pretrained stage and result in utility drop on OOD data. Our task of DP image classification is different from the OOD task [43] but still shares some common intuition. After Phase I, our feature extractor has learned some useful priors while the classifier head has a random initialization. If we directly update the full network, too much noise is added to the full network, which may distort features learned in Phase I and lead to suboptimal performance. We also conduct experiments and find that, without noise prior (therefore model is randomly initialized), the two-stage training pipeline would not significantly improve the performance compared to directly training the full parameters (See Appendix H).
We leverage images generated from random processes and propose a three-phase training DP-RandP to optimize the use of noise prior. The evaluation across multiple datasets including the benchmark datasets CIFAR10 and ImageNet shows that DP-RandP can improve the performance of DP-SGD. For example, DP-RandP improves the previous best reported accuracy on CIFAR10 from  $60.6\%$  to  $72.3\%$  at  $\varepsilon = 1$ . DP-RandP is a general framework for different datasets, models, and representation learning methods. Future improvements of designs in each of these categories would potentially improve the performance of DP-RandP. DP-RandP makes use of priors from synthetic images. It would be interesting to study whether DP-RandP would improve the priors for DP with public data. Also, investigating the priors beyond image domains, e.g., language and speech tasks, for differentially private training would also be of great interest.
We are grateful to anonymous reviewers at NeurIPS for valuable feedback. This work was supported in part by the National Science Foundation under grant CNS-2131938, the ARL's Army Artificial Intelligence Innovation Institute (A2I2), Schmidt DataX award, and Princeton E-affiliates Award.
[66] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
[67] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv:2109.12298, 2021.
[68] Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In International Conference on Learning Representations, 2021.
[69] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large scale private learning via low-rank reparametrization. In Proceedings of the 38th International Conference on Machine Learning, pages 12208-12218. PMLR, 2021.
[70] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations, 2022.
[71] Yaodong Yu, Maziar Sanjabi, Yi Ma, Kamalika Chaudhuri, and Chuan Guo. Vip: A differentially private foundation model for computer vision. arXiv preprint arXiv:2306.08842, 2023.
A Ablation study for Phase I
We present our main results by using the WRN-16-4 model pretrained on Style-GAN dataset by Baradad et al. [5] with representation learning [64] for CIFAR10/CIFAR100/DermaMNIST. In this section, we provide more ablation study on the different synthetic data and different representation learning methods in Phase I. We use CIFAR10 for the evaluation.
A.1 Results on different synthetic data
A number of synthetic datasets have been proposed by prior work. Baradad et al. [5] consider a family of synthetic datasets generated by random processes, and report that the best synthetic dataset in terms of downstream performance is generated by an untrained StyleGAN with a specific initialization, denoted as StyleGAN-Oriented. Based on this prior work, we also use StyleGAN-Oriented as our main synthetic dataset throughout the main body of the paper. We also considered the Shaders dataset proposed in Baradad et al. [6] for our ImageNet experiments. In this subsection we consider different choices of synthetic datasets and find that multiple synthetic datasets can provide good performance. That is, our results and analysis extend beyond StyleGAN-Oriented and can be applied to future proposed synthetic datasets.
We first use the feature extractor checkpoint<sup>4</sup> provided by Baradad et al. [5] pretrained on different synthetic datasets (Please refer Baradad et al. [5] for the full description of these datasets).
Note that Baradad et al. [5] uses a small AlexNet [42] as feature extractor, which includes BatchNorm [36]. We freeze the feature encoder and only train the last linear model, therefore the feature extractor does not break our differential privacy guarantee. When we use the feature extractor in the main body we use WideResNet without BatchNorm.
Tab. 10 summarizes the results on different synthetic datasets. As in Baradad et al. [5], the best synthetic dataset is StyleGAN-oriented. However, even the worst-performing synthetic dataset (Dead leaves) performs similarly to the best prior work [61, 15]. Tab. 10 suggests that one potential future direction of DP-RandP is better synthetic data.
Table 10: Test accuracy (\%) of private linear probing ( $\varepsilon = 1$ , training for 100 steps, that is the same as with WRN-16-4) on a small AlexNet trained on different synthetic images. StyleGAN-Oriented achieves the best performance. Note that all these images are generated without access to real-world images. Evaluation on CIFAR10 task.
Dead leaves textures	Stat Color+WMM	Untrained StyleGAN	Feature Vis Dead leaves
Sparse	High freq	Oriented
Accuracy	59.92	64.59	64.34	64.08	67.79	59.32
A.2 Results on different representation learning methods
In Tab. 11, we compare the representation learning method of Wang and Isola [64] (results already in the main body) to MoCo [30] (we use default hyperparameters in the official repository) for use in Phase I. For Phase II and III, we use the same hyperparameters as in Tab. 13.
Similar to the main results, DP-RandP achieves significant improvements compared to baseline [15]. We find that using either of contrastive learning methods [64, 30] can achieve  $72\%$  accuracy at  $\varepsilon = 1$ . Also, Tab. 11 shows that DP-RandP consistently improves upon DP-RandP w/o Phase II when using either of Wang and Isola [64] or He et al. [30] in Phase I. We note that while DP-RandP is robust to the two representation learning choices of for Phase I, there is a small gap between the two methods as  $\varepsilon$  increases. This suggests a future direction for further improving our method with a principled choice of contrastive learning method for Phase I.
<sup>4</sup>https://github.com/mbaradad/learning_with_noise.
Table 11: Test accuracy (%) of different representation learning methods in Phase I. Evaluation on CIFAR10 task.
Method for Phase I	Phases	ε = 1	ε = 2	ε = 3	ε = 4	ε = 6	ε = 8	ε = ∞
Wang and Isola [64]	DP-RandP	72.32	77.25	79.99	81.88	84.01	85.26	91.69
DP-RandP w/o Phase II	69.03	75.31	78.44	80.56	82.96	84.45	91.69
MoCo [30]	DP-RandP	72.79	76.26	78.32	79.78	81.64	82.84	90.84
DP-RandP w/o Phase II	69.48	73.82	76.60	78.89	80.56	82.64	90.84
-	Phase III only (De et al. [15])	56.8	64.9	69.2	71.9	71.0	79.5	88.9