SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinyu Tang* Ashwinee Panda* Vikash Sehwag Prateek Mittal  
Princeton University
In privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) performs worse than SGD due to per-sample gradient clipping and noise addition. A recent focus in private learning research is improving the performance of DP-SGD on private data by incorporating priors that are learned on real-world public data. In this work, we explore how we can improve the privacy-utility tradeoff of DP-SGD by learning priors from images generated by random processes and transferring these priors to private data. We propose DP-RandP, a three-phase approach. We attain new state-of-the-art accuracy when training from scratch on CIFAR10, CIFAR100, MedMNIST and ImageNet for a range of privacy budgets  $\varepsilon \in [1,8]$ . In particular, we improve the previous best reported accuracy on CIFAR10 from  $60.6\%$  to  $72.3\%$  for  $\varepsilon = 1$ . Our code is available at https://github.com/inspire-group/DP-RandP.
Figure 1: Our proposed DP training pipeline (DPRandP) has three distinct phases. In Phase I, we sample images from random processes and train a feature extractor with representation learning to embed image priors beneficial for visual tasks. In Phase II, we spend a small privacy budget to train a linear classifier on top of extracted features of private data. In Phase III, we update all parameters with our remaining privacy budget. We demonstrate that incorporating image priors in Phase-I significantly improves DP training and adopting Phase II before training the whole network in Phase III can further improve test accuracy in DP training.
Figure 2: Comparing our results on CI-FAR10 and previous state-of-the-art for  $(\varepsilon, 10^{-5})$ -DP setup. Our full DP-RandP outperforms all previous works on this commonly benchmarked dataset and reduces the privacy cost needed to achieve  $80\%$  accuracy from  $\varepsilon = 6$  to  $\varepsilon = 3$ . Even our private linear probing from noise prior outperforms all previous work at  $\varepsilon = 1$  but crucially has diminishing returns, a shortcoming that we address with our proposed three-phase DP training framework (DP-RandP).
Machine learning models are susceptible to a range of attacks that exploit data leakage from trained models for objectives such as training data reconstruction and membership inference [58, 4]. Differential Privacy (DP) is the gold standard for quantifying privacy risks and providing provable guarantees against attacks [20, 21]. DP implies that the outputs of an algorithm e.g., the final weights trained by stochastic gradient descent (SGD) do not change much (given by the privacy budget  $\varepsilon$ ) across two neighboring datasets  $D$  and  $D'$  that differ in a single entry.
Definition 1 (Differential Privacy) A randomized mechanism  $\mathcal{M}$  with domain  $\mathcal{D}$  and range  $\mathcal{R}$  preserves  $(\varepsilon, \delta)$ -differential privacy iff for any two neighboring datasets  $D, D' \in \mathcal{D}$  and for any subset  $S \subseteq \mathcal{R}$  we have  $\operatorname*{Pr}[\mathcal{M}(D) \in S] \leq e^{\varepsilon} \operatorname*{Pr}[\mathcal{M}(D') \in S] + \delta$ .
Differentially Private Stochastic Gradient Descent (DP-SGD) [59, 1] is the standard privacy-preserving training algorithm for training neural networks on private data, with an update rule given by  $w^{(t + 1)} = w^{(t)} - \frac{\eta_t}{|B|} \left( \sum_{i\in B} \frac{1}{c} \mathrm{clip}_c(\nabla \ell(x_i, w^{(t)})) + \sigma \xi \right)$  where the changes to SGD are the per-sample gradient clipping  $\mathrm{clip}_c(\nabla \ell(x_i, w^{(t)})) = \frac{c \times \nabla \ell(x_i, w^{(t)})}{\max(c, ||\nabla \ell(x_i, w^{(t)})||_2}$  and addition of noise sampled from a  $d$ -dimensional Gaussian distribution  $\xi \sim \mathcal{N}(0, 1)$  with standard deviation  $\sigma$ . DP-SGD introduces bias and variance into SGD and therefore degrades utility, creating a challenging privacy-utility tradeoff. For example, the state-of-the-art accuracy in private training is only  $60.6\%$  on CIFAR-10 at  $\varepsilon = 1$  [35], while Dosovitskiy et al. [19] obtains  $99.5\%$  accuracy non-privately.
Theoretical analysis of the DP-SGD update yields that noise addition is especially harmful to convergence at the start of training [24], and that pretraining on public data can greatly improve convergence in this initial phase of optimization [46] by providing a better initialization [25]. Previous works have improved the privacy-utility tradeoff of DP-SGD by pre-training on large publicly available datasets, such as ImageNet [16], to learn visual priors [50, 49, 54, 9]. Other works assume that a small subset of in-distribution data is publicly available [68, 2, 45, 52].
Interestingly, previous work has uncovered that synthetic data learned from random processes [44, 10, 22, 38] can be used in representation learning [30, 12, 11] to learn highly useful visual priors [39, 5]. Because there is a large distribution shift between synthetic images and natural images, training on synthetic images does not incur any privacy cost. In this work, we leverage noise priors learned from synthetic images to boost the performance of DP training, and make the following key contributions:
- We find that while noise priors are only marginally helpful in non-private training, we can unlock their full potential to improve private training with a carefully calibrated training framework.
- We provide empirical evidence that priors learned from random processes become more critical as the privacy budget decreases, because priors provide fast convergence at the start of training, and have a much larger impact on private training than non-private training.
- We find that training a single linear layer (linear probing) on top of a pretrained feature extractor that has learned noise priors is more robust to large amounts of noise addition than end-to-end training of the entire network. We demonstrate this insight by linear probing with a small privacy budget  $\varepsilon = 0.1$  to  $57.1\%$  on CIFAR10, achieving nontrivial performance with lower privacy cost than previous work has considered.
- We observe that while linear probing from noise prior has diminishing returns on performance as the privacy budget increases, end-to-end training of the entire network continues to improve with large  $\varepsilon$  but critically struggles for small privacy budgets.
- We harness our insights by proposing a privacy allocation strategy that combines the benefits of learning from priors, linear probing, and full training into our full method DP-RandP (Visualized in Fig. 1). Our proposed approach pretrains a feature extractor on synthetic data to learn priors from random processes without paying privacy cost, then pays a small privacy cost for linear probing and makes the best use of our remaining privacy budget by updating all parameters to adapt our learned features to the private data.
- We evaluate DP-RandP against previous work and unlock new SOTA performance across CIFAR10, CIFAR100, MedMNIST and ImageNet across all evaluated privacy budgets  $\varepsilon \in [1,8]$ . We provide a snapshot of our results in Fig. 2.
2 DP-RandP: Mitigating the effects of noise in DP-SGD with noise prior
Synthetic image generation without natural images. Recent progress in computer vision shows that pretraining models on synthetic images without natural images [5, 6, 39] can learn visual priors that are competitive to priors from natural images. The synthetic images can either be generated from texture and fractal-like noise [6, 39], or structure priors extracted from an untrained StyleGAN [5].
Noise prior from synthetic images for private training. We consider pretraining on synthetic data, generated by random processes [5, 6] (example images are in Fig. 3), as a noise prior in differentially private training. We use contrastive representation learning [12, 11, 30, 64], which aims to learn features invariant to common image transformations, to learn good visual features using synthetic images. At a high level, rather than using a random or 'cold' initialization for our downstream private dataset, we are using representation learning to obtain a 'warm initialization' that encodes priors learned from random processes. Across common natural vision tasks, noise priors also ensure that there is no privacy leakage from the pre-training data into this 'warm initialization'.
(a) StyleGAN [5] (b) Shaders [6]
Figure 3: We use synthetic data generated from random processes in representation learning to learn useful image priors. Due to the high distribution shift from real images, we do not incur privacy costs when learning noise priors from these images.
Figure 4: Comparison of training from a random initialization (cold) and pretrained encoder on synthetic dataset (warm) across different privacy budgets (the x-axis is the corresponding  $\sigma$ ).
Figure 5: We zoom in on  $\sigma = 0, \sigma = 9.3$  with learning rate  $= 0.4$  and find that the improvement of the warm start over cold start is mostly due to the initialization using synthetic dataset.
How much noise prior benefits private training? As an initial exploration, we obtain a warm initialization via representation learning on synthetic images and compare it with a competitive DP-SGD baseline [15] that uses cold (random) initialization (Fig. 4). We find that while the warm initialization only improves performance by  $2.7\%$  when  $\sigma = 0$ , i.e., non-private training, the warm initialization improves performance by  $12.5\%$  when  $\sigma = 9.3$  (equivalent to  $\varepsilon = 1$ ). In Fig. 5, we zoom in on a single point of comparison between  $\sigma = 0$  (dashed) and  $\sigma = 9.3$  (solid) for warm and cold initializations. We find that over the course of training, as more noise is added, both the  $\sigma = 9.3$  warm and cold initializations not only converge at the same rate (given an appropriately small learning rate) but also diverge from the non-private runs at similar rates. Although warm and cold start obtain different results in the private setting, the difference is mostly due to the initialization and is therefore magnified at smaller privacy budgets.
We support this claim by drawing a connection to previous works. Ganesh et al. [25] prove the existence of out-of-distribution public datasets that can achieve small test loss on target private datasets. Bu et al. [8] prove that in the gradient flow setting for the NTK regime, that is, when we are taking very small steps  $\eta \rightarrow 0$ , noise does not impact convergence. Mehta et al. [50], Panda et al. [54] propose scaling the learning rate  $\eta$  inversely with the noise  $\sigma$ . We combine our analysis with these previous works by noting that if we start training from fixed initialization and set the learning rate near-zero for small privacy budgets [50, 54], we enter the regime [8] where the noise does not impact convergence, and achieving nontrivial performance for these small privacy budgets provides the first empirical evidence for the theory [25] that only the initialization matters.
We gather our insights into a design goal that will enable us to achieve nontrivial performance under strict privacy constraints. We want to encode a learned prior into our initialization and then adapt this prior to private data. We now introduce DP-RandP, a method that achieves this key design goal.
2.1 Three-phase differentially private training framework
We propose a three-phase DP training framework DP-RandP that has two phases after pretraining on synthetic images, and significantly improves the performance of DP training (Fig. 1). We first learn noise priors by training a feature extractor on synthetic data (Phase-I). We then split our private training into 1) Learning the head classifier with frozen features (Phase-II) and 2) End-to-end training of the entire network to co-adapt the feature extractor and head classifier (Phase-III).
Our design is motivated by the strengths and weaknesses of linear probing and end-to-end training in the private setting. First, the  $\ell_2$  norm of the Gaussian noise added to gradients in each DP-SGD step scales with the number of parameters in the model. Because the head classifier typically has far fewer parameters than the feature extractor, updating this linear layer reduces the amount of added noise. Second, there is a large distribution shift between synthetic data from random processes and natural images found in private datasets. Linear probing merely inherits the frozen pre-trained features, but end-to-end training of all layers can improve the pre-trained features by adapting them to the private dataset. Because adapting to our private dataset may require many end-to-end training steps, we improve the convergence by first learning a task-specific linear classifier that can be trained quickly, and then training the entire network end-to-end. DP-RandP satisfies our previously stated design goal by first obtaining a good initialization with pretraining, then transferring the prior encoded in this initialization to the private dataset with fast-converging linear probing, and finally end-to-end training to adapt to the private dataset as much as possible for a given privacy budget.
We further consider the design choice of how to allocate the privacy budget for the two private training phases  $(\varepsilon_{1}$  and  $\varepsilon_{2})$ . Recall that our model is pre-trained on synthetic data in Phase I, thus it doesn't incur any privacy cost. In Phase II, we use DP-SGD with privacy budget  $\varepsilon_{1}$  to train the linear classifier. We observe diminishing returns in accuracy with increasing  $\varepsilon_{1}$ , so we spend a smaller privacy budget on linear classifier training and allocate the rest to training the full network in Phase III. Allocating a higher budget to full training does not have diminishing returns; consider that  $\varepsilon_{2} = \infty$  we recover non-private accuracy. We now rigorously evaluate the performance of DP-RandP.
3 Evaluation
To outline the evaluation section we first overview our experimental setup and then evaluate the performance of DP-RandP on CIFAR10/CIFAR100/DermaMNIST in Sec. 3.2. We find that our method outperforms all previous works across multiple datasets, architectures, and privacy budgets. In Sec. 3.3 we find that our principled allocation of privacy budget  $\varepsilon_{1},\varepsilon_{2}$  between linear probing and end-to-end training is robust to different choices of  $\varepsilon_{1}$  and  $\varepsilon_{2}$ . We also find that the private linear probing is a strong computationally efficient baseline. Specifically, we provide a new SOTA on ImageNet with private linear probing. We next provide a quantitative comparison of DP-RandP to DP with public data. Finally we discuss the computational costs of our method and find that DP-RandP can provide computational savings over previous methods.
Learning priors from images generated by random processes with representation learning. In Phase I we sample images from StyleGAN-oriented [5], and train a feature extractor on these synthetic datasets with representation learning [12, 64] with the loss function proposed in Wang and Isola [64]. Although our method can accommodate any kind of synthetic data and representation learning method, we focus our evaluation on these datasets and methods. We consider other kinds of synthetic data and other representation learning in Appendix A.
Datasets and models. We evaluate DP-RandP on CIFAR10/CIFAR100 [41], DermaMNIST in MedMNIST [65, 66] and private linear probing version of DP-RandP on ImageNet [16]. For CIFAR10/CIFAR100, we use WRN-16-4 following De et al. [15]. For MedMNIST, we use ResNet-9 following Holzl et al. [34]. We choose WRN-16-4 and ResNet-9 because these architectures for the corresponding datasets achieve the most compelling results in previous works [15, 34]. For ImageNet, we use a ViT-base [19] feature extractor pretrained on Shaders-21k [6] provided by Yu et al. [71] and train a linear classifier. We provide the results on CIFAR10, CIFAR100, DermaMNIST in Sec. 3.2 and results on ImageNet in Sec. 3.3. We also report results of WRN-40-4 on CIFAR10 in Appendix B.
Implementation details. To ensure a fair comparison with previous works [15, 57, 35], we use standard DP-SGD [1] and make use of multiple data augmentations and exponential moving average (EMA) as proposed by De et al. [15], that are now standard techniques used in DP-SGD work [57, 35]. We allocate small privacy budget  $\varepsilon_{1}$  to Phase II and remaining privacy budget  $\varepsilon_{2}$  to Phase III according to the strategy detailed in Sec. 3.3. We report our results across different privacy costs and use  $\delta = 10^{-5}$  for CIFAR10/CIFAR100/DermaMNIST by following previous works [15, 34]. When we report results, we report the standard deviation and accuracy averaged across 5 independent runs with different random seeds. We report implementation details for CIFAR10/CIFAR100/DermaMNIST in Appendix C and for ImageNet in Appendix G.
3.2 Evaluation of DP-RandP
We report the results of DP-RandP in Tab. 1, Tab. 2 and Tab. 3 for CIFAR10, CIFAR100 and DermaMNIST datasets, respectively.
DP-RandP outperforms all previous works across all privacy budgets. In Tab. 1 we find that DP-RandP obtains higher performance than previous works [15, 35, 61, 40, 18, 69, 55] on CIFAR10 across the standard evaluated privacy budgets  $\varepsilon \in [1,8]$ .
We first compare DP-RandP to De et al. [15] as our CIFAR10 model, optimizer and hyperparameters follow De et al. [15]; the only difference is the use of Phase I and Phase II in DP-RandP to learn a prior from synthetic data and allocate a small privacy budget to linear probing. Crucially DP-RandP outperforms De et al. [15] by more than  $15\%$  for the important privacy budget  $\varepsilon = 1$ .
Tramér and Boneh [61] use a ScatterNet [53] to encode invariant image priors and Hölzl et al. [35] use equivariant CNNs [14] to learn transform invariant features. DP-RandP shares the same intuition of leveraging invariant image priors as these works [61, 35]. Instead of leveraging model architecture design for invariant features, we achieve this intuition by learning priors from images generated from random processes and design our three-phase framework to optimize use of this prior. Although DP-RandP shares this intuition of leveraging invariant image priors, DP-RandP achieves  $12\%$  improvement over previous works [61, 35] who both achieve  $60\%$  at  $\varepsilon = 1$ . Our improvement comes both from leveraging the priors from synthetic images and our design of three learning phases that makes the best use of priors. We provide a detailed comparison to Tramér and Boneh [61] in Sec. 4 where we explain why and how the feature prior we learn from synthetic data provides better results than the feature prior provided by their architectures.
Table 1: Test accuracy (\%) of DP-RandP and comparison to previous work on CIFAR10. Not shown in this table are Klause et al. [40], Dörmann et al. [18], Papernot et al. [55], Yu et al. [69] because they achieve  $71.5\%$  at  $\varepsilon = 7.5, 70.1\%$  at  $\varepsilon = 7.42, 66.2\%$  at  $\varepsilon = 7.53$  and  $63.4\%$  at  $\varepsilon = 8$  respectively, that are not on the pareto frontier of previous work.
Method	ε = 1	ε = 2	ε = 3	ε = 4	ε = 6	ε = 8	ε = ∞
Tramèr and Boneh [61]	60.3	67.2	69.3	-	-	-	73.8*
De et al. [15]	56.8	64.9	69.2	71.9	77.0	79.5	88.9
Hölzl et al. [35]	60.59	71.86	75.96	78.27	80.26	81.62	-
DP-RandP	72.320.22	77.250.07	79.990.21	81.880.27	84.010.23	85.260.11	91.69
CIFAR100 results. Previous work has not provided results on training from scratch for CIFAR100, perhaps because as we find in Tab. 2 the DP-SGD baseline following De et al. [15] does not perform well for  $\varepsilon = 3$ . We believe this to be a competitive baseline, but DP-RandP outperforms it by more than  $10\%$  across  $\varepsilon \in [3,8]$ . Although CIFAR10 and CIFAR100 are both benchmark computer vision datasets, we find CIFAR100 to be a much more challenging task for private learning. One possible
2We run experiments for  $\varepsilon = \infty$  with per-sample gradient clipping but without noise, because we are interested in the ability of DP-RandP to mitigate variance. We note that there is accuracy degradation from the non-private baseline even at  $\varepsilon = \infty$  due to the bias introduced by per-sample gradient clipping [37]. When we report  $\varepsilon = \infty$  results from previous work we mark them with  $*$  when previous work does not report whether the non-private baseline uses clipping.
explanation for this is that private classifiers struggle to distinguish between many classes because the added noise is more likely to shift the decision boundary. We use the same hyperparameters for CIFAR10 and CIFAR100, that are likely suboptimal for CIFAR100, and the performance gap between  $\varepsilon = 8$  and  $\varepsilon = \infty$  may be mitigated if we train on CIFAR100 for longer than we do on CIFAR10. We encourage the use of CIFAR100 as a standard benchmark for private learning in the future because we find limited room for improvement on CIFAR10. In particular, the private and non-private gap between  $\varepsilon = 8$  and  $\varepsilon = \infty$  for CIFAR10 is only  $\approx 6\%$  for DP-RandP.
Table 2: Test accuracy (%) of DP-RandP and comparison to DP-SGD baseline on CIFAR100.
Method	ε = 3	ε = 4	ε = 6	ε = 8	ε = ∞
DP-SGD	30.73	34.45	39.66	44.22	66.68
DP-RandP	43.330.15	46.400.31	51.530.13	55.020.21	71.68
**DermaMNIST results.** We have shown that DP-RandP outperforms previous work on the standard CV benchmarks of CIFAR10 and CIFAR100, and now consider the privacy sensitive medical dataset DermaMNIST. Although CIFAR is a standard CV benchmark, there is limited previous work that evaluates on privacy sensitive data in CV such as medical images. In Tab. 3 we find that DP-RandP achieve improvements from  $\varepsilon = 1$  to  $\varepsilon = 7.42$  by up to  $2.78\%$  over the DP-SGD baseline that we evaluate. Also, DP-RandP can achieve similar accuracy at  $\varepsilon = 4$  as the result of DP-SGD at  $\varepsilon = 7.42$ , which reduces the privacy cost from  $\varepsilon = 7.42$  to  $\varepsilon = 4$ . We also include the results of Holzl et al. [34], that use equivariant neural networks [14]. DP-RandP is a uniform framework applicable to any model architectures. We leave the systematic investigation of more model architectures, such as equivariant neural network [14, 35] in DP-RandP, for future work.
Table 3: We follow previous work [34] and report the validation accuracy  $(\%)$  of DP-RandP on DermaMNIST. We also report the test accuracy in Appendix D.
ε	ε = 1	ε = 4	ε = 7.42	ε = ∞
baseline in Hölzl et al. [34]	-	-	72.41	78.48*
best in Hölzl et al. [34]	-	-	74.17	77.84*
DP-SGD we evaluated	69.000.37	71.780.87	74.080.41	77.27
DP-RandP	71.780.40	74.820.55	75.910.29	79.26
We find that the prior we learn from synthetic data is applicable to standard benchmarks and medical images. One direction for future work is to validate the robustness of this prior across more datasets.
3.3 Allocating privacy budget in DP-RandP
In this subsection we analyze the privacy budgets of Phase II ( $\varepsilon_1$ , linear probing) and Phase III ( $\varepsilon_2$ , full training) and how to allocate the overall privacy budget  $\varepsilon$  among Phase II and Phase III. Specifically, we will present a new STOA on ImageNet with additional designs on linear probing.
In Fig. 6 we observe that DP-RandP is robust to the key algorithmic design choice of how much privacy budget to allocate to Phase II and Phase III (Please check Appendix E for more details of Fig. 6 and more results on different  $\varepsilon$ ).
In particular, we find that even the worst choice of  $\varepsilon_1 / \varepsilon$  provides results comparable to the previous SOTA on CIFAR10 at  $\varepsilon = 1$ .<sup>3</sup> We first investigate the behavior at each extrema and conclude a general allocation strategy that provides good performance across CIFAR10, CIFAR100 and DermaMNIST.