SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since DP-MIX $_{\text{DIFF}}$  uses a diffusion model to generate synthetic examples, this could make the comparison unfair because other methods do not use synthetic samples. To avoid this, we purposefully use the exact same pre-training data (i.e., LAION-2B) to pre-train models as was used to train the diffusion model. This avoids the issue of the synthetic examples somehow "containing" data that other methods do not have access to. Moreover, we conducted experiments (check Table 6) to show that the synthetic examples themselves do not boost performance. It is the way they are used by DP-MIX $_{\text{DIFF}}$  that boosts performance. Finally, out of the six datasets we used for evaluation, none of them overlap with the LAION-2B dataset (to the best of our knowledge).
5.1 Training from Scratch with DP
Since DP-MIX $_{\text{DIFF}}$  relies on synthetic samples from diffusion models, it would not be fair to compare it to prior SoTA methods that do not have access to those samples in the training from scratch setting. Therefore, we focus this evaluation on DP-MIX $_{\text{SELF}}$ . For this, we use a WRN-16-4 model. Our baseline from this experiment is De et al. [9] who use a combination of techniques to improve performance, but mostly self-augmentation. Results in Table 2 show that DP-MIX $_{\text{SELF}}$  consistently provides improvements across datasets and privacy budgets.
We observed differences between the results reported by De et al. [9] using JAX and the reproduction by Sander et al. [34] on Pytorch. In our supplemental materials, Table 11 presents our reproduction of De et al. [9] using their original JAX code alongside the DP-MIX<sub>SELF</sub> implementation.
Table 2: Test accuracy (%) of a WRN-16-4 model trained from scratch: Our proposed DP-MIX $_{\text{SELF}}$  technique significantly outperforms De et al. [9] Self-Aug method (baseline and prior SoTA) in all privacy budget settings and all three datasets considered.
Dataset	Method	ε = 1	ε = 2	ε = 4	ε = 8
CIFAR-10	Self-Aug	56.8 (.5)	62.9 (.3)	69.5 (.4)	78.7 (.5)
DP-MIXSELF	57.2 (.4)	64.6 (.4)	70.47 (.4)	79.8 (.3)
CIFAR-100	Self-Aug	13.3 (.5)	20.9 (.4)	31.8 (.2)	39.2 (.4)
DP-MIXSELF	14.1 (.4)	21.5 (.4)	33.3 (.3)	40.6 (.3)
EuroSAT	Self-Aug	75.4 (.3)	81.1 (.1)	85.8 (.2)	89.7 (.3)
DP-MIXSELF	75.7 (.2)	82.8 (.3)	87.4 (.2)	90.8 (.2)
5.2 Finetuning with DP
We consider two pretrained models Vit-B-16 and ConvNext. As baselines/prior SoTA we consider De et al. [9] and the Mixed Ghost Clipping technique of Bu et al. [5]. Results are shown in Table 3.
We observe that our proposed DP-MIX<sub>SELF</sub> method yields significant improvements over all prior methods across datasets and privacy budgets. We also observe that DP-MIX<sub>DIFF</sub>, which uses samples from the diffusion model, significantly outperforms DP-MIX<sub>SELF</sub> on datasets such as Caltech256, SUN397, and Oxford-IIIT Pet. Notably, when the privacy budget is limited, such as  $\varepsilon = 1$ , we observe remarkable improvements compared to DP-MIX<sub>SELF</sub> (e.g.,  $8.5\%$  on Caltech256 using Vit-B-16). This shows the benefits of incorporating diverse images from the diffusion model via mixup.
On the other hand, we observe that diffusion examples do not provide a benefit for datasets such as CIFAR-10 and EuroSAT. We investigate the cause of this empirically in Appendix B (supplementary materials). On these datasets, DP-MIX $_{\text{DIFF}}$  only sometimes outperforms the prior baselines, but DP-MIX $_{\text{SELF}}$  provides a consistent improvement, suggesting that samples from the diffusion model
Table 3: Test accuracy (\%) of fine-tuned Vit-B-16 and ConvNext: We fine-tune Vit-B-16 and ConvNext models on CIFAR-10, CIFAR-100, EuroSAT, Caltech256, SUN397 and Oxford-IIT Pet datasets using different  $\varepsilon$  with  $\delta = 10^{-5}$ , and report the test accuracy and standard deviation. Our proposed methods, DP-MIX $_{\text{SELF}}$  and DP-MIX $_{\text{DIFF}}$ , outperform the baselines in all cases.
Dataset	Method	Vit-B-16	ConvNext
ε = 1	ε = 2	ε = 4	ε = 8	ε = 1	ε = 2	ε = 4	ε = 8
CIFAR-10	Mixed Ghost	95.0 (.1)	95.0 (.2)	95.3 (.4)	95.3 (.2)	94.6 (.1)	94.6 (.1)	94.7 (.0)	94.7 (.1)
Self-Aug	96.5 (.1)	97.0 (.0)	97.1 (.0)	97.2 (.0)	95.9 (.0)	96.4 (.1)	96.5 (.1)	96.5 (.0)
DP-MIXSELF	97.2 (.3)	97.4 (.2)	97.4 (.2)	97.6 (.3)	96.8 (.1)	96.9 (.1)	96.9 (.1)	97.3 (.1)
DP-MIXDIFF	97.0 (.2)	97.1 (.1)	97.2 (.1)	97.3 (.1)	96.3 (.1)	96.5 (.1)	96.6 (.1)	96.7 (.1)
CIFAR-100	Mixed Ghost	78.2 (.4)	78.5 (.1)	78.4 (.3)	78.4 (.1)	74.9 (.3)	75.1 (.1)	75.5 (.2)	75.8 (.1)
Self-Aug	79.3 (.2)	83.2 (.3)	83.5 (.3)	84.2 (.1)	75.8 (.3)	80.0 (.1)	81.4 (.2)	82.2 (.2)
DP-MIXSELF	81.8 (.2)	83.5 (.1)	84.5 (.1)	84.6 (.2)	78.2 (.3)	80.9 (.2)	82.3 (.1)	82.3 (.1)
DP-MIXDIFF	82.0 (.1)	83.8 (.1)	84.0 (.1)	84.3 (.1)	79.4 (.2)	81.4 (.2)	81.6 (.1)	81.8 (.2)
EuroSAT	Mixed Ghost	84.0 (.1)	84.8 (.2)	84.9 (.1)	85.0 (.2)	85.5 (.2)	86.9 (.1)	87.0 (.6)	87.6 (.3)
Self-Aug	93.3 (.2)	94.1 (.2)	95.4 (.2)	95.5 (.2)	93.5 (.2)	94.5 (.4)	95.2 (.1)	95.2 (.1)
DP-MIXSELF	94.3 (.1)	94.9 (.2)	95.6 (.2)	95.6 (.1)	94.6 (.1)	94.7 (.1)	95.4 (.2)	95.5 (.1)
DP-MIXDIFF	92.6 (.1)	93.4 (.1)	93.9 (.2)	93.9 (.1)	92.8 (.2)	93.2 (.1)	93.6 (.2)	93.8 (.1)
Caltech256	Mixed Ghost	79.7 (.2)	88.2 (.2)	91.4 (.2)	92.3 (.2)	79.2 (.2)	87.4 (.2)	88.0 (.1)	88.2 (.2)
Self-Aug	80.4 (.1)	89.7 (.2)	92.0 (.1)	93.2 (.2)	80.0 (.2)	88.2 (.1)	91.0 (.1)	92.2 (.1)
DP-MIXSELF	81.2 (.2)	90.1 (.2)	92.2 (.2)	93.4 (.1)	81.0 (.1)	88.7 (.1)	91.3 (.1)	92.4 (.1)
DP-MIXDIFF	89.7 (.2)	91.8 (.2)	92.9 (.1)	93.9 (.1)	88.7 (.2)	91.8 (.2)	92.6 (.2)	93.3 (.1)
SUN397	Mixed Ghost	70.7 (.2)	71.2 (.1)	72.2 (.2)	72.5 (.2)	64.3 (.2)	65.0 (.2)	65.3 (.1)	65.3 (.1)
Self-Aug	72.7 (.1)	76.0 (.1)	78.0 (.1)	79.6 (.2)	72.2 (.1)	76.5 (.1)	78.0 (.2)	78.9 (.1)
DP-MIXSELF	73.2 (.1)	76.5 (.2)	78.7 (.2)	79.6 (.1)	72.5 (.1)	76.8 (.1)	78.5 (.0)	79.5 (.1)
DP-MIXDIFF	75.1 (.2)	77.8 (.1)	79.5 (.2)	80.6 (.1)	75.0 (.2)	77.5 (.1)	79.3 (.1)	80.0 (.1)
Oxford Pet	Mixed Ghost	71.2 (.2)	79.1 (.2)	80.4 (.2)	81.0 (.2)	65.2 (.2)	78.2 (.2)	79.1 (.2)	79.9 (.2)
Self-Aug	72.2 (.2)	82.1 (.2)	85.8 (.3)	88.2 (.1)	68.1 (.2)	81.3 (.2)	85.5 (.1)	87.0 (.1)
DP-MIXSELF	72.5 (.2)	82.5 (.2)	86.8 (.2)	88.7 (.2)	68.8 (.2)	81.7 (.2)	86.3 (.2)	87.7 (.2)
DP-MIXDIFF	83.2 (.3)	86.3 (.2)	88.3 (.2)	89.4 (.2)	80.5 (.2)	86.2 (.2)	88.2 (.2)	88.8 (.2)
can inhibit performance. On CIFAR-100, DP-MIX $_{\text{DIFF}}$  outperforms other methods only at low privacy budgets ( $\varepsilon \leq 2$ ), with DP-MIX $_{\text{SELF}}$  providing the best performance in the high privacy budget regime.
6 Ablation Study: Why Does DP-Mix Improve Performance?
We perform ablation experiments to better understand why our methods consistently and significantly outperform the prior SoTA techniques.
6.1 Understanding Self-Augmentations
Since DP-MIX $_{\text{SELF}}$  does not alter the label and involves applying mixup to (augmentations of) a single original training example, the method could be viewed as just another single-sample data augmentation technique. The question then becomes what augmentation techniques improve performance when using differential privacy and why.
De et al. [9] use flipping and cropping as self-augmentation (Self-Aug). We perform experiments using several widely used augmentation techniques (based on Chen et al. [7] — Colorjitter, Translations and Rotations, Cutout, Gaussian Noise, Gaussian Blur, and Sobel Filter) in addition to Self-Aug to compare them to DP-MIXSELF. We set  $K_{\mathrm{BASE}} = 16$  as in previous experiments. Results are shown in Table 4, where the second to last column combines all augmentations together (randomly chosen). We observe that these augmentations do not enhance performance to the same degree as DP-MIXSELF. In a few cases, the augmentations improve performance slightly above Self-Aug, but some of them also diminish performance.
6.2 Number of Self-Augmentations
Increasing the number of self-augmentations beyond 16 does not improve performance [9]. This was also pointed out recently by Xiao et al. [41]. In fact, sometimes increasing the number of
Table 4: Test accuracy (\%) of Self-Aug with other Augmentations: Vit-B-16 model performance on CIFAR-10 and CIFAR-100 with  $\varepsilon = 1$  with different augmentations. We observe that these augmentations do not enhance performance to the same degree as DP-MIX $_{\text{SELF}}$ .
Dataset	Self-Aug	+Jitter	+Affine	+Cutout	+Noise	+Blur	+Sobel	+All	DP-MIXSELF
CIFAR-10	96.5 (.1)	96.3 (.1)	96.2 (.1)	96.5 (.1)	95.8 (.2)	96.7 (.2)	72.5 (.1)	96.5 (.2)	97.2 (.3)
CIFAR-100	79.3 (.2)	78.1 (.2)	73.9 (.2)	79.6 (.1)	73.5 (.2)	80.0 (.1)	9.5 (.2)	79.4 (.1)	81.8 (.2)
self-augmentations hurts performance. We perform experiments by varying the number of self-augmentations ( $K_{\mathrm{BASE}}$ ) on CIFAR-10. Results are shown in Table 5. We observe that  $K_{\mathrm{BASE}} = 16$  is optimal for the self-augmentations proposed by De et al. [9]. However, we obtain significantly better performance DP-MIXSELF for  $K = 32$  ( $K_{\mathrm{BASE}} = 16$ ,  $K_{\mathrm{SELF}} = 16$  and  $K_{\mathrm{DIFF}} = 0$ ). Recall that setting  $K_{\mathrm{DIFF}} = K_{\mathrm{SELF}} = 0$  in Algorithm 1 recovers the Self-Aug method of [9].
Table 5: Increase  $K_{\text{BASE}}$  from 16 to 36 for Self-Aug. We conduct experiments on CIFAR-10 with  $\varepsilon = 8$  and  $\delta = 10^{-5}$  in two settings: train a WRN-16-4 model from scratch and fine-tune a pre-trained Vit-B-16. We can observe that for both cases, there are no substantial performance improvements when increasing  $K_{\text{BASE}}$ .
Training method	Total # of Aug	WRN-16-4	Vit-B-16 (pretrained)
Self-Aug	16	78.8 (.5)	97.2 (.0)
24	78.5 (.4)	97.0 (.1)
32	78.4 (.3)	97.0 (.1)
36	78.6 (.4)	97.0 (.1)
DP-MIXSELF	32	79.8 (.3)	97.6 (.3)
6.3 Pretraining or DP-MIXDIFF?
Since DP-MIX $_{\text{DIFF}}$  uses synthetic samples from a diffusion model, we consider pretraining the model on the synthetic samples  $\mathcal{D}$  from the diffusion model with SGD prior to fine-tuning with DP. The results, presented in Table 6, indicate that pretraining on  $\mathcal{D}$  does not improve the model's performance. This indicates that the improved performance of DP-MIX $_{\text{DIFF}}$  results from the way it uses the diffusion samples, not just from having access to diffusion samples.
Table 6: Test accuracy (\%) with and without pretraining on synthetic diffusion samples: We compare the performance of fine-tuned Vit-B-16 and ConvNext with DP on Caltech256 with versus without pretraining on  $\mathcal{D}$  (using SGD). We set  $\varepsilon = 2$  and  $\delta = 10^{-5}$ .
Pretrained on D	Vit-B-16	ConvNext
Yes	81.5 (.0)	80.9 (.1)
No (Ours)	91.8 (.2)	91.8 (.2)
6.4 Effect of DP-Mix on Training Data
In Table 7, we compare the distributions of the effective train sets produced by different data augmentation techniques against those of the original train and test sets, i.e., we measure the Fréchet Inception Distance (FID), where lower FID values indicate more similarity between the datasets. Note, the FIDs are consistent with test accuracy across datasets and methods. For example, the FID of Self-Aug + Colorjitter is significantly larger than FID of Self-Aug, explaining why adding Colorjitter decreases test accuracy: it results in data less similar to train and test sets. Similarly, the FID for DP-MIX $_{\text{DIFF}}$  for EuroSAT is much larger compared to the other datasets, which explains reduced test accuracy relative to the other methods. In Table 8, we compare the FID between the original train and test sets with the FID between the original train set and the synthetic samples generated by the text-to-image diffusion model that we employ for DP-MIX $_{\text{DIFF}}$ . Note the much greater FID between the train set and the diffusion samples. This large domain gap may explain why, as shown in Table 6,
Table 7: FID for different methods' generated images compared to training set and test set.
Dataset	Train	Test
Self-Aug	+Jitter	DP-MIXSELF	DP-MIXDIFF	Self-Aug	+Jitter	DP-MIXSELF	DP-MIXDIFF
CIFAR-10	2.6	6.0	3.1	3.3	3.1	6.4	3.5	3.7
CIFAR-100	3.1	6.1	3.3	3.5	3.5	6.5	3.8	3.9
EuroSAT	2.2	6.1	3.0	6.6	4.1	8.3	6.1	10.9
Caltech256	1.0	2.9	1.3	1.5	2.5	4.2	2.8	2.9
Table 8: FID between datasets for different FID models. We compute FID values between the training set and test set, and training set and text-to-images diffusion model generated images based on different FID models (InceptionV3 and Vit-B-16).
Dataset	InceptionV3	Vit-B-16
Train vs Test	Train vs Diffusion	Train vs Test	Train vs Diffusion
CIFAR-10	3.2	30.5	0.4	30.1
CIFAR-100	3.6	19.8	0.5	21.9
EuroSAT	7.4	164.0	7.3	82.9
Caltech256	6.8	25.1	1.6	37.4
pretraining on the diffusion samples result in much worse performance compared to our proposed DP-MIXDIFF.
6.5 Influence of  $K_{\mathrm{BASE}}$ ,  $K_{\mathrm{SELF}}$  and  $K_{\mathrm{DIFF}}$
Recall,  $K_{\mathrm{BASE}}$  is the number of base self-augmentations,  $K_{\mathrm{SELF}}$  is the number of mixups, and  $K_{\mathrm{DIFF}}$  is the number of synthetic diffusion samples used. We vary  $K_{\mathrm{BASE}}$ ,  $K_{\mathrm{SELF}}$  and  $K_{\mathrm{DIFF}}$  values and report their performance in Table 9. We observe that selecting  $K_{\mathrm{DIFF}} = 2$  or  $K_{\mathrm{DIFF}} = 4$  and setting  $K_{\mathrm{SELF}} \leq K_{\mathrm{BASE}}$  gives good results overall. In this paper, we used  $K_{\mathrm{BASE}} = 16$ ,  $K_{\mathrm{SELF}} = 18$  and  $K_{\mathrm{DIFF}} = 2$  for most cases as it provides good overall results across many datasets and settings<sup>3</sup>.
Table 9: Influence of  $K_{\mathrm{BASE}}$ ,  $K_{\mathrm{SELF}}$ , and  $K_{\mathrm{DIFF}}$  for fine-tuning Vit-B-16 model on Caltech256 with  $\varepsilon = 1$  and  $\delta = 10^{-5}$ .
KBASE	8	8	8	8	8	8	8	8	8	16	16	16	16	16
KSELF	8	8	8	16	16	16	24	24	24	8	8	8	16	16
KDIFF	0	2	4	0	2	4	0	2	4	0	2	4	0	2
Acc. (%)	80.4	85.0	87.2	77.1	84.5	87.2	77.2	84.2	86.4	77.7	84.2	86.7	81.2	83.8
KBASE	16	16	16	16	24	24	24	24	24	24	24	24	24
KSELF	16	24	24	24	8	8	8	16	16	16	24	24	24
KDIFF	4	0	2	4	0	2	4	0	2	4	0	2	4
Acc. (%)	86.3	77.3	82.6	85.8	77.9	82.7	85.5	77.9	82.1	85.1	80.1	81.5	84.5
We studied the application of multi-sample data augmentation techniques such as mixup for differentially private learning. We show empirically that the most obvious way to apply mixup, using microbatches, does not yield models with low generalization errors as microbatching is at an inherent disadvantage. We then demonstrate how to harness mixup without microbatching by applying it to self-augmentations of a single training example. This provides a significant performance increase over the prior SoTA. Finally, we demonstrate that producing some augmentations using text-to-image diffusion models further enhances performance when combined with mixup.
<sup>3</sup>We use larger  $K_{\mathrm{DIFF}}$  i.e. 24 for Caltech256 and Oxford Pet as it provides better performance.
This work was supported in part by the National Science Foundation under CNS-2055123. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
[40] H. Xiao and S. Devadas. Towards understanding practical randomness beyond noise: Differential privacy and mixup. Cryptology ePrint Archive, 2021.
[41] H. Xiao, Z. Xiang, D. Wang, and S. Devadas. A theory to instruct differentially-private learning via clipping bias reduction. In 2023 IEEE Symposium on Security and Privacy (SP), pages 2170–2189. IEEE Computer Society, 2023.
[42] J. Xiao, K. A. Ehinger, J. Hays, A. Torralba, and A. Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119:3-22, 2016.
[43] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[44] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[45] S. Zagoruyko and N. Komodakis. Wide residual networks. British machine vision conference (bmvc), 2016.
[46] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[47] L. Zhang, Z. Deng, K. Kawaguchi, A. Ghorbani, and J. Zou. How does mixup help with robustness and generalization? 2020.
A Algorithm: DP-SGD
Algorithm 2 DP-SGD from Abadi et al. [2]
Require: Training data  $x_{1},\ldots ,x_{N}$  loss function  $\mathcal{L}(\theta) = \frac{1}{N}\sum_{i}\mathcal{L}(\theta ,x_{i})$  Parameters: learning rate  $\eta_t$  noise scale  $\sigma$  group size  $B$  gradient norm bound  $C$  Initialize  $\theta_0$  randomly for  $t\in [T]$  do Take a random sample  $B_{t}$  with sampling probability  $B / N$  for each  $i\in B_{t}$  do Compute  $\mathbf{g}_t(x_i)\gets \nabla_\theta \mathcal{L}(\theta_t,x_i)$ $\bar{\mathbf{g}}_t(x_i)\gets \mathbf{g}_t(x_i) / \max (1,\frac{||\mathbf{g}_t(\mathbf{x}_i)||_2}{C})$  end for  $\tilde{\mathbf{g}}_t\gets \frac{1}{B}\sum_i(\bar{\mathbf{g}}_t(x_i) + \mathcal{N}(0,\sigma^2 C^2\mathbf{I}))$ $\theta_{t + 1}\gets \theta_t - \eta_t\tilde{\mathbf{g}}_t$  end for
Ensure:  $\theta_T$  and compute the overall privacy cost  $(\varepsilon, \delta)$  using a privacy accounting method
B.1 How useful is the synthetic data?
Let  $S_{1}$  and  $S_{2}$  denote equally sized disjoint subsets of the private dataset, and  $\mathcal{D}$  an equally sized set of synthetic samples generated via diffusion, as described in Section 3.1.
For this experiment, we consider three different training methods.
Method A: Fine-tune on  $S_{1}$  using DP-MIXSELF.
- Method B: Fine-tune on  $S_{1}$  using DP-MIX $_{\text{DIFF}}$  with mixup images sampled from  $\mathcal{D}$ .
- Method C: Same as Method B, but replace set  $\mathcal{D}$  with set  $\hat{S}_2$ . Note, we assume that  $S_2$  is publicly available, so accessing it during training does not incur a privacy cost.
The intuition behind this experiment is that method C provides an upper bound for both A and B, since, in the best case, the distribution of the synthetic data would exactly match that of the private data. The experimental results, presented in Table 10, validate this intuition.
For CIFAR-100, Caltech256, SUN397, and Oxford Pet, method C outperforms method A. Consistent with this, method B, our proposed DP-MIXDIFF, also provides a performance boost. On the other hand, for CIFAR-10 and EuroSAT, method C, despite directly using private data for mixup, does not meaningfully improve performance. Similarly, method B also does not improve performance. For EuroSAT, method B slightly decreases performance, due to the large domain gap (large FID) between the synthetic and private data.
Table 10: Test accuracy (\%) using different training methods with Vit-B-16 and ConvNext on various datasets after fine-tuning. We set the  $\varepsilon = 2$  and  $\delta = 10^{-5}$ .
Dataset	Vit-B-16	ConvNext
A	B	C	A	B	C
CIFAR-10	96.9 (.1)	96.9 (.0)	97.0 (.1)	96.2 (.1)	96.1 (.1)	96.2 (.1)
CIFAR-100	81.3 (.3)	82.0 (.2)	82.1 (.1)	76.7 (.3)	78.0 (.1)	78.4 (.1)
EuroSAT	93.7 (.1)	91.4 (.1)	93.9 (.2)	93.8 (.2)	92.3 (.1)	93.9 (.2)
Caltech 256	76.0 (.4)	81.9 (.1)	82.3 (.2)	76.4 (.3)	81.1 (.1)	81.6 (.3)
SUN397	70.8 (.2)	72.4 (.1)	73.8 (.1)	70.1 (.2)	72.3 (.1)	73.6 (.2)
Oxford Pet	65.1 (.3)	68.3 (.1)	68.3 (.1)	64.6 (.2)	66.0 (.1)	68.1 (.1)
B.2 Reproducing De et al. [9] and DP-MIXSELF in JAX
For completeness, we train WRN-16-4 on CIFAR10 using Self-Aug and our proposed DP-MIXSELF using [9]'s official JAX code (https://github.com/deepmind/jax PRIVacy). As in [9], we repeat each experiment 5 times and report median test accuracy in Table 11. The results are consistent
with those presented in the rest of our paper — DP-MIX $_{\text{SELF}}$  outperforms Self-Aug for different privacy budgets.
Table 11: Performance comparison on JAX.
Method	ε = 1	ε = 8
Self-Aug	56.3 (.3)	79.4 (.1)
DP-MIXSELF	57.1 (.4)	80.0 (.2)
B.3 Pure-DP-MIXDIFF
To demonstrate the influence of  $K_{\mathrm{BASE}}$  in our method, we set  $K_{\mathrm{BASE}} = 0$  and call it Pure-DP-MIXDIFF. In effect Pure-DP-MIXDIFF is simply mixing up the synthetic examples themselves. We test it on CIFAR-100 and represent it in Table 12. We can see that Pure-DP-MIXDIFF offers much worse performance than both DP-MIXSELF and DP-MIXDIFF, although it still offers better performance than Self-Aug due to the beneficial effects of mixup. More generally, we think that Pure-DP-MIXDIFF will tend to worsen an overfitting problem whenever there is a large domain gap between the original training data and the diffusion samples. DP-MIXDIFF does not suffer from this problem because it ensures that (augmented versions) of the original training data samples are seen during training.
Table 12: Test accuracy (\%) of Pure-DP-MIXDIFF  $(K_{\mathrm{BASE}} = 0)$  on CIFAR-100 with Vit-B-16 model. We set  $\delta = 10^{-5}$  and  $\varepsilon = 1$ . We can observe that Pure-DP-MIXDIFF does not improve performance, which shows the necessity of using base augmentations  $(K_{\mathrm{BASE}} > 0)$ .
Method	Test accuracy
Self-Aug	79.3 (.2)
DP-MIXSELF	81.8 (.2)
DP-MIXDIFF	82.0 (.1)
Pure-DP-MIXDIFF	80.9 (.2)
B.4 Running time
We provide the running time for different methods in Table 13. All experimental runs utilized a single A100 GPU and were based on the same task of finetuning the Vit-B-16 model on the Caltech256 dataset for 10 epochs. Due to additional augmentation steps, the training time of our methods is longer than prior work.
Table 13: Running time for different methods of the same task (fine-tuning Vit-B-16 on Caltech256 for 10 epochs). We use one A100 GPU for each training method.
Method	Self-Aug	DP-MIXSELF	DP-MIXDIFF
Running time	2h 12min	7h 33min	7h 40 min
B.5 Effect of Mixup on Gradients
We study what happens to gradients and parameter updates during training for our methods versus Self-Aug. Fig. 2 plots the per-parameter gradient magnitude averaged over samples at each epoch (prior to clipping and noise adding). The histogram shows the data averaged over all training epochs and the X% color lines show that data only for the epoch at X% of the total training process. There are 10 epochs for this experiment – for example, the line for 20% shows the data for epoch 2.
The figure shows more concentrated values for our methods compared to the Self-Aug baseline, which suggests more stable training and faster convergence. Standard deviations for CIFAR-10 with Self-Aug, DP-Mix $_{\text{SELF}}$  and DP-Mix $_{\text{DIFF}}$  are:  $2.16 \cdot 10^{-3}$ ,  $9.76 \cdot 10^{-4}$  and  $9.59 \cdot 10^{-4}$ , respectively. For Caltech256 they are:  $1.43 \cdot 10^{-3}$ ,  $1.07 \cdot 10^{-3}$  and  $9.32 \cdot 10^{-4}$ , respectively. This is consistent with experimental results of test accuracies for each method.
Figure 2: Per-parameter gradient magnitude before clipping and adding noise on CIFAR-10(a) and Caltech256(b) with fine-tuning Vit-B-16 model with  $\varepsilon = 1$  and  $\delta = 10^{-5}$ . The different curves represent different training stages. Values for our proposed DP-MIXSELF and DP-MIXDIFF are more concentrated suggesting more stable training and faster convergence.
(b) Caltech256
C Additional Experimental Details
Datasets We use the following datasets: