SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we establish the generalization bound for bGMM based on the general bound proposed in Theorem 3.1. To derive such a bound, the main task is to bound terms  $M$ ,  $\beta_{m_T}$ ,  $\mathcal{D}_{\mathrm{TV}}(\mathcal{D}, \mathcal{D}_G(S))$  and  $\mathcal{T}(m_S, m_G)$  in Theorem 3.1. For  $M$  (Lemma B.5) and  $\beta_{m_T}$  (Lemma B.6), we mainly use the concentration property of the multivariate Gaussian variable (Lemma B.4). In addition, inspired by previous works on naive Bayes [48], we bound  $\mathcal{D}_{\mathrm{TV}}(\mathcal{D}, \mathcal{D}_G(S))$  (Lemma B.7) by discussing the distance between the estimated parameters and the true parameters of bGMM. Besides, the concentration property of  $\mathcal{T}(m_S, m_G)$  (Lemma B.9) can be induced by the preceding discussion. Finally, we can obtain the following results.
Theorem 3.2 (Generalization bound for bGMM, proof in Appendix B.2). Consider the setting introduced in Section 3.2.1. Given a set  $S$  with  $m_S$  i.i.d. samples from the bGMM distribution  $\mathcal{D}$  and an augmented set  $S_G$  with  $m_G$  i.i.d. samples drawn from the learned Gaussian mixture distribution, then with high probability at least  $1 - \delta$ , it holds that
$$
| \text {G e n - e r r o r} | \lesssim \left\{ \begin{array}{l l} \frac {\log (m _ {S})}{\sqrt {m _ {S}}} & \text {i f f i x d a n d m _ {G} = 0 ,} \\ \frac {\log^ {2} (m _ {S})}{\sqrt {m _ {S}}} & \text {i f f i x d a n d m _ {G} = \Theta (m _ {S}) ,} \\ \frac {\log (m _ {S})}{\sqrt {m _ {S}}} & \text {i f f i x d a n d m _ {G} = m _ {G , o r d e r} ^ {*}} \\ d & \text {i f f i x m _ {S}}. \end{array} \right. \tag {2}
$$
Remark. Explicit upper bound of generalization error. (19) in Appendix B.2 gives us an explicit form to predict the generalization error in the bGMM setting. In Section 4.1, we will see that (19) predicts the order and trend of true generalization error well, which verifies the correctness of the proposed learning guarantee in the bGMM setting.
Remark. Negative learning rate of GDA. Even though we estimate the sufficient statistics of the Gaussian mixture distribution  $(\mu$  and  $\sigma^2)$  directly in this special case, we can not hope to enjoy a better learning rate. Things could be worse when we model the distribution in reality (e.g., images, texts), which suggests that when original samples are abundant, further performing GDA based on existing generative models can not improve the generalization. Theorem 3.3 supports this viewpoint.
Remark. Improvement at a constant level matters a lot when overfitting happens. From (2) we know that when  $m_S$  is small and  $d$  is large, the curse of dimensionality happens, which leads to an awful generalization error. In this case, though GDA can only improve it at a constant level by controlling the generalization error w.r.t. mixed distribution, the effect is obvious due to the large scale of  $d$ . We note that it is challenging to obtain an explicit form of the constant-level improvement due to the complexity of the generalization bound. Therefore, we clarify this more clearly by comparing the cases where  $m_G = 0$  (without GDA) and  $m_G \to +\infty$  in Corollary B.1 of Appendix B.2.
3.3 Implications on deep generative models
Nowadays, data augmentation with deep generative models is widely used and received lots of attention. Therefore, benefiting from the recent advances in the generative adversarial network (GAN) [2; 49] and SGD [35; 50], we discuss implications of our theory on real problems, which will be verified by the empirical experiments in Section 4.2.
3.3.1 Learning setup
We consider the general binary classification task in the deep learning era. In this part, we introduce the setup of data distribution, deep neural classifier, learning algorithm, and deep generative model.
Distribution setting. We assume that input space satisfies  $\mathcal{X} \subseteq [0,1]^d$ , and our analysis can be easily extended to any bounded input space. This assumption generally holds in many practical problems, for example, image data satisfies  $\mathcal{X} \subseteq [0,255]^d$ . Similarly to bGMM, we let  $\mathcal{Y} = \{-1,1\}$  and assume that the distribution of  $y$  is known.
Deep neural classifier. We consider a general  $L$ -layer multi-layer perception (MLP) or convolutional neural network (CNN)  $f(\mathbf{w}, \cdot): \mathcal{Z} \to \mathbb{R}$ , where  $\mathbf{w}$  denotes its weights and  $\mathbf{w}_l$  denotes the weights in the  $l$ -th layer. Its abstract architecture is consistent with that in [50], and details can be found
in Appendix A.1. In addition, we suppose the deep neural classifier satisfies smoothness and boundedness assumptions, which are adopted by many previous works [34; 35; 50; 51].
Assumption 3.1 (Smoothness). We assume that  $f(\mathbf{w}, \cdot)$  is  $\eta$ -smooth with respect to  $\mathbf{w}$ , that is,  $|\nabla f(\mathbf{w}_1, \cdot) - \nabla f(\mathbf{w}_2, \cdot)| \leq \eta \| \mathbf{w}_1 - \mathbf{w}_2 \|_2$  for any  $\mathbf{w}_1$  and  $\mathbf{w}_2$ .
Assumption 3.2 (Boundedness). We assume that for all  $l \in [L]$ , there exists a constant  $W_{l}$  which satisfies  $\| \mathbf{w}_l\| _2 \leq W_l$ .
Learning algorithm for the deep neural classifier. The setting of the learning algorithm is conformed to the practice. We assume that the loss function is the binary cross-entropy loss  $\ell(f, (\mathbf{x}, y)) = \log(1 + \exp(-y f(\mathbf{w}, \mathbf{x})))$  and it is optimized by SGD. For the  $t$ -th step, we set the step size as  $\frac{c}{\eta t}$  for some positive constant  $c$ . Besides, we assume that the total iteration number  $T = O(m_T)$ . These configurations are adopted by past works on the stability of SGD [34; 35].
Deep generative model. We choose GAN as our deep generative model, which is parameterized by MLP. Its abstract architecture is the same as that in Theorem 19 of [49], and details are placed in Appendix A.2. Besides, due to the lack of conditional generative model theory, we make a naive approximation here by assuming that each category is learned by a GAN, respectively.
3.3.2 Theoretical results
Similarly to the bGMM setting, we establish a generalization bound for the deep learning setup. To reach this goal, we bound terms  $M$ ,  $\beta_{m_T}$ , and  $\mathcal{D}_{\mathrm{TV}}(\mathcal{D},\mathcal{D}_G(S))$  based on the recent results on GAN [49] and SGD [29; 50]. First, boundedness and Lipschitzness of classifier  $f$  can be induced from Assumption 3.2 (Lemma B.10). Second, the boundedness of  $f$  directly implies the upper bound for  $M$  because the binary cross-entropy loss is 1-Lipschitz with respect to  $f$ . Third, by combining the Lipschitzness and smoothness of  $f$ , we can bound  $\beta_{m_T}$  for SGD (Lemma B.11). Finally,  $\mathcal{D}_{\mathrm{TV}}(\mathcal{D},\mathcal{D}_G(S))$  can be bounded by the result in [49] (Lemma B.12).
Theorem 3.3 (Generalization bound for GAN, proof in Appendix B.3). Consider the setup introduced in Section 3.3.1. Given a set  $S$  with  $m_S$  i.i.d. samples from any distribution  $\mathcal{D}$  and an augmented set  $S_G$  with  $m_G$  i.i.d. examples sampled from the distribution  $\mathcal{D}_G(S)$  learned by GANs, then for any fixed  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , it holds that
$$
\mathbb {E} | \text {G e n - e r r o r} | \lesssim \left\{ \begin{array}{l l} \frac {1}{\sqrt {m _ {S}}} & \text {i f f i x W , L , d , l e t m _ {G} = 0 ,} \\ \max  \left(\left(\frac {\log (m _ {S})}{m _ {S}}\right) ^ {\frac {1}{4}}, \log m _ {S} \mathfrak {T} (m _ {S}, m _ {G})\right) & \text {i f f i x W , L , d , l e t m _ {G} = \Theta (m _ {S}) ,} \\ \left(\frac {\log (m _ {S})}{m _ {S}}\right) ^ {\frac {1}{4}} & \text {i f f i x W , L , d , l e t m _ {G} = m _ {G , o r d e r} ^ {*}}, \\ d L ^ {2} \left(\prod_ {l = 1} ^ {L} \| W _ {l} \| _ {2}\right) ^ {2} & \text {i f f i x m _ {S} .} \end{array} \right.
$$
Remark. Slow learning rate with GDA. Upper bounds in Theorem 3.3 show that when we perform GDA, the order with regard to  $m_S$  strictly becomes worse. Therefore, it implies that when  $m_S$  is large enough, it is hopeless to boost the performance obviously by augmenting the train set based on GANs. On the contrary, GDA may make the generalization worse.
Remark. GDA matters a lot when overfitting happens. From Theorem 3.3, we know that as the data dimension and model capacity become larger, the deep neural classifier trained with SGD becomes easier to overfit the train set and gain terrible generalization performance. In this case, a constant-level improvement of generalization caused by GDA will be significant. Similarly to the bGMM setting, we clarify the constant-level improvement by comparing the cases where  $m_{G} = 0$  (without GDA) and  $m_{G} \to +\infty$  in Corollary B.2 of Appendix B.3.
In this section, we conduct experiments to verify the results in Section 3, which are two-folded:
- We conduct simulations in the setting of bGMM and validate the results in Theorem 3.2.
- We empirically study the effect of GDA on the real CIFAR-10 dataset [52], which supports our theoretical implications on GANs.
(a)  $d = 1$  ,truth
(b)  $(d,m_S) = (1,40)$ , truth
(c)  $(d,m_S) = (1,40)$ , prediction
(d)  $m_S = 10$  ,truth
(e)  $(d,m_S) = (50,10)$ , truth
(f)  $(d,m_S) = (50,10)$ , prediction
Figure 1: Simulations results on the bGMM setting. We do not integrate the truths and predictions due to the difference between their y-axis scale.
4.1 Simulations on bGMM
We let  $\pmb{\mu} = (1 / \sqrt{d},\dots ,1 / \sqrt{d})^{\top}$  to satisfy  $\| \pmb {\mu}\| _2 = 1$ $\sigma^2 = 0.6^2$  , and randomly generate 10,000 samples according to the Gaussian mixture distribution as the test set. We approximate the Gen-error by the gap between the training error and the test error. To eliminate randomness, we average over 1,000 random runs and report the mean results. We denote  $\gamma = m_G / m_S$  in this section.
First, we investigate the case that data dimension  $d$  is fixed. To verify the order is near to  $O(1 / \sqrt{m_S})$  ( $\log m_S$  can be ignored with respect to  $\sqrt{m_S}$ ), we fix  $d = 1$ , and change  $m_S$  from 20 to 500. For each selected  $m_S$ , we adjust  $\gamma$  from 0 to 50 to generate new samples in different levels. The result is presented in Figure 1a, which shows that the generalization error decreases in a near  $O(1 / \sqrt{m_S})$  order. Besides, generalization error without GDA is always (near) optimal, which empirically proves that GDA is ineffective when  $m_S$  is large enough.
Second, we conduct simulations in the case that  $m_S$  is fixed as a small constant. To verify the order is  $O(d)$ , we fix  $m_S = 10$ , and change  $d$  from 2 to 100. For each selected  $d$ , we also adjust  $\gamma$  from 0 to 50. The result is displayed in Figure 1d, which shows that the generalization error increases in a  $O(d)$  order. In addition, when  $d$  is large (e.g., 100) and the curse of dimensionality happens, generalization error with larger  $\gamma$  is better by a big margin, which suggests that though GDA could only enhance it at a constant level, the effect is significant when overfitting occurs.
Third, we design experiments to validate whether the upper bound in Theorem 3.2 can predict the trend of generalization error well. Similarly to previous theoretical works (e.g. [53]), we find an approximation of (19) in Appendix B.2 as our prediction by replacing  $\log(a / \delta)$  with  $\log(a)$  if  $a \neq 1$  else 1. We plot the ground truths and predictions in the case that  $(d, m_S) = (1, 40)$  and  $(50, 10)$ , respectively. Results in Figure 1 show that our bound predicts the trend of generalization error well. Therefore, an approximation of the optimal augmentation size  $m_G^*$  can be found by minimizing (19).
4.2 Empirical results on CIFAR-10
In this part, we conduct experiments on the real CIFAR-10 dataset with ResNets [54] and various deep generative models, including conditional DCGAN (cDCGAN) [55], StyleGAN2-ADA [56] and elucidating diffusion model (EDM) [30]. Details of experiments (e.g. motivation, model architecture, training) can be found in Appendix D.
To validate our theoretical implications in Section 3.3, we are supposed to discuss two cases, where one  $m_S$  is small and the other  $m_S$  is large. The two cases can be approximated by whether performing another data augmentation. We additionally use the standard data augmentation in [54] to approximate the case with large  $m_S$ . Then, for each selected ResNet and generative model, we set  $m_G$  from 0 to
Table 1: Accuracy on the CIFAR-10 test set, where S.A. denotes standard augmentation.
Generator	Classifier	S.A.	GDA (mg)
0	100k	300k	500k	700k	1M
cDCGAN [55]	ResNet18	×	85.76	86.8	87.83	87.59	87.52	86.47
✓	94.4	93.92	93.41	93.81	93.01	92.6
ResNet34	×	85	86.9	87.93	87.56	87.17	86.28
✓	94.59	94.83	94.21	93.64	93.69	93.18
ResNet50	×	82.85	87.49	88.59	86.67	86.3	85.2
✓	94.69	94.43	93.86	93.74	93.12	92.63
StyleGAN2-ADA [56]	ResNet18	×	85.76	90.22	91.33	91.37	91.25	91.38
✓	94.4	94.68	94.46	94.4	94.11	94.12
ResNet34	×	85	90.24	91.23	91.45	91.56	90.91
✓	94.59	95.05	94.9	94.4	94.43	94.21
ResNet50	×	82.85	90.85	92.29	92.29	92.29	91.61
✓	94.69	94.74	95.04	94.56	94.76	94.28
EDM [30]	ResNet18	×	85.76	92.8	94.87	95.43	96.24	96.28
✓	94.4	96.15	96.74	97.09	97.28	97.5
ResNet34	×	85	93.42	94.93	95.59	96.14	96.44
✓	94.59	96.47	96.96	97.36	97.53	97.51
ResNet50	×	82.85	93.29	95.29	95.95	96.1	96.64
✓	94.69	96.09	96.87	97.28	97.6	97.74
1M and record the accuracy of the trained classifier on the CIFAR-10 test set. Results are presented in Table 1. We further empirically verify our theory by estimating the generalization error directly. By definition, given a trained neural classifier, the generalization error of Theorem 3.3 can be estimated by the absolute gap between the mean cross-entropy loss on the training set (with generated data) and the mean cross-entropy loss on the test set. We place the results and analysis with this estimator at Table 3 in Appendix D.7. In the following, we interpret our experimental results.
GANs improve the test performance of classifiers when overfitting occurs. When standard augmentation is not used, ResNets trained on the train set consistently suffer from overfitting. However, this can be relieved by data augmentation based on GANs, though cDCGAN can not generate high-quality images. This phenomenon supports the implications from Theorem 3.3.
We can not have an obvious improvement by using GANs when  $m_S$  is approximately large. When standard augmentation is used, deep neural classifiers trained on the CIFAR-10 dataset achieve non-trivial performance. In this case, GDA with cDCGAN always damages the generalization ability. Though we use StyleGAN2-ADA, which achieves state-of-the-art conditional image generation performance on the CIFAR-10 dataset, we can not boost the performance of classifiers obviously, and even consistently obtain worse test accuracy when  $m_G$  is 500k or 1M.
Diffusion probabilistic models are promising for GDA. As diffusion models show their excellent ability on image generation, a natural question emerges: are diffusion models more suitable for GDA? We choose the EDM that achieves state-of-the-art FID scores as the generator. Table 1 in Appendix D.7 shows that EDM improves the test accuracy obviously, even though the standard augmentation has been utilized. This suggests that diffusion models enjoy  $\mathcal{D}_{\mathrm{TV}}\left(\mathcal{D},\mathcal{D}_G(S)\right)$  with a faster convergence rate than GANs, and shows the promise of diffusion models in GDA.
Data augmentation practice and theory. Data augmentation [57; 58] is a universal method to improve the generalization ability of deep neural networks in the case of insufficient training data. Classical data augmentation methods include geometric transformations [54], color space transformations [59], kernel filters [60], mixing images [61], random erasing [62], feature space augmentation [63], etc. There are also many theoretical works studying the effect of classical data augmentation methods from different perspectives [64; 65; 66; 67; 68].
With the advance of deep generative models, GDA becomes a novel and promising data augmentation technique. For example, [16] shows that augmenting the ImageNet training set [69] with samples from the conditional diffusion models significantly improves the classification accuracy. However, little work has investigated the theory of GDA. Both empirical success and theoretical opening encourage us to study the role of GDA.
Algorithmic stability theory. Classical results [25; 26] introduced dually in Section 2 has various extensions. Prominent work [34] focuses on the uniform stability of SGD and derive generalization bounds for it. [35] improves the results in [34] and obtains tight guarantees for the stability of SGD, which is used in Theorem 3.3.
Establishing stability bounds under non-i.i.d. settings has also received a surge of interest in recent years. A major line models the dependencies by mixing models [70; 71] and derives stability bounds with mixing coefficients [27; 28; 72]. However, it is usually difficult to estimate the mixing coefficients quantitatively. To avoid this problem, another line qualitatively models the dependencies by graphs. Recently, [29] derive a general stability bound for dependent settings characterized by forest complexity of the dependency graph. However, it is hard to use these techniques to derive a better bound than Theorem 3.1 for GDA, which is discussed dually in Appendix C.
Convergence of deep generative models. In addition to the bound for  $\mathcal{D}_{\mathrm{TV}}\left(\mathcal{D},\mathcal{D}_G(S)\right)$  with respect to GANs [49] we used in Theorem 3.3, there are attempts to derive such a bound for diffusion models [73; 74; 75; 76]. Informally, they mainly assume that estimation error of score function is bounded, then with an appropriate choice of step size and iteration number, diffusion models output a distribution which is close to the true distribution. However, it is still unclear how to derive learning guarantees with respect to the train set size  $m_S$  directly. Once such learning guarantees are established, we can directly analyze the effect of GDA with diffusion models by Theorem 3.1.
6 Impacts and limitations
This paper is mainly a theoretical work and a first step towards understanding the GDA, and it can give some insights to the practice. Theorem 3.1 implies that improving the distribution approximation performance of the generative models is important for the GDA, which motivates people to design better generative models. Besides, it shows that stabilizing the training of the generative models can bring benefits to the GDA, which motivates us to improve the stability of the training of generative models (e.g. GAN). Furthermore, it implies that if we can estimate terms in the bound, then the optimal augmentation size can be approximated. With the emergence of more advanced theory, our results can be extended to other settings (e.g., diffusion models, self-supervised learning) and give more guidance to the practice. One limitation is that our results do not enjoy tightness guarantees, though they benefit from the same nearly optimal guarantee shown by [26] when  $m_{G} = 0$ . The derivation of lower bounds can be left to future work.
In this paper, we attempt to understand modern GDA techniques. To realize this goal, we first establish a general algorithmic stability bound in this non-i.i.d. setting. It suggests that GDA enjoys a faster learning rate when the divergence term  $\mathcal{D}_{\mathrm{TV}}\left(\mathcal{D},\mathcal{D}_G(S)\right) = o(\max \left(\log (m)\beta_m,1 / \sqrt{m}\right))$ . Second, We specify the learning guarantee to the bGMM and GANs settings. Theoretical results show that, in both cases, though GDA can not enjoy a faster learning rate, it is effective when terrible overfitting happens, which suggests its promise in learning with limited data. Finally, experimental results support our theoretical conclusions and further show the promise of diffusion models in GDA.
This work was supported by NSF of China (Nos. 62076145, 62206159); Beijing Outstanding Young Scientist Program (No. BJWZYJH012019100020098); Shandong Provincial Natural Science Foundation (No. ZR2022QF117); Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (No. 22XNKJ13); the Fundamental Research Funds of Shandong University. C. Li was also sponsored by Beijing Nova Program (No. 20220484044).
[1] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[2] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[4] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[5] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, pages 8821-8831, 2021.
[6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877-1901, 2020.
[8] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1):5485-5551, 2020.
[9] Christoph Leiter, Ran Zhang, Yanran Chen, Jonas Belouadi, Daniil Larionov, Vivian Fresen, and Steffen Eger. Chatgpt: A meta-analysis after 2.5 months. CoRR, abs/2302.13795, 2023.
[10] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. CoRR, abs/2208.10442, 2022.
[11] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. CoRR, abs/2303.06555, 2023.
[12] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
[13] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In ECCV, volume 11206, pages 218-234, 2018.
[14] Toan Tran, Trung Pham, Gustavo Carneiro, Lyle J. Palmer, and Ian D. Reid. A bayesian data augmentation approach for learning deep models. In NIPS, pages 2797-2806, 2017.
[15] Shin'ya Yamaguchi, Sekitoshi Kanai, and Takeharu Eda. Effective data augmentation with multi-domain learning gans. In AAAI, pages 6566-6574, 2020.