SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chenyu Zheng $^{1,2}$ , Guoqiang Wu $^{3}$ , Chongxuan Li $^{1,2*}$
$^{1}$  Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China  
 $^{2}$  Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China  
 $^{3}$  School of Software, Shandong University, Shandong, China  
{chenyu.zheng666, guoqiangwu90}@gmail.com; chongxuanli@ruc.edu.cn
Generative data augmentation, which scales datasets by obtaining fake labeled examples from a trained conditional generative model, boosts classification performance in various learning tasks including (semi-)supervised learning, few-shot learning, and adversarially robust learning. However, little work has theoretically investigated the effect of generative data augmentation. To fill this gap, we establish a general stability bound in this not independently and identically distributed (non-i.i.d.) setting, where the learned distribution is dependent on the original train set and generally not the same as the true distribution. Our theoretical result includes the divergence between the learned distribution and the true distribution. It shows that generative data augmentation can enjoy a faster learning rate when the order of divergence term is  $o(\max \left( \log(m) \beta_m, 1 / \sqrt{m} \right))$ , where  $m$  is the train set size and  $\beta_m$  is the corresponding stability constant. We further specify the learning setup to the Gaussian mixture model and generative adversarial nets. We prove that in both cases, though generative data augmentation does not enjoy a faster learning rate, it can improve the learning guarantees at a constant level when the train set is small, which is significant when the awful overfitting occurs. Simulation results on the Gaussian mixture model and empirical results on generative adversarial nets support our theoretical conclusions. Our code is available at https://github.com/ML-GSAI/Understanding-GDA.
Deep generative models [1; 2; 3; 4] have achieved great success in many fields, including computer vision [5; 6], natural language processing [7; 8; 9], and cross-modal learning [10; 11; 12] in the recent years. A promising usage built upon them is generative data augmentation (GDA) [13; 14; 15], which scales the train set by producing synthetic examples with labels based on advanced conditional generative models. Empirically, it has been observed that GDA can improve classification performance in lots of settings, including supervised learning [16; 17], semi-supervised learning [18; 19; 20], few-shot learning [21], zero-shot learning [22], adversarial robust learning [23; 24], etc.
Although promising algorithms and applications of GDA emerge in different learning setups, our experiments in Section 4 show that GDA does not always work, such as in the case with a rich train set or standard augmentation methods (e.g., flip). Besides, the number of augmented data has a significant impact on the performance while is often tuned manually. These phenomena motivate us to study the effect of GDA. Unfortunately, little work has investigated this technique from a theoretical perspective. Therefore, in this paper, we take a first step towards understanding it. Specially, we consider the supervised classification setting, and try to answer the following questions rigorously:
- Can we establish learning guarantees for GDA and explain when it works precisely?
*Correspondence to Chongxuan Li.
Our first main contribution is to propose a general algorithmic stability bound for GDA in Section 3.1. The main technical challenge is that GDA breaks the primary i.i.d. assumption of the classical results [25; 26] because the distribution learned by the generative model is dependent on the sampled train set and generally not the same as the true distribution. Besides, it is unclear whether the existing general non-i.i.d. stability bounds [27; 28; 29] are suitable to derive meaningful guarantees for GDA. Informally, our result (Theorem 3.1) can be presented as follows:
$|\text{Gen-error}|\lesssim$  distributions' divergence  $^+$  generalization error w.r.t. mixed distribution,
where Gen-error means the generalization error of GDA, and  $a \lesssim b$  means  $a = O(b)$ . The distributions' divergence term on the right hand is caused by the divergence between the distribution learned by the generative model and the true distribution. In addition, the remaining generalization error w.r.t. mixed distribution vanishes as we increase the augmentation size. Comparing this bound to the classical result without GDA (Theorem 2.1), we can obtain an exact condition for GDA to be effective: GDA can enjoy a faster learning rate when the order of divergence term is  $o(\max \left(\log(m)\beta_m, 1/\sqrt{m}\right))$ , where  $m$  is the train set size and  $\beta_m$  is the corresponding uniform stability constant. This means the performance of the chosen generative model matters a lot.
Our second main contribution is to particularize the general results to the binary Gaussian mixture model (bGMM) and generative adversarial nets (GANs) [2] in Section 3.2 and Section 3.3, respectively. Our theoretical results (Theorems 3.2 and 3.3) show that, in both cases, the order of the divergence term in the obtained upper bound is  $\Omega \left(\max \left(\log (m)\beta_{m},1 / \sqrt{m}\right)\right)$ . Therefore, when the train set size is large enough, it is hopeless to use GDA to boost the classification performance by a large margin. Worse still, GDA may damage the generalization of the learning algorithm. However, when the train set size is small and awful overfitting happens, GDA can improve the learning guarantee at a constant level, which is significant in this situation. These theoretical implications show the promise of GDA in real-world problems with limited data.
Finally, experiments presented in Section 4 validate our theoretical findings. In particular, in the bGMM setting, experimental results show that our generalization bound (Theorem 3.2) predicts the order and trend of true generalization error well. Besides, in our empirical study on the real image dataset, we find that GANs can not boost the test performance obviously and even damage the generalization when standard data augmentation methods are used to approximate the case with a large train set. In contrast, GANs improve the performance by a large margin when the train set size is small and terrible overfitting occurs. All these experimental results support our theoretical implications in Section 3. Furthermore, we also conduct experiments with the state-of-the-art diffusion model [30]. Empirical results show the promise of the diffusion model in GDA and suggest it could have a faster learning rate than GAN.
Let  $\mathcal{X} \subseteq \mathbb{R}^d$  be the input space and  $\mathcal{Y} \subseteq \mathbb{R}$  be the label space. We denote by  $\mathcal{D}$  the population distribution over  $\mathcal{Z} = \mathcal{X} \times \mathcal{Y}$ . The  $L_p$  norm of a random variable  $X$  is denoted as  $\| X \|_p = (\mathbb{E}|X|^p)^{\frac{1}{p}}$ . Given a set  $S = \{\mathbf{z}_1, \mathbf{z}_2, \ldots, \mathbf{z}_m\}$ , we define  $S^{\backslash i}$  as the set after removing the  $i$ -th data point in the set  $S$ , and  $S^i$  as the set after replacing the  $i$ -th data point with  $\mathbf{z}_i'$  in the set  $S$ . Let  $[m] = \{1, 2, \ldots, m\}$ , then for every set  $V \subseteq [n]$ , we define  $S_V = \{\mathbf{z}_i : i \in V\}$ . In addition, for some function  $f = f(S)$ , we denote its conditional  $L_p$  norm with respect to  $S_V$  by  $\| f \|_p (S_V) = (\mathbb{E}[||f||^p | S_V])^{\frac{1}{p}}$ . Besides, we denote the total variation distance by  $\mathcal{D}_{\mathrm{TV}}$  and KL divergence by  $\mathcal{D}_{\mathrm{KL}}$ , respectively.
We let  $(\mathcal{Y})^{\mathcal{X}}$  be the set of all measurable functions from  $\mathcal{X}$  to  $\mathcal{Y}$ ,  $\mathcal{A}$  be a learning algorithm and  $\mathcal{A}(S) \in (\mathcal{Y})^{\mathcal{X}}$  be the hypothesis learned on the dataset  $S$ . Given a learned hypothesis  $\mathcal{A}(S)$  and a loss function  $\ell: (\mathcal{Y})^{\mathcal{X}} \times \mathbb{Z} \to \mathbb{R}_{+}$ , the true error  $\mathcal{R}_{\mathcal{D}}(\mathcal{A}(S))$  with respect to the data distribution  $\mathcal{D}$  is defined as  $\mathbb{E}_{\mathbf{z} \sim \mathcal{D}}[\ell(\mathcal{A}(S), \mathbf{z})]$ . In addition, the corresponding empirical error  $\widehat{\mathcal{R}}_S(\mathcal{A}(S))$  is defined as  $\frac{1}{m} \sum_{i=1}^{m} \ell(\mathcal{A}(S), \mathbf{z}_i)$ .
- Can we obtain theoretical insights on hyperparameters like the number of augmented data?
2.2 Generative data augmentation
In this part, we describe the process of GDA in a mathematical way. Given a training set  $S$  with  $m_S$  i.i.d. examples from  $\mathcal{D}$ , we can train a conditional generative model  $G$ , and denote the model distribution by  $\mathcal{D}_G(S)$ . We note that the randomness from training the generative model is ignored in this paper. In addition, we define the expectation of the model distribution with regard to  $S$  as  $\mathcal{D}_G = \mathbb{E}_S[\mathcal{D}_G(S)]$ . Based on the trained generative model, we can then obtain a new dataset  $S_G$  with  $m_G$  i.i.d. samples from  $\mathcal{D}_G(S)$ , where  $m_G$  is a hyperparameter. Typically, we consider the case that  $m_G = \Omega(m_S)$  if GDA is utilized. We denote the total number of the data points in augmented set  $\widetilde{S} = S \cup S_G$  by  $m_T$ . Besides, we define the mixed distribution after augmentation as  $\widetilde{\mathcal{D}}(S) = \frac{m_S}{m_T} \mathcal{D} + \frac{m_G}{m_T} \mathcal{D}_G(S)$ . As a result, a hypothesis  $\mathcal{A}(\widetilde{S})$  can be learned on the augmented dataset  $\widetilde{S}$ . To understand the effect of GDA, we are interested in the generalization error  $|\mathcal{R}_{\mathcal{D}}(\mathcal{A}(\widetilde{S})) - \widehat{\mathcal{R}}_{\widetilde{S}}(\mathcal{A}(\widetilde{S}))|$  with regard to the learned hypothesis  $\mathcal{A}(\widetilde{S})$ . For convenience, we denote it by Gen-error in the remaining paper. Technically, we establish bounds for Gen-error using the algorithmic stability introduced in the next subsection. As far as we know, this is the first work to investigate the learning guarantees for GDA.
2.3 Generalization via algorithmic stability
Algorithmic stability analysis is a important tool to provide guarantees for the generalization of machine learning models. A key advantage of stability analysis is that it exploits particular properties of the algorithm and provides algorithm-dependent bounds. Different notations of stability have been proposed and used to establish high probability bounds for generalization error [25; 31; 32; 33]. Among them, uniform stability is the most widely used and has been utilized to analyze the generalization of many learning algorithms, including regularized empirical risk minimization (ERM) algorithms [25] and stochastic gradient descent (SGD) [32; 34; 35]. The uniform stability is defined as the following.
Definition 2.1 (Uniform stability). Algorithm  $\mathcal{A}$  is uniformly  $\beta_{m}$ -stable with respect to the loss function  $\ell$  if the following holds
$$
\forall S \in \mathcal {Z} ^ {m}, \forall \mathbf {z} \in \mathcal {Z}, \forall i \in [ m ], \sup  _ {\mathbf {z}} \left| \ell (\mathcal {A} (S), \mathbf {z}) - \ell (\mathcal {A} (S ^ {i}), \mathbf {z}) \right| \leq \beta_ {m}.
$$
Given a  $\beta_{m}$ -stable learning algorithm, the milestone work [25] provides a high probability generalization bound that converges when  $\beta_{m} = o(1 / \sqrt{m})$ . This condition may fail to hold in some modern machine learning settings [36], which leads to meaningless guarantees. In recent years, some works [37; 38; 26] improved the classical bound by establishing novel and tighter concentration inequalities. Especially, [26] proposed a moment bound and obtained a nearly optimal generalization guarantee, which only requires  $\beta_{m} = o(1 / \log m)$  to converge. It is listed in the next theorem.
Theorem 2.1 (Corollary 8, [26]). Assume that  $\mathcal{A}$  is a  $\beta_{m}$ -stable learning algorithm and the loss function  $\ell$  is bounded by  $M$ . Given a training set  $S$  with  $m$  i.i.d. examples sampled from the distribution  $\mathcal{D}$ , then for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , it holds that
$$
\left| \mathcal {R} _ {\mathcal {D}} (\mathcal {A} (S)) - \widehat {\mathcal {R}} _ {S} (A (S)) \right| \lesssim \log (m) \beta_ {m} \log \left(\frac {1}{\delta}\right) + M \sqrt {\frac {1}{m} \log \left(\frac {1}{\delta}\right)}. \tag {1}
$$
We note that all generalization bounds mentioned above require a primary condition: data points are drawn i.i.d. according to the population distribution  $\mathcal{D}$ . However, it no longer holds in the setting of GDA. On the one hand, the distribution  $\mathcal{D}_G(S)$  learned by the generative model is generally not the same as the true distribution  $\mathcal{D}$ . On the other hand, the learned  $\mathcal{D}_G(S)$  is heavily dependent on the sampled dataset  $S$ . This property brings obstacles to the derivation of the generalization bound for GDA. Furthermore, though there exists some non-i.i.d. stability bounds [27; 28; 29], it is still unclear whether these techniques are suitable in the GDA setting.
3 Main theoretical results
In this section, we present our main theoretical results. In Section 3.1, we establish a general generalization bound (Theorem 3.1) for GDA. Built upon the general learning guarantee, we then
particularize the learning setup to the bGMM introduced in Section 3.2.1 and derive a specified generalization bound (Theorem 3.2). Finally, we discuss our theoretical implications on GANs in real-world problems (Theorem 3.3). Notably, to the best of our knowledge, this is the first work to investigate the generalization guarantee of GDA.
3.1 General generalization bound
To understand GDA, we are interested in studying the generalization error of the hypothesis  $\mathcal{A}(\widetilde{S})$  learned on the dataset  $\widetilde{S}$  after augmentation. Formally, we need to bound  $|\mathcal{R}_{\mathcal{D}}(\mathcal{A}(\widetilde{S})) - \widehat{\mathcal{R}}_{\widetilde{S}}(\mathcal{A}(\widetilde{S}))|$ , which has been defined as Gen-error in Section 2.2. Recall that  $\widetilde{\mathcal{D}}(S)$  has been defined as the mixed distribution after augmentation, to derive such a bound, we first decomposed Gen-error as
$$
| \text {G e n - e r r o r} | \leq \underbrace {\left\lfloor \mathcal {R} _ {\mathcal {D}} (\mathcal {A} (\widetilde {S})) - \mathcal {R} _ {\widetilde {\mathcal {D}} (S)} (\mathcal {A} (\widetilde {S})) \right\rfloor} _ {\text {D i s t r i b u t i o n s} ^ {\prime} \text {d i v e r g e n c e}} + \underbrace {\left\lfloor \mathcal {R} _ {\widetilde {\mathcal {D}} (S)} (\mathcal {A} (\widetilde {S})) - \widehat {\mathcal {R}} _ {\widetilde {S}} (\mathcal {A} (\widetilde {S})) \right\rfloor} _ {\text {G e n e r a l i z t i o n e r r o r w . r . t . m i x e d d i s t r i b u t i o n}}.
$$
The first term on the right hand can be bounded by the divergence (e.g.,  $\mathcal{D}_{\mathrm{TV}},\mathcal{D}_{\mathrm{KL}}$ ) between the mixed distribution  $\widetilde{\mathcal{D}}(S)$  and the true distribution  $\mathcal{D}$ . It is heavily dependent on the ability of the chosen generative model. For the second term, we note that classical stability bounds (e.g. Theorem 2.1) can not be used directly, because points in  $\widetilde{S}$  are drawn non-i.i.d.. We mainly use a core property of  $\widetilde{S}$ , that is,  $S$  satisfies the i.i.d. assumption, and  $S_{G}$  satisfies the conditional i.i.d. assumption when  $S$  is fixed. Inspired by this property, we furthermore decompose this term and utilize sharp moment inequalities [39; 26] to obtain an upper bound. Finally, we conclude with the following result.
Theorem 3.1 (Generalization bound for GDA, proof in Appendix B.1). Assume that  $\mathcal{A}$  is a  $\beta_{m}$ -stable learning algorithm and the loss function  $\ell$  is bounded by  $M$ . Given an set  $\widetilde{S}$  augmented as described in Section 2.2, then for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , it holds that
$$
\begin{array}{l} | \text {G e n - e r r o r} | \lesssim \underbrace {\frac {m _ {G}}{m _ {T}} M \mathcal {D} _ {\mathrm {T V}} \left(\mathcal {D} , \mathcal {D} _ {G} (S)\right)} _ {\text {D i s t r i b u t i o n s} ^ {\prime} \text {d i v e r g e n c e}} + \frac {M (\sqrt {m _ {S}} + \sqrt {m _ {G}}) + m _ {S} \sqrt {m _ {G}} \beta_ {m _ {T}}}{m _ {T}} \sqrt {\log \left(\frac {1}{\delta}\right)} \\ + \frac {\beta_ {m _ {T}} (m _ {S} \log m _ {S} + m _ {G} \log m _ {G}) + m _ {S} \log m _ {S} M \mathfrak {T} (m _ {S} , m _ {G})}{m _ {T}} \log \left(\frac {1}{\delta}\right), \\ \end{array}
$$
where  $\mathfrak{T}(m_S, m_G) = \sup_i \mathcal{D}_{\mathrm{TV}} \left( \mathcal{D}_G^{m_G}(S), \mathcal{D}_G^{m_G}(S^i) \right)$ .
Remark. Tightness of the proposed upper bound. Let  $m_G = 0$ , we observe that Theorem 3.1 degenerates to Theorem 2.1. Therefore, our stability bound includes the i.i.d. setting as a special case and benefits from the same nearly optimal guarantee shown by [26]. Further analysis of the tightness of our guarantee when  $m_G > 0$  is left to future work.
Remark. Comparison with the existing non-i.i.d. stability bounds. Detailed introduction for non-i.i.d. stability bounds is placed in Section 5. We note that previous results [27; 28; 29] are proposed for the general non-i.i.d. case. Therefore, they may fail to give awesome guarantees in this special case. In Appendix C, we show that it is hard to derive a better bound than Theorem 3.1 by using the existing non-i.i.d. stability results directly.
Remark. Stability of the learned distribution  $\mathcal{D}_G(S)$ .  $\mathfrak{T}(m_S, m_G)$  in Theorem 3.1 reflects the stability of the learned distribution with regard to changing one data point in the training set received by the generative model. Our bound suggests that the more stable the model distribution is, the better performance can be achieved by GDA. As far as we know, though uniformly stability of some generative learning algorithms has been studied [40], the new notation  $\mathfrak{T}(m_S, m_G)$  emerging in our bound has not been studied yet.
Remark. Selection of augmentation size. We first consider the order of the upper bound with respect to  $m_S$ . Observing Theorem 3.1, we find that the distributions' divergence term can not be controlled by increasing  $m_G$  while the remaining generalization error w.r.t. mixed distribution will vanish. We note that there exists a trade-off between the fast learning rate and augmentation consumption. Typically, the augmentation consumption is caused by additional sampling, training, and storage. When the order of the divergence term is smaller than that of the remains, increasing  $m_G$  can induce a faster convergence. Otherwise, increasing  $m_G$  can not lead to a faster convergence but a larger
consumption. Therefore, an efficient augmentation size  $m_{G,\text{order}}^*$  with regard to the order of  $m_S$  can be defined as follows:
$$
m _ {G, \text {o r d e r}} ^ {*} = \inf  _ {m _ {G}} \left\{\text {g e n e r a l i z a t i o n e r r o r w . r . t . m i x e d d i s t r i b u t i o n} \lesssim \text {d i s t r i b u t i o n s} \text {’ d i v e r g e n c e} \right\}.
$$
Furthermore, without considering the cost, the optimal augmentation number  $m_G^*$  can be achieved by minimizing the upper bound directly. Unfortunately, it is difficult to calculate an explicit form of  $m_{G,\text{order}}^*$  and  $m_G^*$  here due to the ignorance of  $\beta_{m_T}$  and  $\mathfrak{T}(m_S,m_G)$ . We will discuss them more concretely in the specified cases.
Remark. Sufficient conditions for GDA with (no) faster learning rate. We still consider the order of the learning guarantee with respect to  $m_S$  here. Let  $m_G = m_{G,\text{order}}^*$ , it can be found that divergence  $\mathcal{D}_{\mathrm{TV}}(\mathcal{D},\mathcal{D}_G(S))$  plays an important role in deciding whether GDA can enjoy a faster learning rate. Comparing Theorem 3.1 with Theorem 2.1 (without augmentation), we can conclude sufficient conditions as follows.
Corollary 3.1. Assume that the loss function  $\ell$  is bounded by  $M$ , we have
- if  $\mathcal{D}_{\mathrm{TV}}\left(\mathcal{D},\mathcal{D}_G(S)\right) = o\left(\max \left(\log (m)\beta_m,1 / \sqrt{m}\right)\right)$ , then GDA enjoys a faster learning rate.
- if  $\mathcal{D}_{\mathrm{TV}}\left(\mathcal{D},\mathcal{D}_G(S)\right) = \Omega \left(\max \left(\log (m)\beta_m,1 / \sqrt{m}\right)\right)$ , then GDA can not enjoy a faster learning rate.
Notably, as we will present in Section 3.2 and 3.3, though GDA can not enjoy a faster learning rate in both special case, it is possible to improve the generalization guarantee at a constant level when  $m_S$  is small, which is important when awful overfitting happens.
3.2 Theoretical results on bGMM
The bGMM is a classical but non-trivial setting, which has been widely studied in literature [41; 42; 43]. In this section, we investigated it in the context of GDA. Simulations will be conducted in Section 4.1 to verify these results.
3.2.1 Setting of bGMM
In this part, we introduce the data distribution configuration in the bGMM, as well as the corresponding linear classifier and conditional generative model. Similar setups of distribution and classifier have been adopted by many previous works [44; 45; 46].
Distribution setting. We consider a binary task where  $\mathcal{Y} = \{-1,1\}$ . Given a vector  $\pmb{\mu} \in \mathbb{R}^d (\|\pmb{\mu}\|_2 = 1)$  and noise variance  $\sigma^2 > 0$ , we assume that the distribution satisfies  $y \sim \mathrm{uniform}\{-1,1\}$  and  $\mathbf{x} \mid y \sim \mathcal{N}(y\pmb{\mu}, \sigma^2 I_d)$ . Besides, similarly to [47], we assume that the distribution of  $y$  is known, which is satisfied in conditional learning with labels.
Simple linear classifier. We consider a linear classifier parameterized by  $\pmb{\theta} \in \mathbb{R}^d$  in the form of prediction  $\widehat{y} = \mathrm{sign}(\pmb{\theta}^\top \mathbf{x})$ . Given  $m$  samples,  $\pmb{\theta}$  is learned by performing ERM with respect to the negative log-likelihood loss function, that is,
$$
l (\boldsymbol {\theta}, (\mathbf {x}, y)) = \frac {1}{2 \sigma^ {2}} (\mathbf {x} - y \boldsymbol {\theta}) ^ {\top} (\mathbf {x} - y \boldsymbol {\theta}).
$$
As a result, this learning algorithm will return  $\widehat{\pmb{\theta}} = \frac{1}{m}\sum_{i = 1}^{m}y_{i}\mathbf{x}_{i}$ , which satisfies  $\mathbb{E}[\widehat{\pmb{\theta}}] = \pmb{\mu}$ .
Conditional generative model. We consider a simple generative model parameterized by  $\mu_y, \sigma_k^2$ , where  $y \in \{-1, 1\}$  and  $k \in [d]$ . It learns the parameters of Gaussian mixture distribution directly. Given  $m$  data points, let  $m_y$  be the number of samples in class  $y$ , it returns
$$
\widehat {\boldsymbol {\mu}} _ {y} = \frac {\sum_ {y _ {i} = y} \mathbf {x} _ {i}}{m _ {y}}, \quad \widehat {\sigma} _ {k} ^ {2} = \sum_ {y} \frac {m _ {y}}{m} \frac {\sum_ {y _ {i} = y} (x _ {i k} - \widehat {\mu} _ {y k}) ^ {2}}{m _ {y} - 1},
$$
which are unbiased estimators of  $\pm \pmb{\mu}$  and  $\sigma^2$ , respectively. Based on the learned parameters, we can perform GDA by generating new samples from the distribution  $y \sim \mathrm{uniform}\{-1, 1\}, \mathbf{x} \mid y \sim \mathcal{N}(\widehat{\pmb{\mu}}_y, \widehat{\Sigma})$ , where  $\widehat{\Sigma} = \mathrm{diag}(\widehat{\sigma}_1^2, \ldots, \widehat{\sigma}_d^2)$ .
3.2.2 Theoretical results