SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training. We trained  $f_{\theta}$  by the Nesterov momentum SGD for 200 epochs with a momentum of 0.9, and an initial learning rate of 0.01; we decayed the learning rate by 0.1 at 60, 120, and 160 epochs. We trained  $F_{\phi}$  by the Adam optimizer for 200 epochs with a learning rate of  $1.0 \times 10^{-4}$ . We used mini-batch sizes of 64 for  $\mathcal{D}$  and 64 for  $\mathcal{D}_{\mathrm{p}}$ . The input samples were resized into a resolution of  $224 \times 224$ ;  $x_{\mathrm{p}}$  was resized by differentiable transformations. For synthetic samples from  $G_{\Phi}$  in PCR and GDA+SSL, the strong transformation  $T$  was RandAugment [21] by following [16], and it was implemented with differentiable transformations provided in Kornia [32]. We determined the hyperparameter  $\lambda$  by grid search among [0.1, 1.0] with a step size of 0.1 for each method by  $\mathcal{D}_{\mathrm{val}}$ . To avoid overfitting, we set the hyperparameters of MGR that are searched with only applying PCR i.e., we did not use meta-learning to choose them. We used  $\lambda_{\mathrm{KL}}$  of 0.01. We selected the final model by checking the validation accuracy for each epoch. We ran the experiments three times on a 24-core Intel Xeon CPU with an NVIDIA A100 GPU with 40GB VRAM and recorded average test accuracies with standard deviations evaluated on the final models.
4.2 Evaluation on Multiple Datasets
We confirm the efficacy of MGR across multiple datasets. Table 1a shows the top-1 accuracy scores of each method. As reported in [7], GDA degraded the base model on many datasets; it slightly improved the base model on only one dataset.  $\mathrm{GDA + MH}$ , which had decoupled classifier heads for GDA, exhibited a similar trend to GDA. This indicates that simply decoupling the classifier heads is not a solution to the performance degradation caused by synthetic samples. In contrast, our MGR stably and significantly outperformed the baselines and achieved the best results. The ablation of MGR discarding PCR or MPS is listed in Table 1a. We confirm that both PCR and  $\mathrm{GDA + MPS}$  improve GDA. While  $\mathrm{GDA + SSL}$  underperforms the base models, PCR outperforms the base models. This indicates that using unsupervised loss alone is not sufficient to eliminate the negative effects of the synthetic samples and that discarding the classifier  $h_\omega$  from the regularization is important to obtain the positive effect. MPS yields only a small performance gain when combined with GDA, but it significantly improves its performance when combined with PCR i.e., MGR. This suggests that there is no room for performance improvements in GDA, and MPS can maximize the potential benefits of PCR.
4.3 Evaluation on Small Datasets
A small dataset setting is one of the main motivations for utilizing generative data augmentation. We evaluate the effectiveness of MGR on smaller datasets. Table 1b shows the performance when reducing the Cars dataset into a volume of  $\{10,25,50,75\}\%$ . Note that we trained  $G_{\Phi}$  on each reduced dataset, not on  $100\%$  of Cars. In contrast to the cases of the full dataset (Table 1a), no baseline methods outperformed the base model in this setting. This is because  $G_{\Phi}$  trained on the small datasets generates low-quality samples with less reliability on the conditional label  $y_{\mathrm{p}}$  that are not appropriate in supervised learning. On the other hand, MGR improved the baselines in large margins. This indicates that, even when the synthetic samples are not sufficient to represent the class categories, our MGR can maximize the information obtained from the samples by utilizing them to regularize feature extractors and dynamically finding useful samples.
Figure 5: UMAP visualization of feature spaces on training. The plots in the figures represent real and synthetic samples in the feature spaces. Our methods (PCR and PCR+MPS) can help the feature extractors separate real sample clusters. In contrast, the existing method (GDA) confuses the feature extractor by leaking synthetic samples out of the clusters.
4.4 Visualization of Feature Spaces
In this section, we discuss the effects of PCR and MPS through the visualizations of feature spaces in training. To visualize the output of  $g_{\psi}$  in 2D maps, we utilized UMAP [33] to reduce the dimensions. UMAP is a visualization method based on the structure of distances between samples, and the low dimensional embeddings can preserve the distance between samples of the high dimensional input. Thus, the distance among the feature clusters on UMAP visualization of  $g_{\psi}$  can represent the difficulty of the separation by  $h_{\omega}$ . We used the official implementation by  $[33]^5$  and its default hyperparameters. We plotted the UMAP embeddings of  $g_{\psi}(x)$  and  $g_{\psi}(x_{\mathrm{p}})$  at  $\{5,15,30\}$  epochs, as shown in Figure 5; we used ResNet-18 trained on the Cars dataset. At first glance, we observe that GDA and PCR formed completely different feature spaces. GDA forms the feature spaces by forcing to treat the synthetic samples the same as the real samples through cross-entropy loss and trying to separate the clusters of samples according to the class labels. However, the synthetic samples leaked to the inter-cluster region at every epoch because they could not represent class categories perfectly as discussed in Sec. 1. This means that the feature extractor might be distorted to produce features that confuse the classifier. On the other hand, the synthetic samples in PCR progressively formed a cluster at the center, and the outer clusters can be seen well separated. Since UMAP can preserve the distances between clusters, we can say that the sparse clusters that exist far from the center are considered easy to classify, while the dense clusters close to the center are considered difficult to classify. In this perspective, PCR helps  $g_{\psi}$  to leverage the synthetic samples for learning feature representations interpolating the dense difficult clusters. This is because the synthetic samples tend to be in the middle of clusters due to their less representativeness of class categories. That is, PCR can utilize the partial but useful information contained in the synthetic samples while avoiding the negative effect. Further, we observe that applying MPS accelerates the convergence of the synthetic samples into the center.
For a more straightforward visualization of class-wise features, we designed a simple binary classification task that separates Pets [31] into dogs and cats, and visualized the feature space of a model trained on this task. Fig. 6 shows the feature space after one epoch of training. While GDA failed to separate the clusters for each class, MGR clearly separated the clusters. Looking more closely, MGR helps samples to be dense for each class. This is because PCR makes the feature extractor learn slight differences between the synthetic samples that interpolate the real samples. From these observations, we conclude that our MGR can help models learn useful feature representations for solving tasks.
5 https://github.com/lmcinnes/umap
(a) GDA
(b) MGR
(b) FID
Figure 7: Statistics in training (Cars)
Figure 6: UMAP visualization (ResNet-18). We used the Pets dataset by modifying the class definition to the binary classes (dogs and cats). The visualization results are plotted after one epoch training with 2048 real samples and 2048 synthetic samples.
(a) Uniform
Figure 8: Synthetic samples (class: Hummer)
(b) MPS
4.5 Analysis of MPS
Evaluation of validation loss. We investigate the effects on validation losses when using MPS. Through the meta-optimization by Eq. (7), MPS can generate samples that reduce the validation loss of  $f_{\theta}$ . We recorded the validation loss per epoch when applying uniform sampling (Uniform) and when applying MPS. We used the models trained on Cars and applied the PCR loss on both models. Figure 7a plots the averaged validation losses. MPS reduced the validation loss. In particular, MPS was more effective in early training epochs. This is related to accelerations of converging the central cluster of synthetic samples discussed in Section 4.4 and Figure 5. That is, MPS can produce effective samples for regularizing features and thus speed up the entire training of  $f_{\theta}$ .
Quantitative evaluation of synthetic samples. We evaluate the synthetic samples generated by MPS. To assess the characteristics of the samples, we measured the difference between the data distribution and distribution of the synthetic samples. We leveraged the Fréchet Inception distance (FID, [34]), which is a measurement of the distribution gap between two datasets using the closed-form computation assuming multivariate normal distributions:
$$
\operatorname {F I D} \left(\mathcal {D}, \mathcal {D} _ {\mathrm {p}}\right) = \| \mu - \mu_ {\mathrm {p}} \| _ {2} ^ {2} + \operatorname {T r} \left(\Sigma + \Sigma_ {\mathrm {p}} - 2 \sqrt {\Sigma \Sigma_ {\mathrm {p}}}\right),
$$
where  $\mu$  and  $\Sigma$  are the mean and covariance of the feature vectors on InceptionNet for input  $\{x^i\}$ . Since FID is a distance, the lower  $\mathrm{FID}(\mathcal{D},\mathcal{D}_{\mathrm{p}})$  means that  $\mathcal{D}_{\mathrm{p}}$  contains more high-quality samples in terms of realness. We computed FID scores using 2048 samples in  $\mathcal{D}$  and 2048 synthetic samples every epoch; the other settings are given in Section 4.1. The FID scores in training are plotted in Figure 7b. We confirm that MPS consistently produced higher-quality samples than Uniform. This indicates that the sample quality is important for generalizing  $f_{\theta}$  even in PCR, and uniform sampling can miss higher quality samples in generative models. Since the performance gain by GDA+MPS in Table 1a did not better than MGR, the higher-quality samples by MPS can still contain uninformative samples for the cross-entropy loss, but they are helpful for PCR learning good feature representations.
Qualitative evaluation of synthetic samples. We evaluate the qualitative properties of the synthetic samples by visualizing samples of a class. The samples generated by Uniform and MPS are shown in Figure 8a and 8b, where the dataset was Cars and the class category was Hummer. Compared with
Table 2: Performance of MGR varying  $G_{\Phi}$  (ResNet-18, Cars).
GΦ	Conditional	FID	Top-1 Acc. (%)
None (Base Model)	-	-	85.50±.10
Real CR	-	-	86.16±.02
FastGAN [35]	No	23.1	86.30±.16
BigGAN [14]	Yes	15.6	86.86±.06
StyleGAN2-ADA [9]	Yes	9.5	87.22±.15
StyleGAN-XL [36]	Yes	6.2	88.37±.20
Table 3: Performance comparison between MGR and data augmentation methods (ResNet-18, Cars, Top-1 Acc.  $(\%)$ ).
Data Augmentation	Base Model	+MGR
None	85.50±.10	87.22±.15
MixUp [37]	86.87±.30	87.60±.46
CutMix [38]	86.13±.19	87.80±.51
AugMix [39]	86.25±.11	87.65±.03
RandAugment [21]	87.47±.05	88.67±.10
SnapMix [40]	87.11±.20	88.21±.13
TrivialAugment [25]	87.83±.16	89.10±.19
Uniform, MPS produced samples with more diverse body colors and backgrounds. That is, MPS focuses on the color and background of the car as visual features for solving classifications. In fact, since Hummer has various colors of car bodies and can drive on any road with four-wheel drive, these selective generations of MPS are considered reasonable in solving classification tasks.
4.6 Effect of Generative Models
Here, we evaluate MGR by varying the generative model  $G_{\Phi}$  for producing synthetic samples. As discussed in Sec. 3.1 and 3.2, MGR can use arbitrary unconditional/conditional generative models as  $G_{\Phi}$  unless it has a latent space. To confirm the effects when changing  $G_{\Phi}$ , we tested MGR with FastGAN [35], BigGAN [18], and StyleGAN-XL [36]. Table 2 shows the results on Cars. The unconditional FastGAN achieved similar improvements as conditional cases. However, since unconditional generative models are generally of low quality in FID, they are slightly less effective. For the conditional generative models, we observe that MGR performance improvement increases as the quality of synthetic samples improves. These results suggest the potential for even greater MGR gains in the future as the performance of the generative model improves. We also evaluate MGR with recent diffusion models in Appendix B.3. Meanwhile, to evaluate the value of synthetic samples in the consistency regularization, we compare MGR with the case where we use real samples in Eq. (6) (Real CR). Our method outperformed Real CR. This can be because interpolation by the generators helps the feature extractor to capture the difference between images. Furthermore, our method searches the optimal synthetic samples by using latent vectors of generative models with MPS. In contrast, Real CR cannot search the optimal real samples for CR loss because real data is fixed in the data space.
4.7 Combination of MGR and Data Augmentation
To assess the practicality of MGR, we evaluate the comparison and combination of MGR and existing data augmentation methods. Data augmentation (DA) is a method applied to real data and is an independent research field from generative data augmentation. Therefore, MGR can be combined with DA to improve performance further. Table 3 shows the evaluation results when comparing and combining MGR and DA methods; we used MixUp [37], CutMix [38], AugMix [39], RandAugment [21], SnapMix [40], and TrivialAugment [25] as the DA methods. The improvement effect of MGR is comparable to that of DA, and the highest accuracy was achieved by combining the two methods. In particular, MGR was stably superior to sample-mix-based such as MixUp and AugMix. This result also indicates that synthetic samples, which non-linearly interpolate the real samples, elicit better performance than linearly interpolating real samples by the sample-mix-based DA methods. We consider this strength to be an advantage of using generative models.
We briefly review generative data augmentation and training techniques using generative models.
The earliest works of generative data augmentation are [19, 41, 42]. They have demonstrated that simply adding synthetic samples as augmented data for classification tasks can improve performance in few-shot learning, person re-identification, and medical imaging tasks. Tran et al.[20] have
proposed a generative data augmentation method that simultaneously trains GANs and classifiers for optimizing  $\theta$  to maximize the posterior  $p(\theta |x)$  by an EM algorithm. Although this concept is similar to our MPS in terms of updating both  $G_{\Phi}$  and  $f_{\theta}$ , it requires training the specialized neural architectures based on GANs. In contrast, MPS is formalized for arbitrary existing generative models with latent variables, and it requires no restrictions to the training objectives of generative models.
On the analysis of generative data augmentation, Shmelkov et al. [7] have pointed out that leveraging synthetic samples as augmented data degrades the performance in general visual classification tasks. They have hypothesized that the cause of the degradation is the less diversity and fidelity of synthetic samples from generative models. Subsequent research by Yamaguchi et al. [8] have shown that the scores related to the diversity and fidelity of synthetic samples (i.e., SSIM and FID) are correlated to the test accuracies when applying the samples for generative data augmentation in classification tasks. Based on these works, our work reconsiders the training objective and sampling method in generative data augmentation and proposes PCR and MPS.
More recently, He et al. [43] have reported that a text-to-image generative model pre-trained on massive external datasets can achieve high performance on few-shot learning tasks. They also found that the benefit of synthetic data decreases as the amount of real training data increases, which they attributed to a domain gap between real and synthetic samples. In contrast, our method does not depend on any external dataset and successfully improves the accuracy of the classifier even when synthetic data are not representatives of class labels (i.e., having domain gaps).
One of the limitations of our method is the requirement of bilevel optimization of classifier and finder networks. This optimization is computationally expensive particularly when used with generative models that require multiple inference steps, such as diffusion models as discussed in Appendix B.3. We have tried other objective functions not requiring bilevel optimization, but at this time, we have not found an optimization method that outperforms MPS (see Appendix B.4). Nevertheless, since recent studies rapidly and intensively focus on the speedup of diffusion models [44, 45, 46], we can expect that this limitation will be negligible in near the future. Additionally, applying MGR to pre-trained text-to-image diffusion models (e.g., [44]) is also important for future work because they have succeeded in producing effective samples for training downstream task models in a zero-shot manner [43].
This paper presents a novel method for generative data augmentation called MGR. MGR is composed of two techniques: PCR and MPS. To avoid the degradation of classifiers, PCR utilizes synthetic samples to regularize feature extractors by the simple consistency regularization loss. MPS searches the useful synthetic samples to train the classifier through meta-learning on the validation loss of main tasks. We empirically showed that MGR significantly improves the baselines and brings generative data augmentation up to a practical level. We also observed that the synthetic samples in existing generative data augmentation can distort the decision boundaries on feature spaces, and the PCR loss with synthetic samples dynamically generated by MPS can resolve this issue through visualization. We consider that these findings will help future research in this area.
We thank the members of the Kashima Laboratory and the PRMU community for discussing the initial concepts of this paper and giving useful advice.
[1] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 2019.
[41] Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
[42] Francesco Calimeri, Aldo Marzullo, Claudio Stamile, and Giorgio Terracina. Biomedical data augmentation using generative adversarial neural networks. In International Conference on Artificial Neural Networks. Springer, 2017.
[43] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574, 2022.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[45] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022.
[46] Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, 2022.
[47] Ron Mokady, Omer Tov, Michal Yarom, Oran Lang, Inbar Mosseri, Tali Dekel, Daniel Cohen-Or, and Michal Irani. Self-distilled stylegan: Towards generation from internet photos. In ACM SIGGRAPH 2022 Conference Proceedings, 2022.
[48] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
The following manuscript provides the supplementary materials of the main paper: Regularizing Neural Networks with Meta-Learning Generative Models.
A Algorithm of Meta Generative Regularization
Algorithm 1 Meta Generative Regularization
Require: Training dataset  $\mathcal{D}$ , validation dataset  $\mathcal{D}_{\mathrm{val}}$  main model  $f_{\theta}$ , generator  $G$ , finder  $F_{\phi}$ , training batchsize  $B$ , pseudo batchsize  $B_{\mathrm{p}}$ , validation batchsize  $B_{\mathrm{val}}$ , step size  $\eta$  and  $\xi$ , hyperparameter  $\lambda$  and  $\lambda_{\mathrm{KL}}$
Ensure: Trained main model  $f_{\theta}$
1: while not converged do
2:  $\{(x^i,y^i)\}_{i = 1}^B\sim \mathcal{D}$
3:  $\{z^i\}_{i = 1}^{B_{\mathrm{P}}}\sim \mathcal{N}(0,I)$
4: //Updating  $\phi$  for MPS
5:  $\{(x_{\mathrm{val}}^{\mathrm{i}},y_{\mathrm{val}}^{\mathrm{i}})\}_{i = 1}^{B_{\mathrm{val}}}\sim \mathcal{D}$
6:  $\{x_{\mathrm{p}}^{i}\}_{i = 1}^{B_{\mathrm{p}}} = \{G_{\Phi}(F_{\phi}(z^{i}),y_{\mathrm{p}}^{i})\}_{i = 1}^{B_{\mathrm{p}}}$
7:  $\theta^{\prime}\gets \theta -\eta \nabla_{\theta}(\frac{1}{B}\ell (f_{\theta}(x^{i}),y^{i}) + \frac{\lambda}{B_{\mathrm{p}}}\ell_{\mathrm{PCR}}(x_{\mathrm{p}}^{i};\psi))$
8:  $\phi \gets \phi -\xi \nabla_{\phi}\left(\frac{1}{B_{\mathrm{val}}}\ell (f_{\theta^{\prime}}(x_{\mathrm{val}}),y_{\mathrm{val}}) + \lambda_{\mathrm{KL}}(D_{\mathrm{KL}}(p_{\phi}(z)\| p(z))))$
9: //Updating  $\theta$  with PCR
10:  $\{x_{\mathrm{p}}^{i}\}_{i = 1}^{B_{\mathrm{p}}} = \{G_{\Phi}(F_{\phi}(z^{i}),y_{\mathrm{p}}^{i})\}_{i = 1}^{B_{\mathrm{p}}}$
11:  $\theta \gets \theta -\eta \nabla_{\theta}\left(\frac{1}{B}\ell (f_{\theta}(x^{i}),y^{i}) + \frac{\lambda}{B_{\mathrm{p}}}\ell_{\mathrm{PCR}}(x_{\mathrm{p}}^{i};\psi)\right)$
12: end while
B.1 Evaluation of Gradient Approximation
Here, we evaluate the gradient approximation by Eq. (9). As shown in Table 4, Eq. (9) well approximated the second-order gradients in speeding up over  $10\%$  with 0.08 of the accuracy drop.
Table 4: Performance comparison between MPS with 2nd-order gradients and 1st-order approximated gradients (ResNet-18, Cars).
Method	Top-1 Acc. (%)	Wall Clock Time (hours)
2nd-Order	87.30±.39	6.55
1st-Order Approx.	87.22±.15	5.79
B.2 Ablation study of  $F_{\phi}$
In Section 3.2, we introduce  $F_{\phi}$  for meta-optimized parameters and the residual architectures with MLP defined by Eq. (10). We performed an ablation study of MPS with respect to the meta-optimized parameters and the architectures of  $F_{\phi}$ . We compared MPS with a variant of MPS optimizing  $G_{\Phi}$  instead of  $F_{\phi}$ . We also attempted other architectures for  $F_{\phi}$  including Linear:  $W_{\phi}(z) + b$ , MLP:  $\mathrm{MLP}_{\phi}(z)$ , and Residual+Shallow:  $z + \tanh(W_{\phi}(z) + b)$ . The results of these variations are shown in Table 5. We observed that MPS with  $G_{\Phi}$  caused failures of training  $f_{\theta}$  and degraded the accuracy. On the other hand, all variants of MPS with  $F_{\phi}$  succeeded in boosting the models without MPS. Thus, restricting the number of optimized parameters is important, and determining an optimal  $z$  with the finder  $F_{\phi}$  is effective on the optimization problems of MPS. For the variants of MPS with  $F_{\phi}$ , we observed that the residual architectures and regularization by  $D_{\mathrm{KL}}(p_{\phi}(z)\| p(z))$  contributed to the successes. Interestingly, MPS with Linear  $F_{\phi}$  outperformed MPS with MLP  $F_{\phi}$ , i.e., significantly transforming the input  $z \sim p(z)$  by complex functions results in low accuracy. These results suggest that better latent vectors in  $\mathcal{Z}$  to train  $f_{\theta}$  can exist near the uniformly sampled input  $z$ . Thus, limiting the search range by tanh in the residual architectures can help in finding better latent vectors.
Table 5: Ablation study of MPS (ResNet-18, Cars).1
Method	Top-1 Acc. (%)
Without MPS (PCR)	86.32±.07
MPS	87.22±.15
MPS with GΦ	84.47±.05
MPS with Linear Fφ	86.51±.09
MPS with MLP Fφ	86.35±.13
MPS with Residual+Shallow Fφ	86.88±.16
MPS w/o DKL(pφ(z)||p(z))	86.92±.22
B.3 MGR with Diffusion Models