Regularizing Neural Networks with Meta-Learning Generative Models
Authors: Shin'ya Yamaguchi
Daiki Chijiwa
Sekitoshi Kanai
Atsutoshi Kumagai
Hisashi Kashima
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Learning
Generative Models
Generative Data Augmentation
Regularization
Meta-Learning
TL;DR: Improving performance of neural networks by regularizing with synthetic samples from meta-learning generative models.

Abstract: This paper investigates methods for improving generative data augmentation for deep learning. Generative data augmentation leverages the synthetic samples produced by generative models as an additional dataset for classification with small dataset settings. A key challenge of generative data augmentation is that the synthetic data contain uninformative samples that degrade accuracy. This can be caused by the synthetic samples not perfectly representing class categories in real data and uniform sampling not necessarily providing useful samples for tasks. In this paper, we present a novel strategy for generative data augmentation called *meta generative regularization* (MGR). To avoid the degradation of generative data augmentation, MGR utilizes synthetic samples for regularizing feature extractors instead of training classifiers. These synthetic samples are dynamically determined to minimize the validation losses through meta-learning. We observed that MGR can avoid the performance degradation of naive generative data augmentation and boost the baselines. Experiments on six datasets showed that MGR is effective particularly when datasets are smaller and stably outperforms baselines by up to 7 percentage points on test accuracy.

Regularizing Neural Networks with Meta-Learning Generative Models
Shin'ya Yamaguchi Daiki Chijiwa' Sekitoshi Kanai"
Atsutoshi Kumagai" Hisashi Kashima 1TT *Kyoto University
Abstract
This paper investigates methods for improving generative data augmentation for deep learning. Generative data augmentation leverages the synthetic samples produced by generative models small dataset settings. A key challenge generative data augmentation is that the synthetic data contain uninformative samples that degrade accuracy. This is because the synthetic samples do not perfectly represent class categories in real data and uniform sampling does not necessarily provide useful samples for tasks. In this paper, we present a novel strategy for generative data augmentation called meta generative regularization (MGR). To avoid the degradation of generative data augmentation, MGR utilizes synthetic samples in the regularization term for feature extractors instead of in the loss function, e.g. cross-entropy. These synthetic samples are dynamically determined to minimize the validation losses through meta-learnig. We observed that MGR can avoid the performance degradation of naïve generative data augmentation and boost the baselines. Experiments on six datasets showed that MGR effective particularly when datasets are smaller and stably outperforms baselines.
additional dataset for classification with
Introduction
While deep neural networks achieved impressive performance on various machine learning tasks, training them still requires large amount of labeled training data in supervised learning. The labeled datasets are expensive when few experts can annotate the data, e.g., medical imaging. In such scenarios, generative data augmentation is a promising option for improving the performance of models. Generative data augmentation basically adds pairs of synthetic samples from conditional generative models and their target labels into real training datasets. The expectations of generative data augmentation are that the synthetic samples interpolate missing data points and perform as oversampling for classes with less real training samples [ This simple method can improve the performance of several tasks with less diversity of inputs such medical imaging tasks [2] 5),
However, in general cases that require the recognition of more diverse inputs (e.g., CIFAR datasets (6],, generative data augmentation degrades rather than improves the test accuracy [7]. Previous studies have indicated that this can be caused by the low quality of synthetic samples in terms of the diversity and fidelity 77 8. If this hypothesis is correct, we can expect high-quality generative models (e.g., StyleGAN2-ADA ()) resolve the problem; existing generative data augmentation methods adopt earlier generative models e.g., ACGAN [TO] and SNGAN [I]]. Contrary to the expectation, this not the case. We observed that generative data augmentation fails to improve models even when using a high-quality StyleGAN2-ADA (Figure Dp. Although the samples partially appear to be real to humans, they are not yet sufficient to train classifiers in
Coorresponding author. Email: e yww.ucchiettttccomm
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Top-1 Accuracy (%)
Regularizing with synthesized samples
70
60
Generative Model
Main Model
Generative Data Augmentatio
(Ours
40
80
80
1001
Dataset Size (%)
Meta-lcarning by validation performance
Figure Accuracy gain using meta generative regularization on Cars I121 with ResNet-18 clas-
Figure 2: Meta generative regularization
sifier 133 and Sty 9 (FID: 9.5)
existing generative data augmentation methods. This paper investigates methodologies for effectively extracting useful information from generative models to improve model performance.
We address this problem based the following hypotheses. First, synthetic samples actually informative but do nol perfectly represent class categories in real data. This is based on a finding by Brock et al. [14] called "class leakage," where class conditional synthetic sample contains attributes of other classes. For example, they observed failure samples including an image of "tennis ball" containing attributes of "dogs" (Figure 4(d) of (14) These class leaked samples do not perfectly represent the class categories in the real dataset. If we use such class leaked samples for updating classifiers, the samples can distort the decision boundaries, shown in Figure 3 Second, regardless of the quality, the generative models originally contain uninformative samples to solve the tasks. This is simply because the generative models are not explicitly optimized to generate informative samples for learning the conditional distribution p[y|æ); they are optimized only for learning the data distribution p(z). Further, the generative models often fail capture the entire data distribution precisely due to their focus on the high-density regions. These characteristics of generative models might disturb the synthesis of effective samples for generative data augmentation. maximize the gain from synthetic samples, we should select appropriate samples for training tasks.
In this paper, we present a novel regularization method called meta generative regularization (MGR). Based on the above hypotheses, MGR is composed of two techniques for improving generative data augmentation: pseudo consistency regularization (PCR) and meta pseudo sampling (MPS). PCR is regularization term using synthetic samples in training objectives for classifiers. Instead of supervised learning with negative log- likelihood log p(y| i.e., cross-entropy, on synthetic samples, we regularize the feature extractor to avoid the distortions on decision boundaries. That is, PCR leverages synthetic samples only for learning feature spaces. PCR penalizes the feature extractors by minimizing the gap between variations of synthetic sample, which is inspired by consistency regularization in MPS corresponds to the second hypothesis and its objective is to select useful samples for training tasks by dynamically searching optimal latent vectors of the generative models. Therefore, we formalize MPS as a bilevel optimization framework of a classifier and finder that is neural network for searching latent vectors. Specifically, this framework updates the finder through meta-learning to reduce the validation loss and then updates the classifier to reduce the PCR loss (Figure 2). By combining PCR and MPS, the existing generative data augmentation degrades the performance (Figure TT.
eemi-supervised learning [5
improve the performance even when
We conducted experiments with multiple vision datasets and observed that MGR can stably improve baselines on various settings by up to percentage points of test accuracy. Further, through the visualization studies, we confirmed that MGR utilizes the information in synthetic samples to learn feature representations through PCR and obtain meaningful samples through MPS.
Preliminary
2.1 Problem Setting
We consider a classification problem in which we train a neural network model fo X -+ y on labeled dataset D {(x', v²) X V)N, where X and y are the input and output label spaces,

---

1. Generating semples with
2. Meta-learning of Fo.
Trie boundary of "dog" Distorted boundary by 4,04019)
Fig: finder
2
J/p "ball"
manimize Hirk
[(0)
PCRC (9, 4)
minimize w.rt
Gp
volidate
Ap
Eyal (8')
Figure 3: Distortion of decision boundary caused by generative data augmentation with conditional synthetic samples leaking another class attribute. If the synthetic sample Tp is "tennis ball dog" (reprinted from (18) with its conditional label y/p "ball", the supervised learner of task head hw distorts the decision boundary of "dog" and
Figure 4: Meta pseudo sampling framework. We meta-optimize the finder F& to generate a useful latent vector (=(=) for training a model parameter through minimizing the validation loss with the once updated parameter #' by real sample and synthetic sample Tp = Go (F)()).
"ball" to classify up as "ball".
respectively. Here, we can use generative model G. 2 x y - X' which is trained on D. We assume that G generates samples from latent vector Z and conditions on samples with a categorical label and y is uniformly sampled from ) We refer to the classification task on D as the main task, and So as the main model. So is defined by a composition of feature extractor Jø and classifier hw, i.e., So bw Dv and
Y, where z is sampled from standard Gaussian distribution p(z) ~o0 I)
w]] To validate fe, we can use a small validation dataset
Deal = (( Vaa) E X' y) Mij which has no intersection with D (i.e., DD Dval = 0).
2.2 Generative Data Augmentation
A typical generative data augmentation trains main model fe with both real data and synthetic data from the generative models [19]. We first generate synthetic samples be utilized as additional training data for the main task. Most previous studies on generative data augmentation 19 20 adopt conditional generative models for Gg. and generate pseudo dataset Dp as
Dp {< v/)) = Go(2", 3
(1)
where 2 is sampled from prior distribution p(z), and y is uniformly sampled from y. Subsequently. fo is trained on both of 'D and Dp using the following objective function.
min (()) + (C)
(2)
(()) = (( y), (0)
(3)
((p,,,)) (fo (zp),
where l is loss function of the main task (e.g., cross-entropy). Ep a loss function for the synthetic samples, and À is a hyperparameter for balancing L and Lp- In previous works, 4p is often set the same as f. Although optimizing Eq. (2) with respect 0 is expected to boost the test performance by interpolating or oversampling conditional samples []]. its naïve application degrades the performance of fe on general settings 77. In this paper, we explore methods to resolve the degradation of generative data augmentation and maximize the performance gain from Dp.
Proposed Method
In this section, we describe our proposed method, MGR. The training using MGR is formalized as alternating optimization as shown in Figure 2 To maximize the gain from synthetic samples, MGR regularizes feature extractor 92 of fo using PCR by effectively sampling useful samples for the generalization from G using MPS. We show the overall algorithm of MGR in Appendix
main model and finder network for searching latent vectors of Go,
Although we basically use conditional G for comparing our method and generative data augmentation, our method can be used with unconditional Ga (see Sec. 4.6).

---

3.1 Pseudo Consistency Regularization
As discussed in Secion we hypothesize that the synthetic samples do not perfectly represent class categories, and training classifiers using them can distort the decision boundaries. This is because Up is not reliable due to Dp can contain class leaked samples [4]. To avoid the degradation caused by the distortion, we propose utilizing Tp to regularize only the feature extractor Py of fo by discarding a conditional label y/- For the regularization, we borrow the concept of consistency regularization, which was originally proposed for semi-supervised learning (SSL. [5 16 17 These SSL. methods were designed to minimize the dissimilarity between the two logits (i.e., the output of lww) of strongly and weakly transformed unlabeled samples to obtain robust representations. By following this concept, the PCR loss is formalized as
(5) where T is a strong transformation such as RandAugment [21], which is similar to the one used in UDA [16] for SSL. The difference between PCR and UDA is that PCR penalizes only Gø. whereas UDA trains the entire fo = how Ge. ÉPCR can be expected to help Ie learns features of inter-cluster interpolated by up without distorting the decision boundaries. Using fyCR. we rewrite Eq. (4) as
(xp v) (T Ie (:
LPCR (0) EmpED, EPCR (Ipi v)).
(6)
3.2 Meta Pseudo Sampling
Most generative data augmentation methods generate a synthetic sample xp from Ga with a randomly sampled latent vector z. This is not the best option G4 is not optimized for generating useful samples to train Se in predicting the conditional distribution p(ulz)); it is originally optimized to replicate p(xlg) [y) or p(z). The main concept of MPS is to directly determine useful samples for training fe by optimizing an additional neural network called finder F : Z - Z. F' takes a latent vector 2 p(z) as input and outputs vector of the same dimension as By using F.g. we generate a synthetic sample as Tp = (F (2).y)) y). The role of Fg is to find the optimal latent vectors that improve the training of fo through Tp. Although we can consider optimizing G. instead of Fa. optimize Fg because the previous work showed that transforming latent variables according to loss functions efficiently reaches the desired outputs [22) and we observed that optimizing G4 is unstable and causes the performance degradations of Jo (Section 4.5).
The useful samples for generalization should reduce the validation loss of Se by using them for optimization. Based on this simple suggestion, we formalize the following bilevel optimization problem for Fs.
min (0)) = (Ber))=B((nnn ((sunna ((for (Ivat), Bval)
subject to 0+ argmin C(0) + ACPCR(0,
Note that the finder parameter & is added the arguments of LPCR- We can optimize Fe with a standard gradient descent algorithm because Fs. Ga. and fo are all composed of differentiable
functions as well existing meta-learning methods such as MAML (3) and DARTS (24]. We
approximate 0* because the exact computation of the gradient V Cm (0*) is expensive 23 24]:
8* = 0'
(LV)( + ACvcR(O,
(8)
where 7) is inner step size. Thus, we update F& by using 0' that is updated for single step and then alternately update 0 by applying Eq. (6) with Tp generated from the updated F The overall optimization flow is shown Figure
Approximating gradients with respect to dh. Computing Lv requires the product com- putation including second-order gradients: V CrcR (0, V) Cva This causes a computation complexity of (( + [o))))). To avoid this computation, we approximate the term using the finite difference method (24] as
V VoLreR (++ .)) VoL (0 .0)
2
0.0 CPCR (0,0) V ~
25
where 0+ is 0 updated by 0 ( is defined by for e based on 24. This approximation reduces the computation complexity to O1$5) + |d]] + 10)) We confirm the speedup by Eq. (9) in Appendix B.I
const, [
We used 0.01 of the constant

---

Techniques for improving finder We introduce two techniques for improving the training of F& in terms of the architectures and a penalty term for the outputs.
While arbitrary neural architectures can be used for the implementation of FØ, we observed that the following residual architecture produces better results.
(2 tanh(MLP
(10)
where MLP is multi layer perceptron.
To ensure that (2) does not diverge too far from the distribution p(z), add Kullback-Leible
divergence term DrL (pp(())lp(z)) where po(z) is the distribution of (z)z) into Eq. (7). When p(=)
follows the standard Gaussian (0, 1). DRL (po(2)))
can be computed by
1 (1 + os n3 -os),
DRL (po (p<(<))pp(())
-
(11)
where j& and a is the mean and variance of (Fo(2 No In Appendix B.2 discuss the effects of design choice based on the ablation study.
4 Experiments
In this section, we evaluate our MGR (the combination of PCR and MPS with the experiments on multiple image classification datasets. We mainly aim to answer three research questions with the experiments: (1) Can PCR avoid the negative effects of Tp in existing methods and improve the performance of fo? (2) Can MPS find better samples for training than those by uniform sampling? (3) How practical is the performance of MGR? We compare MGR with baselines including conven- tional generative data augmentation and its variants in terms of test performance (Sec. 4.2 and 4.3). Furthermore, we conduct a comprehensive analysis of PCR and MPS such as the visualization of trained feature spaces (Sec. (44)) quantitative/qullittiie evaluations of the synthetic samples (Sec. 4.5), performance studies when changing generative models (Sec. 4.6), and comparison to data augmentation methods such TrivialAugment [25] (Sec. 4.7
4.1 Settings
We compare our method with the following baselines. Base Model: training sa with only D. Generative Data Augmentation (GDA): training fo with D and Go using Eq. ( GDA+MH: training Sa with D and G& by decoupling the heads into hw for 'D and hw, for D MH denotes multi-head, This is a naïve approach to avoid the negative effect of Xp on hw by not passing Tp through h.o GDA+MH optimizes the parameters as argmin (x))(x)))) 8) + ACCh (g) (xp)), Vp).
Baselines.
training fo with D and G& by applying an SSL loss for Dp that utilizes the output of hh unlike PCR. This method was originally proposed by Yamaguchi et al. (26) for transfer learning, but we note that G was trained on the main task dataset in contrast to the original paper. By following 266.. we used UDA [16] as the SSL loss and the same strong transformation T as of MGR for the consistency regularization.
GDA+SSL:
Datasets. We used six image datasets for classification tasks in various domains: Cars 12] Aircraft (27). Birds [28). DTD [29, Flowers [30], and Pets [3I]. Furthermore, to evaluate smaller dataset cases, we used subsets of Cars that were reduced by {10, 25, 50, 75%% in volume; reduced them by random sampling on a fixed random seed. We randomly split a dataset into 9 1 and used
the former as D and the latter D
Architectures. We used ResNet- 18 B3 as fo and generators of conditional StyleGAN2-ADA for 256 x 256 images [18] Ga. F& was composed of three-layer perceptron with a leaky-ReLU activation function. We used the ImageNet pre-trained weights of ResNet-18 distributed by PyTorch For StyleGAN2-ADA, we did not use pre-trained weights. We trained Go on each DD from scratch according the default setting of the implementation of StyleGAN2-ADA Note that we used the same G in the baselines and our proposed method.
https//giihbb.oo//yttrovviiiii https//gittub.com///lab/////ssge
5

---

Table 1: Top-1 accuracy (%) of ResNet18. Underlined scores outperform that of Base Model, and Bolded scores are the best among the methods.
(a) Multiple Datasets
(b) Small Datasets (Cars)
Method Dunset
Clas
Airerat
Blads
DTD
Flower
Pets
Base Mode!
65,50*
62.61**
72.24
65.16
94.18-16
87. 87.21
Medbod: Dutneet
25%
50%
GDA
24.50
61,29--55
67.554-11
67.685:37
93.46-11
87.37
Rase Medel
20111200
49.33824
72,91223
GDA-MH
65.774-9
58.16--4
71.95-.0
65.49441
03.83-11
87.75-18
GDA
18.9125
46.56210
66.81-5
80667:11
GDA-SSL
$5.75+21
02.87--4
71.28
05.00
93.66-44
97.75
GDA-MII
18,00
46.63911
61.25+ 71.1
GDA-MPS
65.08+-25
50.11--34
72.254-1
65.87--2
94. 14
87.02341
GDA-SSI
19:71
47,80-20
71,991.4
B1 61,55333
PCR
M6.26
bd.12-21
22.00--0
00.12--11
01.00
35.05
MGR
23.49
53.16
25.24
K3.13
MUR (POX+MPSS)
$7.32*
05.11
74,34*
09,8
95.42-19
NK.05--
Training. We trained fo by the Nesterov momentum SGD for 200 epochs with momentum of 0.9, and an initial learning rate of 0.01; we decayed the learning rate by 0.1 at 60, 120, and 160 epochs. We trained F& by the Adam optimizer for 200 epochs with learning rate of 1.0 x 10-4, We used mini-batch sizes of 64 for D and 64 for pp. The input samples were resized into a resolution of 224 x 224; Tp was resized by differentiable transformations. For synthetic samples from Go in PCR and GDA+SSL, the strong transformation was RandAugment (2 by following [16], and it was implemented with differentiable transformations provided in Kornia [32]. We determined the hyperparameter À by grid search among [0.1, 1.0] with a step size of 0.1 for each method by D To avoid overfitting, we set the hyperparameters of MGR that are searched with only applying PCR i.e., we did not use meta-learning to choose them. We used AKL. of 0.01. We selected the final model by checking the validation accuracy for each epoch. We ran the experiments three times 24-core Intel Xeon CPU with an NVIDIA A100 GPU with 40GB VRAM and recorded average test accuracies with standard deviations evaluated on the final models.
4.2 Evaluation on Multiple Datasets
We confirm the efficacy of MGR across multiple datasets. Table Ia shows the top-1 accuracy scores of each method. As reported in 77. GDA degraded the base model on many datasets; it slightly improved the base model on only one dataset. GDA+MH, which had decoupled classifier heads for GDA, exhibited similar trend to GDA. This indicates that simply decoupling the classifier heads is not a solution to the performance degradation caused by synthetic samples. contrast, our MGR stably and significantly outperformed the baselines and achieved the best results. The ablation of MGR discarding PCR or MPS is listed in Table Ia We confirm that both PCR and GDA+MPS improve GDA. While GDA+SSL underperforms the base models, PCR outperforms the base models. This indicates that using unsupervised loss alone not sufficient to eliminate the negative effects of the synthetic samples and that discarding the classifier hi from the regularization is important to obtain the positive effect. MPS yields only a small performance gain when combined with GDA, but it significantly improves its performance when combined with PCR i.e., MGR. This suggests that there is no room for performance improvements in GDA, and MPS can maximize the potential benefits of PCR.
4.3 Evaluation on Small Datasets
A small dataset setting one of the main motivations for utilizing generative data augmentation. We evaluate the effectiveness of MGR on smaller datasets. Table[[b]shows the performance when reducing the Cars dataset into a volume of {10, 25, 50, 75)%. Note that we trained Go on each reduced dataset, not on 100% of Cars. In contrast to the cases of the full dataset (Table [Ta, no baseline methods outperformed the base model in this setting. This because G trained on the small datasets generates low-quality samples with less reliability on the conditional label yp that are not appropriate in supervised learning. On the other hand, MGR improved the baselines in large margins. This indicates that, even when the synthetic samples are not sufficient to represent the class categories, our MGR can maximize the information obtained from the samples by utilizing them to regularize feature extractors and dynamically finding useful samples.

---

5 epocc
15 epoch
30 epoch
Sentwer
Systata
GDA
-10
23.
PCR
Ayttee
PCR+MPS
38
28 26 25
24
Figure 5: UMAP visualization of feature spaces training. The plots in the figures represent real and synthetic samples in the feature spaces. Our methods (PCR and PCR+MPS) can help the feature extractors separate real sample clusters. In contrast, the existing method (GDA) confuses the feature extractor by leaking synthetic samples out of the clusters.
4.4 Visualization of Feature Spaces
In this section, we discuss the effects of PCR and MPS through the visualizations of feature spaces in training. To visualize the output of ga in 2D maps, utilized UMAP (3 to reduce the dimensions. UMAP is a visualization method based on the structure of distances between samples, and the low dimensional embeddings can preserve the distance between samples of the high dimensional input. Thus, the distance among the feature clusters on UMAP visualization of Go. can represent the difficulty of the separation by h We used the official implementation by [33 and its default hyperparameters. We plotted the UMAP embeddings of gu (a) and (ap) (5, 10,30) epochs, as shown in 5 we used ResNet-18 trained on the Cars dataset. At first glance, we observe that GDA and PCR formed completely different feature spaces. GDA forms the feature spaces by forcing to treat the synthetic samples the same as the real samples through cross-entropy loss and trying to separate the clusters of samples according to the class labels. However, the synthetic samples leaked to the inter-cluster region at every epoch because they could not represent class categories perfectly as discussed Sec. [ This means that the feature extractor might be distorted to produce features that confuse the classifier. On the other hand, the synthetic samples in PCR progressively formed a cluster at the center, and the outer clusters can be seen well separated. Since UMAP can preserve the distances between clusters, we can say that the sparse clusters that exist far from the center are considered easy to classify, while the dense clusters close to the center are considered difficult to classify. In this perspective, PCR helps gy to leverage the synthetic samples for learning feature representations interpolating the dense difficult clusters. This because the synthetic samples tend to be in the middle of clusters due to their less representativeness of class categories. That is, PCR can utilize the partial but useful information contained in the synthetic samples while avoiding the negative effect. Further, we observe that applying MPS accelerates the convergence of the synthetic samples into the center.
For more straightforward visualization of class-wiss features, we designed simple binary clas- sification task that separates Pets (II] into dogs and cats, and visualized the feature space of a model trained on this task. Fig. [6 shows the feature space after one epoch of training. While GDA failed to separate the clusters for each class, MGR clearly separated the clusters. Looking more closely, MGR helps samples to be dense for each class. This is because PCR makes the feature extractor learn slight differences between the synthetic samples that interpolate the real samples. From these observations, conclude that our MGR can help models learn useful feature representations for solving tasks.
htts://gthub..com/mmcineeuuggggg
7

---

Real Dog
8
Dog
Real Cat
Real Cat
Synthetic Dog
Synthetic Dog
Synthetic Cat
Synthetic Cat
1-
-2
2
8
10
(a) GDA
(b) MGR
Figure 6: UMAP visualization (ResNet-18) 18). We used the Pets dataset by modifying the class definition to the binary classes (dogs and cats). The visualization results are plotted after one epoch training with 2048 real samples and 2048 synthetic samples.
32
69
apcce
eacch
(a) Validation Loss
(b) FID
(a) Uniform
(b) MPS
Figure 7: Statistics in training (Cars)
Figure 8: Synthetic samples (class: Hummer)
4.5 Analysis of MPS
Evaluation of validation loss. We investigate the effects on validation losses when using MPS. Through the meta-optimization by Eq. (7). MPS can generate samples that reduce the validation loss of fe. We recorded the validation loss per epoch when applying uniform sampling (Uniform) and when applying MPS. We used the models trained on Cars and applied the PCR loss on both models. Figure plots the averaged validation losses. MPS reduced the validation loss. In particular, MPS was more effective in early training epochs. This related to accelerations of converging the central cluster of synthetic samples discussed in Section 4.4 and Figure 5] That is, MPS can produce effective samples for regularizing features and thus speed up the entire training of fe.
Quantitative evaluation of synthetic samples. We evaluate the synthetic samples generated by MPS. To assess the characteristics of the samples, we measured the difference between the data distribution and distribution of the synthetic samples. We leveraged the Fréchet Inception distance (FID, (34) which is a measurement of the distribution gap between two datasets using the closed- form computation assuming multivariate normal distributions:
FID Pp) = || M 1122 + Tr Σ E, 22 EEE,
where y and E are the mean and covariance of the feature vectors on InceptionNet for input fa*). Since FID is distance, the lower FID(D, Dp) means that Dp contains more high-quality samples terms of realness. We computed FID scores using 2048 samples D and 2048 synthetic samples every epoch; the other settings are given in Section 4.1 The FID scores in training are plotted in Figure We confirm that MPS consistently produced higher-quality samples than Uniform. This indicates that the sample quality is important for generalizing fa even in PCR, and uniform sampling can miss higher quality samples in generative models. Since the performance gain by GDA+MPS in Table Ia did not better than MGR, the higher-quality samples by MPS can still contain uninformative samples for the cross-entropy loss, but they are helpful for PCR learning good feature representations.
Qualitative evaluation of synthetic samples. We evaluate the qualitative properties of the synthetic samples by visualizing samples of a class. The samples generated by Uniform and MPS are shown in Figure Sa and 8b where the dataset was Cars and the class category was Hummer. Compared with

---

Table Performance comparison between MGR and data augmentation methods (ResNet- 18, Cars,
Table 2: Performance of MGR varying G
(ResNet-18. 18, Cars).
Top-1 1 Acc. (%)).
Go
Conditional
FID
Top-1 Acc. (%)
Data Augmentation
Base Model
+MGR
None (Base Model)
85.50=
None
85.50
87.22
Real CR
86. 16
MixUp
86.87 1.30
87.60
FastGAN 35
No
23.1
86.30#
CutMix
86.13
87 97.80tt.11
BigGAN
Yes
15.6
86.86 m
Aug
30
86,25
87. 87.65
StylGGAN2-ADA
Yes
87,22 15
RandAugment
21
87.47
88.67
StyleGAN-XL 36
Yes
6.2
88,374.30
SnapMix
40
87.1 87.11
88 28.21 88.21
TrivialAugment
25
87.83
89.10
Uniform, MPS produced samples with more diverse body colors and backgrounds. That is, MPS focuses the color and background of the car as visual features for solving classifications. In fact, since Hummer has various colors of car bodies and can drive on any road with four-whee drive, these selective generations of MPS considered reasonable in solving classification tasks.
4.6 Effect of Generative Models
Here, evaluate MGR by varying the generative model G for producing synthetic samples. As discussed in Sec. [3. and 3.2, MGR can use arbitrary unconditional/condtiona generative models Go unless it has latent space. To confirm the effects when changing Ga, we tested MGR with FastGAN 35 BigGAN [8], and StyleGAN-XI. [36]. Table 2 shows the results on Cars. The unconditional FastGAN achieved similar improvements conditional cases. However, since unconditional generative models are generally of low quality in FID, they are slightly less effective. For the conditional generative models, we observe that MGR performance improvement increases as the quality of synthetic samples improves. These results suggest the potential for even greater MGR gains in the future as the performance of the generative model improves. We also evaluate MGR with recent diffusion models in Appendix [333 Meanwhile, to evaluate the value of synthetic samples the consistency regularization, we compare MGR with the case where we use real samples in Eq. (6) (Real CR). Our method outperformed Real CR. This can be because interpolation by the generators helps the feature extractor to capture the difference between images. Furthermore, our method searches the optimal synthetic samples by using latent vectors of generative models with MPS. In contrast, Real CR cannot search the optimal real samples for CR loss because real data fixed in the data space.
4.7 Combination of MGR and Data Augmentation
To assess the practicality of MGR, evaluate the comparison and combination of MGR and existing data augmentation methods. Data augmentation (DA) is a method applied to real data and is an independent research field from generative data augmentation. Therefore, MGR can be combined with DA to improve performance further. Table 3 shows the evaluation results when comparing and combining MGR and DA methods; used MixUp (37). CutMix (38,, AugMix [39], RandAugment (2], SnapMix 40), and TrivialAugment [25] as the DA methods. The improvement effect of MGR comparable to that of DA, and the highest accuracy
achieved by combining the two methods. In particular, MGR was stably superior to sample-mix-based such as MixUp and AugMix. This result also indicates that synthetic samples, which non-linearly interpolate the real samples, elicit better performance than linearly interpolating real samples by the sample-mix-basee DA methods. We consider this strength to be advantage of using generative models.
Related Work
We briefly review generative data augmentation and training techniques using generative models.
The earliest works of generative data augmentation are 19 simply adding synthetic samples as augmented data for classification tasks can improve performance in shot learning, person re-identification,, and medical imaging tasks. Tran et al. (20) have
42 They have demonstrated that

---

proposed a generative data augmentation method that simultaneously trains GANs and classifiers for optimizing 0 to maximize the posterior p(0]z) by an EM algorithm. Although this concept is similar to our MPS in terms of updating both Ge and fo. it requires training the specialized neural architectures based on GANs. contrast, MPS formalized for arbitrary existing generative models with latent variables, and it requires no restrictions to the training objectives of generative models.
On the analysis of generative data augmentation, Shmelkov al. have pointed out that leveraging synthetic samples as augmented data degrades the performance in general visual classification tasks. They have hypothesized that the cause of the degradation the less diversity and fidelity of synthetic samples from generative models. Subsequent research by Yamaguchi et al. [8] have shown that the scores related the diversity and fidelity of synthetic samples (i.e., SSIM and FID) are correlated to the test accuracies when applying the samples for generative data augmentation in classification tasks. Based on these works, our work reconsiders the training objective and sampling method in generative data augmentation and proposes PCR and MPS.
More recently, He et al. [43 have reported that lext-to-image generative model pre-trained on massive external datasets can achieve high performance on few-shot learning tasks. They also found that the benefit of synthetic data decreases as the amount of real training data increases, which they attributed to a domain gap between real and synthetic samples. In contrast, our method does not depend on any external dataset and successfully improves the accuracy of the classifier even when synthetic data are not representatives of class labels (i.e., having domain gaps).
6 Limitation
One of the limitations of our method is the requirement of bilevel optimization of classifier and finder networks. This optimization is computationally expensive particularly when used with generative models that require multiple inference steps, such as diffusion models as discussed in Appendix [B.3] We have tried other objective functions not requiring bilevel optimization, but at this time, we have not found optimization method that outperforms MPS (see Appendix [B.4). Nevertheless, since recent studies rapidly and intensively focus on the speedup of diffusion models 45 46), we can expect that this limitation will be negligible in near the future. Additionally, applying MGR to pre-trained text-to-image diffusion models (e.g., [44]) is also important for future work because they have succeeded in producing effective samples for training downstream task models in zero-shot
manner 43
7 Conclusion
This paper presents novel method for generative data augmentation called MGR. MGR is composed of two techniques: PCR and MPS. To avoid the degradation of classifiers, PCR utilizes synthetic samples to regularize feature extractors by the simple consistency regularization loss. MPS searches the useful synthetic samples to train the classifier through meaa-learning on the validation loss of main tasks. We empirically showed that MGR significantly improves the basclines and brings generative data augmentation up to a practical level. We also observed that the synthetic samples in existing generative data augmentation with synthetic samples dynamically generated by MPS can resolve this issue through visualization. We consider that these findings will help future research in this area.
distort the decision boundaries
feature spaces, and the PCR loss
Acknowledgements
We thank the members of the Kashima Laboratory and the PRMU community for discussing the initial concepts of this paper and giving useful advice.
References
[1] Connor Shorten and Taghi Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 2019.
10

---

[2] Francesco Calimeri, Aldo Marzullo, Claudio Stamile, and Giorgio Terracina. Biomedical data augmentation using generative adversarial neural networks. In International conference on artificial neural networks, pages 626-634. Springer, 2017.
[3] Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Gan-based synthetic medical image augmentation for increased enn performance in liver lesion classification. Neurocomputing, 2018.
[4] Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Synthetic data augmentation using gan for improved liver lesion classification. In 2018 IEEE 15th International Symposium on Biomedical Imaging, 2018.
[5] Abdul Waheed, Muskan Goyal, Deepak Gupta, Ashish Khanna, Fadi Al-Turjman, and Plá- cido Rogerio Pinheiro. Covidgan: data augmentation using auxiliary classifier gan for improved covid-19 19 detection. leee Access, 2020.
[6] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[7] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In Proceedings of the European Conference Computer Vision, 2018.
[8] Shin's ya Yamaguchi, Sekitoshi Kanai, and Takeharu Eda. Effective data augmentation with multi-domain learning gans. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[9] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems, 2020.
[10] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International Conference Machine Learning, 2017.
[11] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
[12] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition, Sydney, Australia, 2013.
[13] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
[14] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[15] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances neural information processing systems, 2014.
[16] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data aug- mentation for consistency training. In Advances in Neural Information Processing Systems, 2020.
[17] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang. Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, 2020.
[18] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018.
[19] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv: 2017.
[20] Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid. A bayesian data augmentation approach for learning deep models. Advances Neural Information Processing Systems 30, 2017.
11

---

[21] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition Workshops, pages 702-703, 2020.
[22] Ruoshi Liu, Chengzhi Mao, Purva Tendulkar, Hao Wang. and Carl Vondrick. Landscape learning for neural network inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[23] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International conference on machine learning. 2017.
[24] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search.
In International Conference on Learning Representations, 2019.
[25] Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 774-782, 2021.
[26] Shin's ya Yamaguchi, Sekitoshi Kanai, Atsutoshi Kumagai, Daiki Chijiwa, and Hisashi Kashima. Transfer learning with pre-trained conditional generative models. arXiv preprint arXiv:2204. /2833, 2022.
[27] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. enn-ggrined visual classification of aircraft. arXiv, 2013.
[28] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical report, California Institute of Technology, 2010.
[29] M. Cimpoi, S. Maji, Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2014.
[30] M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
[31] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs, In IEFE Conference on Computer Vision and Pattern Recognition, 2012.
[32] Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020.
[33] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03344 2018
[34] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
[35] Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations, 2021.
[36] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH, 2022.
[37] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
[38] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision. 2019.
[39] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: simple data processing method to improve robustness and uncertainty. In Proceedings of the International Conference Learning Representations, 2020.
[40] Shaoli Huang, Xinchao Wang, and Dacheng Tao. Snapmix: Semantically proportional mix- ing for augmenting fine-grained data. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
12

---

[41] Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
[42] Francesco Calimeri, Aldo Marzullo, Claudio Stamile, and Giorgio Terracina. Biomedical data augmentation using generative adversarial neural networks. In International Conference
Artificial Neural Networks. Springer, 2017.
[43] Ruifei Hc, Shuyang Sun, Xin Yu, Chubui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint 2rXiv:22/0.07574, 2022.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[45] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022.
[46] Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVE international conference on computer vision, 2022.
[47] Ron Mokady, Omer Tov, Michal Yarom, Oran Lang, Inbar Mosseri, Tali Dekel, Daniel Cohen- Or, and Michal Irani. Self-distilled stylegan: Towards generation from internet photos. ACM SIGGRAPH 2022 Conference Proceedings, 2022.
[48] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
13

---

Appendix
The following manuscript provides the supplementary materials of the main paper: Regularizing Neural Networks with Meta-Learning Generative Models.
A
Algorithm of Meta Generative Regularization
Algorithm Meta Generative Regularization
Require:
Training dataset validation dataset Dua main model fe. generator finder Fs. training batchsize B. pseudo batchsize Bp. validation batchsize Bud. step size and Ç. hyperparameter and AKL.
Ensure: Trained main model fo
1: while not converged do
2: {(a', oo),
(2 Bp (O
3:
4:
Updatiing for MPS
5:
((aha ) Bgg D
6:
{ [Ga(Fa(z") ). Rp
=
7:
0' o't 0 Ve (')) B
a (rek(eti
8:
-+ -- (Vo( (1_ (ffor (Ira),, vaal Ara (Dau(po()))))))))
Bud
9:
// Updating P PCR
PCR
10:
[- (Fo (())
11:
0 0 (V
Dp toC (upi v))
12:
end while
B Additional Experiments
B.1 Evaluation Gradient Approximation
Here, we evaluate the gradient approximation by Eq. (9). As shown in Table 4 Eq. 9 well approximated the second-order gradients in speeding up over 10% with 0.08 of the accuracy drop.
Table 4: Performance comparison between MPS with 2nd-order gradients and Ist-order approximated gradients (ResNet-18, Cars).
Method
Top-1 I Acc. (%) Wall Clock Time (hours)
2nd-Order
L..30
6.55
Ist-Order Approx.
87.22
5.79
B.2
Ablation study of Fs
introduce Fa for meta-optimized parameters and the residual architectures with MLP defined by Eq. (TO). We performed an ablation study of MPS with respect to the meta-optimized parameters and the architectures of Fg. We compared MPS with variant of MPS optimizing G instead of Fg- We also attempted other architectures for F& including Linear: W²(z) + b. ML.P: shown in Table 5 We observed that MPS with G4, caused failures of training fo and degraded the accuracy, On the other hand, all variants of MPS with F's succeeded boosting the models without MPS. Thus, restricting the number of optimized parameters is important, and determining an optimal a with the finder F& effective on the optimization problems of MPS. For the variants of MPS with Fø. we observed that the residual architectures and regularization by D (2))(())) contributed to successes. Interestingly, MPS with Linear Fa outperformed MPS with MLP F i.e., significantly transforming the input p(z) by complex functions results in low accuracy. These results suggest that better latent vectors in train fo can exist near the uniformly sampled input z. Thus, limiting the search range by tanh in the residual architectures can help in finding better latent vectors.
In Section 3.2
MLP (2), and Residual-Shallow: z + ( + b). The results of these variations
14

---

Table Ablation study of MPS (ResNet- 18, Cars).1
Method
Top-1 I Acc. (%)
Without MPS (PCR)
86.32t.07
MPS
87.22 1..2 1.11
MPS with G4
84.47-.05
MPS with Linear Fo
86.51 +.00
MPS with MLP F MPS with Residual+Shallow Fs
86.35
86. 86.88-1.1 16
MPS w/o DKL (k((p())ll((z)
86.92
B.3 MGR with Diffusion Models
We tested our method on EDM 45). recent diffusion model. Due the computation cost, we used 10% reduced CIFAR- 10 as the dataset. We optimized F& to search the first step noise of the diffusion process. Table B-4 shows that our method with EDM improves Base Model. However, the overhead of incorporating diffusion models was significant; it takes more than ten times longer training than GANs, In future work, we will investigate gigher--weigt methods using the diffusion model.
Table 6: Performance studies on Diffusion Model (ResNet-18 Cars)
Method
Top-1 Acc. (%)
Base Model
86.49#t48
GDA (EDM)
85.80#.30
MGR
88.49
B.4 Updating Fs without Meta-optimization
MPS consists of meta-lcarning validation losses requiring bi-level optimization, which is relatively heavy computation. One can consider if F& could be trained without meta-optimization. Here, we try alternative methods other than meta-learning update Fg. Instead of meta-learning, used a strategy of choosing hard examples via optimizing Fg. That is, we optimize Fş by maximizing the training cross-entropy (CE) loss and the PCR loss synthetic samples. Note that, in both cases, we used the PCR loss for synthetic samples when training classifiers. Table A-2 shows the results. Optimizing Fa with CE and PCR slightly improved the baselines but significantly underperformed our method (MGR). This result can justify using meta-optimizing F& to generate useful samples for classifiers. Nevertheless, this idea could inspire a sampling method that does not require bi-level optimization in future work.
Table 7: Performance comparison of updating strategies for F& (ResNet-18 on Cars)
Method
Top-1 Acc. (%)
Base Model
85.50)..10
PCR
$6.36#.08
Optimizing Fs w/ CE
86.52
Optimizing F& w/ PCR MGR
86.44
87.22 1.15
B.5 ImageNet Classification
We evaluated our MGR on ImageNet by randomly initializing ResNet- 18 as the classifier and the pre-trained BigGAN as the generator. Table shows that MGR successfully improves top-1 accuracy, while naive generative data augmentation (GDA) does not; this is the same trend Table (a) of the main paper. This result indicates that our method consistently works on complex and large-scale datasets. We will add the result in multiple trials with standard deviation to the paper.
15

---

Table 8: ImageNet Classification (ResNet-18
Method
Top-1 Acc. (%)
Base Model
68. 10
GDA
64.74
MGR
70.55
B.6 Latent Augmentation
Some readers may wonder why not also utilize augmentation in the generative model's latent space instead of data space. We implemented this approach by adding Gaussian noise to the latent vector 2' = + s, where (0, 10 3). Then, compute the consistency regularization between g(G(z)) and g(G(z')). instead ((())) and (z)()) We call this variant LatentAugment, We found that LatentAugment improves the performance of our MGR (Table()). Interestingly, LatentAug- ment can improve when it used solely without image data augmentation 7, i.e., RandAugment. This obtain meaningful variants by perturbing latent vectors, which is challenging for conventional data augmentation. However, this approach doubles the number of generators' forward computations and thus leads to increased computation time and memory footprint.
indicates that
Table Classification on Cars (ResNet- 18)
Method
Top-1 Acc. (%)
Base Model
85.50
MGR (LatentAugment)
86.49
MGR (RandAugment)
87.22 4..1
MGR (RandAugment + LatentAugment
87.85
B.7 Comparison to Existing Sampling Technique
Here, evaluate MPS by comparing it with existing sampling technique called multi-modal truncation sampling [47]. which was originally proposed for sampling better quality samples for image generation. Table shows the result of combining multi-modal truncation sampling and PCR. Although the FID score of multi-modal truncation sampling certainly outperforms MPS, the gain of the classification accuracy underperforms MPS. This implies that improving sample quality is necessary condition for performance improvements, not a sufficient condition. Meanwhile, since MPS explicitly searches for samples that minimize the validation loss, can improve classifiers more directly than incorporating existing sampling methods.
Table 10: Classification Cars (ResNet-18)
Method
Running Mean FID
Top-1 Acc. (%)
PCR
22.96
86.36#.08
Multi-modal Truncation + PCR MGR (MPS + PCR)
21.12
86.51 #:21
22.08
87.22
B.8
Performance Study when using Pre-trained Generators
We tried to ImageNet pre-trained BigGAN and confirmed that this does not solve the degradation problem of GDA (Table TT). However, the use pre-trained models can enhance our method.
B.9 Evaluation Robustness toward Natural Corruption
Our method can improve the robustness against natural corruption and MPS does not generate biased samples toward training distribution. We tested the robustness of our method on CIFAR-10-C [48), which is test set for CIFAR-10 corrupted by various transformations. Table 12 shows the results.
16

---

Table 11: Classification on Cars (ResNet-18 18,
ImageNet pre-trained BigGAN), Underlined scores outperform that of Base Model, and Bolded scores are the best among the methods.
Method Dataset size
10%
25%
50%
100%
Base Model
20.11
49.33
72.91 2..91
85.80
GDA
18.82
46.38 L.59
70.23 1.66
86. 16.16
MGR
24.55
53.41 1..8
75.46 2.11
87.17 1.118
While GDA degraded the performance for all corruptions, PCR significantly improved the base model. Furthermore, MGR achieved even higher robustmess. This indicates MPS in MGR can provide samples that are useful for generalization through meta-optimization.
Table 12: Classification on CIFAR-100 R--C (ResNet-18)
tepdecten
plan
Imc
Erghizn
Beo
11:15
51.6
15:11
51.11
3832
B6A2
$2.11
13.0
6533
28.0
1183
68.55
71.00
17


Reviews:
[{"review_id": "6RW5Q0PNkP", "reviewer": "Reviewer_ZFum", "summary": "This paper leverages synthetic images from generative models to train classifier models, effectively using them as an augmentation tool. However, instead of incorporating these images in a simplistic fashion, the synthetic images are utilized as a regularizer. The paper posits that synthetic samples may not always perfectly mirror class categories found in real data distribution. As a response, a meta-learning framework is employed to dynamically ascertain which synthetic samples should be used to minimize validation losses. The paper introduces a feature-based consistency regularization loss for the generated images, a method similar to self-supervised techniques. The unique contribution of this paper is the proposition of a finder network that is trained within a meta-learning context, this network is designed to select images that will enhance the performance of the classification model. Through extensive experimentation, the paper illustrates that the proposed methodology can effectively circumvent the performance deterioration often linked with naive generative data augmentation while concurrently improving the baselines.", "questions": "*The major novelty of the paper resides in the proposition of a sampling network, trained via a meta-learning method, to generate synthetic samples aimed at enhancing the classification model's performance. The authors further highlight that a superior FID score from the generative model correlates to improved performance in classification models (Table 2), and the FID score derived from their sampling network significantly surpasses that of uniform sampling (Fig 6b). In the generative models' literature, there exist works that advocate for effective sampling strategies from pre-trained GAN models, utilizing the multimodal truncation method, such as Makady et al. 2022's 'Self-Distilled StyleGAN: Towards Generation from Internet Photos.' How would their sampling approach fare when combined with the consistency regularization loss? It could significantly bolster the paper's case if authors demonstrated that the proposed sampling network exceeds the performance of other sampling methods, like the multimodal truncation method.", "limitations": "The author themselves pointed out that the limitation of the method is the requirement of bilevel optimization of classifier and finder networks. This optimization is computationally expensive and may be less practical when used in generative models that require multiple inference steps, such as diffusion models.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-presented and easy to follow.\n* The paper presents a novel sampling network that is trained in a meta-learning setting to select synthetic images that will enhance the performance of the classification model.\n* Through a series of experiments, the paper successfully illustrates the effectiveness of the overall approach, as well as the contributions of different aspects of the methodology, i.e., consistency regularization loss and Meta Pseudo Sampling.\n* The proposed generative model augmentation method complements conventional data augmentation methods and can be employed together to enhance performance further.", "weaknesses": "* The paper showcases its experiments predominantly on simple, fine-grained classification datasets. To bolster the credibility of their methodology, the authors would benefit from demonstrating how their approach enhances classification performance on more complex, large-scale datasets, such as ImageNet.\n* In Table 1, I suggest including an additional baseline: the base model + SSL. This means applying a consistency regularization loss on the real training images without utilizing any synthetic images. This will highlight the degree of improvement achieved using the generative model and the proposed model. The author acknowledges that training the sampling network in a meta-learning setting is computationally expensive, so this comparison would clarify whether the benefits of this approach justify its cost, particularly if the gains are minimal.\n* For the consistency loss, the paper solely employs augmentation in the image space. However, one might ask why not also utilize augmentation in the generative model's latent space by introducing small noise to the latent vector z. Given that the latent space of the generators implicitly represents the image manifold, performing augmentation in this latent space could lead to meaningful augmentation in the image space. Such augmentation could potentially be more challenging to achieve using conventional image augmentation methods.\n* Figure 5 could be improved further by incorporating a visualization of different class features along the decision boundary of the classifier. This enhancement would provide further insight into how these synthetic images influence the decision boundary under different cases.\n* (Just a suggestion) In lines 45-50, I would like to introduce an additional argument. Despite generative models being trained to model p(x), they frequently fail to capture the entire distribution of the training set, focusing mainly on the high-density region of the distribution. Therefore, the information extracted from these generative models is often less comprehensive than the information found in the actual dataset distribution. Furthermore, these synthetic images frequently exhibit disfigured object shapes; for instance, a majority of human shapes in StyleGAN-XL images appear distorted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful reading and many insightful comments.\n\n### **W1: Is the proposed method effective on more complex, large-scale datasets such as ImageNet?**\n**Yes.** By following your suggestion, we evaluated our methods on ImageNet. We confirmed the same trend as Table 1 even on ImageNet, which is a large-scale and complex dataset. Please refer to the general response and Table R-1.\n\n---\n\n### **W2: Additional baseline of consistency regularization loss with real data**\nThank you for this suggestion. We examined this baseline in Table R-2 in the above general response and found that **the consistency regularization (CR) with synthetic samples with meta-learning is more effective than one with real samples**. We would emphasize that our method has the advantages of performance improvements by meta-learning of the sampling networks and switching the generative model to a better one, which could not be achieved by CR with real samples. From these facts, we conclude that the proposed approach is beneficial for the cost.\n\n---\n\n### **W3: Why not also utilize augmentation in the generative model's latent space?**\nThis is a very interesting idea. We implemented this approach by adding Gaussian noise to the latent vector as $z^\\prime = z + s$ where $s\\sim\\mathcal{N}(0,10^{-3})$. Then, we compute the consistency regularization between $g(G(z))$ and $g(G(z^\\prime))$, instead of $g(G(z))$ and $g(T(G(z)))$. We call this variant Latent Augmentation. We found that Latent Augmentation improves the performance of our MGR (Table R-4). Interestingly, Latent Augmentation can improve when it is used solely without image data augmentation $T$ (RandAugment). This indicates that we can obtain meaningful variants by perturbing latent vectors, which is challenging for conventional data augmentation, as you said. However, this approach doubles the number of generators' forward computations and thus leads to increased computation time and memory footprint. We would like to present the latent augmentation technique in the paper as a promising option to improve MGR.\n\n\n**Table R-4. Classification on Cars (ResNet-18)**\n|| Top-1 Acc.|\n| :- | :- |\n|Base Model| 85.80 $\\pm$ .10 |\n|MGR (Latent Augment)| 86.49 $\\pm$ .33 |\n|MGR (RandAugment)| 87.22 $\\pm$ .15|\n|MGR (RandAugment + Latent Augment)| 87.85 $\\pm$ .53 |\n\n---\n\n### **W4: UMAP visualization should be improved (Fig. 5)**\nThank you for the helpful comments. According to your suggestion for a more straightforward visualization of the decision boundary, we conducted the additional visualization study by using a binary classification dataset created by modifying Pets. In Fig. I of the attached PDF, we can see how the proposed method effectively enlarges the margins of the class boundaries. Please also see the general response. We would be happy to receive any comments on the modified visualization results.\n\n---\n\n### **W5: Suggestion of additional explanation for the second hypothesis (L45-50)**\n> Despite generative models being trained to model p(x), they frequently fail to capture the entire distribution of the training set, focusing mainly on the high-density region of the distribution. Therefore, the information extracted from these generative models is often less comprehensive than the information found in the actual dataset distribution. Furthermore, these synthetic images frequently exhibit disfigured object shapes; for instance, a majority of human shapes in StyleGAN-XL images appear distorted.\n\nWe appreciate your detailed suggestion with the concrete instance. It makes sense that the na\\\"ive synthetic samples lack detailed information due to focusing on the high-density regions. In fact, from our sample visualization results in Fig. 7, it appears that uniform sampling produces samples with similar features, not comprehensive. The proposed method seems to reduce the negative effect of such disfigured samples by PCR and remove the disfigured samples by MPS. We would gladly incorporate your suggestions into the paper!\n\n--- \n\n### **Q1: How would multi-modal truncation sampling [c] fare when combined with PCR?**\nThank you for the question. Through the additional experiments, we found that multi-modal truncation sampling is somewhat helpful for improving PCR, but MPS is more effective. Table R-5 shows the result of combining multi-modal truncation sampling and PCR. **Although the FID score of multi-modal truncation sampling certainly outperforms MPS, the gain of the classification accuracy underperforms MPS**. This implies that improving sample quality is a necessary condition for performance improvements, not a sufficient condition. Meanwhile, since MPS explicitly searches for samples that minimize the validation loss, it can improve classifiers more directly than incorporating existing sampling methods. We will add this analysis to the paper.\n\n**Table R-5. Classification on Cars (ResNet-18)**\n|| Running Mean FID | Top-1 Acc.|\n| :- | :- | :- |\n|PCR|22.96| 86.36$\\pm$.08|\n|Multi-modal truncation sampling + PCR|21.12|86.51$\\pm$.21|\n|MGR (MPS + PCR)|22.08|87.22$\\pm$.15|\n\n[c] Mokady, Ron, et al. \"Self-distilled stylegan: Towards generation from internet photos.\" ACM SIGGRAPH 2022 Conference Proceedings. 2022."}, {"review_id": "hoJ88Hy3C9", "reviewer": "Reviewer_g8gt", "summary": "The authors propose a method for using synthetic images from GANs to augment training image classifiers. The naive approach to this problem is to generate samples for each class and treat these as supervised examples, but this can degrade performance due to image artifacts. Instead, the authors propose to use the generated data only for consistency regularization of the featurizer (”PCR”). Further, they meta-learn the codes to use for generation (”MPS”). On 6 datasets, the authors see performance improvements from using synthetic data this way, rather than performance drops.", "questions": "A point of confusion for me is why experiments avoided using pretrained generative models / finetuning from them. As noted in the related work, massive pretraining may help produce higher quality samples. The authors attribute degradations after training on generated data to *class leakage*, i.e. samples where artifacts of multiple classes are combined into the same image, such as the tennis ball with a dog face generated by BigGAN. This seems like the kind of artifact that improved generative models can remove. Does generative data augmentation with large pretrained models result in such performance drops? Have the authors tried their method with finetuned GANs?", "limitations": "The authors note that the meta-learning loop renders the method unusable for computationally intensive generative models, e.g. diffusion models.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The motivation is clear, the method is clean, and the approach makes intuitive sense. The overall paper presentation is very good.\n- The authors select a nice set of baselines and show their method outperforms them. In particular, GDA + MH was an important comparison to have.\n- The paper contains a solid set of ablations, including: the importance of PCR v. MPS (either can improve performance, both are best), the effect of MPS on sample quality as measured by FID, experiments with multiple generative models for one dataset, and experiments showing that gains are additive with standard data augmentations.", "weaknesses": "- A missing baseline is consistency regularization without generative data augmentation, i.e. equation 5 applied to real examples $\\mathcal D$ rather than $\\mathcal D_p$. How does the method compare to training with this objective, and are gains additive?\n- Standard training with TrivialAugment / RandAugment outperform this method on Cars, presumably with much lower computational cost — though the authors do show these can be combined with the method to obtain further gains. I’d like to check if this trend holds for other datasets as well.\n- Because of the metalearning loop, the method is too slow to apply with diffusion models, as noted in the Limitations and Appendix B.3. All experiments are done with GANs trained from scratch on the target datasets.\n- Experiments are done on relatively small datasets. It would be interesting to compare this method on ImageNet.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your carefully reading and thoughtful feedback.\n\n### **W1: How does the proposed method compare to the method using consistency regularization on real samples?**\nThank you for your insightful suggestion. Although the consistency regularization with real samples brings some improvement, **the proposed method using synthetic samples and meta-learning outperforms this baseline**. We would emphasize that our method has the advantages of performance improvements by meta-learning of the sampling networks and switching the generative model to a better one, which could not be achieved by CR with real samples. For details, we would be happy if you take a look at the general response and Table R-2.\n\n---\n\n### **W2: What if comparing our method with DA on other datasets?**\nThank you for the reasonable comment. Table R-6 and R-7 shows Aircraft and Birds; DA and our MGR were competitive and the combination achieved further accuracy improvements. Thus, **our method provides practical performance regardless of the datasets**. MGR outperformed the DA methods solely for some datasets and consistently achieved the best accuracy when it is combined with DA. We will add these results to the paper.\n\n**Table R-6. Classification on Aircraft (ResNet-18)**\n| |No TDA|AugMix|RandAug|Trivial Aug|\n| :- | :- | :- | :- | :- |\n|Base Model|62.61$\\pm$.79|64.53$\\pm$.70|63.12$\\pm$.52|66.14$\\pm$.24|\n|MGR|65.11$\\pm$.57|65.65$\\pm$.12|64.98$\\pm$.09|68.17$\\pm$.34|\n\n**Table R-7. Classification on Birds (ResNet-18)**\n|| No TDA          | AugMix          | RandAug         | Trivial Aug     |\n|:-|:-|:-|:-|:-|\n|Base Model|72.24$\\pm$.32|72.44$\\pm$.11|70.80$\\pm$.37|73.61$\\pm$.19|\n|MGR|74.24$\\pm$.34|74.92$\\pm$.62|71.14$\\pm$.23|75.02$\\pm$.29|\n\n---\n\n### **W3: Limitations of the computational cost of meta-learning**\nWe thank you for carefully reading Limitation and Appendix B.3. As we mentioned in Limitation, diffusion models are rapidly speeding up by recent intensive efforts. We can easily imagine that diffusion models achieve a sampling speed comparable to GANs in the near future because the speed-up issue is also important in other applications, such as real-time rendering. For example, a recent work [d] proposed a diffusion-like generative model called consistency model, which achieves high-quality samples (6.2 of FID in ImageNet) in a few steps. In this perspective, the limitation of training speed on diffusion models will be resolved in the near future.\n\n[d] Song, Yang, et al. \"Consistency Models.\" International conference on machine learning (2023).\n\n---\n\n### **W4: It would be interesting to compare this method on ImageNet**\nThank you for the suggestion. We provide the ImageNet results in Table R-1 in the general response. We confirm that **our method stably performs even on the large-scale dataset**. We would be happy if you could see the general response for more details.\n\n---\n\n### **Q1: Does naive generative data augmentation with large pretrained models result in such performance drops? Have the authors tried their method with finetuned GANs?**\n**Yes and Yes. We have tried to use ImageNet pre-trained BigGAN and confirmed that this does not solve the degradation problem of GDA (Table R-8). However, the use of pre-trained models can enhance our method.** We omitted this result from the paper to separate the proposed method's effects from the transfer learning effects by the pre-trained models. Even so, since it is an important fact that fine-tuning does not solve the problem, we will add this result to Appendix. Thank you for pointing out this.\n\n**Table R-8. Classification on Cars (ResNet-18, ImageNet pre-trained BigGAN)**\n|| 10% | 25% | 50% | 100%\n-- | -- | -- | -- | --\nBase Model|20.11$\\pm$.03|49.33$\\pm$.54|72.91$\\pm$.38|85.80$\\pm$.18\nGDA|18.82$\\pm$.22|46.38$\\pm$.59|70.23$\\pm$.66|86.11$\\pm$.16\nMGR|24.55$\\pm$.24|53.41$\\pm$.89|75.46$\\pm$.12|87.17$\\pm$.18"}, {"review_id": "22dHWkkVxf", "reviewer": "Reviewer_gssr", "summary": "The paper proposed meta-generative regularization (MGR) for improving generative data augmentation. MGR is optimized by alternative training between the main and finder network. To train the main network, contrastive learning is used. To train the finder network, the authors propose the bilevel optimization and approximate the solution.", "questions": "Please refer to the weaknesses.\n\n===\n\nI update my rate from 4 to 5 because my concerns are addressed and the authors will reflect the discussion below to the paper.", "limitations": "The limitation is described in Sec. 6 in the main paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method does not update the generative model, but rather updates the finder network. Thus, the foundation generative model can be used.\n2.\tI think that the proposed method does not depend on the type of generative models such as normalizing flow, auto-regressive, gan, and score-based generative models.\n3.\tFDM to reduce the computation complexity.\n4.\tThe effectiveness of the reduced datasets", "weaknesses": "1. CutMix [A] and MixUp [B] rather than AugMix are the strong baselines. These methods are simple, easily implemented, no training, and effective. A comparison is needed.\n2. In Sec. 4.4, I am not sure about the interpretation of UMAP. From the visualization, the diversity of the proposed method is reduced. Thus, the loss term of contrastive learning, L_PCR, might be important. How much the performance is improved when the main model is trained on cross entropy and contrastive loss using only real data?\n3. The result of diffusion model is in the supplementary. However, the experiment of recent generation models, including text-to-image generator, makes the paper more convincing because it shows promising results [C, D, E, F].\n4. How does this method affect the evaluation in terms of robustness, generalizability, or bias? These is possibility to generate the biased samples.\n\n[A] CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features, ICCV 2019\n\n[B] mixup: Beyond Empirical Risk Minimization, ICLR 2018\n\n[C] IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION?, ICLR 2023\n\n[D] Fake it till you make it: Learning transferable representations from synthetic ImageNet clones, 2023\n\n[E] Synthetic Data from Diffusion Models Improves ImageNet Classification, 2023\n\n[F] Generative models improve fairness of medical classifiers under distribution shifts, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments and suggestions.\n\n### **W1: Comparing MGR with CutMix and MixUp**\nThank you for this suggestion. We provide the comparison in Table R-3. **MGR outperformed CutMix and Mixup**. As well as the cases of other DA methods, **the combination of MGR and CutMix/Mixup produces further improvements**. We will add these results in Table 3 of the paper. We would appreciate it if you take a look at the general responses for more details.\n\n---\n\n### **W2: Interpretation of UMAP visualization and the performance of consistency regularization loss with real data**\n> In Sec. 4.4, I am not sure about the interpretation of UMAP. From the visualization, the diversity of the proposed method is reduced. \n\nThank you for the comments. For a more straightforward interpretation, we additionally tried the UMAP visualization on a simple binary classification task (Fig. I in the attached PDF). MGR clearly separates the clusters for each class and forms each cluster to be dense, indicating that **MGR reduces the distance between samples in the same class rather than reduces the diversity in the entire feature space**.\n\n> the loss term of contrastive learning, L_PCR, might be important. How much the performance is improved when the main model is trained on cross entropy and contrastive loss using only real data?\n\nWe agree with your comment: $\\mathcal{L}_\\text{PCR}$ plays an important role in regularizing classifiers. We compared our methods and the consistency regularization with real data (CR on real data) in Table R-2. **Our PCR and MGR outperformed CR on real data**. This implies that synthetic samples help train classifiers with useful features that are not in the real data. Further, we would emphasize that MPS also contributes to performance improvements by meta-learning, which could not be achieved by CR with real samples. Please also see the general response for more details.\n\n---\n\n### **W3: On leveraging pre-trained text-to-image generator**\n> the experiment of recent generation models, including text-to-image generator, makes the paper more convincing because it shows promising results [C, D, E, F].\n\nThank you for the comments. Indeed, the studies of training classifiers with text-to-image generators are increasing and have become popular in the research community. However, we would respectfully point out that our work has a different research question from these works: solving the performance degradation problem of generative data augmentation. The text-to-image generators such as Stable Diffusion used in [C] are pre-trained on large-scale multi-modal datasets. This means that using these models contains the effect of transfer learning; transfer learning across datasets is not included in our claims. Thus, to separate the effects of our method and transfer learning, we focused on the generative models trained on target data only. Since our results show that the synthetic samples are useful even when the target dataset is only available, our contribution is fundamental for leveraging synthetic samples. In future work, we will develop sampling methods taking transfer learning and text-prompting into account. We will add the above discussions in the Limitation section.\n\nOn a related note, the performance study when using fine-tuned generators is discussed in the response for Reviewer g8gt (Q1). Please take a look if you are interested. \n\n---\n\n### **W4: How does MGR affect to robustness, generalizability, or bias? These is possibility to generate the biased samples.**\nThank you for your interesting comments. In short, **our method can improve the robustness against natural corruption and MPS does not generate biased samples toward training distribution.** We tested the robustness of our method on CIFAR-10-C [e], which is a test set for CIFAR-10 corrupted by various transformations. Table R-9 shows the results. While GDA degraded the performance for all corruptions, PCR significantly improved the base model. Furthermore, MGR achieved even higher robustness. This indicates MPS in MGR can provide samples that are useful for generalization through meta-optimization. We will add this result to the paper.\n\n**Table R-9. Classification on CIFAR-10-C (severity: 5)**\n|            | clean | gaussian noise | shot noise | impulse noise | defocus blur | glass blur | motion blur | zoom blur | snow  | frost | fog   | brightness | contrast | elastic transform | pixelate | jpeg compression | mean  |\n| :--------- | :---- | :------------- | :--------- | :------------ | :----------- | :--------- | :---------- | :-------- | :---- | :---- | :---- | :--------- | :------- | :---------------- | :------- | :--------------- | :---- |\n| Base Model | 86.49 | 53.32          | 53.12      | 39.01         | 34.06        | 37.15      | 31.08       | 36.14     | 46.68 | 36.38 | 19.14 | 52.45      | 10.51    | 41.14             | 46.59    | 53.02            | 42.27 |\n| GDA        | 84.11 | 52.70          | 52.98      | 39.94         | 29.39        | 38.66      | 28.97       | 28.82     | 45.04 | 40.08 | 24.00 | 50.29      | 13.07    | 40.91             | 45.11    | 52.85            | 41.68 |\n| PCR        | 87.06 | 59.35          | 60.65      | 37.71         | 47.51        | 50.89      | 41.71       | 47.11     | 63.94 | 57.33 | 31.13 | **73.43**      | 13.66    | 58.00             | 63.77    | 68.55            | 53.86 |\n| **MGR**        | **88.02** | **61.69**          | **62.53**      | **41.62**        | **52.21**        | **51.91**      | **47.94**       | **51.09**     | **65.78** | **59.50** | **34.49** | 73.23      | **14.83**    | **60.65**             | **65.32**    | **71.10**            | **56.37** |\n\n[e] Hendrycks, Dan, and Thomas Dietterich. \"Benchmarking neural network robustness to common corruptions and perturbations.\" International Conference on Learning Representations (2019)."}, {"review_id": "ZRusqYzuEc", "reviewer": "Reviewer_Ja7o", "summary": "This paper proposed a regularization method 'Meta generative regularization' based on the bi-level optimization frame addressed for the generative data augmentation. The MGR is consisited of two terms: pseudo consistency regularization (PCR) and meta pseudo sampling (MPS). The training using MGR is formalized  as alternating optimization of a main classification model and finder network for searching latent vectors of generative model(eg, StyleGAN).To maximize the gain from synthetic samples, MGR regularizes a feature extractor part of the classification model using PCR by effectively sampling useful samples for the generalization from GAN using MPS.", "questions": "See Weaknesses. \n- Eq.(9) is wrong, since the numerator does not contain $\\epsilon$.", "limitations": "The studied problem of this paper is somewhat out-of-date. The effectiveness is limited among existing researches. Especically, the proposed method does not show the effectiveness in some problems with sparse data, e.g., medical imaging.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Use pseudo consistency regularization to address the distortion of decision boundary. \n- Introduce a subnetwork called a finder to improve the training of classifier, and address the unstable training of generator.\n\n", "weaknesses": "-  The data-driven data augmentation is not novel for the community, e.g., AutoAugment [1], Population Based Augmentation [2], Fast AutoAugment [3], ect. The proposed method is expensive in computation, and only achieve comparable or even worse performance than \nhand-designed data augmentation methods, e.g., SnapMix [4]. The advantage of proposed method is not clear to readers.\n- The used meta-learning technique is similar to Generative Teaching Networks [5].\n- The experimental results can not support the effectiveness of proposed method. The compared method should contain other hand-designed data augmentation methods, e.g., mixup, cutmix, ect; and the data-driven data augmentation methods, e.g., AutoAugment [1], Population Based Augmentation [2], Fast AutoAugment [3], ect. Meanwhile, the hand-designed data augmentation methods SnapMix [4] can achieve a significant improvement on CUB, Cars, Aircraft datasets, while does not introduce additional expensive computation.\n- What if the classfication model totally training from scratch on the  synthetic samples generated by the trained finder network for the StyleGAN?\n- Compared with existing data-driven data augmentation methods, proposed method is limited in transferability for other classification tasks.\n\n[1] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. AutoAugment: Learning Augmentation Policies from Data. In CVPR, 2018.\n[2] D. Ho, E. Liang, I. Stoica, P. Abbeel, and X. Chen. Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules. In ICML, 2019.\n[3] S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim. Fast AutoAugment. In NIPS, 2019.\n[4] Huang S, Wang X, Tao D. Snapmix: Semantically proportional mixing for augmenting fine-grained data. In AAAI, 2021, 35(2): 1628-1636.\n[5] Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, Jeff Clune. Generative Teaching Networks.  ICML 2020\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments on the various points of view.\n\n### **W1: The data-driven data augmentation is not novel. What is the advantage of the proposed method over existing data augmentation methods?**\nFirst of all, **the novelty of our work is mainly in solving the performance degradation of generative data augmentation (GDA), not in proposing a new data-driven data augmentation method**. As discussed in Sec. 4.7, data augmentation (DA) and generative data augmentation (GDA) are independent research fields. Thus, they can be combined easily. In this regard, the comparison with DA methods is an indicator of practicality. We have shown that our MGR achieves comparable performance to DA methods and the combination achieves the best performance in Table 3. MGR can outperform the DA baselines by switching the generator to StyleGAN-XL (Table 2), indicating that it continues to improve as generative models evolve in the future. Therefore, **the advantage is a performance improvement that cannot be obtained with DA**. Note that we selected AugMix, RandAugment, and TrivialAugment as the DA baselines because they are more lightweight and powerful baselines than data-driven DA methods such as AutoAugment [f].\n\n[f] Müller, Samuel G., and Frank Hutter. \"Trivialaugment: Tuning-free yet state-of-the-art data augmentation.\" CVPR. 2021.\n\n---\n\n### **W2: Difference between the proposed method and Generative Teaching Network (GTN) [g]**\nThank you for providing related work. **MGR is different and superior to GTN in terms of (I) meta-optimization objective, (II) classifier training, and (III) computation efficiency**. First, MGR meta-optimizes only the finder network for searching optimal samples for classifier training, whereas GTN meta-optimizes entire generators. As a result, MGR avoids the overfitting caused by updating entire generators as reported [h]. Second, MGR trains a classifier with both real and synthetic samples simultaneously, whereas GTN trains it with only synthetic samples. This is also the reason why GTN cannot be a baseline of GDA. This difference comes from the difference in purpose between MGR and GTN: the former is for regularizing classifiers, and the latter is for meta-learning \"data generation\" for fast adaptation. For the regularization purpose, MGR is a more straightforward method than GTN and GTN could not solve the performance degradation problem of GDA. Third, MGR efficiently computes the objective function through approximating the second-order gradient (Eq. (9)), whereas GTN na\\\"ively computes meta-gradients. We will add this discussion to related work.\n\n[g] Such, Felipe Petroski, et al. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. ICML. 2020.\n\n[h] Tero Karras, et al. Training generative adversarial networks with limited data. NeurIPS. 2020\n\n---\n\n### **W3: The experimental results can not support the effectiveness of proposed method. The compared method should contain other hand-designed DA methods, e.g., Mixup, CutMix, SnapMix.**\nWe would respectfully point out that **we have shown the effectiveness in comparison with DA methods in Sec. 4.7 and Table 3**. Additionally, we provide the results with Mixup and CutMix on Table R-3, indicating **our MGR outperformed Mixup, CutMix, and SnapMix**. Please see also the general response for more details. We will add the results to the paper.\n\n---\n\n### **W4: What if the classification model totally training from scratch?**\nThank you for the comment. **MGR can stably perform even when training classifiers from scratch**. Table R-10 shows the same trend as Table 1. We will add the results to the paper.\n\n**Table R-10. Classification on Cars (Scratch ResNet-18)**\n||Top-1 Acc.|\n|:-|:-|\n|Base Model|64.29$\\pm$.40|\n|GDA|62.93$\\pm$.82|\n|MGR|70.62$\\pm$.43|\n\n---\n\n### **W5: The proposed method is limited in transferability for other classification tasks**\n**We have confirmed that MGR can stably perform on multiple classification datasets in Sec. 4.2 and Table 1 (a)**. Therefore, we consider that MGR has task-wide transferability. If we misunderstand something, we would be happy if you could provide additional comments.\n\n---\n\n### **Q1: Eq. (9) is wrong since the numerator does not contain $\\epsilon$**\nWe appreciate you for pointing out this. In L138, $\\theta^{\\pm}$ was incorrectly defined and caused confusion. The correct definition is $\\theta^{\\pm} = \\theta \\pm \\epsilon\\eta\\nabla_\\theta\\mathcal{L}_\\mathrm{val}(\\theta)$. With this modification, $\\epsilon$ appears in the numerator, and Eq.(9) becomes the correct definition.\n\n---\n\n### **L1: The studied problem of this paper is somewhat out-of-dated**\nWe respectfully disagree with this opinion. As the research workshop was held in NeurIPS 2022 [i], machine learning with synthetic samples is an on-going active research topic, not out-of-dated. We believe that our contribution provides new options for the use of synthetic samples and will significantly help develop this research trend.\n\n[i] SyntheticData4ML Workshop. NeurIPS 2022.\n\n---\n\n### **L2: The proposed method does not show the effectiveness in medical imaging.**\nWe are afraid that it is not the case. We evaluated our method on the Chaoyang dataset [j], which is a medical imaging dataset for classifying cancers. Table R-11 shows the result. We confirm that **our MGR effectively performs on medical imaging dataset**.\n\n**Table R-11. Classification on Chaoyang (ResNet-18, StyleGAN2-ADA)**\n|| Top-1 Acc.|\n|:- | :- |\n|Base Model| 83.56$\\pm$.57|\n|GDA|84.23$\\pm$.57|\n|MGR|**87.48**$\\pm$**.15**|\n\n[j] Zhu, Chuang, et al. Hard Sample Aware Noise Robust Learning for Histopathology Image Classification. IEEE transactions on medical imaging."}]

Author rebuttal:
# General Response
We greatly appreciate the reviewers for providing many constructive and insightful comments. We are happy to find all reviewers give scores of 3 (good) or better for the soundness and presentation of our paper. We are also pleased that most reviewers recognize the effectiveness of our method in the Strength section. The feedback from the reviewers concentrates on the suggestions to add experiments to enhance our claims. Thus, we have conducted as many additional experiments as possible to address the concerns. The evaluations on new datasets (e.g., **ImageNet** and **medical imaging**) and baselines (e.g., **CutMix** and **Mixup**) further clarify the performance superiority of our method besides our original claims. For details, please refer to the individual responses. 

In the rest of this response, we provide the parts of additional experiments for answering the shared concerns among the reviewers.

## **Additional experimental results for shared concerns**
---

### **Is the proposed method effective on ImageNet? (Reviewers ZFum and g8gt)**
**Yes, our method (MGR) improves top-1 accuracy on ImageNet.** We evaluated our meta generative regularization (MGR) on ImageNet by randomly initialized ResNet-18 as the classifier and the pre-trained BigGAN as the generator. The experimental setup follows [a]. Table R-1 shows that MGR successfully improves top-1 accuracy, while naive generative data augmentation (GDA) does not; this is the same trend as Table 1 (a) of the main paper. This result indicates that our method consistently works on complex and large-scale datasets. We will add the result in multiple trials with standard deviation to the paper.

**Table R-1. ImageNet Classification (ResNet-18)**
||Top-1 Acc. (ImageNet Val.)|
| :-- | :--|
|Base Model |68.10|
|GDA|64.74|
|MGR (Ours)|**70.55**|

[a] Pytorch - Models and Pre-trained Weights

---
### **What if using real data instead of synthetic samples in consistency regularization? (Reviewers ZFum and g8gt)**
Reviewers **ZFum** and **g8gt** recommended assessing the models' performance with the consistency regularization (CR) computed on *real* datasets. We agree that this baseline is important to evaluate the value of synthetic samples in the regularization. Table R-2 shows the performance of the real consistency regularization (we refer to CR on real data) on Cars dataset; the setting is shared with Table 1 of the main paper. For CR on real data, we used the training dataset for computing Eq. (5). **Our pseudo consistency regularization (PCR) slightly outperformed CR on real data**. This can be because interpolation by the generators helps the feature extractor to capture the difference between images. **Furthermore, PCR enables MGR to search the optimal synthetic samples by using latent vectors of generative models**. In contrast, CR on real data cannot search the optimal real samples for CR loss because real data is fixed in the data space. We will add CR on real data results to Table 1 and the above discussions to the paper.

**Table R-2. Classification on Cars (ResNet-18)**
|| Top-1 Test Acc.|
| :--------- | :-------------- |
| Base Model | 85.80 $\pm$ .10 |
| CR on real data | 86.16 $\pm$ .02 |
| PCR | 86.36 $\pm$ .08 |
| MGR | 87.22 $\pm$ .15 |
| MGR (StyleGAN-XL)| **88.37** $\pm$ **.20**|

---

### **Can the proposed method outperform CutMix, MixUp, and SnapMix? (Reviewers gssr and Ja7o)**
**Yes.** We evaluated CutMix, MixUp, and SnapMix on Cars with ResNet-18 (Table R-3). Our MGR outperformed them. Additionally, MGR boosted the performance of them. This trend is consistent with Table 3. This result also indicates that synthetic samples, which non-linearly interpolate the real samples, elicit better performance than linearly interpolating real samples in the input space by CutMix, Mixup, and SnapMix. We consider this strength is an advantage of using generative models.

**Table R-3. Classification on Cars (ResNet-18)**
||Top-1 Acc.|
|:-|:-|
|Base Model|85.50$\pm$.10|
|CutMix|86.13$\pm$.19|
|Mixup|86.87$\pm$.30|
|SnapMix|87.11$\pm$.20|
|**MGR**|87.22$\pm$.15|
|**MGR+CutMix**|87.80$\pm$.51|
|**MGR+Mixup**|87.60$\pm$.46|
|**MGR+SnapMix**|88.21$\pm$.13|

---

### **Additional UMAP visualization on binary classification dataset (Reviewers ZFum and gssr)**
Reviewers **ZFum** and **gssr** commented on the modification and interpretation of UMAP visualization in Fig. 5. For a more straightforward visualization, we designed a simple binary classification task that separates Pets [b] into dogs and cats, and visualized the feature space of a model trained on this task. Fig. I in the attached PDF shows the feature space after one epoch of training. While GDA failed to separate the clusters for each class, **MGR clearly separates the clusters**. Looking more closely, MGR helps samples to be dense for each class. This is because PCR makes the feature extractor learn slight differences between the synthetics samples that interpolate the real samples.

[b] Parkhi, Omkar M., et al. "Cats and dogs." 2012 IEEE conference on computer vision and pattern recognition. IEEE (2012).

---

Meta-review: The paper received mixed ratings, there was a rebuttal: Four knowledgeable reviewers recommended: Borderline Accept, Accept, Accept, Borderline Accept, and Reject. On balance, it is our recommendation to accept the paper. No basis to overturn the reviews. Authors should attend to the main points in the reviews. when preparing a final version. No basis to overturn the reviews.
Decision: Accept (poster)