Title: Regularizing Neural Networks with Meta-Learning Generative Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

Generative Models

Generative Data Augmentation

Regularization

Meta-Learning
Abstract: This paper investigates methods for improving generative data augmentation for deep learning. Generative data augmentation leverages the synthetic samples produced by generative models as an additional dataset for classification with small dataset settings. A key challenge of generative data augmentation is that the synthetic data contain uninformative samples that degrade accuracy. This can be caused by the synthetic samples not perfectly representing class categories in real data and uniform sampling not necessarily providing useful samples for tasks. In this paper, we present a novel strategy for generative data augmentation called *meta generative regularization* (MGR). To avoid the degradation of generative data augmentation, MGR utilizes synthetic samples for regularizing feature extractors instead of training classifiers. These synthetic samples are dynamically determined to minimize the validation losses through meta-learning. We observed that MGR can avoid the performance degradation of naive generative data augmentation and boost the baselines. Experiments on six datasets showed that MGR is effective particularly when datasets are smaller and stably outperforms baselines by up to 7 percentage points on test accuracy.

Regularizing Neural Networks with Meta-Learning Generative Models

Abstract

This paper investigates methods for improving generative data augmentation for deep learning. Generative data augmentation leverages the synthetic samples produced by generative models as an additional dataset for classification with small dataset settings. A key challenge of generative data augmentation is that the synthetic data contain uninformative samples that degrade accuracy. This is because the synthetic samples do not perfectly represent class categories in real data and uniform sampling does not necessarily provide useful samples for tasks. In this paper, we present a novel strategy for generative data augmentation called meta generative regularization (MGR). To avoid the degradation of generative data augmentation, MGR utilizes synthetic samples in the regularization term for feature extractors instead of in the loss function, e.g., cross-entropy. These synthetic samples are dynamically determined to minimize the validation losses through meta-learning. We observed that MGR can avoid the performance degradation of naïve generative data augmentation and boost the baselines. Experiments on six datasets showed that MGR is effective particularly when datasets are smaller and stably outperforms baselines.

1 Introduction

While deep neural networks achieved impressive performance on various machine learning tasks, training them still requires a large amount of labeled training data in supervised learning. The labeled datasets are expensive when a few experts can annotate the data, e.g., medical imaging. In such scenarios, generative data augmentation is a promising option for improving the performance of models. Generative data augmentation basically adds pairs of synthetic samples from conditional generative models and their target labels into real training datasets. The expectations of generative data augmentation are that the synthetic samples interpolate missing data points and perform as oversampling for classes with less real training samples [1]. This simple method can improve the performance of several tasks with less diversity of inputs such as medical imaging tasks [2, 3, 4, 5].

However, in general cases that require the recognition of more diverse inputs (e.g., CIFAR datasets [6]), generative data augmentation degrades rather than improves the test accuracy [7]. Previous studies have indicated that this can be caused by the low quality of synthetic samples in terms of the diversity and fidelity [7, 8]. If this hypothesis is correct, we can expect high-quality generative models (e.g., StyleGAN2-ADA [9]) to resolve the problem; existing generative data augmentation methods adopt earlier generative models e.g., ACGAN [10] and SNGAN [11]. Contrary to the expectation, this is not the case. We observed that generative data augmentation fails to improve models even when using a high-quality StyleGAN2-ADA (Figure 1). Although the samples partially appear to be real to humans, they are not yet sufficient to train classifiers in

40

60

80

100

Dataset Size (%)

50

60

70

80

Top-1 Accuracy (%)

MGR (Ours)

Regularizing with synthesized samples

Meta-learning by validation performance

existing generative data augmentation methods. This paper investigates methodologies for effectively extracting useful information from generative models to improve model performance.

We address this problem based on the following hypotheses. First, synthetic samples are actually informative but do not perfectly represent class categories in real data . This is based on a finding by Brock et al. [14] called 'class leakage," where a class conditional synthetic sample contains attributes of other classes. For example, they observed failure samples including an image of 'tennis ball" containing attributes of 'dogs" (Figure 4(d) of [14]). These class leaked samples do not perfectly represent the class categories in the real dataset. If we use such class leaked samples for updating classifiers, the samples can distort the decision boundaries, as shown in Figure 3. Second, regardless of the quality, the generative models originally contain uninformative samples to solve the tasks. This is simply because the generative models are not explicitly optimized to generate informative samples for learning the conditional distribution p ( y | x ) ; they are optimized only for learning the data distribution p ( x ) . Further, the generative models often fail to capture the entire data distribution precisely due to their focus on the high-density regions. These characteristics of generative models might disturb the synthesis of effective samples for generative data augmentation. To maximize the gain from synthetic samples, we should select appropriate samples for training tasks.

In this paper, we present a novel regularization method called meta generative regularization (MGR). Based on the above hypotheses, MGR is composed of two techniques for improving generative data augmentation: pseudo consistency regularization (PCR) and meta pseudo sampling (MPS). PCR is a regularization term using synthetic samples in training objectives for classifiers. Instead of supervised learning with negative log-likelihood -log p ( y | x ) , i.e., cross-entropy, on synthetic samples, we regularize the feature extractor to avoid the distortions on decision boundaries. That is, PCR leverages synthetic samples only for learning feature spaces. PCR penalizes the feature extractors by minimizing the gap between variations of a synthetic sample, which is inspired by consistency regularization in semi-supervised learning [15, 16, 17]. MPS corresponds to the second hypothesis and its objective is to select useful samples for training tasks by dynamically searching optimal latent vectors of the generative models. Therefore, we formalize MPS as a bilevel optimization framework of a classifier and a finder that is a neural network for searching latent vectors. Specifically, this framework updates the finder through meta-learning to reduce the validation loss and then updates the classifier to reduce the PCR loss (Figure 2). By combining PCR and MPS, we can improve the performance even when the existing generative data augmentation degrades the performance (Figure 1).

We conducted experiments with multiple vision datasets and observed that MGR can stably improve baselines on various settings by up to 7 percentage points of test accuracy. Further, through the visualization studies, we confirmed that MGR utilizes the information in synthetic samples to learn feature representations through PCR and obtain meaningful samples through MPS.

2 Preliminary

2.1 Problem Setting

We consider a classification problem in which we train a neural network model f θ : X → Y on a labeled dataset D = { ( x i , y i ) ∈ X × Y} N i =1 , where X and Y are the input and output label spaces,

𝑦

!

= 'ball'

True boundary of

'dog' and 'ball'

Distorted boundary

by

ℓ

!

(𝑥

!

, 𝑦

!

)

𝒵

𝒵

𝐹

!

:

finder

𝐺

!

𝑥

"

𝑥

"

𝑥

ℒ(𝜃)

𝜆ℒ

#$%

(𝜃, 𝜙)

𝜃

"

ℒ

#$%

(𝜃

"

)

+

minimize w.r.t.

𝜃

validate

minimize w.r.t.

𝜙

1. Generating samples with

𝐹

!

2. Meta-learning of

𝐹

!

respectively. Here, we can use a generative model G Φ : Z × Y → X , which is trained on D . We assume that G Φ generates samples from a latent vector z ∈ Z and conditions on samples with a categorical label y ∈ Y , where z is sampled from a standard Gaussian distribution p ( z ) = N (0 , I ) and y is uniformly sampled from Y 2 . We refer to the classification task on D as the main task, and f θ as the main model. f θ is defined by a composition of a feature extractor g ψ and a classifier h ω , i.e., f θ = h ω ◦ g ψ and θ = [ ψ, ω ] . To validate f θ , we can use a small validation dataset D val = { ( x i val , y i val ) ∈ X × Y} N val i =1 , which has no intersection with D (i.e., D∩D val = ∅ ).

2.2 Generative Data Augmentation

A typical generative data augmentation trains a main model f θ with both real data and synthetic data from the generative models [19]. We first generate synthetic samples to be utilized as additional training data for the main task. Most previous studies on generative data augmentation [19, 20, 8] adopt conditional generative models for G Φ , and generate a pseudo dataset D p as

where z i is sampled from a prior distribution p ( z ) , and y i p is uniformly sampled from Y . Subsequently, f θ is trained on both of D and D p using the following objective function.

where ℓ is a loss function of the main task (e.g., cross-entropy), ℓ p is a loss function for the synthetic samples, and λ is a hyperparameter for balancing L and L p . In previous works, ℓ p is often set the same as ℓ . Although optimizing Eq. (2) with respect to θ is expected to boost the test performance by interpolating or oversampling conditional samples [1], its naïve application degrades the performance of f θ on general settings [7]. In this paper, we explore methods to resolve the degradation of generative data augmentation and maximize the performance gain from D p .

3 Proposed Method

In this section, we describe our proposed method, MGR. The training using MGR is formalized as alternating optimization of a main model and finder network for searching latent vectors of G Φ , as shown in Figure 2. To maximize the gain from synthetic samples, MGR regularizes a feature extractor g ψ of f θ using PCR by effectively sampling useful samples for the generalization from G Φ using MPS. We show the overall algorithm of MGR in Appendix A.

3.1 Pseudo Consistency Regularization

As discussed in Secion 1, we hypothesize that the synthetic samples do not perfectly represent class categories, and training classifiers using them can distort the decision boundaries. This is because y p is not reliable due to D p can contain class leaked samples [14]. To avoid the degradation caused by the distortion, we propose utilizing x p to regularize only the feature extractor g ψ of f θ by discarding a conditional label y p . For the regularization, we borrow the concept of consistency regularization, which was originally proposed for semi-supervised learning (SSL) [15, 16, 17]. These SSL methods were designed to minimize the dissimilarity between the two logits (i.e., the output of h ω ) of strongly and weakly transformed unlabeled samples to obtain robust representations. By following this concept, the PCR loss is formalized as

where T is a strong transformation such as RandAugment [21], which is similar to the one used in UDA [16] for SSL. The difference between PCR and UDA is that PCR penalizes only g ψ , whereas UDA trains the entire f θ = h ω ◦ g ψ . ℓ PCR can be expected to help g ψ learns features of inter-cluster interpolated by x p without distorting the decision boundaries. Using ℓ PCR, we rewrite Eq. (4) as

3.2 Meta Pseudo Sampling

Most generative data augmentation methods generate a synthetic sample x p from G Φ with a randomly sampled latent vector z . This is not the best option as G Φ is not optimized for generating useful samples to train f θ in predicting the conditional distribution p ( y | x ) ; it is originally optimized to replicate p ( x | y ) or p ( x ) . The main concept of MPS is to directly determine useful samples for training f θ by optimizing an additional neural network called a finder F ϕ : Z → Z . F ϕ takes a latent vector z ∼ p ( z ) as input and outputs a vector of the same dimension as z . By using F ϕ , we generate a synthetic sample as x p = G Φ ( F ϕ ( z ) , y ) . The role of F ϕ is to find the optimal latent vectors that improve the training of f θ through x p . Although we can consider optimizing G Φ instead of F ϕ , we optimize F ϕ because the previous work showed that transforming latent variables according to loss functions efficiently reaches the desired outputs [22] and we observed that optimizing G Φ is unstable and causes the performance degradations of f θ (Section 4.5).

The useful samples for generalization should reduce the validation loss of f θ by using them for optimization. Based on this simple suggestion, we formalize the following bilevel optimization problem for F ϕ .

Note that the finder parameter ϕ is added to the arguments of L PCR. We can optimize F ϕ with a standard gradient descent algorithm because F ϕ , G Φ , and f θ are all composed of differentiable functions as well as existing meta-learning methods such as MAML [23] and DARTS [24]. We approximate θ ∗ because the exact computation of the gradient ∇ ϕ L val ( θ ∗ ) is expensive [23, 24]:

where η is an inner step size. Thus, we update F ϕ by using θ ′ that is updated for a single step and then alternately update θ by applying Eq. (6) with x p generated from the updated F ϕ . The overall optimization flow is shown in Figure 4.

Approximating gradients with respect to ϕ . Computing ∇ ϕ L val ( θ ′ ) requires the product computation including second-order gradients: ∇ 2 ϕ,θ L PCR ( θ, ϕ ) ∇ θ ′ L val ( θ ′ ) . This causes a computation complexity of O (( | Φ | + | ϕ | ) | θ | ) . To avoid this computation, we approximate the term using the finite difference method [24] as

where θ ± is θ updated by θ ± εη ∇ θ L val ( θ ) . ε is defined by const. ∥∇ θ L val ( θ ) ∥ 2 . We used 0 . 01 of the constant for ε based on [24]. This approximation reduces the computation complexity to O ( | Φ | + | ϕ | + | θ | ) . We confirm the speedup by Eq. (9) in Appendix B.1.

Techniques for improving finder We introduce two techniques for improving the training of F ϕ in terms of the architectures and a penalty term for the outputs.

While arbitrary neural architectures can be used for the implementation of F ϕ , we observed that the following residual architecture produces better results.

where MLP is multi layer perceptron.

To ensure that F ϕ ( z ) does not diverge too far from the distribution p ( z ) , we add a Kullback-Leibler divergence term D KL ( p ϕ ( z ) ∥ p ( z )) , where p ϕ ( z ) is the distribution of F ϕ ( z ) into Eq. (7). When p ( z ) follows the standard Gaussian N (0 , I ) , D KL ( p ϕ ( z ) ∥ p ( z )) can be computed by

where µ ϕ and σ ϕ is the mean and variance of { F ϕ ( z i ) } N p i =1 . In Appendix B.2, we discuss the effects of design choice based on the ablation study.

4 Experiments

In this section, we evaluate our MGR (the combination of PCR and MPS with the experiments on multiple image classification datasets. We mainly aim to answer three research questions with the experiments: (1) Can PCR avoid the negative effects of x p in existing methods and improve the performance of f θ ? (2) Can MPS find better samples for training than those by uniform sampling? (3) How practical is the performance of MGR? We compare MGR with baselines including conventional generative data augmentation and its variants in terms of test performance (Sec. 4.2 and 4.3). Furthermore, we conduct a comprehensive analysis of PCR and MPS such as the visualization of trained feature spaces (Sec. 4.4), quantitative/qualitative evaluations of the synthetic samples (Sec. 4.5), performance studies when changing generative models (Sec. 4.6), and comparison to data augmentation methods such as TrivialAugment [25] (Sec. 4.7).

4.1 Settings

Baselines. Wecompare our method with the following baselines. Base Model : training f θ with only D . Generative Data Augmentation (GDA) : training f θ with D and G Φ using Eq. (2). GDA+MH : training f θ with D and G Φ by decoupling the heads into h ω for D and h ω p for D p . MH denotes multi-head. This is a naïve approach to avoid the negative effect of x p on h ω by not passing x p through h ω . GDA+MH optimizes the parameters as argmin L ( f θ ( x ) , y ) + λ L ( h ω p ( g ψ ( x p )) , y p ) .

θ

GDA+SSL : training f θ with D and G Φ by applying an SSL loss for D p that utilizes the output of h ω unlike PCR. This method was originally proposed by Yamaguchi et al. [26] for transfer learning, but we note that G Φ was trained on the main task dataset D in contrast to the original paper. By following [26], we used UDA [16] as the SSL loss and the same strong transformation T as of MGR for the consistency regularization.

Datasets. We used six image datasets for classification tasks in various domains: Cars [12], Aircraft [27], Birds [28], DTD [29], Flowers [30], and Pets [31]. Furthermore, to evaluate smaller dataset cases, we used subsets of Cars that were reduced by { 10 , 25 , 50 , 75 } % in volume; we reduced them by random sampling on a fixed random seed. We randomly split a dataset into 9 : 1 and used the former as D and the latter as D val .

Architectures. We used ResNet-18 [13] as f θ and generators of conditional StyleGAN2-ADA for 256 × 256 images [18] as G Φ . F ϕ was composed of a three-layer perceptron with a leaky-ReLU activation function. We used the ImageNet pre-trained weights of ResNet-18 distributed by PyTorch. 3 For StyleGAN2-ADA, we did not use pre-trained weights. We trained G Φ on each D from scratch according to the default setting of the implementation of StyleGAN2-ADA. 4 Note that we used the same G Φ in the baselines and our proposed method.

(a) Multiple Datasets

(b) Small Datasets (Cars)

Training. We trained f θ by the Nesterov momentum SGD for 200 epochs with a momentum of 0.9, and an initial learning rate of 0.01; we decayed the learning rate by 0.1 at 60, 120, and 160 epochs. We trained F ϕ by the Adam optimizer for 200 epochs with a learning rate of 1 . 0 × 10 -4 . We used mini-batch sizes of 64 for D and 64 for D p . The input samples were resized into a resolution of 224 × 224 ; x p was resized by differentiable transformations. For synthetic samples from G Φ in PCR and GDA+SSL, the strong transformation T was RandAugment [21] by following [16], and it was implemented with differentiable transformations provided in Kornia [32]. We determined the hyperparameter λ by grid search among [0 . 1 , 1 . 0] with a step size of 0 . 1 for each method by D val. To avoid overfitting, we set the hyperparameters of MGR that are searched with only applying PCR i.e., we did not use meta-learning to choose them. We used λ KL of 0 . 01 . We selected the final model by checking the validation accuracy for each epoch. We ran the experiments three times on a 24-core Intel Xeon CPU with an NVIDIA A100 GPU with 40GB VRAM and recorded average test accuracies with standard deviations evaluated on the final models.

4.2 Evaluation on Multiple Datasets

We confirm the efficacy of MGR across multiple datasets. Table 1a shows the top-1 accuracy scores of each method. As reported in [7], GDA degraded the base model on many datasets; it slightly improved the base model on only one dataset. GDA+MH, which had decoupled classifier heads for GDA, exhibited a similar trend to GDA. This indicates that simply decoupling the classifier heads is not a solution to the performance degradation caused by synthetic samples. In contrast, our MGR stably and significantly outperformed the baselines and achieved the best results. The ablation of MGR discarding PCR or MPS is listed in Table 1a. We confirm that both PCR and GDA+MPS improve GDA. While GDA+SSL underperforms the base models, PCR outperforms the base models. This indicates that using unsupervised loss alone is not sufficient to eliminate the negative effects of the synthetic samples and that discarding the classifier h ω from the regularization is important to obtain the positive effect. MPS yields only a small performance gain when combined with GDA, but it significantly improves its performance when combined with PCR i.e., MGR. This suggests that there is no room for performance improvements in GDA, and MPS can maximize the potential benefits of PCR.

4.3 Evaluation on Small Datasets

A small dataset setting is one of the main motivations for utilizing generative data augmentation. We evaluate the effectiveness of MGR on smaller datasets. Table 1b shows the performance when reducing the Cars dataset into a volume of { 10 , 25 , 50 , 75 } % . Note that we trained G Φ on each reduced dataset, not on 100% of Cars. In contrast to the cases of the full dataset (Table 1a), no baseline methods outperformed the base model in this setting. This is because G Φ trained on the small datasets generates low-quality samples with less reliability on the conditional label y p that are not appropriate in supervised learning. On the other hand, MGR improved the baselines in large margins. This indicates that, even when the synthetic samples are not sufficient to represent the class categories, our MGR can maximize the information obtained from the samples by utilizing them to regularize feature extractors and dynamically finding useful samples.

5 epoch

15 epoch

30 epoch

GDA

10

5

0

5

10

15

5

0

5

10

15

20

Real

Synthetic

10

5

0

5

10

15

20

15

10

5

0

5

10

15

Real

Synthetic

15

10

5

0

5

10

15

15

10

5

0

5

10

15

Real

Synthetic

PCR

5

0

5

10

15

20

10

5

0

5

10

15

Real

Synthetic

5

0

5

10

15

20

25

10

5

0

5

10

15

20

Real

Synthetic

5

0

5

10

15

20

25

10

5

0

5

10

15

20

Real

Synthetic

PCR+MPS

5

0

5

10

15

20

25

15

10

5

0

5

10

15

Real

Synthetic

5

0

5

10

15

20

25

10

5

0

5

10

15

20

Real

Synthetic

5

0

5

10

15

20

25

15

10

5

0

5

10

15

Real

Synthetic

1

4.4 Visualization of Feature Spaces

In this section, we discuss the effects of PCR and MPS through the visualizations of feature spaces in training. To visualize the output of g ψ in 2D maps, we utilized UMAP [33] to reduce the dimensions. UMAP is a visualization method based on the structure of distances between samples, and the low dimensional embeddings can preserve the distance between samples of the high dimensional input. Thus, the distance among the feature clusters on UMAP visualization of g ψ can represent the difficulty of the separation by h ω . We used the official implementation by [33] 5 and its default hyperparameters. We plotted the UMAP embeddings of g ψ ( x ) and g ψ ( x p ) at { 5 , 15 , 30 } epochs, as shown in Figure 5; we used ResNet-18 trained on the Cars dataset. At first glance, we observe that GDA and PCR formed completely different feature spaces. GDA forms the feature spaces by forcing to treat the synthetic samples the same as the real samples through cross-entropy loss and trying to separate the clusters of samples according to the class labels. However, the synthetic samples leaked to the inter-cluster region at every epoch because they could not represent class categories perfectly as discussed in Sec. 1. This means that the feature extractor might be distorted to produce features that confuse the classifier. On the other hand, the synthetic samples in PCR progressively formed a cluster at the center, and the outer clusters can be seen well separated. Since UMAP can preserve the distances between clusters, we can say that the sparse clusters that exist far from the center are considered easy to classify, while the dense clusters close to the center are considered difficult to classify. In this perspective, PCR helps g ψ to leverage the synthetic samples for learning feature representations interpolating the dense difficult clusters. This is because the synthetic samples tend to be in the middle of clusters due to their less representativeness of class categories. That is, PCR can utilize the partial but useful information contained in the synthetic samples while avoiding the negative effect. Further, we observe that applying MPS accelerates the convergence of the synthetic samples into the center.

For a more straightforward visualization of class-wise features, we designed a simple binary classification task that separates Pets [31] into dogs and cats, and visualized the feature space of a model trained on this task. Fig. 6 shows the feature space after one epoch of training. While GDA failed to separate the clusters for each class, MGR clearly separated the clusters. Looking more closely, MGR helps samples to be dense for each class. This is because PCR makes the feature extractor learn slight differences between the synthetic samples that interpolate the real samples. From these observations, we conclude that our MGR can help models learn useful feature representations for solving tasks.

2

0

2

4

6

8

10

0

1

2

3

4

5

6

7

Real Dog

Real Cat

Synthetic Dog

Synthetic Cat

(a) GDA

5

0

5

10

15

2

0

2

4

6

8

Real Dog

Real Cat

Synthetic Dog

Synthetic Cat

(b) MGR

0

50

100

150

200

0.7

0.8

0.9

1.0

1.1

1.2

1.3

1.4

val_loss

Uniform

MPS

(a) Validation Loss

0

50

100

150

200

20

21

22

23

24

25

FID

Uniform

MPS

(b) FID

4.5 Analysis of MPS

Evaluation of validation loss. We investigate the effects on validation losses when using MPS. Through the meta-optimization by Eq. (7), MPS can generate samples that reduce the validation loss of f θ . We recorded the validation loss per epoch when applying uniform sampling (Uniform) and when applying MPS. We used the models trained on Cars and applied the PCR loss on both models. Figure 7a plots the averaged validation losses. MPS reduced the validation loss. In particular, MPS was more effective in early training epochs. This is related to accelerations of converging the central cluster of synthetic samples discussed in Section 4.4 and Figure 5. That is, MPS can produce effective samples for regularizing features and thus speed up the entire training of f θ .

Quantitative evaluation of synthetic samples. We evaluate the synthetic samples generated by MPS. To assess the characteristics of the samples, we measured the difference between the data distribution and distribution of the synthetic samples. We leveraged the Fréchet Inception distance (FID, [34]), which is a measurement of the distribution gap between two datasets using the closedform computation assuming multivariate normal distributions:

where µ and Σ are the mean and covariance of the feature vectors on InceptionNet for input { x i } . Since FID is a distance, the lower FID ( D , D p ) means that D p contains more high-quality samples in terms of realness. We computed FID scores using 2048 samples in D and 2048 synthetic samples every epoch; the other settings are given in Section 4.1. The FID scores in training are plotted in Figure 7b. We confirm that MPS consistently produced higher-quality samples than Uniform. This indicates that the sample quality is important for generalizing f θ even in PCR, and uniform sampling can miss higher quality samples in generative models. Since the performance gain by GDA+MPS in Table 1a did not better than MGR, the higher-quality samples by MPS can still contain uninformative samples for the cross-entropy loss, but they are helpful for PCR learning good feature representations.

Qualitative evaluation of synthetic samples. We evaluate the qualitative properties of the synthetic samples by visualizing samples of a class. The samples generated by Uniform and MPS are shown in Figure 8a and 8b, where the dataset was Cars and the class category was Hummer . Compared with

(a) Uniform

(b) MPS

Uniform, MPS produced samples with more diverse body colors and backgrounds. That is, MPS focuses on the color and background of the car as visual features for solving classifications. In fact, since Hummer has various colors of car bodies and can drive on any road with four-wheel drive, these selective generations of MPS are considered reasonable in solving classification tasks.

4.6 Effect of Generative Models

Here, we evaluate MGR by varying the generative model G Φ for producing synthetic samples. As discussed in Sec. 3.1 and 3.2, MGR can use arbitrary unconditional/conditional generative models as G Φ unless it has a latent space. To confirm the effects when changing G Φ , we tested MGR with FastGAN [35], BigGAN [18], and StyleGAN-XL [36]. Table 2 shows the results on Cars. The unconditional FastGAN achieved similar improvements as conditional cases. However, since unconditional generative models are generally of low quality in FID, they are slightly less effective. For the conditional generative models, we observe that MGR performance improvement increases as the quality of synthetic samples improves. These results suggest the potential for even greater MGR gains in the future as the performance of the generative model improves. We also evaluate MGRwith recent diffusion models in Appendix B.3. Meanwhile, to evaluate the value of synthetic samples in the consistency regularization, we compare MGR with the case where we use real samples in Eq. (6) (Real CR). Our method outperformed Real CR. This can be because interpolation by the generators helps the feature extractor to capture the difference between images. Furthermore, our method searches the optimal synthetic samples by using latent vectors of generative models with MPS. In contrast, Real CR cannot search the optimal real samples for CR loss because real data is fixed in the data space.

4.7 Combination of MGR and Data Augmentation

To assess the practicality of MGR, we evaluate the comparison and combination of MGR and existing data augmentation methods. Data augmentation (DA) is a method applied to real data and is an independent research field from generative data augmentation. Therefore, MGR can be combined with DA to improve performance further. Table 3 shows the evaluation results when comparing and combining MGR and DA methods; we used MixUp [37], CutMix [38], AugMix [39], RandAugment [21], SnapMix [40], and TrivialAugment [25] as the DA methods. The improvement effect of MGR is comparable to that of DA, and the highest accuracy was achieved by combining the two methods. In particular, MGR was stably superior to sample-mix-based such as MixUp and AugMix. This result also indicates that synthetic samples, which non-linearly interpolate the real samples, elicit better performance than linearly interpolating real samples by the sample-mix-based DA methods. We consider this strength to be an advantage of using generative models.

5 Related Work

We briefly review generative data augmentation and training techniques using generative models.

The earliest works of generative data augmentation are [19, 41, 42]. They have demonstrated that simply adding synthetic samples as augmented data for classification tasks can improve performance in few-shot learning, person re-identification, and medical imaging tasks. Tran et al.[20] have

proposed a generative data augmentation method that simultaneously trains GANs and classifiers for optimizing θ to maximize the posterior p ( θ | x ) by an EM algorithm. Although this concept is similar to our MPS in terms of updating both G Φ and f θ , it requires training the specialized neural architectures based on GANs. In contrast, MPS is formalized for arbitrary existing generative models with latent variables, and it requires no restrictions to the training objectives of generative models.

On the analysis of generative data augmentation, Shmelkov et al. [7] have pointed out that leveraging synthetic samples as augmented data degrades the performance in general visual classification tasks. They have hypothesized that the cause of the degradation is the less diversity and fidelity of synthetic samples from generative models. Subsequent research by Yamaguchi et al. [8] have shown that the scores related to the diversity and fidelity of synthetic samples (i.e., SSIM and FID) are correlated to the test accuracies when applying the samples for generative data augmentation in classification tasks. Based on these works, our work reconsiders the training objective and sampling method in generative data augmentation and proposes PCR and MPS.

More recently, He et al. [43] have reported that a text-to-image generative model pre-trained on massive external datasets can achieve high performance on few-shot learning tasks. They also found that the benefit of synthetic data decreases as the amount of real training data increases, which they attributed to a domain gap between real and synthetic samples. In contrast, our method does not depend on any external dataset and successfully improves the accuracy of the classifier even when synthetic data are not representatives of class labels (i.e., having domain gaps).

6 Limitation

One of the limitations of our method is the requirement of bilevel optimization of classifier and finder networks. This optimization is computationally expensive particularly when used with generative models that require multiple inference steps, such as diffusion models as discussed in Appendix B.3. We have tried other objective functions not requiring bilevel optimization, but at this time, we have not found an optimization method that outperforms MPS (see Appendix B.4). Nevertheless, since recent studies rapidly and intensively focus on the speedup of diffusion models [44, 45, 46], we can expect that this limitation will be negligible in near the future. Additionally, applying MGR to pre-trained text-to-image diffusion models (e.g., [44]) is also important for future work because they have succeeded in producing effective samples for training downstream task models in a zero-shot manner [43].

7 Conclusion

This paper presents a novel method for generative data augmentation called MGR. MGR is composed of two techniques: PCR and MPS. To avoid the degradation of classifiers, PCR utilizes synthetic samples to regularize feature extractors by the simple consistency regularization loss. MPS searches the useful synthetic samples to train the classifier through meta-learning on the validation loss of main tasks. We empirically showed that MGR significantly improves the baselines and brings generative data augmentation up to a practical level. We also observed that the synthetic samples in existing generative data augmentation can distort the decision boundaries on feature spaces, and the PCR loss with synthetic samples dynamically generated by MPS can resolve this issue through visualization. We consider that these findings will help future research in this area.

References

Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data , 2019.

Francesco Calimeri, Aldo Marzullo, Claudio Stamile, and Giorgio Terracina. Biomedical data augmentation using generative adversarial neural networks. In International conference on artificial neural networks , pages 626-634. Springer, 2017.

Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Gan-based synthetic medical image augmentation for increased cnn performance in liver lesion classification. Neurocomputing , 2018.

Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Synthetic data augmentation using gan for improved liver lesion classification. In 2018 IEEE 15th International Symposium on Biomedical Imaging , 2018.

Abdul Waheed, Muskan Goyal, Deepak Gupta, Ashish Khanna, Fadi Al-Turjman, and Plácido Rogerio Pinheiro. Covidgan: data augmentation using auxiliary classifier gan for improved covid-19 detection. Ieee Access , 2020.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In Proceedings of the European Conference on Computer Vision , 2018.

Shin'ya Yamaguchi, Sekitoshi Kanai, and Takeharu Eda. Effective data augmentation with multi-domain learning gans. In Proceedings of the AAAI Conference on Artificial Intelligence , 2020.

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems , 2020.

Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International Conference on Machine Learning , 2017.

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations , 2018.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition , Sydney, Australia, 2013.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2016.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2019.

Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in neural information processing systems , 2014.

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems , 2020.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. In Advances in Neural Information Processing Systems , 2020.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2018.

Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv:1711.04340 , 2017.

Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid. A bayesian data augmentation approach for learning deep models. In Advances in Neural Information Processing Systems 30 , 2017.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 702-703, 2020.

Ruoshi Liu, Chengzhi Mao, Purva Tendulkar, Hao Wang, and Carl Vondrick. Landscape learning for neural network inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , 2017.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations , 2019.

Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 774-782, 2021.

Shin'ya Yamaguchi, Sekitoshi Kanai, Atsutoshi Kumagai, Daiki Chijiwa, and Hisashi Kashima. Transfer learning with pre-trained conditional generative models. arXiv preprint arXiv:2204.12833 , 2022.

S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv , 2013.

P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical report, California Institute of Technology, 2010.

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2014.

M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing , 2008.

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition , 2012.

Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2020.

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems , 2017.

Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations , 2021.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH , 2022.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , 2019.

Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. In Proceedings of the International Conference on Learning Representations , 2020.

Shaoli Huang, Xinchao Wang, and Dacheng Tao. Snapmix: Semantically proportional mixing for augmenting fine-grained data. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In Proceedings of the IEEE International Conference on Computer Vision , 2017.

Francesco Calimeri, Aldo Marzullo, Claudio Stamile, and Giorgio Terracina. Biomedical data augmentation using generative adversarial neural networks. In International Conference on Artificial Neural Networks . Springer, 2017.

Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems , 2022.

Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision , 2022.

Ron Mokady, Omer Tov, Michal Yarom, Oran Lang, Inbar Mosseri, Tali Dekel, Daniel CohenOr, and Michal Irani. Self-distilled stylegan: Towards generation from internet photos. In ACM SIGGRAPH 2022 Conference Proceedings , 2022.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2019.

Appendix

The following manuscript provides the supplementary materials of the main paper: Regularizing Neural Networks with Meta-Learning Generative Models.

A Algorithm of Meta Generative Regularization

Algorithm 1 Meta Generative Regularization

Require: Training dataset D , validation dataset D val main model f θ , generator G , finder F ϕ , training batchsize B , pseudo batchsize B p , validation batchsize B val, step size η and ξ , hyperparameter λ and λ KL

Ensure: Trained main model f θ

1: while not converged do

2: { ( x i , y i ) } B i =1 ∼ D

3: { z i } B p i =1 ∼ N (0 , I )

4: // Updating ϕ for MPS

5: { ( x i val , y i val ) } B val i =1 ∼ D

6: { x i p } B p i =1 = { G Φ ( F ϕ ( z i ) , y i p ) } B p i =1

7: θ ′ ← θ -η ∇ θ ( 1 B ℓ ( f θ ( x i ) , y i ) + λ B p ℓ PCR ( x i p ; ψ ))

8: ϕ ← ϕ -ξ ∇ ϕ ( 1 B val ℓ ( f θ ′ ( x val ) , y val ) + λ KL ( D KL ( p ϕ ( z ) ∥ p ( z ))))

9: // Updating θ with PCR

10: { x i p } B p i =1 = { G Φ ( F ϕ ( z i ) , y i p ) } B p i =1

11: θ ← θ -η ∇ θ ( 1 B ℓ ( f θ ( x i ) , y i ) + λ B p ℓ PCR ( x i p ; ψ ))

12: end while

B Additional Experiments

B.1 Evaluation of Gradient Approximation

Here, we evaluate the gradient approximation by Eq. (9). As shown in Table 4, Eq. (9) well approximated the second-order gradients in speeding up over 10% with 0 . 08 of the accuracy drop.

B.2 Ablation study of F ϕ

In Section 3.2, we introduce F ϕ for meta-optimized parameters and the residual architectures with MLP defined by Eq. (10). We performed an ablation study of MPS with respect to the meta-optimized parameters and the architectures of F ϕ . We compared MPS with a variant of MPS optimizing G Φ instead of F ϕ . We also attempted other architectures for F ϕ including Linear : W ϕ ( z ) + b , MLP : MLP ϕ ( z ) , and Residual+Shallow : z +tanh( W ϕ ( z ) + b ) . The results of these variations are shown in Table 5. We observed that MPS with G Φ caused failures of training f θ and degraded the accuracy. On the other hand, all variants of MPS with F ϕ succeeded in boosting the models without MPS. Thus, restricting the number of optimized parameters is important, and determining an optimal z with the finder F ϕ is effective on the optimization problems of MPS. For the variants of MPS with F ϕ , we observed that the residual architectures and regularization by D KL ( p ϕ ( z ) ∥ p ( z )) contributed to the successes. Interestingly, MPS with Linear F ϕ outperformed MPS with MLP F ϕ , i.e., significantly transforming the input z ∼ p ( z ) by complex functions results in low accuracy. These results suggest that better latent vectors in Z to train f θ can exist near the uniformly sampled input z . Thus, limiting the search range by tanh in the residual architectures can help in finding better latent vectors.

B.3 MGRwith Diffusion Models

We tested our method on EDM [45], a recent diffusion model. Due to the computation cost, we used a 10% reduced CIFAR-10 as the dataset. We optimized F ϕ to search the first step noise of the diffusion process. Table B-4 shows that our method with EDM improves Base Model. However, the overhead of incorporating diffusion models was significant; it takes more than ten times longer training than GANs. In future work, we will investigate lighter-weight methods using the diffusion model.

B.4 Updating F ϕ without Meta-optimization

MPS consists of meta-learning on validation losses requiring bi-level optimization, which is a relatively heavy computation. One can consider if F ϕ could be trained without meta-optimization. Here, we try alternative methods other than meta-learning to update F ϕ . Instead of meta-learning, we used a strategy of choosing hard examples via optimizing F ϕ . That is, we optimize F ϕ by maximizing the training cross-entropy (CE) loss and the PCR loss on synthetic samples. Note that, in both cases, we used the PCR loss for synthetic samples when training classifiers. Table A-2 shows the results. Optimizing F ϕ with CE and PCR slightly improved the baselines but significantly underperformed our method (MGR). This result can justify using meta-optimizing F ϕ to generate useful samples for classifiers. Nevertheless, this idea could inspire a sampling method that does not require bi-level optimization in future work.

B.5 ImageNet Classification

We evaluated our MGR on ImageNet by randomly initializing ResNet-18 as the classifier and the pre-trained BigGAN as the generator. Table 8 shows that MGR successfully improves top-1 accuracy, while naive generative data augmentation (GDA) does not; this is the same trend as Table 1 (a) of the main paper. This result indicates that our method consistently works on complex and large-scale datasets. We will add the result in multiple trials with standard deviation to the paper.

B.6 Latent Augmentation

Some readers may wonder why not also utilize augmentation in the generative model's latent space instead of data space. We implemented this approach by adding Gaussian noise to the latent vector as z ′ = z + s , where s ∼ N (0 , 10 -3 ) . Then, we compute the consistency regularization between g ( G ( z )) and g ( G ( z ′ )) , instead of g ( G ( z )) and g ( T ( G ( z ))) . We call this variant LatentAugment. We found that LatentAugment improves the performance of our MGR (Table 9). Interestingly, LatentAugment can improve when it is used solely without image data augmentation T , i.e., RandAugment. This indicates that we can obtain meaningful variants by perturbing latent vectors, which is challenging for conventional data augmentation. However, this approach doubles the number of generators' forward computations and thus leads to increased computation time and memory footprint.

B.7 Comparison to Existing Sampling Technique

Here, we evaluate MPS by comparing it with an existing sampling technique called multi-modal truncation sampling [47], which was originally proposed for sampling better quality samples for image generation. Table 10 shows the result of combining multi-modal truncation sampling and PCR. Although the FID score of multi-modal truncation sampling certainly outperforms MPS, the gain of the classification accuracy underperforms MPS. This implies that improving sample quality is a necessary condition for performance improvements, not a sufficient condition. Meanwhile, since MPS explicitly searches for samples that minimize the validation loss, it can improve classifiers more directly than incorporating existing sampling methods.

B.8 Performance Study when using Pre-trained Generators

We tried to use ImageNet pre-trained BigGAN and confirmed that this does not solve the degradation problem of GDA (Table 11). However, the use of pre-trained models can enhance our method.

B.9 Evaluation of Robustness toward Natural Corruption

Our method can improve the robustness against natural corruption and MPS does not generate biased samples toward training distribution. We tested the robustness of our method on CIFAR-10-C [48], which is a test set for CIFAR-10 corrupted by various transformations. Table 12 shows the results.

While GDA degraded the performance for all corruptions, PCR significantly improved the base model. Furthermore, MGR achieved even higher robustness. This indicates MPS in MGR can provide samples that are useful for generalization through meta-optimization.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a regularization method 'Meta generative regularization' based on the bi-level optimization frame addressed for the generative data augmentation. The MGR is consisited of two terms: pseudo consistency regularization (PCR) and meta pseudo sampling (MPS). The training using MGR is formalized  as alternating optimization of a main classification model and finder network for searching latent vectors of generative model(eg, StyleGAN).To maximize the gain from synthetic samples, MGR regularizes a feature extractor part of the classification model using PCR by effectively sampling useful samples for the generalization from GAN using MPS.\n\nSTRENGTHS:\n- Use pseudo consistency regularization to address the distortion of decision boundary. \n- Introduce a subnetwork called a finder to improve the training of classifier, and address the unstable training of generator.\n\nWEAKNESSES:\n-  The data-driven data augmentation is not novel for the community, e.g., AutoAugment [1], Population Based Augmentation [2], Fast AutoAugment [3], ect. The proposed method is expensive in computation, and only achieve comparable or even worse performance than \nhand-designed data augmentation methods, e.g., SnapMix [4]. The advantage of proposed method is not clear to readers.\n- The used meta-learning technique is similar to Generative Teaching Networks [5].\n- The experimental results can not support the effectiveness of proposed method. The compared method should contain other hand-designed data augmentation methods, e.g., mixup, cutmix, ect; and the data-driven data augmentation methods, e.g., AutoAugment [1], Population Based Augmentation [2], Fast AutoAugment [3], ect. Meanwhile, the hand-designed data augmentation methods SnapMix [4] can achieve a significant improvement on CUB, Cars, Aircraft datasets, while does not introduce additional expensive computation.\n- What if the classfication model totally training from scratch on the  synthetic samples generated by the trained finder network for the StyleGAN?\n- Compared with existing data-driven data augmentation methods, proposed method is limited in transferability for other classification tasks.\n\n[1] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. AutoAugment: Learning Augmentation Policies from Data. In CVPR, 2018.\n[2] D. Ho, E. Liang, I. Stoica, P. Abbeel, and X. Chen. Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules. In ICML, 2019.\n[3] S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim. Fast AutoAugment. In NIPS, 2019.\n[4] Huang S, Wang X, Tao D. Snapmix: Semantically proportional mixing for augmenting fine-grained data. In AAAI, 2021, 35(2): 1628-1636.\n[5] Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, Jeff Clune. Generative Teaching Networks.  ICML 2020\n\nQUESTIONS:\nSee Weaknesses. \n- Eq.(9) is wrong, since the numerator does not contain $\\epsilon$.\n\nLIMITATIONS:\nThe studied problem of this paper is somewhat out-of-date. The effectiveness is limited among existing researches. Especically, the proposed method does not show the effectiveness in some problems with sparse data, e.g., medical imaging.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed meta-generative regularization (MGR) for improving generative data augmentation. MGR is optimized by alternative training between the main and finder network. To train the main network, contrastive learning is used. To train the finder network, the authors propose the bilevel optimization and approximate the solution.\n\nSTRENGTHS:\n1.\tThe proposed method does not update the generative model, but rather updates the finder network. Thus, the foundation generative model can be used.\n2.\tI think that the proposed method does not depend on the type of generative models such as normalizing flow, auto-regressive, gan, and score-based generative models.\n3.\tFDM to reduce the computation complexity.\n4.\tThe effectiveness of the reduced datasets\n\nWEAKNESSES:\n1. CutMix [A] and MixUp [B] rather than AugMix are the strong baselines. These methods are simple, easily implemented, no training, and effective. A comparison is needed.\n2. In Sec. 4.4, I am not sure about the interpretation of UMAP. From the visualization, the diversity of the proposed method is reduced. Thus, the loss term of contrastive learning, L_PCR, might be important. How much the performance is improved when the main model is trained on cross entropy and contrastive loss using only real data?\n3. The result of diffusion model is in the supplementary. However, the experiment of recent generation models, including text-to-image generator, makes the paper more convincing because it shows promising results [C, D, E, F].\n4. How does this method affect the evaluation in terms of robustness, generalizability, or bias? These is possibility to generate the biased samples.\n\n[A] CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features, ICCV 2019\n\n[B] mixup: Beyond Empirical Risk Minimization, ICLR 2018\n\n[C] IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION?, ICLR 2023\n\n[D] Fake it till you make it: Learning transferable representations from synthetic ImageNet clones, 2023\n\n[E] Synthetic Data from Diffusion Models Improves ImageNet Classification, 2023\n\n[F] Generative models improve fairness of medical classifiers under distribution shifts, 2023\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\n===\n\nI update my rate from 4 to 5 because my concerns are addressed and the authors will reflect the discussion below to the paper.\n\nLIMITATIONS:\nThe limitation is described in Sec. 6 in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for using synthetic images from GANs to augment training image classifiers. The naive approach to this problem is to generate samples for each class and treat these as supervised examples, but this can degrade performance due to image artifacts. Instead, the authors propose to use the generated data only for consistency regularization of the featurizer (”PCR”). Further, they meta-learn the codes to use for generation (”MPS”). On 6 datasets, the authors see performance improvements from using synthetic data this way, rather than performance drops.\n\nSTRENGTHS:\n- The motivation is clear, the method is clean, and the approach makes intuitive sense. The overall paper presentation is very good.\n- The authors select a nice set of baselines and show their method outperforms them. In particular, GDA + MH was an important comparison to have.\n- The paper contains a solid set of ablations, including: the importance of PCR v. MPS (either can improve performance, both are best), the effect of MPS on sample quality as measured by FID, experiments with multiple generative models for one dataset, and experiments showing that gains are additive with standard data augmentations.\n\nWEAKNESSES:\n- A missing baseline is consistency regularization without generative data augmentation, i.e. equation 5 applied to real examples $\\mathcal D$ rather than $\\mathcal D_p$. How does the method compare to training with this objective, and are gains additive?\n- Standard training with TrivialAugment / RandAugment outperform this method on Cars, presumably with much lower computational cost — though the authors do show these can be combined with the method to obtain further gains. I’d like to check if this trend holds for other datasets as well.\n- Because of the metalearning loop, the method is too slow to apply with diffusion models, as noted in the Limitations and Appendix B.3. All experiments are done with GANs trained from scratch on the target datasets.\n- Experiments are done on relatively small datasets. It would be interesting to compare this method on ImageNet.\n\nQUESTIONS:\nA point of confusion for me is why experiments avoided using pretrained generative models / finetuning from them. As noted in the related work, massive pretraining may help produce higher quality samples. The authors attribute degradations after training on generated data to *class leakage*, i.e. samples where artifacts of multiple classes are combined into the same image, such as the tennis ball with a dog face generated by BigGAN. This seems like the kind of artifact that improved generative models can remove. Does generative data augmentation with large pretrained models result in such performance drops? Have the authors tried their method with finetuned GANs?\n\nLIMITATIONS:\nThe authors note that the meta-learning loop renders the method unusable for computationally intensive generative models, e.g. diffusion models.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages synthetic images from generative models to train classifier models, effectively using them as an augmentation tool. However, instead of incorporating these images in a simplistic fashion, the synthetic images are utilized as a regularizer. The paper posits that synthetic samples may not always perfectly mirror class categories found in real data distribution. As a response, a meta-learning framework is employed to dynamically ascertain which synthetic samples should be used to minimize validation losses. The paper introduces a feature-based consistency regularization loss for the generated images, a method similar to self-supervised techniques. The unique contribution of this paper is the proposition of a finder network that is trained within a meta-learning context, this network is designed to select images that will enhance the performance of the classification model. Through extensive experimentation, the paper illustrates that the proposed methodology can effectively circumvent the performance deterioration often linked with naive generative data augmentation while concurrently improving the baselines.\n\nSTRENGTHS:\n* The paper is well-presented and easy to follow.\n* The paper presents a novel sampling network that is trained in a meta-learning setting to select synthetic images that will enhance the performance of the classification model.\n* Through a series of experiments, the paper successfully illustrates the effectiveness of the overall approach, as well as the contributions of different aspects of the methodology, i.e., consistency regularization loss and Meta Pseudo Sampling.\n* The proposed generative model augmentation method complements conventional data augmentation methods and can be employed together to enhance performance further.\n\nWEAKNESSES:\n* The paper showcases its experiments predominantly on simple, fine-grained classification datasets. To bolster the credibility of their methodology, the authors would benefit from demonstrating how their approach enhances classification performance on more complex, large-scale datasets, such as ImageNet.\n* In Table 1, I suggest including an additional baseline: the base model + SSL. This means applying a consistency regularization loss on the real training images without utilizing any synthetic images. This will highlight the degree of improvement achieved using the generative model and the proposed model. The author acknowledges that training the sampling network in a meta-learning setting is computationally expensive, so this comparison would clarify whether the benefits of this approach justify its cost, particularly if the gains are minimal.\n* For the consistency loss, the paper solely employs augmentation in the image space. However, one might ask why not also utilize augmentation in the generative model's latent space by introducing small noise to the latent vector z. Given that the latent space of the generators implicitly represents the image manifold, performing augmentation in this latent space could lead to meaningful augmentation in the image space. Such augmentation could potentially be more challenging to achieve using conventional image augmentation methods.\n* Figure 5 could be improved further by incorporating a visualization of different class features along the decision boundary of the classifier. This enhancement would provide further insight into how these synthetic images influence the decision boundary under different cases.\n* (Just a suggestion) In lines 45-50, I would like to introduce an additional argument. Despite generative models being trained to model p(x), they frequently fail to capture the entire distribution of the training set, focusing mainly on the high-density region of the distribution. Therefore, the information extracted from these generative models is often less comprehensive than the information found in the actual dataset distribution. Furthermore, these synthetic images frequently exhibit disfigured object shapes; for instance, a majority of human shapes in StyleGAN-XL images appear distorted.\n\nQUESTIONS:\n*The major novelty of the paper resides in the proposition of a sampling network, trained via a meta-learning method, to generate synthetic samples aimed at enhancing the classification model's performance. The authors further highlight that a superior FID score from the generative model correlates to improved performance in classification models (Table 2), and the FID score derived from their sampling network significantly surpasses that of uniform sampling (Fig 6b). In the generative models' literature, there exist works that advocate for effective sampling strategies from pre-trained GAN models, utilizing the multimodal truncation method, such as Makady et al. 2022's 'Self-Distilled StyleGAN: Towards Generation from Internet Photos.' How would their sampling approach fare when combined with the consistency regularization loss? It could significantly bolster the paper's case if authors demonstrated that the proposed sampling network exceeds the performance of other sampling methods, like the multimodal truncation method.\n\nLIMITATIONS:\nThe author themselves pointed out that the limitation of the method is the requirement of bilevel optimization of classifier and finder networks. This optimization is computationally expensive and may be less practical when used in generative models that require multiple inference steps, such as diffusion models.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}