SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We tested our method on EDM [45], a recent diffusion model. Due to the computation cost, we used a  $10\%$  reduced CIFAR-10 as the dataset. We optimized  $F_{\phi}$  to search the first step noise of the diffusion process. Table B-4 shows that our method with EDM improves Base Model. However, the overhead of incorporating diffusion models was significant; it takes more than ten times longer training than GANs. In future work, we will investigate lighter-weight methods using the diffusion model.
Table 6: Performance studies on Diffusion Model (ResNet-18 on Cars)
Method	Top-1 Acc. (%)
Base Model	86.49±.48
GDA (EDM)	85.80±.30
MGR	88.49±.12
B.4 Updating  $F_{\phi}$  without Meta-optimization
MPS consists of meta-learning on validation losses requiring bi-level optimization, which is a relatively heavy computation. One can consider if  $F_{\phi}$  could be trained without meta-optimization. Here, we try alternative methods other than meta-learning to update  $F_{\phi}$ . Instead of meta-learning, we used a strategy of choosing hard examples via optimizing  $F_{\phi}$ . That is, we optimize  $F_{\phi}$  by maximizing the training cross-entropy (CE) loss and the PCR loss on synthetic samples. Note that, in both cases, we used the PCR loss for synthetic samples when training classifiers. Table A-2 shows the results. Optimizing  $F_{\phi}$  with CE and PCR slightly improved the baselines but significantly underperformed our method (MGR). This result can justify using meta-optimizing  $F_{\phi}$  to generate useful samples for classifiers. Nevertheless, this idea could inspire a sampling method that does not require bi-level optimization in future work.
Table 7: Performance comparison of updating strategies for  $F_{\phi}$  (ResNet-18 on Cars)
Method	Top-1 Acc. (%)
Base Model	85.50±.10
PCR	86.36±.08
Optimizing Fφw/ CE	86.52±.21
Optimizing Fφw/ PCR	86.44±.68
MGR	87.22±.15
B.5 ImageNet Classification
We evaluated our MGR on ImageNet by randomly initializing ResNet-18 as the classifier and the pre-trained BigGAN as the generator. Table 8 shows that MGR successfully improves top-1 accuracy, while naive generative data augmentation (GDA) does not; this is the same trend as Table 1 (a) of the main paper. This result indicates that our method consistently works on complex and large-scale datasets. We will add the result in multiple trials with standard deviation to the paper.
Table 8: ImageNet Classification (ResNet-18)
Method	Top-1 Acc. (%)
Base Model	68.10
GDA	64.74
MGR	70.55
B.6 Latent Augmentation
Some readers may wonder why not also utilize augmentation in the generative model's latent space instead of data space. We implemented this approach by adding Gaussian noise to the latent vector as  $z' = z + s$ , where  $s \sim \mathcal{N}(0, 10^{-3})$ . Then, we compute the consistency regularization between  $g(G(z))$  and  $g(G(z'))$ , instead of  $g(G(z))$  and  $g(T(G(z)))$ . We call this variant LatentAugment. We found that LatentAugment improves the performance of our MGR (Table 9). Interestingly, LatentAugment can improve when it is used solely without image data augmentation  $T$ , i.e., RandAugment. This indicates that we can obtain meaningful variants by perturbing latent vectors, which is challenging for conventional data augmentation. However, this approach doubles the number of generators' forward computations and thus leads to increased computation time and memory footprint.
Table 9: Classification on Cars (ResNet-18)
Method	Top-1 Acc. (%)
Base Model	85.50±.10
MGR (LatentAugment)	86.49±.33
MGR (RandAugment)	87.22±.15
MGR (RandAugment + LatentAugment)	87.85±.53
B.7 Comparison to Existing Sampling Technique
Here, we evaluate MPS by comparing it with an existing sampling technique called multi-modal truncation sampling [47], which was originally proposed for sampling better quality samples for image generation. Table 10 shows the result of combining multi-modal truncation sampling and PCR. Although the FID score of multi-modal truncation sampling certainly outperforms MPS, the gain of the classification accuracy underperforms MPS. This implies that improving sample quality is a necessary condition for performance improvements, not a sufficient condition. Meanwhile, since MPS explicitly searches for samples that minimize the validation loss, it can improve classifiers more directly than incorporating existing sampling methods.
Table 10: Classification on Cars (ResNet-18)
Method	Running Mean FID	Top-1 Acc. (%)
PCR	22.96	86.36±.08
Multi-modal Truncation + PCR	21.12	86.51±.21
MGR (MPS + PCR)	22.08	87.22±.15
B.8 Performance Study when using Pre-trained Generators
We tried to use ImageNet pre-trained BigGAN and confirmed that this does not solve the degradation problem of GDA (Table 11). However, the use of pre-trained models can enhance our method.
B.9 Evaluation of Robustness toward Natural Corruption
Our method can improve the robustness against natural corruption and MPS does not generate biased samples toward training distribution. We tested the robustness of our method on CIFAR-10-C [48], which is a test set for CIFAR-10 corrupted by various transformations. Table 12 shows the results.
Table 11: Classification on Cars (ResNet-18, ImageNet pre-trained BigGAN). Underlined scores outperform that of Base Model, and Bolded scores are the best among the methods.
Method / Dataset size	10%	25%	50%	100%
Base Model	20.11±.03	49.33±.54	72.91±.38	85.80±.18
GDA	18.82±.22	46.38±.59	70.23±.66	86.11±.16
MGR	24.55±.24	53.41±.89	75.46±.12	87.17±.18
While GDA degraded the performance for all corruptions, PCR significantly improved the base model. Furthermore, MGR achieved even higher robustness. This indicates MPS in MGR can provide samples that are useful for generalization through meta-optimization.
Table 12: Classification on CIFAR-10-C (ResNet-18).
Method	clean	gaussian noise	shot noise	impulse noise	defocus blur	glass blur	motion blur	zoom blur	snow	frost	fog	brightness	contrast	elastic transform	pixelate	jpeg compression	mean
Base Model	86.49	53.32	53.12	39.01	34.06	37.15	31.08	36.14	46.68	36.38	19.14	52.45	10.51	41.14	46.59	53.02	42.27
GDA	84.11	52.70	52.98	39.94	29.39	38.66	28.97	28.82	45.04	40.08	24.00	50.29	13.07	40.91	45.11	52.85	41.68
PCR	87.06	59.35	60.65	37.71	47.51	50.89	41.71	47.11	63.94	57.33	31.13	73.43	13.66	58.00	63.77	68.55	53.86
MGR	88.02	61.69	62.53	41.62	52.21	51.91	47.94	51.09	65.78	59.50	34.49	73.23	14.83	60.65	65.32	71.10	56.37