SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We next analyze GIF-SD. As mentioned in Appendix D.2, we conduct channel-wise noise perturbation for latent optimization in GIF-SD. As analyzed in Appendix B.2, the channel-level latent feature encodes more subtle style information, and conducting channel-level noise perturbation diversifies the style of images while maintaining its content integrity. Therefore, our guided optimization particularly diversifies the style of the created images, without changing the content semantics of the latent features after diffusion (cf. Figure 19). Moreover, the prompt-guided diffusion with our explored prompts helps to enrich image styles further (e.g., cartoon or oil painting). Hence, combining both of them enables GIF-SD to create new samples with much higher diversity (cf. Figure 19).
Figure 19: Visualization of the generated images by SD with our explored guided optimization and designed prompts.
We then investigate the individual influence of our guidance criteria on GIF-SD on the basis of our prompt-guided diffusion. As shown in Table 21, both the class-maintained informativeness guidance  $S_{inf}$  and the diversity promotion guidance  $S_{div}$  contribute to model performance. One interesting thing is that, unlike GIF-DALLE that does not work without  $S_{inf}$ , GIF-SD can work well using only the diversity promotion guidance  $S_{div}$ . The key reason is that GIF-SD conducts channel-level noise perturbation over latent features and particularly diversifies the style of the created images without changing the content semantics of the latent features after diffusion. Therefore, the class semantics can be maintained well when only promoting sample diversity. Moreover, combining both guidance criteria enables GIF-SD to achieve the best expansion effectiveness (cf. Table 21), leading to promising performance gains (i.e.,  $13.5\%$  accuracy improvement on average over six natural image datasets) compared to unguided expansion with SD (cf. Table 19).
Table 21: Ablation of guidance and prompts in GIF-SD for expanding CIFAR100-Subset by  $5 \times$ .
Method	Designed prompts	Sinf	Sdiv	CIFAR100-Subset
GIF-SD	52.9±0.8
✓	56.2±1.0
✓	✓	59.6±1.1
✓	✓	59.4±1.2
✓	✓	✓	61.1±0.8
F.5.3 Discussions on the constraint of the perturbed feature in GIF
The hyper-parameter  $\varepsilon$  is used to ensure that the perturbed feature does not deviate from the input feature significantly, and its value depends on the prior model and target dataset. As described in Appendix D, for GIF-SD, we set  $\varepsilon = 0.8$  for most natural image datasets and further adjust  $\varepsilon = 4$  for Caltech101 to increase its dataset diversity for better performance. Once determined for a given prior model and target dataset,  $\varepsilon$  remains fixed for various expansion ratios. As shown in the following table, there is no need to increase when the expansion ratio becomes larger.
Table 22: Ablation of hyper-parameter  $\varepsilon$  on Caltech101 for GIF-SD.
ε on Caltech101	2	4	8
5×-expanded by GIF-SD	53.0	54.4	53.6
10×-expanded by GIF-SD	59.2	59.3	58.2
F.6 Discussion of training models with only expanded images
It is interesting to know how the model performs when trained with only the created images by our method. To this end, we train ResNet-50 from scratch using only images generated by GIF-DALLE on the CIFAR100-Subset and compare the result with a model trained on the real images of the CIFAR100-Subset.
We report the results regarding  $1 \times$  expansion in Table 23. We find that the model trained with only  $1 \times$  synthetic images performs worse than the model trained with the original dataset, indicating that the quality of synthetic data still lags behind that of real images. Please note that this does not degrade our contribution, since our work aims to expand small datasets rather than replace them entirely. Moreover, mixing the original images with the created images to the same size as the original dataset can lead to better performance than using only the original dataset. This suggests that the created images are not a simple repetition of the original dataset but offer new information that is useful for model training. Lastly, the model trained on the complete 1x-expanded dataset significantly outperforms the models trained either only on the original dataset or solely on the generated images, underscoring the potential of synthetic images in expanding small-scale datasets for model training.
Table 23: Performance of the model trained with only the expanded data of the  $5 \times$ -expanded CIFAR100-Subset dataset by GIF-DALLE.
CIFAR100-Subset	Data amount	Accuracy
Training with real images in original dataset	10,000	35.0±1.7
Training with only the 1×-created data by GIF-DALLE	10,000	21.0±0.7
Training with mixing original data and 1×-created data by GIF-DALLE	10,000	37.2±0.8
Training with 1×-expanded dataset by GIF-DALLE	20,000	45.6±1.1
We next report the results regarding  $5 \times$  expansion in Table 24. The model trained with  $5 \times$  synthetic images has already performed comparably to the model trained with real images. This result further verifies the effectiveness of our explored dataset expansion method. Moreover, the model trained with the full  $5 \times$ -expanded dataset performs much better than that trained with only the original dataset or with only the generated images. This further shows that using synthetic images for model training is a promising direction. We expect that our innovative work on dataset expansion can inspire more studies to explore this direction in the future.
Table 24: Performance of the model trained with only the expanded data of the  $5 \times$  -expanded CIFAR100-Subset dataset by GIF-DALLE.
CIFAR100-Subset	Data amount	Accuracy
Training with real images in original dataset	10,000	35.0±1.7
Training with only the 5×-created data by GIF-DALLE	50,000	35.2±1.3
Training with 5×-expanded dataset by GIF-DALLE	60,000	54.5±1.1
F.7 Effectiveness on long-tailed classification dataset
In previous experiments, we have demonstrated the effectiveness of our proposed method on relatively balanced small-scale datasets. However, real-world classification datasets are usually class imbalanced and even follow a long-tailed class distribution. Therefore, we further apply GIF-SD to expand a long-tailed dataset, i.e., CIFAR100-LT [5] (with the imbalance ratio of 100), to see whether it is also beneficial to long-tailed learning. Here, we train ResNet-50 from scratch with the cross-entropy loss or the balanced softmax loss [35] for 200 epochs, where Balanced Softmax [35, 102] is a class re-balancing loss designed for long-tailed learning.
As shown in Table 25, compared to training with cross-entropy directly on the original CIFAR100-LT dataset,  $20 \times$  expansion by our GIF-SD leads to a  $13.5\%$  model accuracy gain. This demonstrates the effectiveness of our proposed method in long-tailed learning. More encouragingly, our GIF expansion boosts the performance of few-shot classes more than many-shot classes, which means that GIF helps to address the issue of class imbalance.
Besides the cross-entropy loss, our dataset expansion is also beneficial to model training with long-tailed losses, such as Balanced Softmax. As shown in Table 25,  $20 \times$  expansion by GIF-SD boosts the accuracy of the Balanced Softmax trained model by  $14.8\%$ , and significantly improves its tail-class performance by  $26.8\%$ . These results further demonstrate the applicability of our GIF to long-tailed learning applications. We expect that this work can inspire more long-tailed learning studies to explore dataset expansion since information lacking is an important challenge in long-tailed learning [103].
Table 25: Effectiveness of GIF-SD for expanding CIFAR100-LT (imbalance ratio 100) by  $10\times$ , where all models are trained for 200 epochs. Here, Balanced Softmax [35, 102] is a class re-balancing losses designed for long-tailed learning.
CIFAR100-LT	Training losses	Many-shot classes	Medium-shot classes	Few-shot classes	Overall
Original	Cross-entropy	70.5	41.1	8.1	41.4
20×-expanded by GIF-SD	Cross-entropy	79.5 (+9.0)	54.9 (+13.8)	26.4 (+18.3)	54.9 (+13.5)
Original	Balanced Softmax	67.9	45.8	17.7	45.1
20×-expanded by GIF-SD	Balanced Softmax	73.7 (+5.8)	59.2 (+13.4)	44.5 (+26.8)	59.9 (+14.8)
F.8 Effectiveness on larger-scale dataset
In previous experiments, we have demonstrated the effectiveness of our proposed method on small-scale natural and medical image datasets. In addition to that, one may also wonder whether our method can be applied to larger-scale datasets. Although expanding larger-scale datasets is not the goal of this paper, we also explore our method to expand the full CIFAR100 by  $5 \times$  for model training. As shown in Table 26, compared to direct training on the original CIFAR100 dataset, our GIF-SD leads to a  $9.4\%$  accuracy gain and GIF-DALLE leads to an  $8.7\%$  accuracy gain. Such encouraging results verify the effectiveness of our methods on larger-scale datasets.
Table 26: Effectiveness of GIF for expanding the full CIFAR100.
Dataset	CIFAR100
Original	70.9±0.6
Expanded
5×-expanded by GIF-DALLE	79.6±0.3
5×-expanded by GIF-SD	80.3±0.3
F.9 Safety check
Ethical considerations, especially in AI research and data generation, are indeed paramount. Our approach is constructed with care to avoid negative implications, as evidenced in the following points:
- Controlled generation: In our approach, the generation of synthetic data is driven by our expansion guidances, which ensure that new data is derived directly and meaningfully from the original dataset. This controlled mechanism minimizes the risks of creating unrelated or potentially harmful images.
- No personal or sensitive data: It is also worth noting that our method primarily focuses on publicly available datasets like CIFAR, Stanford Cars, and similar, which do not contain personal or sensitive information. As such, the risks related to privacy breaches or misrepresentations are substantially diminished.
Following this, we further employ the Google Cloud Vision  $\mathsf{API}^4$  to perform a safety check on the 50,000 images generated during 5x-expansion of CIFAR100-S by GIF-SD. The Google Cloud Vision API is a tool from Google that uses deep learning to analyze and categorize content in images, commonly used for safety checks. It evaluates the likelihood of the image containing adult themes such as nudity or sexual activities, alterations made for humor or offensiveness (spoof), medical relevance, violent content, and racy elements which could include suggestive clothing or poses. This assessment aids in ensuring that images adhere to content standards and are appropriate for their target audiences.
As evidenced by Table 27, the synthetic images by our method are safe and harmless. To be specific, the majority of our generated images are categorized as either "Very unlikely" or "Unlikely" across all five metrics. Moreover, for categories like "Adult" and "Medical", the likelihood is almost negligible. Moreover, the visualized images in Appendix G also highlight the benign nature of the images produced by our method.
Table 27: Safety check of the generated images of CIFAR100-S by our GIF-SD, in terms of different metrics of Google Cloud Vision API.
Metrics	Very unlikely	Unlikely	Neutral	Likely	Very likely
Adult	96%	4%	0%	0%	0%
Spoof	82%	15%	3%	0%	0%
Medical	86%	14%	0%	0%	0%
Violence	69%	31%	0%	0%	0%
Racy	66%	25%	9%	0%	0%
<sup>4</sup>https://cloud.google.com/vision/docs/detecting-safe-search
G More Visualization Results
This appendix provides more visualized results for the created samples by our methods on various natural image datasets. Specifically, we report the synthetic images by GIF-SD on Caltech101 in Figure 20, those by GIF-DALLE in Figure 21 and those by GIF-MAE in Figure 22. The visualized results show that our GIF-SD and GIF-DALLE can create semantic-consistent yet content-diversified images well, while GIF-MAE can generate content-consistent yet highly style-diversified images. The visualization of GIF-SD and GIF-DALLE on other natural image datasets are shown in Figures 23-32.
G.1 Visualization of the expanded images on Caltech101
G.1.1 Visualization of the expanded images by GIF-SD on Caltech101
Input
Figure 20: Visualization of the created samples on Caltech101 by GIF-SD.
Our GIF-SD expansion
G.1.2 Visualization of the expanded images by GIF-DALLE on Caltech101
Figure 21: Visualization of the created samples on Caltech101 by GIF-DALLE.
Our GIF-DALLE expansion
G.1.3 Visualization of the expanded images by GIF-MAE on Caltech101
Figure 22: Visualization of the created samples on Caltech101 by GIF-MAE.
Our GIF-MAE expansion
G.2 Visualization of the expanded images on Cars
G.2.1 Visualization of the expanded images by GIF-SD on Cars
Figure 23: More visualization of the synthetic samples on Cars by GIF-SD.
G.2.2 Visualization of the expanded images by GIF-DALLE on Cars
Figure 24: More visualization of the synthetic samples on Cars by GIF-DALLE.
G.3 Visualization of the expanded images on Flowers
G.3.1 Visualization of the expanded images by GIF-SD on Flowers
Figure 25: More visualization of the synthetic samples on Flowers by GIF-SD.
G.3.2 Visualization of the expanded images by GIF-DALLE on Flowers
Figure 26: More visualization of the synthetic samples on Flowers by GIF-DALLE.
G.4 Visualization of the expanded images on Pets
G.4.1 Visualization of the expanded images by GIF-SD on Pets
Figure 27: More visualization of the synthetic samples on Pets by GIF-SD.
G.4.2 Visualization of the expanded images by GIF-DALLE on Pets
Figure 28: More visualization of the synthetic samples on Pets by GIF-DALLE.
G.5 Visualization of the expanded images on CIFAR100-Subset
G.5.1 Visualization of the expanded images by GIF-SD on CIFAR100-Subset
Input
Figure 29: More visualization of the synthetic samples on CIFAR100-Subset by GIF-SD.
G.5.2 Visualization of the expanded images by GIF-DALLE on CIFAR100-Subset
Figure 30: More visualization of the synthetic samples on CIFAR100-Subset by GIF-DALLE. Note that the resolution of the input CIFAR100 images is small (i.e.,  $32 \times 32$ ), so their visualization is a little unclear.
G.6 Visualization of the expanded images on DTD
G.6.1 Visualization of the expanded images by GIF-SD on DTD
Figure 31: More visualization of the synthetic samples on DTD by GIF-SD.
G.6.2 Visualization of the expanded images by GIF-DALLE on DTD
Figure 32: More visualization of the synthetic samples on DTD by GIF-DALLE.