SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Please refer to Appendix C for proofs. This theorem shows that the generalization error is bounded by the inverse of  $\delta$ -diversity. That is, the more diverse samples are created, the more improvement of generalization performance would be made in model training. In real small-data applications, data scarcity leads the covering radius  $\delta$  to be very large and thus the  $\delta$ -diversity is low, which severely affects model generalization. Simply increasing the data number (e.g., via data repeating) does not help generalization since it does not increase  $\delta$ -diversity. In contrast, our GIF applies two key criteria (i.e., "informativeness boosting" and "sample diversity promotion") to create informative and diversified new samples. The expanded dataset thus has higher data diversity than random augmentation, which helps to increase  $\delta$ -diversity and thus boosts model generalization. This advantage can be verified by Table 2 and Figure 4.
Implementing GIF with different prior models. To enable effective expansion, we explore three prior models for guided imagination: DALL-E2 [56] and Stable Diffusion [59] are advanced image generative methods, while MAE [24] is skilled at reconstructing images. We call the resulting methods GIF-DALLE, GIF-SD, and GIF-MAE, respectively. We introduce their high-level ideas below, while their method details are provided in Appendix D.
GIF-DALLE adheres strictly to the above pipeline for guided imagination, while we slightly modify the pipeline in GIF-SD and GIF-MAE since their image encoders are different from the CLIP image encoder. Given a seed sample, GIF-SD and GIF-MAE first generate the latent feature via the encoders of their prior models, and then conduct random noise perturbation following Eq. (1). Here, GIF-SD has one more step than GIF-MAE before noise perturbation, i.e., conducting prompt-guided diffusion for its latent feature, where the rule of the prompt design will be elaborated in Appendix D.2. Based on the perturbed feature, GIF-SD and GIF-MAE generate an intermediate image via their decoders, and apply CLIP to conduct zero-shot predictions for both the seed and intermediate images to compute the informativeness guidance (i.e., Eq. (2)) for optimizing latent features. Note that, in GIF-SD and GIF-MAE, we execute channel-level noise perturbation since we find it facilitates the generation of content-consistent samples with a greater variety of image styles (cf. Appendix B.2).
Table 1: Accuracy of ResNet-50 trained from scratch on small datasets and their expanded datasets by various methods. Here, CIFAR100-Subset is expanded by  $5 \times$ , Pets is expanded by  $30 \times$ , and all other natural image datasets are expanded by  $20 \times$ . All medical image datasets are expanded by  $5 \times$ . Moreover, MAE, DALL-E2 and SD (Stable Diffusion) are the baselines of directly using them to expand datasets without our GIF. In addition, CLIP indicates its zero-shot performance. All performance values are averaged over three runs. Please see Appendix F for more comparisons.
Dataset	Natural image datasets	Medical image datasets
Caltech101	Cars	Flowers	DTD	CIFAR100-S	Pets	Average	PathMNIST	BreastMNIST	OrganSMNIST	Average
Original	26.3	19.8	74.1	23.1	35.0	6.8	30.9	72.4	55.8	76.3	68.2
CLIP	82.1	55.8	65.9	41.7	41.6	85.4	62.1	10.7	51.8	7.7	23.4
Distillation of CLIP	33.2	18.9	75.1	25.6	37.8	11.1	33.6	77.3	60.2	77.4	71.6
Expanded
Cutout [15]	51.5	25.8	77.8	24.2	44.3	38.7	43.7 (+12.8)	78.8	66.7	78.3	74.6 (+6.4)
GridMask [8]	51.6	28.4	80.7	25.3	48.2	37.6	45.3 (+14.4)	78.4	66.8	78.9	74.7 (+6.5)
RandAugment [12]	57.8	43.2	83.8	28.7	46.7	48.0	51.4 (+20.5)	79.2	68.7	79.6	75.8 (+7.6)
MAE [24]	50.6	25.9	76.3	27.6	44.3	39.9	44.1 (+13.2)	81.7	63.4	78.6	74.6 (+6.4)
DALL-E2 [56]	61.3	48.3	84.1	34.5	52.1	61.7	57.0 (+26.1)	82.8	70.8	79.3	77.6 (+9.4)
SD [59]	51.1	51.7	78.8	33.2	52.9	57.9	54.3 (+23.4)	85.1	73.8	78.9	79.3 (+11.1)
GIF-MAE (ours)	58.4	44.5	84.4	34.2	52.7	52.4	54.4 (+23.5)	82.0	73.3	80.6	78.6 (+10.4)
GIF-DALLE (ours)	63.0	53.1	88.2	39.5	54.5	66.4	60.8 (+29.9)	84.4	76.6	80.5	80.5 (+12.3)
GIF-SD (ours)	65.1	75.7	88.3	43.4	61.1	73.4	67.8 (+36.9)	86.9	77.4	80.7	81.7 (+13.5)
Figure 4: Accuracy of ResNet-50 trained from scratch on the expanded datasets with different expansion ratios. The results on other datasets are reported in Appendix F.1.
Datasets. We evaluate GIF on six small-scale natural image datasets and three medical datasets. Natural datasets cover a variety of tasks: object classification (Caltech-101 [18], CIFAR100-Subset [41]), fine-grained classification (Cars [40], Flowers [49], Pets [50]) and texture classification (DTD [10]). Here, CIFAR100-Subset is an artificial dataset for simulating small-scale datasets by randomly sampling 100 instances per class from the original CIFAR100. Moreover, medical datasets [90] cover a wide range of image modalities, such as breast ultrasound (BreastMNIST), colon pathology (PathMNIST), and Abdominal CT (OrganSMNIST). Please refer to Appendix E for data statistics.
Compared methods. As there is no algorithm devoted to dataset expansion, we take representative data augmentation methods as baselines, including RandAugment, Cutout, and GridMask [8]. We also compare to directly using prior models (i.e., DALL-E2, SD, and MAE) for dataset expansion. Besides, CLIP has outstanding zero-shot abilities, and some recent studies explore distilling CLIP to facilitate model training. Hence, we also compare to zero-shot prediction and knowledge distillation (KD) of CLIP on the target datasets. The implementation details of GIF are provided in Appendix D.
5.1 Results of small-scale dataset expansion
Expansion effectiveness. As shown in Table 1, compared with the models trained on original datasets, GIF-SD boosts their accuracy by an average of  $36.9\%$  across six natural image datasets and  $13.5\%$  across three medical datasets. This verifies the superior capabilities of GIF over other methods for enhancing small datasets, particularly in the data-starved field of medical image understanding. This also suggests that dataset expansion is a promising direction for real small-data applications. Here, the reason why GIF-SD outperforms GIF-DALLE is that GIF-DALLE only exploits the image-to-image variation ability of DALL-E2, while GIF-SD further applies text prompts to diversify samples.
Expansion efficiency. Our GIF is more sample efficient than data augmentations, in terms of the accuracy gain brought by each created sample. As shown in Figure 4,  $5 \times$  expansion by GIF-SD outperforms even  $20 \times$  expansion by various data augmentations on Cars and DTD datasets, implying
Table 2: Corruption accuracy of ResNet-50 trained from scratch on CIFAR100-S and our  $5 \times$  expanded dataset, under 15 types of corruption in CIFAR100-C with the severity level 3. More results regarding other severity levels are provided in Appendix F.2.
Dataset	Noise	Blur	Weather	Digital
Gauss.	Shot	Impul.	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elastic	Pixel	JPEG	Average
Original	12.8	17.0	12.5	30.5	31.7	25.2	28.6	26.5	19.0	18.6	28.3	11.5	29.5	33.6	28.8	23.6
5×-expanded by RandAugment	16.7	21.9	27.5	42.2	42.5	35.8	40.2	36.9	31.9	30.0	43.1	20.4	41.2	44.7	37.6	34.2 (+10.6)
5×-expanded by GIF-SD	29.7	36.4	32.7	51.9	32.4	39.2	46.0	45.3	38.1	47.1	55.7	37.3	48.6	53.2	49.4	43.3 (+19.7)
20×-expanded by GIF-SD	31.8	39.2	34.7	58.4	33.4	43.1	51.9	51.7	47.4	55.0	63.3	46.5	54.9	58.0	53.6	48.2 (+24.6)
Table 3: Accuracy of various architectures trained on  $5 \times$  expanded Cars. The results on other datasets are given in Appendix F.3.
Dataset	ResNeXt-50	WideResNet-50	MobileNet-v2	Avg.
Original	18.4±0.5	32.0±0.8	26.2±4.2	25.5
Expanded
RandAugment	29.6±0.8	49.2±0.2	39.7±2.5	39.5 (+14.0)
GIF-DALLE	43.7±0.2	60.0±0.6	47.8±0.6	50.5 (+25.0)
GIF-SD	64.1±1.3	75.1±0.4	60.2±1.6	63.5 (+38.0)
Table 4: Comparison between our methods and directly fine-tuning CLIP models on three medical image datasets.
Dataset	PathMNIST	BreastMNIST	OrganSMNIST
Original dataset	72.4±0.7	55.8±1.3	76.3±0.4
Linear-probing of CLIP	74.3±0.1	60.0±2.9	64.9±0.2
fine-tuning of CLIP	78.4±0.9	67.2±2.4	78.9±0.1
distillation of CLIP	77.3±1.7	60.2±1.3	77.4±0.8
5×-expanded by GIF-SD	86.9±0.3	77.4±1.8	80.7±0.2
Table 5: Benefits of dataset expansion to CLIP fine-tuning on CIFAR100-S. Moreover, ID indicates in-distribution performance, while OOD indicates out-of-distribution performance on CIFAR100-C.
Methods	ID Accuracy	OOD Accuracy
Training from scratch on original dataset	35.0	23.6
Fine-tuning CLIP on original dataset	75.2 (+40.2)	55.4 (+31.8)
Fine-tuning CLIP on 5x-expanded dataset by GIF-SD	79.4 (+44.4)	61.4 (+37.8)
our method is at least  $4 \times$  more efficient than them. The limitations of these augmentations lie in their inability to generate new and highly diverse content. In contrast, GIF leverages strong prior models (e.g., SD), guided by our discovered criteria, to perform data imagination. Hence, our method can generate more diversified and informative samples, yielding more significant gains per expansion.
Benefits to model generalization. Theorem 4.1 has shown the theoretical benefit of GIF to model generalization. Here, Table 2 demonstrates that GIF significantly boosts model out-of-distribution (OOD) generalization on CIFAR100-C [27], bringing  $19.3\%$  accuracy gain on average over 15 types of OOD corruption. This further verifies the empirical benefit of GIF to model generalization.
Versatility to various architectures. We also apply the  $5 \times$  -expanded Cars dataset by GIF to train ResNeXt-50, WideResNet-50 and MobileNet V2 from scratch. Table 3 shows that the expanded dataset brings consistent accuracy gains for all architectures. This underscores the versatility of our method: once expanded, these datasets can readily be applied to train various model architectures.
Comparisons with CLIP. As our method applies CLIP for dataset expansion, one might question why not directly use CLIP for classifying the target dataset. In fact, our GIF offers two main advantages over CLIP in real-world small-data applications. First, GIF has superior applicability to the scenarios of different image domains. Although CLIP performs well on natural images, its transferability to non-natural domains, such as medical images, is limited (cf. Table 4). In contrast, our GIF is able to create samples of similar nature as the target data for dataset expansion, making it more applicable to real scenarios across diverse image domains. Second, GIF supplies expanded datasets suitable for training various model architectures. In certain scenarios like mobile terminals, hardware constraints may limit the supportable model size, which makes the public CLIP checkpoints (such as ResNet-50, ViT-B/32, or even larger models) unfeasible to use. Also, distilling from these CLIP models can only yield limited performance gains (cf. Table 1). In contrast, the expanded datasets by our method can be directly used to train various architectures (cf. Table 3), making our approach more practical for hardware-limited scenarios. Further comparisons and discussions are provided in Appendix F.4.
Benefits to model fine-tuning. In previous experiments, we have demonstrated the advantage of dataset expansion over model fine-tuning on medical image domains. Here, we further evaluate the benefits of dataset expansion to model fine-tuning. Hence, we use the 5x-expanded dataset by GIF-SD to fine-tune the pre-trained CLIP VIT-B/32. Table 5 shows that our dataset expansion significantly improves the fine-tuning performance of CLIP on CIFAR100-S, in terms of both in-distribution and out-of-distribution performance.
Figure 5: Visualization. (a) Examples of the created samples for Caltech101 by augmentation and GIF. Please see Appendix G for the visualization of more datasets. (b) Failure cases by GIF-SD.
Figure 6: Effects of different perturbation noise on MAE for expanding CIFAR100-Subset by  $5\times$
Figure 7: Visualization of the created medical images by GIF-SD, where SD is fine-tuned on medical data before dataset expansion (Please see the analysis of fine-tuning in Appendix B.6).
5.2 Analyses and discussions
We next empirically analyze GIF. Due to the page limit, we provide more analyses of GIF (e.g., mixup, CLIP, image retrieval, and long-tailed data) in Appendix B and Appendix F.
Effectiveness of zero-shot CLIP in GIF. We start with analyzing the role of zero-shot CLIP in GIF. We empirically find (cf. Appendix B.5) that GIF with fine-tuned CLIP performs only comparably to that with zero-shot CLIP on medical datasets. This reflects that zero-shot CLIP is enough to provide sound guidance without the need for fine-tuning. Additionally, a random-initialized ResNet50 performs far inferior to zero-shot CLIP in dataset expansion, further highlighting the significant role of zero-shot CLIP in GIF. Please refer to Appendix B.5 for more detailed analyses.
Effectiveness of guidance in GIF. Table 1 shows that our guided expansion obtains consistent performance gains compared to unguided expansion with SD, DALL-E2 or MAE, respectively. For instance, GIF-SD obtains  $13.5\%$  higher model accuracy on natural image datasets than unguided expansion with SD. This verifies the effectiveness of our criteria in optimizing the informativeness and diversity of the created samples. More ablation analyses of each criterion are given in Appendix F.5.
Pixel-wise vs. channel-wise noise. GIF-SD and GIF-MAE inject perturbation along the channel dimension instead of the spatial dimension. This is attributed to our empirical analysis in Appendix B.2. We empirically find that the generated image based on pixel-level noise variation is analogous to adding pixel-level noise to the original images. This may harm the integrity and smoothness of image content, leading the generated images to be noisy (cf. Figure 10(d) of Appendix B.2). Therefore, we decouple latent features into two dimensions (i.e., token and channel) and particularly conduct channel-level perturbation. As shown in Figure 10(e), optimizing channel-level noise variation can generate more informative data, leading to more effective expansion (cf. Figure 6).
Visualization. The samples we created are visualized in Figures 5a and 7. While GridMask obscures some image pixels and RandAugment randomly alters images with pre-set transformations, both fail to generate new image content (cf. Figure 5a). More critically, as shown in Figure 7, RandAugment may crop the lesion location of medical images, leading to less informative and even noisy samples. In contrast, our method can not only generate samples with novel content (e.g., varied postures and backgrounds of water lilies) but also maintains their class semantics, and thus is a more effective way to expand small-scale datasets than traditional augmentations, as evidenced by Table 1.
Table 6: Consumption costs. Time and costs are calculated from the expansion of 10,000 images. Accuracy improvements are compared to the original, small dataset.
Method	Expansion speed	Time	Costs	Accuracy gains
Human collection	121.0s per image	2 weeks	$800	-
Cutout	0.008s per image	76 seconds	$0.46	+12.8
GridMask	0.007s per image	72 seconds	$0.43	+14.4
RandAugment	0.008s per image	82 seconds	$0.49	+20.5
GIF-MAE (ours)	0.008s per image	80 seconds	$0.48	+23.5
GIF-SD (ours)	6.6s per image	2 hours	$40	+36.9
Table 7: Relation analysis between the domain gap (FID) and model accuracy.
Datasets	FID	Accuracy (%)
CIFAR100-S	-	35.0
RandAugment	24.3	46.7
Cutout	104.7	44.3
GridMask	104.8	48.2
GIF-MAE	72.3	52.7
GIF-DALLE	39.5	54.5
GIF-SD	81.7	61.1
Failure cases. Figure 5b visualizes some failure cases by GIF-SD. As we use pre-trained models without fine-tuning on the natural images, the quality of some created samples is limited due to domain shifts. For example, the face of the generated cat in Figure 5b seems like a lion face with a long beard. However, despite seeming less realistic, those samples are created following our guidance, so they can still maintain class consistency and bring new information, thus benefiting model training.
Computational efficiency and time costs. Compared to human data collection, our GIF offers substantial savings in time and costs for small dataset expansion. As shown in Table 6, GIF-MAE achieves a  $5 \times$  expansion per sample in just 0.04 seconds, while GIF-SD accomplishes the same in 33 seconds. To illustrate, according to rates from Masterpiece Group², manually annotating 10,000 images takes two weeks and costs around $800. In contrast, GIF-SD generates the same amount of labeled data in a mere two hours, costing roughly $40 for renting 8 V100 GPUs³. Moreover, with a slight model performance drop, GIF-MAE can create 10,000 labeled data in just 6 seconds, at a cost of about $0.48 for renting 8 V100 GPUs for 80 seconds. Specifically, GIF-MAE has a time cost within the same magnitude as data augmentation, but it delivers much better performance gains. The slight time overhead introduced by MAE is offset by GPU acceleration, resulting in competitive time costs. For those prioritizing performance, GIF-SD becomes a more attractive option. Although it involves a longer time due to its iterative diffusion process, it provides more significant performance gains. Note that our method only requires one-time expansion: the resultant dataset can be directly used to train various models (cf. Table 3), without the need for regeneration for each model.
Relation analysis between the domain gap and model performance. We further compute the Fréchet Inception Distance (FID) between the synthetic data generated by different methods and the original data of CIFAR100-S. Interestingly, while one might initially assume that a lower FID implies better quality for the expanded data, the actual performance does not consistently follow this notion. As shown in Table 7, even though GIF-SD has a worse FID than GIF-DALLE, it achieves better performance. Likewise, despite having nearly identical FIDs, Cutout and GridMask lead to different performance. These results suggest that the effectiveness of dataset expansion methods depends on how much additional information and class consistency the generated data can provide to the original dataset, rather than the distribution similarity between those samples and the original data. This discussion may spark further research into the relationship between expansion effectiveness and data fidelity (as measured by metrics like FID), potentially guiding the development of even more effective dataset expansion techniques in the future.
This paper has explored a novel task, dataset expansion, towards resolving the data scarcity issue in DNN training. Inspired by human learning with imagination, we presented a novel guided imagination framework for dataset expansion. Promising results on small-scale natural and medical image datasets have verified its effectiveness. Despite its encouraging results, there is still room to improve. That is, using only the generated samples for model training is still worse than using real samples of equivalent size, suggesting huge potential for algorithmic data generation to improve. We expect that our work can inspire further exploration of dataset expansion so that it can even outperform a human-collected dataset of the same size. Please refer to Appendix D.5 for a more detailed discussion on limitations and broader impact of our work.
2https://mpg-myanmar.com/annotation
<sup>3</sup>https://cloud.google.com/compute/gpus-pricing#gpu-pricing
This work was partially supported by the National Research Foundation Singapore under its AI Singapore Programme (Award Number: AISG2-TC-2021-002 and AISG2-PhD-2021-08-008).
[1] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in Brief, 28:104863, 2020.
[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Sahara, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023.
[3] S Bao, Y Tang, HH Lee, R Gao, S Chiron, I Lyu, LA Coburn, KT Wilson, JT Roland, BA Landman, et al. Random multi-channel image synthesis for multiplexed immunofluorescence imaging. Proceedings of Machine Learning Research, 156:36-46, 2021.
[4] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022.
[5] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems, 2019.
[6] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Generalizing dataset distillation via deep generative prior. In Computer Vision and Pattern Recognition, 2023.
[7] Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and Jose Alvarez. Data-free knowledge distillation for object detection. In Winter Conference on Applications of Computer Vision, pages 3289-3298, 2021.
[8] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Gridmask data augmentation. arXiv preprint arXiv:2001.04086, 2020.
[9] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In International Conference on Computer Vision, pages 4794-4802, 2019.
[10] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[11] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Computer Vision and Pattern Recognition, pages 113-123, 2019.
[12] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems, volume 33, 2020.
[13] Gang Dai, Yifan Zhang, Qingfeng Wang, Qing Du, Zhuliang Yu, Zhuoman Liu, and Shuangping Huang. Disentangling writer and character styles for handwriting generation. In Computer Vision and Pattern Recognition, pages 5977-5986, 2023.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, pages 248-255, 2009.
[15] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017.
[16] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pages 8780-8794, 2021.
[104] Yifan Zhang, Ying Wei, Qingyao Wu, Peilin Zhao, Shuaicheng Niu, Junzhou Huang, and Mingkui Tan. Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE Transactions on Image Processing, 29:7834-7844, 2020.
[105] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Computer Vision and Pattern Recognition, pages 10145–10155, 2021.
[106] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674–12685, 2021.
[107] Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. arXiv preprint arXiv:2204.07513, 2022.
[108] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
[109] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI Conference on Artificial Intelligence, 2020.
[110] Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. In International Conference on Computer Vision, pages 17205-17216, 2023.
[111] Yongchao Zhou, Hshmat Sahak, and Jimmy Ba. Training on thin air: Improve image classification with generated data. arXiv preprint arXiv:2305.15316, 2023.
[112] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European Conference on Computer Vision, pages 592-608, 2020.
A More Related Studies