Expanding Small-Scale Datasets with Guided Imagination
Authors: Yifan Zhang
Daquan Zhou
Bryan Hooi
Kai Wang
Jiashi Feng
Venue: NeurIPS 2023 poster (2023)
Keywords: Dataset Expansion
Guided Imagination
TL;DR: We explore a new task of dataset expansion for small-data applications. Relying on two discovered criteria, we develop a novel guided imagination framework for effective and efficient dataset expansion.

Abstract: The power of DNNs relies heavily on the quantity and quality of training data. However, collecting and annotating data on a large scale is often expensive and time-consuming. To address this issue, we explore a new task, termed dataset expansion, aimed at expanding a ready-to-use small dataset by automatically creating new labeled samples. To this end, we present a Guided Imagination Framework (GIF) that leverages cutting-edge generative models like DALL-E2 and Stable Diffusion (SD) to "imagine" and create informative new data from the input seed data. Specifically, GIF conducts data imagination by optimizing the latent features of the seed data in the semantically meaningful space of the prior model, resulting in the creation of photo-realistic images with new content. To guide the imagination towards creating informative samples for model training, we introduce two key criteria, i.e., class-maintained information boosting and sample diversity promotion. These criteria are verified to be essential for effective dataset expansion: GIF-SD obtains 13.5% higher model accuracy on natural image datasets than unguided expansion with SD. With these essential criteria, GIF successfully expands small datasets in various scenarios, boosting model accuracy by 36.9% on average over six natural image datasets and by 13.5% on average over three medical datasets. The source code is available at https://github.com/Vanint/DatasetExpansion.

Expanding Small-Scale Datasets with Guided Imagination
Yifan Zhang
Daquan Zhou Bryan Hooi
Kai Wang' *ByteDance
Jiashi Feng?
National University of Singapore
Abstract
The power of DNNs relies heavily on the quantity and quality of training data. However, collecting and annotating data large scale is often expensive and time- consuming. To address this issue, we explore a new task, termed dataset expansion, aimed expanding ready-to-use small dataset by automatically creating new labeled samples. To this end, we present a Guided Imagination Framework (GIF) that leverages cutting-edge generative models like DALL-E2 and Stable Diffusion (SD) to "imagine" and create informative new data from the input seed data. Specifically, GIF conducts data imagination by optimizing the latent features of the seed data in the semantically meaningful space of the prior model, resulting in the creation of photo-realistic images with new content. To guide the imagination towards creating informative samples for model training, we introduce two key criteria, i.e., mlais-mmintaieed information boosting and sample diversity promotion. These criteria are verified to be essential for effective dataset expansion: GIF-SD obtains 13.5% higher model accuracy on natural image datasets than unguided expansion with SD. With these essential criteria, GIF successfully expands small datasets in various scenarios, boosting model accuracy by 36.9% 9% on average over six natural image datasets and by 13.5% on average three medical datasets. The source code is available at https: : //ggt/gttb//////////////// com/
Introduction
substantial number of training samples is essential for unleashing the power of deep networks [14]. However, such requirements often impede small-scale data applications from fully leveraging deep learning solutions. Manual collection and labeling of large-scale datasets are often expensive and time-consuming in small-scall scenarios [52]. To address data scarcity while minimizing costs, we explore a novel task, termed Dataset Expansion. As depicted Figure (left), dataset expansion aims to create automatic data generation pipeline that can expand small dataset into a larger and more informative one for model training. This task particularly focuses on enhancing the quantity and quality of the small-scale dataset by creating informative new samples. This differs from conventional data augmentations that primarily focus on increasing data size through transformations, often without creating samples that offer fundamentally new content. The expanded dataset is expected to be versatile, fit for training various network architectures, and promote model generalization.
We empirically find that naive applications of existing methods cannot address the problem well (cf. Table 1 and Figure 4). Firstly, data augmentation [12, 15, 109], involving applying pre-set transformations to images, can be used for dataset expansion. However, these transformations primarily alter the surface visual characteristies of an image, but cannot create images with novel content (cf. Figure 5a). As result, the new information introduced is limited, and the performance gains tend to saturate quickly as more data is generated. Secondly, we have explored pre-trained generative models (e.g., Stable Diffusion (SD) [59]) to create images for model training. However,
"Corresponding to: yifan. waang..aa@uuu.. nus nus. zhoudaquan21 @gmail............ com com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Small datavel
Expanded dariser
68-
Dur ecperded calsett
15
dataset expansion
Mose pattorant
drg
+304
Aulo-created data with new Information
167
Coersegrrane object Fireegrinned object
Teclure
Pathology
Llbreccund
Figure 1: Dataset Expansion aims to create data with new information to enrich small datasets for training DNNs better (left). As shown, the created images by our method are all class-relevant but diverse (e.g,, new sitting postures of cats, new patterns of cushions). This enables ResNett-0 trained on our expanded datasets to perform much better than the one trained on the original datasets (right).
these pre-trained generative models are usually category-agnosse to the target dataset, so they cannot ensure that the synthetic samples carry the correct labels and are beneficial to model training.
Different from them, our solution is inspired by human learning with imagination. Upon observing an object, humans can readily imagine its different variants in various shapes, colors, or contexts, relying on their prior understanding of the world [75, 80J. Such an imagination process is highly valuable for dataset expansion as it does not merely tweak an object's appearance, but leverages rich prior knowledge to create object variants infused with new information. In tandem with this, recent generative models like SD and DALL-E2 [56] have demonstrated exceptional abilities in capturing the data distribution of extremely vast datasets [4, 66] and generating photo-realistic images with rich content. This motivates us to explore their use as prior models to establish our computational data imagination pipeline for dataset expansion, i.e., imagining different sample variants given seed data. However, the realization of this idea is non-trivial and is complicated by two key challenges: how to generate samples with correct labels, and how to ensure the created samples boost model training.
To handle these challenges, we conduct a series of exploratory studies (cf. Section 3) and make two key findings. First, CLIP [54], which offers excellent zero-shot classification abilities, can map latent features of category-agnostcc generative models to the specific label space of the target dataset. This enables us to generate samples with correct labels. Second, we discover two informativeness criteria that are crucial for generating effective training data: 1) class masistaaintt information boosting to ensure that the imagined data are class-consto consistent with the seed data and bring new information; 2) sample diversity promotion to encourage the imagined samples to have diversified content.
In light of the above findings, we propose a Guided Imagination Framework (GIF) for dataset expansion. Specifically, given a seed image, GIF first extracts its latent feature with the prior (generative) model. Unlike data augmentation that imposes variation over raw images, GIF optimizes the "variation" over latent features. Thanks to the guidance carefully designed by our discovered criteria, the latent feature is optimized to provide more information while maintaining its class. This enables GIF to create informative new samples, with class-consistent semanties yet higher content diversity, for dataset expansion. Such expansion is shown to benefit model generalization.
As DALL-E2 [56] and SD are powerful in generating images, and MAE [24] excels reconstructing images, we explore their use as prior models of GIF for data imagination. We evaluate our methods on small-scale natural and medical image datasets. As shown in Figure 1 (right), compared to ResNet- 50 trained on the original dataset, our method improves its performance by a large margin across various visual tasks. Specifically, with the designed guidance, GIF obtains 36.9% accuracy gains on average over six natural image datasets and 13.5% gains on average over three medical datasets. Moreover, the expansion efficiency of GIF is much higher than data augmentation, i.e., 5x expansion by GIF-SD outperforms even 20x expansion by Cutout, GridMask and Kand/Aumment on Cars and DTD datasets. In addition, the expanded datasets also benefit distribttti performance of models, and can be directly used to train various architectures (e.g., ResNeXt [85], WideResNet [97], and MobileNet [64]), leading to consistent performance gains. We also empirically show that GIF is more applicable than CLIP to handle real small-data scenarios, particularly with nan-natural image domains (e.g., medicine) and hardware constraints (e.g., limited supportable model sizes). Please note that GIF is much faster and more cost-effective than human data collection for dataset expansion.
2

---

2 Related Work
Learning with synthetic images. Training with synthetic data is a promising direction [26, 34, 110]. For example, DatasetGANs [42, 105] explore GAN models [17, 33] to generate images for segmentation model training. However, they require a sufficiently large dataset for in-domain GAN training, which is not feasible in small-data scenarios. Also, as the generated images without labels, they need manual annotations on generated images to train a label generator for annotating synthetic images. Similarly, many recent studies [2, 3, 22, 39, 63, 78, 82, 94] also explored generative models to generate new data for model training. However, these methods cannot ensure that the synthesized data bring sufficient new information and accurate labels for the target small datasets. Moreover, training GANs from scratch [3, 39, 63, 78, 94], especially with very limited data, often fails converge or produce meaningful results. In contrast, our dataset expansion aims expand a small dataset into larger labeled one in fully automatic manner without involving human annotators. As such, our method emerges as a more effective way to expand small datasets.
Data augmentation. Augmentation employs manually specified rules, such as image manipula- tion [91]. erasing [15, 109], mixup [28, 100, 101], and transformation selection [11, 12] to boost model generalization [69]. Despite certain benefits, these methods enrich images by pre-defined transformations, which only locally vary the pixel values of images and cannot generate images with highly diversified content. Moreover, the effectiveness of augmented data is not always guaranteed due to random transformations. In contrast, our approach harnesses generative models trained on large datasets and guides them to generate more informative and diversified images, thus resulting in more effective and efficient dataset expansion. For additional related studies, please refer to Appendix A.
3 Problem and Preliminary Studies
Problem statement. To address data scarcity, we explore a novel datasel expansion task. Without loss of generality, consider image classification, where small-scale training dataset Dj = { y4 )= is given. Here, Hú denotes an instance with class label yı, and n. denotes the number of samples. The goal of dataset expansion is to generate set of new synthetic samples D. fæ', v = to enlarge the original dataset, such that DNN model trained on the expanded dataset D. D, outperforms the model trained on D, significantly. The key is that the synthetic sample set D. should be highly related to the original dataset D, and bring sufficient new information to boost model training.
;==1
3.1
A proposal for computational imagination models
Given an object, humans can easily imagine its different variants, like the object in various colors, shapes, or contexts, based on their accumulated prior knowledge about the world [75, 80). This imagination process is highly useful for dataset expansion, as it does not simply perturb the object's appearance but applies rich prior knowledge to create variants with new information. In light of this, we seek build a computational model simulate this imagination process for dataset expansion.
Deep generative models, known for their capacity to capture the distribution of a dataset, become our tool of choice. By drawing their prior distribution knowledge, we can generate new samples resembling the characteristics of their training datasets. More importantly, recent generative models, such Stable Diffusion (SD) and DALL-E2, have demonstrated remarkable abilities in capturing the distribution of extremely large datasets and generating photo-realistic images with diverse content. This inspires us explore their use as prior models to construct our data imagination pipeline.
Specifically, given a pre-trained generative model G and a seed example (I, y) from the target dataset, we formulate the imagination of x' from G to transform the raw image into an embedding for imagination with the generative model. is perturbation applied to fx) such that G can generate x' different from A simple choice of d would be Gaussian random noise, which, however, cannot generate highly informative samples. In provide useful guidance.
x' = Gf + 6). Here, f(-) is an image encoder of
the following subsection, will discuss how to optimize
worth noting that we do not aim to construct a biologically plausible imagination model that strictly follows the dynamics and rules of the human brain. Instead, we draw inspiration from the imaginative activities of the human brain and propose a pipeline to leverage well pre-trained generative models to explore datasct expansion.
3

---

Random Cubout
Input
DALLE expansion
Guided DALLE expansion
Randon Ronshugrior expansion
Selective Randk.gme expansion
Mww petorerro
E
22
25.0
orgnnl ddladd Cubuu Cutsul espaniion Pandkggmat opension
(a)
(b)
Figure 2: Effectiveness of the informativeness criteria for sample creation. (a) Comparison
between random expansion and our selective expansion on CIFAR100-Subse (b) Visualization of DALLE expansion with and without diversity guidance.
3.2 How to guide imagination for effective expansion?
Our data imagination pipeline leverages generative models to create new samples from seed data. However, is unclear what types of samples are effective and how optimize accordingly in the pipeline to create data that are useful for model training. Our key insight is that the newly created sample x' should introduce new information compared the seed sample z, while preserving the same class semantics as the seed sample. To achieve these properties, we explore the following preliminary studies and discover two key criteria: (1) mlass-maintained information boosting, and (2) sample diversity promotion.
informativeness boosting. When enhancing data informativeness, it is non-trivial to ensure that the generated x' has the same label y as the seed sample x, since it is difficult to maintain the class semantics after perturbation in the latent space (ar) To overcome this,
explore CLIP [54] for its well-known image-text alignment ability: CLIP's image encoder can project an image x' an embedding space aligned with the language embedding of its class name y [72, 81]. Therefore, we can leverage CLIP's embedding vectors of all class names as a zero-shot classifier to guide the generation of samples that maintain the same class semantics as seed data. Mcanwhile, the entropy of the zero-shot prediction can serve measure to boost the classification informativeness of the generated data.
To pinpoint whether the criteria of class-maintained information boosting helps to generate more informative samples, we with exploratory experiments subset of CIFAR [41]. Here, the subset is built for simulating small-scale datasets by randomly sampling 100 instances per class from CIFAR100. We first synthesize samples based on existing data augmentation methods (i.e.. Ragdnnt and Cutout [15]) and expand CIFAR 100-Subset by 5x. Mcanwhile, we conduct selective augmentation expansion based on our criteria (i.e., selecting the samples with the same zero-shot prediction but higher prediction entropy compared to seed samples) until we reach the required expansion ratio per seed sample. As shown Figure 2a, selective expansion outperforms random expansion by 1.3% to 1.6%, meaning that the selected samples are more informative for model training. Compared to random augmentation, selective expansion filters out the synthetic data with lower prediction entropy and those with higher entropy but inconsistent classes. The remaining data thus preserve the same class semantics but bring more information gain, leading to better expansion effectiveness.
Sample diversity promotion. To prevent the "imagination collapse" issue that generative models yield overly similar or duplicate samples, we delve further into the criterion of sample diversity promotion. To study its effectiveness, we resort to a powerful generative model (i.e., DALL-E2) as the prior model to generate images and expand CIFAR 100- (00-Subset by 5x, where the guided expansion scheme and the implementation of diversity promotion will be introduced in the following section. As shown in Figure 2b, the generated images with diversity guidance are more diversified: starfish images have more diverse object numbers, and motorbike images have more diverse angles of view and even new driver. This leads to 1.4% additional accuracy gains on CIFAR 100-Subset (cf. Table 20 in Appendix F.5), demonstrating that the criterion of sample diversity promotion is effective bringing diversified information to boost model training.

---

imagined
feature
repested
optimzed
CLIP image encoder
DALL-E2
b
decoder
perturbed
informatvenees guidance
names "cat" "dog"
CLIP text encoder
coa 1dog CaI
'co''
zero- -hot classifior
Figure 3: Illustration of the proposed GIF method based on DALL-E2 [56], which expands small datasets by creating informative new samples with guided imagination. Here, we resort to DALL-E2 as the prior model, in which the image/text encoders CLIP image/text encoders while the decoder the diffusion model of DALL-E2. Moreover, + denotes guided residual multiplicative perturbation. More implementation details of GIF-DALLE, GIF-SD and GIF-MAE are provided in Appendix D.
4 GIF: A Guided Imagination Framework for Dataset Expansion
In light of the aforementioned studies, we propose Guided Imagination Framework (GIF) for dataset expansion. This framework guides the imagination of prior generative models based on the identified criteria. Given seed image æ from a target dataset, we first extract its latent feature f(x) via the encoder of the generative model. Different from data augmentation that directly imposes variations over raw RGB images, GIF optimizes the "variations" over latent features. Thanks the aforementioned criteria as guidance, the optimized latent features result in samples that preserve the correct class semanties while introducing new information beneficial for model training.
Overall pipeline. To detail the framework, we use DALL-E2 [56] as prior generative model for illustration. As shown in Figure 3, DALL-E2 adopts CLIP image/text encoders fCLP and fcuP as its image/text encoders and uses a pre-trained diffusion model G as its image decoder. To create set of new images x' from the seed image x, GIF first repeats its latent feature f fCLIP-(1)) (1) for K times, with K being the expansion ratio. For each latent feature f. we inject perturbation over it with randomly initialized noise: z ~ 4((0, 1) and bias b ~ (0, 1). Here, prevent out-of-control imagination, we conduct residual multiplicative perturbation on the latent feature f and enforce an e-ball constraint on the perturbation follows:
= ().c((1+22]] b),
where (-) means to project the perturbed feature f' to the e-ball of the original latent feature, i.e. II," -1loo S e. Note that each latent feature has independent z and b. Following our explored criteria, GIF optimizes and b over the latent feature space as follows:
2.6 3 +-
arg max Sinf + Saie. 2.b
where S and Saie correspond to the class-maintained informativeness score and the sample diversity score, respectively, which will be elaborated below. This latent feature optimization is the key step for achieving guided imagination. After updating the noise z' and bias 6' for each latent feature, GIF obtains a set of new latent features by Eq. (1), which are then used to create new samples through the decoder G.
Class-maintained informativeness. To boost the informativeness of the generated data without changing class labels, we resort to CLIP's zero-shot classification abilities. Specifically, first use fCUP. to encode the class name y of a sample x and take the embedding wy fCLIP.(()) as the zero-shot classifier of class y. Each latent feature f can be classified according to its cosine similarity to Wy, i.e., the affinity score of x belonging to class is 8, cos( (x), w.y), which forms a prediction probability vector
o a[ ......... .c]] for the total C' classes of the target dataset based on softmax (-)). The prediction of the perturbed feature can be obtained in the same way. We then design Sinf f to improve the information entropy of the perturbed feature while maintaining its class semantics as the seed sample:
Sinf = s' + (s log(s) - s' log(s')). )), s.t. j = arg max(s).
Specifically, sf, denotes the zero-shot classification score of the perturbed feature s' regarding the predicted label of the original latent feature j arg max(s). Here, the zero-shot prediction of the
5

---

original data serves as an anchor to regularize the class semantics of the perturbed features in CLIP's embedding space, thus encouraging class consistency between the generated samples and the seed sample. Moreover, log(s) s' log(s') means contrastive entropy increment, which encourages the perturbed feature to have higher prediction entropy and helps to improve the classification informativeness of the generated image.
Sample diversity. To promote the diversity of the generated samples, we design Sai as the Kullback-eeibler (KL) divergence among all perturbed latent features of a seed sample: Saie = DfL(f")))) where f denotes the current perturbed latent feature and f indicates the mean over the K perturbed features of the seed sample. To enable measuring KL divergence between features, inspired by [84]. we apply the softmax function transform feature vectors into probability vectors for KL divergence.
Theoretical analysis. We then analyze our method conceptually understand its benefits model generalization. We resort to ó-cover [62] to analyze how data diversity influences the generalization error bound. Specifically, "a dataset E' a 8-cover
dataset S" means set of balls with radius d centered at each sample of the dataset E' can cover the entire dataset S. According to the property of 8-cover, we define the dataset diversity by d-diversity the inverse of the minimal òmins i.e. Following the same assumptions of [67], we have the following result.
ddiv
Theorem 4.1. Let A denote a learning algorithm that outputs a set of parameters given a dataset D (2i, 3: (m) with i..dd samples drawn from distribution Pz. Assume the hypothesis function X"-Lipschitz continuous, the loss function E(x, y) is --isschiitz continuous for all and is bounded by L, with El yr: A) = 0 for all i E [n]. If constitutes a ô-cover of Pz, then with probability least 7/ the generalization error bound satisfies:
1 Σ Th iE(m)
(xi, ya; A)D C *+XLLC
B 10,,, A)]
ddiv
where C' a constant, and the symbol S indicates "smaller than" up to an additive constant.
Please refer Appendix C for proofs. This theorem shows that the generalization error is bounded by the inverse of 5-diversity. That is, the more diverse samples are created, the more improvement of generalization performance would be made in model training. In real small-data applications, data scarcity leads the covering radius ō to be very large and thus the 8-diversity is low, which severely affects model generalization. Simply increasing the data number (e.g., via data repeating) does not help generalization since it does not increase d-diversity. In contrast, our GIF applies two key criteria (i.e,, "informativeness boosting" and "sample diversity promotion") to create informative and diversified new samples. The expanded dataset thus has higher data diversity than random augmentation, which helps to increase d-diversity and thus boosts model generalization. This advantage can be verified by Table 2 and Figure 4.
Implementing GIF with different prior models. To enable effective expansion, we explore three prior models for guided imagination: DALL-E2 [56] and Stable Diffusion [59] are advanced image generative methods, while MAE [24] is skilled at reconstructing images. We call the resulting methods GIF-DALLE, GIF-SD, and GIF-MAE, respectively. We introduce their high-level ideas below, while their method details are provided in Appendix D.
GIF-DALLE adheres strictly to the above pipeline for guided imagination, while we slightly modify the pipeline in GIF-SD and GIF-MAE since their image encoders are different from the CLIP image encoder. Given a seed sample, GIF-SD and GIF-MAE first generate the latent feature via the encoders of their prior models, and then conduct random noise perturbation following Eq. (1). Here, GIF-SD has one more step than GIF-MAE before noise perturbation, i.e., conducting prompt- guiit- diffusion for its latent feature, where the rule of the prompt design will be elaborated in Appendix D.2. Based on the perturbed feature, GIF-SD and GIF-MAE generate an intermediate image via their decoders, and apply CLIP to conduct zero-shot predictions for both the seed and intermediate images to compute the informativeness guidance (i.e., Eq. (2)) for optimizing latent features. Note that, in GIF-SD and GIF-MAF, we execute channel-level noise perturbation since we find facilitates the generation of content-consistent samples with a greater variety of image styles (cf. Appendix B.2).

---

Table I: Accuracy of ResNet-50 trained from scratch on small datasets and their expanded datasets by various methods. Here, CIFAR 100-Subset is expanded by 5x, Pets is expanded by 30x, and all other natural image datasets are expanded by 20x. All medical image datasets are expanded by 5x. Moreover, MAE, DALL-E2 and SD (Stable Diffusion) are the baselines of directly using them to expand datasets without our GIF. In addition, CLIP indicates its zero-shot performance. All performance values are averaged over three runs, Please see Appendix F for more comparisons.
Dataset
Natural image datasets
Medical image datasets
Caltech101
Cans
Flower
DTD
CIFARI00:S
Pets
Averaee
PaRMNNIT
BreastNNSST
OrganSMNIST
Average
Original
74.1
23.1
6.8
72.4
55.8
76.3
CLIP
$2.1
55.8
65.9
41.7
41.6
85.4
10.7
51.8
73
2344
Distillaionn of CLIIP
13.2
75.1
25.6
37.8
11.1
33.5
77.3
00.2
77.4
71.6
Expanded
Catore [1S]
77.8
24.2
38.7
(+
78.8
66.7
78.3
(466(644)
GrilMask [S]
90.7
25.3
48.2
37.6
(-
78.4
06.8
78.9
(+6.5)
RandAaggent (12)
83.8
28.7
46.7
48.0
51.4 (+20.5)
79.3
68.7
79.6
(4766)
[24]
76.3
27.6
393
2413.2
81.7
63.4
78.6
74/6(6644
DALL-E2 (56]
61.3
84.1
34.5
$2.1
61.7
57,0
82.8
70.8
79.3
(7.6
ST [19]
51.1
51.7
78.8
33.2
$2.0
$7.9
54.36-23-49
85.1
73.8
78.0
5277
5234"
820
90.6
GI DALLE (ours)
63.0
53.1
88.2
39.5
54.5
65.4
60.8
88.4
76.6
80.5
50.5 (+12.5)
GIP SD (oun)
65.1
75.7
88.3
43.4
61.1
73.4
67.8 (+
86.9
77.4
86.7
$1.7 (+135)
Cuisut
GridMas
RandAugment
--- GIF-MAE (surs)
GIP-DALLE (oum)
GIF-5D Jouns)
80
Standard Cars
80
Pets
70-
70
20.5
71
60
50
Mode
40
150
42
505
531
665
103
00
52
52.4
30
36.
26
243
281
20.5
40
321
316
377
20
+10x
20
+20x
30
+10x
+20x
+30x
Figure Accuracy of ResNet-50 trained from scratch on the expanded datasets with different expansion ratios. The results on other datasets are reported in Appendix F.1.
5
Experiments
Datasets. We evaluate GIF on six small-scale natural image datasets and three medical datasets. Nat-
ural datasets cover variety of tasks: object classification (Caltech-10) 101 [18]. CIFARRO0-Subset [41]),
fine-grained classification (Cars [40]. Flowers [49], Pets [50]) and texture classification (DTD [10]).
Here, CIFARRO0-Subsel is an artificial dataset for simulating small-scale datasets by randomly sampling 100 instances per class from the original CIFARIO0, Moreover, medical datasets [90]
cover a wide range of image modalities, such as breast ultrasound (BreastMNISST), colon pathology (PathMNIST), and Abdominal CT (OrganSMNIST), Please refer to Appendix E for data statistics.
Compared methods. As there is algorithm devoted dataset expansion, we take representative data augmentation methods as baselines, including RandAugment, Cutout, and GridMask [8]. We also compare to directly using prior models (i.e., DALL-E2, SD, and MAE) for dataset expansion. Besides, CLIP has outstanding zero-shot abilities, and some recent studies explore distilling CLIP to facilitate model training. Hence, we also compare to zero-shot prediction and knowledge distillation (KD) of CLIP on the target datasets. The implementation details of GIF are provided in Appendix D.
5.1 Results of small-scale dataset expansion
Expansion effectiveness. As shown in Table I, compared with the models trained on original datasets, GIF-SD boosts their accuracy by average of 36.9% across six natural image datasets and 13.5% across three medical datasets. This verifies the superior capabilities of GIF over other methods for enhancing small datasets, particularly in the data-starved field of medical image understanding. This also suggests that dataset expansion is promising direction for real small-data applications. Here, the reason why GIF-SD outperforms GIF-DALLE is that GIF-DALLE only exploits the maae--t-imman variation ability of DALL-E2, while GIF-SD further applies text prompts to diversify samples.
Expansion efficiency. Our GIF is more sample efficient than data augmentations, in terms of the accuracy gain brought by each created sample. As shown in Figure 4, 5x expansion by GIF-SD outperforms even 20x expansion by various data augmentations on Cars and DTD datasets, implying
7

---

Table 2: Corruption aceuracy of ResNet-50 trained from scratch on CIFARIO0-S and our 5x expanded dataset, under 15 types of corruption in CIFAR 100-C with the severity level 3. More results regarding other severity levels are provided in Appendix F.2.
Nosse
Bhr
Westter
Digitl
Tatreet
Game
Shot
Impul
Dafre:
Class
Metion
Zoom
Snow
Froe
Fog
Bei
Cuntr
Flastcc
Pool
IPEG
Aszrgge
Original
12.8
170
12.5
M1.
31.7
25.2
75.6
26.5
19.0
18.6
28.3
11.5
28.8
33.6
26.8
25.6
5x cauudaa by Rundeng
16.3
21.
27.5
42.2
42.5
35.8
46.2
35.0
31.9
30.0
43.1
204
41.3
44.7
37.5
34.2
(+106)
Seoegnaneed CiI-SD
518
10.4
46.0
45.5
16.1
471
56.3
57-1
48.
452
48-(+19
expanded GIF SD
34.7
58A
33.4
51.9
51.7
47.4
55.0
46.5
58.0
(1.2()
Table 3: Accuracy of various architectures trained on 5x expanded Cars. The results on other datasets given in Appendix F.3.
Table 4: Comparison between our methods and directly fine-tuning CLIP models on three medical image datasets.
Damser
ResNeXt 50
WiteRenNeet
MobineNet
Avg.
Dataset
PIIMNIST
BreasMNNSST
OrgmSMNNIS
Original
184
26.2,42
25.5
Original
72.4.u7
$5.8
76. 36.3
lnnar-prohing CLIP
74.3
600
Expanded
fine-tuning of CLIP
78.4.00
67.2
78.9.0
Randeuggennt
GI DALLE
29.6
43.7au
60.0.am
49.2.am
39.7an
47.8
50.5 (+25.0)
29.5 (+16.0)
distillation of CLIP
77.3u17
60.2gt.3
77.4
GIP-SD
64.1a3
75.1.a4
69.210
63.5 (+38.0)
5x-expanded GIF-SD
86.
77.
80.7
Table 5: Benefits of dataset expansion CLIP fine-tuning on CIFARIO0-S. Moreover, ID indicates in-distribution performance, while OOD indicates out-of-distribution performance on CIFAR 100-C.
Methods
ID Accuracy
OOD Accuracy
Training from scratch on original dataset
35.0
23.6
Fine-tuning CLIP on original dataset Fine-tuning CLIP 5x-expanded dataset by GIF-SD
75.2 (+40.2)
(+31.88
79.4 (+44.4)
61.4 (437.8)
our method is at least 4x more efficient than them. The limitations of these augmentations lie in their inability to generate new and highly diverse content. In contrast, GIF leverages strong prior models (e.g., SD), guided by our discovered criteria, to perform data imagination. Hence, our method can generate more diversified and informative samples, yielding more significant gains per expansion.
Benefits to model generalization. Theorem 4.1 has shown the theoretical benefit of GIF to model generalization. Here, Table 2 demonstrates that GIF significantly boosts model out-of-distriuttio (OOD) generalization CIFAR 100-C [27], bringing 19.3% accuracy gain average over 15 types of OOD corruption. This further verifies the empirical benefit of GIF to model generalization.
Versatility to various architectures. We also apply the 5x ex-enndded Cars dataset by GIF to train ResNeXt-50, WideResNet-50 and MobileNet V2 from scratch. Table 3 shows that the expanded dataset brings consistent accuracy gains for all architectures. This underscores the versatility of our method: once expanded, these datasets can readily be applied to train various model architectures.
Comparisons with CLIP. As our method applies CLIP for dataset expansion, one might question why not directly use CLIP for classifying the target dataset. In fact, our GIF offers two main advantages over CLIP in real-world small-data applications. First, GIF has superior applicability to the scenarios of different image domains. Although CLIP performs well on natural images, its transferability to non-natural domains, such as medical images, is limited (cf. Table 4). In contrast, our GIF is able to create samples of similar nature as the target data for dataset expansion, making it more applicable to real scenarios across diverse image domains. Second, GIF supplies expanded datasets suitable for training various model architectures. In certain scenarios like mobile terminals, hardware constraints may limit the supportable model size, which makes the public CL.IP checkpoints (such as ResNet-50, ViT-B/32, or even larger models) unfeasible to use. Also, distilling from these CLIP models can only yield limited performance gains (cf. Table 1). In contrast, the expanded datasets by method can be directly used to train various architectures (cf. Table 3), making our approach more practical for hardware-limited scenarios. Further comparisons and discussions are provided in Appendix F.4.
Benefits to model fine-tuning, In previous experiments, we have demonstrated the advantage of dataset expansion over model fine-tuning on medical image domains. Here, we further evaluate the benefits of dataset expansion to model fine-tuning. Hence, we use the 5x-expanded dataset by GIF-SD to fine-tune the pre-trained CLIP VIT-B/32. Table 5 shows that our dataset expansion significantly improves the fine-tuning performance of CLIP on CIFARIOG 100-S, in terms of both in-distribution and out-of-distributio performance.
8

---

Input
GndWork
Ranskugmeen
Our GIF DNLLE
Our GIF SD
Input
Fallure cases
(a)
(b)
Figure 5: Visualization. (a) Examples of the created samples for Caltech101 by augmentation and GIF. Please see Appendix G for the visualization of more datasets. (b) Failure cases by GIF-SD.
Input
RandAugment
Our GIF-SD
55
62.7
50-
40.9
45-
Model
40-
35-
orignna
MAE
Road plood mile
fxed channell noive lwannd tikan noke
leumadd chennol
Figure 7: Visualization of the created medical images by GIF-SD, where SD fine-tuned on medical data before dataset expansion (Please the analysis of fine-tuning in Appendix B.6).
Figure 6: Effects of different perturbation noise on MAE for expanding CIFAR100-Subset by 5x.
5.2 Analyses and discussions
We next empirically analyze GIF. Due to the page limit, we provide more analyses of GIF (e.g., mixup. CLIP, image retrieval, and long-tailed data) in Appendix and Appendix F.
Effectiveness of zero-shot CLIP GIF. We start with analyzing the role zero-shot CLIP in GIF We empirically find (cf. Appendix B.5) that GIF with fine-tuned CLIP performs only comparably to that with zero-shot CLIP on medical datasets. This reflects that zero-shot CLIP is enough to provide sound guidance without the need for fine-tuning. Additionally, random- iinttiaiiiit ResNet50 performs far inferior to zero-shot CLIP dataset expansion, further highlighting the significant role of zero-shot CLIP in GIF, Please refer Appendix B.5 for more detailed analyses.
Effectiveness of guidance GIE. Table shows that our guided expansion obtains consistent performance gains compared to unguided expansion with SD, DALL-E2 or MAE, respectively. For instance, GIF-SD obtains 13.5% higher model accuracy on natural image datasets than unguided expansion with SD. This verifies the effectiveness of our criteria in optimizing the informativeness and diversity of the created samples. More ablation analyses of each criterion are given in Appendix F.5.
Pixel-wise vs. channel-wise noise. GIF-SD and GIF-MAE inject perturbation along the channel di- mension instead of the spatial dimension. This is attributed to our empirical analysis Appendix B.2. We empirically find that the generated image based on pixel-level noise variation is analogous to adding pixel-level noise to original images. This may harm the integrity and smoothness of image content, leading the generated images to be noisy (cf. Figure 10(d) of Appendix B.2). Therefore, we decouple latent features into two dimensions (i.e., token and channel) and particularly conduct channel-level perturbation. As shown in Figure 10(e), optimizing channel-level noise variation can generate more informative data, leading to more effective expansion (cf. Figure 6).
Visualization. The samples we created are visualized in Figures 5a and 7. While GridMask obscures some image pixels and Rand/Augment randomly alters images with pre-set transformations, both fail to generate new image content (cf. Figure 5a). More critically, as shown in Figure 7, RandAugment may crop the lesion location of medical images, leading to less informative and even noisy samples. In contrast, our method can not only generate samples with novel content (e.g., varied postures and backgrounds of water lilies) but also maintains their class semantics, and thus a more effective way to expand small-scale datasets than traditional augmentations, evidenced by Table 1.

---

Table 6: Consumption costs. Time and costs are calculated from the expansion of 10,000 images. Accuracy improve- ments are compared to the original, small dataset.
Table Relation analysis between the
domain gap (FID) and model accuracy.
Datasets
FID
Accuracy (%)
Method
Expansion speed
Time
Costs
Accuracy gains
CIFAR100-S
35.0
Human collection
121.0s per image
2 weeks
$800
RandAugment
24.3
46.7
Cutout
104.7
44.3
Cutout
GridMask
0,008s image
per imuge
72 seconds
76 seconds
$0.46
S0.43
+12,8
+14.4
GridMask
104.8
48.2
RandAugment
0,008s image
82 seconds
$0.49
+20.5
GIF-MAE
72.3
52.7
GUI-MAE (ours)
per imuge
80 seconds
S0.48
+23.5
GIF-DALLE
39.5
54.5
GIF SD (ours)
6.68 per image
2 hours
$40
+36.9
GIF-SD
81.7
61.1
Failure cases. Figure 5b visualizes some failure cases by GIF-SD. As use pre-trained models without fine-tuning on the natural images, the quality of some created samples is limited due to domain shifts. For example, the face of the generated cat in Figure 5b seems like lion face with long beard. However, despite seeming less realistic, those samples are created following our guidance, so they can still maintain class consistency and bring new information, thus benefiting model training.
Computational efficiency and time costs. Compared to human data collection, our GIF offers substantial savings in time and costs for small dataset expansion. As shown in Table 6, GIF-MAE achieves a 5x expansion per sample in just 0.04 seconds, while GIF-SD accomplishes the same in 33 seconds. To illustrate, according to rates from Masterpiece Group', manually annotating 10,000 images takes two weeks and costs around $800. In contrast, GIF-SD generates the same amount of labeled data in mere two hours, costing roughly $40 for renting V100 GPUs². Moreover, with a slight model performance drop, GIF-MAE can create 10,000 labeled data in just 6 seconds, at a cost of about $0.48 for renting 8 V 100 GPUs for 80 seconds. Specifically, GIF-MAE has time cost within the same magnitude as data augmentation, but it delivers much better performance gains. The slight time overhead introduced by MAE is offset by GPU acceleration, resulting in competitive time costs. For those prioritizing performance, GIF-SD becomes a more attractive option. Although it involves longer time due to its iterative diffusion process, provides more significant performance gains. Note that our method only requires one-time expansion: the resultant dataset can be directly used train various models (cf. Table 3), without the need for regeneration for each model.
Relation analysis between the domain gap and model performance. We further compute the Fréchet Inception Distance (FID) between the synthetic data generated by different methods and the original data of CIFAR 100-S. Interestingly, while one might initially assume that a lower FID implies better quality for the expanded data, the actual performance does not consistently follow this notion. As shown in Table 7, even though GIF-SD has a worse FID than GIF-DALLLE, achieves better performance. Likewise, despite having nearly identical FIDs, Cutout and GridMask lead to different performance. These results suggest that the effectiveness of dataset expansion methods depends on how much additional information and class consistency the generated data can provide to the original dataset, rather than the distribution similarity between those samples and the original data, This discussion may spark further research into the relationship between expansion effectiveness and data fidelity (as measured by metrics like FID), potentially guiding the development of even more effective dataset expansion techniques in the future.
Conclusion
This paper has explored novel task, dataset expansion, towards resolving the data scarcity issue in DNN training. Inspired by human learning with imagination, we presented a novel guided imagination framework for dataset expansion. Promising results small-scale natural and medical image datasets have verified its effectiveness. Despite its encouraging results, there is still room to improve. That is, using only the generated samples for model training is still worse than using real samples of equivalent size, suggesting huge potential for algorithmic data generation to improve. We expect that work can inspire further exploration of dataset expansion that it even outperform human-collected dataset of the same size. Please refer to Appendix D.5 for a more detailed discussion on limitations and broader impact of our work.
Pnttpa://mpg hhttpe://mpg-myarmarr wwtamaar..cmaammaaeae com/annotation
hhttps:/ccoodd google. cww/ cowpute/ppteeeceeeeecccceceeeeec gpus cettss-priiinngguu pricing
10

---

Acknowledgments
This work was partially supported by the National Research Foundation Singapore under its AI Singapore Programme (Award Number: AISG2-TC-2021 1000 and 10--G-2-DD2221-0-00000
References
[1] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in Brief, 28: 28:104863, 2020.
[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304. 2.X0:22304.08466, 2023.
[3] S Bao, Y Tang, HH Lee, R Gao, S Chiron, Lyu, LA Coburn, KT Wilson, JT Roland, BA Landman, al. Random multi-channel image synthesis for multiplexed immunofluores- cence imaging, Proceedings of Machine Learning Research, 156:36-46, 2021.
[4] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022.
[5] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbal- anced datasets with paneel-istribtton-aveeeee margin loss. In Advances in Neural Information Processing Systems, 2019.
[6] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and LunYan Zhu. Generalizing dataset distillation via deep generative prior. In Computer Vision and Pattern Recognition, 2023.
[7] Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and Jose Alvarez. Data-free knowledge distillation for object detection. In Winter Conference Applications of Computer Vision, pages 3289-3298, 2021.
[8] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Gridmask data augmentation. arXiv preprint arXiv:2001.0440 2020.
[9] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Interna- tional Conference on Computer Vision, pages 7794-4802 2019.
[10] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[11] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment: Learning augmentation strategies from data. In Computer Vision and Pattern Recognition, pages 113-123, 2019.
[12] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems, volume 33, 2020.
[13] Gang Dai, Yifan Zhang, Qingfeng Wang, Qing Du, Zhuliang Yu, Zhuoman Liu, and Shuang- ping Huang. Disentangling writer and character styles for handwriting generation. In Computer Vision and Pattern Recognition, pages 5977-5986, 2023.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, pages 248-255 2009.
[15] Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708 04552, 2017.
[16] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pages 5790-8794, 2021.

---

[17] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[18] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Computer Vision and Pattern Recognition Workshop, 2004.
[19] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, pages 1180-1189, 2015.
[20] Yunhao Ge, Harkirat Behl, Jiashu Xu, Suriya Gunasekar, Neel Joshi, Yale Song, Xin Wang. Laurent Itti, and Vibhav Vineet. Neural-sim: Learning to generate training data with nerf. In European Conference on Computer Vision, 2022.
[21] Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation:
A survey. International Journal of Computer Vision, 129(6): 1789)11789-119, 2021.
[22] Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, and Sangdoo Yun. Compodiff: Versatile composed image retrieval with latent diffusion. arXiv preprint arXiv:2303. 1916, 2023.
[23] Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learn- ing for pre-trained language model fine-tuning. In International Conference Learning Representations, 2021.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition, pages 770-778, 2016.
[26] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and XIAOJUAN QI. Is synthetic data from generative models ready for image recognition? In International Conference on Learning Representations, 2023.
[27] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations, 2019.
[28] Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations, 2019.
[29] Ralf Herbrich. Learning kernel classifiers: theory and algorithms. MIT press, 2001.
[30] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503. 150.00253, 2015.
[31] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851, 2020.
[32] Xun Huang and Serge Belongie. Arbitrary style transfer real-time with adaptive instance normalization. In International Conference on Computer Vision, pages 1501-1510, 2017.
[33] Phillip Isola, Jun-Y Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Computer Vision and Pattern Recognition, pages 1125-11343 1134, 2017.
[34] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as data source for multiview representation learning. In International Conference on Learning Repre- sentations, 2022.
[35] Ren Jiawei, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems, 2020.
12

---

[36] Jakob Nikolas Kather, Johannes Krisam, Pompimol Charoentong, Tom Luedde. Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios Valous, Dyke Ferber, al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Medicine, 16(1)=e1002730 2019.
[37] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Computer Vision and Pattern Recognition, pages 2426-2435, 2022.
[38] Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Founda- tions and Trends in Machine Learning, (2(4):307-392,000 2019.
[39] Quan Kong, Bin Tong, Martin Klinkigt, Yuki Watanabe, Naoto Akira, and Tomokazu Mu- rakami. Active generative adversarial network for image classification. In AAAI Conference on Artificial Intelligence, pages 4090-4097, 2019.
[40] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. In Workshop on Fine-Grained Visual Categorization, 2013.
[41] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
[42] Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba. Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations. In Computer Vision and Pattern Recognition, pages 21330-21344 2022.
[43] Pu Li, Xiangyang Li, and Xiang Long. Fencemask: A data augmentation approach for pre-extracted image features. arXiv preprint arXiv:2006.07877, 2020.
[44] Xingjian Li, Haoyi Xiong, et al. Delta: Deep learning transfer using feature map with attention for convolutional networks. In International Conference on Learning Representations, 2019.
[45] Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Advances in Neural Information Processing Systems, volume 32, 2019.
[46] Hongbin Lin, Yifan Zhang, Zhen Qiu, Shuaicheng Niu, Chuang Gan, Yanxia Liu, and Mingkui Tan. Prototype-guided continual adaptation for class-incremental unsupervised domain adapta- tion. In European Conference on Computer Vision, pages 351-368, 2022.
[47] B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
[48] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc- Grew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning. 2022.
[49] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing. 2008.
[50] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Computer Vision and Pattern Recognition, 2012.
[51] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In International Conference Computer Vision, pages
2021.
[52] Guo-Jun Qi and Jiebo Luo. Small data challenges in big data era: A survey of recent progress on unsupervised and remi-supervised methods. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2020.
13

---

[53] Zhen Qiu, Yifan Zhang, Hongbin Lin, Shuaicheng Niu. Yanxia Liu, Qing Du, and Mingkui Tan. Source-free domain adaptation via avatar prototype generation and adaptation. In International Joint Conference on Artificial Intelligence, pages 2921-2927, 2021.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 7743.8763. PMIR, 2021.
[55] Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in Neural Information Processing Systems, 32, 2019.
[56] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
[57] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Muchine Learning, pages 8821-8831 2021.
[58] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-2I1 pretraining for the masses, In Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[60] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. /2242, 2022.
[61] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-ine text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
[62] Claude Sammut and Geoffrey I Webb. Encyclopedia of machine learning and data mining. Springer Publishing Company, Incorporated, 2017.
[63] Veit Sandfort, Ke Yan, Perry Pickhardt, and Ronald M Summers. Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in segmentation tasks. 2019.
Scientific Reports,
[64] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiol Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Computer Vision and Pattern Recognition, pages 4520- 2018.
[65] Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In Computer Vision and Pattern Recognition, 2023.
[66] Christoph Schuhmann, Robert Kaczmarezyk; Aran Komatsuzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Jenia Jitsev, Theo Coombes, and Clayton Mullis. Laion-400m: Open dataset of clip-filtred 400 million image-text pairs. NeurlPS Workshop Datacentric AI, 2021.
[67] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. In International Conference on Learning Representations, 2017.
[68] Mingjia Shi, Yuhao Zhou, Kai Wang, Huaizheng Zhang, Shudong Huang, Qing Ye, and Jiancheng Lx. Prior: Personalized prior for reactivating the information overlooked in federated learning. In Neural Information Processing Systems, 2023.
14

---

[69] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[70] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
[71] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson.
Does knowledge distillation really work? In Advances in Neural Information Processing Systems, volume 34, pages 6906-6919, 2021.
[72] Changyao Tian, Wenhai Wang, Xizhou Zhu, Xiaogang Wang, Jifeng Dai, and Yu Qiao. VI-Itr: Learning class-wise visual- -ongu-linguiitic representation for toigetailed visual recognition. In European Conference on Computer Vision, 2022.
[73] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. arXiv preprint arXiv:2306. 2023.
[74] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Computer Vision and Pattern Recognition, pages 7167-7176, 2017.
[75] Andrey Vyshedskiy. Neuroscience of imagination and implications for human evolution. Current Neurobiology, 2019.
[76] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text- and-image driven manipulation of neural radiance fields. In Computer Vision and Pattern Recognition, pages 3835-3844, 2022.
[77] Pei Wang, Yijun Li, Krishna Kumar Singh, Jingwan Lu, and Nuno Vasconcelos. Imagine: Im- age synthesis by image-guide guided model inversion. In Computer Vision and Pattern Recognition, pages 3681-36999 2021.
[78] Sihan Wang, Fuping Wu, Lei Li, Zheyao Gao, Byung-Wov Woo Hong, and Xiahai Zhuang. Un- supervised cardiac segmentation utilizing synthesized images from anatomical labels. In International Workshop on Statistical Atlases and Computational Models of the Heart, pages 349-358, 2022.
[79] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 100559, 2018.
[80] Mary Warnock and Cean-Paul Sartre. The psychology of the imagination. Routledge, 2013.
[81] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong. et al. Robust fine-tuning of zero-shot models. In Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
[82] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In International Conference on Computer Vision, 2023.
[83] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[84] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, pages 487, 2016.
[85] Saining Xie, Ross Girshick. Piotr Dollár, Zhuowen Tu, and Kaiming He Aggregated residual transformations for deep neural networks. In Computer Vision and Pattern Recognition, 2017.
[86] Austin Xu, Mariya I Vasileva, and Arjun Seshadri. Handsoff: Labeled dataset generation with no additional human annotations. In NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research, 2022.
15

---

[87] Haohang Xu, Shuangrui Ding, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Masked autoencoders are robust data augmentors. arXiv preprint arXiv:2206.044: 04846, 2022.
[88] Xuanang Xu, Fugen Zhou, Bo Liu, Dongshan Fu, and Xiangzhi Bai. Efficient multiple organ localization et image using 3d region proposal network. IEEE Transactions on Medical Imaging, 38(8): 1851188888 1898, 2019.
[89] Beining Yang. Kai Wang. Qingyun Sun, Cheng Ji, Xingcheng Fu, Hao Tang, Yang You, and Jianxin Li. Does graph distillation see like vision dataset counterpart? In Neural Information Processing Systems, 2023.
[90] Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In IEEE International Symposium on Biomedi- cal Imaging, pages 191-195 2021.
[91] Suorong Yang, Weikang Xiao, Mengcheng Zhang, Suhan Guo, Jian Zhao, and Furao Shen. Image data augmentation for deep learning: A survey, arXiv preprint Fr..ix2204.0861000 2022.
[92] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Computer Vision and Pattern Recognition, pages 8715-8724, 2020.
[93] Alex Yu, Vickie Yc, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[94] Biting Yu, Luping Zhou, Lei Wang, Yinghuan Shi, Jurgen Fripp, and Pierrick Bourgeat. Ea- gans: edge-aware generative adversarial networks for cross- moss-modallty mr image synthesis. IEEE Transactions on Medical Imaging. 387))1150-1162, 2019.
[95] Longhui Yu, Yifan Zhang, Lanqing Hong, Fei Chen, and Zhenguo Li. Dual-curricuum teacher for Glomain-inconsiseen object detection in autonomous driving. In British Machine Vision Conference, 2022.
[96] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision, pages 6023-6032, 2019.
[97] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
[98] Manzil Zaheer, Ankit Singh Rawat, Seungyeon Kim, Chong You, Himanshu Jain, Andreas Veit, Rob Fergus, and Sanjiv Kumar. Teacher guided training: An efficient framework for knowledge transfer. arXiv preprint arXiv:2208.066: 06825, 2022.
[99] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[100] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani. and James Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representations, 2021.
[101] Yifan Zhang, Bryan Hooi, Dapeng Hu, Jian Liang, and Jiashi Feng. Unleashing the power of contrastive self-supervised visual models via contrast-regularizes fine-tuning. In Advances in Neural Information Processing Systems, volume 34, pages 29848-29860, 2021.
[102] Yifan Zhang, Bryan Hooi, HONG Lanqing, and Jiashi Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. In Advances in Neural Information Processing Systems, 2022.
[103] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
16

---

[104] Yifan Zhang, Ying Wei, Qingyao Wu, Peilin Zhao, Shuaicheng Niu, Junzhou Huang, and Mingkui Tan. Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE Transactions on Image Processing, 29:7834-7844 2020.
[105] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francoff Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Computer Vision and Pattern Recognition, pages 10145-10155 2021.
[106] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674-12685, 2021.
[107] Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. arXiv preprint arXiv:2204.0755 2022.
[108] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
[109] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI Conference on Artificial Intelligence, 2020.
[110] Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. In International Conference on Computer Vision, pages 17205-17216, 2023.
[111] Yongchao Zhou, Hshmat Sahak, and Jimmy Ba. Training on thin air: Improve image classifi- cation with generated data. arXiv preprint arXiv:2305. 15316, 2023.
[112] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European Conference on Computer Vision, pages 592-608 2020.
17

---

A More Related Studies
Image synthesis. Over the past decade, image synthesis [2, 13, 65, 73, 111] has been extensively
explored, with four main approaches leading the way: generative adversarial networks (GANs) [17, 33], auto-regressive models [38, 57], diffusion models [16, 31 and neural radiance fields [20, 47, 93]. Recently, diffusion techniques, such as DALL-E2 [56], Imagen [61], and Stable Diffusion [59], have demonstrated exceptional capabilities in producing photo-realistic images. In practice, these techniques can serve as prior models in our GIF framework for dataset expansion.
Additionally; CLIP [54], thanks its text-image matching ability, has been used to guide image generation [37, 48, 51, 76]. In these approaches, CLIP matches a generated image with given text. In contrast, our work uses CLIP align the latent features of category-agnostic generative models with the label space of the target dataset. This alignment enables GIF to perform guided data expansion, generating informative new samples specific to target classes.
Furthermore, model inversion [83, 86] is another technique that has been investigated for image generation by inverting trained classification network [77, 92] or GAN model [112]. Although we currently apply only two advanced generative models (DALL-E2 and Stable Diffusion) and a reconstruction model (MAE) within the GIF framework in this study, model inversion methods could also be incorporated into our framework for dataset expansion. This opens up exciting avenues for future research.
More discussion on data augmentation. Image data augmentation has become a staple in enhanc- ing the generalization of DNNs during model training [69, 91]. Based on technical characteristics, image data augmentation can be categorized into four main types: image manipulation, image erasing, image mix, and auto augmentation.
Image manipulation augments data through image transformations like random flipping, rotation, scaling, cropping, sharpening, and translation [91]. Image erasing, on the other hand, substitutes pixel values in certain image regions with constant or random values, as seen in Cutout [15], Random Erasing [109], GridMask [8], and Fenchmask [43]. Image mix combines two more images sub-regions into single image, as exemplified by Mixup [100], CutMix [96], and AugMix [28].
Lastly, Auto Augmentation utilizes a search algorithm random selection to determine augmentation operations from a set of random augmentations, such as AutoAugment [11], Fast AutoAugment [45], and RandAugment [12].
While these methods have shown effectiveness in certain applications, they primarily augment data by applying pre-defined transformations each image. This results in only local variations in pixel values and does not generate images with significantly diversified content. Furthermore, as most methods employ random operations, they cannot ensure that the augmented samples are informative for model training and may even introduce noisy augmented samples. Consequently, the new information brought about is often insufficient for expanding small datasets, leading to low expansion efficiency (cf. Figure 4). In contrast, our proposed GIF framework utilizes powerful generative models (such DALL-E2 and Stable Diffusion) trained on large-scale image datasets, guiding them to optimize latent features in accordance with our established criteria (i.e., class-maintained information boosting and sample diversity promotion). This results in the creation of images that are both more informative and diversified than those from simple image augmentation, thereby leading to more efficient and effective dataset expansion.
We note that the work [87] also explores MAE for image augmentation based on its reconstruction capability. It first masks some sub-regions of images and then feeds the masked images into MAE for reconstruction. The recovered images with slightly different sub-regions are then used as augmented samples. Like other random augmentation methods, this approach only varies pixel values locally and cannot ensure that the reconstructed images are informative and useful. In contrast, our GIF-MAE guides MAE to create informative new samples with diverse styles through our guided latent feature optimization strategy. Therefore, GIF-MAE is capable of generating more useful synthetic samples, effectively expanding the dataset.
18

---

Contrasting with dataset distillation. Dataset distillation, also known as dataset condensation, is task that seeks to condense a large dataset into smaller set of synthetic samples that are comparably effective [6, 68, 79, 89, 106, 107, 108, 110]. The goal of this task is to train models to achieve performance comparable to the original dataset while using significantly fewer resources. Such task is diametrically opposed to our work on dataset expansion, which strives to expand a smaller dataset into a larger, richer, and more informative one. We achieve this by intelligently generating new samples that are both informative and diverse. Hence, dataset distillation focuses on large-data applications, whereas our focus lies on expanding dataset diversity and information richness for more effective deep model training in small-data applications.
Contrasting with transfer learning. Numerous studies have focused on model transfer Iearning techniques using publicly available large datasets like ImageNet [14, 58J. These approaches include model fine-tuning [23, 44, 101], knowledge distillation [21, 30J, and domain adaptation [19, 46, 53, 74, 95, 104].
Despite effectiveness in certain applications, these model transfer learning paradigms also suffer from key limitations. For instance, the study [55] found that pre-training and fine-tuning schemes do not significantly enhance model performance when the pre-trained datasets differ substantially from the target datasets, such when transferring from natural images to medical images. Moreover, model domain adaptation often necessitates that the source dataset and the target dataset share same highly similar label spaces, requirement that is often unmet in small-data application scenarios due to the inaccessibility of a large-scale and labeled source domain with a matching label space. In addition, the work [7 I] found that knowledge distillation does not necessarily work if the issue of model mismatch exists [9]. i.e., large discrepancy between the predictive distributions of the teacher model and the student model. The above limitations of model transfer learning underscore the importance of the dataset expansion paradigm: if small dataset is successfully expanded, it can be directly used to train various model architectures.
We note that some data-free knowledge distillation studies [7, 92, 98] also synthesize images, but their goal is particularly to enable knowledge distillation in the setting without data. In contrast, our task is independent of model knowledge distillation. The expanded datasets are not method seehod-dependent or model-dependeer and, thus, can train various model architectures to perform better than the original small ones.
19

---

B More Preliminary Studies
B.1 Sample-wise expansion or sample-agnostic expansion?
When we design the selective expansion strategy in Section 3.2, another question appears: should we ensure that each sample is expanded by the same ratio? To determine this, we empirically compare RandAngment expansion with sample-wise selection and sample-agnostic selection according to one expansion criteria, Lee., class-maintained information boosting. Figure 8 shows that sample-wist -isee expansion performs much better than sample-agnoste expansion. To find out the reason for this phenomenon, visualize how many times sample is expanded by sample-agnostic expansion. As shown in Figure 9, the expansion numbers of different samples by sample-agnostic expansion present a long-tailed distribution [103]. with many image samples not expanded at all. The main reason for this is that, due to the randomness of RandAugment and the differences among images, not all created samples are informative and it is easier for some samples to be augmented more frequently than others. Therefore, given fixed expansion ratio, the sample-agnostic expansion strategy, as it ignores the differences in images, tends to select more expanded samples for more easily augmented images. This property leads sample-agnostic expansion to waste valuable original samples for expansion (iee. loss of information) and also incurs in Figure 8. In contrast, sample-wise wise expansion can fully take advantage of all the samples in the target dataset and thus more effective than sample-agnostic expansion, which should be considered when designing dataset expansion approaches.
imsalance problem, thus resulting in worse performance
48.0
Expemsion numioe per sampie (loraa))
47.5
15.0
Model parformencs
47.0
12.5
46.7
665
10.0
7.5
These samples are not axpanded
46.0
45.8
5.0
455
2.5
45.0
0.0
Random expersiin Data-agnosto guidance Data-wise guidance
6000
10000
Sample indox
Figure 8: Comparison of model performance between samples-wise selection and sample- agnostic selection for RandAugment expansion on
Figure 9: Statistics of the expansion numbers of different data in CIFAR 100-Subset by sample- agnostic selective expansion with RandAugment, which presents long-tailed distribution.
B.2 Pixel-level noise or channel-level noise?
In our preliminary studies exploring the MAE expansion strategy, we initially used pixel-level noise to modify latent features. However, this approach did not perform well. To understand why, we analyze the reconstructed images. An example of this is presented in Figure 10(d). We find that the generated image based on pixel-level noise variation is analogous to adding pixel-level noise to the original images. This may harm the integrity and smoothness of image content, leading the reconstructed images to be noisy and less informative. In comparison, as shown in Figure 10(b), a more robust augmentation method like RandAugment primarily alters the style and geometric positioning of images but only slightly modifies the content semantics. As a result, it better preserves content consistency. This difference inspires us to factorize the influences on images into two dimensions: image styles and image content. In light of the findings in [32], we know that the channel-leve latent features encode more subtle style information, whereas the token-level latent features convey more content information. We thus decouple the latent features of MAE into two dimensions (i.e., a token dimension and channel dimension), and plot the latent feature distribution change between the generated image and the original image in these two dimensions.
20

---

(a) original image
(b) RandAugment
(c) MAE reconstruction
nolsed-added MAE
(e) Guided MAE
Figure 10: An illustrated visualization of the generated images by (b) RandAugment, (c) MAE recon- struction, (d) random pixel-level variation over latent features, and our guided MAE expansion. We find our guided MAE can generate content-consistent images of diverse styles.
Figure shows the visualization of this latent feature distribution change. The added pixel-Ievel noise changes the token-level latent feature distribution more significantly than RandAugment (cf. Figure Il))).. However, it only slightly changes the channel-level latent feature distribution (cf. Figure 11)).. This implies that pixel-level noise mainly alters the content of images but slightly changes their styles, whereas RandAugment mainly influences the style images while maintaining their content semanties. In light of this observation and the effectiveness of Rand/Augent, motivated to disentangle latent features into the two dimensions, and particularly conduct channel- level noise to optimize the latent features in our method. As shown in Figure 11, the newly explored channel-level noise variation varies the channel-level latent feature more significantly than the token- level latent feature. It thus can diversify the style of images while maintaining the integrity of image content. This innovation enables the explored MAE expansion strategy to generate more informative samples compared pixel-level noise variation (cf. Figure 10(d) vs. Figure 10(c)), leading to more effective dataset expansion, as shown in Figure 6. In light of this finding, we also conduct channel-level noise variation for GIF-SD.
0.20
Pixcl-leval
Pixel
RandAugment
RandAugment
Our guided channel-level nolee
Our guided channel-eeeel noss
0.15
Difinence of charmel-lovwe distrbbtton
niferrence of tokon- features
0.05
0.
0.00
-2
0.05
0.10
50
100
150
200
200
400
600
800
1000
Token index
Channel index
(8) Difference of token-level feature distribution
(b) Difference of channel-tevel feature distribution
Figure 11: Changes of the latent feature distributions along the token dimension and the channel dimension, between the latent feature of the generated image and that of the original image.
21

---

B.3 How to design prompts for Stable Diffusion?
Text prompts play an important role image generation of Stable Diffusion. The key goal of prompts in dataset expansion is to further diversify the generated image without changing its class semantics. We find that domain labels, class labels, and adjective words are necessary to make the prompts semantically effective. The class label is straightforward since we need to ensure the created samples have the correct class labels. Here, we show the influence of different domain labels and adjective words on image generation of Stable Diffusion.
Domain labels. We first visualize the influence of different domain prompts on image generation. As shown in Figure 12, domain labels help to generate images with different styles, We note that similar domain prompts, like "a sketch of" and "a pencil sketch of", tend generate images with similar styles. Therefore, it is sufficient to choose just one domain label from a set of similar domain prompts, which does not influence the effectiveness of dataset expansion but helps reduce the redundancy of domain prompts. In light of this preliminary study, we design the domain label set by ("an image of", real-world photo of", "a cartoon image of", "an oil painting of", "a sketch of").
Prompts
Input
Images generated by pre-trained Stable Diffusion
"An image samoyed"
*An oil painting of samoyed"
"A cartoon image samoyed
"A sketch of samoyed"
"A pencil sketch of samoyed"
Figure 12: The influence of the domain prompts on image generation of pre-trainee traiiraied Stable Diffusion. The input image is selected from the Pets dataset. Here, the strength hyper-panee parameter is set to 0.9, and the scale is set 20.
22

---

Adjective words. We next show the influence of different adjective words on image generation of Stable Diffusion. As shown in Figure 13, different adjectives help diversify the content of the generated images further, although some adjectives may lead to similar effects on image generation. Based on the visualization exploration, we design the adjective set by [" "colorful", "stylized", "high-contrast", "low-contrast", "posterized", "solarized", "sheared", "bright", "dark").
Prompts
Input
Images generated by pre-trained Stable Diffusion
"An image of samoyed"
"An image colorful samoyed"
"An image of stylized samoyed
"An image of bright samoyed"
'An image of sheared samoyed"
"An image of solarized samoyed
"An image of posterized samoyed"
"An image of high- contrast samoyed"
Figure 13: The influence of the adjective prompts on image generation of pre-trained Stable Diffusion. The input image is selected from the Pets dataset. Here, the strength hyper-parameter is set to 0.9, and the scale is set 20.
23

---

B.4 How to set hyper-parametery for Stable Diffusion?
B.4.1 Hyper-parameter of strength
The hyper-parameter of the nosing strength controls to what degree the initial image is destructed. Setting strength to 1 corresponds to the full destruction of information in the input image while setting strength to 0 corresponds to no destruction of the input image. The higher the strength value is, the more different the generated images would be from the input image. In dataset expansion, the choice of strength depends on the target dataset, but we empirically find that selecting the strength value from [0.5, 0.9] performs better than other values. A too-small value of strength (like 0.1 or 0.3) brings too little new information into the generated images compared the seed image. At the same time, too too-large value (like 0.99) may degrade the class consistency between the generated images and the seed image when the hyper-ppauee parammeerr of scale is large.
Strength
Input
Images generated by pre-trained Stable Diffusion
0
0.1
0.3
0.5
0.7
0.9
0.99
Figure 14: The influence of the "strength" hyper-parameter on image generation of pre-trained Stable Diffusion. The input image is selected from the Pets dataset. The prompt is "an image of colorful samoyed", while the scale is set to 20.
24

---

B.4.2 Hyper-paranee parameter of scale
The hyper-parameter of scale controls the importance of the text prompt guidance image generation of Stable Diffusion. The higher the scale value, the more influence the text prompt has on the generated images. In dataset expansion, the choice of strength depends the target dataset, but empirically find that selecting the strength value from [5, 50] performs better than other values. A too-small value of scale (like I) brings too little new information into the generated images, while too-large value (like 100) may degrade the class information of the generated images.
Scale
Input
Images generated by pre-trained Stable Diffusion
0
1
5
10
20
50
100
Figure 15: The influence of the "scale" hyper- syper-parameter on image generation of pre-trained Stable Diffusion. The input image is selected from the Pets dataset. The prompt is "an image of colorful samoyed", while the strength is set to 0.9.
25

---

B.5 More discussions on the effectiveness of zero-shot CLIP
In GIF, we exploit the zero-shot discriminability of the pre-trained CLIP to guide dataset expansion. In Table 1, we have found that the zero-shot performance of CLIP is not significantly good, particularly on medical image datasets. is interesting to know whether further fine-tuning CLIP on the target medical dataset can bring further improvement. To determine this, we further compare the results of GIF-MAE with fine-tuned CLIP and with zero-shot CLIP based on OrganSMNIST. To be specific, we add a linear classifier on the top of the CLIP image encoder and fine-tune the CLIP model.
As shown in Table 8, GIF-MAE with fine-tuned CLIP performs only comparably to that with zero-shot CLIP, which reflects that the CLIP's zero-shot classifier is enough to provide sound guidance. The reason is that, although the zero-shot performance is not that good, CLIP still plays an important anchor effect in maintaining the class semantics of the generated samples and helps to bring new information. Let us first recall the class maintanaid informativeness score: Sin/ = s + (s log(s) - s' log(s')). Specifically, no matter whether CLIP zero-shot classifier is accurate or not, maximizing s/1 essentially uses the prediction of the seed data as an anchor in the CLIP semantic space to regularize the class semanties of the perturbed features. This ensures the created data maintain the correct class, which is highly important for effective dataset expansion. In addition, maximizing the entropy difference, i.e., 8 log(s) s' log(s'). encourages the perturbed feature to have higher entropy regarding CLIP zero-shot prediction. When CLIP zero-shot classifier is accurate, the entropy increment enables the created data to become more difficult to classify regarding CLIP zero-shot discrimination and thus brings more information for classification model training. When CLIP zero-shot classifier is not that accurate, the entropy increment introduces variations into the created data and makes them different from the seed data. Under the condition that the true class is maintained, this optimization is beneficial to boosting the diversity of the expanded dataset, which is helpful for model training. Hence, CLIP's zero-shot abilities are useful for guided imagination in various image domains.
Afterwards, given that zero-shot CLIP can provide valuable guidance despite its limited accuracy, one may wonder whether aadom-initialized deep model could serve similar function. However, as shown in Table 8, using random-initialinn performs much worse than zero-shot CLIP (i.e., 79.0 vs. 80.6). This could be attributed to the fact that, although the classifiers of both random ResNet50 and zero-shot CLIP struggle with the target medical classes, the CLIP's pre-training results in a feature space that is more semantically meaningful and representative than a mandomly-nitializzz zero-shot CLIP to better anchor the class semantics of synthetic samples, thereby leading to more effective dataset expansion. These empirical observations further verify the effectiveness of using zero-shot CLIP in guiding dataset expansion.
ResNer50 as the guidance model for dataset expansion
ResNet50. This distinction allows
Table 8: Comparison between the model performance by GIF-MAE expansion with zero-shot CLIP guidance and fine-tuned CLIP guidance, well random-initiale ResNet-50 guidance, based on the OrganSMNIST medical image dataset. All results are averaged over three runs.
OrganSMNIST
Guidance model
Guidance model accuracy
Model accuracy
Original dataset
76.3
Random-initializz ResNet50
7.1
79.0 (+2.7)
5 5x-expanded by GIE-MEE
Fine-tuned CLIP Zero-shot CLIP (ours)
75.6
80.7 (+4.4)
7.7 20.0
80.6 (+4.3)
26

---

B.6 Do we need to fine-tune generative models on medical image datasets?
Stable Diffusion (SD) and DALL-E2 are trained on large-scall datasets consisting natural image and text pairs, showing powerful capabilities in natural image generation and variation. However, when we directly apply them expand medical image datasets, find the performance improvement is limited, compared to MAE as shown in Table 9.
Table 9: Accuracy of ResNet-50 trained the 5x 5x-expanded medical image datasets by GIF based on SD and DALLE wo and w/ fine-tuning, All results are averaged over three runs.
Dataset
PathMNIST
BreaSIMNIST
OrganSMNIST
Average
Original
72.4
55.8
76.3
68.2
GIF-MAE
82.020.7
73.3
80.620.6
78.6
GIF-DALLE (w/o luning)
78.4
59.3
76,4
71.4
GIF-DALLE (w/ tuning)
84.4
76.6
80.5
80.5
GIF-SD (w/o tuning)
80.8
59.4
79.5
73.2
GIF-SD (w/ tuning)
86.9
77.4
80.7
81.7
To pinpoint the reason, we visualize the generated images by SD on PathMNIST. As shown in Figure 16(top), we find that SD) fails to generate photo-realistic medical images, particularly when the hyper-parameter of strength is high. For example, the generated colon pathological images by pre-trained SD look more like a natural sketch and lack medical nidus areas found the input image. This implies that directly applying SD suffers from significant domain shifts between natural and medical images, preventing the generation of photo-realistic and informative medical samples using its image variation abilities. This issue also happens when applying DALL-E2 for medical dataset expansion. In contrast, MAE is reconstruction model and does not need to generate new content for the target images, it has much less negative impact by domain shifts. To address the issue, when applying SD and DALL-E2 to medical domains, we first fine-tune them on target medical datasets, followed by dataset expansion. Specifically, DALL-E2 is fine-tuned based image reconstruction, while SD fine-tuned based on Dreambooth [60]. As shown in Figure 16(bottom), the fine-tuned SD is able to generate medical images that more domain-similar to the input colon pathological image. Thanks to the fine-tuned SD and DALL-E2, GIF is able to bring more significant performance gains over GIE-MAE (ef. Table 9), and thus expands medical image datasets better.
Input
Generated images
Pre-trained
Stable Diffusion
Fined-tuned Stable Diffusion
Visualization of the synthetic medical colon pathological images by Stable Diffusion (SD) with or without fine-tuning. Here, the prompt of SD is "a colon pathological sketch of colorful debris", while the strength is set to 0.5. We find that SD suffers from severe domain shifts between natural and medical images and cannot generate photo-realistic and informative medical samples. In contrast, the generated medical images by the fine-tuned SD are more domain -imilar to the input colon pathological image.
Figure 16:
27

---

B.7 Visualization of created medical images
In the main paper, we visualize the created medical samples by GIF-SD. Here, we further visualize the created medical samples by GIF-MAL and discuss them. As shown in Figure 17, RandAugment randomly varies the medical images based on set of pre-defined transformations. However, due to its randomness, RandAugment may crop the lesion location of medical images and cannot guarantee the created samples be informative, even leading to noise samples. In contrast, our GIF-MAE can generate content-consisten images with diverse styles, so it can enrich the medical images while maintaining their lesion location unchanged. Therefore, GIF-MAE is able expand medical image datasets better than RandAugment, leading higher model performance improvement (cf. Table 1). However, GIF-MAE unable to generate images with diverse content, which limits its effectiveness. In comparison, SD, after fine-tuning, is able generate class-maintaiin content and styles, and thus achieves better expansion effectiveness (cf. Table 1). To summarize, our methods can expand medical image datasets more effectively than data augmentation.
samples with more diverse
Input
RandAugment
Our GIF-MAE
Our GIF-SD
Figure 17: Examples of the created samples for PathMNIST by RandAugment and GIF,
28

---

C Theoretical Analysis
In this appendix, we seek to analyze the benefits of our dataset expansion model generalization performance. Inspired by [67], we resort to the concept of 5-cover [62, 29] to analyze how data diversity influences the generalization error bound. Specifically, "a dataset a 8-cover of a dataset S" means set of balls with radius centered at each sample of the dataset E can cover the entire dataset S.
Definition C.I. (ō-cover [62]) Let (M, be a metric space, let S S M and let μ 0. set E S M is 8-cover for S, if for every s E S, there is an e E E such that p(s, e) s 6. The minimal ó regarding S and E is denoted by 8min.
In this work, we follow the assumptions of the work [67] and extend its Theorem I to the version of the generalization error bound. Let A be learning algorithm that outputs set of parameters, given training dataset D {-i, We VeEn)) with n. i.i.d. samples drawn from the data distribution Pz. Assume that the hypothesis function is A*-Lipschitz continuous, the loss function f(z, y) is X --pisschitz continuous for all y and bounded by L, and E(x,, yi: A) = for Vi E [n]. If the training set D is 8-cover of Pz, with probability at least 7. the generalization error bound satisfies:
1 Σ KE(M)
Pu [((x, yr A)]
((ri, A)I & Emin (X X*LC),
T2
where C' is constant, and the symbol indicates "smaller than" up an additive constant. According to the property of the d-cover, we then define the dataset diversity, called 5-diversity, by the inverse of the minimal :min:
Definition C.2. (6-diversity) If dataset E' is a 8-cover of the full dataset S, then the 8-diversity of the set E' regarding the full set ddie Jmsn
The ō-diversity is easy to understand: given a training set D = {ær,, 3i Heen that is a d-cover of the data distribution Pz, if the radius 6mi is high, the diversity of this dataset must be low. Then, have:
Theorem C.1. Let A denote a learning algorithm that outputs a set of parameters given a dataset D = {xi, Vijjejn) with i.i.d. samples drawn from distribution Pz. Assume the hypothesis function Xi-Lipschitz continuous, the loss function f(x, y) is X --pischhitz continuous for all y, and is bounded by L, with E(x,, ys; A) = 0 for all i E [n]. If constitutes ô-cover of Pz, then with probability least 7/ the generalization error bound satisfies:
- ((x.8iAA) & X +ALLC YI (E(m)
dáiv
e where C' is a constant, and the symbol S indicates "smaller than" up to an additive constant.
This theorem shows that the generalization error is bounded by the inverse of 6-diversity. That is, the more diverse samples are created by dataset expansion method, the more improvement of generalization performance would be made model training. In real small-data applications, the data limitation issue leads the covering radius 5 to be very large and thus the 5-diversity is low, which severely affects the generalization performance of the trained model. More critically, simply increasing the data number (e.g., via data repeating) does help the generalization since it does not increase 5-diversity. Instead of simply increasing the number of samples, our proposed GIF framework adopts two key imagination criteria (i.e., "class-maintained informativeness boosting" and "sample diversity promotion") to guide advanced generative models (e.g.. DALL-E2 and Stable Diffusion) to synthesize informative and diversified new samples. Therefore, the expanded dataset would have higher data diversity than random augmentation, which helps to increase d-diversity and thus improves model generalization performance.
29

---

D More Method and Implementation Details
D.1 Method details of GIF-DALLE
Thanks to strong image generation abilities, GIF-DALLE applies DALL-E2 [56] as its prior model which follows the pipeline described in Section 4. Its pseudo-code is provided in Algorithm where the image embedding obtained by fCLP44 serves as diffusion guidance to help the diffusion decoder to generate new images. GIF-DALLE conducts guided imagination on the CLIP embedding space.
We further clarify the implementation of the proposed guidance. Specifically, class mass-maintained informativeness f encourages the consistency between the predicted classification scores and s', and improves the information entropy for the predicted score of the generated sample s':
Sinf = = + (s log(s) - s' log(s')) s.L., j = arg max(s).
(5)
Here, j = argmaz(()) is the predicted class label of the original latent feature. Such a criterion helps to keep the class semantics of the optimized feature the same as that of the original one in the CLIP embedding space while encouraging the perturbed feature to have higher information entropy regarding CLIP zero-shot predictions. This enables the generated samples be more informative for follow-up model training. To promote sample diversity, the diversity Sai is computed by the Kullback-Leite (KL) divergence among all perturbed latent features of seed sample as follows:
Saie DKL
(f log(o(f))/o())
(6)
where f' denotes the current perturbed latent feature and f indicates the mean over the K perturbed latent features of this seed sample. implementing diversity promotion Sdies we measure the dissimilarity of two feature vectors by applying the softmax function o(-) to the latent features, and then measuring the KL divergence between the resulting probability vectors.
Algorithm 1: GIF-DALLE Algorithm
Input:
Original small dataset Do; CLIP image encoder fcur-()); DALL-E2 diffusion decoder G(+); CLIP zero-shot classifier w((); Expansion ratio K; Perturbation constraint
Initialize: Synthetic data set D, = 0:
for E D. do
S 0:
f fcup)(x);
// latent feature encoding for seed sample // CLIP zero-shot prediction for seed sample
8 = w(f))
for :=/ do
Initialize noise U(0, 1) and bias & ~N0, = = + =f b.);
noise perturbation (Eq. (1))
w(f]:
// CLIP zero-shot prediction
Sunf += of lo) 8' log(s')), st.j arg max( // clase-maintained informativeness (Eq. (5))
end
mean
Saie E,
- (o(o((()) // diversity (Eq. (6))
b 1== t arg max Sint Sat // guided latent optimization (Eq. (2))
for i= do
f. = Pp.c((1 2fff b));
// guided noise perturbation (Bq. (1)) // sample creation
1 G(
Add x5¹ D..
end end Output: Expanded dataset Do D..
More implementation details. In our experiment, DALL-E2 is pre-trained on Laion-400M [66] and then used for dataset expansion. The resolution of the created images by GIF-DALLE is 64x64 for model training without further super-resolution. Only when visualizing the created images, we use super-resolution up-sample the generated images to 256 for clarification. Moreover, we set è = 0.1 in the guided latent feature optimization. During the diffusion process, we set the guidance scale as 4 and adopt the DDIM sampler [70] for 100-step diffusion. For expanding medical image datasets, it is necessary to fine-tune the prior model for alleviating domain shifts.
30

---

D.2 Method details of GIF-SD
GIF-SD applies Stable Diffusion (SD) [59] as its prior model. As its encoder differs from the CLIP image encoder, we slightly modify the pipeline of GIF-SD.
Pipeline. As shown in Algorithm 2. GIF-SD first generates latent feature for the seed image via its image encoder. Following that, GIF-SD conducts prompt-based diffusion for the latent feature, where the generation rule of prompts will be elaborated in Eq. (7). Please note that, with suitable prompt design, the prompt-based diffusion helps to create more diversified samples. Afterward, GIF-SD) conducts channel-wise noise perturbation. Here, the latent feature of SD has three dimensions: two spatial dimensions and one channel dimension. As discussed in our preliminary (cf. Appendix B.2), the channel-level latent feature encodes more subtle style information, whereas the spatial-level latent features encode more content information. In light of the findings in this preliminary study, we particularly conduct channel-level noise to optimize the latent features in GIF-SD for further diversifyyng the style of the generated images while maintaining the content semantics of the latent features (after prompt-guice
diffusion) unchanged. Based on the randomly perturbed feature, GIF- SD generates an intermediate image via its image decoder and applies CLIPP to conduct zero-shot prediction for both the seed and the intermediate image to compute the guidance. With the guidance, GIF-SD optimizes the latent features for creating more style-diverse samples. Here, GIF-SD conducts guided imagination on its own latent space.
Algorithm 2: GIF-SD Algorithm
Input: Original small dataset Do; SD image encoder f()) and image decoder G(-): SD diffusion module faa( (prompt)): );; CLIP image encoder fcuru()); CLIP zero-shot classifier wn-);; Expansion ratio K; Perturbation constraint E.
Initialize: Synthetic data set DD = 0;
for E D. do
Sins = f= f(x);
// latent feature encoding for seed sample // Prompt generation (Eq. (7))
Randomly sample a [prompt]
f = [fm [prompt))
// SD latent diffusion 1/ CLIP zero-shot prediction for seed sample
w (x)c for i=I..., K do
Initialize noise 21 U(0,) 1) and hias ~N(0 f = (1 2.) b.):
// noise perturbation (Eq. (1)) // CLIP zero-shot prediction // maint-mmintained
8' = w Smn += of (slog(s) s' log(s')), s.t. arg max(s); informativeness (Eq. (5))
end
mean(
Saiv = = (P = E() (og(o((2)/(())) // diversity (E)
{2/, 1= t arg max, San + San
// guided latent optimization (Eq. (2))
for := do
J [1 + z\). b))=
// guided noise perturbation (Eq. (1))
x G(f();
// sample creation
Add x + D..
end end
Output: Expanded dataset D. U D..
Rule of prompt design. In our preliminary studies in Appendix B.3, we find that domain labels, class labels, and adjective words are necessary to make the prompts semantically effective. Therefore, design the prompts using the following rule:
Prompt == [domain] of a(n) [adj] [class].
For example, "an oil painting of colorful fox". To enable the prompts to be diversified, inspired by our preliminary studies, we design a set of domain labels and adjective words for natural image datasets as follows.
Domain label set: ["an image of", real-world photo of", cartoon image of". "an oil painting of". "a sketch of"]
31

---

Adjective word set: [" ", "colorful", "stylized", "high- thigh-contrast", "low-contrast', "posterized", "solarized", "sheared". "bright". "dark"]
For a seed sample, we randomly sample a domain label and an adjective word from the above sets to construct prompt. Note that, for medical image datasets, cancel the domain label set and replace it as the modality of the medical images, e.g., ["Abdominal CT image of"), ["Colon pathological image jf").
Implementation details. In our experiment, we implement GIF-SD based on CLIP VIT-B/32 and Stable Diffusion v1-4, which are pre- pre-trained on large datasets and then used for dataset expansion. Here, we use the official checkpoints of CLIP VIT-B/32 and Stable Diffusion v1-4. The resolution of the created images by GIF-SD is 512x512 for all datasets. Moreover, for guided latent feature optimization in GIF-SD, we set e = 0.8 for natural image datasets and = 0.1 for medical image datasets. Here, further adjust performance. During the diffusion process, we adopt the DDIM sampler [70] for 50-step latent diffusion. Moreover, the hyper-parameter of strength and scale in SD depend on datasets, while more analysis is provided in Appendix B.4. Note that, for expanding medical image datasets, it is necessary to fine-tune the prior model for alleviating domain shifts.
4 for Caltech101
increase image diversity for better
32

---

D.3 Method details of GIF-MAE
Thanks to strong image reconstruction abilities, our GIF-MAE applies the MAE-trained model [24] as its prior model. As its encoder is different from the CLIP image encoder, we slightly modify the pipeline of GIF-MAE.
Pipeline. As shown in Algorithm 3, GIF-MAR first generates a latent feature for the seed image via its encoder, and conducts channel-wise noise perturbation. Here, the latent feature of MAE has two dimensions: spatial dimension and channel dimension. As discussed our preliminary (cf. Appendix B.2), the channel-level latent feature encodes more subtle style information, whereas the token-level latent feature encodes more content information. Motivated by the findings in this preliminary study, we particularly conduct channel-level noise to optimize the latent features in our GIF-MAE method for maintaining the content semantics of images unchanged. Based on the perturbed feature, GIF-MAE generates an intermediate image via its decoder and applies CLIP to conduct zero-shot prediction for both the seed and the intermediate image to compute the guidance. With the guidance, GIF-MAE optimizes the latent features for creating content- consssset samples of diverse styles. Here, GIF-MAF conducts guided imagination on its own latent space.
Algorithm 3: GIF-MAE Algorithm
Input: Original small dataset Do; MAE image encoder f(-) and image decoder G(-); CLIP image encoder fcur-1-);; CLIP zero-shot classifier w(-); Expansion ratio K; Perturbation constraint s.
Initialize: Synthetic data set D,
for x E D. do
Sinf = 0: f f(x); $ w(fcru(())
// latent feature encoding for seed sample 1/ CLIP zero-shot prediction for seed sample
for j= do
Initialize noise ~ 2(00 1) and bias by ~ (00, 1);
P),[([1+ (2)f((++)+++)) +)f+ b.):
// channel-level noise perturbation (Eq. (1)) // intermediate image generation
E = G(J):
8' w/ (x(c)r
S.nf += of (s log(s) s' log(s')), st arg max(s): informativeness (Eq. (5))
// class-miintained
end
mean(
Sae = [.DKKo(o(()))0(())))
o(f) (og((())//(())) // diversity (Eq. (6))
(z/, 6. (- arg max, ,s Sinf Sai
// guided latent optimization (Eq. (2))
for 1=I.... do
J :P, (11+ z\). b1) // guided channel-wize wise noise perturbation (Eq. (1)) // sample creation
x G(f.);
Add x + D..
end end
Output: Expanded dataset D. U D..
Implementation details. In our experiment, we implement GIF-MAE based on CLIP VIT-B/32 and MAE VIT-L/16, which are pre-trate mre-trained large datasets and then fixed for dataset expansion. Here, we use the official checkpoints of CLIP VIT-B/32 and MAE VIT-L/16. The resolution the created images by GIF-MAE is 224x224 224 for all datasets. Moreover, set for guided latent feature optimization in GIF-
33

---

D.4 Implementation details of model training
We implement GIF in PyTorch based on CLIP VIT-B/32, DALL-E2, MAE VIT-L/I6, and Stable Diffusion (SD) V1-4, which are pre-trained on large datasets and then fixed for dataset expansion. We use the official checkpoints of CLIP VIT-B/32, MAE VIT-L/16. and SD v1-4, and use the DALL-E2 pre-trained on Laion-400M [66]. On medical datasets, since DALL-E2 and SD were initially trained on natural images and suffer from domain shifts to medical domains (please see the discussion in Appendix B,6), we fine-tune them on the target dataset before dataset expansion.
To fairly evaluate the expansion effectiveness of different methods, we use them expand the original small datasets by the same ratios, followed by training models from scratch on the expanded dataset with the same number of epochs and the same data pre-processing. In this way, the models are trained with the same number of update steps, so that all expansion methods are fairly compared. The expansion ratio depends on the actual demand of real applications. In the experiment of Table CIFAR100-Subsee is expanded by 5x, Pets is expanded by 30x, and all other datasets are expanded by x Moreover, all medical image datasets are expanded by 5x. In addition, all augmentation baselines expand datasets with the same expansion ratio for fair comparisons.
After expansion, we train ResNet-50 [25] from scratch for 100 epochs based the expanded datasets. During model training, we process images via random resize to x24 through bicubic sampling, random rotation, and random flips. If not specified, we use the SGD optimizer with a momentum of 0.9, We set the initial learning rate (LR) to 0.01 with cosine LR decay, except the initial LR of CIFARI00 CIFARI00-Subsee and OrganSMNIST is 0.1. The model performance is averaged over three runs in terms of micro accuracy on natural image datasets and macro accuracy on medical image datasets.
D.5 Discussions on limitations and broader impact
Limitations. We next discuss the limitations of method.
1. Performance of generated samples. The expanded samples are still less informative than real samples. For example, a ResNet-50 trained from scratch on our 5x-expanded CIFAR MAAR100-SA -u0-Subset achieves accuracy of 61.1%, which lags behind the 71.0% accuracy on the original CIFAR 100. This gap signals the potential for advancing algorithmic dataset expansion. Please Appendix F.6 for detailed discussions. We expect that this pioneering work can inspire more studies to explore dataset expansion so that it can even outperform a human-collected dataset of the same size.
2. Quality of generated samples. Some samples might have noise, exemplified in Figure 5b. Despite seeming less realistic, those samples are created following our guidance (e.g.. class-maintainee informativeness boosting). This ensures the class consistency of these samples, mitigating potential negative effects on model training. Nonetheless, refining the expansion method to address these noisy cases further enhance the effectiveness of dataset expansion.
3. Scope of work. Our current focus is predominantly on image classification. Exploring the adaptability of our method to other tasks, such object detection and semantic segmentation, is an intriguing next step.
Broader impact. We further summarize our broader impact. Our method can offer a notable reduction in the time and cost associated with manual data collection and annotation for dataset expansion, as discussed in Section 5. This can revolutionize how small datasets are expanded, making deep learning more accessible to scenarios with limited data availability (cf. Table 1).
34

---

E Dataset Statisties
The statisties of natural image datasets. We evaluate our method on six small-scale natural image datasets, including Caltech-101 [18], SEPAR10--Susse [41], Standard Cars [40], Oxford 102 Flowers [49], Oxford-IIIT Pets [50] and DTD [10]. Here, 1E-AR100-Subset is an artificial dataset for simulating small-scale datasets by randomly sampling 100 instances per class from the original CIFARI 100 10 dataset, and the total sample number 10,000. These datasets cover a wide range of classification tasks, including Doarse-grained object classification (i.e., CIFARI 100-uusset and Caltech-10h 101), fine-grained object classification (i.e., Cars, Flowers and Pets) and texture classification (i.e., DTD). The data statistics of these natural image datasets are given in Table 10. Note that the higher number of classes or the lower number average samples per class dataset has, the more challenging the dataset is.
Table 10: Statistics of small-scale natural image datasets.
Dutusets
Tusks
Classes
Sumples
# Average samples per class
10
Course-grained object classification
102
3,060
30
CIFAR100-Sube Subset
Coarse grained object classification
100
10,000
100
Standard Cars
Fine- Fine-grained ohject classification
Oxford 102 Flowers
Fine-graine Fine-grained object classification
102
6,552
64
Oxford-IIIT Pets
Fine-grained object classification
37
3,842
104
Describable Textures (DTD)
Texture classification
3,760
The statisties of medical image datasets. To evaluate the effect of dataset expansion medical images, we conduct experiments on three small-scas medical image datasets. These datasets cover wide range of medical image modalities, including breast ultrasound (i.e., BreastMNIST [i]), colon pathology (i.e., PathMNIST [36]), and Abdominal CT (i.e., OrganSMNIST [88]). We provide detailed statisties for these datasets in Table 11.
Table 11: Statistics small-scale medical image datasets. To better simulate the scenario of small medical datasets, we use the validation sets of BreastMNIST and PathMNIST for experiments instead of training sets, whereas OrganSMNIST is based on its training set.
Datasets
Data Modality
Classes
# Samples
Average samples per class
BreastMNIST [1.
Breast Ultrasound
78
PathMNIST [36, 90]
Colon Pathology
1.112 10,004
OrganSMNIST [88, 90]
Abdominal CT
11
13,940
1,267
35

---

F More Experimental Results and Discussion
E1 More comparisons expansion with augmentations
F1111 More results on expansion efficiency
In Figure 4, we have demonstrated the expansion efficiency of our proposed GIF over Cutout, GridMask and RandAugment on the Cars, DTD and Pets datasets. Here, we further report the results on Caltech101, Flowers, and CIFAR 100-Subset datasets. As shown in Figure 18, 5 expansion by GIF-SD and GIF-DALLE has already performed comparably to 20x expansion of these augmentation methods, while 10x expansion by GIF-SD and GIF-DALLE outperforms 20x expansion by these data augmentation methods a lot. This result further demonstrates the effectiveness and efficiency of our GIF, and also reflects the importance of automatically creating informative synthetic samples for model training.
Cutou
GridMask
Randkugment
SinDEX-MAE (ours)
SimDEX-DALLE
GIF-SD
101
Oxford
CIFAR100-Subser
65
Vodel per merformance
60
Model performenca
86
85.4
58
54.5
.55
56
55.2
5/0
84
82
11.0
83
837
Model
R
48.
80
80
48.3
78
75.1
+5c
+10x
+00%
+5c
+10x
+20k
+1x
+2%
+5k
Expansion dize
Exponsion size
Expansion size
Figure 18: Accuracy of ResNet-50 trained from scratch on the expanded datasets with different expansion ratios based on Caltech101, Flowers, and CIFAR SIFARR00-SSubel datasets.
F.1.2 Comparison to Mixup and CutMix
We further compare our method to more advanced augmentation methods. Specifically, we apply Mixup-based methods, i.e., Mixup [99] and CutMix [96], to expand SEFAR100-Subse by 5 x and use the expanded dataset to train the model from scratch. As shown Table 12, GIF-SD performs much better than Mixup and CutMix, further demonstrating the superiority of our method over augmentation-base based expansion methods.
Table 12: Comparison between GIF and Mixup methods for expanding CIFAR100-Subsee by 5x.
CIFARI S0o-bsettt
Accuracy
Original dataset
35.0
Expanded dataset
5 exeaanddedd by Mixup [99]
45.6
5x eexpanded by CutMix [96]
50.7
5x -expanded by GIF-SD
61.1+0.8
36

---

E.1.3 Comparison with an advanced generative method
We further compare our method with an advanced generative method [26] for dataset expansion. This method includes strategies like language enhancement (LE) [26] and CLIP Filter (CF) [26]. We use this method to expand the CIFAR 100-S dataset based on Stable Diffusion (SD). As shown in the following table, SD combined with the method [26] still noticeably inferior to our GIF-SD for both training from scratch and CLIP tuning. This further demonstrates the superiority of our method.
Table 13: Comparison between GIF and the method [26] for expanding CIFAR100-Subsel by 5x.
CIFARI00-S
Training from scratch
CLIP fine-tuning
Original dataset
35.0
75.2
5x 5x-expanded dataset by SD+method [26]
55.1 (+20.1)
77.0 (+1.8)
5x-expanded dataset by GIF-SD (ours)
61.1 (+26.1)
79.4 (+4.2)
F.1.4 Comparison to infinite data augmentation
The training time varies based on the specific datasets. However, it is pivotal to note that all dataset expansion methods were compared based on the same expansion ratio, thus ensuring consistent training time/cost and fair comparisons. We acknowledge that training on an expanded dataset will inevitably take longer than training on the original dataset. However, as shown in Table 1 (cf. Section 5.1), the significant improvement in model performance (i.e., by 36.9% on average over six natural image datasets and by 13.5% on average over three medical datasets) makes the increased investment training time worthwhile.
Despite this, one may wonder how the explored dataset expansion would perform compared to training with infinite data augmentation. Therefore, in this appendix, further evaluate the performance of infinite data augmentation on the CIFAR 100-Sut Specifically, based on RandAugment, we train ResNet-50 using infinite online augmentation for varying numbers of epochs from 100 to 700. As shown in Table 14, using RandAugment to train models for more epochs leads to better performance, but gradually converges (around 51 % accuracy at 500 epochs) and keeps fluctuating afterward. By contrast, our proposed method proves advantageous with the same training consumption costs: training the model on the original CIFAR100-S dataset for 5x more epochs performs much worse than the model trained on our 5x 5x-expanded dataset. This comparison further underscores the effectiveness of our method in achieving higher accuracy without inflating training costs.
Table 14: Comparison between GIF-SD and infinite data augmentation on CIFAR100-Subset. Here, consumption costs equal data number x training epoch.
Methods
Epochs
Consumption
Accuracy
Original
Standard training Training with RandAugment
100
million
35.0
100
million
39.6
Training with RandAugment
200
2 million
46.9
Training with RandAugment
300
million
48.1 10.6
Training with RandAugment
400
4 million
49.6.1.4
Training with RandAugment
500
million
51.3
Training with Rand/ugment
600
6 million
51.1
Training with RandAugment
700
million
50.6+1.1
Expanded
5 S -xxpannded by GIF-SD
100
6 million
61.1
37

---

F11.5 Discussion of picking related samples from larger datasets
Picking and labeling data from larger image datasets with CLIP is an interesting idea for dataset expansion. However, such a solution is limited in real applications, since a large-scale related dataset may be unavailable in many image domains (e.g. medical image domains). Moreover, selecting data from different image domains (e.g. from natural images to medical images) is unhelpful for dataset expansion. Despite the above limitations in real applications, we also evaluate this idea on CIFAR 10-Subbett and investigate whether helps dataset expansion when there is larger dataset of the same image nature, e.g., ImageNet. Here, we use CLIP to select and annotate related images from ImageNet to expand CIFAR 100-Subset. Specifically, we scan over all ImageNet images and use CLIP to predict them to the class of CIFARIOO-Siu Soobbuee.. We select the samples with the highest prediction probability higher than 0.1 and expand each class by 5x. As shown in Table 15, the idea of picking related images from ImageNet makes sense, but performs worse than our proposed method. This result further demonstrates the effectiveness and superiority our method. In addition, how to better transfer large-scale datasets expand small datasets is interesting open question, and expect to explore it in the future.
Table 15: Comparison between GIF and picking related data from ImageNet for expanding CIFAR100- Subset by 5x.
CIFARIO0-Subset
Accuracy
Original dataset
35.0
Expanded dataset
5 x expanded by picking data from ImageNet with CLIP
40.9
x exppanddd GIF-SD
61.1
38

---

F.2 More results of benefits to model generalization
In CIFAR100-C [27], there are 15 types of OOD corruption (as shown Table 16), i.e., Gaussian noise, shot noise, impulse noise, defocus blur, glass blue, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic transformation, pixelation. and JPEG compression. Each corruption type has 5 different severity levels: the larger severity level means more severe distribution shifts between CIFAR 100 and CTFAR 100-C. In Table the main paper, we have shown the empirical benefit of our method to model
(OOD) generalization based on CIFAR 100-C with the severity level Here, we further report its performance on CIFAR100-C with other severity levels. As shown in Table 16, our method is able to achieve consistent performance gains across all severity levels, which further verifies the benefits of GIF to model OOD generalization.
Table 16: Corruption Accuracy of ResNet-50 trained from scratch on CIFARI00-S and our 5x expanded dataset, under 15 types of corruption in CIFARI00-C with various severity levels.
(a) CIFARI00-C with the severity level
Weather
Diginl
Doineet
Canss:
Shot
Impui.
Defoc.
Glass
Motion
Zoom
Snow
Frost
Comm:
Elastic
Pixet
JPEG
Original
25,6
20.3
25.0
34.2
32.2
31.7
30.9
32.3
25.3
51.8
33.7
20.2
31.7
34.1
30.9
30.7
5% expanded by GIP SD
50.3
34.6
50.5
.9.2
29.4
53.7
51.9
53.1
54.0
58.7
59.5
57.1
52.5
57.9
54.7
53.2 (+22.5)
20x expandedd GIF-SD
$5.0
60.5
$4.8
66.1
30.2
$6.0
58.0
61.1
62.2
65.1
66.2
64.3
39.2
63.8
60.8
58.9 (+27.2)
(b) CIFARI00-C with the severity level
Noise
Blar
Weather
Diginl
Classs
Iupul.
Defix:
Zusen
Smow
Froe:
Foy
Average
Original
Sx-erpandedd
(-21
31.2
(425.7)
(c) CIFARIO0-C with the severity level
Nuise Shot
Wealler
Diginl
Datact
Cunss
Inpui
Defox
Class
Mosion
7oem
Snow
Froe:
Fog
Bii
Com:
Flastic
Pisel
1P80
Averggs
Original
12,8
17.0
12.5
30.5
31.7
25.2
28.5
25.5
19.0
15.6
28.3
11.5
29.5
33.6
28.8
23.6
5x expanded GIP SD
29.1
36.4
32.7
31.9
12.4
39.2
46,0
45.3
38.1
47.1
55.7
37.3
45.6
$3.2
49.4
43.3 (+19.3)
20x exuadedd GIl SD
31.8
39.2
34.7
58.4
33.4
43.1
51.9
51.7
47.4
55.0
63.
46.5
54.9
58.0
53.6
48.3 (+24.6)
(d) CIFAR100-4 with the severity level
Noise
Blur
Weather
Digini
Glass
Motion
Elustic
Orgiiaal
10.8
14.3
7.7
28.5
29.3
25.2
27.8
23.5
19.5
16.1
24.9
7.4
29.0
33.0
28:1
21.5
5xcespanded GIF-
25.3
31.2
18.0
45.1
21.4
39.6
42.5
41.7
37.7
40.2
52.1
26.1
44.2
47.8
48.2
37.4 (-15.))
20x 30x-equaded by GIF -D
27.4
33.7
20.2
50.7
21.7
43.9
47.8
48.8
46.7
47.6
68.7
35.3
47.9
49.3
51.2
42.2 (1-007)
(e) CIFAR 100-C with the severity level
Noise
Blur
Weather
Digitl
Clnses
Impul.
Fog
Nverage
Original
9.4
10.7
18.9
19.4
16.6
29.0
27.3
exponded GII
21.4
23.8
35.1
24.7
35.6
(+11.1)
20x seganded GIF-SD
22,9
25.
33.8
46.4
38.4
31.1
40.4
$2.0
48.8
(+14.7)
39

---

F.3 More results of applicability to various model architectures
In Table 3, we have demonstrated the generalizability of our expanded Cars dataset various model architectures. Here, we further apply the expanded Caltech101, Flowers, DTD, CIFAR 100- S, and Pets datasets (5x x expansion ratio) by GIF-SD and GIF-DALLE to train ResNeXt-50 [85], WideResNet-50 [97] and MobileNet V2 [64] from scratch. Table 17 shows that our expanded datasets bring consistent performance gains for all the architectures
datasets. This further affrrm the versatility of our expanded datasets, which, once expanded, are readily suited for training various model architectures.
Table 17: Model performance of various model architectures trained 5x expanded natural image datasets by GIF.
Caltech101 [18]
Dataset
ResNet-50
ResNeXt-50
WideResNet-50
MobilteNet-v2
Avg.
Original dataset
26.3
32.610.5
34.7
33.841.1
31.9
5x 5x-expanded by GIF-DALLE
57.3
55.240.1
61.8
59,4
58.4 (+26.5)
5x 5x-expanded by GIF-SD
54.4
52.8+1.1
60.7
55.640.5
55.9 (+24.0)
Cars [40]
Dataset
ResNet-50
ResNeXt-50
WideResNet-50
MobilteNet-v2
Avg.
Original dataset
19.8 19.8
18.420.5
32.0.0.8
26.2+4.2
24.1
5x 5x-expanded by LIF-DALLE
53. 53.1
43.740.2
60.0
47.8
51.2 (+27.1) I)
Sx 5x-exponded by GIF-SD
60.6
64.1
75.1
60.2
65.0 (+40.9)
Flowers [49]
Dataset
ResNet-50
ResNeXt-50
WideResNet-50
MobilteNet-v2
Avg.
Original dataset
74.1
75.8112
79.3
85.5
78.7
Sx Sxx-epaaded by GIF-DALLE
82.8
81.640.
84.6
88.8
84.4 (+5.7)
5x 5x-exponded by GIF-SD
82.1
82.0
85.0
89.0
84.5 (+5.8)
DTD [10]
Dataset
ResNet-50
ResNeXt-50
WideResNet-50
MobilteNet-v2
Avg.
Original dataset 5x 5x-exponded by GIF-DALLE
23.1
25.440.6
26.1
28.1
25.7
31.2+0.9
30.640.1
35.3
37.4
33.6 (+7.9)
5x 5x-exponded by GIF-SD
33.9
33.3
40.6
40.8
37.2 (+11.5)
CIFARI00-S [41]
Dataset
ResNet-50
ResNeXt-50
WideResNet-50
MohilteNet-v2
Avg.
Original dataset
35.0
36.3
42.0
50.9
41.1
Sx Sx-expanded by GIF-DALLE 54.5
52.440.7
55.3
56.2
54.6 (+13.5)
Sx Sx-expanded by GIF-SD
61.1
59.0
64.4
62.4
61.4 (420.3)
Pets [50]
Dataset
Re
ResNeXt-50
WideResNet-500
MobilteNet-12
Avg.
Original datase!
6.8_1.8
19.04
20.5
37.5
21.4
5x ex-andede by GIF-DALLE
46.2
52.34
66.22a.1
60.3
56.3 (+34.9)
5x 5x-expanded by GIF-SD
65.8
56.5
20.4
60.6
63.5 (+42.1)
40

---

F.4
OrganSMNIST More discussions on CLIP
In the following subsections, we provide more discussions on the comparisons with CLIP.
F.4.1
Why not directly transfer CLIP models to target datasets?
In our proposed GIF framework, we leverage the pre-trained CLIP model to guide dataset expansion. An inevitable question might be: why not directly use or transfer the CLIP model to the target dataset, especially given its proven effectiveness on many natural image datasets? Before delving into that, it is important to note that we aim to tackle small-data scenarios, where only a limited-size dataset is available and there are no large-scale external datasets with similar nature to the target dataset. Consequently, training new CLIP model on the target dataset (e.g., in the medical image domains) not feasible. Therefore, we rely on publicly available CLIP models for dataset expansion. Compared to directly using or transferring CLIP models, our dataset expansion introduces a necessary new paradigm for two primary reasons as follows.
First, our GIF method has better applicability to scenarios across various image domains. While CLIP demonstrates good transfer performance on certain natural image datasets, it struggles to achieve this performance on other domains, such medical image datasets. To illustrate this, we test the linear- probing and fine-tuming performance of the CLIP-trainee ResNet-50 model on three medical datasets. As shown in Table 18, directly employing or transferring the CLIP model yielded unsatisfactory results or only marginally improved performance significantly underperforming compared to our dataset expansion approach. The limited transfer performance is attributed the fact that, when the pre-trained datasets are highly different from the target datasets, the pre-training weights do not significantly bolster performance compared to training from scratch [55]. Such an issue cannot be resolved by conducting CLIP pre-training on these domains, since there is no large-scale dataset of similar data nature to the target dataset in real scenarios. In contrast, our GIF framework capable of generating images of similar nature as the target data for dataset expansion, enhancing its applicability to real-world scenarios across diverse image domains.
Second, our dataset expansion can provide expanded datasets suitable for training various network architectures. In certain practical scenarios, such as mobile terminals, the permissible model size is severely limited due to hardware constraints. Nonetheless, the publicly available CLIP checkpoints are restricted to ResNet-50, ViT-B/32, or even larger models, which may not be viable in these constrained settings. In contrast, the expanded dataset by our method can be readily employed to train various range of model architectures (cf. Table 3), making it more applicable to scenarios with hardware limitations. One might suggest using CLIP in these situations by conducting knowledge distillation from large CLIP models to facilitate the training of smaller model architectures. However, as indicated in Table and Table 18, although knowledge distillation of CLIP does enhance model performance on most datasets, the gains are limited. This arises from two key limitations of CLIP knowledge distillation. First, distillation can only yield marginal improvements when the performance of CLIP on the target dataset (e.g., medical image domains) is not good. Second, distillation tends to be ineffective when there is a mismatch between the architectures of student and teacher models [9, 71]. This comparison further underscores the advantages of our method for training various network architectures, while the CLIP model architectures are fixed and not editable.
Table 18: Comparison between our methods and directly fine-tuning CLIP models on three medical image datasets. All results are averaged over three runs.
Dataset
PathMNIST
BreastMNIST
Original dataset
72.4
55,8
76.3
CLIP linear probing
74.3
60.0=2.9
64.9
CLIP fine-tuning
78.4
67.2
78.9 10.1
CLIP knowledge distillation
77.3
60.2+13
77. 77.4-0.8
5 x 5x-exxanded by GIF-SD
86.9 L..3
77.4 1.8
80.7 10.2
41

---

F.4.2 Discussion on when to use GIF over zero-shot CLIP models
In Table I, it is noted that while zero-shot CLIP performs well on datasets like Caltech 101 and Pets, it struggles with medical image datasets. This poses the question: when should we prefer GIF over pre-trained CLIP models? Although zero-shot CLIP outperforms our GIF-SD the Caltech 101 and Pets datasets, our method demonstrates superior overall performance across six natural image datasets, well as medical image datasets (see Table 1). Thus, we recommend using our methud as the primary option.
Meanwhile, if the target dataset has high distributional similarity with the CLIP training dataset, may also be beneficial to consider CLIP alternative and see whether it can achieve better performance. Nevertheless, it is important to note that, as discussed in Section 5.1 and Appendix F.4.1, CLIP less effective in some specific application scenarios. For instance, its performance on non- natural image domains like medical images is limited (as shown in Table 4). Additionally, publicly available CLIP checkpoints are restricted to larger models like ResNet--5 and ViT-B/32, making them unsuitable for scenarios with hardware constraints (e.g, mobile terminals) where smaller model sizes are necessary. these scenarios, our proposed method exhibits promising performance (as shown in Tables 3 and 4), offering a more versatile solution.
42

---

E,5 More ablation studies
F55.1
The effectiveness of guidance in GIF-DALLE
GIF optimizes data latent features for informative sample creation by maximizing the designed objective functions of guidance (i.e. class-maintaines informativeness Sinf and sample diversity ) which are essential for effective dataset expansion. With these essential guidance criteria, as shown in Table 19, our guided expansion framework obtains consistent performance gains compared to unguided expansion with SD, DALL-E2, or MAE, respectively. This verifies the effectiveness of our criteria in optimizing the informativeness and diversity of the created samples.
Table 19: Accuracy of ResNet-50 trained from scratch on small datasets and their expanded datasets by various methods, Here, CIFAR 100-Subset is expanded by 5x, Pets is expanded by 30x, and all other natural image datasets are expanded by 20x. All medical image datasets are expanded by 5x. Moreover, MAE, DALL-E2 and SD (Stable Diffusion) are the baselines of directly using them to expand datasets without our GIF, All results are averaged over three runs.
Datuoet
Natral image danses
Medical image daasets
Catech101
Flowers
DTD
CIFAR10005
Pets
Average
PUIMNIST
BicasMNNSST
OrpoSMNIST
Averase
Drightal
26.3
74.1
23.1
35.0
72.4
55.8
76.3
Exuadedd by MAE
$0.6
25.9
76,3
27.6
443
39.9
(11.2)
81.7
63.4
78.6
746(1644)
Eepanded by CII-MAF (sans)
58.4
44.5
14.4
14.2
50.9
51.4
54.4 (+23.5)
82.0
75.5
00.6
Eppusted by DALL--02
61.
M1
57,01-36113
82.8
7088
79.3
Exuadedd GIP C.F.DALLE (ouus)
63.0
59.2
39.5
60.8 (1299)
84.4
76.6
803 (+12.3)
Equaded
51.1
78.8
54.3 (-23.4)
85.1
73.8
78.9
(+31l)
Ripusted Gill-D (osn)
45.1
18.3
(+3669
86.9
77.4
81.7
81.7
In this appendix. we further explore the individual influence of these criteria on GIF-DALLE. Specifically, mentioned in Appendix D.I, GIF-DALLE conducts guided imagination the CLIP embedding space, which directly determines the content of the created samples. With the aforementioned essential criteria, shown in Figure 2b, our GIF-DALLE is able to create motorbike images with more diverse angles of view and even a new driver compared to unguided DALLE expansion. Here, we further dig into how different criteria influence the expansion effectiveness of GIF-DALLE. As shown in Table 20, boosting the class-maintained informativeness Siny is the foundation of effective expansion, since it makes sure that the created samples have correct labels and bring new information. Without it, only Saie cannot guarantee the created samples to be meaningful, although the sample diversity improved, even leading to worse performance. In contrast, with S diversity promotion Sate can further bring more diverse information to boost data informativeness and thus achieve better performance (cf. Table 20). Note that contrastive entropy increment log(s) ) - s' log(s') ( in class-maintaines informativeness plays different roles from diversity promotion Sate. Contrastive entropy increment promotes the informativeness of each generated image by increasing the prediction difficulty over the corresponding seed image, but this guidance cannot diversify different latent features obtained from the same image. By contrast, the guidance of diversity promotion encourages the diversity of various latent features of the same seed image, but cannot increase the informativeness of generated samples regarding prediction difficulty. Therefore, using the two guidance together leads the generated images to be more informative and diversified, thus bringing higher performance improvement (cf. Table 20). As a result, as shown in Table 19, with these two essential criteria as guidance, the model accuracy by GIF-DALLE is 3.3% accuracy higher than unguided data generation with DALL-E2.
Table 20: Ablation of guidance in GIF-DALLE for expanding CIFARI00-Subset by 5x.
Method
Sinf
Saso
CIFAR100-Subset
52.1
/
53.1
GIF-DALLE
V
51.8
/
54.5
43

---

F.5.2 The effectiveness of guidance in GIF-SD
We next analyze GIF-SD. As mentioned in Appendix D.2, we conduct channel-wise noise perturbation for latent optimization in GIF-SD. As analyzed in Appendix B.2, the channel-level latent feature encodes more subtle style information, and conducting channel-level noise perturbation diversifies the style of images while maintaining its content integrity. Therefore, our guided optimization particularly diversifies the style of the created images, without changing the content semantics of the latent features after diffusion (cf. Figure 19). Moreover, the prompt-guided diffusion with our explored prompts helps to enrich image styles further (e.g. cartoon or oil painting). Hence, combining both of them enables GIF-SD to create new samples with much higher diversity (cf. Figure 19).
Methods
Input
Generated images
Stable Diffusion
our guided optimization
our designed prompts
our guided optimization designed prompts
Figure 19: Visualization of the generated images by SD with our explored guided optimization and designed prompts.
We then investigate the individual influence of our guidance criteria GIF-SD on the basis of our prompt-guided diffusion. As shown in Table 21, both the blass-maintaines informativeness guidance Sinf and the diversity promotion guidance Saie contribute to model performance. One interesting thing is that, unlike GIF-DALLE that does not work without S.nf- GIF-SD can work well using only the diversity promotion guidance S The key reason is that GIF-SD conducts channel-level noise perturbation over latent features and particularly diversifies the style of the created images without changing the content semantics of the latent features after diffusion. Therefore, the class semantics can be maintained well when only promoting sample diversity. Moreover, combining both guidance criteria enables GIF-SD achieve the best expansion effectiveness (cf. Table 21), leading to promising performance gains (i... 13.5% accuracy improvement on average over six natural image datasets) compared unguided expansion with SD (cf. Table 19).
Table 21: Ablation of guidance and prompts in GIF-SD for expanding CIFARIO0-Subset by 5x.
Method
Designed prompts Smn
Saso
CIFAR100-Subset
52.940.8
56.2+1.
GIF-SD
59.6
59.441.2
61.1
44

---

F,5.3 Discussions on the constraint of the perturbed feature in GIF
The hyper-pann parerraareeterrr is used to ensure that the perturbed feature does not deviate from the input feature significantly. and its value depends on the prior model and target dataset. As described in Appendix D, for GIF-SD, set E = 0.8 for most natural image datasets and further adjust = 4 for 1a to increase dataset diversity for better performance. Once determined for a given prior model and target dataset, remains fixed for various expansion ratios. As shown in the following table, there no need increase when the expansion ratio becomes larger.
Table 22: Ablation of hyper-parameter on Caltech101 for GIF-SD.
on Caltech 101
2
4
8
x -xxpandded by GIF-SD
53,0
54.4
53.6
10x -ex-aaxadded by GIF-SD
59.2
59.3
58.2
45

---

F6 Discussion of training models with only expanded images
is interesting to know how the model performs when trained with only the created images by our method. To this end, we train ResNet-50 from scratch using only images generated by GIF-DALLE on the CIFAR100-Subset and compare the result with a model trained on the real images of the CIFAR 100-Subset.
We report the results regarding 1 x expansion in Table 23. We find that the model trained with only 1 x synthetic images performs worse than the model trained with the original dataset, indicating that the quality of synthetic data still lags behind that of real images. Please note that this does not degrade our contribution, since our work aims to expand small datasets rather than replace them entirely. Moreover, mixing the original images with the created images the same size as the original dataset can lead to better performance than using only the original dataset. This suggests that the created images are not simple repetition of the original dataset but offer new information that is useful for model training. Lastly, the model trained on the complete Ix- Ix-expanded dataset significantly outperforms the models trained either only on the original dataset or solely on the generated images, underscoring the potential of synthetic images in expanding small-scale datasets for model training.
Table 23: Performance of the model trained with only the expanded data of the 5x-expanded CIFAR 100-Subset dataset by GIF-DALLE.
CIFARIO0-
Data amount
Accuracy
Training with real images in original dataset
10,000
35.0
Training with only the Ix-created data by GIF-DALLE Training with mixing original data and I x xx-rreated data by GIF-DALLE
10,000
21.0+0.7
10,000
37.2
Training with Ix-expanded dataset by GIF-DALLE
20,000
45.6+1.1
We next report the results regarding 5x expansion in Table 24. The model trained with 5x synthetic images has already performed comparably to the model trained with real images. This result further verifies the effectiveness of our explored dataset expansion method. Moreover, the model trained with the full 5 5x-exandeed dataset performs much better than that trained with only the original dataset or with only the generated images. This further shows that using synthetic images for model training is promising direction. We expect that our innovative work on dataset expansion can inspire more studies to explore this direction in the future.
Table 24: Performance of the model trained with only the expanded data of the 5x- CIFART00-S 100-ubet dataset by GIF-DALLE.
CIFAR100-Subeer
Data amount
Accuracy
Training with real images original dataset
10,000
35.0+17
Training with only the 5x Sx-created data by DALLE
50,000
35.2+1.3
Training with 5x e-expanded dataset by GIF-DALLE
60,000
54.5
46

---

F.7 Effectiveness on long-tailed classification dataset
In previous experiments, we have demonstrated the effectiveness of our proposed method on relatively balanced small-scale datasets. However, real-world classification datasets are usually class imbalanced and even follow a long-tailed class distribution. Therefore, we further apply GIF-SD expand a long-tailed dataset, i.e., CIFAR 100LT [5] (with the imbalance ratio of 100), to see whether it is also beneficial to long-tailed learning. Here, we train ResNet-50 from scratch with the cross-entropy loss or the balanced softmax loss [35] for 200 epochs, where Balanced Softmax [35, 102] is class re-balancing loss designed for long-tailed learning.
As shown in Table 25, compared to training with cross-entropy directly on the original CIFARIO0-LT dataset, 20x expansion by our GIF-SD leads a 13.5% model accuracy gain. This demonstrates the effectiveness of our proposed method in long-tailed learning. More encouragingly, our GIF expansion boosts the performance of few-shot classes more than many-shot classes, which means that GIF helps to address the issue of class imbalance.
Besides the cross-entropy loss, our dataset expansion is also beneficial to model training with long- tailed losses, such as Balanced Softmax. As shown in Table 25, 20 x expansion by GIF-SD boosts the accuracy of the Balanced Softmax trained model by 14.8%, and significantly improves its tail-class performance by 26.8%. These results further demonstrate the applicability of our GIF to long-tailed learning applications. We expect that this work can inspire more long-tailed learning studies to explore dataset expansion since information lacking is an important challenge in long-tailed learning [103].
Table 25: Effectiveness of GIF-SD for expanding CIFAR 100-LT (imbalance ratio 100) by 10x, where all models are trained for 200 epochs. Here, Balanced Softmax [35, 102] is a class re-balancing losses designed for tailed learning.
CIFARI00-LT
Training losses
Many-shot classes
Medium-shot classes
Few-shot classes
Overall
Original
Cross-entropy Cross-entropy
70.5
41.1 54,9 (+13.8)
8.1
41.4
20x expanded by GIF-SD
(+9.0)
26.4 (+18.3)
54.9 (+13.))
Original
Balanced Softmax
67.9
45.8 59.2 (+13.4)
17.7
45.1
20x expanded by GIF-SD
Balanced Softmax
73.7 (+5.8)
44.5 (+26.8)
59.9 (+14.8)
F88 Effectiveness on larger-scale dataset
In previous experiments, we have demonstrated the effectiveness of our proposed method on small- scale natural and medical image datasets. In addition to that, one may also wonder whether our method can be applied to larger-scale datasets. Although expanding larger-scale datasets is not the goal of this paper, we also explore our method to expand the full CIFAR 100 by 5x for model training. As shown in Table 26, compared to direct training on the original CIFAR 100 dataset, our GIF-SI) leads to a 9.4% accuracy gain and GIF-DALLE leads to an 8.7% accuracy gain. Such encouraging results verify the effectiveness of our methods on larger-scale datasets.
Table 26: Effectiveness of GIF for expanding the full CIFAR100.
Dataset
CIFAR100
Original
70.9
Expanded
--eeaandded by GIF-DALLE x -x-epanneed by GIF-SD
79.6_1..
80.3
47

---

F.9
Safety check
Ethical considerations, especially Al research and data generation, are indeed paramount. Our approach is constructed with care to avoid negative implications, as evidenced in the following points:
Controlled generation: In our approach, the generation of synthetic data is driven by our expansion guidances, which ensure that new data derived directly and meaningfully from the original dataset. This controlled mechanism minimizes the risks of creating unrelated or potentially harmful images.
No personal sensitive data:
It is also worth noting that method primarily focuses on publicly available datasets like CIFAR, Stanford Cars, and similar, which do not con- tain personal sensitive information. As such, the risks related to privacy breaches mistepresentations are substantially diminished.
Following this, we further employ the Google Cloud Vision API to perform a safety check on the 50,000 images generated during 5x-expansion of CIFARI00-3 by GIF-SD. The Google Cloud Vision API is a tool from Google that uses deep learning to analyze and categorize content in images, commonly used for safety checks. It evaluates the likelihood of the image containing adult themes such as nudity or sexual activities, alterations made for humor or offensiveness (spoof), medical relevance, violent content, and racy elements which could include suggestive clothing or poses. This assessment aids in ensuring that images adhere to content standards and are appropriate for their target audiences.
As evidenced by Table 27, the synthetic images by our method are safe and harmless. To be specific, the majority of our generated images categorized either "Very unlikely" or "Unlikely" across all five metrics. Moreover, for categories like "Adult" and "Medical", the likelihood almost negligible. Moreover, the visualized images in Appendix G also highlight the benign nature of the images produced by our method.
Table 27: Safety check of the generated images of CIFARI00-S by our GIF-SD, in terms of different
metries of Google Cloud Vision API.
Metrics
Very unlikely
Unlikely
Neutral
Likely
Very likely
Adult
96%
4%
0%
0%
0%
Spoof
82%
15%
3%
0%
0%
Medical
86%
14%
0%
0%
0%
Violence
69%
31 %
0%
0%
0%
Racy
66%
25%
9%
0%
0%
http:://....uoooooooooooo/////// seach
48

---

G More Visualization Results
This appendix provides more visualized results for the created samples by our methods on various natural image datasets. Specifically, we report the synthetic images by GIF-SD Caltech101 in Figure 20, those by GIF-DALLE in Figure 21 and those by GIF-MAE in Figure 22. The visualized results show that our GIF-SD and GIF-DALLE can create semantic- consssseennntttet yet content-diversifeed images well, while GIE-MAE can generate content-consistent yet highly style-diversified images. The visualization of GIF-SD and GIF-DALLE on other natural image datasets are shown in Figures 23-32.
G.1
Visualization of the expanded images on Caltech101
G.I.I Visualization of the expanded images by GIF-SD on Caltech101
Input
Our GIF-SD expansion
Figure 20: Visualization of the created samples on Caltech101 by GIF-SD.
49

---

G.1.2
Visualization of the expanded images by GIF-DALLE on Caltech101
Input
Our GIF-DALLE expansion
Figure 21: Visualization of the created samples on Caltech101 by GIF-DALLE.
50

---

G.1.3 Visualization of the expanded images by GIF-MAE on Caltech101
Input
Our GIF-MAE expansion
X
Figure 22: Visualization of the created samples on Caltech101 by GIF-MAE
51

---

G.2
Visualization of the expanded images on Cars
G.2.1
Visualization of the expanded images by GIF-SD on Cars
Input
Our GIF-SD expansion
D2L
Figure 23: More visualization of the synthetic samples on Cars by GIF-SD.
52

---

G.2.2 Visualization of the expanded images by GIF-DALLE on Cars
Input
Our GIF-DALLE expansion
7030
Figure 24: More visualization of the synthetic samples on Cars by GIF-DALLE.
53

---

G.3
Visualization of the expanded images on Flowers
G.3.1
Visualization of the expanded images by GIF-SD on Flowers
Input
Our GIF-SD expansion
Figure 25: More visualization of the synthetic samples on Flowers by GIF-SD.
54

---

G.3.2
Visualization of the expanded images by GIF-DALLE on Flowers
Input
Our GIF-DALLE expansion
Figure 26: More visualization of the synthetic samples on Flowers by GIF-DALLE.
55

---

G.4
Visualization of the expanded images on Pets
G.4.1
Visualization of the expanded images by GIF-SD on Pets
Input
Our GIF-SD expansion
Figure 27: More visualization of the synthetic samples on Pets by GIF-SD.
56

---

G.4.2 Visualization of the expanded images by GIF-DALLE on Pets
Input
Our GIF-DALLE expansion
Figure 28: More visualization of the synthetic samples on Pets by GIF-DALLE.
57

---

G.5
Visualization of the expanded images on CIFAR100-Subset
G.5.1
Visualization of the expanded images by GIF-SD on CIFAR100-Subsel
Input
Our GIF-SD expansion
Figure 29: More visualization of the synthetic samples on CIFAR100-Subset by GIF-SD.
58

---

G.5.2
Visualization of the expanded images by GIF-DALLE on CIFARI0O-Subset
Input
Our GIF-DALLE expansion
Figure 30: More visualization of the synthetic samples on SEFFAR00-Subsee by GIF-DALLE. Note that the resolution of the input CIFAR100 images is small (i.e. 32x x2),, so their visualization is little unclear.
59

---

G.6
Visualization of the expanded images on DTD
G.6.1 Visualization of the expanded images by GIF-SD on DTD
Input
Our GIF-SD expansion
Figure 31: More visualization of the synthetie samples DTD by GIF-SD.
60

---

G.6.2 Visualization of the expanded images by GIF-DALLE on DTD
Input
Our GIF-DALLE expansion
Figure 32: More visualization of the synthetic samples on DTD by GIF-DALLE.
61


Reviews:
[{"review_id": "041EQWJfBA", "reviewer": "Reviewer_7xaH", "summary": "This paper explores new dataset expansion task that solves data scarcity of small datasets while minimizing costs.\nThey leveraged generative models to create an automatic data generation pipeline.", "questions": "- I wonder the impact/synergy of the proposed method for pretrained imagenet models (like ViT), compared to the results in Table 1. Previous methods for data augmentation also explains their superiority under finetuning protocol.\n\n- I guess the linear-probing or finetuning of CLIP will show comparable results for Table 2.\n", "limitations": "No.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Dataset expansion is interesting and useful research topic for small-scale domain.\n- The proposed methodology is simple and easy to utilize. \n- Extensive experiments demonstrate the proposed method helps performance improvement in small benchmark as well as model generalization.", "weaknesses": "- My major concern is the novelty of the task. I agree that new sample generation can help classifier to be more robust. However, this task (dataset expansion) is not a novel task to me, and I don't think the experimental results are surprising or interesting. Recent works [A,B,C,D,E] for various computer vision tasks have explored generative modeling to solve the limited labeled data.\n\n\n  - [A] W. Wu et al., \"DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models\" \n  - [B] S. Azizi et al, \"Synthetic Data from Diffusion Models Improves ImageNet Classification\"\n  - [C] G. Gu et al., \"CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion\" \n  - [D] Q. Kong et al., \"Active Generative Adversarial Network for Image Classification\"\n  - [E] V. Sandfort et al., \"Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks\"\n\n- Although authors explain expansion efficiency in Fig.4 and Sec.5.1, generation time of diffusion model and conversion speed using GIF in training may be slow. So, I wonder the training efficiency in terms of cost and time.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the comments, particularly for recognizing that the task is useful and the proposed method is simple and easy to use. We address the concerns as follows.\n\n--- \n \n>**Q1. Concern about the novelty of our work**\n \nAlthough expanding datasets is not a completely new concept, our work still presents valuable contributions to the community in the following aspects:\n\n- **Task importance**: Automatic dataset expansion, especially for small-data scenarios, bears great significance. While the concept might not be brand-new, our work provides a unique perspective on formally defining and tackling this task. The proposed dataset expansion can substantially reduce the time and costs involved in human data collection/annotation (cf. Section 5.2, Lines 343-350), and significantly enhances the model performance in small-data scenarios (cf. Table 1 in Section 5.1). The importance of this task has been highly recognized by other reviewers, such as Reviewer 1CUd \"*the task is meaningful*\", Reviewer UbXT \"*the dataset expansion task could contribute to the development of academia and industry*\" and Reviewer 6GPi \"*the work has the potential to attract wide attention*\".\n\n- **Method novelty**: Although using generative models to create training data is not a new idea, our method introduces a distinct design. The crux of our innovation lies in the concept of guided imagination (cf. Section 3.1), coupled with two critical expansion criteria (cf. Section 3.2). These insights are underpinned by both empirical observations and theoretical analysis (cf. Section 3, Section 5.2, Appendix B, Theorem 4.1). Based on these criteria, our approach can guide generative models to create informative new samples with novel content and correct class labels for expanding datasets. While the mentioned methods [A-E] also introduce new training data, they cannot ensure the synthesized data bring sufficient new information and accurate labels for the target small datasets. Moreover, training GANs from scratch, especially with very limited data, often fails to converge or produce meaningful results, making the method like [D,E] less effective in small-data scenarios. As such, our method emerges as a more effective way to expand small datasets. The contribution of our method has been highly recognized by Reviewer 1CUd \"*the proposed framework is intuitive, and the criteria are well-motivated*\", and Reviewer J4xT \"This paper presents a promising solution\".\n\nWe hope that this clarification can illustrate the unique contributions of our work and address the concern adequately.\n\n\n[A] DiffuMask. Arxiv, 2023-03-21  \n[B] Synthetic Data from Diffusion Models Improves ImageNet Classification. Arxiv, 2023-04-17  \n[C] CompoDiff. Arxiv, 2023-03-21  \n[D] Active Generative Adversarial Network for Image Classification. In AAAI, 2019  \n[E] Data augmentation using CycleGAN. Scientific Reports, 2019  \n\n---\n\n>**Q2. Question about the cost and time** \n \nThanks for the comment. Please refer to [General Response G1](https://openreview.net/forum?id=82HeVCqsfh&noteId=e7B6qvrwYr) for the detailed response.\n\n--- \n\n>**Q3. The effectiveness in model fine-tuning**\n \nThanks for mentioning the potential synergy between dataset expansion and model fine-tuning. They indeed complement each other. As discussed in Appendix F.4.3 (cf. Table 15), our dataset expansion also improves the fine-tuning performance of pre-trained CLIP VIT-B/32 on CIFAR100-S. For clarity, we provide the related results below.\n\n| Methods | CIFAR100-S |\n| ---------- |:---------------:| \n| Training from scratch | 35.0 | \n| Zero-shot CLIP VIT-B/32 | 41.6 (+6.6) | \n| Fine-tuning CLIP VIT-B/32 on original dataset | 75.2 (+40.2) | \n| Fine-tuning CLIP VIT-B/32 on 5x-expanded dataset by GIF-SD | 79.4 (+44.4) | \n\nDuring the rebuttal, we further find that our dataset expansion also boosts the fine-tuning performance of models like ImageNet pre-trained ResNet-50 on Cars, as evident in the following table.\n\n| Methods | Cars | \n| ---------- |:---------------:| \n| Fine-tuning ImageNet pre-trained ResNet-50 on original dataset | 87.6 | \n| Fine-tuning ImageNet pre-trained ResNet-50 on 5x-expanded dataset by GIF-SD | 88.9 | \n \n\nIn summary, our dataset expansion works effectively with model fine-tuning across various datasets and architectures, which further verifies our effectiveness.\n\n---\n\n>**Q4. I guess the finetuning of CLIP will show comparable results for Table 2** \n \nTable 2 aims to fairly compare dataset expansion methods in the out-of-distribution (OOD) setting, so we train all models from scratch. We agree with the reviewer that fine-tuning a pre-trained CLIP, benefiting from its extensive pre-training on large datasets, would undoubtedly yield superior OOD performance. However, as discussed in the response to **the above Q3**, dataset expansion and model transfer are distinct yet complementary paradigms. They are not in competition, but rather can be synergistically combined. Therefore, our method can also enhance the OOD performance of fine-tuned CLIP models on CIFAR100-C, as demonstrated in the following table. Thanks again for the insightful feedback, and we will incorporate the new result into the revised manuscript.\n \n| CIFAR100-S dataset | OOD Accuracy on CIFAR100-C | \n| ---------- |:---------------:| \n| Training from scratch on original dataset | 23.6 | \n| Training from scratch on 5x-expanded dataset by GIF-SD | 43.3 (+19.7)| \n| CLIP fine-tuning on original dataset | 55.4 (+31.8) | \n| CLIP fine-tuning on 5x-expanded dataset by GIF-SD | 61.4 (+37.8) | \n\n---\n\nWe deeply value the reviewer's dedication to assessing our paper. Based on the constructive feedback, we have endeavored to address every concern and have incorporated new results and discussions to that end. We kindly ask the reviewer to re-evaluate our paper in light of these enhancements, which enable our work to better align with the conference's standards. Should additional concerns arise, we are glad to resolve them."}, {"review_id": "gQTk8ibx8D", "reviewer": "Reviewer_1CUd", "summary": "This paper describes a new task called dataset expansion, which aims at expanding the size of small datasets to boost the performance of data-driven AI models on tasks like object classification. The paper proposes a framework called Guided Imagination Framework (GIF) to achieve it by utilizing pre-trained large-scale generative models to synthesize new informative samples according to the images in the dataset. Specifically, the method perturbs the latent feature of the examplar image in the dataset and designs two criteria, i.e., *class-maintained information boosting* and *sample diversity promotion*, to optimize the noise added to the latent feature of the exemplar image. The paper conducts extensive experiments and verifies the effectiveness of the proposed method in boosting the classification performance on small datasets of natural images and medical images.", "questions": "1. I have some questions about GIF-SD. Does GIF-SD first sample several Gaussian noises $z$ and then perform the inverse diffusion process with the text prompt condition to obtain latent features $f$ which are perturbed by Eq.1 to obtain $f^{'}$ and then fed into the image decoder? In that case, since $f^{'}$ is not the direct perturbation of the feature of the real image sample in the dataset, I think GIF-SD is quite different from GIF-DALLE and GIF-MAE and may fail to guarantee the resemblance to the image samples in the dataset. Or does GIF-SD use a strategy similar to the real guidance (RG) strategy in [22]?\n\n2. The compared methods include DALL-E2, SD, and MAE, but how exactly these methods are used to generate samples are not clear. Have the authors tried their best to optimize the way these models generate samples, e.g., as does in [22] using language enhancement (LE) and CLIP Filter (CF)? This is important since it can faithfully reflect the proposed method's superiority.\n\n3. I suggest the authors report the domain gap between the real image samples and the synthetic image samples, e.g., using FID or LPIPS. And the analysis of how the domain gap affects the performance is favored.", "limitations": "The authors are encouraged to discuss the limitations of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-organized and may be one of the first research to explore the effectiveness of synthetic images in boosting classification performance. The task is meaningful, the proposed framework is intuitive, and the criteria to maintain the class information and to encourage diversity are well-motivated. The experiments are systematically conducted and show that the proposed framework helps improve performance significantly on various backbones. The ablation study in the appendix shows the effectiveness of the proposed two criteria.", "weaknesses": "1. Though the authors give two reasons for not using CLIP for classifying the target dataset: (1) the transferability: Line#298-300; (2) GIF can benefit various model architectures. However, as one of the state-of-the-art backbones for classification, the proposed method's effectiveness in boosting the performance of CLIP on natural images needs to be justified.\n2.  I think the comparison with the few-shot setting in [22] is required to prove the superiority of the method, as the few-shot setting in [22] shares similarities with the paper's task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful feedback, particularly for recognizing the significance of the studied task and our proposed method. We next address the concerns as follows.\n\n--- \n \n>**Q1. Effectiveness in boosting CLIP fine-tuning**\n \nThanks for pointing this out. In fact, we have evaluated the effectiveness of our method in boosting CLIP fine-tuning. As shown in Table 15 (cf. Appendix F.4.3), our method significantly enhances the fine-tuning performance of CLIP VIT-B/32 on CIFAR100-S, elevating the accuracy from 75.2% to 79.4%. \n\nDuring the rebuttal, we further find that dataset expansion is also beneficial to the out-of-distribution robustness of the fine-tuned CLIP model, as shown in the following table.\n\n| CIFAR100-S dataset | Accuracy on CIFAR100-C | \n| ---------- |:---------------:| \n| Training from scratch on original dataset | 23.6 | \n| CLIP fine-tuning on original dataset | 55.4 (+31.8) | \n| CLIP fine-tuning on 5x-expanded dataset by GIF-SD | 61.4 (+37.8) | \n\nThese results further underscore the potency of our method. \n\n--- \n\n>**Q2. Does GIF-SD use a strategy similar to the real guidance (RG) strategy in [22]?** \n\nYes, GIF-SD is implemented with a strategy similar to the real guidance (RG) strategy in [22], which has been detailed in Appendix D.2, including the pseudo-code and other implementation details. To offer further clarity, we summarize the overall pipeline here. GIF-SD first conducts text-guided latent diffusion based on the latent feature of the seed data as the starting point (instead of random noise). After text-guided diffusion, GIF-SD conducts guided imagination based on Eqs. (1-2) and the image decoder. This ensures that the generated samples are class-maintained (cf. visualization results in Appendix G) and bring sufficient new information for boosting model performance (cf. Table 1 in Section 5.1). \n\n---\n\n>**Q3. Comparison with the few-shot setting in [22]？** \n \nFollowing the constructive suggestion, we adopt the advanced few-shot strategies in [22] (i.e., language enhancement (LE) and CLIP Filter (CF)) to expand the CIFAR100-S dataset based on Stable Diffusion (SD) and real guidance (RG). As shown in the following table, SD combined with these strategies [22] is still noticeably inferior to our GIF-SD for both training from scratch and CLIP tuning. This further demonstrates the superiority of our method. \n\n| CIFAR100-S | Training from scratch | CLIP fine-tuning | \n| ---------- |:---------------:| :---------------:| \n| Original dataset | 35.0 | 75.2 |\n| 5x-expanded dataset by SD+LE+CF [22] | 55.1 (+20.1)| 77.0 (+1.8)\n| 5x-expanded dataset by GIF-SD (ours) | 61.1 (+26.1)| 79.4 (+4.2) |\n\n---\n\n>**Q4. Implementation of DALL-E2,SD and MAE baselines** \n \nThese baselines were set up identically to their GIF counterparts for ablated evaluation, with the only exception being the absence of the guided imagination optimization (cf. Section 4). To ensure a consistent comparison, we did not leverage other strategies, like LE and CF [22], for both the baselines and our methods. \n\nWe agree with the reviewer that integrating these strategies might potentially enhance the quality of data generation. Nevertheless, as shown in the table responding to **the above Q3**, SD using these strategies still performs worse than our GIF-SD, which further verifies our superiority. \n\n---\n\n>**Q5. Analyzing the relations between the domain gap and model performance** \n \nThanks for the insightful suggestion. In response, we further compute the Fréchet Inception Distance (FID) between the synthetic data generated by different methods and the original data of CIFAR100-S. The results are summarized in the table below:\n\n| Datasets | FID | Accuracy |\n| ---------- |:---------:| :--------:| \n| Original CIFAR100-S dataset | - | 35.0 |\n| 5x-expanded dataset by RandAugment | 24.3 | 46.7 |\n| 5x-expanded dataset by Cutout | 104.7 | 44.3 |\n| 5x-expanded dataset by Gridmask | 104.8 | 48.2 |\n| 5x-expanded dataset by GIF-MAE | 72.3 | 52.7 |\n| 5x-expanded dataset by GIF-DALLE | 39.5 | 54.5 | \n| 5x-expanded dataset by GIF-SD | 81.7 | 61.1 |\n \nInterestingly, while one might initially assume that a lower FID implies better quality for the expanded data, the actual performance does not consistently follow this notion. For instance, even though GIF-SD has a worse FID than GIF-DALLE, it achieves better performance. Likewise, despite having nearly identical FIDs, Cutout and Gridmask lead to different performance. These results suggest that the effectiveness of dataset expansion methods depends on how much additional information and class consistency the generated data can provide to the original dataset, rather than the distribution similarity between those samples and the original data.\n\nIn summary, we are grateful for the thought-provoking question. We believe this discussion will spark further research into the relationship between expansion effectiveness and data fidelity (as measured by metrics like FID), potentially guiding the development of even more effective dataset expansion techniques in the future.\n\n---\n\n>**Q6. The authors are encouraged to discuss the limitations of the paper** \n \nThanks for the valuable suggestion. We will enrich our supplementary with a limitation discussion. Due to the word constraints of this rebuttal, we kindly direct the reviewer to our detailed response to [Reviewer 6GPi's Q7](https://openreview.net/forum?id=82HeVCqsfh&noteId=CerYJBcKIX), where we delve into this discussion.\n\n---\n\nThanks again for the thoughtful feedback. We have diligently addressed each concern in our rebuttal, incorporating fresh experimental results and deeper discussions. The reviewer's insights have been pivotal in enriching our work, which aligns more closely with the conference's standards. We hope our updates address the concerns and might encourage a positive reconsideration of the initial assessment within the review panel."}, {"review_id": "uXHqSd6tiv", "reviewer": "Reviewer_UbXT", "summary": "This paper proposes to expand a small dataset by automatically creating new labeled samples with pre-trained generative models, such as DALL-E2 and Stable Diffusion. The proposed framework, namely Guided Imagination Framework, contains two key parts, i.e., class-maintained information boosting and sample diversity promotion. The experiments of GIF-SD obtain improvements on multiple datasets. ", "questions": "Please see the weaknesses", "limitations": "Please see the weaknesses", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe key idea is interesting and dataset expansion task could contribute to the development of academia and industry.\n2.\tHigher accuracy can be achieved over multiple image datasets. \n", "weaknesses": "The proposed solution to dataset expansion is too easy. The class-maintaining informativeness strategy aims to generate perturbed features with seed sample class consistency. However, how to evaluate the quality and accuracy of generated samples with perturbed features?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the effort in reviewing our paper. We are glad that the interesting idea and the important task are appreciated. We address the concerns point by point as follows.\n\n---\n\n>**Q1. The proposed solution to dataset expansion is too easy** \n \nThanks for the feedback on the simplicity of our solution. We believe that simplicity, when paired with effectiveness, is a strength, not a limitation.\n\n- **Simplicity as a virtue**: Simple solutions are generally more intuitive to understand, so they are usually more adaptable and more likely to be used by the community. Although our method might seem straightforward, achieving such simplicity with effectiveness requires substantial underlying effort and insights, as depicted in Section 3, Section 5.2, Appendix B, Appendix F and Theorem 4.1. These empirical observations and theoretical foundations not only justify our approach but also provide invaluable insights for future research. \n\n- **Significance of automatic dataset expansion**: Our straightforward method holds significant value. It showcases notable advancements in various small-data scenarios, such as in-distribution generalization (Table 1), out-of-distribution robustness (Table 2), and long-tailed problems (Table 22 in Appendix F.7). More importantly, it can drastically reduce the cost and time associated with human data collection/annotation for expanding small datasets (cf. the discussion in Section 5.2, Lines 343-350).\n\nIn summary, while our method might appear straightforward at first glance, the depth of thought, experiments, and insights behind it attest to its uniqueness and efficacy. Hence, our method offers an important contribution to addressing the small-data challenges. Its value was also recognized by Reviewer 7xaH \"*The proposed methodology is simple and easy to utilize*\", Reviewer 1CUd \"*the proposed framework is intuitive, and the criteria are well-motivated*\", and Reviewer J4xT \"This paper presents a promising solution\".\n\n---\n\n>**Q2. The class-maintaining informativeness strategy aims to generate perturbed features with seed sample class consistency. However, how to evaluate the quality and accuracy of generated samples with perturbed features?** \n \nThe goal of our method is to create informative new samples that, when used in conjunction with the original data, improve the model performance in small-data scenarios. Directly assessing the perturbed features might not provide an accurate reflection of how beneficial the generated samples are when it comes to training a model. Therefore, a more effective way to judge the quality of the generated samples is to directly observe their enhancement in the final model performance. In light of this, this work evaluated various dataset expansion methods by directly measuring the performance of models trained on their expanded datasets. \n\nIn addition, we would like to highlight that the class-maintaining property of our method is pivotal. This consistency is reinforced by the objective function outlined in Lines 207-208, ensuring that perturbed features remain aligned with the seed sample class. By guaranteeing this class consistency, our method can bring novel information to model training without the need to concern class misalignment of the generated samples. The ablation results of GIF-DALLE (cf. Table 17 in Appendix F.5.1) further demonstrate the effectiveness and importance of our class-maintaining strategy. To clarify, we provide the related result below (please refer to Appendix F.5.1 for more detailed analyses of the result). \n\n| Methods | Class-maintained strategy | Diversity promotion strategy| Accuracy of 5x-expanded CIFAR100-S |\n| ---------- |:---------------:|:---------------:|:---------------:| \n| GIF-DALLE | ✖ | ✖ | 52.1 |\n| GIF-DALLE | ✔ | ✖ | 53.1 |\n| GIF-DALLE | ✖ | ✔ | 51.8 |\n| GIF-DALLE | ✔ | ✔ | 54.5 | \n\n--- \n\nWe genuinely appreciate the effort the reviewer has dedicated to reviewing our work. We have diligently addressed the concerns in the rebuttal, emphasizing the novel contributions and the potential value of our work. We respectfully request the reviewer to reconsider our paper with these clarifications in mind. A positive re-evaluation would be immensely beneficial to our efforts. We remain dedicated to addressing any further questions."}, {"review_id": "XvWkbJPo3N", "reviewer": "Reviewer_J4xT", "summary": "This paper introduces a novel task that addresses the expansion of a limited dataset into a larger one by utilizing a Guided Imagination Framework. The proposed approach harnesses latent features to generate new data instances while placing emphasis on preserving class-specific information and enhancing sample diversity. By employing these criteria, the generated data effectively facilitates the improvement of the model's learning process. The efficacy of the proposed method is demonstrated through rigorous validation on multiple datasets, showcasing its superior performance.", "questions": "- It would be valuable to receive further insights into the time required to complete the generating-training procedure. Understanding the time constraints associated with the proposed method is essential for assessing its practicality and scalability. Insights into the time requirements would enable a more comprehensive evaluation of the proposed approach in terms of efficiency and resource allocation.\n- A comprehensive discussion on the training process and underlying structures of the listed methods, including GIF-MAE and GIF-SD, would greatly enhance the readers' understanding of these approaches. Detailed explanations of the methodologies employed, such as the specific techniques utilized for data generation and the architectures of the models, would provide valuable insights into the novelty and effectiveness of these methods. Expanding on these aspects would facilitate a more thorough comparative analysis of the proposed approach against other techniques.\n- Gaining clarity on whether pretrained weights were utilized in all of the proposed methods would be beneficial for assessing the fairness of the comparison. Understanding the extent to which prior knowledge is leveraged across different approaches is crucial for interpreting and contextualizing the results. Specifically, differentiating between methods that start from scratch without any pretrained weights and those that employ pretrained models, such as CLIP, would provide a more accurate understanding of the experimental setup and potential biases.\n- To further validate the proposed approach, it would be beneficial to compare it with other methods that also generate synthesized data, as referenced in the provided papers (https://arxiv.org/pdf/2301.06043.pdf, https://ieeexplore.ieee.org/document/8629301, https://proceedings.mlr.press/v156/bao21a/bao21a.pdf). Conducting such comparisons would offer valuable insights into the relative strengths and weaknesses of the proposed method and provide a broader perspective on its performance compared to existing state-of-the-art techniques.", "limitations": "- The generation method employed to expand dataset volumes is a time-consuming process that necessitates various preprocessing steps. Furthermore, training with a substantially larger amount of data (at least 5 times more) adds to the complexity. This undertaking should be approached with careful consideration, acknowledging the resource and time requirements involved.\n\n- In order to provide a comprehensive evaluation, it would be advantageous for the authors to conduct fair comparisons with existing models and methods, taking into account factors such as memory footprint. Considering these aspects would enable a more thorough assessment of the proposed approach in relation to its counterparts, providing a clearer understanding of its advantages and limitations.\n\n- Given that the proposed method involves the generation of new data, it is highly recommended that an ethical review be conducted to ensure compliance with ethical guidelines and considerations. The potential impact and implications of generating synthetic data should be carefully evaluated to ensure that it aligns with ethical principles, privacy regulations, and legal requirements. Such a review would demonstrate a responsible approach towards the development and application of the proposed method.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper presents a promising solution to the challenge of generating reliable supplementary training data, distinguishing itself from conventional data augmentation methods by generating unique content distinct from the existing dataset.\n- The paper introduces Guided Imagination Framework that leverages prior knowledge of latent features to generate additional data while maintaining alignment with class labels, ensuring consistency throughout the generated samples.\n- Experimental results show the remarkable effectiveness of this proposed approach in significantly enhancing the accuracy of image classification tasks, substantiating its superiority over alternative methods.", "weaknesses": "- A major concern arises from the results presented in Table 1, particularly regarding the Stanford Cars dataset (https://ai.stanford.edu/~jkrause/papers/fgvc13.pdf). The comparison reveals that certain alternative methods achieve superior results (>90) without the need for additional data, as demonstrated in (https://arxiv.org/pdf/2102.05918.pdf). This raises skepticism regarding the efficacy of the augmented data in genuinely improving the model's performance, as the reported results fall below the mentioned benchmark (<80).\n\n- Another point of consideration is the authors' exclusive focus on employing ResNet-50 as the sole backbone architecture for their experiments. This singular choice could introduce bias, as it remains plausible that only ResNet performs well within the proposed framework. To establish the robustness and generalizability of the proposed method, it would be valuable for the authors to explore and validate its performance across multiple alternative backbone architectures, thereby mitigating potential biases.\n\n- It is important to acknowledge that the proposed method significantly increases the training time due to the inclusion of a generative model and the utilization of substantially larger amounts of data (at least 5 times more) for training the classification model. This extensive time requirement may pose considerable challenges, particularly in classification tasks where efficiency is a crucial aspect to consider and optimize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comprehensive and constructive comments, particularly for recognizing that our solution is promising and remarkably effective. We address all the concerns as follows.\n\n---\n \n>**Q1. Concern on the results in Table 1, particularly regarding Cars** \n\nThe experimental results are indeed reasonable based on the following reasons:\n\n- **Different training methodology**: Our focus is to expand small-scale datasets, and as such, **all the models are trained from scratch for evaluation**. This allows us to fairly compare the effectiveness of various expansion methods, without biases from pre-trained models. Please note that our task is orthogonal and complementary to the model transfer task in Align (https://arxiv.org/pdf/2102.05918.pdf), where the transfer performance inevitably surpasses training from scratch due to its extensive pre-training on large datasets. \n\n- **Additional fine-tuning results**: To address this concern, we further fine-tune an ImageNet pre-trained ResNet50 on Cars. Using the original dataset, the fine-tuned ResNet-50 achieves a performance of 87.6. Remarkably, with the expanded dataset by GIF-SD, the performance further increases to 88.9, highlighting the effectiveness of our method in enhancing model fine-tuning. The additional results further substantiate the reasonableness of this paper.\n \n---\n\n>**Q2. Concern on the exclusive focus on using ResNet-50** \n\nThanks for the comment, but it seems that there might have been a misunderstanding. Our study did indeed consider the robustness and generalizability of our method across various model architectures, not solely focusing on ResNet-50. Specifically, Table 3 (cf. Section 5.1) and Table 13 (cf. Appendix F.3) demonstrate our method's performance on various architectures, such as ResNext, WideResNet, and MobileNet. These results highlight an important advantage of our method: the expanded dataset can be used with various network structures without needing to regenerate, thereby affirming its broad applicability.\n\n--- \n\n>**Q3. Question about the time and costs** \n \nThanks for the comment. Please refer to [General Response G1](https://openreview.net/forum?id=82HeVCqsfh&noteId=e7B6qvrwYr) for the detailed response.\n\n---\n\n>**Q4. Details on GIF-MAE and GIF-SD** \n\nThanks for the feedback. Due to the page limitations, we had to put the method details, implementation details, and pseudo-code of all proposed methods in Appendix D. We understand that the supplementary is not mandatory reading, but it is intended to provide a comprehensive understanding of what the reviewer suggests. This concern might be addressed by delving into this appendix. If the reviewer has any further questions after referring to these details, we would be more than happy to provide additional clarification. \n\nFurthermore, according to the NeurIPS guidelines, we are permitted to add one more page to the camera-ready. In such a case, we will move more crucial method details to the main text to enhance its clarity and comprehension. We hope these measures can sufficiently address the concern.\n\n---\n\n>**Q5. Clarification on whether pre-trained weights were utilized during model training** \n\nDuring the model training phase, the models are trained from scratch for all dataset expansion methods (**including ours**) in all tables (including Table 4), ensuring no pre-trained weights are used for maintaining a fair comparison. The only exceptions are **CLIP-related baselines** (in Tables 1,4, and 15), where the pre-trained checkpoint of CLIP was used. We believe this clarification should offer a more accurate understanding of our experimental setup.\n\n---\n\n>**Q6. Further comparisons to advanced generative methods** \n\nThanks for suggesting a comparison with the mentioned generative methods. We recognize its importance, but there are two primary challenges in executing this comparison: (1) The mentioned GAN-based generative methods have not made their model checkpoints publicly available. (2) Training GANs from scratch, particularly when data is limited, frequently results in non-convergence or yields unmeaningful outcomes, making it unsuitable for small dataset expansion.\n\nMeanwhile, recent research [A] has indicated that diffusion models, thanks to iterative denoising, outperform GANs in image generation. As such, we opted for diffusion models in our work, as their checkpoints are readily available. \n\nEven so, we agree with the value of comparisons with more advanced generative methods. Therefore, we further compare our method with a recent ICLR work [B], as recommended by Reviewer 1CUd. Specifically, the method [B] proposes strategies like language enhancement (LE) and CLIP filters (CF) to enhance generative models for generating training data. For a fair comparison, we use this method with SD and our GIF-SD to expand the CIFAR100-S dataset by 5x. The results, as shown in the following table, further affirm the superiority of our method.\n\n| CIFAR100-S | Accuracy | \n| ---------- |:---------:| \n| Original | 35.0 | \n| 5x-expanded by SD | 52.9 (+17.9)| \n| 5x-expanded by SD + method [B] | 56.0 (+21.0)| \n| 5x-expanded by GIF-SD (ours) | 61.1 (+26.1)| \n \n[A] Diffusion models beat gans on image synthesis. In NeurIPS, 2021  \n[B] Is synthetic data from generative models ready for image recognition? In ICLR, 2023  \n\n---\n\n>**Q7. Question on potential influences of synthetic data** \n\nWe appreciate the question on the ethical implications. Please refer to [General Response G2](https://openreview.net/forum?id=82HeVCqsfh&noteId=e7B6qvrwYr) for the detailed response.\n\n---\n\nThanks for the valuable feedback. Through our rebuttal, we have clarified ambiguities, provided additional results, and deepened our discussions to address all concerns. We believe our revised work aligns more closely with the conference's standards. A positive reconsideration of the initial assessment would be greatly beneficial to our efforts. Should any further questions, we are glad to answer them. "}, {"review_id": "otWhLIoYvj", "reviewer": "Reviewer_6GPi", "summary": "The paper proposes an image generation framework for expanding small-scale datasets. The proposed Guided Imagination Framework (GIF) leverages large language-vision models (i.e., CLIP) and generative models (i.e., DALL-E2, Stable Diffusion, and MAE) based on two criteria that help to generate informative new images with (i) class-consistent semantics but (ii) higher content diversity. Experimental analyses and ablation studies on several natural and medical image datasets show that GIF is effective and efficient in boosting the accuracy and generalization of the models trained on artificially expanded datasets.", "questions": "- How are the prior generative models (DALL-E2, SD, and MAE) trained as baselines? Are they configured exactly the same way as the proposed methods but simply without the GIF components?\n- While fine-tuning CLIP, how much effort was put into to its proper optimization, for example, was something similar to robust fine-tuning of CLIP as presented in [A] followed?\n- How much more is there to be gained if GIF is applied to large-scale datasets? Not asking for more experiments here but looking for some intuition/speculation/expert opinion on this.\n- Would it be possible to expand the literature review with a few more relevant papers on text-driven dataset generation?\n- Would it be possible to reorganize the paper to include some of the technical details about the method/algorithm in the main paper?\n- References must be updated to include only those cited in the main paper.", "limitations": "There is no discussion on limitations and broader impact of the work. Error analysis is a bit superficial. A more detailed error analysis would be helpful. Is there a risk of mode collapse? Is there a risk of exacerbating biases while generating images? Cutting down the required human effort and cost of data collection can be highlighted as broader impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Significance\nCombining large language-and-vision models with generative models such as stable diffusion to generate synthetic images has become a popular topic that can support various downstream tasks. To this end, the paper has the potential to attract wide attention.\n\n### Originality\nThe proposed guided imagination framework for dataset expansion combines CLIP with several SOTA generative models in a unique way to optimize the variation over the latent space so that they can generate images with more diverse content but still within the same semantic concept/category.\n\n### Clarity\nThe two core criteria of class-maintained informativeness boosting and sample diversity promotion are defined and illustrated with clear examples. The paper is written well in general although there is just too much material that could not be fit into the main paper and hence moved to supplementary material (but this leaves holes in the paper).\n\n### Quality\nThe experimental evaluations and ablation studies are quite extensive and informative. Main findings are summarized adequately in the main text and detailed discussion is presented in the supplementary material due to page constraints. The contribution of the proposed framework is supported with quantitative evidence from various perspectives.\n", "weaknesses": "### Experiments\nSome of the important technical and algorithmic details are not included in the paper, which leaves the reader a bit hanging in the air and pushes them to read the supplementary material (which becomes rather like a \"mandatory material\"). Not sure if there is an easy way to fix this but it would be good to re-organize the material to remedy this concern.\n\nHow are the baseline DALL-E2, SD, and MAE implemented? Are they configured the same way as the proposed methods but without the actual GIF components (i.e., ablated)?\n\nWhile fine-tuning CLIP, how much effort was put into to its proper optimization, for example, was something similar to robust fine-tuning of CLIP as presented in [A] followed?\n\nAlso it would be interesting to see how much more is there to be gained if GIF is applied to large-scale datasets?\n\n### Literature Review & Citations\nLiterature review is too brief. It would be good to squeeze in a bit more of the other relevant text-driven dataset generation papers (e.g., [33, 44, 48, 69, B-D]). At the end of the day, GIF-SD is also driven by text prompts. To this end, some of the redundancy across sections (e.g., Sections 1, 3, and 4) with repeated statements can be eliminated.\n\nThe references contain papers that are not cited in the main paper. Actually, only 43 (see the list below) of the 98 papers are referenced in the main paper and the others are probably mentioned (didn't fully check) in the supplementary material. The bibliography must include only the references that are cited in the main paper. Supplementary material can have its own references.\n\nCitations that appear in the main paper: 2, 6, 8, 9, 10, 11, 12, 14, 15, 20, 22, 23, 24, 29, 30, 35, 36, 37, 45, 46, 47, 49, 50, 52, 55, 59, 60, 61, 62, 63, 66, 68, 72, 73, 74, 76, 77, 81, 82, 86, 89, 93, 97\n\nThere are some additional recent work that might be good to include in the paper or in the supplementary [B-E].\n\n### Minor Concerns\n76: Better to include references to Cutout, GridMask, RandAugment, Cars and DTD here because they are mentioned for the first time in the paper.\n\n208: s' should be f'\n\n328: conducts <== conduct\n\n329: There is no Figure 10(e) in the main paper.\n\n### Suggested References\n[A] Wortsman, Mitchell, et al. \"Robust fine-tuning of zero-shot models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[B] Sarıyıldız, Mert Bülent, et al. \"Fake it till you make it: Learning transferable representations from synthetic ImageNet clones.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[C] Zhou, Yongchao, Hshmat Sahak, and Jimmy Ba. \"Training on Thin Air: Improve Image Classification with Generated Data.\" arXiv preprint arXiv:2305.15316 (2023).\n\n[D] Tian, Yonglong, et al. \"StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners.\" arXiv preprint arXiv:2306.00984 (2023).\n\n[E] Azizi, Shekoofeh, et al. \"Synthetic data from diffusion models improves imagenet classification.\" arXiv preprint arXiv:2304.08466 (2023).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the highly constructive comments. We are glad to see that the task significance and the method originality are appreciated. We answer all questions point by point as follows.\n\n---\n \n \n>**Q1. Would it be possible to put more technical details in the main paper?** \n\nWe highly appreciate the constructive suggestion on the organization of technical details. Due to page constraints, some method details were placed in the supplementary material (cf. Appendix D). However, per the NeurIPS guidelines (https://nips.cc/Conferences/2023/CallForPapers), we are allowed an additional page in the camera-ready version if accepted. Given this opportunity, we plan to incorporate more essential method details into the main text to enhance its clarity and accessibility. We hope this adjustment can address the concern.\n\n---\n\n>**Q2. Are the baselines (DALL-E2, SD, and MAE) configured the same way as the proposed methods but without the actual GIF components?** \n\n\nYes, these baselines were configured identically to their GIF counterparts for ablated evaluation, with the only exception being the absence of the guided imagination optimization (cf. Section 4). Without the crucial GIF components, these baselines cannot ensure that the synthesized data bring sufficient new information, content diversity, and accurate labels for small-scale dataset expansion. This is evident in Table 1 (cf. Ssction 5.1), where their performance is markedly inferior compared to our GIF methods.\n\n--- \n\n>**Q3. The implementation of CLIP fine-tuning** \n\nIn this work, we did not use any sophisticated fine-tuning strategies like the robust fine-tuning of CLIP as presented in [A]. Instead, we kept it straightforward and used cross-entropy for standard fine-tuning. It is worth noting that this cross-entropy training is consistent with other training-from-scratch baselines of dataset expansion methods. By doing so, we ensure a fair comparison across different methods and baselines.\n\n---\n\n>**Q4. Can GIF be applied to large-scale datasets?** \n\nThanks for raising this question. In fact, we have discussed this in Appendix F.8, where our GIF method is adaptable to expanding larger-scale datasets, e.g., the original CIFAR dataset. The results in Table 23 (cf. Appendix F.8) further demonstrate the effectiveness of our approach.\n\n--- \n\n>**Q5. Literature review is too brief. Please review more text-driven dataset generation methods and the studies [B-E]**\n \nThank you for the suggestions. Due to page limitations, we put a detailed review of related work in Appendix A. Should our work be accepted, the extra page granted for the camera-ready will allow us to expand the literature review in the main text. We will then incorporate a more comprehensive discussion on text-driven dataset generation papers and include the related studies [B-E] mentioned by the reviewer.\n \n\n--- \n\n>**Q6. References must be updated to include only those cited in the main paper** \n\nThanks for pointing this out. We will revise the reference in the main paper to include only the ones cited, and ensure the supplementary has its own reference list.\n\n--- \n\n>**Q7. Discussion on limitations and broader impact** \n\nIn light of the suggestion, we will enrich our supplementary with the following discussions:\n\n**Limitation**: \n\n- **(1) Performance of generated samples**: The expanded samples are still less informative than real samples. For example, a ResNet-50 trained from scratch on our 5x-expanded CIFAR100-Subset achieves an accuracy of 61.1%, which lags behind the 71.0% accuracy on the original CIFAR100. This gap signals the potential for advancing algorithmic dataset expansion. We expect that this pioneering work can inspire more studies to explore dataset expansion so that it can even outperform a human-collected dataset of the same size. \n\n- **(2) Quality of generated samples**: Some samples might have noise, as exemplified in Figure 5b. Despite seeming less realistic, those samples are created following our guidance (e.g., class-maintained informativeness boosting). This ensures the class consistency of these samples, mitigating potential negative effects on model training. Nonetheless, refining the expansion method to address these noisy cases can further enhance the effectiveness of dataset expansion.\n\n- **(3) Scope of work**. Our current focus is predominantly on image classification. Exploring the adaptability of our method to other tasks, such as object detection, is an intriguing next step.\n\n**Broader impact**. Our method can offer a notable reduction in the time and cost associated with manual data collection and annotation for dataset expansion, as discussed in Section 5.2 (cf. Lines 343-350). This can revolutionize how small datasets are expanded, making deep learning more accessible to scenarios with limited data availability (cf. Table 1). \n\n--- \n\n>**Q8. Other minor issues** \n\nThanks for noting these details. We appreciate the meticulous review. We will make the necessary corrections following the suggestions.\n\n---\n\nWe deeply appreciate the insightful feedback from the reviewer. Thanks to these constructive suggestions, our paper's quality has been greatly elevated to meet the conference's high standards. We kindly ask the reviewer to take into account the refinements and enhancements we have incorporated. The reviewer's endorsement is influential and could potentially influence other reviewers in recognizing our paper's merits. "}]

Author rebuttal:
**General Response**
---

We deeply appreciate all the reviewers for dedicating time and effort to reviewing our paper. Here, we first address the general question below, and subsequently, we will provide detailed responses to each reviewer's specific questions and comments. 

---

>**G1. Concern about the time and cost of dataset expansion (Reviewers 7xaH and J4xT)** 
 
The primary goal of dataset expansion is to mitigate the time and cost of human data collection/annotation and boost model performance in small-data scenarios. Please note that improving model performance on small-scale datasets inevitably incurs certain costs. For instance, in the context of transfer learning, the total cost—including time and resources for collecting and pre-training on large-scale datasets (such as large-scale medical image datasets)—could be significantly high. In this work, we introduce a complementary task paradigm to mitigate the human labor and financial costs associated with data collection, and also bolsters model performance. 

In this context, dataset expansion and model training are treated as two distinct phases, and we differentiate the analysis for each phase as detailed below.

**I. Dataset expansion**: As discussed in Section 5.2 (Lines 343-350), GIF-SD can expand an image by 5 times in just 33 seconds using a V100 GPU, so it can significantly save time and cost of dataset expansion compared to manual data collection/annotation. Specifically, manually annotating 10,000 images, according to Masterpiece Group (https://mpg-myanmar.com/annotation), would typically **take 2 weeks and cost around \$800**. In contrast, GIF-SD can generate the same volume of labeled data in a **mere 2 hours, costing roughly \$40 for renting 8 V100 GPUs**. Moreover, if higher efficiency is pursued with an acceptable performance drop, GIF-MAE can create 10,000 labeled data in **just 80 seconds, at a cost of about \$0.48 for renting 8 V100 GPUs**. Note that once the dataset is expanded, it can be directly utilized to train various models, removing the need for regeneration with each model and thereby further enhancing efficiency. 

| Methods | Expansion speed | Time (10,000 images) | Costs (10,000 images) |
| ---------- |:---------------:| :---------------:| :---------------:| 
| Human data collection | 120.96s per image | 2 weeks | $800 |
| GIF-MAE (ours) | 0.008s per image | 80 seconds | $0.48 |
| GIF-SD (ours) | 6.6s per image | 2 hours | $40 |

**II. Model training**: The training time varies based on the specific datasets. However, it is pivotal to note that all dataset expansion methods were compared based on the same expansion ratio, thus ensuring consistent training time/cost and fair comparisons. We acknowledge that training on an expanded dataset will inevitably take longer than training on the original dataset. However, as shown in Table 1 (cf. Section 5.1), the significant improvement in model performance (i.e., by 36.9% on average over six natural image datasets and by 13.5% on average over three medical datasets) makes the increased investment in training time worthwhile. More importantly, a detailed analysis presented in Appendix F.1.3 (cf. Table 10) shows that, even with the same training consumption (in terms of sample number × training epochs), our proposed method still proves advantageous. More specifically, as shown in the following table, training the model on the original CIFAR100-S dataset for 5x more epochs performs much worse than the model trained on our 5x-expanded dataset. This comparison further underscores the effectiveness of our method in achieving higher accuracy without inflating training costs.


| CIFAR100-S | Training epoch |    Consumption (data number x epoch)    | Accuracy |
| ---------- |:---------------:|  :---------------:  | :---------------:| 
| Training on original dataset | 100 | 1 million | 35.0 |
| Training on original dataset with RandAugment | 100 | 1 million |39.6 |
| Training on original dataset with RandAugment | 600 | 6 million| 51.1 |
| Training on 5x-expanded dataset by GIF-SD (ours) | 100 | 6 million| 61.1 |

To summarize, based on the comprehensive analysis, we firmly believe that the efficiency and cost-effectiveness of our method outweigh the concerns related to computational consumption.
 

---

>**G2. Question on potential influences of the synthetic data (Reviewer J4xT)** 

Ethical considerations, especially in AI research and data generation, are indeed paramount. Given the importance of this topic, we have opted to address it in the general response, even though it was specifically raised by Reviewer J4xT. In fact, our approach is constructed with care to avoid negative ethical implications, as evidenced in the following points:

- **Controlled generation**: In our approach, the generation of synthetic data is driven by our expansion guidances, which ensure that new data is derived directly and meaningfully from the original dataset. This controlled mechanism minimizes the risks of creating unrelated or potentially harmful images. 

- **No personal or sensitive data**: It is also worth noting that our method primarily focuses on public available datasets like CIFAR, Stanford Cars, and similar, which do not contain personal or sensitive information. As such, the risks related to privacy breaches or misrepresentations are substantially diminished. 

- **Ethical commitment**: While our research aims to showcase the technical capabilities of our method, we agree with the significance of upholding ethical standards in technology. In future applications, if applying our method to more sensitive domains, we commit to seek ethical evaluations to ensure responsible use. 

We hope these clarifications underscore our commitment to responsible AI and address the concern.

Meta-review: This paper initially received mixed ratings, and the authors submitted rebuttals. Reviewers read the rebuttals and further discussed with the authors. Most reviewers were satisfied by the authors' rebuttals and responses. As a result, all reviewers except one raised their ratings. The AC read the paper, the authors' rebuttals and the discussions between authors and reviewers. The AC agrees with the majority of the reviewers on their final ratings, and the decision is to accept.
Decision: Accept (poster)