Expanding Datasets With Guided Imagination
Authors: Yifan Zhang
Zhou Daquan
Bryan Hooi
Kai Wang
Jiashi Feng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Dataset Expansion
Guided Imagination

Abstract: The power of Deep Neural Networks (DNNs) depends heavily on the training data quantity, quality and diversity. However, in many real scenarios, it is costly and time-consuming to collect and annotate large-scale data. This has severely hindered the application of DNNs. To address this challenge, we explore a new task of dataset expansion, which seeks to automatically create new labeled samples to expand a small dataset. To this end, we present a Guided Imagination Framework (GIF)  that leverages the recently developed big generative models (e.g., DALL-E2) to ``imagine'' and create informative new data from seed data to expand small datasets. Specifically, GIF conducts imagination by optimizing the latent features of seed data in a semantically meaningful space, which are fed into the generative models to generate photo-realistic images with new contents. For guiding the imagination towards creating samples useful for model training, we exploit the zero-shot recognition ability of CLIP and introduce three criteria to encourage informative sample generation, i.e., prediction consistency, entropy maximization and diversity promotion.  With these essential criteria as guidance, GIF works well for expanding datasets in different domains,  leading to 29.9\% accuracy gain on average over six natural image datasets, and 10.4\% accuracy gain on average over three medical image datasets. The source code will be made public. 

Under review as a conference paper at ICLR 2023
EXPANDING SMALL-SCALE DATASETS WITH GUIDED IMAGINATION
Anonymous authors Paper under double-blind review
ABSTRACT
The power of Deep Neural Networks (DNNs) depends heavily on the training data quantity, quality and diversity. However, many real scenarios, is costly and time-consuming to collect and annotate large-scale data. This has severely hin- dered the application of DNNs. To address this challenge, we explore a new task of dataset expansion, which seeks to automatically create new labeled samples expand a small dataset. To this end, we present a Guided Imagination Framework (GIF) that leverages the recently developed big generative models (e.g., DALL- E2) and reconstruction models (e.g.. MAE) to "imagine" and create informative new data from seed data to expand small datasets. Specifically, GIF conducts imagination by optimizing the latent features of seed data in a semantically mean- ingful space, which are fed into the generative models generate photo-realistic images with new contents. For guiding the imagination towards creating samples useful for model training. we exploit the zero-shot recognition ability of CLIP and introduce three criteria to encourage informative sample generation, i.e., pre- diction consistency, entropy maximization and diversity promotion. With these essential criteria as guidance, GIF works well for expanding datasets in differ- ent domains, leading to 29.9% accuracy gain on average over six natural image datasets, and 10.4% accuracy gain average over three medical image datasets.
INTRODUCTION
Having sufficient amount of training data is crucial for unleashing the power of deep neural net- works (DNNs) (Deng et al., 2009; Qi & Luo, 2020). However, many fields, collecting large-seale datasets is expensive and time-consuming (Qi & Luo, 2020; Zhang et al., 2020), resulting in limited dataset sizes which make it difficult to fully utilize DNNs. To address this data limitation issue and reduce the cost of manual data collection, aanottation, we explore dataset expansion in this work, which seeks to build an automatic data generation pipeline for expanding a small dataset into a larger and more informative one, as illustrated in Figure 1 (left).
There are some research attempts that could be applied to dataset expansion. Among them, data augmentation Vries & Taylor, 2017: Cubuk al., 2020; Zhong et al., 2020) applies pre-defined transformations to each image for enriching datasets. However, these transformations mostly affect the surface visual characteristies of image, but have minimal effect on the actual image content. Therefore, the brought new information is limited, and cannot sufficiently address the limited-data issue in small datasets. Besides, some recent studies (Zhang al., 2021c; Li al., 2022) utilize generative adversarial networks (GANs) (Goodfellow et al., 2014; Brock et al., 2018) to synthesize images for model training. They, however, require a sufficiently large dataset for in-domain GAN training, which is not feasible in the small-data scenario. Moreover, the generated images are often not well-annotated, limiting their utility for DNN training. Therefore, both of them are unable to effectively resolve the dataset expansion problem.
For an observed object, humans can easily imagine its different variants in various shapes, colors or contexts, relying their accumulated prior understanding of the world (Warnock & Sartre, 2013; Vyshedskiy, 2019). Such an imagination process is highly useful for dataset expansion, since it does not simply perturb the object's 's appearance but applies rich prior knowledge to create object variants with new information. Meanwhile, recent breakthroughs in large-scale generative models (e.g., DALL-E2 (Ramesh et al., 2022)) have demonstrated that generative models can effectively capture

---

Under review as a conference paper at ICLR 2023
Small datasel
Expanded delsset
90-
-0:
Our rxpooded conoce
datasel expansion
Node partomane
Exp
created data
5a.
C3Re01071 OHARICO
CH15
Pe5p
MTD
Cuarn-grained object Fine-graineel odpaci
Teatur
Pathokagy Ultnasouaa
Figure 1: Dataset expansion aims create data with new information enrich small datasets for training DNN models better (left). The ResNet50 trained on the expanded datasets by our proposed method performs much better than the one trained on the original small datasets (right).
the sample distribution of extremely large datasets (Schuhmann et al., 2021; Byeon et al., 2022) and show encouraging abilities in generating photo-realistic images with a rich variety of contents. This motivates us to explore their capabilities as prior models to develop a computational data imagination pipeline for dataset expansion, by imagining different sample variants from seed data. However, deploying big generative models for dataset expansion is highly non-trivial, complicated by several key challenges, including how generate samples with correct labels, and how to make sure the created samples are useful for model training.
To handle these challenges, conduct series of studies (ef. Section 3), from which make two important findings. First, the CLIP model (Radford et al., 2021), which offers excellent zero-shot classification abilities, can map latent features of category-agnostic generative models to the specific label space of the target small dataset. This is helpful for generating samples with correct labels. Second, we empirically find three informativeness criteria crucial for generating effective training data: 1) zero-shot prediction consistency to ensure that the imagined image is class-consistent with the seed image; 2) entropy maximization to encourage the imagined images to bring more informa- tion; 3) diversity promotion to encourage the imagined images to have diversified contents.
In light of the above findings, we propose the Guided Imagination Framework (GIF) for dataset ex- pansion. Specifically, given a seed image, GIF first extracts its latent feature with the prior generative model. Different from data augmentation that imposes variation over the raw image, GIF optimizes the variation over the latent feature. Thanks to the guidance carefully designed by our discovered criteria, the latent feature is optimized to provide more information while maintaining its class se- mantics. This enables GIF to create informative new samples, with classsconsittent semanties yet higher content diversity, to expand small datasets for model training. Considering that DALL-E2 have been shown be powerful in generating images and MAE (He et al., 2022) is excellent at reconstructing images, we explore their use as prior models for imagination in this work.
We evaluate the proposed method on both small-scale natural and medical image datasets. As shown in Figure (right), compared to the ResNet50 model trained on the original dataset, our method improves the model performance by a large margin across variety visual tasks, including fine- grained object classification, texture classification, cancer pathology detection, and ultrasound image classification. More specifically, GIF obtains 29.9% accuracy gain on average over six natural im- age datasets, and 10.4% accuracy gain on average over three medical image datasets. Moreover, we show that the expansion efficiency of our method is much higher than expansion with existing aug- mentation methods. For example, 5x expansion by our GIF-DALLE method already outperforms 20x expansion by Cutout, GridMask and RandAugment on the Cars and DTD datasets. In addition, the expanded datasets can be directly used to train different model architectures (e.g., ResNeXt, WideResNet and MobileNet), leading to consistent performance improvement.
2 RELATED WORK
Learning with synthetic images. Training models with synthetic images a promising direc- tion (Jahanian et al., 2022). DatasetGANs (Zhang et al., 2021c; Li et al., 2022) explore GAN mod- els (Isola et al., 2017; Esser et al., 2021) to generate images for segmentation model training. How- without labels, they need manual annotations generated images to train a label generator for annotating synthetic images. In contrast, our dataset expansion aims to expand real small dataset to a larger labeled one in fully automatic manner, without involving
ever, the generated images
2

---

Under review as a conference paper at ICLR 2023
human annotators. In this work, we did not explore designing more advanced generative models as this is not our focus, and we leave the discussion on image synthesis to Appendix A for reference.
Data augmentation. Data augmentation has been widely used to improve the generalization of DNNs (Shorten & Khoshgoftaar, 2019), which typically generates new images with manually spec- ified rules, e-g., image manipulation (Yang et al., 2022), image erasing VDeVries & Taylor, 2017; Zhong et al., 2020), image mixup (Zhang et al., 2021a; Hendrycks et al., 2019), and learning to select from set of transformations for augmentation (Cubuk et al., 2019; 2020). Despite their ef- fectiveness in some applications, most augmentation methods apply pre-defined transformations to enrich datasets, which only locally varies the pixel values of images and cannot generate images with highly diversified contents. Moreover, most methods cannot guarantee the effectiveness of the augmented samples for model training. As a result, they cannot effectively address the issue of lack of information in small datasets, and their efficiency of dataset expansion is low. In comparison, our GIF framework leverages powerful generative models trained on large datasets and guides them to generate more informative and diversified images, and thus can expand datasets more efficiently. More discussions on data augmentation are provided in Appendix A.
PROBLEM STATEMENT AND PRELIMINARY STUDIES
Problem statement. To address the common data scarcity challenge when deploying DNN models, this paper explores a new task, dataset expansion. Without loss of generality, we consider image classification problems. We are given a small-scale image dataset D,= 24, ye 3 where de- notes the number of samples, and x; denotes an instance with class label yı- Dataset expansion aims to generate a set of new synthetic samples D.= =(Zy v 1==1 to enlarge the original dataset, such that a DNN model trained on the expanded dataset D, U DD outperforms the model trained on DD signif- icantly. The key is that the synthetic sample set DD should be highly-related to the original dataset Do while bringing sufficient new information beneficial for model training.
3.1 PROPOSAL FOR COMPUTATIONAL IMAGINATION MODELS
Given an observed object, humans can easily imagine its different variants, like the object in various colors, shapes or contexts, based on their accumulated prior knowledge about the world (Warnock & Sartre, 2013; Vyshedskiy, 2019). Inspired by this, we attempt build a computational model to simulate this imagination process for dataset expansion. It is known that deep generative models are trained to capture the full distribution of training dataset and can well maintain its distribution knowledge. We can query well-trained generative model to generate new samples with similar characteristics presented by its training dataset. More crucially, recent deep generative models (e.g., DALL-E2) have shown impressive abilities in capturing the sample distribution of extremely large datasets and generating photo-realistic images with various contents, which inspires us to explore them as the prior model to build the data imagination pipeline.
Specifically, given a pre-trained generative model G, and a seed example (I. y) from the small dataset to expand, formulate the imagination of x from as x' = Gff(x) + 8). Here f(-) is an image encoder of the generative model to transform the raw image into an embedding for imagination with the generative model, and is perturbation applied to f(x) such that G can generate x' different from æ. A simple choice of 8 would be a random noise sampled from a Gaussian distribution. We will discuss how optimize to provide useful guidance in the following section.
Dataset expansion requires the created samples to have correct category labels for model training- However, ensuring the generated x' to have the same label y as x is highly non-trivial, because it is hard to maintain the class semantics of the seed sample in the embedding space of f()) af- ter perturbation and the pre-trained generative models are usually category-agnosie for the target dataset. We resort to CLIP (Radford et al., 2021) to address this issue. CLIP is trained by contrastive learning on a large-scale image-text dataset, such that its image encoder can project images to an embedding space well aligned with rich natural language corpus (Radford et al., 2021; Tian et al., 2022: Wortsman et al., 2022). We propose to leverage the CLIP image encoder in our computational imagination model to map any sample z from the target dataset to the embedding aligned with the embedding of its category name y, and thus it is convenient to take the embedding of label as a reference anchor to regularize the image embedding (ar avoid changing its class.
3

---

Under review as a conference paper at ICLR 2023
55-
Rardom Guised usparale
54.5
50
Ransilugreet
4BLT
52-
4G.
42.2
45-
48
3500
orgiial
w'
(a)
Figure 2: Elfectiveness of the informativeness criteria for sample creation. (a) Comparison between random expansion and our selective expansion on CIFAR CIFAR100-Subbl 10-Subet.... (b) Comparison be- tween the guided DALLE expansion with and without diversity promotion on CIFAR100-Sube -00-Set
is worth noting that in this work we do not seek to develop a biologically plausible imagination model that exactly follows true human brain dynamics and rules. Instead, we draw inspiration from the imagination activities of human brains and propose a pipeline to leverage the well pre-trained big generative models to explore dataset expansion.
3.2 How TO GUIDE THE IMAGINATION FOR EFFECTIVE DATASET EXPANSION?
The proposed data imagination pipeline leverages big generative models to create new samples from the seeds to expand datasets. However, it is unclear what kind samples are effective and helpful for model training. To determine this question and optimize 8 accordingly in the above pipeline, we conduct several preliminary studies and make the following observations.
To guide generative models to create data that are useful for model training, the sample should have the same class semantics (without label change from the seed sample) but bring new information. To achieve these properties, we leverage the zero-shot prediction ability of CLIP and design three informativeness criteria based on the CLIP-mapped label space: zero-shot prediction consistency, entropy maximization and diversity promotion. The criterion of prediction consistency means that the zero-shot predictions for the synthetic image and the seed image should be the same, which en- sures that the class semantics of the imagined image is consistent with that of the seed one. Entropy maximization encourages the synthetic image to have larger zero-shot prediction entropy than the seed image, and thus it is more challenging for classification and brings new information for model training. Moreover, diversity promotion encourages the synthetic images for the same seed image to be as diversified as possible so that they can bring more diverse information.
Class consistency and entropy maximization. We first start with exploratory experiments on subset of CIFARI00 (Krizhevsky et al., 2009) to pinpoint whether achieving the criteria of class consistency and entropy maximization leads to more informative samples. Here, CIFARIO00 100-subeet built for simulating small-scale datasets by randomly sampling 100 instances per class from the original CIFAR 100, and the total sample number is 10,000. We synthesize samples based on existing data augmentation methods (i.e., RandAugment and Cutout VDeVsies & Taylor, 2017)) and expand CIFAR 100-subset by 5x. Meanwhile, we conduct selective augmentation expansion by selecting the samples with the same zero-shot prediction but higher prediction entropy compared to their seed samples, until we reach the required expansion ratio per seed sample. The goal is examine whether the selective synthetic samples are more useful for model training. As shown in Figure 2a, the selective expansion strategy outperforms random expansion by 1.3% to 1.6%, meaning that the selected samples are more informative for model training. Compared to random augmentation, selective expansion can filter out the synthetic samples with lower prediction entropy and those with higher entropy but different predictions. The remaining samples thus contain more information while preserving the same class semantics, leading to better expansion effectiveness.
Diversity promotion. To avoid "imagination collapse" in which the generative models generate excessively similar or duplicate samples, we further introduce the criterion of diversity promotion. To study its effectiveness, we resort to powerful generative model (i.e., DALL-E2) the prior model to generate images and expand CIFAR 100-subet by 5x, where the guided expansion scheme and the implementation of diversity promotion will be introduced in the following section. As shown in Figure 2b, diversity promotion can further improve accuracy by 1.3%, demonstrating that it can bring more diverse information to boost model training.
4

---

Under review as a conference paper at ICLR 2023
Input
Imagined images
Iatent feature
repeated
oplimized
CLIP image encoder
DALL-E2 decoder
①
perturbed
informativnnes guidance
dlss names
CLIP encoder
"ang"
d0g
"car"
zero-shol classifier
Figure 3: Overview of our proposed GIF-DALLEE method, which expands small datasets by creating informative new samples with guided imagination. Here, we resort to DALL-E2 as the prior gen- crative model, in which the image/text encoders are CLIP image/text encoders while the decoder is the diffusion model of DALL-E2. Moreover, + denotes guided residual multiplicative perturbation.
GIF: A GUIDED IMAGINATION FRAMEWORK FOR DATASET EXPANSION
In light of the above studies, propose simple GIF Framework for dataset expansion, by guiding the imagination method built on big pre-trained generative models via the aforementioned criteria.
Given seed image I from target small dataset, we first extract its latent feature f(x) based on the encoder of a pre-trained model (e.g., the CLIP image encoder fCLP-1). Different from data augmentation that imposes variation over the raw RGB images, our proposed pipeline explores optimizing the variation over the sample latent features. Thanks to the careful method designed with the criteria investigated above, the varied latent features are able to maintain the sample class semantics while providing more new information for model training.
To detail our proposed framework, we utilize DALL-E2 as the prior generative model for illustration. As shown in Figure 3, DALL-E2 is built by adopting CLIP image/text encoders fCLIP- and fcuP- as its image/text encoders and using a pre-trained diffusion model G as its decoder. To create a set of new images x' from the seed image GIF first repeats its latent feature f = fCUP.1(>)) for K times, with K being the expansion ratio. For each latent feature, we inject perturbation over the latent feature f with randomly initialized noise ~ U(O, 1) and bias ~ (O,,, 1). Here, to prevent out-of-control imagination, we conduct residual multiplicative perturbation on the latent feature f and enforce an e-ball constraint the perturbation follows:
P/.c ((1 2)f + b),
where (..c(.) means to project the perturbed feature f' to the e-ball of the original latent feature, i.e. 11 flloo S c. Note that each latent feature has independent and b.
Following explored criteria, GIF optimizes and b over the latent feature space as follows:
z',6 b'
arg max Scon + Sent + Saie-
(2)
2,6
Here, Scons Sent and Saiv correspond to the class consistency, entropy difference and diversity, respectively. To compute these objectives, we resort to CLIPP's zero-shot classification abilities. Specifically, we first use fCLPPT encode the class name y of sample æ and take the embedding w, = (C) as the zero-shot classifier of class Each latent feature f(x) can be classified according its cosine similarity to Wys i.e., the affinity score of x belonging cos( f (z), Wy), which forms classification prediction vector [si...., scl for the total C' classes of the target dataset. The prediction of the perturbed feature s' can be obtained in the same way.
class y is 8y
With the zero-shot prediction, we design the objective functions as follows: the prediction consis- tency Scon encourages the consistency between the predicted classification scores on and s', so we define Sea.. s/c where i argmar(s)) is the predicted class of the original latent feature. The cri- terion of entropy maximization Sens seeks to improve the informativeness of the generated image, so we define Sent Entropy (s') -Entoppyss) to encourage the perturbed feature to have higher predic- tion entropy. To promote sample diversity, the diversity Saiv is computed by the Kullback-Leiin Leibler (KL) divergence among all perturbed latent features of seed sample: Saie KL(f': (fff: f), where f denotes the current perturbed latent feature and f indicates the mean over the K perturbed latent features of this seed sample.
5

---

Under review as a conference paper at ICLR 2023
Table I: Accuracy of ResNet-50 trained from scratch on small datasets and their expanded datasets by various methods. Here, CIFAR 100-Subset is expanded by 5x, Pets is expanded by 30x, and all other datasets are expanded by x0x. Moreover, MAE and DALL-E2 are the baselines of directly using them to expand datasets without our guidance.
Dataset
Caltech101
Cars
Hlowers
DID
CIFARI00SS
Pets
Avg.
Original
26.3+1.0
19,8 18.9
44.1
23. 23.1
35.0+17
6.8418
30.9 33.6 (+2.7)
KD with CLIP
33.2g1.
25.6g0 25.
37.823.1
11.1a0
Exponded
Cutout (DeVries Taylor,2011
51.51u7
25.81us
77.8
24.2
44.3
38.7
(+12.8)
GridMask (Chen et 2020)
28440s
80.720a
25.3
48.220.7
37.6.u.3
(+14.4
RandAugment (Cabuk et al., 2020)
57.8 $7.840.9
43.2+2.1
838820
28.7+12
46.7=1.5
48.0z04
51.4 (+20.5)
MAE al. 2022)
50.64008
25.9+1.4
76.340.
27.6+02
44.3+1.7
39.9.u.7
44.1 (+13.2)
DALL E2 (Ramesh et al., 2022)
61.340.2
48.3+0.3
84.1
34.5+04
52.1=0.
61.7=0.
(+.06+)).
GII-MAE (ours)
58.4g1.0
44.5
84,420a
34,2.0a
52.721.6
52.400
54.4 (+23.5)
GIF DIF-DALLEE (ours)
63.0gg.5
53.1
88.220.3
39.5.0.7
54.5.1.
66.420.4
60.8 (+29.9)
Note that the above guided latent feature optimization is the key step for achieving guided imagina- tion. After updating the noise 2' and bias D' for each latent feature, GIF obtains set of new latent features by Eq. (I), which are then used to create new samples through the decoder G. In this way, a small-scale dataset can be effectively expanded to a larger and more informative one.
Considering that DALL-E2 have proven to be powerful in generating images and MAE (He et al., 2022) is skilled at reconstructing images, we use them as prior models for imagination. We call the resulting methods GIF-DALLE and GIF-MAE, respectively.
GIE-DALLE follows exactly the aforementioned pipeline for guided imagination, while we slightly modify the pipeline of GIF-MAE because its encoder is not the CLIP image encoder. Specifically, GIF-MAE first generates a latent feature for the seed image based on its encoder, and conducts ran- dom channel-level noise perturbation following the way of Eq. (1). Based on the perturbed feature, GIF-MAE generates an intermediate image via its decoder, and applies CLIP to conduct zero-shot predictions for both the seed and the intermediate image to compute the guidance Eq. (2) for op- timizing latent features. In this way, GIF-MAE can create content-consistent samples of diverse styles. More details of the two methods are provided in their pseudo codes in Appendix C.
5 EXPERIMENTS
5.1 EXPANSION ON NATURAL IMAGE DATASETS
Settings. We first evaluate the effectiveness of our proposed method on six small-scale natu- ral image datasets, including object classification (Caltech-101 (Fei-Fei et al., 2004), CIFARIO0- Subset (Krizhevsky et al., 2009)), fine-gran fine-grained object classification (Standard Cars (Krause et al., 2013), Oxford 102 Flowers (Nilsback & Zisserman, 2008), Oxford-IIIT Pets (Parkhi et al., 2012)) and texture classification (DTD (Cimpoi al., 2014)). Here, CIFAR 100-subset is an artificial dataset for simulating small-scale datasets by randomly sampling 100 instances per class from the original CIFAR 100 dataset. Their data statistics are given in Appendix D.
We implement GIF in Py Torch based on CLIP VIT-B/32, DALL-E2 and MAE VIT-L/16, which are pre-trained on large datasets and then fixed for dataset expansion. We use the official checkpoints of CLIP VIT-B/32 and MAE VIT-L/16, and use the DALL-E2 pre-trained on Laion-400M (Schuhmann et al., 2021). After expansion, we train ResNet-50 (He et al., 2016) from scratch for 100 epochs on the expanded datasets. More implementation details are provided in Appendix C.
As there is no algorithm devoted dataset expansion, we take representative data augmentation methods as baselines, including RandAugment, Cutout, and GridMask (Chen et al., 2020). Besides, CLIP has demonstrated outstanding zero-shot ability thanks to its pre-training on extremely large- scale datasets, and some recent works explore distilling CLIP to facilitate model training. Hence, we also compare our method with knowledge distillation (KD) of CLIP on the original dataset.
Comparisons with data augmentation. As shown in Table 1, compared with the model trained on the original datasets, GIF-DALLE leads to 29.9% accuracy gain on average over six datasets, showing promising capabilities for expanding small datasets. Such improvement not only shows that dataset expansion is a promising direction for boosting DNNs on real small-date applications, but also verifies the effectiveness of our guided imagination in creating informative new samples.

---

Under review as a conference paper at ICLR 2023
GricMask
Ranskugment
(ound)
GIF-DALLE (ours)
Standard Cars
DTD
38.5
Oxtord HIT Pets
50
pertamanne
42.5
455
40-
50.7
51.
5/4
25-
254
28
50
469
06.7
48.0
395
2/
25
243
40
366
267
25
241
20
223
274
37.5
+20x
120x
Figure 4: Accuracy of ResNet-50 trained from scratch on the expanded datasets with different ex-
pansion ratios. The performance is averaged over 3 runs. More results reported in Appendix F.
Input
GridMask
Randkugment
Our GF DALLE
Figure 5: Examples of the created samples for Caltech101 by data augmentation and GIF.
In addition, our expansion method more sample efficient than data augmentations, in terms of the accuracy gain brought by each created sample. As shown in Figure 4, on Cars and DTD datasets, 5x expansion by GIF-DALLE even outperforms 20x expansion by various data augmentation methods, implying GIF-DALLE is 4x more sample efficient than them. This is because these augmentation methods cannot generate images with new and highly diversified content. In contrast, our GIF leverages strong prior models (e.g.. DALL-E2 and CLIP) trained from large datasets, and learns to perform imagination with the guidance from our discovered criteria. Hence, can generate more diversified and informative images, leading
significant performance gain.
Visualization. We further visualize the created samples. As shown in Figure 5, GridMask masks partial pixels of the inputs, while RandAugment randomly varies the images with set of transfor- mations. Both of them cannot create new content from the inputs, whereas our method can create images with new contents from the seed images, e.g, water lilies with different postures and back- grounds. This further shows the superiority of our method. See more visualization in Appendix F.
Comparisons knowledge distillation by CLIP. As shown in Table 1, although KD of CLIP indeed improves model performance on most datasets, it only leads to limited performance gain on those small-scale datasets. This reveals that simply distilling the knowledge of big models does not necessarily work well for the small-data regime. This also verifies the value of dataset expansion by creating informative new samples for model training on small datasets.
5.2 ANALYSIS
Expanded datasets can boost various model architectures. We further apply the expanded Cars dataset (5x expansion ratio) by GIF-DALLE to train ResNeXt-50 (Xie et al., 2017), WideResNet- 50 (Zagoruyko & Komodakis, 2016) and MobileNet V2 (Sandler et al., 2018) from scratch. Table shows that the expanded dataset brings consistent performance gain for all the architectures. This clearly demonstrates the effectiveness of the expanded dataset and the generalizability the created samples by our method. Once expanded, the datasets are readily used for training other models.
Table 2: Performance of different model architectures trained on 5x expanded Cars by GIF.
Dataset
Re
ResNeXi.50
WideResNet-50
MobilteNet-22
Avg.
Original
19.8.0.
18.4
32.0an.
26.2441
24.1
Exponded
RandAugment
43.2
29.6
49.2 49.2
39.7
40.4 (+9.5)
GIF-DALLE
53.1
43.7
60.0gn.
47.8
51.2 (+27.1)

---

Under review as a conference paper at ICLR 2023
Input
DALLE expansion
Guided DALLE expansion
input
failure cases
(b) Figure 6: More examples created samples by GIF-DALLE. (a) Visualization of DALLE expansion with and without our guidance. (b) Visualization of failure cases.
(a)
Effectiveness of guided latent feature optimization. Based on our guidance criteria, GIF opti- mizes the variation over the latent features for creating new samples. As shown in Table our guided framework obtains consistent performance gain on all datasets compared to direct expansion with MAE DALL-E2 respectively, which further demonstrates the effectiveness criteria in optimizing informativeness and diversity of the created samples. More specifically, com- pared to direct DALLE expansion, as shown in Figure 6a, our GIF-DALLE with guidance can create starfish images with more diverse object numbers, and create motorbike images with more diverse angles of view and even a new driver. Moreover, compared to direct MAE expansion, GIF-MAE generate informative samples with more diverse style while maintaining image content, as shown in Figure 10 (cf. Appendix B). More ablation results of these criteria are given in Table (cf. Appendix E.2).
our guidance
Pixel-wise vs. channel-wise noise. GIF-MAE injects perturbation along the channel dimen- sion instead of spatial dimension. Note that the generated image based pixel-level noise variation is analogous to adding pixel-level noise the original images. This may harm the integrity and smoothness of image con- tent, leading the generated images to noisy (cf. Figure 10(d)). In contrast, GIF decou- ples latent features into two dimensions (i.e., token and channel), and particularly conducts lhaenel-level perturbation. As shown in Fig- 10(e), optimizing channel-level noise vari- ation can generate more informative data, lead- ing more effective expansion (cf. Figure 7).
55
Model performanc
44.99
45.3
46.2
eriginal
faxe pluel noha
Sund chienne nohe
learned takan nale
leamed channel nons
Figure 7: Effects of different kinds of noise on MAE for expanding CIFARIOO-Subset by 5x.
Failure cases. We visualize some failure cases in Figure 6b. As we use pre-trained DALL-E2 without fine-tuning, the quality of some created samples may be limited due to domain shifts. For example, the face of the generated dog image in Figure 6b seems like a horse face. However, despite seeming less realistic, those samples are generated following our guidance criteria, so they can still maintain the class consistency and bring new information, thus being beneficial to model training.
5.3 APPLICATIONS TO MEDICAL IMAGE DATASETS
Medical image understanding severely data-lacking field. We thus explore apply our method to expand three typical scall-scale medical image datasets (Yang et al., 2021), including BreastM- NIST (Al-Dhabyani al., 2020), PathMNIST (Kather et al., 2019), and OrganSMNIST (Xu et al., 2019). These datasets cover wide range of medical image modalities, including breast ultrasound (BreastMNIST), colon pathology (PathMNIST) and Abdominal CT (OrganSMNIST), See their de- tailed statistics in Appendix D. Considering that DALL-E2 was initially trained natural images and suffers from domain shifts to medical domains (see discussion in Appendix E.3), we thus first fine-tune it on the target medical datasets, followed by dataset expansion. Similar natural image
8

---

Under review as a conference paper at ICLR 2023
Table 3: Accuracy of ResNet-50 trained from scratch
Table 4: Accuracy on the expanded Or- ganSMNIST with various expansion ratios.
on small medical image datasets and their 5 5x-expanded datasets by various methods. All the performance is av- eraged over 3 runs.
Expanded dataset
5x
10x
20x
RandAugment
79.6
80.1
80.5
Datasct
PathMNIST
BreasMNNIST
OrgarSMNIST
Avg.
GIF-MAE
80.6
81.1
81.2
Original
72.4
55.8
76.3
68.2
Table 5: Comparison of model fine-tuning on OrganSMNIST with 2 expansion.
KD with CLIP
60.2
71.6
Exponded
Cuiout
78.8
66.7
78.3
74.5 74.6(+6.4)
Original
36.3
RandAugment
GIF MAE
GridMask
78.4
$2.0
79.2
66.8
68.7
73.3
79.6
80.6
78.9
78.6 (+10.-)
74.7 (16.5)
75.8 (+7.6)
Fine tuning (CLIP pre trained)
Fine- tuning (ImageNet pee- pre-trained)
77.9
78.9
GIF DALLE
84.4
76.6
80.5
(+
Expanded by GIF-MAE
81.2
datasets, we train ResNet-50 from scratch based on the expanded datasets. The model performance is averaged over 3 runs in terms of macro accuracy.
Comparisons with data augmentation. As shown in Table 3, our GIF has a good ability to expand small-scale medical image datasets. Compared to the model trained on the original datasets, GIF- MAE brings 12.3% performance gains and GIF-MAE brings 10.4% performance gains on average over three medical image datasets. This further demonstrates the effectiveness and practicability of our method. Moreover, the result also reveals that, based on GIF, the prior MAE model pre- praitrained on large natural image datasets can also expand small medical image datasets well. In addition, our method also presents higher expansion efficiency than RandAugment on medical image datasets. As shown in Table 4, on OrganSMNIST, 10x expansion by GIF-MAE already outperforms 20x expansion by RandAugment. This further verifies the importance of generating more informative samples for medical dataset expansion, where we believe there is still huge room for improvement.
Visualization. We visualize the created medical samples in Figure 14 (cf. Appendix E.4). We find that RandAugment randomly varies the medical images based on a set of pre-defined transformations and may crop the lesion location of medical images, so it cannot guarantee the created samples to be informative and may even generate noisy samples. In contrast, our GIF-MAE can generate content- consistent images with diverse styles, and thus enrich medical images while retaining their lesion locations. Therefore, GIF-MAE expands medical datasets more effectively.
Comparison with model fine-tuning. The pre- and fine-tuning scheme is widely used to transfer the representation ability learned from large datasets to small datasets. Here we also compare this scheme with our dataset expansion. As shown in Table fine-tuning both the CLIP pre-trained or the ImageNet pre-trained ResNet-50 only leads to limited performance gain, which is worse than dataset expansion. That is. when the pre-trained datasets are highly different from the target dataset (e.g., from natural images to medical images here), the pre-training and fine-tuning scheme does not significantly help performance, which was also verified by Raghu et al. (2019). In contrast, GIF can effectively exploit the knowledge of MAE and guide it to generate informative medical images, thus leading to better expansion effectiveness. This also reflects the importance of creating informative new samples for boosting real small-data medical applications.
6 CONCLUSIONS
This work has explored new task, dataset expansion, towards resolving the challenging data scarcity issue for DNN training. Inspired by human learning with imagination, we presented a simple framework for dataset expansion via guided imagination. Promising empirical results on both small-scale natural and medical image datasets have demonstrated the effectiveness of the pro- posed method. Despite its encouraging results, there is significant room improve our method. Specifically, as GIF does not fine-tune the pre-trained generative models on the target dataset, it may generate poor-quality images when there are huge domain shifts. Further fine-tuning the generative models can improve the quality of the expanded datasets, which is worth exploring in the future. Moreover, the expanded samples are still less informative than real samples. For example, we ob- served that ResNet- 50 trained from scratch on our 5x -x-xpandeed CIFAR Sub-betett 1..15#1.1) per- forms worse than on the original CIFAR 100 dataset (71 O..6-.)), indicating huge headroom for algo- rithmic dataset expansion to improve. We expect that this pioneering work can inspire future studies on dataset expansion, e.g., how to leverage increasingly powerful generative models to achieve even better performance than human data collection.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy, Dataset of breast ultra- sound images. Data in Brief. 28: 28:104863, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference Learning Representations, 2018.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. Coyo-700m: Image-text pair dataset, 2022.
Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and Jose Alvarez. Data-free knowledge distillation for object detection. Winter Conference on Applications of Computer Vision, pp. -298, 2021.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Gridmask data augmentation. arXiv preprint wrXiv:2001.04080 2020.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In International Conference on Computer Vision, pp. 4794 4802, 2019.
Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Computer Vision and Pattern Recognition, pp- 113-123, 2019.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems, volume 33, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Terrance Driese and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Ad- vances Neural Information Processing Systems, volume 34, pp. 8780-8794, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Computer Vision and Pattern Recognition, pp. 12873-12883, 2021.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Computer Vision and Pattern Recognition Workshop, 2004.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference Machine Learning, pp. 1180-1188 2015.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural
Information Processing Systems, 2014.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 12966):189-11899 2021.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre- trained language model fine-tuning. In International Conference Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Computer Vision and Pattern Recognition, pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Computer Vision and Pattern Recognition, pp. 16000- 16009, 2022.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations, 2019.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531, 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer real-time with adaptive instance normal- ization. In International Conference Computer Vision, pp. 1501-1510, 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In Computer Vision and Pattern Recognition, pp. 1125-1134, 2017.
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. In International Conference on Learning Representations, 2022.
Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, et al. Pre- dicting survival from colorectal cancer histology slides using deep learning: A retrospective mul- ticenter study. PLoS Medicine, 16(1)=e1002730, 2019.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Computer Vision and Pattern Recognition, pp. 2426-2435, 2022.
Diederik Kingma, Max Welling, et al. An introduction variational autoencoders. Foundations and Trends@ in Machine Learning, 12(4):307-392, 2019.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine- grained cars. 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Daiqing Li. Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba. Big- datasetgan: Synthesizing imagenet with axxe-wise annotations. Computer Vision and Pattern Recognition, pp. (1330-21340) 2022.
Lu Xiangyang Li, and Xiang Long. Fencemask: A data augmentation approach for pre-extracted image features. arXiv preprint arXiv:2006.07877 2020.
Xingjian Li. Haoyi Xiong, et al. Delta: Deep learning transfer using feature map with attention for convolutional networks. In International Conference on Learning Representations, 2019.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Advances in Neural Information Processing Systems, volume 32, 2019.
Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European conference on computer vision, 2020.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, 2022.
11

---

Under review as a conference paper at ICLR 2023
Maria-Elen Nilsback and Andrew Zisserman. Automated flower classification large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Computer Vision and Pattern Recognition, 2012.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In International Conference on Computer Vision, pp- 2085-2094, 2021.
Guo-Jun Qi and Jiebo Luo. Small data challenges in big data era: A survey of recent progress on unsupervised and semi-supervised methods. IEEE Transactions Putlern Analysis and Machine Intelligence, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging, Advances in Neural Information Processing Systems, 32, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. pp. 8821-8831,, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manoy Imagenet-21k pretraining for the masses. In Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic -maage diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chinh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. In International Conference on Learning Representations, 2017.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander Alemi, and Andrew G Wilson. Does knowledge distillation really work? In Advances in Neural Information Processing Systems, volume 34, pp. 6906-6919 6919, 2021.
Changyao Tian, Wenhai Wang, Xizhou Zhu, Xiaogang Wang. Jifeng Dai, and Yu Qiao. VI-Itr: Learning class-wise visual-linguistic representation for long-tailed visual recognition. Euro- pean Conference on Computer Vision, 2022.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
12

---

Under review as a conference paper at ICLR 2023
Andrey Vyshedskiy. Neuroscience of imagination and implications for human evolution. Current Neurobioloyy, 2019.
Can Wang. Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and- image driven manipulation of neural radiance fields. In Computer Vision and Pattern Recognition, pp. 3835-3844, 2022.
Pei Wang, Yijun Li, Krishna Kumar Singh, Jingwan Lu, and Nuno Vasconcelos. Imagine: Image synthesis by image-guided model inversion. In Computer Vision and Pattern Recognition, pp- 3681-3690, 2021.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei Efros. Dataset distillation. arXiv preprint arXiv: 10959, 2018.
Mary Warnock and Jean-Paul Sartre. The psychology of the imagination. Routledge, 2013.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongscok Namkoong, et al. Robust fine-tuning of zero-shot models. In Computer Vision and Pattern Recognition, pp. 7959-7971, 2022.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Computer Vision and Pattern Recognition, 2017.
Haohang Xu, Shuangrui Ding, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Masked autoencoders robust data augmentors. arXiv preprint arXiv:2206.04844 2022.
Xuanang Xu, Fugen Zhou, Bo Liu, Dongshan Fu, and Xiangzhi Bai. Efficient multiple organ local- ization ct image using 3d region proposal network. IEEE Transactions on Medical Imaging, 38(8):1885-1898, 2019.
Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight toml benchmark for medical image analysis. In IEEE International Symposium on Biomedical Imaging, pp. 191-195, 2021.
Suorong Yang, Weikang Xiao, Mengcheng Zhang, Suhan Guo, Jian Zhao, and Furao Shen. Image data augmentation for deep learning: A survey. arXiv preprint arXiv:2204.08610, 2022.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Computer Vision and Pattern Recognition, pp. 7724,,72, 2020.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. In Computer Vision and Pattern Recognition, pp. 4578 4587, 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe. and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. Interna- tional Conference on Computer Vision, pp. 6023-6032. 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference Learning Representa- tions, 2021a.
Yifan Zhang, Ying Wei, et al. Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE Transactions Image Processing, 2020.
Yifan Zhang, Bryan Hooi, Dapeng Hu, Jian Liang, and Jiashi Feng. Unleashing the power of con- trastive self-supervised visual models via rontrast-r fine-tuning. In Advances Neural Information Processing Systems, volume 34, pp.
2021b.
13

---

Under review as a conference paper at ICLR 2023
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Dean--Fancoos Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Computer Vision and Pattern Recognition, pp. 10145-10155, 2021c.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp. 12674-12685, 2021.
Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. arXiv preprint arXiv:2204.077 07511, 2022.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. (2):3, 2021.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In AAAI conference on artificial intelligence, volume 34, pp. 13001-13008, 2020.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European Conference on Computer Vision, pp. 592-608, 2020.
14

---

Under review as a conference paper at ICLR 2023
APPENDIX
A MORE RELATED STUDIES
Image synthesis. In the past decade, image synthesis has been widely studied following four main approaches: GANs (Isola et al., 2017; Esser et al., 2021), auto-regressive models (Kingma et al., 2019; Ramesh et al., 2021), diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021), and neural radiance fields (Mildenhall et al., 2020; Yu et al., 2021). For example, DALL-E2 (Ramesh et al., 2022) and Imagen (Saharia et al., 2022) trained on large-scale datasets have shown promising abilities to generate photo-realistic images, and thus can serve as generative models in our work for dataset expansion. Recently, CLIP (Radford et al., 2021), thanks to its text-image matching ability. has been used to guide image generation (Patashnik et al., 2021; Nichol et al., 2022; Kim et al., 2022; Wang et al., 2022), where CLIP is applied to match the generated image and given text. In contrast, our work explores using CLIP to map latent features of generative models to the label space of the target small dataset, which enables them to conduct dataset expansion. In addition, model inversion (Xia et al., 2022) was also explored to generate images by inverting a trained classification network (Yin et al., 2020; Wang et al., 2021) 1) a GAN model (Zhu et al., 2020). Although our GIF framework only explores DALL-E2 and MAE (He et al., 2022) in this work, model inversion approaches can also be incorporated into our framework for dataset expansion, which is worth exploring in the future.
More discussion data augmentation. Image data augmentation has been widely used im- prove the generalization of DNNs during model training (Shorten & Khoshgoftaar, 2019; Yang et al., 2022). According the technical characteristics, data augmentation for images can be divided into four categories: image manipulation, image erasing, image mix and auto augmentation. Specif- ically, image manipulation augments data via image transformations, including random flipping, random rotation, random scaling ratio, random cropping, random sharpening, random translation and so on (Yang et al., 2022). Image erasing augments data by randomly replacing pixel values of some image regions with constant values or random values, e.g., Cutout VDes & Taylor, 2017), random erasing (Zhong et al., 2020), GridMask (Chen et al., 2020) and Fenchmask (Li et al., 2020). Image mix augments data by randomly mixing two or more images sub-regions into one image, egg. Mixup (Zhang et al., 2021a), CutMix (Yun et al., 2019), and AugMix (Hendrycks et al., 2019). Auto augmentation seeks to augment images by automatically searching randomly selecting aug- mentation operations from a set of random augmentations, including Autonngment (Cubuk et al., 2019), Fast AutoAutment (Lim al., 2019) and RandAugment (Cubuk et al., 2020).
Despite effectiveness in some applications, most of these augmentation methods impose pre-defined transformations very each sample for enriching datasets, which only locally varies the pixel values of images and cannot generate images with highly diversified content. Moreover, as most methods are random augmentation operations, they cannot make sure the augmented samples are informative for model training and may even introduce noisy augmented samples. As a result, the brought new information can be limited small datasets and the efficiency of dataset expansion is low. In comparison, the proposed GIF framework resorts to powerful generative models (e.g., DALL-E2) trained on large-scale image datasets and guides them to conduct dataset expansion based on guided latent feature optimization designed by our discovered criteria (i.e., zero-shol prediction consistency, entropy maximization and diversity promotion). As a result, the created images are more informative and diversified than simple image augmentation, thus leading to more effecttvee and efficient dataset expansion.
Note that, Xu et al. (2022) also explores MAE for image augmentation based its reconstruction ability. Specifically, it augments images by first masking some sub-regions of images and then sending the masked images into MAE for image reconstruction. The reconstructed images with the recovered but slightly different sub-regions are then used for model training as augmented images. Like the above mentioned random augmentation methods, this approach (Xu et al., 2022) cannot generate images with new content and cannot ensure the reconstructed images to be informative and useful. In contrast, our GIF-MAE guides MAE to create informative new samples of diverse styles through our guided latent feature optimization strategy. Therefore, GIF-MAE is able to generate more useful synthetic samples for effective dataset expansion.
15

---

Under review as a conference paper at ICLR 2023
Difference from dataset distillation. Dataset distillation (Wang et al., 2018; Zhao et al., 2021; Zhao & Bilen, 2021; 2022), also called dataset condensation, aims to condense a large dataset to small/ comparable set of synthetic samples, which is expected to train models to have lossless performance compared to the original dataset. Such a task is in the opposite direction of our dataset expansion task, which aims expand small dataset to larger and more informative by automatically generating informative and diversified new samples.
Difference from model transfer learning. Based on existing large datasets (e.g., ImageNet (Deng et al., 2009; Ridnik et al., 2021)), numerous studies have explored model transfer learning, such as model fine-tuning (Li al., 2019; Gunel et al., 2021; Zhang al., 2021b), knowledge distilla- tion (Hinton et al., 2015; Gou et al., 2021), and domain adaptation (Ganin & Lempitsky. 2015; Tzeng al., 2017). Despite effectiveness in some applications, these model transfer learning paradigms also suffer from key limitations. For example, Raghu et al. (2019) found that the pre-training and fine-tuning scheme does not significantly help model performance when the pre-trained datasets are very different from the target datasets (e.g., from natural images to medical images). Moreover, model domain adaptation usually requires that the source dataset and the target dataset are paired with the same highly overlapping label spaces, which is usually unsatisfiable in small-data ap- plication scenarios. In addition, Stanton et al. (2021) found that knowledge distillation does not necessarily work if the issue of model mismatch exists (Cho & Hariharan, 2019), i.e., large discrep- ancy between the predictive distributions of the teacher model and the student model. The above limitations of model transfer learning also reflect the importance of effective dataset expansion: if small dataset is effectively expanded, then various models can be directly trained on it. We note that some data-free knowledge distillation studies (Yin al., 2020; Chawla et al., 2021) also synthesize images, but their goal is particularly to enable knowledge distillation in the setting without data. In contrast, our task is independent of model knowledge distillation. The expanded datasets are not method-dependent or model-dependent, and thus can train various model architectures to perform better than the original small ones.
16

---

Under review as a conference paper at ICLR 2023
B MORE PRELIMINARY STUDIES
Sample-wise expansion sample-agnostic expansion? When design the selective expansion strategy in Section 3.2, another question appears front of us: should we ensure that each sample expanded by the same ratio? To determine this, empirically compare RandAugment expan- sion with sample-wist wise selection and sample-agnostic selection according to two expansion criteria, L.., zero-shot prediction consistency and entropy maximization. Figure 8 shows that sample- wise expansion performs much better than sample-agnostic expansion. To find out the reason for this phenomenon, we visualize how many times sample is expanded by sample-agnostic expansion. As shown in Figure 9, the expansion numbers of different samples by sample-agnostic expansion present long-tailed distribution, where many image samples are even not expanded. The main reason is that, due to the randomness of RandAugment and the differences among images, not all created samples are informative and it easier for some samples be augmented more effectively than others. Therefore, given a fixed expansion ratio, the sample-agnostic expansion strategy, as it ignores the differences of images, tends to select more expanded samples for those easy-to-augment images. This leads sample-agnostic expansion to waste some valuable original samples for expan- sion (i.e. loss of information) and also incurs a caass-iballanee problem, thus resulting in worse performance in Figure 8. In contrast, sample-wise expansion can fully take the advantage of all the samples in the target small dataset and thus is more effective than sample-agnostic expansion, which should be considered when designing dataset expansion approaches.
48.0
20.0
45.0
Expersion rumber per sample (ootad))
17.5
47.
Model parbereenss
47.0
12.5
10.0
465
7.5
There semples are not expanded
46.0
5.0
45.
2.5
0.0
45.0
Random expersion
Dats-agrosic guidance Dass-wise guidance
2000
4000
6000
8000
10000
Sample
Figure 8: Comparison of model performance between samples-wise selection and sample- agnostic selection for RandAugment expansion on CIFAR 100-Subset..
Figure 9: Statistics of the expansion numbers of different data in CIFAR 1ARR100-Subeet by sample- agnostic selective expansion with RandAugment, which presents long-tailed distribution.
Pixel-level noise or channel-level noise? When we explore the MAE-based expansion strategy in preliminary studies, first explore pixel-level noise to vary sample latent features, which, however, does not perform well. We dig the reason behind it by visualizing the reconstructed images. One illustrated example is given in Figure 10(d), from which we find that the generated image based on pixel-level noise variation is analogous to adding pixel-level noise to the original images. This may harm the integrity and smoothness of image content, leading the reconstructed images to be noisy and less informative. In comparison, as shown in Figure 10(b), the strong augmentation method (i.e., RandAAumeet) mainly varies the style and geometric position of images, but slightly changes the content semantics of images, so it can better maintain the content consistency. This difference inspires us to factorize the influences on images into two dimensions: image styles and image con- tents. In light of Huang & Belongie (2017), we know that the channel-level latent feature encodes more subtle style information, whereas the levelllevel latent feature encodes more content informa- tion. We thus decouple the latent feature of MAE into two dimensions (i.e.. token dimension and a channel dimension), and plot the latent feature distribution change between the generated image and the original image in these two dimensions.
17

---

Under review as a conference paper at ICLR 2023
(a) original image
(b) RandAugment
(c) MAE reconstruction
noised-added MAE
(0) Guided MAE
Figure 10: An illustrated visualization of the generated images by (b) RandAugment, (c) MAE reconstruction, (d) random pixel-level variation over latent features, and (e) guided MAE expan- sion. We find our guided MAE can generate content-consisten images diverse styles.
We report the visualization in Figure 11. The added pixel-Ievel noise changes the token-level latent feature distribution more significantly than RandAugment (cf. Figure 1 I())),, but it only slightly changes the channel-level latent feature distribution (cf. Figure 11()). This implies that pixel-level noise mainly alters the content of images but slightly changes their styles, whereas RandAugment mainly influences the style images while maintaining their content semantics. In light of this observation and the effectiveness of RandAugment, we motivated disentangle latent features into the two dimensions, and particularly conduct channel-level noise to optimize the latent features in method. As shown in Figure 11, the newly explored channel-level noise variation varies the channel-level latent feature more significantly than the token-level latent feature, and thus can diversify the style of images while maintaining the integrity of image content. This innovation enables the explored MAE expansion strategy to generate more informative samples compared to pixel-level noise variation (ef. Figure 10(d) vs Figure 10(e)), leading more effective dataset expansion, as shown in Figure 7 in the main paper.
0.20
Pexe-eevll random noise
Pixel-lvvel random noise
Rand/ugment
RandAugment
guided channel-level noise
4
guided charnel-tavel nake
0.15
Diffrance chhenna-eevel disintuution
Dfferencee oo loud featurs
0.00
2.
0.05
-0100
50
100
160
200
0
200
400
600
800
1000
Token index
Channel Index
(a) Difference of token-level feature distribution
(b) Difference of channoi-eeval feature distribution
Figure 11: Changes of the latent feature distributions along the token dimension and the channel dimension, between the latent feature of the generated image and that of the original image.
18

---

Under review as a conference paper at ICLR 2023
C MORE IMPLEMENTATION DETAILS
C.I IMPLEMENTATION DETAILS OF GIF-DALLE
Our GIF-DALLE applies DALL-E2 (Ramesh al., 2022) as its prior generative model, which follows the pipeline described in Section 4. Its pseudo-code is provided in Algorithm 1. Specifically, DALL-E2 built by using CLIP image/text encoders fCLP and fCLP-T as its image/text encoders and adopting a diffusion model G as its decoder. Here, GIF-DALLE conducts guided imagination on CLIP embedding space.
Here, we further clarify the implementation of the proposed guidance. Specifically, the prediction consistency Scon encourages the consistency between the predicted classification scores on s and s':
Scon 2/ s.t.,
arg max(s),
where i argmar(s) is the predicted class of the original latent feature. Such a loss helps to keep the prediction of the optimized feature to be the same as that of the original one, The criterion of entropy maximization Sent seeks to improve the informativeness of the generated image follows:
Sent Entropy(s') (s') Entropy(s) (>) -s log(s') + log(s),
(4)
which encourages the perturbed feature to have higher information entropy regarding CLIP zero-shot predictions. To promote sample diversity, the diversity Saio is computed by the Kullback-Leibler (KL) divergence among all perturbed latent features of seed sample as follows:
Sai f))
(5)
where f' denotes the current perturbed latent feature and f indicates the mean over the K perturbed latent features of this seed sample. In implementing diversity promotion Saie. measure the dissimilarity of two feature vectors by applying the softmax function to the latent features, and then measuring the KL divergence between the resulting probability vectors.
In our experiment, the DALL-E2 model is pre-trained on Laion-400M (Schuhmann et al., 2021) and then fixed for dataset expansion. The resolution of the created images by GIF-DALLE is 64x64 for all datasets. Moreover, we set E = 0.1 in the guided latent feature optimization.
Algorithm 1: GIF-DALLE Algorithm
Input: Original small dataset Do; CLIP image encoder fcuuu(-);; DALL-F2 diffusion decoder G(.); CLIIP zero-shot classifier w()); Expansion ratio K' Perturbation constraint e.
Initialize: Synthetic data set D, 0:
for E D, do
Scon = Sent = 0;
f = ;
// latent feature 1/ CLIP zero-shot prediction
w(f) for i=l..... K do
Initialize noise =; ~ 2(00 1) and bias NNO,, 1);
f = PI, + 21) b.):
// noise perturbation
8 = w(f:):
Scon += o'j,
j arg max(s):
prediction consistency score // entropy difference score
Sent += Entropy(s') - Entropy(s):
end
f= mewm (fF))
Saie sum(fKL(fl: :
// diversity score
KK t- arg max, , Som + S.nt + Saco
for i=I..., do
f = -P,,.(11 =f),f - b();
x G(f");
Add x + DD;
end end
Output: Expanded datasct D. D...
19

---

Under review as a conference paper at ICLR 2023
C.2 IMPLEMENTATION DETAILS OF GIF-MAE
Our GIF-MAE applies the MAE-trained model (He et al., 2022) as its prior generative model based on its strong image reconstruction abilities. As its encoder is not the CLIP image encoder, slightly modify the pipeline of GIF-MAE. As shown in Algorithm 2, GIF-MAE first generates la- tent feature for the seed image based on its encoder, and conducts channel-wise noise perturbation. Here, the latent feature of MAE has two dimensions: spatial dimension and channel dimension. As discussed in our preliminary (cf. Appendix B), the channel-level latent feature encodes more subtle style information, whereas the token-level latent feature encodes more content information. Based on finding this preliminary study, we particularly conduct channel-level noise to optimize the latent features in our GIF-MAE method for maintaining the content semantics of images unchanged. Based on the perturbed feature, GIF-MAE generates an intermediate image via its decoder, and ap- plies CLIP conduct vero-shot prediction for both the seed and the intermediate image compute the guidance Spons Sent and S With these guidance, GIF-MAE optimizes the latent features for creating content-consistent samples of diverse styles. Here, GIF-MAE conducts guided imagination on its own latent space.
In experiment, implement GIF-MAE based on CLIP VIT-B/32 and MAE VIT-L/16, which are pre-trained on large datasets and then fixed for dataset expansion. Here, we use the official checkpoints of CLIP VIT-B/32 and MAE VIT-L/16. The resolution of the created images GIF- MAE is x 224 for all datasets. Moreover, we set e = 5 for guided latent feature optimization in GIF-MAE.
Algorithm 2: GIF-MAE Algorithm
Input: Original small dataset Do; MAE image encoder f(-) and image decoder G(-); CLIP image encoder fc):-(()); CLIP zero-shot classifier w(-); Expansion ratio K; Perturbation constraint Initialize: Synthetic data set D,
for E D. do
Smm S 0:
f f(x);
// latent feature // CLIP zero-shot prediction
w (x))
for i=l.....K
Initialize noise 2, U(0, 1) and bias b, ~O 1);
PI. 2)) b:);
chanrel-level noise perturbation // intermedi ate image generation
= G(fi);
w (z)c::
Scan += 8hj, w.r.t. j arg max(s);
// prediction consistency score // entropy difference score
Sent + = Entropy(s') - Entropy(s);
end
mwwm (( Saiv = sum((KL((]: 1));
11 diversiiy score
12/, 1- + arg max, Scon Sent + Sacii
for i=l...,KK do
f = (1. =4ff b%);;
r/ G(J');
Add x!" + D.:
end end
Output: Expanded dataset Do U D..
20

---

Under review as a conference paper at ICLR 2023
C.3 MORE IMPLEMENTATION DETAILS OF MODEL TRAINING
We implement both GIF-DALLE and GIF-MAE in PyTorch. To fairly evaluate the expansion ef- fectiveness of different methods, we use these methods expand the original small datasets by the same ratios, followed by training models from scratch on the expanded dataset with the same number of epochs and same data pre-processiie processiin. In this way, the models are trained with the same number of update steps, so that all expansion methods are fairly compared.
The expansion ratio depends on the actual demand of real applications. In the experiment of Ta- ble 1, CIIARIOO-SSus 100-Subse is expanded by 5x, Pets is expanded by 30x, and all other datasets are expanded by 20x. Moreover, the experiment of Table 3, all medical image datasets are expanded by 5x. Moreover, all augmentation baselines expand datasets with the same expansion ratio for fair comparisons. After expansion, we train ResNet-50 (He al., 2016) from scratch for 100 epochs based the expanded datasets. During model training, 224x 224 through bicubic sampling. random rotation and random flips. If not specified, we use the SGD optimizer with momentum of 0.9. We set the initial learning rate (LR) to 0.01 with cosine LR decay, except the initial LR of CIFARIOU subsee and OrganSMNIST is 0.1.
process images via random resize to
21

---

Under review as a conference paper at ICLR 2023
D DATASET STATISTICS
The statistics of natural image datasets. We evaluate our method on six small-scale natural image datasets, including Caltech-101 (Fei-Fei al., 2004),
(Krizhevsky et al., 2009), Standard Cars (Krause et al., 2013), Oxford 102 Flowers (Nilsback & Zisserman, 2008), Oxford- IIIT Pets (Parkhi et al., 2012) and DTD (Cimpoi al., 2014). Here, CIFARI00-subset is an artificial dataset for simulating small-scale datasets by randomly sampling 100 instances per class from the original CIFAR 100 dataset, and the total sample number 10,000. These datasets cover wide range of classification tasks, including coarse- -rained object classification (i.e., CIFAR 100-subset and Caltech-101 101), fine-grained object classification (i.e., Cars, Flowers and Pets) and texture clas- sification (i.e., DTD). The data statisties of these natural image datasets are given in Table 6. Note that the higher number classes or the lower number of average samples per class dataset has, the more challenging the dataset is.
Table Statistics of small-scale natural image datasets.
Datasets
Tasks
# Classes
Samples
* Average samples class
Caltech101
Coarse Course-grained object classification
102
3,060
CIFAR100-Subset
Coarse- Coarse-gruined object classification
100
10,000
100
Standard Cars
Fine-grained object classification
196
8.144
42
Oxford Flowers
Fine-grained object classification
102
6,552
Oxford-IIT Pets
Fine-grained object classification
37
3,842
104
Describable Textures (DTD)
Texture classification
47
3,760
The statisties of medical image datasets. To evaluate the effect of dataset expansion on medical images, we conduct experiments on three small-scale medical image datasets. These datasets cover wide range of medical image modalities, including breast ultrasound (i.e., BreastMNIST (Al- Dhabyani et al., 2020)), colon pathology (i.e., PathMNIST (Kather et al., 2019)) and Abdominal CT (i.e., OrganSMNIST (Xu al., 2019)). We provide detailed statistics for these datasets in Table
Table 7: Statistics of small-scale medical image datasets. To better simulate the scenario of small medical datasets, we use the validation sets of BreastMNIST and PathMNIST for experiments stead of training sets, whereas OrganSMNIST is based on its training set.
Datasets
Data Modality
Classes
Samples
# Average samples per class
HreasMNNST (Al-Dhahyani al., 2020; Yang et al., 2021)
Breast Ultrasound
2
78
PathMNIST (Katber al., 2019; Yang et al., 2021)
Colon Pathology
9
1.112 13,940
10,004
OrgSnSMNIST (Ku 2019; Yang al,, 2021)
Abdominal CT
1,267
22

---

Under review as a conference paper at ICLR 2023
E MORE EXPERIMENTAL RESULTS AND DISCUSSION
E. MORE COMPARISONS TO DATASET EXPANSION WITH AUGMENTATIONS
In Figure 4, we have demonstrated the expansion efficiency of our proposed GIF-DALLE over Cutout, GridMask and RandAugment on Cars and DTD and Pets datasets. Here, we further report the results on Caltech101, Flowers and CIFARIOR 100-Subset datasets. As shown in Figure 12, 10x expansion by GIF-DALLE outperforms x expansion of these augmentation methods lot, while 5x expansion by GIF-DALLE has already performed comparably to 20x expansion of these aug- mentation methods. This result further demonstrates the effectiveness and efficiency of our proposed method, and also reflects the importance of automatically creating informative synthetic samples for model training.
Cutout
GrdMaakk
Randkugment
MAE
GIRDDALLE (ours)
10
Oxford 132 Flowers
CIFAR1 Do-ubaat
54.5
65.0
81.3
09
Model purberanns
60
55.4
Model performmnce
05
84.9
Model performence
150
40
482
52
55.2
84
02
0.55
83.4
85.6
45
40.3
50
80
76.
81.1
73.8
40
36.
78-
77.7
45
78
77.1
+5%
+10x
+20%
+5x
10xx
200
+1x
Expansson
Expension size
Exparssoon
Figure 12: Accuracy of Re trained from scratch on the expanded datasets with different
expansion ratios based on Caltech101, Flowers and CIFAR100-Subset datasets.
E.2 MORE ABLATION STUDIES ON THE GUIDANCE
GIF optimizes sample latent features for informative sample generation by maximizing the designed objective functions, i.e., zero-shot prediction consistency Scon, prediction entropy difference Sext. and diversity promotion Saiv. In Table 1, we have demonstrated their effectiveness in guiding DALL-E2 and MAE to generate informative samples for expansion. In this appendix, we further investigate their individual influence on GIF-DALLE for expanding CIFARI00-Subsee by 5x.
As Scon and Sext are closely coupled based on CLIP's zero-shot prediction for consistent-class information promotion, we regard them as a whole. As shown in Table Scon + Sent the foun- dation of effective expansion, since it makes sure that the created samples have correct labels and bring more information. Without it, only Saie cannot guarantee the created samples to be mean- ingful, although the sample diversity is improved, leading to even worse performance. In contrast, with Spon + Sents diversity promotion Saio can further bring more diverse information to boost data informativeness and thus achieve better performance (ef. Table 8). Note that entropy maximization Sent and diversity promotion Saiv play different roles. Entropy maximization promotes the informa- tiveness of each generated image by increasing the prediction difficulty over the corresponding seed image, but this guidance cannot diversify different latent features obtained from the same image. By contrast, the guidance of diversity promotion encourages the diversity of various latent features of the same seed image, but it cannot increase the informativeness of generated samples regarding prediction difficulty. Therefore, using the two guidance together leads the expanded images to be more informative and diversified, thus bringing higher performance improvement.
Table 8: Ablation of guidance in GIF-DALLE for expanding CIFAR 100-Subset by 5x.
Method
Scon
Sent
52.8 S
CIFAR100-Subse:
52. 10.99
/
52.4 /
GIF-DALLE
$1.8
53.1
54.5
23

---

Under review as a conference paper at ICLR 2023
E.3 ANALYSIS OF DALL-E2 ON MEDICAL IMAGES
DALL-E2 is trained on a large-scale dataset consisting of natural image and text pairs, and has shown powerful capabilities in natural image generation and variation. However, when directly apply to expand medical image datasets, we find the performance improvement is limited, compared MAE as shown in Table To pinpoint the reason, we visualize the generated images on Or- ganSMNIST in Figure 13 and find that it fails to generate photo-realistic medical images. The main reason is that DALL-E2 suffers from domain shifts between natural and medical images and cannot generate photo-realistic and informative medical samples based on its image variation abilities. In contrast, MAE reconstruction model and does not need to generate new content for the target images as DALL-E2, so it has much less negative impact by domain shifts. To address the issue, when applying DALL-E2 to medical domains, we first fine-tune it on target medical datasets, fol- lowed by dataset expansion. As shown in Table 9, based on the fine-tuner DALL-E2, GIF-DALLE can bring more significantly performance gains over GIF-MAE, and thus expands medical image datasets better.
Table 9: Accuracy of ResNet-50 trained on the 5x -x-xxaadded medical image datasets by GIF based on DALLE w/o and w/ fine-tuning. The performance is averaged over 3 runs.
Datiset
PabMNIST
BeeaMMNIST
ORgeSMNIST
Figure 13: Visualization of the synthetic medi- cal images by DALL-E2 via its variation abili- ties the OrganSMNIST dataset.
Original
7244nT
76.3.n
GIF-MAE
82.0gut
GIE-DALLE twio tuiing)
78.4.10
59.3 76.6114
76.463
GIF-DALLE (w/ tuning)
84.4gs
E.4 VISUALIZATION OF CREATED MEDICAL IMAGES
We visualize the created medical samples by different methods. As shown in Figure 14, RandAug- ment randomly varies the medical images based on set of pre-defined transformations. However, due to its randomness, RandAugment may crop the lesion location of medical images and cannot guarantee the created samples be informative, even leading to noise samples. contrast, our GIF- MAE can generate while maintaining their lesion location unchanged. Therefore, GIF-MAE able to expand medical image datasets better than RandAngment, leading to higher model performance improvement (cf. Table in the main paper).
images with diverse styles, so it can enrich the medical images
Input
RandAugment
Our GIF-MAE
Figure 14: Examples of the created samples for PathMNIST by RandAugment and GIF-MAF.
24

---

Under review as a conference paper at ICLR 2023
E.5 DISCUSSION ON THE USE OF ZERO-SHOT CLIP
In GIF, we exploit the zero-shot discriminability of the pre-trained CLIP to guide dataset expansion. is interesting to know whether fine-tuning CLIP on the target small dataset can bring further improvement. To determine this, we further compare the results of GIF-MAE with fine-tuned CLIP and that with zero-shot CLIP based on OrganSMNIST. To be specific, we add an additional linear classifier on the top of the CLIP image encoder and fine-tune the CLIP model on OrganSMNIST, where the accuracy of the fine-tuned CLIP is 78.9. As shown in Table 10, GIF-MAE with fine- tuned CLIP performs only comparably to that with zero-shot CLIP. This reflects that the CLIP text classifier is enough to provide sound guidance based on its text-image matching abilities, whereas simply fine-tuning CLIP does not work better on small-scale medical datasets. Even so, we expect to further improve CLIP with more effective adaptation and prompt engineering in the future, which may better unleash the power of CLIP to guide dataset expansion.
Table 10: Comparison between the model performance by GIF-MAE expansion with zero-shot CLIP and fine-tuned CLIP based on the OrganSMNIST medical image dataset. Here, the expansion ratios are (5x, 10x, 20x). The performance is averaged over runs.
Dataset
OrganSMNIST
Original
76.3
Expanded
5x
10x
20x
with fine-tuned CLIP
80. 10.3
80.6
81.0go.5
with zero-shot CL.IP (ours)
80.6
81.1 81.1
81.2g0.4
E.6 WHY NOT DIRECTLY TRANSFER CLIP MODELS TO TARGET DATASETS?
In our proposed GIF framework, we resort to the pre-trained CLIP to guide dataset expansion. One may wonder why not to directly pre-train and fine-tune CLIP models to the target datasets, which has shown effectiveness in many natural image datasets. Before discussing this, we would like to highlight that explore dataset expansion to handle real small-data scenarios, where only small- size dataset is available without any large-scale external dataset of similar image nature. Therefore, pre-training models with CLIP or other self-supervised methods on large-scale target datasets is inapplicable. In our proposed method, we resort to publicly available CLIP models for dataset expansion. Compared directly transferring CLIP models, our dataset expansion is necessarily new paradigm for the following two key reasons.
First, our GIF method has better applicability the scenarios of different image domains. Although CLIP has strong transfer performance on some natural image datasets, its transfer performance on other domains like medical image datasets limited. Here, we report the fine-tuning performance of CLIP-traines ResNet-50 model on three medical datasets. As shown in Table 11, transferring the CLIP model only leads to limited performance gains, which are significantly worse than our dataset expansion. The reason is that, when the pre-trained datasets are highly different from the target dataset, the pre-training and fine-tuning scheme does not significantly help performance (Raghu et al., 2019). In contrast, our GIF framework can generate images of similar nature the target dataset for expansion, and thus GIF is more beneficial to real applications in various image domains.
Comparison between our methods and directly fine-tuning CLIP models on three medical image datasets. The performance is averaged over runs.
Table 11:
Dataset
PathMNIST
BreastMNIST
OrganSMNIST
Original dataset
72.4
55.8
76.3_p.4
Fine-tuned CLIP
78.4
18.9_o.1 67.2+2.4
GIF-MAE
82.0
80.6 76.6
73.3
GII-DALLE
84.4
80.5
25

---

Standard training Under review as a conference paper at ICLR 2023
Our dataset expansion can provide expanded datasets ready for training various network architec- tures. In some real application scenarios like mobile terminals, the supportable model size is very limited due to the constraints of device hardware. However, the publicly available checkpoints pro- vided by CLIP are only ResNet50, ViT-B/32 or even larger models, which may not be allowed to use in those scenarios. By contrast, the expanded dataset by our method can be directly used train various model architectures (cf. Table 2), and thus is more applicable to the application scenarios with hardware constraints. Also, once the datasets are expanded, they can be released the public to facilitate future studies.
Please note that the goal of this paper is to show the huge potential of dataset expansion as a promis- ing future direction instead of completely resolving it. We expect that the performance of dataset expansion can be further improved in future research.
E.7 EFFECTIVENESS ON LARGER LARGER-SCALE DATASETS
In previous experiments, we have demonstrated the effectiveness of our proposed method on small- scale natural and medical datasets. One may be interested whether our method can be applied to larger-scale datasets. Although expanding larger-scale datasets is not the goal of this paper, also explore our method to expand the full CIFARI00 by 5x for model training from scratch. As 8.7% accuracy gain compared to direct training on
shown in Table12, our GIF-DALLE leads the original CIFAR100 dataset. Such encouraging results verify the effectiveness of our methods on larger-scale datasets, and we expect apply our methods even larger datasets like ImageNet and other tasks in the future.
Table 12: Effectiveness of GIF-DALLE for expanding CIFAR100 by 5x.
Dataset
CIFAR100
Original
70.9
Expanded
GIF-MAE
77.0=0.3
GIF-DALLE
79.6
E.8 COMPARISON TO INFINITE DATA AUGMENTATION
In previous experiments, we have demonstrated that our proposed GIF framework is more effective and efficient than existing data augmentation methods in expanding small datasets. Despite this, one may also wonder how the explored dataset expansion would perform compared to training with infinite data augmentation. Therefore, in this appendix, we further evaluate the performance of infinite data augmentation on 10--AR0.ssuseet. Specifically, based on RandAugment, we train ResNet-50 using infinite online augmentation for varying numbers of epochs from 100 to 700. As shown in Table 13, using RandAugment to train models for more epochs leads to better performance, but gradually converges (around 51 % accuracy at 500 epochs) and keeps fluctuating afterward. By contrast, our GIF-DALLE can achieve better performance when only training 100 epochs, which further demonstrates the effectiveness of our method in generating informative synthetic data for model training.
Table 13: Comparison between GIF-DALLE and infinite data augmentation on CIFAR100-suber
Methods
Epochs
Accuracy
Original
100
35.0
Training with RundAugment
100
39.642.
Training with RundAugment
200
46.940.9
Training with RandAugment Training with RandAngment Training with RandAugment
300
48. 1
400
49.649.4
500
51.3
Training with RandAugment Training with Rand/uugent
600
51.1
700
50.641.1
Expanded
x -x-xxandded by GIF-DALLE
100
14.5
26

---

Under review as a conference paper at ICLR 2023
E.9 DISCUSSION OF PICKING RELATED SAMPLES FROM LARGER DATASETS
Picking and labeling data from larger image datasets with CLIP is an interesting idea for dataset expansion. However, such solution is limited in real applications, since a large-scale related dataset may be unavailable in many image domains. Moreover, selecting data from different image domains (e.g., from natural images medical images) is unhelpful for dataset expansion.
Despite the above limitations in real applications, we also evaluate this idea on CIFAR 100-subset and investigate whether it helps dataset expansion when there is a larger dataset of the same image nature, e.g.. ImageNet. Here, we use CLIP to select and annotate related images from ImageNet to expand SEFAR100-subsee predict them to the class of CIFAR 1u0-su We select the samples with the highest prediction probability higher than 0.1 and expand each class by 5x. As shown Table 14, the idea of picking related images from ImageNet makes sense, but performs worse than proposed method. This result further demonstrates the effectiveness and superiority of our method. In addition, how to better transfer large-scale datasets expand small datasets is interesting open question, and expect to explore it in the future.
Specifically, we scan over all ImageNet images and use CLIP to
Table 14:
Comparison between GIF and picking related data from ImageNet for expanding CIFAR 100-subset by 5x.
CIFARIO0-Subse
Accuracy
Original dataset
35.0
Expanded dataset
5 x expanded by picking data from ImageNet with CLIP
50.9
5 5x-expanded by GIF-DALLE
54.5
E. DISCUSSION OF TRAINING MODELS WITH ONLY EXPANDED IMAGES
It is interesting to know how the model would perform when trained with only the expanded images by our method. More specifically, we use only the expanded images by GIF-DALLE on CIFAR subset to train ResNet-50 from scratch, and compare it to the model trained on real images of CIFARI00-Subset. As shown in Table 15, the model trained with our synthetic images performs comparably to the model trained with real images. This result further verifies the effectiveness of our explored dataset expansion method, Moreover, the model trained with the full expanded dataset performs much better than that trained with only the original dataset with only the generated images. That is, the generated images are not a simple repetition of the original dataset, but bring new information to the expanded dataset for model training. This further shows that using synthetic images for model training is a promising direction. We expect that our work on dataset expansion can inspire more studies to explore this direction in the future.
Table 15: Performance of the model trained with only the expanded images on the 5x-expanded
dataset of CIFAR SEPAR100-subse by GIF-DALLE.
TEARR00-Subsset
Accuracy
Training with real images in original dataset Training with 5 5x-expanded dataset by GIF-DALLE
35.0
54.5+1.1
Training with only the expanded images by GIF-DALLE
35.2
27

---

Under review as a conference paper at ICLR 2023
E.11 COMPARISON TO CUTMIX FOR DATASET EXPANSION
In this appendix, we further apply a more advanced augmentation method, i.e., CutMix (Yun et al., 2019), expand
by 5 times and use the expanded dataset train the model from scratch. The results in Table 16 further demonstrate the superiority of our method over augmentation-basse expansion methods.
Table 16:
Comparison between GIF-DALLE and CutMix (Yun et al., 2019) for expanding CIFAR100-subee by 5x.
CIFAR100-Subset
Accuracy
Original dataset
35.0
Expanded dataset x expaanddd by CutMix
50.7
5 -xepandded by GIF-DALLE 54.5
28

---

Under review as a conference paper at ICLR 2023
F MORE VISUALIZATION RESULTS
This appendix provides more visualized results for the created samples by our methods on various natural image datasets. Specifically, we report the synthetic images by GIF-DALLE on Caltech101 in Figure 15 and those by GIF-MAE in Figure 16. The visualized results show that our GIF-DALLE can create semantic-consistent yet -iieent-diversifed images well, while GIF-MAE can generate content-consisten yet highly style-dive dlyeediiersified images. In addition, the visualization of GIF-DALLE on other natural image datasets are shown in Figures 17-21.
VISUALIZATION OF THE EXPANDED IMAGES ON CALTECH
Input
Our GIF-DALLE expansion
Figure 15: Visualization of the created samples on Caltech101 by GIF-DALLE.
29

---

Under review as a conference paper at ICLR 2023
Input
Our GIF-MAE expansion
XXXIX
Figure 16: Visualization of the created samples on Caltech101 by GIF-MAE.
30

---

Under review as a conference paper at ICLR 2023
F.2 VISUALIZATION OF THE EXPANDED IMAGES ON CARS
Input
Our GIF-DALLE expansion
7330
Figure 17: More visualization the synthetic samples on Cars by GIE-DALLE.
31

---

Under review as a conference paper at ICLR 2023
F.3 VISUALIZATION OF THE EXPANDED IMAGES ON FLOWERS
Input
Our GIF-DALLE expansion
Figure 18: More visualization of the synthetic samples on Flowers by GIF-DALLE.
32

---

Under review as a conference paper at ICLR 2023
F.4 VISUALIZATION OF THE EXPANDED IMAGES ON PETS
Input
Our GIF-DALLE expansion
Figure 19: More visualization of the synthetic samples on Pets by GIF-DALLE.
33

---

Under review as a conference paper at ICLR 2023
F.5 VISUALIZATION OF THE EXPANDED IMAGES ON CIFAR CIFAR1OO-SORR OO-SUSETTTT
Input
Our GIF-DALLE expansion
Figure 20: More visualization of the synthetic samples on CIFARI00-SULE 100-Subset by GIF-DALLE. Note
that the resolution of CIFARIO0 100 images is small (i.e. 32x32), so their visualization is little unclear.
34

---

Under review as a conference paper at ICLR 2023
F.6 VISUALIZATION OF THE EXPANDED IMAGES ON DTD
Input
Our GIF-DALLE expansion
Figure 21: More visualization of the synthetic samples on DTD by GIF-DALLE.
35

---

Under review as a conference paper at ICLR 2023
G EXPLORATORY THEORETICAL ANALYSIS
In this work, we explore the empirical possibility of using recent generative models (e.g., DALL-E2) for small dataset expansion, and we have empirically demonstrated the effectiveness of our proposed GIF method in previous experiments. Although theoretical analysis is not the focus of this paper, it also interesting theoretically analyze the proposed dataset expansion. However, rigid theoretical analysis for dataset expansion is complex and non-trie non-trivial, so here we provide an exploratory idea to analyze the benefits of our dataset expansion to generalization performance.
Inspired by Sener & Savarese (2017), we resort to the concept of 8 cover to analyze how the diversity of data influences the generalization error bound. Specifically, "a dataset is 5 cover of a dataset 8" means set of balls with radius 8 centered at each sample of the dataset 8 can cover the entire dataset In our work, we only consider the small target dataset and its true data distribution, so we follow the assumptions of the work (Sener & Savarese, 2017) and extend its Theorem I to the version of the generalization error bound.
Corollary Let A be a learning algorithm and C be a constant. Given a training set = {2i, Ye hetn with *Li.d * samples drawn from the true data distribution Pz. 1fe training set 'D is cover of the true distribution Pz, the hypothesis function is -Lissscciitz continuous, the loss function [(x, y) is X -ipiscchitt continuous for all y and bounded by L, and (( A) for Vi E [n]. with the probability at least 1 7,, the generalization error bound satisfies:
ChayP=l((x,,) y: A)]
Σ iE(m)
E(a,, yi: A) 58(x XTTC).
(6)
7
This corollary shows that the generalization error is bounded by the covering radius 6. In real small- data applications, the data limitation issue leads 5 to very large and thus severely affects the generalization performance of the trained model. More critically, simply increasing the data number (e.g., via data repeating) does not help the generalization since it does not decrease the covering radius 6. Rather than simply increasing the number of samples, our proposed GIF framework guides recent generative models (e.g., DALL-E2) to synthesize informative and diversified new images for expanding the original small dataset. Therefore, the expanded dataset would have higher diver- sity of data, which helps to decrease the covering radius and thus improves the trained model's generalization performance.
36


Reviews:
[{"review_id": "xItaWAvEqyZ", "reviewer": "Reviewer_cRYy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work improves the performance of low-shot learning by expanding the small-scale dataset by utilizing powerful generative models (i.e. decoders). To expand the dataset in a more meaningful way, this work proposes three conditions to ensure that the synthetic samples should be consistent with the real samples in zero-shot prediction perspective and diverse enough to improve the final performance of the models trained on the expanded dataset. Experiments on the proposed methods based on MAE and DALL-E 2 (without upscaling modules) show that this expansion policy indeed improves the classification performance in a low-shot setting. ", "summary_of_the_review": "I’m leaning towards rejection, since I’m not sure that the performance gain achieved by this work is really significant or not, compared to the simple recipes used in many SSL methods. ", "strength_and_weaknesses": "**Strengths**:\n\n* Expanding the dataset based on a powerful decoder is a very intuitive and interesting direction to few-shot learning. In addition, this work proposes three criteria on how to create effective synthetic samples. And, these have been empirically validated in several experiments.  \n* The appendix helps me understand the details of the algorithm to reproduce the results. And, all experiments have been conducted on publicly available checkpoints, so the accessibility of this approach is quite high. \n\n**Weaknesses**: \n\nMy biggest concern in this work is the low performance of the main experiments, shown in Table 1. Compared to the transfer learning or linear evaluation results from ALIGN and CLIP, the numbers in Table 1 aren’t convincing enough to argue that this kind of data expansion policy is practically useful than the standard protocol, i.e. training an encoder by SSL objectives then fine-tuning the encoder on small-scale datasets. \n\nI summarize the performance gap between this method and CLIP and ALIGN below: \n\n| Method | Caltech101 | Cars | Flowers | Pets | \n| ---------- | -------------- | -------------- | -------------- | -------------- | \n| GIF-MAE (proposed one) | 58.4 | 44.5 | 84.4 | 52.4 |\n| GIF-DALLE2 (proposed one) | 63.0 | 53.1 | 88.2 | 66.4 | \n| CLIP (ViT-B/32) - linear eval.    | 93.0  | 81.8  | 96.9 | 90.0 |\n| CLIP (ViT-L/14, 334px) - linear eval.    | 96.0 | 91.5 | 99.2 | 95.1 |\n| ALIGN - transfer learning | - | 96.1 | 99.7 | 96.2 | \n\nALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision, https://arxiv.org/abs/2102.05918\n\n**More questions**\n\nQ1. In my understanding, the roles of the entropy maximization and diversity promotion seem to be very similar to each other. Could you elaborate a little bit more on the reason why both conditions are required instead of using the diversity promotion? \n\nQ2. In table 1, GIF-DALLE consistently performs better than GIF-MAE, though the resolution of the synthetic samples based on GIF-MAE is much higher than the ones on GIF-DALLE (224px vs. 64px.), as described in Appendix C.1. It would be helpful to describe the reason why GIF-DALLE performs better than the MAE in the revised manuscript. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R0GUJyPP-QH", "reviewer": "Reviewer_nKUi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to tackle a new task of dataset expansion, which is to automatically find new labeled data to augment an existing dataset.\nTo do so, they propose to generate them using a generative model (DALL-E2) using CLIP guidance.\nThe idea is that using DALL-E2 would be able to generate variations around existing samples, and CLIP can be used to guide the generated samples to only generate useful data.\nIn addition to DALL-E2, they show they can also use Masked Auto-Encoders (MAE) as a backbone, with the same CLIP guidance.\nThey compare their method of expansion against standard data-augmentation techniques on a set of small natural image datasets, as well as medical image datasets.", "summary_of_the_review": "The paper tackles an interesting problem in a novel way, showing promising results. It contains a lot of information, shedding light on the capabilities of current generative models and text embedding to synthesize training data.\n\nMy two main concerns are 1) about the integration of the GIF-MAE model in the narrative of the paper and 2) that the current experiments are not sufficient to be convinced about the usefulness of the method in practice. Additionally, I would also like some discussions regarding the limitations of the method in the main paper.", "strength_and_weaknesses": "Strengths:\n- The paper tackles a very relevant problem of training in a low-data regime. Using generative models for this purpose is an intuitive solution that shows promises but has so far been struggling to obtain significant success. The paper is a step in that direction.\n- The diverse set of experiments provides good insights into the method. I found the results on medical images and the corresponding discussion in Appendix E regarding the limitations of GIF-DALLE especially important, and believe there should be more discussions in the main paper on that subject.\n\nWeaknesses:\n- It is not clear how GIF-MAE fits in the narrative.\n  - In most sections the main focus is GIF-DALLE and the MAE variant is barely mentioned, almost like an afterthought. In the experiments, however, it takes a very significant portion.\nThe authors might want to fix this imbalance. It could be done by integrating GIF-MAE earlier in the motivation, or instead by treating it even more entirely as a side contribution in the later sections.\n  - The sentence in the introduction \"Considering that DALL-E2 and MAE (He et al., 2022) have been shown to be powerful in generating and reconstructing images, we explore their use as prior generative models for imagination in this work.\" seems particularly misleading. Can the authors clarify if they consider MAE a powerful image generator, and if so, in which sense?\n- The framing of the results is useful but seems artificial. As such the presented results are insufficient to draw conclusions regarding the usefulness of the method in practice.\n  - Comparing their methods against standard data augmentations with a fixed number of samples is fine, but it would be useful to have the performance of the baselines with a virtually infinite number of augmented samples.\nIndeed, a key difference between the two approaches is that, in practice, data generation has significant memory and a computation footprint while it should be very easy to add infinite standard data augmentation in the training pipeline with little cost.\n  - This comparison might also not be fair as the GIF-DALLE has had access to Laion-400M. Another interesting baseline would have been to simply automatically select and label additional unlabelled data using CLIP embeddings. This could also serve as some sort of ablation study to check if the generator is actually useful, or if directly using the raw dataset with CLIP pseudo-labeling is enough.\n\nQuestion:\n- It is mentioned that GIF-DALLE generates images of size 64x64 (Appendix C.1) while the downstream models use inputs of size 224x224 (Appendix C.3). These numbers seem off considering the shown samples. Can the authors confirm these numbers are used for all datasets?  If so, how exactly are the outputs of GIF-DALLE upsampled and processed, especially with regard to the random cropping mentioned in C.3?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ob-4VQ487F", "reviewer": "Reviewer_sxmr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Guided Imagination Framework (GIF) for generating more data samples for small datasets. The proposed model generates more image samples using two large-scale models trained with very large datasets (e.g., CLIP ViT-B/32 trained with private 400M image-text pairs and Dall-E2 decoder trained with LAION-400M). To generate new images, GIF first encodes the reference image using the CLIP ViT-B/32 encoder, and decodes a perturbed version of the encoded feature using the Dall-E2 decoder. This paper also proposes three optimization objectives based on CLIP features (both image and label text features) to seek the best perturbations for generating new samples: (1) zero-shot prediction consistency (2) entropy maximization and (3) diversity promotion. The experiments on six small natural image datasets and three small medical image datasets show that the proposed GIF shows improvements in the small dataset regime.", "summary_of_the_review": "This paper is well-written, and the method looks fairly novel. However, the performance of the proposed method is much worse than CLIP zero-shot or linear probing, where they even use much fewer computational resources than the proposed method (no need to generate images, no need to tune the full models for x20 data samples). I think the empirical contribution and the practicalness of this paper are not enough to be accepted at the ICLR conference.\n\n---\n\nDuring the reviewer-author discussion period, the authors provide more information than the original paper. For example, the authors showed that the CLIP zero-shot is worse than the proposed method in some medical domains. However, as my following comments, I don't think the current version of the paper is not enough to be accepted at the ICLR conference. Even if we consider the additional content provided during the discussion period, the expected revision will be very significant (the main message in introduction will be revised, the main table will be revised, following explanations and the analysis will be revised); another review process will be required.\n\nI am also not convinced by the additional experimental results. Especially, I am not positive to the additional experiments on medical domains because CLIP zero-shot shows almost random (10%) accuracy, while the quality check of the proposed generative process is done by the low-performance CLIP zero-shot. I cannot agree with the following comment\n> \"Although the zero-shot performance of CLIP is not good, its feature space is already relatively discriminative (given that CLIP linear-probing performance is good)\" \n\nThe proposed method uses the zero-shot performance (zero-shot prediction consistency) for the \"informativeness guidance\". If the CLIP model has no power to discriminate the \"ground-truth\" images, then how can it guide whether the generated images are informative or not? Overall, I think this paper will need additional careful discussions related to my initial concerns.", "strength_and_weaknesses": "## Strength\n\n- This paper leverages the knowledge from the models trained with large-scale datasets by generating more data samples\n- Compared to previous data augmentation methods, the generated images look more \"realistic\" and seem helpful\n- This paper provides enough analysis to show the effectiveness of the proposed method (e.g., various architectures, where to optimize, pixel-wise vs. channel-wise noise, ...)\n- The paper is well-written and easy to follow.\n\n## Weakness\n\n### Comparison with CLIP zero-shot / linear-probe results\n\nI think it is the most critical weakness of this paper. Note that the proposed method heavily relies on CLIP ViT-B/32 zero-shot performance and Dall-E2 decoder where both models are trained with 400M image-text pairs (CLIP is trained with private samples and Dall-E2 decoder is trained with LAION-400M, respectively). I checked the zero-shot performance of CLIP ResNet-50 and ViT-B/32 models and found that the proposed method is much worse than CLIP zero-shot performances (from the original CLIP paper [R1] Table 17):\n\n| Methods                         | Caltech | Cars | Flowers | DTD | CIFAR100-S | Pets | Avg. |\n|-----------------------------|-----------|------|----------|-------|---------------|-------|------|\n| GIF-Dall-E (ResNet-50)   | 63.0       | 53.1 | 88.2.     | 39.5  | 54.5            | 66.4  | 60.8 |\n| CLIP RN50 Zero-shot     | 82.1       | 55.8 | 65.9      | 41.7 | 41.6             | 85.4  | 62.1 |\n| CLIP ViT-B/32 Zero-shot | 87.9      | 59.4  | 66.7     | 44.5 | 65.1              | 87.0  | 68.4 |\n| CLIP ViT-B/32 Linear probe | 93.0 | 74.9  | 96.9     | 76.5  | 80.5 (full data) | 90.0 | 85.3 |\n\nNote that directly comparing CLIP RN50 and the conventional RN50 could be unfair because CLIP ResNet-50 is not exactly the same as the conventional ResNet-50 (e.g., attention pooling, antialias pooling, ...). However, this paper uses ViT-B/32 CLIP for generating new images, and it outperforms the proposed method (except Flowers). Furthermore, if we allow access to the training datasets, then the CLIP model outperforms GIF, CLIP zero-shot with a significantly large gap (85.3 vs. 60.8 in the average accuracy). I attached CLIP ViT-B/32 linear probe results from Table 10 of the original CLIP paper [R1]. Note that the linear probe result for CIFAR100-S is made by the full training dataset, but it is known that the linear probe is less sensitive to the dataset size than the full model fine-tuning. However, I acknowledge that the comparison is not fair for CIFAR100-S (zero-shot results are fair because they are not trained with the training dataset).\n\nWhy do we need dataset expansion if CLIP zero-shot performance (or linear probe) is better than dataset expansion? If we just use CLIP zero-shot predictions, then (1) we do not need Dall-E2 or MAE decoders (2) we do not need to generate more image samples (3) we do not need to train models on the generated images. If we allow (3), the CLIP linear probe shows significant results.\n\nI do not have CLIP results for the three medical datasets, but unless CLIP results are very poor for the medical datasets, I do not think the proposed scenario is not practical.\n\n### Too heavy computation resources\n\nEven if we assume that the big models trained with large-scale datasets (e.g., CLIP encoder, Dall-E2, or MAE decoder) are publicly accessible from the web, so that no extra resource is required, the proposed method still needs heavy computations to (1) generate images (2) train with x20 images. Note that the data augmentation methods usually assume the number of the total iterations is not changed, e.g., the vanilla training and Cutout augmentation for the given training dataset needs the same iterations. Following the terminology of this paper, it is common to use x1 for data augmentation. Data augmentation with xN is often called \"repeated augmentation\" [R2]. Usually, when we use repeated augmentation, it is not recommended to use \"epochs\" because a model is updated N times more using SGD optimizer, and it is known that more update brings a better performance [R3].\n\nTo summarize, the proposed method needs three additional heavy computation resources (1) training big models with large-scale datasets -- it could be ignored if we only consider image datasets, but if we consider other domains, it limits the usage of the method (2) generating images using the CLIP encoder, Dall-E2 or MAE decoder with optimized perturbations (3) the dataset expansion itself amplifies the total computational resource if we do not limit the number of iterations (i.e., x20 needs 20 times more computations than x1).\n\n### References\n\n[R1] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International Conference on Machine Learning. PMLR, 2021.\n[R2] Touvron, Hugo, et al. \"Training data-efficient image transformers & distillation through attention.\" International Conference on Machine Learning. PMLR, 2021.\n[R3] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n### Minor comments\n\n- It looks that the bib for the COYO dataset in page 1 is broken (\"Minwoo Byeon, 2022\").\n- It could be interesting if there exists the analysis of dropping the original images, but only using the augmented images (i.e., using only $\\mathcal D_S$ in Problem statement (Section 3)\n- The details of $S_{con}$ $S_{ent}$ $S_{div}$ could be more well described, e.g., showing the exact formulations using `\\equation`\n    - The formulation for the entropy minimization is not clear. As far as the reviewer understood, it calculates the entropy of the predicted probability (e.g., in this case, normalized cosine similarity). I cannot understand how encouraging higher prediction entropy (i.e., showing a more \"uniform\" prediction) helps the optimization. Is there any guess from the authors?\n    - The formulation for the $S_{div}$ is not clear. How can we compute KL between features? I presume that it is achievable by optimizing the lower bound of KL divergence (as VAE)\n- I wonder if the method works for larger datasets, such as CIFAR-100 (full dataset) or ImageNet-1K. Especially, I wonder about CIFAR-100 results. ImageNet-1K would not be necessary considering the scope of this paper.\n- [Not necessary] It could be interesting if there are more comparisons with more advanced image augmentation methods, such as mixed sample data augmentation (e.g., Mixup, CutMix).\n- [Minor] I suggest clarifying that the method is designed for \"a small dataset\" from the title, e.g., \"Expanding Datasets With Guided Imagination for a small dataset\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3BMxeVsM1Gu", "reviewer": "Reviewer_DLM1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a Guided Imagination Framework (GIF) for data augmentation. The original small-set dataset can be expanded by generating new images from powerful generative models such as DALL-E or MAE. Compared with the non-parametric data augmentations, such generative-models-guided ones bring extra new information due to the nature of the models themselves. Diversity and fidelity are the two crucial factors for success. This paper proposed three criteria as guidance: prediction consistency, entropy maximization, and diversity promotion. GIF is verified on multiple natural or medical image datasets with a stable boost compared with the baselines.", "summary_of_the_review": "Despite many concerns about this paper, I am still positive due to its novelty and the promising results in many small datasets.", "strength_and_weaknesses": "Pros:\n+ The setting of this paper is new and interesting. A similar idea has been applied in the GAN models, which seems not to be working. The latest progress in diffusion models is significant. It has shown a great enhancement in the quality of synthetic images. Applying such generative models to help discriminative tasks is a natural and straightforward idea. To the best of my knowledge, this is a pioneering work in this area.\n\n+ This paper has achieved very promising results over many benchmarks, with a 29.9% accuracy gain on average over six natural image datasets and a 10.4% accuracy gain on average over three medical ones. Such an improvement is significant compared with non-parametric data augmentations.\n\n+ With the help of three criteria as guidance, there is a further accuracy gain due to the increase of new information in generated data. The proposed disturbance is simple and effective.\n\nCons:\n- One of my major concerns is that all the evaluations are conducted over small datasets where MAE or DALL-E should easily cover such domains. I am not very surprised by the improvement in these datasets. I am wondering if this will be helpful for other tasks/domains on a larger scale. \n\n- There is no sufficient theoretical analysis in this paper. The whole paper is based on empirical studies with solid experiments whose quality can be further improved with more discussion of analysis in the theoretical aspect.\n\n- Despite the expansion efficiency of GIF claimed in Sec. 5.1, I am doubtful about the time and cost efficiency due to the low speed and high computation of the diffusion model in inference.\n\nQuestions and Other Concerns:\n1. I am wondering what is the open-source plan of this paper since the DALL-E is not an open-sourced model.\n\n2. It would be good if the authors shared the time and computational costs needed for dataset expansion.\n\n3. The images generated by MAE seem to have bad visual quality. How to explain the benefits of such noisy images for augmentation? Is it any possibility that they could be harmful to accuracy if the image quality is pretty bad and why?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject