IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION?
Authors: Ruifei He
Shuyang Sun
Xin Yu
Chuhui Xue
Wenqing Zhang
Philip Torr
Song Bai
XIAOJUAN QI
Venue: ICLR 2023 notable top 25% (2023)
Keywords: data generation
image recognition
text-to-image synthesis

Abstract: Recent text-to-image generation models have shown promising results in generating high-fidelity photo-realistic images. Though the results are astonishing to human eyes, how applicable these generated images are for recognition tasks remains under-explored. In this work, we extensively study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks, and focus on two perspectives: synthetic data for improving classification models in the data-scare settings (i.e. zero-shot and few-shot), and synthetic data for large-scale model pre-training for transfer learning. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks. Code: https://github.com/CVMI-Lab/SyntheticData. 

Published as a conference paper at ICLR 2023
IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION?
Ruifei He!
Shuyang Sun
Xin Yu!
Chuhui Xue
Wenqing Zhang'
Philip Torr
Song Bai Xiaojuan Qili
'The University of Hong Kong Universityy of Oxford 'ByteDance
ABSTRACT
Recent text-to-ing text-to-image generation models have shown promising results in gener- ating high-fidelity photo-realistic images. Though the results are astonishing to human eyes, how applicable these generated images are for recognition tasks re- mains under-explored. In this work, we extensively study whether and how syn- thetic images generated from state-of-the-un text-to-image generation models can be used for image recognition tasks, and focus on two perspectives: synthetic data for improving classification models in data-scarce settings (i.e. zero-shot and few- shot), and synthetic data for large-scale model pre-training for transfer learning. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks. Code:
INTRODUCTION
Over the past decade, deep learning powered by large-scale annotated data has revolutionized the field of image recognition. However, it is costly and time-consuming to manually collect a large- scale labeled dataset, and recent concerns about data privacy and usage rights further hinder this process. In parallel, generative models that aim to model real-data distributions can now produce high-fidelity photo-realistic images. In particular, recent text-to-image generation models (Nichol et al., 2021; Ramesh et al., 2022; Saharia et al., 2022b) have made major breakthroughs in synthe- sizing high-quality images from text descriptions. This promotes ask: is synthetic data from generative models ready for image recognition tasks?
There are a few early attempts at exploring synthetic data from generative models for image recogni- tion tasks, Besnier et al. (2020) use a class-conditional GAN (BigGAN (Brock et al., 2018) trained for ImageNet- 1000 classes) to generate images for training image classifiers. Zhang al. (2021) leverage StyleGAN (Karras et al., 2019) to produce synthetic labeled data for object-part segmen- tation. Jahanian et al. (2021) manipulate the latent space of a GAN model to produce multi-view images for contrastive learning. Albeit promising, early works either address tasks small scale or only for a specific setting. Plus, they all focus on GAN-based models and none explore the revo- lutionary text-to-iman image generation models, which hold more promises to benefit recognition tasks.
In this paper, we present the first study on the state-of-the-art text-to-image generation models for image recognition. With the power of text-to-image generation, we could hopefully not only gener- ate massive high-quality labeled data, but also achieve domain customization by generating synthetic data targeted for specifie label space, i.e. the label space of downstream task. Our study is car- ried out on one open-sourced text-to-image generation model, GLIDE (Nichol et al., 2021) !. We attempt uncover the benefits and pitfalls of synthetic data for image recognition through the lens of investigating the following two questions: 1) is synthetic data from generative models ready for improving classification models? 2) whether synthetic data can be a feasible source for transfer learning (i.e. model pre-training)? It is worth noting that for 1), we only studied the zero-shot and few-shot settings because the positive impact of synthetic data diminishes as more shots are present. And, we build most of our investigations on the of-te-of-the-rr method CLIP (Radford et al., 2021) with the feature extractor initialized with large-scale pre-trained weights frozen.
Part of the work is done during internship ByteDance. Email: ruifeihe@ hkc.hku.hk
Corresponding authors: songhai. site @ gma, xjqi sogi@eee.hku.hk hku.hk
At the beginning of this project, GLIDE is the only open-sourced text-to-image synthesis model that also delivers high-quality synthesis results.

---

Published as a conference paper at ICLR 2023
Our Findings. First, in the zero-shot setting, i.e. no real-world data are available, we demonstrate that synthetic data can sigmificantly improve classification results on 17 diverse datasets: the per- formance is increased by 4.31% in top-1 accuracy on average, and even improved by as much as 17.86% on the EuroSAT dataset. To better leverage synthetic data in this setting, we also investigate useful strategies to increase data diversity, reduce data noise, and enhance data reliability. This is achieved by designing diversified text prompts and measuring the correlation of text and synthesized data with CLIP features.
Second, in the few-shot setting, i.e. a few real images are available, albeit not as significant as in the zero-shot task, synthetic data are also shown to be beneficial and help us achieve new state of the art. Our observation shows that the domain gap between synthetic data and downstream task data is one challenge on further improving the effectiveness of synthetic data on classifier learning- Fortunately, in this setting, the accessibility of real data samples can provide useful information about the data distribution of the downstream task. We thus propose to use real images as guidance in the generation process to reduce domain gaps and improve effectiveness.
Third, in large-scale model pre-training for transfer learning, our study shows that synthetic data are suitable and effective for model pre-training, delivering superior transfer learning performance and even outperforming ImageNet pre-training. Especially, synthetic data work surprisingly well in unsupervised model pre-training. and favor ViT-based backbones. We also demonstrate that by increasing the label space (i.e. text prompts) for data generation, the enlarged data amount and diversity could further bring performance boosts. Besides, synthetic data work collaboratively with real data (i.e. ImageNet) where we obtain improved performance when the model is initialized with ImageNet pre-trained weights.
2 RELATED WORKS
Synthetic Data for Image Recognition. There are mainly two forms of synthetic data for image recognition, iee. 1) synthetic datasets generated from a traditional simulation pipeline; 2) synthetic images output from generative models.
The first type, synthetic datasets (Dosovitskiy et al., 2015; Peng et al., 2017; Richter et al., 2016), are usually generated from traditional pipeline with a specific data source, e.g.synthetic 2D renderings of 3D models or scenes from graphics engines. However, this traditional way of generating synthetic datasets has several drawbacks: 1) manually defined pipeline generated synthetic data may have a certain gap with real-world data; 2) taking up huge physical space to store and huge cost to share and transfer; 3) data amount and diversity bounded by the specific data source.
Compared with synthetic datasets, generative models are more efficient means of synthetic data representation, exhibiting favorable advantages: 1) could produce high-fidelity photorealistic im- ages closer to real data since they are trained on real-world data; 2) highly condensed compared to synthetic data itself, and take up much reduced storage space; 3) potentially unlimited synthetic data size. Only recently, few works attempt to explore synthetie data generated from generative models for image recognition. Besnier et al. (2020) use a class-conditional GAN to train classifiers of the same classes. Zhang al. (2021) leverage the latent code of StyleGAN (Karras et al., 2019) to produce labels for object part segmentation. While they achieve promising results, both works are task-wise and only employed on small scale. Jahanian et al. (2021) use GAN-based generator to generate multiple views to conduct unsupervised contrastive representation learning. These works, however, explore upon the traditional GAN-based models; in contrast, our work investigates with the best released text-to-ing -maae generation model, which demonstrates new customization ability for different downstream label space.
Text-to-Imee Image Diffusion Models. Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Nichol & Dhariwal, 2021) have recently emerged as class of promising and powerful generative models. As a likelihood- based model, the diffusion model matches the underlying data distribution ((xo) by learning to reverse noising process, and thus novel images can be sampled from a prior Gaussian distribution via the learned reverse path. Because of the high sample quality, good mode coverage and promising training stability, diffusion models are quickly becoming a new trend in both unconditional (Ho et al., 2020; Nichol & Dhariwal, 2021; Ho et al., 2022) and conditional (Dhariwal & Nichol, 2021; Rombach et al., 2022: Lugmayr et al., 2022; Saharia et al., 2022a; Meng et al., 2021; Saharia et al., 2022c) image synthesis fields.
2

---

Published as a conference paper at ICLR 2023
In particular, text-to-image generation can be treated as a conditional image generation task that requires the sampled image to match the given natural language description. Based upon the for- mulation of the diffusion model, several text-to-image models such as Stable diffusion (Rombach et al., 2022), DALL-F2 (Ramesh et al., 2022), Imagen (Saharia et al., 2022b) and GLIDE (Nichol al., 2021) deliver unprecedented synthesis quality, largely facilitating the development of the AI- for-Art community. Despite achieving astonishing perceptual results, their potential utilization for high-level tasks is yet under-explored. In this paper, we utilize the state-of-the-ar model GLIDE and showcase its powerfulness and shortcomings for synthesizing data for recognition tasks.
IS SYNTHETIC DATA READY FOR IMAGE RECOGNITION?
In the following sections, we answer the question by studying whether synthetic data can benefit recognition tasks and how to better leverage synthetic data to address different tasks. We carry out our exploration through the lens of two basic settings with three tasks: synthetic data for improving classification models in the data-scarce setting (i.e. zero-shot and few-shot) (see Sec. 3.1 and Sec. 3.2) and synthetic data for model pre- tre-trainin for transfer learning (see Sec. 3.3).
Model Setup for Data-scarce (i.e. Zero-shot and Few-shot) Image Classification. As CLIP (Radford et al., 2021) is the state-of-the-ar approach for zero-shot learning, we conduct our study for zero-shot and few-shot settings upon pre-trained CLIP models, aiming to better understand synthetic data upon strong baselines. There have been a few attempts on better tuning pre-trained CLIP for data-scarce image classification, such as CoOp (Zhou et al., 2022b), CLIP Adapter (Gao et al., 2021), and Tip Adapter (Zhang et al., 2022), where the image encoder is frozen for better preserving the pre- trained feature space. We argue that different tuning methods could all be regarded as different ways of learning classifier weights, e.g. CoOp optimizes learnable prompts for better learning classifiers.
Here, we adopt a simple tuning method, Classifier Tuning (CT), a baseline method introduced in Wortsman et al. (2022). Concretely, for k-way classification, we input the class names C = {c1, (c.,..,,+]] with prompt 8; = "a photo of a {ci}" into the text encoder of CLIP to obtain the text features h(s;) Then the text features h(s;) could be used to construct classifier weights W e Raxk where d is the dimension of text features. Finally, we combine the image encoder g with the classifier weights W to obtain a classification model f(x) g(x)"W. We empirically show that CT performs comparably with other tuning methods. Compared with complex designed tuning methods, we hope use simpler method for better investigating the effectiveness of synthetic data.
3.1 IS SYNTHETIC DATA READY FOR ZERO-SHOT IMAGE RECOGNITION?
Our aim is to investigate to what degree synthetic data are beneficial to zero-shot tasks and how to better leverage synthetic data for zero-shot learning.
Zero-shot Image Recognition. We study the inductive zero-shot learning setting where no real training images of the target categories are available. CLIP models are pre-trained with large-scale image-caption pairs, and the similarities between paired image features (from an image-encoder g) and text features (from a text-encoder h)) are maximized during pre-training. The pre-trained feature extractor can then be used to solve zero-shot tasks where given an image, its features from are compared with text features of different classes from and the image is further assigned the class that has the largest similarity in the CLIP text-image feature space.
Synthetic Data for Zero-shot Image Recognition. Though CLIP models exhibit strong zero-shot performance thanks to the large-scale vision- dataset for pre-training, there are still several shortcomings when the model deployed for downstream zero-shot classification task, which may be attributed to unavoidable data noise in CLIP's pre-training data or the label space mismatch between pre- preitrann and the zero-shot task. Hence, with given label space for a zero-shot task, we study whether synthetic data can be used to better adapt CLIP models for zero-shot learning.
How to generate the data? Given a pre-trained eott-to-image generation model, to synthesize novel samples, the basic (B) strategy is to the label names of the target categories to build the language input and generate a corresponding image. Then, the paired label names and synthesized data can be employed to train the classifier with the feature extractor frozen.
How to enrich diversity? Only using the label names as inputs might limit the diversity of synthe- sized images and cause bottlenecks for validating the effectiveness of synthetic data. Hence,
3

---

Published as a conference paper at ICLR 2023
Dataset
Task
CLIP-RN50
CLIP-RN50+SYN
CLIP-VIT-B/16
CLIP-WIT-B/G+-SYN
CIFAR-IO
70.31
80.06 (+9.75)
90.80
02.37(++.55)
CIFAR-100
35.35
45.69 (+10.34)
68.22
70.71 (+2.49)
101
0
86.09
87.74 (+1.65)
92.98
94.16 (+1.18)
C5ltech256
73.36
75.74 (42.38)
80.14
(+1.29)
ImageNet
60.33
60.78 (+0,45)
68.75
69.16 (+0.41)
SUN397
58.51
60.07 (+1.56)
62.51
63.79 (+1.28)
Aircraft
17.34
21.94 (+4.60)
24.81
30.78 (+5.97)
Birdsnap
34.33
38.05 (43.72)
41.90
46.84 (+4.94)
Cars
f
55.63
56.93 (+1.30)
65.23
66.86 (+1.63)
CUB
f
46.69
(410.22
55.23
63.79 (+8.56)
Flower
f
66.08
67.05 (+0.97)
71.30
72.60 (+1.30)
Food
80.34
80.35 (+0.01)
88.75
88.83 (+0.08)
Pets
85.80
86.81 (+1.01)
89.10
90.41 (+131)
DTD
42.23
43.19 (+0.96)
44,39
44.92 (+0,53)
EuroSAT
si
37.51
55.37 (+17.86)
47.77
59.86 (+12.09)
ImageNet-Sketch
33.29
36.55 (+3.26)
46.20
48,47 (+2.27)
ImageNet-R
56.16
59.37 (43.21)
74.01
76.41 (+2.40)
Average
55.13
59.47 (+4.31)
65.42
68.32 (+2,90)
Table 1: Main Results on Zero-shot Image Recognition. All results are top-1 accuracy test set.
o: object-level. st scene-level. f: fine-graie grained. t: textures. si: satellite images. r: robustness.
leverage an off-the-shelf word-to-sentence TS model (pre-trained on "Colossal Clean Crawled Cor- pus" dataset (Raffel et al., 2020) and finetuned on CommonGen dataset (Linn et al., 2019)) to increase the diversity of language prompts and the generated images, namely language enhancement (LE), hoping to better unleash the potential of synthesized data. Concretely, we input the label name of each class to the word-to-sentence model which generates diversified sentences containing the class names as language prompts for the extttto-image generation process. For example, if the class label is "airplane", then the enhanced language prompt from the model could be "a white airplane hover- ing over a beach and a city"" The enhanced text descriptions introduce rich context descriptions.
How reduce noise and enhance robustness? ? It's unavoidable that the synthesized data may contain low-quality samples. This is even more severe in the setting with language enhancement as it may introduce undesired items into language prompts (see Figure A.2 in Appendix for visual examples). Hence, we introduce a CLIP Filter (CF) strategy to rule out these samples. Specifically, CLIP zero-shot classification confidence used to assess the quality of synthesized data, and the low- confidence ones are removed. Besides, as soft-target is more robust than hard-target in countering sample noise, we study whether soft cross-entropy loss (SCE, see Sec. C.4 in Appendix) which uses the normalized clip scores as a target could be used to enhance robustness against data noise.
Experiment Setup. We select 17 diverse datasets covering object-level (CIFAR- 10 and CIFAR-100 ((Krizhevsky et al., 2009). Caltech101 (Fei-Fei et al., 2006), Caltech256 (Griffin et al., 2007). Im- ageNet (Deng et al., 2009)), scene-level (SUN397 (Xiao et al., 2010)), fine-grained (Aircraft (Maji al., 2013), Birdsnap (Berg et al., 2014), Cars (Krause al., 2013), CUB (Wah et al., 2011), Flower (Nilsback & Zisserman, 2008), Food (Bossard et al., 2014), Pets (Parkhi et al., 2012)), textures (DTD (Cimpoi al., 2014)). satelite images (EuroSAT (Helber et al., 2019)) and robustness (ImageNet- Sketch (Wang et al., 2019), ImageNet-R (Hendrycks et al., 2021)) for zero-shot image classification. For synthetic data amount, we generate 2000 (study of synthetic image number in Appendix Sec. B.3) synthetic images for each class in B and LE. For LE, we generate 200 sentences for each class.
Main Results: 1) zero-shot classification results on 17 datasets; study of synthetic data diversity;
3) study of synthetic data reliability; 4) study of model/classifier tuning; 5) study of the behavior of synthetic data for zero-shot classification in the training from scratch settings.
Synthetic data can significantly improve the performance of zero-shot learning. Our main studies in zero-shor shot settings are conducted with CLIP-RN50 (ResNet-50 (He al., 2016) and CLIP-Vir CLT- B/16 (ViT-B/16 (Dosovitskiy et al.. 2020)) as CLIP backbone), and we report results with our best strategy of LEF+CF As shown in Table 1, on 17 diverse downstream zero-shot image clas- sification datasets, we achieve remarkable average gain of 4.31 % for CLIP-RN50 and 2.90% for CLIP-VIT-B/16 in terms top-1 accuracy. Significantly, on the Au dataset, we achieve the largest performance boost of 17.86% for CLIP-RN5O in top-1 accuracy. We notice that the perfor- mance gain brought by synthetic data varies differently across datasets, which is mainly related to GLIDE's training data distribution. The training data distribution of the text-to-image generation model GLIDE would exhibit bias and produce different domain gaps with different datasets (see Sec. A.2 in Appendix for more analysis).
4

---

Published as a conference paper at ICLR 2023
LE
LE+CF
Dataset
CLIP
CE
SCE
CE
SCE
CE
SCE
CIFAR-IU 10 70.31 77.39
(+7.08)
78,23 (+7.92)
77.20 (+6.89)
77.55 (+7.24)
80.01 (+9.70)
80,06 (+9.75)
CIFAR-I00
35.35
43.99
(+8.64)
44.25 (+8.90)
44.08 (+8.73)
44,91 (+9.56)
44.55 (+9.20)
45.69 (+10.34)
EuroSAT
37.51 45.64 (+8.13)
48.23 (+10.72)
53.26 (+15.75)
54.94 (+17.43)
54.75 (+17.24)
55.37 (+17.86)
Table 2: Ablation study on Language Enhancement (LE), CLiP-based Filtering (CF), and Soft- target Cross-Entropy (SCE).
Language diversity matters. By introducing more linguistic context into the text input, LE helps increase the diversity of synthetic data. As shown in Table 2, LE can achieve additional performance gains upon B in most cases (0.66) on CIFAR-100, 6.71f on EuroSAT), which demonstrates the efficacy of LE and the importance of synthetic data diversity for zero-shot classification.
Reliability matters. While LE could help increase the diversity of synthetic data, it also introduces the risks of noisy samples. Observed CIFAR-IO in Table 2, LE sometimes even brings perfor- mance drops compared with B (0.68% I on CIFAR- 10), which may attribute to the noise introduced by enhanced language prompts, e.g. the sentence extended from the class name word may contain other class names or confusing objects. Fortunately, with CF to filter out unreliable samples, LE+CF yields consistent improvement upon B. Moreover, SCE generally achieves better performance than CE, showing its better adaptation to label noise.
Classifier tuning is enough for CLIP, while runing with the pre-trained encoder leads to degradation, mainly due to domain gaps. Here, we investigate if only tuning the final classifier is the optimal solution in our setting with synthetic data. As shown in Table 3, we tune different proportions of the full model parameters on synthetic data for EuroSAT (0.02% corresponds to our default case where only the classifier is tuned), and report the zero-shot performance on the test set of EuroSAT. The best results are obtained by only tuning the classifier, and the performance gradually decreases as we gradually incorporate more parameters in the encoder for optimization, which agrees with the traditional strategy. For understanding why synthetic data may harm pre-trained image encoder, experiment with real-world data with domain shifts and find they behave similarly to synthetic data (Appendix Sec. B.2), which suggests that domain gap is the main reason for the phenomenon. We argue that synthetic data might have a better chance to overcome domain shifts in comparison with real-world data since we can customize and keep the label space of the synthetic data in line with the down-stream dataset and use strategies during synthesizing to alleviate domain shifts.
Param Tuned (%)
0.02
0.04
62.50
64.06
69.53
82.81
92.19
Acc
37.51
55.37
55.11
55.28
54.56
54.34
53.63
52.09
Table 3: Parameters tuned v.s. Accuracy. Dataset: EurOSAT.
Real shot
16
32
64
80
90
95
100
Acc
2.48
10.4
14.95
21.96
24.4
25.52
27.99
29.95
Table 4: Setting when training from scratch. Dataset: CIFAR-100.
Synthetic data deliver inferior performance in the training from scratch setting and are much less data-effi data-efficient than real data. To exclude the influence of powerful CLIP initialization in our study of synthetic data, we also conduct a from-scratch setting on the CIFAR- 100 dataset, where we optimize ResNet-50 model from random initialization. Given the label space of the CIFAR-100 dataset, we generate a synthetic dataset of 50k (500 images per class) to train a ResNet-50 model from scratch for image classification. We achieve a performance of 28. 28.74% top-1 accuracy on CIFAR-100 test set, which is much lower than the performance of the pre-trained CLIP model (see Table 1). This might be attributed to the quality and diversity of data. The CLIP model benefits from diverse real- world data. Further, we hope to investigate how many real in-domain training data can match the performance of our 50k synthetic data. As shown in Table 4, training with 95 images per category x95x 100 9.5k) will achieve comparable performance as that of 50k synthetic data. This manifests that synthetic data are not as efficient and effective as real data when solving downstream tasks. requires around 5 times more data in order to achieve comparable performance as that of real data. Note that we find further increasing the amount of synthetic data will not deliver further performance gains for the downstream classification task. We expect that further investigations on synthesis quality will bring new opportunities in this area which will be our future work.
Summary. Current synthetic data from text-to-ine -it-to--iageeeeee generation models could indeed bring sig- nificant performance boosts for a wide range of zero-shot image classification tasks, and is readily applicable with carefully designed strategies such as large-scale pre-trained models. Diversity and
5

---

Published as a conference paper at ICLR 2023
reliability matter for synthetic data when employed for zero-shot tasks. When the model is trained from scratch with synthetic data, synthetic data cannot deliver satisfactory performance and are much less data-efficient and effective for solving the classification task in comparison with real data.
3.2 Is SYNTHETIC DATA READY FOR FEW-SHOT IMAGE RECOGNITION?
In this section, explore the effectiveness of synthetic data for few-shot tasks and how synthetic data impact the performance as more and more shots are included. Also, we design effective strate- gies to better leverage synthetic data.
Few-shot Image Recognition. We adopt the CLIP-based method as the model for few-shot image recognition due to its state-of-the-ard performance (Radford et al., 2021). As discussed previously, various prompt learning based methods can be treated as tuning the classifier weights. We thus study how to tune the classifier weights with synthetic data. In an N-way M-shot case, we are given M real images of each test class, where M (1, 2, 4, 8, 16) in our experiments. With a total of N x M training samples, we hope to achieve favorable performance on a hold-out test set of the N classes.
Synthetic Data for Few-shot Image Recognition. While there have been a few attempts to study how to better adapt CLIP models for few-shot tasks (Zhou et al., 2022b;a; Zhang et al., 2022), they all focus on the model optimization level, and none have explored from the data level. Here, we systematically study whether and how synthetic data can be employed for solving few-shot image recognition tasks.
With the experience from synthetic data for zero-shot tasks, we adopt the best strategy (i.e. LE+CF) in the zero-shot setting the basic strategy (B). Further, as the few-shot real samples can provide useful information on the data distribution of the classification task, we develop two new strategies leveraging the in-domain few-shot real data for better using synthetic data: 1) Real Filtering (RF): given synthetic data of one class c, we use the features of few-shot real samples to filter out synthetic images whose features are very close to the features of real samples that belong to other categories different from class c: 2) Real guidance (RG): we use the few-shot real samples as guidance to generate synthetic images where the few-shot real samples (added noise) replace the random noise at the beginning of the generation to guide the diffusion process (details in Appendix Sec. C.3).
Experiment Setup. For datasets, we carefully select 8 image classification datasets from recent works (Zhou et al., 2022b;a; Zhang et al., 2022) that conduct few-shot learning upon CLIP: Ima- geNet (Deng et al., 2009), Caltech101 (Fei-Fei al., 2006), Pets (Parkhi et al., 2012), Cars (Krause et al., 2013), Aircraft (Maji et al., 2013), SUN397 (Xiao et al., 2010), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019). For synthetic image number, we generate 800 (study of synthetic image number in Appendix Sec. B.3) images per class for RG method to approximately match the number of images in B and RF.
Main Results: 1) few-shot classification results on 8 datasets; 2) ablation study of training strategy; 3) ablation study of synthetic data generation strategy; 4) ablation study of BN strategy.
Synthetic data can boost few- learning and the positive impact of synthetic data will gradually diminish with the increase of real data shots. As shown in Figure 1 (results of more datasets are in the Appendix Sec. B.1), with only few-shot real images for training, our implemented CT w. init (classifier weights initialized from CLIP text embeddings) performs comparably with the state-of- the-art CLIP tuning methods Tip Adapter (Zhang et al., 2022) and CoOp (Zhou et al., 2022b). CT w. Syn represents our results of applying synthetic data with mix training, real image as guidance, and freezing BN strategies. With the help of generated synthetic data, CT w. Syn achieves no- ticeable performance gains upon CT w. init, and achieves a new state-of-the-art few-shot learning performance across different datasets. We argue that for data-scarce few-shot classification, syn- thetic data could help address the insufficient data problem to boost performance. However, we notice that the boost from synthetic data gradually diminishes as the real shot number increases. We state that the efffcctivnness of each sample in real data is high since there's 's no domain gap; in con- trast, synthetic data suffer from domain gaps and perform less efficiently. In addition, the positive effects of the few-shot real data may overlap with that of synthetic data. Thus, with the increase of real data, the overlapping becomes serious and the positive impacts of synthetic data are reduced.
Mix Training fits few-shot learning with synthetic data. Now that we have two parts of data, í.e. few-shot real data and synthetic data, we could either 1) phase-wise train on each part of data with two training phases, or 2) adopt mix training that simultaneously utilizes two parts of data to update

---

Published as a conference paper at ICLR 2023
73
SUN997
100
EuroSAT
71
89
70
Score (96)
80
W. Syn
Score
60
WLw.Syn
init
Tip Adapter
50
Adapter
03
59
Coop
40
Goop
Zero-shot CLI
Zero-shot CUP
301
8 15
12
14
36
Number of Roal abeled examoles per class
Number of Real labeld examples class
Figure 1: Results for few-shot image recognition. Results on all 8 datasets are provided in Appendix.
the model in each iteration. Details of phase-wise/mix training in Appendix Sec. C.5.2. We provide the results in Table 7: we study on the EuroSAT dataset and use synthetic data generated from the RG method; under different shot number settings, mix training performs consistently better than two phase-wise strategies. We suggest that mix training could help learn better classifiers since each part could function as a regularization for the other: synthetic data help alleviate instabilities brought by limited real samples, and real data help address the noise and domain gap of synthetic data.
Employing real data as guidance can alleviate domain differences and boost performance. We com- pare three strategies of synthetic data generation for few-shot tasks. As shown in Table 5, both RF and RG provide performance gains upon which is the best strategy in the zero-shot setting. This demonstrates the importance of utilizing the domain knowledge from few-shot images for preparing the synthetic data. Further, RG significantly outperforms RF, yielding the best performance. This shows utilizing real data as guidance of the diffusion process help reduce the domain gap (visual illustrations in the Appendix Sec. B.8).
RF
RG
Train data Freeze BN? Test Acc
M-shot
Phase-wise - real real - syn training
Mix
87.1 87.33 88.47
Real
75.31
I
63.01
63.32
64,36
Real
85.63
2
72.24
72.85
73.62
Table Ablation for Basic strategy (B), Real Filtering (RF), Real
Syn
44.73
4
78.88
79.21
79.88
8
83.64
83.99
84.57
Syn
55.37
16
87.10
87.44
88.47
Guidance (RG) on
Table 6: Frozen BN works better Table 7: Mix training works bet- ter for few-shot tasks on EuroSAT.
EurOSAT, 16 shot.
for 16-shot settings on EuroSAT.
Frozen BN works better. Lastly, we investigate batch normalization (BN) strategies for our few- shot settings with synthetic data. As shown in Table 6, for both real and synthetic data, freezing the BN layers yields much better performance. We analyze that for real data, it is hard to get good estimation of BN statisties when the number of images is limited. As for synthetic data, we attribute this to the statistical difference between different domains. Hence, we freeze BN layers during tuning for few-shot settings.
Summary. Synthetic data from text-to-image generation models could readily benefit few-shot learning and achieve a new state-of-the-art few-shot classification performance with strategies we present in this paper. However, the positive impact of synthetic data will diminish more shots of real data are available which further confirms our previous claim that synthetic data are still not as effective real data in training classification models.
3.3 IS SYNTHETIC DATA READY FOR PRE-TRAINING'
Finally, we study whether synthetic data are effective in large-scale pre-training. We also present effective strategies to better leverage synthetic data for model pre-training.
Pre-training for Transfer Learning. Recently, it has become a common practice to first pre- prait models on large-scale datasets to obtain a well-trained feature extractor and then fine-tune the models on downstream tasks with labeled data (i.e. transfer learning). There have been various successful pre-training methods, including supervised pre- treiiiii (Joulin et al., 2016; Li et al., 2017; Mahajan et al., 2018; Sun et al., 2017; Kolesnikov et al., 2020), self- sulevsuper pre-training (Chen et al., 2020a; He et al., 2020; Caron et al., 2020: Grill et al., 2020; Chen & Hc, 2021; Zbontar et al., 2021; Ye et al., 2019), and semi-supervised pre-training (Xie al., 2020; Pham al., 2021).
7

---

Published as a conference paper at ICLR 2023
Synthetic data for Pre-training. Since data amount and diversity play important roles in pre- training, we adopt the synthetic data generation strategy LE solely to maximize the scale of syn- thetic pre- pre-training data. We study two settings for generating synthetie data for pre- pre-training: 1) downstream-aware where we have access to the label space of the downstream task, and thus we generate synthetic data according to the label space of the downstream task; 2) downstream-agnnstee where we have no access to downstream tasks in the pre-training stage, and we turn to a relatively general and diverse label space such ImageNet-IIe IK. For pre-training methods, we experiment with supervised pre-training and self-supervised pre-training methods.
Experiment Setup. We compare synthetic pre-trained models with models of random initial- ization and models of ImageNet-IK pre-training in terms of their transfer learning abilities. For downstream-awar settings: we conduct supervised pre-training on synthetic data generated accord- ing to CIFAR- 100 label space and then transfer to CIFAR-100 100 through finetuning for evaluation.
For downstream-agnue agnoosii settings: we perform supervised pre-training and self-supervised pre- training (we adopt Moco v2 (Chen et al., 2020b) framework for its simplicity and reproducibility) on synthetic data generated from ImageNet- label space and evaluate the transfer performance by finetuning the pretrained models on a object detection dataset - PASCAL VOC (Everingham et al., 2010). Further, we experiment with ImageNet-22 label space (original ImageNet- IK and another non-overlapping IK label names randomly selected from ImageNet-21KK to study the factors of data diversity and amount in synthetic pre-training. We use ResNet-50 as the default backbone when not else noted, and also experiment with a ViT-based backbone, i.e. DeiT-S (Touvron et al., 2021).
Results for Downstream-awars settings. We generate synthetic data of different sizes from CIFAR-100 label space, i.e. 1x, 2x, 3x ImageNet-IK data size, concretely 1. 1.2M, 2.4M, 3.6M. We pre-train the model on the generated synthetic labeled set in a supervised manner, and then per- form evaluation after finetuning the model on CIFAR-100. As shown Table 8, with an equivalent amount of data as that of ImageNet- IK (1. (1.2M), synthetic data for pre-training can largely reduce the gap between training from scratch (78.8%% and ImageNet- pre-trained model (84.50%). Moreover, with 2 and 3x synthetic data, pre-training on synthetic data outperforms Image IK pre-training with a noticeable margin. In addition, when we initialize the model from ImageNet- 1K pre- weights and pre-train the model on synthetic data, we obtain extra boosts upon both results.
We conclude that for downstream-aware synthetic pre-training, synthetic data deliver close perfor- mance as that of ImageNet-IK pretraining with the same amount of data, synthetic data amount helps improve the results to outperforming ImageNet-IK pre-training. and synthetic pre-training could further benefit from ImageNet- IK pre-training.
Results for pownstream-agnostee settings. We first experiment with ImageNet- label space with 1x or 2x ImageNet- 1K data size, i.e. 1.2M/2.44 42 IN-IK Syn. We perform supervised pre- training and self-supervised pre-training (i.e. Moco v2) on the generated synthetic data, and evaluate the pre-training results by transferring to the CIFAR- 100 image classification task or the PASCAL VOC detection task. As it is too costly to validate all settings (e.g., it takes more than 1 week to train Moco v2 4.0M synthetic data), we select several representative settings of interest validate the effectiveness of synthetic data without hurting our conclusion.
As shown in Table 10 and 11, with 1.2M IN-1K Syn, both supervised pre-training (79.00%) and self-supervised pre-training (81.55%) could largely approach their IN-IK Real counterparts (su- per.:81.3%; self-super.82..44
and largely outperforms the result without pre-training (66.08%). When increasing the data amount to 2.4M, the transferred results further increase, and the unsu- pervised pre-training method, i.e. Moco v2, performs better utilizing our synthetic data thanks to its independence of labels, yielding a 133% transferred performance which surpasses super- vised pre-training on IN-IK Real (81 (80.3%) and on par with its Moco v2 counterpart at IN-1K K Real (82.44 (82.44%). Next, expand the label space by adding another IK categories, producing IN-2K Syn. The enlarged diversity and data amount further bridge the gap between synthetic pre-training results and IN-1K Real pre-training results. Noticeably, the unsupervised pre-trained model Moco v2 (82.29%) largely approaches the IN-IK Real counterpart (82. (42.4%) with negligible performance drop of 0. 15%. Furthermore, when initialized from IN-1K Real pre-trained weights, both supervised and self-supervised pre-training improve upon both pure real data and synthetic data for pre-training.
While the above results are all obtained with convolutional- backbone i.e. ResNet50, we fur- ther explore with a recent ViT-based backbone i.e. DeiT-S (Touvron et al., 2021). Surprisingly,
8

---

Published as a conference paper at ICLR 2023
pre-trained IN-IK?
Syn. images amount
pre-trained IN-Ik?
Syn. images amount
Data
Data
1.2M 2.4M 3.6M
1.2M 2.4M 4.0M
(None)
78.83
(None)
69.29
C100 Syn
83.90 85.03 85.24
IN-IK Syn
87.98
88.39
(None)
84.50
IN-2K Syn
88.57
88.91
C100 Syn
84,90 85,32 85.52
(None)
88.07
Table 8:
Results CIFAR-100 with downstream-awan supervised pre-training.
Table 9:
Results
CIFAR- 100
with
downstream-agnostic supervised training. Backbone: DeiT-S.
pre-
C100: CIFAR
pre-trained IN-Ik?
Syn. images amount
pre-trained on IN-Ik?
Syn. images amount
Data
Data
1.2M 2.4M 4.0M
1.2M 2.4M 4.0M
(None)
66.08
(None)
66.08
IN-IK Syn
79.00
80.00
IN-1K Syn
81.55 82.13
IN-2K Syn
80.54 80.72
IN-2K Syn
82.22 82.29
(None)
81.30
IN-IK Syn
81.78
(None)
82.44
IN-2K Syn
81.87 81.91
IN-IK Syn
82.47
Table 10: Results for object detection on PAS- CAL VOC with downstream-ague aggnostie super- vised pre-training, all results are reported AP50-
Table Results for object detection on PAS- CAL VOC with wwwnstream-agnostie self- supervised pre-training (Moco v2), all results are reported in AP50-
ViT-based backbone is shown be advantageous compared with convolution- based backbone for synthetic pre-training: outperforming ImageNet pre-training results in the downstream-agnoste settings. Equipped with ViT-based backbone, only 1.2M IN-1K synthetic data, achieve com- parable performance (87. (87.98%) with ImageNet pre-training (88.07%). Further increasing the data amount (88.39%) and label space (88.57%, 88.91%) () of pre-training data leads higher perfor- mance than ImageNet pre-trat training. ViT-based backbones have stronger ability for learning from large-scale data and are more robust (Pinto al., 2021), and thus could better benefit from synthetic pre-training where data are more noisy and data scale could be easily increased.
Conclusion. In terms of transfer abilities, synthetic data from text-to- ext-tto-mmag generation models show surprisingly promising results for model pre-training, which is comparable to the standard ImageNet pre-training. We conclude our findings as follows:
1. Data amount has positive impacts synthetic pre-training; performance could be improved by increasing synthetic data size, but would gradually saturate as the amount of data increases.
Synthetic data for pre-training orthogonal real data for pre-training.
3. For downstream-awar synthetic pre-training, we significantly outperform IN-1K Real (1.2M) pre-training with 2.4M/3.6M synthetic data on CIFAR-100.
4. For downstream-agne aagnosticc synthetic pre-training, we achieve comparable results with ImageNet (IN-1k) Real pre- training; sulf-supervised pre-training performs better than supervised pre-training, and ViT-based backbone performs better than convolutional-biat backbone. Besides, increasing the label space size could further improve the performance.
CONCLUSION
We systematically investigate whether synthetic data from current state-of-the-art text-to-image gen- eration models are readily applicable for image recognition. Our extensive experiments demonstrate that synthetic data are beneficial for classifier learning in zero-shot and few-shot recognition, bring- ing significant performance boosts and yielding new state-of-the-art performance. Further, current synthetic data show strong potential for model pre-training, even surpassing the standard ImageNet pre- treitrainigg. We also point out limitations and bottlenecks for applying synthetic data for image recognition, hoping arouse more future research this direction.
Limitations. all investigated settings, we observe improved performance as the data amount and diversity (label space) increases. However, due to our limited computational resource, we are not able to further scale up data amount, which may take months to train one model. Besides, we are also not able investigate larger model sizes and advanced architectures in the current investigation which is also worth exploring in the future. We present more discussions on limitations and future directions in the appendix.

---

Published as a conference paper at ICLR 2023
Acknowledgement. This work has been supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621) and General Research Fund Scheme (Grant no. 17202422). Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.
REFERENCES
Thomas Berg, Jiongxin Liu, Scung Woo Lee, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proc. Conf. Computer Vision and Pattern Recognition (CVPR), June 2014.
Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: training models from generated images. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2020.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mood-io0 discriminative compo- nents with random forests. In European conference computer vision, pp. 446-461. Springer, 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 11096, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv:2006. 09882, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 2020b.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazıbba, V. Golkov, P. v.d. Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In IEEE Interna- tional Conference on Computer Vision (ICCV), 2015. URL. http://1mm http://mm.i.foff. informatik. umi-free Besibburg.de//ublle.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. N0ir:2010.11922 2020.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE trans- actions on pattern analysis and machine intelligence, (614):594-611, 2006.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Edipter Better vision- languaee models with feature adapters. arXiv preprint arXiv:21 04.4444, 2021.
10

---

Published as a conference paper at ICLR 2023
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. vrXiv:2006.07733 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2225 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of out-of-distribution generalization. ICCV, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23: 1721,2022.
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning, arXiv preprint arXiv:2106.05258, 2021.
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In ECCV, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine- -ree-graied categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ang Li, Allan Jabri, Armand Joulin, and Laurens van der Maaten. Learning visual n-grams from web data. In ICCV, 2017.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense 2019.
reasoning. arXiv preprint 191
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
11

---

Published as a conference paper at ICLR 2023
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine- line-grained visual classification of aircraft. Technical report, 2013.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Maria-Flend Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729. IEFE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint urXivv1710.069244 2017.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In CVPR, 2021.
Francesco Pinto, Philip Torr, and Puneet K Dokania. Are vision transformers always more robust than convolutional neural networks? 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21 (1(140):1-67, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2000:222000061255, 2022.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pp. 102-118. 18. Springer, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/VVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695 2022.
Chitwan Saharia, William Chan, Huiwen Chang. Chris Lee, Jonathan Ho, Tim Salimans. David Fleet, and Mohammad Norouzi. Palette: Image-to-imes -mooimaae diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi. Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint 220.1:2005.11487, 2022b.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022c.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn ing, pp. 2256-2265. PMLR, 2015.
12

---

Published as a conference paper at ICLR 2023
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data deep learning era. In ICCV,, 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-201 dataset. 2011.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518, 2019.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi. Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7971, 2022.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv:2103.03220, 2021.
Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-fres free adaption of clip for few-shot classification. arXiv preprint arXiv:2207.09519. 2022.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10145-10155, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy. and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, pp. 1-12, 2022b.
13


Reviews:
[{"review_id": "pkKcElhjWh", "reviewer": "Reviewer_NhD1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigated the use of synthetic data generated from state-of-the-art text-to-image generation model (GLIDE) to improve image recognition. This was approached from two angles: Fine-tuning the state-of-the-art model for zero-shot learning (CLIP) on synthetic datasets, considering both zero-shot and few-shot settings, and then comparing the performance of pre-trained weights from synthetic datasets with real data to investigate the suitability of synthetic datasets for transfer learning. In addition, several strategies were introduced to improve the diversity and quality of the generated synthetic datasets.", "summary_of_the_review": "The results of this work are insightful and contribute to a better understanding of the capabilities of synthetic datasets. They also shed light on the challenges associated with sample diversity and quality. Depending on the task, the synthesised data from the text-image generation model could simply be full of noise, making this method unlikely to be very useful in practise.", "strength_and_weaknesses": "Strengths:\n- The overall idea aims to investigate the suitability of synthetic datasets for performing image recognition tasks, which can be valuable in a data-scarce environment\n- The language enhancement method for expanding words into sentences is a useful method for generating diverse sentences, which can increase the diversity of synthetic datasets generated\n- The real guidance method appears to be novel in helping the model bridge the domain gap between synthetic and real-world datasets, improving the performance of the model\n- The paper shows a significant improvement over the model CLIP and demonstrates that synthetic datasets can indeed increase the performance of the model\n\nWeekness:\n- The paper mentions some previous work that investigated similar questions, albeit on a smaller scale. A comparison of their results with these methods could further strengthen their work\n- The paper uses 17 datasets to evaluate their method and compares it to the CLIP model. Admittedly, 17 datasets is quite impressive in itself, but the CLIP paper evaluated 30 datasets. Can the authors explain why the remaining 13 datasets were excluded in their analysis?\n- In the paper, 2000 samples per class were synthesised to fine-tune the CLIP model and an average gain of 4.31 was obtained over the original model. Were the 2000 samples chosen arbitrarily, or was the model fine-tuned using different synthetic samples and 2000 determined to be the magic number \n- On page 3 \"...Here, we adopt a simple tuning method, Classifier Tuning (CT) (Wortsman et al., 2022), which directly tunes a classifier attached to CLIP image encoder and initializes the classifier weights with pre-trained text embedding.\" The description of how the classifier tuning (CT) method works does not match my understanding of the paper (Wortsman et al. 2022), which describes the WiSE- FT method as a weighted linear interpolation between the original zero-shot and fine-tuned models. Perhaps the authors could elaborate on how CT uses the pre-trained text embeddings to initialise the classifier weights, or refer me to the relevant paragraph in the Wortsman et al. 2022 paper that does so.\n- On page 6, It is not clear to me how phase-wise training works, nor was the method of mix training well explained, making it difficult to understand the performance gain from the procedure shown later in Table 7. It would be nice if the authors would refer to the relevant literature where these methods are presented or explain in more detail how they work in the appendix ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ioxyPDkTwN", "reviewer": "Reviewer_WkSM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates using synthetic data from generated with GLIDE model for image recognition based on CLIP embedding in zero-shot and few-shot settings. It is shown that synthetic data are beneficial for classifier learning. The synthetic data also show great potential for model pre-training.", "summary_of_the_review": "This paper is interesting but not providing deep insights. More follow up work may be needed.", "strength_and_weaknesses": "+ show state-of-the-art results on zero-shot and few-shot recognition on a range of datasets\n+ the first systematic study on using generative synthetic data for classification\n- foundational models and approaches are limited to GLIDE and CLIP\n- improvement over CLIP baseline not very large\n- time complexity is high", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "blKMMCo9bs", "reviewer": "Reviewer_xQXo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates an interesting problem, whether text-to-image generation model can help image recognition through synthetic data.", "summary_of_the_review": "The paper investigates how will text-to-image generation model helps image recognition. Because of the clarity and meaningful experiments, I lean towards acceptance, but I do hope to see more results or discussions from the authors.", "strength_and_weaknesses": "The problem is interesting, and could be potentially useful in real world applications.\n\nThe paper conduct different experiments to investigate how do the synthetic data from generative models contribute to the image recognition problem under different settings, including zero-shot, few-shot, pre-training then transfer learning settings.\n\nExperimental results are clear and seems to be reasonable.\n\nAlthough the paper is an experiment-based paper which aims to reveal some interesting applications, deeper investigation are expected rather than only conducting experiments. Specifically, what's the reason behind the performance differences (different levels of improvements across datasets are observed, e.g. accuracy is only improved by 0.01% on Food, while the improvement on EuroSAT is 17.86%). If we find a reasonable or intuitive explanation, can we use it to further improve the results by controlling the generation or augmentation process.\n\nIt would be nice if the author could provide some examples of synthetic and ground-truth data. Since the experiments are conducted on many different datasets, intuitively, the more similar synthetic data to ground-truth data, the better performance mode can get, but is it \n true? Some examples along with Table 1 and 2 may help us to better understand why synthetic data contributes to image recognition.\n\nAlthough zero-shot, few-shot problems are important, how about fully-supervised setting? Will the synthetic data benefits the model when lots of ground-truth data are provided or it will be harmful to the model? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2-eegVeel4", "reviewer": "Reviewer_z3p6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work synthesized classification datasets motivated by high-quality images generated by the pre-trained text-to-image model (GLIDE).\nThey used the dataset on three tasks: 1) zero-shot classification, 2) few-shot classification, and 3) pre-training for transfer learning.\nThey tuned the classifier of pre-trained CLIP leveraging the synthesized image-label dataset and improved zero-shot and few-shot classification tasks.\nThey proposed several simple techniques to build the synthesized dataset with appropriate ablation studies for improving diversity and class fidelity.\nPre-training for transfer learning experiments train models with ImageNet class-based synthesized dataset in a supervised and self-supervised manner.\nThe pre-training performance nearly reached the ImageNet1K pre-trained model and showed slightly better performance using the ImageNet pre-trained model as a starting point.\n", "summary_of_the_review": "They fully leverage the open-source pre-trained text-to-image model to generate the synthetic classification dataset.\nExtensive empirical studies support the claim that synthetic datasets can help zero-shot, few-shot classification, and pre-training for transfer learning. \nHowever, the authors did not elaborate sufficiently on the cases when the synthetic images harm tuning the classifier. Specifically, the effect of the proportion of synthesized images are not explained in the paper, which could be the limitation of using the synthetic dataset.\nFurthermore, since the authors mainly conducted experiments with CLIP-Res50, it is highly recommended for the authors to include experiments with variants of CLIPs to demonstrate the scalability of the proposed approach.\n", "strength_and_weaknesses": "Strength - \n\nThe paper is overall well-written and easy to follow.\nThe extensive experiments using the synthetic dataset support the main claim.\nThey proposed a novel approach that utilizes text-to-image pre-trained models for a classification dataset generation that does not use any image and label for zero-shot and few-shot classification.\nThey showed that tuning the classifier on the pre-trained CLIP model with a synthetic dataset increases zero-shot classification performance.\nSupervised and self-supervised learning methods are used in pre-training for transfer learning and boosted the classification CIFAR-100 and object detection on PASCAL VOC 2012.\n\n\nWeaknesses -\n\nThey showed the enhanced results only when tuning on the synthetic dataset. However, in some cases, the synthetic dataset could harm the pre-trained image encoder (e.g., Scratch training on CIFAR-100 with a synthetic dataset and Table 3). The analysis of failed experiments when mixing the real and the synthetic datasets is also an essential part of this systematic paper and can help future research.\n\nWhile tuning CLIP with synthetic dataset does not bring performance degradation, the original CLIP is one of the most important parts in zero-shot and few-shot classification. However, Classifier Tuning (CT), and CT w. init are not explained well in the paper. Please explain them clearly in the paper.\n\nA main concern of the paper is that the authors conducted experiments mainly with CLIP-Res50. To be more specific, there are several variants of CLIP for zero-shot classification, such as CLIP-ViT-S, which shows lower FLOPS and high performance. \nFurthermore, several variants of CLIP show better performance than the results of this paper, which requires additional training with the generated dataset (e.g., CLIP-ViT-B/16, the base model of Vision Transformer has 91.6, 68.7, and 54.1 zero-shot top-1 accuracy on CIFAR-10, CIFAR-100, and EuroSAT, respectively). Therefore, the paper needs to show the scalability to justify that the generated dataset can improve zero-shot classification regardless of the model’s capacity and achieve state-of-the-art zero-shot classification.\n\nIn the pre-training for the transfer learning section, changing the number of synthetic images helps to understand how the synthetic dataset improves the performance. In this respect, further experiments about the number of synthetic images in zero-shot and few-shot classification are necessary to support that the synthetic dataset can help the classification and reveal the limitation of the synthetic dataset. However, the current manuscript does not include experiments for zero-shot and few-shot classification\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%