Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work synthesized classification datasets motivated by high-quality images generated by the pre-trained text-to-image model (GLIDE).
They used the dataset on three tasks: 1) zero-shot classification, 2) few-shot classification, and 3) pre-training for transfer learning.
They tuned the classifier of pre-trained CLIP leveraging the synthesized image-label dataset and improved zero-shot and few-shot classification tasks.
They proposed several simple techniques to build the synthesized dataset with appropriate ablation studies for improving diversity and class fidelity.
Pre-training for transfer learning experiments train models with ImageNet class-based synthesized dataset in a supervised and self-supervised manner.
The pre-training performance nearly reached the ImageNet1K pre-trained model and showed slightly better performance using the ImageNet pre-trained model as a starting point.

strength_and_weaknesses: Strength - 

The paper is overall well-written and easy to follow.
The extensive experiments using the synthetic dataset support the main claim.
They proposed a novel approach that utilizes text-to-image pre-trained models for a classification dataset generation that does not use any image and label for zero-shot and few-shot classification.
They showed that tuning the classifier on the pre-trained CLIP model with a synthetic dataset increases zero-shot classification performance.
Supervised and self-supervised learning methods are used in pre-training for transfer learning and boosted the classification CIFAR-100 and object detection on PASCAL VOC 2012.


Weaknesses -

They showed the enhanced results only when tuning on the synthetic dataset. However, in some cases, the synthetic dataset could harm the pre-trained image encoder (e.g., Scratch training on CIFAR-100 with a synthetic dataset and Table 3). The analysis of failed experiments when mixing the real and the synthetic datasets is also an essential part of this systematic paper and can help future research.

While tuning CLIP with synthetic dataset does not bring performance degradation, the original CLIP is one of the most important parts in zero-shot and few-shot classification. However, Classifier Tuning (CT), and CT w. init are not explained well in the paper. Please explain them clearly in the paper.

A main concern of the paper is that the authors conducted experiments mainly with CLIP-Res50. To be more specific, there are several variants of CLIP for zero-shot classification, such as CLIP-ViT-S, which shows lower FLOPS and high performance. 
Furthermore, several variants of CLIP show better performance than the results of this paper, which requires additional training with the generated dataset (e.g., CLIP-ViT-B/16, the base model of Vision Transformer has 91.6, 68.7, and 54.1 zero-shot top-1 accuracy on CIFAR-10, CIFAR-100, and EuroSAT, respectively). Therefore, the paper needs to show the scalability to justify that the generated dataset can improve zero-shot classification regardless of the model’s capacity and achieve state-of-the-art zero-shot classification.

In the pre-training for the transfer learning section, changing the number of synthetic images helps to understand how the synthetic dataset improves the performance. In this respect, further experiments about the number of synthetic images in zero-shot and few-shot classification are necessary to support that the synthetic dataset can help the classification and reveal the limitation of the synthetic dataset. However, the current manuscript does not include experiments for zero-shot and few-shot classification

clarity,_quality,_novelty_and_reproducibility: Please clarify that the zero-shot classification performance of CLIP-Res50 in the paper is slightly lower than that of the original CLIP paper (~5% in CIFAR-10, CIFAR-100).
In the ablation studies on zero-shot (B, LE, CF) and few-shot (B, RG, RF) classification, qualitative results with FID and classification accuracy with a pre-trained classifier can help to check the diversity and class fidelity. 

summary_of_the_review: They fully leverage the open-source pre-trained text-to-image model to generate the synthetic classification dataset.
Extensive empirical studies support the claim that synthetic datasets can help zero-shot, few-shot classification, and pre-training for transfer learning. 
However, the authors did not elaborate sufficiently on the cases when the synthetic images harm tuning the classifier. Specifically, the effect of the proportion of synthesized images are not explained in the paper, which could be the limitation of using the synthetic dataset.
Furthermore, since the authors mainly conducted experiments with CLIP-Res50, it is highly recommended for the authors to include experiments with variants of CLIPs to demonstrate the scalability of the proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates an interesting problem, whether text-to-image generation model can help image recognition through synthetic data.
strength_and_weaknesses: The problem is interesting, and could be potentially useful in real world applications.

The paper conduct different experiments to investigate how do the synthetic data from generative models contribute to the image recognition problem under different settings, including zero-shot, few-shot, pre-training then transfer learning settings.

Experimental results are clear and seems to be reasonable.

Although the paper is an experiment-based paper which aims to reveal some interesting applications, deeper investigation are expected rather than only conducting experiments. Specifically, what's the reason behind the performance differences (different levels of improvements across datasets are observed, e.g. accuracy is only improved by 0.01% on Food, while the improvement on EuroSAT is 17.86%). If we find a reasonable or intuitive explanation, can we use it to further improve the results by controlling the generation or augmentation process.

It would be nice if the author could provide some examples of synthetic and ground-truth data. Since the experiments are conducted on many different datasets, intuitively, the more similar synthetic data to ground-truth data, the better performance mode can get, but is it 
 true? Some examples along with Table 1 and 2 may help us to better understand why synthetic data contributes to image recognition.

Although zero-shot, few-shot problems are important, how about fully-supervised setting? Will the synthetic data benefits the model when lots of ground-truth data are provided or it will be harmful to the model? 
clarity,_quality,_novelty_and_reproducibility: The paper is clear, has good quality and the experiments seems to be reproducible.
summary_of_the_review: The paper investigates how will text-to-image generation model helps image recognition. Because of the clarity and meaningful experiments, I lean towards acceptance, but I do hope to see more results or discussions from the authors.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates using synthetic data from generated with GLIDE model for image recognition based on CLIP embedding in zero-shot and few-shot settings. It is shown that synthetic data are beneficial for classifier learning. The synthetic data also show great potential for model pre-training.
strength_and_weaknesses: + show state-of-the-art results on zero-shot and few-shot recognition on a range of datasets
+ the first systematic study on using generative synthetic data for classification
- foundational models and approaches are limited to GLIDE and CLIP
- improvement over CLIP baseline not very large
- time complexity is high
clarity,_quality,_novelty_and_reproducibility: The paper writing is very clear. The idea is interesting but not a breakthrough. The experiment results are through, and the results are somehow expected. However, it is hard to extensively evaluation on all recent foundational models.
summary_of_the_review: This paper is interesting but not providing deep insights. More follow up work may be needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigated the use of synthetic data generated from state-of-the-art text-to-image generation model (GLIDE) to improve image recognition. This was approached from two angles: Fine-tuning the state-of-the-art model for zero-shot learning (CLIP) on synthetic datasets, considering both zero-shot and few-shot settings, and then comparing the performance of pre-trained weights from synthetic datasets with real data to investigate the suitability of synthetic datasets for transfer learning. In addition, several strategies were introduced to improve the diversity and quality of the generated synthetic datasets.
strength_and_weaknesses: Strengths:
- The overall idea aims to investigate the suitability of synthetic datasets for performing image recognition tasks, which can be valuable in a data-scarce environment
- The language enhancement method for expanding words into sentences is a useful method for generating diverse sentences, which can increase the diversity of synthetic datasets generated
- The real guidance method appears to be novel in helping the model bridge the domain gap between synthetic and real-world datasets, improving the performance of the model
- The paper shows a significant improvement over the model CLIP and demonstrates that synthetic datasets can indeed increase the performance of the model

Weekness:
- The paper mentions some previous work that investigated similar questions, albeit on a smaller scale. A comparison of their results with these methods could further strengthen their work
- The paper uses 17 datasets to evaluate their method and compares it to the CLIP model. Admittedly, 17 datasets is quite impressive in itself, but the CLIP paper evaluated 30 datasets. Can the authors explain why the remaining 13 datasets were excluded in their analysis?
- In the paper, 2000 samples per class were synthesised to fine-tune the CLIP model and an average gain of 4.31 was obtained over the original model. Were the 2000 samples chosen arbitrarily, or was the model fine-tuned using different synthetic samples and 2000 determined to be the magic number 
- On page 3 "...Here, we adopt a simple tuning method, Classifier Tuning (CT) (Wortsman et al., 2022), which directly tunes a classifier attached to CLIP image encoder and initializes the classifier weights with pre-trained text embedding." The description of how the classifier tuning (CT) method works does not match my understanding of the paper (Wortsman et al. 2022), which describes the WiSE- FT method as a weighted linear interpolation between the original zero-shot and fine-tuned models. Perhaps the authors could elaborate on how CT uses the pre-trained text embeddings to initialise the classifier weights, or refer me to the relevant paragraph in the Wortsman et al. 2022 paper that does so.
- On page 6, It is not clear to me how phase-wise training works, nor was the method of mix training well explained, making it difficult to understand the performance gain from the procedure shown later in Table 7. It would be nice if the authors would refer to the relevant literature where these methods are presented or explain in more detail how they work in the appendix 
clarity,_quality,_novelty_and_reproducibility: Overall, the paper reads well, but the quality of the work could be improved as it presents too much detail in a disorganised manner, for example section 3.1 is too long and could benefit from subsections. The experiments are rigorous and the work appears to be reproducible, although the authors are encouraged to provide their working code.

Minor typo:
- The paragraph "This manifests that synthetic data are not as ..." on page 5.



summary_of_the_review: The results of this work are insightful and contribute to a better understanding of the capabilities of synthetic datasets. They also shed light on the challenges associated with sample diversity and quality. Depending on the task, the synthesised data from the text-image generation model could simply be full of noise, making this method unlikely to be very useful in practise.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper uses a text-to-image model to generate synthetic training data for vision systems, with applications to zero-shot and few-shot learning, as well as pretraining. Labeled synthetic data is created by conditioning on text describing the class label. Experiments show that this approach boosts performance over using no synthetic data, and in the pre-training regime can be competitive with pretraining on real data.

Strengths:
* Timely investigation
* Extensive experiments
* Promising results in several settings

Weaknesses:
* High time complexity relative to modest performance gains


justification_for_why_not_higher_score: All reviewers are in favor of accepting this paper, finding the experiments to be thorough and the results strong. I agree; I think this is very timely work that will be of high interest to the community, hence I suggest a spotlight presentation. I do not think the paper quite reaches the level of oral as there is a fair amount of prior work on using generative models for synthetic training data, and the methods in the current paper may be somewhat expected given this prior work plus the popularity of text-to-image models.
justification_for_why_not_lower_score: See above.
Decision: Accept: notable-top-25%