Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images
Authors: Zhuoran Yu
Chenchen Zhu
Sean Culatana
Raghuraman Krishnamoorthi
Fanyi Xiao
Yong Jae Lee
Venue: Accepted by TMLR ()

Abstract: Recent advances in generative deep learning have enabled the creation of high-quality synthetic images in text-to-image generation. 
While prior research indicates that fine-tuning a pretrained diffusion model on ImageNet and generating synthetic training images can boost an ImageNet classifier's performance, 
when synthetic images start to outnumber real ones in training, the classifier performance starts to degrade, underscoring the scalability challenge of training with synthetic data. 
In this paper, we delve into the necessity of generative fine-tuning for achieving recognition performance improvements and investigate the scalability of training with large-scale synthetic images. 
We find that leveraging off-the-shelf generative models without fine-tuning, while addressing challenges of class name ambiguity, limited prompt diversity, and domain shifts effectively mitigates performance degradation from large-scale synthetic data. 
Specifically, we leverage large language models (LLMs) and CLIP to resolve class name ambiguity. 
To diversify images, we propose contextualized diversification (CD) and stylized diversification (SD) methods, also prompted by LLMs. 
Finally, to mitigate domain shifts, we leverage domain adaptation techniques with auxiliary batch normalization for synthetic images. 
Our framework consistently boosts recognition model performance with increased synthetic data, 
even up to 6 times the original ImageNet size. 
Models trained with our approach demonstrate significant in-domain improvement on ImageNet-val (1.20\% to 2.35\% across various architectures) and strong out-of-domain generalization on ImageNet-Sketch and -Rendition ($\sim$10\% improvement with large vision transformers).

Published in Transactions on Machine Learning Research (01/2025)
Diversify, Don't Fine-Tune: Scaling Up Visual Recognition Training with Synthetic Images
Zhuoran Yu Chenchen Zhu? Sean Culatana? Raghuraman Krishnamoorthi
Fanyi Xiao Yong Jae Lee
University of Wisconsin-Madison
Meta
Reviewed OpenReview: http::///paaaansss..ere/ooottttr
Abstract
Recent advances in generative deep learning have enabled the creation of high-quality syn- thetic images in text-to-image generation. While prior research indicates that fine-tuning a pretrained diffusion model on ImageNet and generating synthetic training images can boost an ImageNet classifier's performance, when synthetic images start to outnumber real ones in training, the classifier performance starts degrade, underscoring the scalability challenge of training with synthetic data. In this paper, we delve into the necessity of generative fine-tuning for achieving recognition performance improvements and investigate the scal- ability of training with large-scale synthetic images. We find that leveraging off-the-shelf generative models without fine-tuning, while addressing challenges of class name ambiguity, limited prompt diversity, and domain shifts effectively mitigates performance degradation from large-scale synthetic data. Specifically, we leverage large language models (L.LMs) and CLIP to resolve class name ambiguity. To diversify images, we propose contextualized diversification (CD) and stylized diversification (SD) methods, also prompted by LLMs. Fi- nally, to mitigate domain shifts, we leverage domain adaptation techniques with auxiliary batch normalization for synthetic images. Our framework consistently boosts recognition model performance with increased synthetic data, even up to 6 times the original ImageNet size. Models trained with our approach demonstrate significant in-domain improvement on ImageNet-val (1.20% to 2.35% across various architectures) and strong out-of-domain generalization on ImagcNet-Sketch and -Rendition (~10% improvement with large vision transformers).
1 Introduction
Recent advances in denoising probabilistic diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021; Ho et al., 2022; Dhariwal & Nichol, 2021; Rombach et al., 2022) have excelled in generating photo-realistic synthetic images in open-vocabulary text-to-image generation (Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022; Nichol et. al., 2021; Yu et al., 2023). These capabilities stem from large-scale image- text datasets like LAION (Schuhmann et al., 2021; 2022) and multimodal models like CLIP (Radford et al., 2021). Ongoing research explores the potential of using synthetic images from such models to improve recognition models (Azizi et al., 2023; He et al., 2022; Sariyildiz et al., 2023), mostly in scenarios with low-shot real training images. Critically, whether such synthetic images can improve a recognition model's performance when real images at the ImageNet-scale are already available remains an open research question.
Recent work (Azizi al., 2023) tries answer this question by delving into the intricate process of fine- tuning the open-vocabulary Imagen model (Saharia al., 2022) ImageNet (Deng et al., 2009) and using the finetuned model to generate synthetic images conditioned on ImageNet class labels. However, designing
" work partially done during internship Meta. I lequal advising.

---

Published in Transactions on Machine Learning Research (01/2025)
Target Class: Crane (bird)
(A)
Naive Prompt
(B)
Naive Prompt w/ Ambiguity Resolved
(C)
Contextual Diversification
(D)
Style Diversification
Figure 1: Prompt augmentation for unambiguous and diversified synthetic data generation. We improve synthetic data generation by augmenting prompts from two perspectives: 1) we resolve the ambiguity for class names to avoid gencrating images with incorrect semantics for the target class (e.g., row A vs. B), and 2) we diversify the prompls used to generate synthetic images both in terms of their contexts (row C) and styles (row D). We automatically achieve both augmentations with LLMs.
the fine-tuning strategy adds an extra layer of complexity and requires repetitive work when applying diverse datasets. More importantly, Azizi et al. (2023) demonstrates that exceeding the ratio of synthetic to real images can lead to significant performance degradation. For instance, training with 4.8 million synthetic images alongside default ImageNet training data yields lower accuracy compared to training solcly with real images.
We argue that the scalability failure of synthetic images stems from the generative fine-tuning process. Fine- tuming pretrained diffussin models on the ImageNet training set aligns the model and the generated synthetic images with the distribution of ImageNet training data. Consequently, training with an excess of synthetic images from fine-tuned models can lead to notable overfitting to the ImageNet training set, resulting in performance degradation.
To validate this argument, instead of fine-tuning a pretrained open-vocabulary diffusson model, we leverage a frozen off-tho-shelf one to generate additional training images to the ImageNet training data. One typical approach to achieving this would be to employ the naive prompt format a photo of a {class name) and train the recognition model with combination of real and synthetic images. However, several notable challenges arise from this simple approach from both the data generation and model training aspects, which lead to sub-optimal recognition performance, as discussed next.
Synthetic Data Generation. Datasets such as ImageNet often contain class names that have multiple meanings. For instance, crane can denote both a construction machine and a bird species. These divergent interpretations can lead to unintended semantic misalignment between synthetic images and class semantics such as generating machine images for a class of bird cranes. Furthermore, adhering to a fixed prompt format can lead to redundancy the generated images, where most of them exhibit similar visual charactcristics. The limited diversity in synthetic images exacerbates the risk of the recognition model overfitting to such images, thereby leading to challenges when scaling up training with synthetic data.
We address these challenges by redesigning the data generation pipeline to solve the label ambiguity and generate diversified images, with the assistance of large language models (LLMs). While existing ap- proaches leverage dataset-specific hierarchical information (Sariyildiz et al., 2023) or manually process class names (Radford et al., 2021) to resolve the ambiguity in class names, we propose a general solution that requires minimal human effort. Specifically, we query LLMs to extract multiple meanings of each class name and select. the meaning with the highest CLIP (Radford et al., 2021) image-text similarity associated with
2

---

Published in Transactions on Machine Learning Research (01/2025)
the actual real images of the class. The selected meaning is used as additional explanation of the class name in the synthetic data generation process.
Next, we generate diversified synthetic images by formulating generation prompts using the knowledge from LLMs. We introduce two distinct methods for diversification: contertualized diversification (CD) and stylized diversification (SD), each based on different principles. CD aims to generate photo-realistic synthetic images featuring class с with diversity introduced through contextual foreground and background objects. In contrast, SD emphasizes the generation of images with varying styles. This whole process of obtaining generation prompts is realized by prompting LLMs, minimizing the need for human labor.
Compared to finetuning diffusion models on the target dataset (Azizi et al., 2023), our approach is able to generale synthetic images leaturing the same class but containing contextual information beyond the original dataset, giving it potential to generalize better on distrr-iistiitution datasets (as we show in Section 4).
Domain Shifts in Recognition Model Training. Another inherent challenge ariscs from the domain shifts between real and synthetic images. While recent difflusion models produce synthetic images of higher quality, they still bear different distribution manifested by their separable features within the feature space (Ojha et al., 2023). When synthetic images start to dominate the real images in training, the domain shifts can lead to suboptimal performance if not properly handled.
We view real and synthetic images as being from two separate domains, and propose to mitigate the domain shifts between them by drawing inspiration from domain adaptation literature. First, we introduce auxiliary batch normalization layers (BN) (Chang et al., 2019; Seo et al., 2020; Li et al., 2018) for recognition models that employ BN to process synthetic images. We further adjust domain-level sampling weights ensuring each batch containing roughly equal number of real and synthetic images during training, avoiding overfitting to synthetic images.
Main Contributions. Through our refined design of the synthetic data generation pipeline and mitigation of domain gaps, our framework yields significant performance improvements over using fine-tuned generative models (Azizi et al., 2023) on ImageNet classification. For example, with ResNet-50, our method achieves +2.53% accuracy improvement over the baseline trained with real images only, and +1.10% compared to training with real and synthotic images from finctuned Imagen (Azizi et al., 2023).
Crucially, our simpler framework (i.e., without generative fine-tuning) consistently enhances results, even when scaling up unique synthetic data to 6x (balanced sampling used in each training batch) of the original training dataset size and further increasing synthetic data only results in minimal performance difference. This is in stark contrast to prior generative fine-tuning work (Azizi et al., 2023), which observed that recognition performance degrades as the synthetic images outnumber real ones in training. This underscores the untapped potential of synthetic data at a larger scale. Additionally, models trained with synthetic images from our framework also show more robust performance for out-of-domain generalization. Specifically, ResNet-50 trained with ImageNet real images and synthetic images from our framework achieves +2.89% +7.55%, and +3.77% absolute accuracy improvement on ImageNet-V2 (Recht et al., 2019), ImageNet- Sketch (Wang et al., 2019), and ImageNet-Rendiine (Hendrycks el al., 2021), respectively; with vision transformers, the improvements on ImageNet- Sketch and ImageNet-Renditton Rendition can even exceed +10%.
2 Related Work
Diffusion Models for Image Generation. Diffusion models (Sohl-Dickstein et al., 2015) have been widely applied in image generation (Kingma et al., 2021; Nichol & Dhariwal, 2021; Ho et al., 2022; Dhariwal & Nichol, 2021; Rombach et al., 2022). With the introduction of large-scale image-text datasets (Schuhmann et al., 2021; 2022) and image-text foundation models like CLIP (Nichol & Dhariwal, 2021), state-of-the-art diffusion models (Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022; Nichol et al., 2021; Yu et al., 2023) can perform texl-to-image generation in an open-vocabulary manner. For example, the latent difTusion model (LDM) (Rombach et al., 2022) conducts diffusion processes in the latent space; Imagen (Saharia et al., 2022), on the other hand, dircetly runs diffusion steps at the pixel level. Recent works extend these models for better control in image generation (Zhang et al., 2023; Bar-Tal Tal et al., 2023; Li et al., 2023b; Huang et al., 2023; Brooks et al., 2023). Our focus is not to improve state-of-the-art diffusion models; instead, we propose
3

---

Published in Transactions on Machine Learning Research (01/2025)
new framework that leverages such models to generate synthetic data to improve recognition model's performance on large-scale datasets.
Improving Recognition Models with Synthetic Data. We position our work within the line of work that studies how to improve visual recognition models with synthetic data from 3D-rendering (Hesse et al., 2023; Fu al., 2023; Yang et al., 2023; Zheng et al., 2023), simulation environments (Richter al., 2016; Dosovitskiy al., 2017; Gan et al., 2020; de Melo et al., 2022), or generative models (Li al., 2021; 2022; Gowal et al., 2021; He et al., 2022; Lin et al., 2023; Ali-Gombe et al., 2018). Early work (Li et al., 2022;
2021; Ali- Gombe et al., 2018) typically trained a generative adversarial network (Goodfellow et al., 2014:
Karras et al., 2019; 2020; Brock et al., 2018) on the target dataset to generate synthetic images. Recent work explores the potential of synthetic images from diffusion models to improve visual recognition (He et al., 2022; Sariyildiz et al., 2023; Lin et al., 2023; Wu et al., 2024). For example, (He et al., 2022; Li et al., 2023a) shows that CLIP classification on target dataset can be improved with synthetic data from diffusion models; Sariyildiz al. (Sariyildiz et al., 2023) uses only synthetic data to train image classification models from scratch; Lin et al. (Lin et al., 2023) extends this study to object, detection by pasting object-centric synthetic images onto training data under the few-shot scenario. This line of research primarily addresses scenarios with limited or no real training data, whereas our work concentrates on investigating the scalability of model training with synthetic data when an ImageNet-scal real training dataset is already accessible. Most related to our work, Azizi et al. (Azizi et al., 2023) recently showed that synthetic data from Imagen (Saharia et al., 2022) finetuned on ImageNet can improve recognition accuracy on ImageNet but it suffers from performance degradation when the synthetic images start to overwhelm the real images. In contrast, our work offers streamlined framework without generative fine-tuning. It effectively mitigates performance degradation when scaling up synthetic images during training, and showcases significantly improved performance in both in-domain and out-of-domain evaluations.
3 Approach
In this section, we introduce our framework from two main aspects: synthetic data generation and using that data for recognition model training. We leverage a latest variant of an off-the-shelf LDM (Rombach et al., 2022) without any finetuning to generate our synthetic images, and train our recognition models using both real training images and generated synthetic images with a carefully designed training recipe.
3.1 Label Ambiguity Resolution
Real-world datasets like ImageNet (Deng et al., 2009), often present a challenge where class labels can carry multiple divergent meanings. For instance, the term crane could refer to a type of construction machine or a species of bird. These dual interpretations lead to generations with siguificantly divergent semantics if not addressed properly: construction machine images are generated when the intended class is bird. While existing solutions tackle this problem by leveraging dataset-specifie hierarchical information (Sariyildiz al., 2023), finetuning (Azizi et al., 2023), manual processing (Radford et al., 2021), we propose more compreheusive and, importantly, manual effort-free approach.
Our approach is centered around each class name c. Rather than manually curating a specific hierarchy for class names, we turn to LLMs for assistance. Given class with N real training images, we quory an LLM to extract K possible meaning phrases ( { associated with e and compute their text embeddings ( K using the CLIP (Radford et al., 2021) text encoder é. We also extract image embeddings (0(15)) for the real training images {19})² using the CLIP visual encoder 0. We then compute the imago-text similarities and select the meaning phrase with highest similarity po:
N 0 ((T)) (PP.)
pe = arg max pc
Σ
(1)
4

---

Published in Transactions on Machine Learning Research (01/2025)
Diffusion Model
A photo of Crane
KV
LAR
(a) Naive Prompt Generation
Diflusion Model
Crane
large long-legged bird
KV
KV
LLM
(b) Naive Prompt Generation with LAR
photo crane taking off from marshy wetland,
rocky river bank, daylight, medium shot photo of crane haunting with its long-neck, plants, golden wide angle shot
Diffusion Model
Q
Contextual Diversification
KV
taking off from marshy wetland, rocky fiver bank, dayllghtt, medium shot A etching of crane taking off from marshy wetland,
painting
rocky river bank, daylight, medium shot
Style Diversification
(c) Generation with LAR and Diversification
Our synthetic data generation pipeline. (a) Generating synthetic data with naive prompts can lead to incorrect semanties for classes with ambiguous names (e.g., the bird us. the machine for "cranc"). (b) Our Label Ambiguity Resolution (LAR) procedure resolves ambiguily in labels while preserving similar semantics in the generated images. (c) Our diversification procedure includes contextual diversification (CD) and style diversification (SD) by prompting an LLM to produce contextualized descriptions of images featuring class ("crane" in this example) that combines different aspects (indicated by different colors in the figure): foreground objects, background objects, lighting condition, camera angle, and different. styles.
Figure 2:
This process is demonstrated Figure 3 and helps pinpoint the most similar meaning Pe which then used as an additional textual description alongside class name c during synthetic data generation (e.g., "A large long-legged bird" for 'Cranc'). By leveraging LLMs and CLIP, we minimize manual effort and ensure that the synthetic data accurately represents the intended semantics even in situations where the class name has multiple meanings; see Figure (b)-
3.2 Diversifying Synthetic Images
The process of diversifying synthetic images goes beyond mere reliance class names (c) and their associated meanings (Pe). Our approach introduces two distinct methods for diversification: contextualized diversification (CD) and stylized diversification (SD), each founded on different principles and mechanisms.
Contextual Diversification (CD) is dedicated to producing photo-realistic synthetic images for each class while maximizing diversity in their contexts. In CD, diversity is achieved through the incorporation of contextual foreground and background objects, which enrich the visual content of the synthetic images. Specifically, we prompt an LLM to produce context ualized descriptions of images featuring class e in a specific way that combines foreground objects, background objects, lighting condition, and camera angle. An example is shown as follows:
Imagine there a photo of {class c). What foreground and background objects can show up together with it? Describe the photo in the following four aspects:
Foreground Background
Lighting Condition
Camera Angle
5

---

Published in Transactions on Machine Learning Research (01/2025)
Image
Train Examples of Crane
4
Tu4 T-b
avg
brrg long logged
Text Encoder
argmax
larg logged bird
A machine for ilng dbjeck
T2 Tu4 TuA T2a T34 2.27
LLM Meanings
Figure 3: Label Ambiguity Resolution. Given multiple meanings of class name (crane in this example), we leverage CLIP to compute average similarity between each meaning and training examples of class c. Then, we use averaged image-text similarity as selection metric to select correct meaning of the c. In this example, a large long-legged bird is selected and used as additional context the rest of our pipeline.
Prompting an LLM in this way results in better responses than simply asking for caption of a specific object class, as the LLM may return a generic description of the class name without much context. To maximize diversity in synthetic images, we ask the LLM for 20 descriptions per prompt and explicitly instruct it to "be creative and avoid repetition". We also use a high temperature of 0.75 when prompting the LLM which helps to provide diverse and creative descriptions. We collect 600 total responses from the LLM for each class and form the generation prompt by joining the descriptions from each aspect with a comma and the prefix photograph of to promote photo-realism. When gencrating synthetic images, scale of 2.0. Since guidance scale controls how strict the diffusion model follows the text prompts, setting a lower guidance scale further encourages generation diversity, and our ablation study indeed shows that such a choice results in the best performance (7c).
guidance
The inclusion of these contextual elements not only provides a more diverse and intricate descriptions of the object class, but more importantly, helps overcome any bias in the original dataset by incorporating knowledge from the LLM in generation. Compared to finetuning diffusion models the target dataset (Azizi et al., 2023), approach can generate synthetic images featuring the same class but containing contextual information beyond the original dataset, giving it potential to generalize better out-of-distribution datasets (as we show in Section 4).
Style diversification (SD) built top of contextual diversification with emphasis on generating synthetic images that exhibit wide range of artistic styles. This approach goes beyond mere realism and delves into the realm of creative expression, allowing for a broader spectrum of synthetic images that cater to diverse aesthetic preferences. Specifically, we query an LLM for 60 different art styles (the list is provided in the Appendix) and replace the keyword "photograph" in the prompts from CD with each art style (e.g.. "a photograph of + "a painting of") to form the new generation prompts. We generate the same number of synthetic images using prompts from CD and SD; see Fig. (c).
The whole process of diversification is summarized in Figure 2. Since both CD and SD are realized with the help of an LLM, it not only automates the process to save human labor, but also ensures that the diversification of synthetic images is achieved in a consistent manner and at scale.
3.3 Mitigating Domain Shifts in Recognition Model Training
While recent diffusion models excel producing high-quality, photo-realistic synthetic images, the distribu- tion gap remains between these synthetic images and their real counterparts. This divergence most notably observed in the nearly separable features that characterize these images within the feature space (Ojha et al., 2023).
As the scale of synthetic images grows, the risk of the recognition model overfitting on synthetic images also grows. Such overfitting can significantly undermine the recognition model performance, especially when synthetic images start to dominate the training data. As shown in prior work (Azizi et al., 2023), the overall accuracy starts to drop when synthetic data outnumbers real ones and leads -2.69% accuracy drop when synthetic data becomes 9x of real data. To circunvent this issue, We view real and synthetic images as from two separate domains inspired by domain adaptation literature. This realization allows

---

Published in Transactions on Machine Learning Research (01/2025)
81
80
Accuracy (%)
79
78
77
76
ResNet-50 (Azizi et al)
75
ResNet-50(ours)
ResNet-152 (ours)
74
0
1.2M
2.4M
4.8M
7.2M
9.6M
12M
Size of Synthetic Data (M)
Figure 4: Top-1 ImageNett Classification Accuracy vs Synthetic Data Size. In contrast to findings in previous work (Azizi et al., 2023) (blue line), our method is able to scale up recognition training, with consistent accuracy improvement as the number of unique synthetic samples increases.
to introduce auxiliary BN layers (Chang et al., 2019; Seo et al., 2020; Li et al., 2018) into the recognition model to separately process synthetic images from real ones. These additional BN layers play a pivotal role in bridging the domain gap as they prevent the batch statistics of synthetic data from disturbing the running means and variances for real images.
In addition, despite the potentially larger scale of synthetic images relative to real images, enforce equal sampling weights for both the real and synthetic images during training so that each training batch roughly contains the same number of real and synthetic images. This avoids synthetic data from dominating each training batch and thus leading to domain overfitting The loss we use to train the recognition model is: L = Eim yi log(f(x;)) - À = Emaatttii yj log(f'(z;)) where f and f' denotes the model weights for real and synthetic images (i.e., they share weights except for BN layers) and X is tunable typer-parameter which enables explicit control of the contribution of synthetic data in training.
4 Experiments
Datasets and Evaluation. We train our recognition model using real images from ImageNet (Deng et al., 2009) and synthetic data generated with our pipeline. Each model is evaluated on ImageNet-val as the in-distribution evaluation. We further consider three ImageNet variations: ImageNet-V2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-Rendition (Hendrycks et al., 2021) to evalu- ale out-of-disttrbbution generalization capability. ImageNet-VVs a reproduced version of ImageNet classes with distribution shifts; SeageNet-SSketde and ImageNet- Renditton contains sketch and rendition versions of ImageNet classes, respectively.
Implementation Details. Following prior work (Axizi et. al., 2023), we evaluate our approach using different CNNS (ResNet-(50, 101, 152}) and vision transformer (DeiT-(S, B, L} (Touvron et al., 2021)) architectures. We use apt-3.5-turla (OpenAI, 2022) from OpenAl as our LLM for contextual diversification and style diversification. For each class, we prompt the LLM to generate 600 contexttnal-diverriffeed prompts and 60 style keywords to form style-diversffied prompts in synthotic data generation. We use the latest variation of LDM (Rombach et al., 2022) to generate our synthetic data. All synthetic images are generated with size of 1024x1024 and downsampled to 256x256 for efficient storage and higher throughput in data loading. To fairly evaluate the impact of generative finetuning when using generated synthetic data in training, we closely follow the training setup from prior work (Azizi et al., 2023), which augments ImageNet training with synthelic images from fine-tuned Imagen (Saharia el al., 2022). Due to the unavailability of Imagen and the finetuned model on ImageNet, we believe this the best practice for fair comparison. Specifically, we train ResNet models for 130 epochs and all vision transformer variants for 300 epochs. We set the synthetic loss weight À 0.6 for all experiments, which selected based on in-domain evaluation of ResNet-50 alone. Full details of our hyper-parameters be found in Appendix.
7

---

Published in Transactions on Machine Learning Research (01/2025)
Bittern Bird
Cardoon
Azizi et (finetuned)
Ours (Diversification)
Figure 5: Qualitative comparison between our diversified synthetic images and synthetic images from (Azizi et al., 2023). Since Azizi et al. (2023) does not share their finctuned model and synthetic data, we use the provided qualitative results their manuscript for analysis. Synthetic images from Azizi et al. (2023) lacks of diversity in both foreground and background whereas our diversified images shows diverse foreground objects in different postures and camera angles and different background environment. The diverse semantic information avoids recognition model overfitting to specific details of synthetic data and prevents performance degradation when scaling up synthetic images.
4.1 Scaling-up Training with Synthetic Data
We first demonstrate how recognition performance changes as we scale up the size of synthetic data from 1.2M to 12M. Since ImageNet has roughly 1.2M real training images, our selected range of synthetic data size increases from 1x ImageNet scale to 10x ImageNet scale. We use the same amount of synthetic images from contextual diversification and style diversification for our models.
Figure 4 shows the results. In prior work (Azizi al., 2023), ResNet-50 suffers performance degradation the size of synthetic data increases: the accuracy becomes even worse than the baseline trained with only real images when synthetic data exceeds 4x ImageNet scale (4.8M). In stark contrast, models trained with our approach demonstrate consistent improvement up to 6x unique synthetic images at ImageNet scale (while balanced sampling is used). Also unlike (Azizi et al., 2023), which observed significant performance degradation when further scaling up the synthetic images, our approach produces consistent model quality even beyond the point of saturation.
This distinction arises from the core contribution of this paper - it's important to avoid generative fine- tuning, done in Azizi et al. (2023), to mitigate overfitting when training synthetic images. Since Azizi et al. (2023) did not release their finetuned model and synthetic training data, can't directly reproduce and ablate their approach. However, we believe scalability challenges arise from generative finetuning, which causes overfitting to the ImageNet training distribution, resulting in less diverse synthetic images. As the evidence, we provide qualitative comparisons between our diversified synthotic images and synthetic images from Azizi et al. (2023) for better understanding of the performance degradation when scaling up synthetic images in training. As can be seen in Figure 5, syuthetic images from Azizi al. (2023) lacks foreground and background diversity of synthetic images) and training with large amount of such images risks overfitting to repetitive details (e.g, layout, fore=/backgroud)) resulting in poor generalization. In contrast, our diversification methods promoted by CD and SD produce more diversified synthetic images in foreground and background and thus avoids overfitting to synthetic images and encourages better generalization.
with low guidance scale (generally, low guidance encourages diversity
4.2
Main Results
We next present the main results of our approach with different recognition architectures and classification tasks. In Table 1, report results of training with ImageNet real images and our diversified synthetic images for each architecture.
When evaluating on ImageNet-val, our approach demonstrates siguificant and consistent improvement over both the baseline model trained with only real data, and the prior state-of-the-art that uses an ImageNet finctuned diffusion model to generate synthetic data (Azizi et al., 2023). For example, our approach achieves +2.53% and +1.10% absolute accuracy improvement over the real image baseline and prior approach (Azizi et al., 2023) with ResNet-50, respectively. Similar improvements are also observed with larger CNN models
8

---

Published in Transactions on Machine Learning Research (01/2025)
ImngeNei-val
ImageNet-V2
InageNei-S
ImageNet-R
Real Only
Axizi al Asici al. (2025)
Oure
Real Ouly
Oux
A
Real Ouly
Ours
A
Real Ouly
Our
A
ResNet-50
76.74
78.17
|2.53
72:20
75.09
12.80
25.01
32.56
17:55
24.55
16.77
ResNet-101
79.01
79.74
80.76
+1.75
74.25
76.39
+2.14
21.92
33.32
+5.40
27,08
31.22
+4.14
ResNet-152
79.28
75.11
77.32
+2.21
29.96
+3.61
25.29
52.45
DefT-S
78.97
$0.49
81.32
+2.35
74.68
77.61
+2.96
28.
42.29
+13.80
27.90
39.08
+11.75
DelT-B
77.35
79.16
+1.81
+11.31
31.47
DeFI-L
82.22
83.05
83.53
+1.31
78.82
79.45
+0.63
39.90
50.01
+10.11
37.21
46.48
+9.47
Table 1: Top-1 Accuracy on ImageNet and out-of-dommin variations. We train each model with real ImageNet training images and synthetic images generated with our pipeline. In-domain evaluation is conducted on ImageNet-vall (underlined) and out-of-domain evaluations are conducted on ImageNet- V2, ImageNet-Sketch and ImageNet-Renditionn real and our synthetic images on these out-of-domain evaluation datasets without further finetuning for evaluation. A denotes the margin of improvement of our approach over bascline models trained with real images only.
Specifically, we apply the same model trained with ImageNet
Training Data
N=100
N=200
N=500
ResNet50
Real Only
47.83
58.76
68.04
ResNet50
Real 2x Syn
60.95
65.19
73.11
ResNet101
Real Only
48.09
59.64
69.25
ResNet101
Real 2x Syn
62.01
67.39
74.87
ResNet152
Real Only
48.05
60.33
70.91
ResNet152
Real + 2x Syn
63.53
69.05
75.63
Table 2: Low-data Regime Evaluation. Our models demonstrate consistent improvement over models trained with only real images. Here, 2x means 2x of the ImageNet data scale (i.e, 2.4M)
like ResNet-101 and ResNet-112, where our approach achieves clear improvements (1.02% and 0.90% respec- tively) over prior work (Azizi et al., 2023). For vision transformer DeīT--(B, L}, which already demonstrate strong performance ImageNet, validate the benefit of our framework in leveraging synthetic data at large scale across model architectures.
method can still further improve their performance. These results
Furthermore, our method shows strong performance on out-of-distribution generalization. When evaluated on ImageNet-V2, ImageNet-Sketch, and ImnageNet-ennditin,, our method achieves +3% to +7% accuracy improvements over the real-only CNN baselines. With vision transformers, the improvement is even more significant: Deit-B trained with our diversifiod synthetic data achieves 11.34% and 9.67% accuracy improve- ment on ImageNet-Sketch and ImageNet-Renditaan over the real baseline, respectively.
These results demonstrate the efficacy of diversified synthetic data to improve both in-domain and out-of- domain generalization.
4.3 Results for Low-data and Long-tail Settings
While the primary focus of this work is to investigate the scalability of training with ImageNet-scale real training data, also demonstrate the effectiveness of our approach in low-data and long-tail scenarios in this section.
Evaluation on Low-data Regime. Here we demonstrate the effectiveness of our method in settings where only limited real training images are available. Specifically, we sample (100, 200, 5003 real training images per class to form the real training data and use 2x additional synthetic data of ImageNet scale (2.4M) from our pipeline. We train each recognition model for 100 epochs as models usually converge faster under the low-data regime.
As shown in Table 2, our improvements on ImageNet transfer well to the low-data regime. With 100 real images per class, additional 2x synthetic data usually brings +10% accuracy improvement. With 500 real images per class, our model achieves 74.87% and 75.63% for ResNet-110 and ResNet-152, which largely closes the gap to training with full ImageNet images (79.01% for ResNet-101 and 79.28% for ResNet-112)..
Evaluation under loig-tail distribution. We also show our pipeline enables strong improvements when the real images follow long-tailed distribution. For comprehensive evaluation under different. imbalance
9

---

Published in Transactions on Machine Learning Research (01/2025)
= 50
100
200
Training Data
Many-shot
Medjum-shot
Fow-shot 45.89
Many-shot
Medlum--hot
Few-shor
Many-shot
Medium-shos
Few-shot
ResNot
Real
78.63
69.08
79.02
68.12
39.35
78.99
06.58
33.31
ResNet-50
Real + 2x Syn
82.19
72.43
51.18
81.36
71.35
45.56
81.61
69.78
39.74
Net-122
Real
80.60
72.00
49.19
80.77
70.25
42.81
80.95
08.72
36.26
ResNet-152
Real + Syn
54.50
83.28
72.88
Table 3: Evaluation of Long-tail Distribution for Real Images. We evaluate each model with an imbalance ratio in {50, 100, 200} by subsampling ImageNet real training images and using 2x additional synthetic data of original ImageNet scale (2.4M) from our pipeline to train the models. Our approach shows consistent performance improvements, especially on few-shot classes.
79.4
75.2
Nalve Template
Accura Ticuracy/ (%)
79.2
75.0
Diversifled Prompts
Accuracy (%)
32
79.0
74.8
31
78.8
74.6
Naive Template
78.6
Accuraz
74.4
30
Diversified Prompts
78.4
74.2
29
78.2
Nave Template
Diversified Prompls
74.0
28
78.0
1.2M
2.4M 3.6M 4.8M 6.0M Size of Syn Data (M)
1.2M 2.4M 3.6M 4.8M 6.0M
1.2M 2.4M 3.6M 4.8M 6.0M Size of Syn Data (M)
Size of Syn Data (M)
(a) -iaaeeNet-Val
(b) ImageNet-VV
(c) ImageNet-SSetchh
Figure 6: Naive vs. Diversified Prompts. As the synthetic images increase from 1.2M to 6M, observe consistent improvement from diversification of synthetic data, and such improvement becomes more distinct as the scale of synthetic data increases. This trend is consistent across different evaluation sets.
ratios, we randomly sample long-tailed distribution with imbalance ratio 7 in {50, 100, 2003 from the original ImageNet training data, following an exponential imbalance function (Cui et al., 2019). Specifically, the number of real images for class k is computed пı n1 - (k-(k11)(KK1) where K is the total number of classes. n1 is the number of labels for the most frequent class and is set to 1300, which is the max number of images per class on ImageNet. Evaluation is conducted on ImageNet-val, which is roughly balanced. We report the accuracy for many-shot classes, medium-shot classes, and few-shot classes. Results are shown in Table 3 and models trained with our synthetic data consistently outperform the real-only models over many-shot, medium-shot and few-shot classes. For few-shot classes, our approach can even improve by 5% to 7% accuracy, which demonstrates the efficacy of diversified synthetic data tackle long-tail problems.
4.4 Ablation Studies
We next present ablation studies. Unless stated otherwise, experiments in this section utilize 2.4M synthetic images generated with both CD and SD.
Importance of Diversifying Synthetic Images. Figure 6 compares the performance of ResNet-50 model on ImageNet-val, -V2, and -Sketch when trained with different amounts of synthetic data generated with naive prompts A photo of {c) vs. our diversified prompts described in Section 3.2. We apply separate BN to train both models for fair comparison. As we increase the size of synthetic images from 1.2M to 6M, we see a consistent improvement from diversified synthetic images on these datasots. The improvement is more significant as the scale of synthetic data increases, which demonstrates the critical role that diversifying synthetic images plays in framework.
Importance of Separate BN in CNNs. Using separate batch norm layers (BN) for real and synthetic images is critical design in our framework and we demonstrate its importance in Table 4. Using separate BN significantly and consistently improves performance for different ResNet architectures, which validates the importance of this design.
Sensitivity to Synthetic Loss Weights. We examine the sensitivity of the tunable hyper-paramen which controls the overall contribution of synthetic data in training. As shown in Figure 7a, the accuracy of
10

---

Published in Transactions on Machine Learning Research (01/2025)
Vanilla ResNet-50
ResNet-101
ResNet-152
78.32
79.92
80.43
Separate BN
78.93
80.34
81.05
Table 4: Vanilla Separate BN. Results generated with 2.4M synthetic images with diversification.
Separate BN is demonstrated to be effective for training realheyn data, across various model architectures.
81
Inception Score(15)
Divertited Prompts
Accuracy (%)
Naive
80
79
13
Resket 50
78
155
0.05
0.2
0.4
0.6
0.8
1.0
0
20
40
60
80
100
Synthetic Loss Weight (A)
Number of Prompts
(a) À vs. Accuracy
(b) IS of synthetic images
79
Accuracy (%)
78
Accuracy (%)
78.0
77
76
77.5
75
74
73 137.5
77.0
2.0
7.5
0.3
0.4
0.5
0.6
0.7
Guidance Scale
Sampling Weight
(c) Guidance scale vs. Accuracy
(d) Sampling weight Accuracy
Figure 7: Ablation Study: (a) Model Sensitivity to Synthetic Loss Weights. The model's performance
remains largely unaffected by synthetic loss weights, except for very low values (c.g., 0.05). (b) Inception Score of Synthetic Images. Increasing the number of diversified prompts consistently enhances the Inception Score, indicating improved diversity in synthetic images. (c) Impact of Guidance Scale. Higher guidance scales impose stricter constraints on diffusion models following text prompts, limiting the diversity of synthetic images. Conversely, low guidance scales (e.g., 1.75) provide too much flexibility, resulting in lower quality generation. Therefore, guidance scale of 2.0 strikes balance between diversity and quality. (d) Impact of sampling weight. Higher sampling weights for synthetic data makes synthetic data more likely to be sampled at each training iteration. As shown in the figure, using a sampling weight of 0.5 (roughly balanced real and synthetic images in training) yields the best performance.
ResNet-50 and ResNet-152 is not affected much by the choice of X unless its value becomes very small (e.g., A < 0.2). Therefore, we simply select À = 0.6 for all our experiments.
Quantifying Diversity with Inception Score. We provide quantitative analysis of our diversification process to generate synthetic images. Specifically, we use the Inception Score (Salimans et al., 2016) as the metric to quantify diversity. We generate a total 100 images using both naive prompt and our diversified prompts. For naive prompts, we repeatedly use the template photo of <class name>!. For diversified prompts, we adjust the number of prompts per class to evenly distribute the generation workload. As shown in Figure 7b, with the increase of different diversified prompts, the Inception Score consistently increases and is higher than that of the baseline using only naive prompt template "a photo of a <class name>", Even only using one diversified prompt from LLM, its Inception Score is still greater than that of a single prompt template demonstrating that diversified prompts not only explicitly maximize the diversity at the prompt level but also better leverage the diversity within pretrained diffusion models. This result demonstrates the efficacy our diversification process.
Impact of Guidance Scale. As described in Section we use low guidance scale of 2.0 encourage diversity in the diffusion model's generations. Figure 7c shows the recognition model's accuracy trained with
11

---

Published in Transactions on Machine Learning Research (01/2025)
Contextual Diversification
Style Diversification
Papillon
(n02086910)
Violin
(nD4536866)
Goldfish
(n01443537)
Magpie
(n01582220)
Golden
Retriever
(n01582220)
Figure 8: Example Generations. We show example synthetic images from our approach. Within the same class, contextual diversification creates diverse synthetic images by introducing varying contextual elements, lighting conditions, and camera angles. In contrast, style diversification alters the visual style of synthetic images. We provide example diversified prompts in Appendix for reference.
synthetic images generated with different guidance scales. The results showcase that diversified synthetic images result in the best recognition performance.
Impact of Sampling Weight. As described in Section 3, we use a sampling weight of 0.5 for synthetic images so that each training batch contains roughly the same number of roal and synthetic images. Figure 7d shows that such choice indeed yields the best recognition model performance while increasing the sampling weight from 0.5 to 0.7 starts hurt the final accuracy.
Visualization of Synthotic Images. Finally, present example synthotic images used in our experiments in Figure 8. Our diversification approach achieves significant levels of diversification through contextual diversification (CD) and style diversification (SD). For the same class, contextual diversification introduces diversity into synthetic images by incorporating different contextual objects, varied lighting conditions and camera angles. In contrast, style diversification focuses on modifying the visual style synthetic images.
Limitation and Broader Impacts We utilize off-tho-shelf diffusion models address scalability challenges in training recognition models with large-scale synthetic images. However, performance is limited by the quality of synthetic images. Future advancements in diffusion models may improve approach. Potential negative societal impacts stem from visual recognition. Enhancing models with synthetic images, generated at minimal human costs, may lead to unintended consequences like increased surveillance.
5 Conclusion
We presented scalable framework to leverage an off-the-shelf diffusion model to generate large-scale syn- thetic data for training visual recognition models. We demonstrated that with our methods label ambigu- ity resolution, contextual and style prompt diversification, and training strategies to mitigate the real/syn thetic domain gap, recognition models trained with our framework significantly outperform prior work that requires finetuning of diffusion models on target datasets. Furthermore, we also demonstrated our framework lcads strong out-of-distribution generalization performance and better ability transfer downstream tasks compared to previous methods. We hope our work can inspire future research on leveraging synthetic data to improve recognition models.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Acknowledgments
This work was supported in part by NSF IIS2404180, Institute of Information & communications Technology Planning & EvaluationnIITP) grants funded by the Korea government (MSIT) (No. 2022-0-00871, Develop- ment of AI Autonomy and Knowledge Enhancement for Al Agent Collaboration) and (No. RS2022-00187233, Development Large Korcan Language Model Technology for Efficient Pro-training), and Microsoft Accel- erale Foundation Models Research Program.
References
Adamu Ali-Gombe, Eyad Elyan, Yann Savoye, and Chrisina Jayne. Few-shot classifier gan. In 2018 Inter- national Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2018.
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Flect. Synthetic data from diffusion models improves imagenet classification. arXiv preprint wxxiv2230.004400 2023.
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for con- trolled image generation. 2023.
Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Swilzerland, September 6-12, 2014, Proceedings, Port VT 13, pp. 446-461. Springer, 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. aixiv preprint arXiv:1809.11000 11066, 2018.
Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the TEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 18392-1888 18402, 2023.
Woong-Gi Gi Chang, Tackgoun You, Sconguk Sco, Suha Kwak, and Bohyung Han. Domain-specific batch nor- malization for unsupervised domain adaptation. In Proceedings of the IEEE/CVE conference on Computer Vision and Paltern Recognition, pp. 7354-7362, 2019.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effec- live number of samples. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognition, pp. 9268 9277, 2019.
Gelso M Melo, Antonio Torralba, Leonidas Guibas, James DiCarlo, Rama Chellappa, and Jessica Hodgins. Next-generation deep learning basod on simulators and synthetic data. Trends in cognétive sciences, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE conference computer vision and pattern recognition, pp. 2009.
Teee,
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780- 07944 2021.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Gang Fu, Qing Zhang, Lei Zhu, Chunxia Xiao, and Ping Li. Towards high-quality specular highlight removal by leveraging large-scale synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf. James Traer, Julian De Fre- itas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007 04944, 2020.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Sven Gowal, Sylvestre- -llves Rebufi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in Neural Information Processing Systems, 34:4218-4233, 2021.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Ts synthetic data from generative models ready for image recognition? arXiv preprint ar Xiw:2210.0PP 0.07774.. 2022.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributio generalization. ICCV, 2021.
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Funnybirds: A synthetic vision dataset for a part- based analysis of explainable ai methods. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3981-3991, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 1-6640-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249 2281, 2022.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXic preprint ar XXXx:2302.09778 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Ailtala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110 8119, 2020.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 1121696-217071 2021.
Bo Li, Haotian Liu, Liangyu Chen, Yong Jae Lee, Chunyuan Li, and Ziwei Liu. Benchmarking and analyzing generative data for visual recognition. arXiv preprint arXie:2307.13697 2023a.
Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised lcarning and strong out-of-dommin generalization. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 8300-8311. 2021.
Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba. Bigdataset- gan: Synthesizing imagenet with pixel-wise annotations. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 21330 21340, 2022.
Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pallern Recognition, 80:109-117, 2018.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 22511 22521, 2023b.
Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao. Explore the power of synthetic data on few-shot object detection. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pallern Recognilion, pp. 638 647, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Πlya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112 10741, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162 8171. PMLR, 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. Tn 2008 Sirth Indian conference on computer vision, graphics & image processing, pp. 722-729. TEFE, 2008.
Utkarsh Ojha, Yuheng Li, and Yong Jac Lee. Towards universal fake image detectors that generalize across generative models. In Proceedings of the IEEE/CVP Conference Computer Vision and Pottern Recog- milion, pp. 24480 24489, 2023.
OpenAT. Introducing chatgpt.
http:///nii..oobbbboo/aaattttttt 2022.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. URL https://orrio. mng/abs/220/ 003204006223 25, 7, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International conference on machine learning, pp. 1389-5440.. PMLR, 2019.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. pp. 102-118. Springer, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion
image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479 36494, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Mert Bulent Sariyildiz, Kartock Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learn- ing transferable representations from synthetic imagenet clones, In CVPR 2023- 1023-IEEE/CT Conference on Computer Vision and Pattern Recognition, 2023.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaezmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint 0rXic:2111.0211, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large- scale dataset for training next generalion image-text models. Advances in Newral Information Processing Systems, 35:25278 25294, 2022.
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In Computer Vision- 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXII 16, pp. 68 83. Springer, 2020.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised lcarning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256 2265. PMLR, 2015.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. In International conference on machine learning, pp. 10347 10357. PMLR, 2021.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European Conference on Computer Vision, pp. 516-533. Springer, 2022.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advonces in Neural Information Processing Systems, pp. 10506-10518, 2019.
Weijia. Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Syuthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Zhitao Yang, Zhongang Cai, Haiyi Mei, Shuai Liu, Zhaoxi Chen, Weiye Xiao, Yukun Wei, Zhongfei Qing, Chen Wei, Bo Dai, et al. Synbody: Syuthetic dataset with layered human models for 3d human perception and modeling. ar preprint arXiv:2303. 17368, 2023.
Yang You, Tgor Gilman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXie preprint ar Xin: 1788. 03888, 6(12):6, 2017.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Beujamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Sholly Sheynin, et al. Scaling autorogressive multi-modal models: Pretraining and instruction tuning, arXiv preprint Xrv: 02591, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19855-19865, 2023.
16

---

Published in Transactions on Machine Learning Research (01/2025)
A
Training Details
We describe the training details of our main experiments in this section (shown in Table 1). For training ResNet models, we keep most hyper-parameters the same as prior work (Azizi et al., 2023) except for batch size and learning rate. Azizi et al. (2023) uses batch size 4096 and learning rate 1.6 and we double them in our experiments to expedite training. Echoing findings in previous work (You et al., 2017), we found this difference only makes training faster without much impact on model accuracy. Also, unlike (Azizi et al., 2023) in which models trained for 200 epochs, we train all models for 130 epochs, as did not observe performance improvements by training them longer. Details can be found in Table 5.
Model Parameter ResNett(50,, 101, 152)
Epochs
130
Batch size
8192
Optimizer
Momentum
Learning rate
3.2
Decay method
Cosine
Weight decay
1e-4
Warmup epochs
5
Label smoothing
0.1
Dropout rate
0.25
Data Augment
RandAug
Table 5: Training Details of ResNet Models.
When training large vision-transformer such as Deit-B and Deit-L (Touvron et al., 2021), we have encountered numerical stability issuet when using the learning rate schedule from Azizi et al (Azizi et al., 2023; Touvron et al., 2021). Therefore, we follow the refined training setup (Touvron et al., 2022) with the same number of epochs from prior work (Azizi et al., 2023; Touvron et al., 2021). The bascline model with real images trained with this new schedule achieves 81 .8% top-1 accuracy for Deit-B, which is slightly lower than the baseline from prior work (Azizi et al., 2023). Note that use the same recipe for both the real-image baselines well as for our method. Details can be found in Table 6.
Model
DeiT-S
DeiT--
DeiT-L
Epochs
300
300
300
Input Size
224
192
192
Batch size
4096
2048
2048
Optimizer
AdamW
Lamb
Lamb
Learning rate
0.004
0.003
0.003
Learning rate decay
Cosine
Cosine
Cosine
Weight decay
0.05
0.05
0.05
Warmup epochs
5
5
Label dmoothing
0.1
0.1
0.1
Drop Path
0.1
0.2
0.45
Rand Augment
9
Mixup prob.
0.8
0.8
0.8
Cutmix prob.
1.0
1.0
1.0
Eval Crop Ratio
0.875
1.0
1.0
Table 6: Training Details of Vision Transformer Models: DeiT-- Touvron et al. (2021), DefT-B Touvron et al. (2021), and D Touvron et al. (2021).
We use 6x synthetic data of ImageNet scale (7.2M) for ResNet models and Deit-S and 2x (2.4)) for Deit-B and Deit-L. We attribute such difference to two reasons. First, larger vision transformer models generally take longer to train per iteration. Second, those models are more sensitive to larger learning rates and linearly scaling up learning rate with batch sizes usually cause numeric stability issues. These two reasons together make training larger vision transformer models with more synthetic data take much longer than
Such difficulty widely observed by other community users:
17

---

Published in Transactions on Machine Learning Research (01/2025)
Training Data
? 50
100
200
ResNet-50
Real
64.65
62.06
59.48
ResNet-50
Real
2x Syn
68.55
66.07
63.73
ResNet-112
Real
67.66
64.91
62.19
ResNet-152
Real + 2x Syn
70.81
68.70
66.31
Table 7: Evaluation of Long-tail Distribution for Real Images. We evaluate cach model with imbal- ance ratio in (50, 100, 200} by subsampling ImageNet real training images and use 2x additional synthetic data of original ImageNet scale (2.4M) from our pipeline to train our models. Compared with baseline trained with real images, our approach shows significant improvement on overall accuracy.
smaller modols (more than a week in our infrastructure with 4 8-A100 GPU nodes). Therefore, we only use 2.4M synthetic data to train those larger models and significant improvement is already observed Table 1.
B Low-data and Long-tail tail Training
As shown in Table 3 and Table 2, diversified synthetic data from our approach can also improve model training under low-data regime and long-tail distribution real images. We provide training details of these experiments in this section. These set of experiments are evaluated with Res.et architectures.
For low-data regime, we sample 100, 200, and 500 real images per class from ImageNet real images construct low-data training set. For models trained with synthetic data, keep most the training setlings same as Table 5, which is used for ImageNet. training except that we use 100 epochs as model usually converge faster under these conditions. For baseline models trained with real images, we use batch size 2048 and learning rate 0.88 to train these models as we found larger batch sizes produce significantly worse results for these models due to data scarcity.
The long-tailed version of ImageNet is constructed by truncating the original ImageNet training set. We use the exponential function (Cui et al., 2019) with different imbalance ratios (50, 100, and 200) to control the level of imbalance of real images. Likewise, train all models for 100 epochs and use smaller batch sizes and learning rates as in low-data regime to train baseline models.
We define many-shot as classes with more than 800 real training images; medium-shot as classes with the number of real training images ranging from 300 to 800; and few-shot classes as classes with less than 300 training images. Results for many-shot, medium-shot, and few-shot classes are already reported in Table Here, we additionally report the overall accuracy of each model Table for further reference.
C Combination of Diversification Methods
As discussed in Section 3, our approach involves generating synthetic data with both contextual diversification (CD) and style diversification (SD). We present the results obtained using different combinations of these diversification techniques. Specifically, we leverage 2x synthetic data of original ImageNet scale for this study.
Table 8 showcases the performance of models trained with various combinations. Using only synthetic images with CD leads to models that struggle to generalize well to datasets like ImayeNel-Sketch or ImageNet- Rendition. Conversely, utilizing only images with SD results in slightly lower performance in in-domain evaluation (indicated by underlined results). Notably, combining CD and SD yields the best performance for both in-domain and out-of-dommin evaluations. Therefore, we keep the same amount of synthetic data with CD and SD for all our experiments.
18

---

Published in Transactions on Machine Learning Research (01/2025)
ImNeet
ImNet-V2
Sketch
Rendition
2x CD
78.17
74.03
27.71
26,91
2x SD
77.95
73.92
30.41
29.67
1x CD 1x SD
78.51
74.68
30.32
29.45
Table 8: Results with different combinations of diversification. We use 2x syutheitc data of original ImageNet scale and evaluate with different combinations of contextual diversification (CD) and style diver- sification (SD). ImageNet-val (underlined) is used for in-domain evaluation and others are for out-of-domain evaluation. Results show that using equal amount of synthetic data with CD and SD yields the best overall results.
D Example of Diversified Generation Prompts
Our prompt diversification procedure involves contextual diversification (CD) and style diversification (SD). To better understand our approach, we provide a list of examples of diversified prompts from CD and SD in this section.
Recall that contextual diversification prompts large language models produce contextualized image de- scriptions featuring a particular object class in following four aspects: foreground objects, background objects, lightning condition, and camera angle. Descriptions of these aspects joined together (each component is separated by comma) to form the lext prompts in syuthetic image generation. See below for examples:
COLDFISH (N01443537
goldfish swimming in fish tank, bubbles, decorative plants, pebbles, artificial aquartum light, medium shot
goldfish in fishbowl wilh treasure chest decoration, table with scallered fish food, room light, low- angle shot goldfish in backyard pond, surrounded by rocks, trees, garden features, natural sunlight, wide shot goldfish én school swimming logether, fish lank with other fish, underwater plants, daylight, panoramic shot
GOLDEN RETRIEVER (N02099601)
golden retriever swimming a lake, blue sky, mountains the distance, morning, wide shot golden retriever playing with a Frisbee, open field, blue sky, afternoon, long shot Golden retriever wearing service dog vest, busy city street, daylight, medium shot golden retriever wearing birthday hat, party decorations, presents, indoor lightning, overhead shot
LEOPARD (N02128385)
resting leopard, tail curled, dense jungle foliage, trees, dappled sunlight, medium shot leopard crouching in hunting position, dense jungle, trees and foliage, golden hour, close-up shot leopard drinking from a river, reflecting water, rocky terrain, distant mountains, harsh daylight, wide shot
leopard groomèng itself, fallen tree trunk, colorful bird perched nearby, late ofternoon sunlight, macro shot
PRETZEL (N07693742)
pretzel held in hand, cup of coffee, newspaper, natural daylight, close-up shot
pretzel with drizzle of chocolate, snacce, plate of assorted desserts, dimmed lightning, medium shot pretzel in bakery display case, freshly baked bread, pastries, soft spollight, close-up shot pretzel being served with side of sausages, traditional German beer garden, outdoor daylight, wide shot
The prompts with contextual diversification are then combined with the prefix " A photograph of to generate photo-realistic synthetic images. On the other hand, the style diversification replaces photograhh with each
19

---

Published in Transactions on Machine Learning Research (01/2025)
Training Data
Flowers
Pets
Food
DTD
KNN
Real
66.75
88.06
62.25
57.18
Real + 6x Synthetic
67.26
90.27
55.44
61.70
Linear Probing
Real
82.12
90.18
67.18
64.89
Real | 6x Syuthetic
82.46
92.26
68.81
66.48
Table 9: Transfer Learning Evaluations. We compare the transfer learning performance of a ResNet-50 model trained with only real ImageNet images, to a model trained with real + 6x synthetic ImageNet images. Results show that models trained with synthetic data (ours) demonstrate stronger capability to transfer to other tasks.
of the style keyword that is crawled from LLMs to is shown below:
form style diversified prompts. The list. of style key words
STYLES FROM LLM
Sketch, Painting, Ilustration, Digital rendering, Print, Comic-style depiction, Manga-style depiction, Pizel art representation, Tattoo design, Graffiti-style portragal, Watercolor, Oil pointing, Charcoal drawinng, Pastel drewing, Stencil art, Collage, Mosaic, Silhouette, Pop art version, Sculpture, Origami, Embroidery, Quilt pattern, Stained glass design, Woodeut, Etching, Lithograph, Screen print, Relief carving, Bronze casting, Glass blowing, Ceramic pottery, Tapestry, Fresco, Murul, Doodle, Cartoon, Animation, 3D model, Wireframe model, CGI, Virtual reality model, Augmented reality model, Hologram, Gouache painting, Ink wash painting, Digital pointing, Stencil graffiti, Airbrush art, Pointilism,, Impasto painting, Engraving, Linocul, Marquetry, Papercul, Batik design, Cross- stitch pattern, Macramé design, Beadwork design, Sand sculpture
E
Synthetic Image Generation Details
We provide details of synthetic data generation addition to the implementation details described Section Specifically, we set the guidance scale to 2.0 and use 50 sampling steps to generate synthetic images. Synthetic images are sampled at resolution of 1024x1024 and downsampled to 256x256 for storage. For faster generation, fp16 inference is used for all synthetic images.
F Transfer Learning for Other Classification Tasks
To examine the quality of representation learned from our synthetic data, we further evaluate the transfer learning performance of our models various classification tasks. Specifically, evaluate four datasets: Oxford Flowers (Nilsback & Zisserman, 2008), Oxford Pets (Parkhi et al., 2012). Food-101 (Bossard et al., 2014), and Describable Textures (Cimpoi et al., 2014). Following the standard transfer learning evaluation protocol, we use both KNN and linear probing to evaluate transfer learning performance.
As shown in Table 9, ResNet-50 models trained with diversified synthetic data demonstrate stronger per- formance in transferring to other classification tasks under both k-ncarest neighbor and lincar probing eval- uation. On average our model achieves 2..61% and +1.41% % accuracy improvement with KNN and linear probing respectively.
20


Reviews:
[{"review_id": "kB1LBqufHr", "reviewer": "Reviewer_sBLh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Eliminating the need for fine-tuning the generative model is a significant simplification that is very useful for end-users.\n- The diversity of the generated images is qualitatively and quantitatively stronger than that of prior work.\n\nWeaknesses\n- The work does not specify the exact text-to-image generative model used, specifically the dataset it was trained on. This is of critical importance since ensuring there is no train-test contamination is crucial to the validity of the results.\n- While the amount of generated data is scaled up to 12x that of real data, the authors weigh the sampling probabilities such that the model sees 50% real data and 50% synthetic data during training. Hence, synthetic data never outnumbers the amount of real samples during training. This makes the results of Figure 3 and section 4.1 potentially misleading or overclaiming. Additionally, the comparison to (Azizi et al., 2023) in Figure 3 is not apples-to-apples, as that work did not artificially up-weight real data.\n- The potency of the long-tail distribution experiments is greatly lessened by the fact that the generative model has seen many examples of the artificially un-likely classes. In order to have rigorous experiments, the un-likely classes should also be un-likely for the generative model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents an improved synthetic data generation pipeline for large scale classification (ImageNet). Specifically they,\n- Improve classification performance by +1.10% on ResNet-50 compared to previous work (Azizi et al., 2023). Improvements are also seen for larger ResNet's and DeiT models.\n- Eliminate the need for fine-tuning text-to-image generators on the target dataset, an important simplification compared to prior work (Azizi et al., 2023). They do this by resolving the label ambiguity problem (e.g. \"crane\" the bird vs \"crane\" the machine) associated with pre-trained generators by using LLMs and CLIP to generate a short un-ambiguous description of the class (e.g. \"crane, a long legged bird\"), used during generation.\n- Qualitatively and quantitatively improve the diversity of generated images by using LLMs to yield diverse generation prompts.\n- Significant classification performance improvements are also seen in out-of-domain tasks such as ImageNet-V2, ImageNet-Sketch and ImageNet-Rendition.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I agree that the method is valuable, as the simplification over prior approaches and increase in diversity of the generated images is significant.\n\nHowever, the first two weaknesses of the work are critical to the validity of the results and the scientific knowledge gained from the paper, so I cannot recommend acceptance unless the following changes are made:\n- **Ensure there is no train-test contamination**: Specify the exact text-to-image generative model used and the dataset it was trained on, and ensure there is no train-test contamination. i.e. make sure the ImageNet validation set is not part of the generator's training set.\n- **Reword the claims regarding synthetic data outnumbering real data**: Since real data is artificially up-weighted to 50%, trained models see equal amounts of real and synthetic data during training. Hence the claims made regarding \"consistent accuracy improvement as the number of synthetic samples increases even after they outnumber the real samples\" is not fully precise. For clarity, I am absolutely in favor of artificially up-sampling real data, and in no way does it diminish the results of the work. My objection is only in the language used. In order to justify the original claims, the authors would need to show similar results without the artificial up-sampling.\n- **Apples-to-apples comparison without artificial up-weighting in Figure 3**: This artificial up-weighting is not done in (Azizi et al., 2023), so the comparison in Figure 3 is not apples-to-apples. I think an additional experiment where artificial up-weighting of real data is not used needs to be added. The main focus of the work are insights regarding synthetic data scaling, so this experiment is critical for the scientific knowledge contributed by the paper.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6BbSjZ5OOz", "reviewer": "Reviewer_uXdr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength:\n1. The paper is well-written and easy to follow\n2. The empirical result in ImageNet-val looks promising and outperforms [1]. \n3. The design choice of specific hyperparameters is well-justified with the corresponding ablation study.\n\n## Weakness:\n1. In the experiment, having a baseline where the image is generated with the same backbone model but without the proposed module would be helpful. The current baseline [1] uses a different backbone model, Imagen, which is not open-sourced, thus making the improvement margin less informative to reflect the true benefit of the proposed methods.\n2. For a similar reason as in the above, the experiment on O.O.D datasets has no baseline but only real data. I assume this is because there are no corresponding O.O.D dataset results reported in the original paper [1].\n3. The paper says, \"... we use the latest variation of LDM ...\" it would be good if the authors explicitly say which model is used, as the definition of \"latest\" may change.\n\n## Summary\nOverall, I think this is valuable work, with good motivation for the method and a solid ablation study on the practical design. I believe this paper can be interesting to researchers interested in synthetic training data generation.\n\n[1] Azizi, S., Kornblith, S., Saharia, C., Norouzi, M. and Fleet, D.J., 2023. Synthetic data from diffusion models improves imagenet classification. TMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a well-designed pipeline for generating diverse synthetic data to train image classifiers. The main technical contribution of the work lies in (1) solving the label ambiguity through curating concept embeddings with the vision-language model, (2) prompt engineering to diversify generation in terms of camera angle, illumination, style, etc., (3) regularization for real-synthetic data domain gap in the loss function. The general generation framework in this work is in line with existing literature, while the proposed pipeline has been empirically proven effective with state-of-the-art performance achieved.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness section.", "broader_impact_concerns": "I don't have any prominent ethical concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R5pUFEt3LK", "reviewer": "Reviewer_JQcx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper is well motivated and timely -- using generated data to train models has the potential to reduce the need for manually annotated data while giving us better models\n- The paper has very clearly defined and targeted insights\n- The method is conceptually intuitive and clearly explained\n- The authors show good empirical results with their method, and their evaluation is reasonably rigorous.\n\n### Weaknesses \n\n- In the comparison with prior work (Azizi et al. 2023): it would be interesting to see how much difference just adding auxiliary batch norm parameters would make to their method.\n\nMinor clarity issues:\n- It's not clear where/how LAR output is used later (Fig. 2)\n- The description of LAR could use a figure (flowchart) to explain the steps exactly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose method to improve models by training them using off-the-shelf generative models. They set up an automated pipeline to generate labelled images while identifying two issues with using generative models -- lack of diversity in generated images and domain shift. To solve these they propose diversification techniques for both context and style, and use auxiliary batch-norm to mitigate domain shifts. They show that using their method, models can improve their performance by training on a much larger proportion of generated images without performance collapse.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Only the ones listed under \"minor clarity issues\" in weaknesses above.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an improved synthetic data generation pipeline for training large scale image classification. The core contributions lie in 
1) eliminating the need for fine-tuning, significantly simplifying the process and improving speed and 
2) disambiguate labels (e.g., crane), and 
3) diversification (e.g., foreground/background objects, lighting, viewpoint, styles).

All three reviewers agree that the paper showcases solid experimental results validating the effectiveness of the proposed synthetic data generation pipeline. Compared to prior method that relies on fine-tuning (Azizi et al., 2023), this approach achieves improved results without fine-tuning will be useful for end-users.
Decision: Accept (poster)