Are Synthetic Classifiers Really as Good as Real Classifiers?

Authors: Jizhe Zhang, Yifei Wang, Stefanie Jegelka, Yisen Wang
Primary area: generative models
Keywords: generative model, representation learning, synthetic data

Abstract:
Foundation models have achieved significant advancements across various domains, yet their training demands vast amounts of real-world data, which is becoming increasingly scarce. To address this challenge, synthetic data has garnered substantial interest as an alternative for augmenting training datasets in fields such as computer vision and natural language processing. However, skepticism remains regarding whether synthetic classifiers can match the performance of those trained on real data. In this paper, we investigate this question by conducting a detailed analysis within the realm of visual tasks, comparing classifiers trained on synthetic versus real data using CLIP and ViT. Our results reveal that synthetic classifiers exhibit deficiencies in a range of challenging real-world scenarios, such as fine-grained classification, extreme object scales and extreme brightness despite achieving comparable overall accuracy to their real-data-trained counterparts. We find that the limitations of synthetic classifiers can be traced back to the limitations of current generative models in capturing the complexity and diversity of real-world data in these aspects. To mitigate these issues efficiently, we explore \textbf{RealTune}, a simple method that enhances synthetic classifiers by finetuning them with a small amount of real data. Experimental evaluations demonstrate that RealTune significantly improves the performance of synthetic classifiers using only a limited real dataset (e.g., 40k images,  3% of ImageNet) with minimal training time (e.g., 1hour on a single NVIDIA RTX 3090 GPU). Our findings indicate that while synthetic data is a valuable resource, integrating real and synthetic data is essential to achieve robust and efficient classifiers. This work underscores the necessity of leveraging both data types to bridge the performance gap and enhance the overall effectiveness of foundation models.

Decision: reject
Reviewer scores: 3, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1729023440629

thread_type:
official_review

<thread_object>
creator: reviewer_rppm

<content>
summary:
The paper investigates whether classifiers trained on synthetic data can match the performance of those trained on real data, especially in visual tasks. Through comparative analysis, the authors find that although synthetic classifiers achieve similar overall accuracy to real-data-trained counterparts, they underperform in challenging scenarios such as fine-grained classification, extreme object scales, and brightness variations. The authors attribute these limitations to the inability of current generative models to fully capture the complexity and diversity of real-world data. To address these issues, they propose **RealTune**, a method that fine-tunes synthetic classifiers with a small amount of real data, significantly improving performance in these complex scenarios. The results demonstrate that combining synthetic and real data is essential to create more robust and efficient classifiers.

strengths:
1. The paper provides a detailed comparison between classifiers trained on synthetic versus real data, especially in challenging scenarios like fine-grained classification and rare situations (e.g., extreme object scales, brightness variations, object occlusion). The authors use a range of quantitative metrics, such as employing YOLO models for object scale assessment and CIELAB color space for brightness evaluation, making their results systematic and reliable. This thorough analysis adds significant value to understanding the limitations of synthetic classifiers and potential improvements for future model development.
2. Generating synthetic data is relatively easy compared to collecting real-world data, but the quality and diversity often fall short. RealTune effectively addresses these shortcomings using minimal real data (around 3% of ImageNet), significantly enhancing the classifier’s performance. The proposed method also reduces computational and energy costs, making it particularly useful in resource-constrained environments. From a review perspective, this solution is practical, cost-effective, and demonstrates a clear advancement in addressing synthetic data challenges.
3. Beyond RealTune, the authors also explore the effectiveness of mixed data pretraining, showcasing the potential of combining real and synthetic data. This strategy not only improves classifier accuracy but also outperforms using either real or synthetic data alone. The experiments on mixed pretraining provide new insights into leveraging limited data resources effectively, which is crucial in scenarios where collecting large-scale real data is impractical. These contributions not only provide an empirical foundation for future research but also serve as valuable guidance for practical training strategies. The discussion of data-mixing approaches adds depth to the paper and offers an innovative direction for making optimal use of synthetic data.

weaknesses:
1. The paper exclusively focuses on vision tasks without exploring other modalities like text or audio, which limits the generalizability of the findings. Given the growing relevance of synthetic data in various domains beyond vision (such as natural language and speech), this is a significant limitation. The authors briefly mention extending the study to other modalities in future work (Section 6), but as it stands, the narrow focus weakens the overall contribution. Extending the experiments to non-visual modalities or providing some preliminary analysis could have increased the paper’s broader applicability.
2. In the evaluation of rare scenarios, such as object occlusion, extreme object scales, and brightness variations, the authors use ImageNet-X (Figure 3). However, the analysis lacks an adequate diversity of datasets that could highlight other real-world challenges (e.g., dynamic backgrounds, motion blur, or domain shifts). ImageNet-X, though useful, does not fully encompass the variety of rare scenarios that might be seen in real-world applications. For a thorough evaluation, incorporating other benchmarks like ImageNet-C or ObjectNet could have provided a more comprehensive assessment of classifier robustness. This would also have helped to better validate the claims about synthetic data limitations.
3. Some figures in the paper are difficult to interpret due to suboptimal presentation choices. For example, Figure 4, which aims to demonstrate class consistency and frequency for fine-grained class confusion, uses bar plots that make it challenging to interpret the differences across models at a glance. The class consistency rates presented are not clearly distinguished, leading to potential confusion. Additionally, the visualization of synthetic data distribution shows class imbalance, but the representation could have been more effective with a clearer breakdown across different class labels to provide insight into how imbalance specifically affects model performance. The authors could benefit from using more readable visualizations like heatmaps or swarm plots that better convey the underlying relationships.
4. While RealTune demonstrates promising improvements, the experimental evaluation lacks adequate baseline comparisons against other established methods for enhancing synthetic classifiers. For example, the paper introduces SynTune as a counterpart to RealTune, but it would have been more informative to compare RealTune with other fine-tuning or data augmentation techniques that are popular in the field. Including a detailed comparison with transfer learning or data distillation methods could have added value to demonstrate RealTune's efficiency more convincingly. Moreover, while Figure 6 shows accuracy improvements, adding more baselines could help to better gauge the significance of the presented results.
5. The mixed training strategy, explored in Section 4.3 and illustrated in Table 3, shows that combining real and synthetic data during pretraining can enhance the performance of classifiers. However, the analysis lacks depth in explaining why certain combinations outperform others and how different ratios of real to synthetic data impact the results. The choice of using only 7.7% of real data (for ImageNet-100) in the mixed dataset is arbitrary and not well justified, limiting the ability to generalize findings to other datasets or settings. Additionally, there are no detailed ablation studies exploring various ratios between real and synthetic data, which would have provided a better understanding of the trade-offs and the optimal way to mix these data types. This omission reduces the experimental rigor and leaves questions regarding the optimal strategy for practical scenarios.

questions:
1.*How well does RealTune generalize across different visual datasets or domains?*
RealTune was tested on ImageNet and its smaller subset, ImageNet-100, which are certainly widely used benchmarks. But I wonder—would the same level of improvement hold if we used a dataset with a lot more variation, like COCO, or even a domain-specific dataset like medical imagery? These types of datasets have different complexities, like intricate scene layouts or highly specialized objects, which might expose weaknesses that did not appear with ImageNet. It would be helpful to know if RealTune is a broadly applicable method for any kind of visual domain or if its effectiveness is more specific to the characteristics of ImageNet.

2.*How does the quality of the fine-tuning data affect RealTune's success?*
The authors showed that RealTune works well with a small amount of real data, but it made me wonder about the specifics of that fine-tuning data. What if the real data used for fine-tuning is biased or lacks diversity? Would RealTune still perform as well? Real-world data collection often has biases, like unbalanced classes or limited examples of certain challenging conditions, and it would be useful to understand how that affects the final model. Does RealTune require a carefully balanced and curated fine-tuning set, or can it adapt well even if the real data is subpar? It would be great to see more on whether the composition of this fine-tuning data matters.

3.*How does RealTune's computational efficiency compare with other methods in terms of energy use and scalability?*
RealTune was described as efficient, running on a single GPU in a relatively short time. But I would like to know more about how it stacks up against other methods in terms of energy usage, especially considering the push for more environmentally friendly AI solutions. How does RealTune compare, for example, to larger pretraining strategies or other fine-tuning techniques when it comes to energy consumption or the practicality of scaling to larger models? Understanding these trade-offs would be really important for someone trying to decide between RealTune and other enhancement methods, especially in scenarios with limited computational resources. It would also help gauge if the method's efficiency benefits hold up when scaling to bigger models or if there are diminishing returns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730453543187

thread_type:
official_review

<thread_object>
creator: reviewer_5xmn

<content>
summary:
This paper investigated the performance of real classifiers and synthetic classifiers. In particular, the authors claim that synthetic classifiers exhibit deficiencies in a range of challenging real-world scenarios, such as fine-grained classification, extreme object scales and extreme brightness despite achieving comparable overall accuracy to their real-data-trained counterparts. To this end, the authors proposed RealTune, a method that enhances synthetic classifiers by finetuning them with a small amount of real data. Experimental evaluations demonstrate that RealTune significantly improves the performance of synthetic classifiers using only a limited real dataset.

strengths:
This paper is well presented and easy to follow.

weaknesses:
In general, this paper conducted both analysis on synthetic classifiers and proposed an approach, however both parts require tremendous extra work to improve for a solid contribution. 

* a) The claim that synthetic classifiers perform worse on fine-grained dataset and rare scenarios is not well justified. From Tab. 1 and Fig. 2, synthetic classifiers and real classifiers show similar trend on both IN results and fine-grained results. Synthetic 371M shows very similar accuracy as real 64M. This is the case for both regular IN images and fine-grained images. In Fig. 3, the so called relative accuracy difference is 0.5%, which does not show significance. Not to mention that there are positive values, indicating poor consistency. 

* b) The fact that generative models fail to generate fine grained is simply not true. There are many works that focus on generating images in the context of fine-grained images. For instance, SphericGAN: Semi-supervised Hyper-spherical Generative Adversarial Networks for Fine-grained Image Synthesis; Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation; The authors should study the synthetic classifiers with data generated with models designed specifically for Fine-Grained Images. 

* c) There is no comparison conducted w.r.t prior arts that work with training models using mixture of synthetic data and real data. As surveyed by the authors themselves in the related work section, yet the authors compared with none of them.

* d) Training models with both synthetic data and real data is simply not novel at all.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730669437110

thread_type:
official_review

<thread_object>
creator: reviewer_urhh

<content>
summary:
This paper performs an analysis of pretraining on synthetic generated images. The authors break down some ways in which these models fall short and how this is reflected in the limits of image generators. For example, an off-the-shelf image generator does not produce images with much variety in image brightness so a classifier trained on these images tends to do worse on very dark or very bright images. Similarly, the image generator will make mistakes and mix up fine-grained semantic categories which leads to noisier supervision.

The authors discuss ways to potentially address these issues when generating images (attempting to adjust settings and prompts) but show there is little to be done on that front. What proves more effective is finetuning on a small set of randomly sampled real data (40k images). It is also helpful to pretrain on a mix of real and synthetic data.

strengths:
This exposition of this paper is quite good, the authors point to very specific limitations and take us through their process to address them. There are quite a number of interesting and detailed analyses sprinkled through the paper, some in particular that stand out to me:
- I like the thought that went into Section 3, and the various ways to pick at how good a generative model is at producing reasonable training data (e.g. the class consistency rate in Figure 4a)
- I was glad the comparison in Table 2 was included that showed the results of training on either randomly sampled real data or a curated subset to address the limitations that had been discussed so far in the paper. It is always a tricky trade-off trying to address rare failure modes directly since by definition they only make up a small fraction of the overall validation setting.

Overall the paper is incredibly thorough, covering a wide variety of evaluation settings, and digging into ways to get more specific tangible insights into where these models do well and poorly. I appreciate the effort it takes to go beyond a simple comparison of overall accuracy on ImageNet.

weaknesses:
- Much of the paper focuses on specific weaknesses that arise due to limitations of the image generator (image brightness, object scale, complex scenes). And the authors argue that these limitations lead to particularly pronounced failure modes in the model trained on synthetic data. I don't know that I agree that these failures are in fact specific to models trained on synthetic data. Looking at Figures 3 and 8, all of the error modes are very highly correlated between models trained on either real or synthetic images. Models trained on real images show the exact same weaknesses to "person blocking" and darker images and smaller objects. Given that this is such a fundamental part of the analysis it is odd to me that it really doesn't stand out as a synthetic data issue in particular, but instead as a more general way these models fail.

- One issue I have is that the authors do not discuss how much data goes into training the image generator. I feel like this is important context to the investigation. As far as I understand, Stable Diffusion was trained on LAION which consists of billions of image-text pairs. And my mental model of these image generators is that it is not unreasonable for them to reproduce source images almost perfectly. So it's not like this is truly "synthetic", some fraction of the data being generated probably bears high similarity to real source images with corresponding captions that match ImageNet labels. If anything, it is surprising that a model trained on that much data still does not know the difference between a rooster and a hen. So do the billions of images used to train StableDiffusion not count when discussing how much data is needed to get these results?

- I find it interesting that some of the specific failure modes seem easily addressed by data augmentation. There is focus in Section 3 on how to prompt the image generator to produce smaller/larger objects or whether other generators don't have the same light/dark limitations, but cropping and resizing and brightness augmentations could all trivially expose the model to more diverse object sizes and image conditions, no?

- The citation provided discussing whether real world data is limited (Villalobos et al 2024) makes a comment about there only being a couple trillion images taken each year while in this investigation we are looking at the impact of tens of thousands of images. How much are we at risk of running out of data?

questions:
While maybe somewhat orthogonal, were any tests of data augmentations done to address any of the effects discussed here? Seems like data augmentation could complement some of what is more difficult to control in the image generation process.

It is interesting that we can train classifiers directly on the output of generative models. Given that, the real question someone might want answered is how to get a strong classifier with large numbers of unlabeled/weakly-labeled images and a small amount of real labeled data. I am not sure the most convincing case is being made that is worth it go through the roundabout process of producing millions of outputs from a generative model while mixing in a little bit of real data. How does performance compare to sampling and training on say, a million LAION images whose captions match up to the imagenet labels? Would that be a reasonable point of comparison? Similarly how would this compare to a model that is pretrained directly on LAION in the style of CLIP?

On a related note, it might helpful context to report how other pretrained models fare when finetuned on a small random subset of ImageNet. I am curious how other modern self-supervised pretraining strategies might do for example.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731146971370

thread_type:
official_review

<thread_object>
creator: reviewer_yzjf

<content>
summary:
The paper presents a quantitative study of closed-set classification accuracy of models trained on synthetic vs real data across various settings, vanilla ImageNet testing, finegrained classification over ImageNet finegrained classes and a list of rare scenarios defined. The paper also ablates the impact of classifier-free guidance scale parameter, text prompt design, generative model over the trained model accuracy using thus generated synthetic data. It also identifies three specific factors that lead to poorer accuracy for models trained purely with synthetic data on object scale, brightness and occlusion by human. Then the paper proposes to fine-tune using a small amount of real data to mitigate the lower accuracy of model trained with synthetic data on these scenarios.

strengths:
The paper covers a number of quantitative tests of comparisons of models trained with real vs synthetic data. The motivation to go deep into the training data generated by SD to understand where potentially the gap of downstream trained classifier could come from makes sense.

weaknesses:
The described scenarios are rather limited and narrow and lack of generalization to broader concepts, e.g., scale, occlusion with human are very specific settings. 

Lack of comprehensive evaluation. There is no comparison of baselines with simple data augmentation on object scale and pixel occlusion. Also, Figure 4b mentions class imbalance as an issue for downstream model training. For this specific case, do class rebalancing techniques help solve the issue already? similarly for scale and occlusions, there is also no comparisons with baselines with simple data augmentation techniques.

In Figure2, the gap between similar ImageNet accuracy models trained with real and synthetic data over the finegrained domain does not appear to be very large - for example. 83.5 of CLIP-REAL-64M vs 83.1 of CLIP-Sync-371M, and 86.21 of ViT-Real-0.25M vs 85.4 of ViT-Sync-2M. It is unclear if these accuracy gaps are statistically significant or within the regime of model sensitivity. Thus it is unclear if the claim of synthetically trained models performing worse on fine-grained categories is technically sound or not.

Lack of technical novelty. The idea of pretraining with synthetic data and then fine-tune using a small amount of real data has been adopted in many prior works [1] [2].

[1] Task2Sim: Towards Effective Pre-training and Transfer from Synthetic Data
[2] From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Bias

questions:
In Figure6, do the vanilla and SynTune baseline also use the additional 40K real samples used in the proposed RealTune setting? if not, it's not surprising that RealTune gets better since it leverages additional source of information.

It seems that here are cases where RealTune is worse than Vanilla, e.g., ViT-Real-1M setting, why is that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732289233763

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Update

comment:
We sincerely thank all reviewers for their thorough reading and valuable feedback. We have thoughtfully addressed their concerns and integrated these suggestions into the revised manuscript. The key revisions include:

- **The ImageNet-X evaluation results are scaled up 100 times (Figure 3, Figure 8, Table 2, Table 5).** We apologize for forgetting to multiply by 100% at first. From the modified results, it is evident that synthetic classifiers perform significantly worse than real classifiers in certain rare scenarios.
- Sec 2.1 Added discussion of the model’s evaluation results on ImageNet-C and ImageNet-3DCC from paper [1].
- Sec 3.1 replaced Figure 4(b) from a predicted label histogram to a fine-grained confusion matrix of 8 snake species.
- Sec 4：Added analysis of the results in Table 3 in section4.3; Added RealTune results on GPT2 in section 4.4.
- Sec 5: A part of the related works are moved to Appendix C due to space constraints.
- **Appendix B: add many new analysis results** on 1) comparison of different mixing methods, 2) results on different ratio of real and synthetic data, 3) ablation study of data augmentation, 4) DINO results 5) results on Caltech101 and EuroSAT 6) results on unbalanced datasets.

> [1] Krishnakant Singh, Thanush Navaratnam, Jannik Holmer, Simone Schaub-Meyer, and Stefan Roth. Is synthetic data all we need? benchmarking the robustness of models trained with synthetic images. In CVPR, 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the classifiers trained on synthetic vs real datasets using CLIP and ViT, on different benchmarks such as fine-grained classification. The results show that synthetic classifiers exhibit deficiencies, and with the proposed RealTune approach, the classifier performance can be significantly improved with a small set of real data. The paper provides a number of evaluations, and the presentation is of high quality. However, the novelty of the work is still limited given the focus of the work is narrow, and the effectiveness of RealTune is also limited in some evaluations. I would recommend rejecting this work.