SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?

Authors: Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: CLIP, synthetic data, generative

Abstract:
We present SynthCLIP, a CLIP model trained on entirely synthetic text-image pairs. Leveraging recent text-to-image (TTI) networks and large language models (LLM), we generate synthetic datasets of images and corresponding captions at scale, with no human intervention. In this work, we provide an analysis on CLIP models trained on synthetic data. We provide insights on the data generation strategy, number of samples required, scaling trends, and resulting properties. We also introduce SynthCI-30M, a purely synthetic dataset comprising 30 million captioned images. Our work focuses on showing the advantages and disadvantages of synthetic data for training CLIP models. Our code, trained models, and data, will be released as open source.

Decision: reject
Reviewer scores: 5, 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1730308592954

thread_type:
official_review

<thread_object>
creator: reviewer_i2n9

<content>
summary:
The paper focuses on training a CLIP-based model using solely synthetic image-text pairs and study the effects of doing so.
SynthCLIP relies on the understanding that curating real multimodal data at scale comes with a cost of quality and alignment between images and their description. To this end, the authors propose to harness the advancement of Text-To-Image models and LLMs to generate a 30M purely synthetic dataset of image-caption pairs. Such a process enables to easily control the distribution of the data and to generate datasets at any scale with no human in the loop.
The authors study the effects of training CLIP with their propose dataset in a various of benchmarks including both vision and vision language tasks and compare it to training with real data.

strengths:
+ The paper is well written and easy to follow. 
+ Utilizing a purely synthetic dataset enables to control the data distribution and to collect data and any scale, without requiring human intervention.
+ Interesting and insightful ablation studies

weaknesses:
- Empirical demonstration of the motivation - the authors claim that in real datasets, increasing their scale comes with a cost of quality and the proposed approach mitigates this and enables collecting quality data in any scale. However, empirically proving this requires comparing the performance against much larger real datasets than CC12M. Outperforming a model trained on CC12M with a 30M dataset doesn't showcase the advantage in quality of the synthetic data. Moreover, the model trained with a10M synthetic data has a worse performance compared to the model trained with a similar amount of real data. Thus, I am afraid that the main claim of the paper was not empirically demonstrated.

- Novelty - The proposed framework is based on existing models and approaches. Unfortunately, constructing the concept bank, which could have been an interesting place for novelty, is taken from an existing work.

- The necessity of "human intervention" - real datasets mainly scraped from the internet and the caption is achieved from the alt-text in the html. Thus, curating large datasets is done automatically. Indeed, such datasets are often noisy and there are various methods for filtering them (for example, by a CLIP-score threshold) or bootstrapping [1]. These methods enable collecting huge scale dataset without requiring human intervention. While relying on alt-text for caption often lead to short and oversimplified captions, there are many works that tackle this by proposing automatic recaptioning [2,3]. Thus, one can utilize real images oriented dataset with high quality without human intervention.

- Fairness of comparison in the experimental section - The authors have stated that the training is done for a fixed number of epochs with a fixed batch size for datasets of different size. From my understanding, this leads to a different number of training iterations for datasets at a different scale, impairing the validity of the comparison.

[1] Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
[2] Improving clip training with language rewrites.
[3] FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions

questions:
- In lines 158-159 the authors state that the captioned are oriented for a single object. How would it effect the performance on tasks that require low-level details understanding? There are many works that try to train on detailed captions to incorporate such an understanding.

- TTI models are currently not good at generating text within images and in understanding relationships between objects. Wouldn't training on generated images result in a model with limited capabilities in such areas?

- Given figure 4, why would we need to generate images and not recaption ones that we can obtain easily by crawling the internet?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730590902449

thread_type:
official_review

<thread_object>
creator: reviewer_xqma

<content>
summary:
The work proposes a synthetic training protocol for CLIP models extending prior work by creating both generated captions and generated images. In the process, the method demonstrates superior performance to common small scale image-text such as CC12M by curating a dataset of 30M synthetic examples. To better understand how different elements of the pipeline affect performance, they also ablate different choices of language models, differences caused by synthetic data sources, and how concept distribution impacts performance.

strengths:
Each element of the synthetic data pipeline is soundly constructed, and follows community norms for composition. Furthermore, the work details a lot of the smaller design choices (e.g. LLM, prompt, and concept distribution) that contribute to the end to end system. The approach itself is noteworthy as it represents a full departure to synthetic data whereas most existing approaches require one of the modalities to be pre-existing.

Model benchmarks are extensive and representative of image-text model capabilities. They show both good diversity in dataset and task.

Some of the most interesting findings come from the paper's ablations. Figure 4a demonstrates the delta in performance that can be attributed to each different synthetic modality. In addition, it shows a potential failure mode of ignored generation commands and how they may be addressed. Additionally, the study on the concept bank is quite interesting providing support for the hypothesis that some of the difference in performance between natural and synthetic data is from the underlying conceptual distribution not a failure in quality. The experiments towards the mitigation of long-tail effects suggest an interesting direction for improving unseen or undersampled concepts in real world training.

weaknesses:
One concern with this work is that there is not sufficient evidence that the method might scale. Certain CLIP pretraining augmentations, like M3AE for example, have been shown to work at small scales, but yield no major benefit at larger scales [1]. Understanding that training frontier CLIP models is prohibitively expensive due to batch size needs, it’s sensible that this data is not available but worth keeping in mind.

The most immediate notice is the difference in data efficiency between real and synthetically drawn samples. Combined with the above, for practical applications it is not quite clear when one would adopt this method as it strictly leads to longer training runs and real training data is abundant (LAION-5B [2] and DataComp [3]). The method would benefit from further analysis into what is causing the reduction in performance, though some beginning analysis is done with respect to the concept distribution the gap is still left largely unexplained. 

[1] Weers et al. 2023 "Masked Autoencoding Does Not Help Natural Language Supervision at Scale"
[2] Schuhmann et al. 2022 "LAION-5B: An open large-scale dataset for training next generation image-text models"
[3] Gadre et al. 2023 "DataComp: In search of the next generation of multimodal datasets"

questions:
1. It isn’t quite clear how MTL is calculated. What is it averaging over?
2. In an effort to understand differences in synthetic distributions versus natural distributions, how does performance change when using a CC3M concept distribution sample equalized to the real CC3M similar to Table 4? Another experiment to get at some of this would be taking CC3M, for each image prompting the model for a single “concept” then creating a caption and image using the proposed pipeline.
3. To understand the differences in scaling, it would be helpful to know the coefficients of the error with respect to dataset size on a log scale. How do natural and synthetic data coefficients compare?
4. This experiment is less pertinent than the above, but with results on improving the long tail distribution there might be interesting robustness properties as a result of concept representation. How does the real versus natural data compare on effective robustness in a framework like that of [1]?

Overall, the work is well presented but would benefit from a coverage of the first three points, and less importantly the fourth, to round out its presentation. I’d be happy to raise my score if the above are addressed.

[1] Nguyen et al. 2023 "Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718699947

thread_type:
official_review

<thread_object>
creator: reviewer_c8nx

<content>
summary:
It's interesting to see the emerging trend of training CLIP models using synthetic data. This work introduces SynthCLIP, a CLIP model training on synthetic data comprising both synthetic captions and images. The paper not only proposed the pipeline for creating synthetic data but also release SynthCI-30M, a comprehensive dataset housing 30 million captioned images generated entirely synthetically. This work unveiled the potential of leveraging synthetic data to enhance CLIP model training.

strengths:
1. The research detailed the pipeline for entirely synthetic creation of image-text pairs and introduced the dataset SynthCI-30M.

2. A comprehensive set of experiments was conducted to elucidate the efficiency and practical value of synthetic data, demonstrating its scalability and effectiveness.

weaknesses:
1. The zero-shot image classification results lack depth to gauge effectiveness comprehensively.

2. The experiments solely compared with CC3M and CC12M. How would results differ if a subset of LAION-400M is employed instead?

questions:
1. I'm curious about the performance on ImageNet variants like ImageNetV2, ImageNet-A, ImageNet-R, and ObjectNet.

2. Will the performance of MLLM understanding tasks be enhanced by employing CLIP trained on SynthCI-30M?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730961319158

thread_type:
official_review

<thread_object>
creator: reviewer_xumb

<content>
summary:
The paper explores the performance of CLIP-style models trained on purely synthetic image-caption pairs (called SynthCLIP) generated by modern text-to-image diffusion models and LLMs. It studies the scaling trends of such models and also provides a dataset of 30 million captioned images.

strengths:
- The paper provides a dataset of 30 million captioned images from diverse set of concepts.
- As the concepts are fixed, one can gather a subset of them to train new models with less concerns about NSFW contents leaking into the training data compared to using real images. 
- The models trained with similar dataset scales using synthetic captioned images show similar performance on down-stream tasks.

weaknesses:
- The main weakness of the paper in my opinion is that the paper is not well-motivated. The introduction section does not provide convincing answers to the questions like "why should we use purely synthetic image-caption datasets? why not a hybrid approach? why is the problem significant?"

- Although controlling the concepts that are present in the dataset can be useful, there is no guarantee that the generated images for each concept are 1) faithful to the content and 2) do not contain NSFW content:
1) faithful to content: The described workflow only filters the captions, not the generated images. It is likely that the generated images contain noisy unrelated images. No workarounds in this regard has been proposed in the paper.

2) NSFW content: The latter may happen because the training of models like Stable diffusion have been on unfiltered datasets like LAION. It is possible that some NSFW contents appear with some concepts in these datasets frequently, resulting in generation of NSFW contents inadvertently. 

- Despite that the paper argues that one can use synthetic images from tail classes to augment the real datasets, I think it is not straightforward to do so. Although the idea seems sound, the Stable Diffusion (SD) model has been trained on real images that have the same long-tailed classes. Therefore, the performance of SD on these classes will not be satisfactory.

questions:
- I suggest that the authors improve the introduction by explaining the motivations and use-cases of employing a purely synthetic dataset to train CLIP models.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper discusses using pure synthetic data for training CLIP models, and shows the performance, scaling property, and analysis compared with real datasets. The paper also introduces a new SynthCI-30M with captions on 30 million images. However, the paper still lacks enough evidence on several key points such as the quality of content of generation images, not enough large-scale experiments and ablations, etc. The motivation of the work also needs to be justified. Therefore, based on the reviews, I recommend rejection of the paper.