SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?

## Abstract

Abstract We present SynthCLIP, a CLIP model trained on entirely synthetic text-image pairs. Leveraging recent text-to-image (TTI) networks and large language models (LLM), we generate synthetic datasets of
images and corresponding captions at scale, with no human intervention. In this work, we provide an analysis on CLIP models trained on synthetic data. We provide insights on the data generation
strategy, number of samples required, scaling trends, and resulting properties. We also introduce SynthCI-30M, a purely synthetic dataset comprising 30 million captioned images. Our code, trained
models, and data, are released at https://github.com/hammoudhasan/SynthCLIP .

## 1 Introduction

Self-supervised training strategies (He et al., , ; Caron et al., , ; Chen et al., , ) are fundamental for many recently released foundation models. These techniques make use of a vast amount of data without incurring a large annotation cost. In particular, contrastive representation learning (Schroff et al., , ) has been successfully employed to extract joint embeddings for heterogeneous data modalities. By using multi-modal training data, CLIP () provides a common representation that links visual and linguistic information. Today, CLIP encoders are included in a wide range of applications, spanning from zero-shot image understanding (; Ren et al., , ), to style transfer (Kwon and Ye, , ), and robotics control (Shridhar et al., , ), among others.

Training CLIP requires large-scale captioned image datasets that are often collected from the web. Unfortunately, retrieving these data from the internet may present significant disadvantages (; Piktus et al., , ; Kang et al., , ). While it is easy to scale the number of unique samples, this comes at an increased difficulty in controlling their quality, which may result in a poor alignment between images and corresponding captions. Moreover, it is challenging to monitor the content of the collected images, resulting in potential concerns for illegal(^1^11We report a recent [article in mainstream news](https://www.telegraph.co.uk/business/2023/12/20/fears-ai-trained-child-abuse-images-thousands-discovered/) on the topic.) or copyrighted content. As an alternative to web-crawled data, some have explored the usage of synthetic data to train supervised (Sariyildiz et al., , ; He et al., , ) or self-supervised (Tian et al., , ) networks.

In this paper, our goal is to investigate the performances of CLIP models trained on fully synthetic data in the form of captioned images. To achieve our objective, we train SynthCLIP, a CLIP model trained exclusively on large-scale generated data. We propose a pipeline that jointly leverages existing text-to-image (TTI) and large language models (LLM) to produce text-image pairs. The captioned images are generated in an end-to-end fashion starting from a large list of concepts which guarantees variability of the synthesized data. We use the LLM to produce captions starting from sampled concepts, and then synthesize their corresponding images using TTI models. Our pipeline brings a significant advantage: we can generate data at any scale, arbitrarily increasing the size of training data depending only on computational power, with no human intervention. This allows for a controlled study on SynthCLIP performance and properties. Our contributions are threefold:

- 1.
We propose SynthCLIP, a CLIP model trained entirely on synthetic data generated with an automatic pipeline that is scalable to any desired dataset size.
- 2.
We provide an extensive study on SynthCLIP, including performance evaluation on five tasks and multiple datasets, and a comprehensive analysis of the resulting properties from training on synthetic data.
- 3.
We release SynCI-30M, an entirely synthetic dataset produced using our generation pipeline. It is composed of 30 million pairs of images and corresponding captions. We also release models trained on different synthetic dataset scales, and the code to generate the dataset.

## 2 Related Work

Representation Learning. Early works in representation learning on images used pre-text tasks such as inpainting, jigsaw puzzle solving, and image rotation prediction (Pathak et al., , ; Noroozi and Favaro, , ; Gidaris et al., , ). Instead, SimCLR (Chen et al., , ) leverages contrastive learning to maximize the similarity between two augmented views of the same image. Alternatively, masked autoencoders (MAE) (He et al., , ) use a masked patch prediction task to learn visual representations. CLIP () and other similar works (Mu et al., , ; Zhai et al., , ; Fini et al., , ) use contrastive learning to learn joint visual and textual representations. Language-image pre-training necessitates high-quality text-image pairs. Its core idea is to maximize the similarity between encoded textual and image representation. We study the possibility of generating synthetic text-image pairs for training CLIP-like models starting from simple concepts only.

Synthetic Captions. Recent works emphasize the importance of high-quality and aligned text-image pairs when training CLIP models, and propose a synthetic caption generation pipeline for improving it. VeCLIP (Lai et al., , ) and CapsFusion () propose methods to produce aligned captions. Both start with a captioning model such as BLIP (Li et al., , ) or LLaVA (), to obtain a semantically-rich synthetic caption. However, captioning models suffer from over-simplification and lack world knowledge, hence they can be compensated by the usage of an LLM (Lai et al., , ; ). LaCLIP (Fan et al., , ) improves the text branch of CLIP by leveraging an LLM to provide multiple rewrites of the same caption to use in contrastive learning. While this improves downstream tasks, it may not reflect the content of the image due to hallucinations (Fan et al., , ). All these works assume the availability of real data, instead, we introduce a fully synthetic pipeline for data generation, allowing arbitrary scalability and control over generated samples.

Synthetic Data. Synthetic data has been used in machine learning fields ranging from audio (Rossenbach et al., , ) to language (Yang et al., , ; ) and vision (Varol et al., , ; Jahanian et al., , ; Zhou et al., , ). In computer vision, they allow for improving models’ performance on several downstream tasks such as semantic segmentation (Richter et al., , ; Ros et al., , ; Chen et al., , ), object detection (Johnson-Roberson et al., , ), and image classification (Yuan et al., , ; Shmelkov et al., , ). Recent works have explored the use of synthetic data from TTI models, to augment training on real data (Azizi et al., , ; Sariyildiz et al., , ; He et al., , ). uses a framework to generate synthetic images, increasing the diversity of existing datasets. All these assume knowledge about object classes in the downstream task, and work with images only. Recently, StableRep (Tian et al., , ) showed that synthetic images generated from Stable Diffusion can be used to train self-supervised methods, however, it still uses real captions, limiting scalability and proper analysis. Parallel works (Fan et al., , ; Tian et al., , ) preliminarily investigate training of vision models on synthetic captions and images. However, they are not specific to CLIP, and they include a small set of classes for generation, preventing appropriate analysis. We discuss our positioning relative to these works in the appendix.

Figure: Figure 1: Pipeline Overview. From a set of concepts $\mathcal{C}$ (left), we obtain a set of synthetic captions $\mathcal{T}$ with an LLM, further refined to $\mathcal{T}^{*}$ by a filtering balanced sampling operation (top). Generated captions are used to prompt a text-to-image model, obtaining synthetic images aligned with the caption (bottom). We then train CLIP on the generated text-image pairs (right).
Refer to caption: x1.png

## 3 Training SynthCLIP

In this section, we present the training procedure for SynthCLIP. We show the pipeline in Figure 1. First, we start by identifying a concept bank containing many raw visual concepts, *i.e*. words that can be associated with their corresponding representations in images. This broad definition covers either common objects, items, and animals (*e.g*. “cat”), proper nouns and specific elements (*e.g*. “Eiffel Tower”), and intangible items associated with specific visual characteristics (*e.g*. “love”, that is often represented with stylized hearts). An LLM is then prompted to generate captions for all the concepts in the concept bank, leading to a set of synthetic captions (Section 3.1). The generated captions are then filtered to a smaller set for improved performance (Section 3.2). These filtered captions are then passed to a text-to-image model to generate corresponding images (Section 3.3). After obtaining our synthetic $\{\text{caption},\text{image}\}$ pairs, a standard CLIP training is carried on the generated data, obtaining the language and text encoders that can be used for downstream tasks (Section 3.4). We now describe each step in detail.

### 3.1 Step 1: Concept-based Captions Generation

The first stage of our pipeline involves the generation of synthetic image captions, that we later aim to use as prompts for text-to-image generators. To achieve this, we utilize an LLM conditioned on our concept bank. The model is prompted to generate captions that describe a scene related to a chosen concept. In our process of generating these captions, we experimented with various prompting techniques, discovering that conditioning the LLM to focus on a particular concept leads to more diverse captions. Concept conditioning ensures that the LLM does not just repeatedly produce captions about a limited set of concepts, over-represented in the LLM training dataset. In other words, this approach helps prevent the model from becoming biased towards certain concepts, and encourages a broader spectrum of caption generation. Limited concept diversity hinders the CLIP training, since contrastive learning highly benefits from variability and more concept coverage (Xu et al., , ). Hence, diversity is a requirement for a proper analysis of SynthCLIP.

We start by introducing our concept bank $\mathcal{C}$ composed by $N_{\mathcal{C}}$ concepts. Unless otherwise stated, we use the MetaCLIP concept bank (Xu et al., , ), that contains over 500,000 concepts drawn from WordNet Synsets and Wikipedia common unigrams, bigrams, and titles. We observe that $N_{C}$ deeply influences CLIP performance, and we investigate this effect in Section 4.3. We then focus on prompt engineering, a critical aspect for generating effective captions for text-to-image generation.

Image generators are sensitive to the quality of the input prompt (Gu et al., , ), which is often a brief text description capturing the characteristics of the desired image. We set specific requirements to ensure that the prompts generated by the LLM are well-suited for the subsequent image generation:

(1) Focus on a Single Concept: Each generated caption should be centered around a single concept, presented in a clear and coherent context.
(2) Brevity and Clarity: The prompts need to be concise yet grammatically correct. The goal is to avoid overly complex or vague inputs that could lead to ambiguous or incorrect images.
(3) Prompt-Only Generation: Our aim is to have the LLM generate prompts without engaging in further reasoning or elaboration. This approach not only saves computational resources, but also simplifies the parsing process.

Assuming $c\in\mathcal{C}$, our designed prompt is:

Formally, we define our LLM generator as $G_{\text{LLM}}$ and the prompt as $p$. Hence, the set of generated captions is $\mathcal{T}=\{t_{c,n}\sim G_{\text{LLM}}(p,c)\},\forall c\in\mathcal{C},
\forall n\in\{1,2,...,N\}$ where $N$ is the number of desired captions for each concept. By looking at the captions in Figure 2, we show how this mechanism results in a highly variable contextual placement of each concept.

Figure: Figure 2: Generation samples. We show generated captions and images pairs for concepts “cat” and “Paris”. Our pipeline provides high variability and realistic contextual placement of input concepts.
Refer to caption: x2.png

### 3.2 Step 2: Captions filtering

When generating captions conditioned on a specific concept $c$, it is typical for other concepts $c^{\prime}\neq c,c^{\prime}\in\mathcal{C}$ to appear within the same caption. This is expected, since even when a sentence is focused on a single concept, other related concepts emerge within the described scene. For example, if $c=\text{``{bird}''}$, a generated caption might be “a bird is resting on a tree”, introducing an additional concept $c^{\prime}=\text{``{tree}''}$. This LLM-specific behavior may create imbalances in the generated data for CLIP training, which benefits from the usage of a balanced amount of concepts (Xu et al., , ).

We create a balanced set of captions, $\mathcal{T}^{*}$, by applying the balancing sampling method proposed in MetaCLIP (Xu et al., , ) to our setting. It consists of two stages: substring matching, and probability balancing. Substring matching determines which concepts from $\mathcal{C}$ appear in each caption within $\mathcal{T}$. This enables us to measure the real frequency of each described concept across the synthesized captions. Probability balancing is then employed to subsample captions $\mathcal{T}^{*}$ from $\mathcal{T}$. It increases the probability of selecting captions with long tail concepts, preventing over-representation of frequently-generated concepts. This yields a subset of captions where both frequent and long-tail concepts are adequately represented. Hence, this approach ensures a diverse and task-agnostic captions set suitable for foundation model pre-training. By sizing the parameters of balanced sampling, we are able to choose the size of the subset $\mathcal{T}^{*}$. For more details, we refer to (Xu et al., , ).

### 3.3 Step 3: Image Generation

Having successfully created a balanced set of synthetic captions $\mathcal{T}^{*}$, our next step is to generate the corresponding images. For this, we utilize a text-to-image generator $G_{\text{TTI}}$. We choose Stable Diffusion (Rombach et al., , ) for this purpose, due to its open-source availability and relatively lower computational demands. For each caption in our set $\mathcal{T}^{*}$, we generate a corresponding image. This process results in a collection of images, $\mathcal{I}^{*}=\{x_{k}\sim G_{\text{TTI}}(t_{k})\}$, where each $x_{k}$ is an image synthesized from the caption $t_{k}\in\mathcal{T}^{*}$. In Figure 2, we show how we generate highly aligned images which correctly capture the described scene and complement it with related realistic information. This proves the efficacy of our caption generation pipeline, leading to appropriate image generation.

### 3.4 Step 4: CLIP Training

Finally, we use the synthetic text-image pairs to train a CLIP model, to explore how effectively a model can learn from entirely synthetic data. We train two encoders, each one dedicated to either the image or text modality, defined as $E_{\text{image}}$ and $E_{\text{text}}$, respectively. We follow the standard CLIP training pipeline (), by applying a contrastive loss on the image and text representations through the encoders. Formally, we extract representations $h=E_{\text{image}}(x_{k}),x_{k}\in\mathcal{I}^{*}$ and $z=E_{\text{text}}(t_{k}),t_{k}\in\mathcal{T}^{*}$, and train by minimizing the CLIP loss $\mathcal{L}_{\text{CLIP}}(h,z)$.

## 4 Experiments

In this section, we evaluate the performance of SynthCLIP. We start by introducing the experimental setup in Section 4.1, presenting datasets, generation models, and downstream tasks. Section 4.2 benchmarks SynthCLIP against baselines trained on real data on multiple tasks. Finally, Section 4.3 includes an analysis of the properties of SynthCLIP and ablation studies for the introduced components.

### 4.1 Experimental Setup

Downstream Tasks
We use five different downstream tasks to assess performance. For ease of evaluation, we categorize the downstream tasks into two categories; (1) Vision Tasks and (2) Vision-Language Tasks. The first focuses on evaluating the capabilities of the frozen vision encoder $E_{\text{image}}$ only, *i.e*., linear probing and few-shot classification.
The second evaluates the synergy between the image encoder $E_{\text{image}}$ and text encoder $E_{\text{text}}$ together. We use image retrieval, text retrieval, and vision-language zero-shot classification tasks as evaluation tasks, following the original CLIP (). Since our evaluation pipeline consists of several tasks whose metrics can behave differenty, we aggregate performance across all tasks using the $\Delta_{\text{MTL}}$ metric (Vandenhende et al., , ), where a model with positive $\Delta_{\text{MTL}}$ indicates an overall better performance compared to a reference baseline.

Datasets We use the real datasets CC3M (Sharma et al., , ) ($3\times 10^{6}$ samples) and CC12M (Changpinyo et al., , ) ($8.8\times 10^{6}$ samples(^2^22The original CC12M is composed of 12M samples. In December 2023, only 8.8M images were available at the linked URLs.)). Real images come at different resolutions, so we resize the shorter edge of the images to 256px. For SynthCLIP, we generate an entirely synthetic dataset, that we call SynthCI (Synthetic Captions-Images) at different scales (number of samples). We refer to SynthCI-3M for a version of SynthCI where $\mathcal{T}^{*}$ and $\mathcal{I}^{*}$ include $3\times 10^{6}$ captions and images, respectively.
For zero-shot evaluation we use ImageNet (Russakovsky et al., , ), for linear probing and few shot we use CIFAR10 (), CIFAR100 (), Aircraft (Maji et al., , ), DTD (Cimpoi et al., , ), Flowers (Nilsback and Zisserman, , ), Pets (Parkhi et al., , ), SUN397 (Xiao et al., , ), Caltech-101 (Fei-Fei et al., , ) and Food-101 (Bossard et al., , ), and for image and text retrieval we use MSCOCO (Lin et al., , ), Flickr8K (Hodosh et al., , ) and Flickr30K (Young et al., , ).

Caption & Image Generation Models
For caption generation, we use Mistral-7B-Instruct V0.2 (Jiang et al., , ) with temperature $0.7$ and top-p set to $0.95$. We also set the presence and frequency penalties at $1$.
For image synthesis, we use Stable Diffusion v1.5 (Rombach et al., , ) with classifier-free guidance set to $2$ and $50$ Denoising Diffusion Implicit Models (DDIM) steps following Tian et al., (). The images are generated at $512\times 512$px and then stored on disk at $256\times 256$px. It takes $0.9$ seconds to generate and save one image on NVIDIA A100 GPU. Image generation was performed on a 48 A100-80GB GPU cluster.

**(a) Vision Tasks**
|  | Network | Data | Samples<br>($\times 10^{6})$ | Synth. data | CIFAR10 | CIFAR100 | Aircraft | DTD | Flowers | Pets | SUN397 | Caltech-101 | Food-101 | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Linear Probing | CLIP | CC3M | 3 | ✗ | 81.8 | 62.7 | 34.7 | 57.3 | 84.1 | 60.5 | 54.3 | 75.6 | 58.7 | 63.3 |
| CC12M | 8.8 | ✗ | 91.3 | 73.0 | 48.5 | 69.6 | 92.2 | 81.3 | 68.9 | 88.2 | 77.7 | 76.7 |  |  |
| SynthCLIP | SynthCI-3M | 3 | ✓ | 80.9 | 60.7 | 36.3 | 60.6 | 85.9 | 59.3 | 55.4 | 73.8 | 60.7 | 63.7 |  |
| SynthCI-8.8M | 8.8 | ✓ | 85.9 | 65.9 | 44.0 | 68.7 | 90.0 | 71.8 | 64.2 | 83.0 | 71.6 | 71.7 |  |  |
| \cdashline3-15 | SynthCI-10M | 10 | ✓ | 86.4 | 67.8 | 44.9 | 68.8 | 90.4 | 71.9 | 64.8 | 85.2 | 72.2 | 72.5 |  |
|  | SynthCI-20M | 20 | ✓ | 87.7 | 68.5 | 47.0 | 70.7 | 92.1 | 75.9 | 68.3 | 86.3 | 75.3 | 74.6 |  |
|  | SynthCI-30M | 30 | ✓ | 88.0 | 69.6 | 45.3 | 71.0 | 92.4 | 77.6 | 69.0 | 86.2 | 76.0 | 75.0 |  |
| Few-shot | CLIP | CC3M | 3 | ✗ | 61.4 | 70.9 | 45.2 | 73.2 | 93.0 | 71.0 | 93.3 | 91.6 | 68.2 | 74.2 |
| CC12M | 8.8 | ✗ | 80.3 | 83.5 | 55.7 | 82.0 | 96.8 | 85.5 | 96.9 | 97.4 | 86.3 | 84.9 |  |  |
| SynthCLIP | SynthCI-3M | 3 | ✓ | 57.6 | 68.8 | 47.2 | 74.3 | 93.5 | 70.8 | 93.5 | 89.9 | 68.3 | 73.8 |  |
| SynthCI-8.8M | 8.8 | ✓ | 62.4 | 73.3 | 56.9 | 79.6 | 95.7 | 80.9 | 95.8 | 95.1 | 78.4 | 79.8 |  |  |
| \cdashline3-15 | SynthCI-10M | 10 | ✓ | 67.0 | 75.1 | 59.3 | 80.4 | 95.9 | 82.8 | 95.9 | 95.4 | 79.4 | 81.2 |  |
|  | SynthCI-20M | 20 | ✓ | 70.6 | 77.4 | 64.4 | 81.4 | 96.7 | 84.7 | 96.6 | 96.1 | 82.8 | 83.4 |  |
|  | SynthCI-30M | 30 | ✓ | 74.0 | 80.8 | 66.1 | 82.5 | 97.2 | 86.2 | 96.8 | 96.5 | 83.6 | 84.9 |  |

### 4.2 Benchmark Evaluation

Performance on the same data scale We evaluate the effectiveness of our entirely synthetic data generation pipeline for training CLIP models compared to training on real data. We use CLIP () trained on CC3M and CC12M as baselines. We first train SynthCLIP on two versions of SynthCI each matching the data scale of CC3M and CC12M, which we call SynthCI-3M and SynthCI-8.8M, respectively.
We report the performance on vision tasks in Table 4.1 and vision-language tasks in Table 4.1, aggregating all metrics with $\Delta_{\text{MTL}}$ (Vandenhende et al., , ) in Table 4.1. As visible in Table 4.1, we obtain lower performance when both datasets are composed by $3\times 10^{6}$ samples (-5.60%) and $8.8\times 10^{6}$ samples (-15.0%), compared to the corresponding real data training with the same dataset size (CC3M and CC12M, respectively). This is expected: considering that real and synthetic data differ in distribution, while training on synthetic samples and testing on real ones, we incur a distribution shift, which ultimately harms performance (Zhou et al., , ; Fan et al., , ).

Scaling SynthCLIP
Our objective now is to analyze the scaling of SynthCLIP, to match performance obtainable by training CLIP on real data. It is indeed known that bigger training datasets help to increase performance (). However, while scaling real datasets necessitates custom collection pipelines from different sources and data curation, we exploit the advantage of our data synthesis pipeline, *i.e*. the capability to scale the size of the training data with no human intervention. In practice, we simply let our generation script run for longer, and re-train SynthCLIP on the larger SynthCI version obtained doing so. In particular, we report performance for SynthCLIP trained on {$10\times 10^{6}$, $20\times 10^{6}$, $30\times 10^{6}$} SynthCI samples, finally matching with 30 million samples the performance of the biggest model we trained on real data (CLIP on CC12M), against which we achieve $\Delta_{\text{MTL}}=\text{\color[rgb]{0.13,0.55,0.13}\definecolor[named]{
pgfstrokecolor}{rgb}{0.13,0.55,0.13}{+0.20\%}}$. We also report a significant increase with respect to CLIP trained on CC3M ($\Delta_{\text{MTL}}=\text{\color[rgb]{0.13,0.55,0.13}\definecolor[named]{
pgfstrokecolor}{rgb}{0.13,0.55,0.13}{{+60.1\%}}}$). From a single task perspective, we outperform CLIP trained on CC12M on image and text retrieval (+2.8% and +5.4%, respectively), while performing competitively with linear probing (-1.7%) and few-shot (+0.00%). While we still underperform in zero-shot evaluation (-3.1%), we attribute this also to additional bias effects that we study in Section 4.3. Ultimately, our experiment shows that by conditioning on a generic list of visual concepts, SynthCLIP can scale and match CLIP trained on large datasets, albeit using more samples due to the distribution shift between our synthetic training data and real test datasets.

**Figure 3: Performance improvement for different SynthCI scales. We show the improvements for all metrics with respect to SynthCLIP trained on SynthCI-3M. Vision-language tasks exhibit better absolute improvements and less saturation with respect to vision ones.**
| Network | SynthCI<br>samples<br>($\times 10^{6})$ | CC12M<br>samples<br>($\times 10^{6})$ | Lin. Prob. | Few-shot | Img Ret. | Text Ret. | IN 0-shot | $\Delta_{\text{MTL}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP | - | 8.8 | 76.7 | 84.9 | 58.9 | 71.7 | 33.6 | $\Rsh$ |
| SynthCLIP | 30 | - | 75.0 | 84.9 | 61.7 | 77.1 | 30.5 | +0.2% |
| Finetuned<br>(5 epochs) | 30 | 0.5 | 76.2 | 86.0 | 67.5 | 74.7 | 34.4 | +4.4% |
| 30 | 2.0 | 76.1 | 86.1 | 67.8 | 79.3 | 35.2 | +6.2% |  |
| 30 | 8.8 | 77.3 | 86.9 | 68.6 | 80.8 | 37.9 | +9.0% |  |
| \cdashline1-9<br>Finetuned<br>(10 epochs) | 30 | 0.5 | 76.0 | 86.0 | 67.2 | 78.7 | 34.5 | +5.4% |
| 30 | 2.0 | 76.5 | 86.0 | 67.9 | 79.8 | 36.0 | +7.0% |  |
| 30 | 8.8 | 77.3 | 86.6 | 68.6 | 80.4 | 38.6 | +9.3% |  |

### 4.3 Analysis

Here, we conduct an analysis to examine key aspects of SynthCLIP. We focus on the importance of textual and visual data modalities, ablate pipeline components (data filtering technique and LLM used for captions generation), and quantify the effects of $\mathcal{C}$ on performance. For all tests, we train on 3 million samples, *i.e*., a similar scale to CC3M, due to the high computational cost of the larger experiments.

Do synthetic captions or synthetic images matter more? SynthCLIP uses entirely synthetic text-image pairs. A key question arises: which has a greater impact on the model’s performance in downstream tasks – synthetic images or synthetic captions? In Table 3(a), we compare the standard CLIP model trained on CC3M, SynthCLIP, and two hybrid CLIP variants. One hybrid uses real captions with synthetic images (CLIP + Text-to-Image), generated using Stable Diffusion v1.5, while the other pairs real images with synthetic captions (CLIP + Captioning), created with the LLaVA () model. Note that these, requiring one real modality, are less scalable than SynthCLIP.

We measure that CLIP + Captioning significantly outperforms standard CLIP in several benchmarks, indicating the effectiveness of synthetic captions in CLIP training. For instance, this approach improves linear probing by 6.5% and text retrieval by 19.5%, though it slightly decreases zero-shot performance by 2.5%. On the other hand, CLIP + Text-to-Image shows less marked improvements and no gains in few-shot performance. This suggests that keeping images real and recaptioning them is more advantageous than generating images for real captions, possibly due to domain shifts and content generation mismatches in synthetic images as noted in Gani et al., (); Wu et al., ().

Hence, we introduce SynthCLIP+Captioning as an extra baseline. Given that TTI models could miss details in text prompts, recaptioning generated images can be beneficial. This is evident in Figure 3(b), where recaptioning corrects alignment issues from the image generation process (*e.g*. the missing bench in the generated image). Comparing SynthCLIP and SynthCLIP+Captioning in Table 3(a) (rows 4 and 5) shows significant gains with captioning, such as a 9.6% improvement in image retrieval. These results advocate for a potential boost in performance from future developments in prompt faithfulness of the TTI model. Moreover, it also opens doors to combinations of caption enrichment techniques such as VeCLIP (Lai et al., , ) and CapsFusion ().

Data Filtering Ablation
In creating our SynthCI-$X$ datasets in Section 4.2, we utilized balanced sampling to select a desired number of captions from a larger set of generated ones. In this section we want to assess how different data sampling strategies affect SynthCLIP’s performance. We focus on the impact of substituting balanced sampling with a more straightforward random sampling approach. For this, we randomly choose a subset of $3\times 10^{6}$ captions from $\mathcal{T}$. The corresponding images for these randomly selected captions are generated using Stable Diffusion v1.5, following the same procedure presented in Section 4.1. We then train SynthCLIP on this newly formed dataset. The results, presented in Table 1(e), indicate a noticeable decline in performance across various tasks with random sampling, especially in retrieval tasks. Here, we observe a drop of 2.7% in both image and text retrieval compared to balanced sampling. These results underline the critical role of balanced concept distribution for CLIP training, highlighting an advantage of synthetic data for data curation.

**(e) Balanced Sampling vs Random Sampling**
| Network | Lin. Prob. | Few-shot | Img Ret. | Text Ret. | IN 0-shot |
| --- | --- | --- | --- | --- | --- |
| SynthCLIP | 63.7 | 73.8 | 33.9 | 46.0 | 9.5 |
| w/ rand. sampling | 61.5 (-2.2) | 72.0 (-1.8) | 31.2 (-2.7) | 43.3 (-2.7) | 9.4 (-0.1) |

Evaluating Different Language Models for Caption Generation
In Table 1(f), we study the effect of changing the language model from Mistral V0.2 7B model to Vicuna 33B. We find that using Mistral V0.2 7B consistently achieves better performance when compared to Vicuna 33B. This might be attributed to Mistral’s superior performance on instruction-following benchmarks such as AlpacaEval (). Indeed, we phrase caption generation as an instruction-following task as previously described in Section 3.1.
This suggests that with increasingly performing models in instruction following, it will be possible to further improve performances of SynthCLIP training.

Concept Bank Impact
We explore how the concept bank size $\mathcal{C}$ and the type of concepts it contains affect the downstream performance of the model. For this, we create two subsets of $\mathcal{C}$. The first subset, $\mathcal{C}_{\text{CC3M}}$, is derived by identifying the concepts that appear in CC3M captions, by performing substring matching with concepts included in $\mathcal{C}$. This results in $40\times 10^{3}$ CC3M-related concepts. The second, $\mathcal{C}_{\text{rand}}$, is formed by randomly selecting the same number of concepts as in $\mathcal{C}_{\text{CC3M}}$ from $\mathcal{C}$.

We generate 3M images for each of $\mathcal{C}_{\text{CC3M}}$ and $\mathcal{C}_{\text{rand}}$ and train SynthCLIP on the generated datasets. We show results in Table 3. Interestingly, we noticed that focusing on CC3M-specific concepts ($\mathcal{C}_{\text{CC3M}}$) enhances performance compared to training with the full $\mathcal{C}$. For example, using $\mathcal{C}_{\text{CC3M}}$ yields a 3.9% improvement in text retrieval and 1.6% in linear probing. We hypothesize that this might be because $\mathcal{C}_{\text{CC3M}}$’s concepts are more aligned with concepts appearing in the downstream tasks, hence indicating a potential distribution bias in CC3M towards concepts prevalent in downstream task images. In contrast, using $\mathcal{C}_{\text{rand}}$ leads to lower performance in all tasks compared to the full $\mathcal{C}$. For example, we observe a 1.2% decrease in text retrieval and 0.8% in linear probing, likely because $\mathcal{C}_{\text{rand}}$’s concepts are less relevant to the downstream tasks. Hence, when specific insights about downstream tasks are unavailable, it is preferable to train on the widest possible range of concepts.

## 5 Discussion

Here, we discuss additional consequential properties of SynthCLIP, resulting from the usage of synthetic data, and we draw conclusions.

Mitigation of long-tail effects. While the distribution shift is preventing matching performance on the same data scale, we argue that the control over $\mathcal{C}$ may be used to mitigate long-tail effects. We provide preliminary insights by evaluating zero-shot classification accuracy on 10 classes (listed in appendix) included in $\mathcal{C}$, but undetected in CC12M with substring matching. Importantly, we compare both SynthCLIP and CLIP trained on $8.8\times 10^{6}$ images. For evaluation, we collect 150 samples per class. We report 44.18/60.04 accuracy for CLIP/SynthCLIP. This gives preliminary insights into how synthetic data could be used for training CLIP models resistant to long-tail distributions.

Data safety
Another aspect resulting from the control of $\mathcal{C}$ is the potential for training SynthCLIP models exclusively on safe data. This is not always possible with web-collected data (Schuhmann et al., , ), due to image-based NSFW detectors having limited performance (Schramowski et al., , ), and for the subjective nature of offensive content. We argue that our concept-based caption generation may help the synthesis of safe images only. In $\mathcal{C}$, we did not modify the concepts proposed by Xu et al., () for reproducibility. However, as a preliminary insight, we process $\mathcal{C}$ with an LLM (more details in appendix), detecting 3.15% NSFW concepts that can be filtered from the original $\mathcal{C}$. Moreover, our data synthesis pipeline could be associated with recent approaches for safe image generation (Schramowski et al., , ). This opens possibilities for safe CLIP trainings.

Concluding remarks We presented SynthCLIP, a CLIP model trained exclusively on synthetic data. Our experiments show SynthCLIP’s scalability and capability to match the performance of models trained on real data, given enough samples, on a large concept corpus. This paves new ways for entirely synthetic training at scale, possibly extending the capabilities of CLIP. Our investigation of the properties of SynthCLIP reveals novel insights into the role of synthetic data in vision-language models.

## Appendix A Comparison with Concurrent Works

We now discuss our positioning with respect to recent concurrent works.

- •
Concepts. In Fan et al., (), they propose a study on large-scale training of synthetic data for supervised learning and CLIP paradigms. They restrict the concept bank to ImageNet classes and provide several techniques to prompt text-to-image models. We refer the reader to section 3.1 of Fan et al., () for more details on the different proposed prompting techniques and TTI generation pipeline. In the supervised setting, it is clear from Table-A7 of their supplementary material that the performance of training on synthetic data becomes comparable when there are 8 times more synthetic data available at classifier free configuration scale of 2. Beyond that, the performance of training on synthetic data starts to saturate. Additionally, in that work, the authors assume some class-priors and restrict their training to ImageNet classes, which has been shown in Hammoud et al., () not to be generalizable to classes that are significantly different than those in ImageNet. Additionally, as mentioned in Section 5, training on a larger concept bank may lead to a better performance on long-tail concepts and hence the importance of covering many concepts during training beyond ImageNet classes. In Tian et al., (), the generation is also conditioned on the downstream classes. We refer the reader to Table-11 of their supplementary material. This gives you an unfair advantage over real data that is not necessarily conditioned on these classes (Section 4.3). For StableRep, they require the existence of real captions which they utilize as prompts for TTI model which is not scalable and have a limited effectiveness compared to synthetic higher quality captions (Section 4.3).
- •
CLIP-specific tasks. The main difference in our work is that we focus on CLIP-related tasks in a holistic manner. Indeed, Fan et al., () only proposes zero-shot evaluation, with no reasoning about how different tasks are impacted. Tian et al., () only proposes very limited experiments with linear probing on ImageNet. Instead, we focus on 5 different tasks, and contextualize separately the effects of synthetic data on different tasks.
- •
Data generation pipeline. Fan et al., () uses multiple strategies for captions generation, that do not necessarily work with large concept banks. We also provide an experiment on this in B.3. Tian et al., () generation pipeline is the most similar to ours, but relies on in-context examples generated by GPT-4. In our preliminary experiments, we verify that caption generation with in-context learning on Mistral 7B causes the model to collapse (Section B), generating variations of the in-context samples by just replacing $c$. Hence, our designed prompt is more adaptable to different models, as we show in Table 1(f) of the main paper.

## Appendix B Additional ablation studies

### B.1 Concept Appearance

In this section, we examine the presence of concepts from our extensive $500\times 10^{3}$ concept bank within real text-image datasets like CC3M and CC12M, as well as our SynthCI synthetic datasets. Our method involves substring matching, where we identify and count the occurrences of each concept within the captions of these datasets. This count reveals how frequently different concepts appear, particularly those occurring more than a specified number of times ($k$).

Table 4 summarizes these findings. Notably, even the smallest SynthCI-3M dataset contains significantly more concepts than the larger real CC12M dataset, surpassing it by nearly 2.5 times in terms of concepts appearing at least once ($k=1$). This trend of broader concept coverage in SynthCI datasets persists even when increasing the threshold to $k=25$ or $k=50$. An intriguing aspect is the average number of samples per concept. The last column of Table 4 shows the average frequency of concept occurrences, considering only those appearing at least 25 times. While CC3M and CC12M, with fewer overall concepts, exhibit a higher average of samples per concept, our SynthCI datasets generally show lower averages. However, SynthCI-30M shows the same average as real datasets, particularly CC12M. This similarity in samples per concept at 30M scale could be a key factor in SynthCI-30M matching the performance of CC12M.

Figure: Table 4: Concept Appearance in Real vs. Synthetic Datasets. We compares the frequency of concept appearances in real datasets (CC3M, CC12M) and their synthetic counterparts. We report the number of concepts that appear at least $k$ times, along with the average appearances for concepts occurring at least $25$ times.

### B.2 Model size

In order to study the effect of model scaling, we train SynthCLIP on SynthCI-3M and SynthCI-10M using a ViT-S/16 backbone and provide comparisons to the used ViT-B/16 in the paper. We report average results in all tasks in Table 5. We notice that bigger backbones perform better, proving the quality of our generated data.

### B.3 LLM usage

We ablate the importance of the LLM by removing it completely from our generation pipeline and replacing that with CLIP templates for zero-shot classification (). The rest of our pipeline is unchanged. We generate 3 million samples and train a CLIP model on top of this newly generated set. We call this model NaiveCLIP. We evaluated vision-language tasks, in particular zero-shot classification (ZS), text retrieval (TR), and image retrieval (IR), as presented in Section 4.1 of the main paper. We report for NaiveCLIP/SynthCLIP 2.0/9.1 for ZS, 2.9/33.9 for IR, 5.1/46.0 for TR. From our results, it is evident how the presence of the LLM is necessary to grant the best performance for vision-language tasks.

### B.4 Prompt engineering

In this section we showcase alternative attempts to generate synthetic captions.

#### Attempt 1 - Generate Captions without Any Conditioning

In our first attempt, we tried to let LLM generate any topic without any conditioning. This was done using the prompt shown in Figure 5. Unfortunately, the captions were overly descriptive and hard for the text-to-image model to generate images for. Moreover, they were always focused on nature, resulting in low variability unsuitable for CLIP training. Examples of generated captions are:

- •
A sunlit garden: vibrant roses bloom against a brick wall, butterflies dance around, water droplets sparkle on leaves, soft focus, balanced composition.
- •
Sunset over tranquil lake: A solitary kayaker paddles through golden reflection, mountains in distance bathed in warm light. Focus on kayaker’s determined face, balanced composition. Soft toned, impressionistic brushstrokes.
- •
A sunlit garden: vibrant roses bloom against a weathered brick wall, butterflies dance around ripe strawberries on a red table, children play nearby, laughter echoes softly. Warmth radiates from every detail.

Figure: Figure 5: Attempt 1 - Captions Generation Prompt

#### Attempt 2 - Generate Captions Using a Topics Bank

Instead of having a concept bank that we generate synthetic captions for, our first attempt was to try having a broader list of topics, *i.e*. a topic bank, used for conditional generation. Particularly, we used the topics shown in Figure 6 and then used the prompt shown in Figure 7 to generate the captions.

Figure: Figure 6: Attempt 2 - Topics Bank

Figure: Figure 7: Attempt 2 - Captions Generation Prompt

The observed issue is that for each topic the LLM had some kind of favourable instance. For example for "Wild Animals", most generated captions were about leopards:

- •
In the desert, a leopard is dragging its kill.
- •
A leopard carries its prey through the arid desert landscape.
- •
The majestic snow leopard roams high within Himalayas mountain range territory.

This issue was not resolvable by adjusting the prompt or parameters of the LLM including the seed, temperature and top-p value. Interestingly, this signals that biases in concept-oriented generations in LLM are significant regardless the amount of data they are trained on. Since we were mostly interested in maximizing the variability of generated concepts, we opted for the concept-based generation pipeline presented in Section 3.1.

Attempt 3 - Using in-context learning Additionally we tried prompting the language model by providing in-context sample prompts to guide the generation. In our tested settings, this causes the model to repeat often the in-context prompt, while modifying the concept $c$. For example, if an in-context sample about $c=\text{{``cat''}}$ is provided as “A beautiful cat sitting near Eiffel tower” and the target concept $c=\text{{``dog''}}$, the generated caption is “A beautiful dog sitting near Eiffel tower”. Instead, our engineered prompt allows us to generate variable samples by only conditioning on $c$.

## Appendix C Details on discussion

Long-tail effects
In Section 5, we highlight the increased performance on long-tail distributions. The concepts we use for the evaluation are: “amber” “chicks” “crystal” “crystal_ball” “loincloth” “rose” “sandcastles” “sundress” “veggies” “x-ray”. Although we outperform CLIP considerably, we still report that 44.18 in zero-shot classification is remarkable for unseen classes. We attribute this behavior to the presence of similar concepts in CC12 (*e.g*. “dress”).

Safety
As anticipated in Section 5, we performed preliminary safety analysis on the concepts in $\mathcal{C}$. We use LLaMA-3 (Meta, , ) to process the input concepts with the following prompt:

This results in 3.15% of concepts being flagged. Upon manual inspection, the classification seems reliable. We report selected examples of filtered concepts (Warning: potentially offensive content): “pornography”, “naked”, “drunk”, “weed”, “escort”.

## Appendix D Training on mixed real data

While mixed training is beyond the scope of our work, we conduct preliminary experiments to understand whether SynthCI could be used jointly to real data for boosting SynthCLIP performance. To do that, we train on a Mixed setup, in which we combine CC12M with $11.2\times 10^{6}$ synthetic samples from SynthCI, totaling $20\times 10^{6}$ captioned images. Note that this allows to compare a SynthCLIP model trained with the same computational costs, trained on SynthCI-20M. We find that this indeed improves the performance over pre-training on SynthCI-30M and CC12M across all benchmarks as seen in Table 6. This means that our synthetic text-image generation pipeline could be used in conjunction with existing large-scale curated datasets to achieve the best performance, in agreement with the literature (Yuan et al., , ; He et al., , ).

**Table 6: Quantitative evaluation on mixed real-synthetic data. Mixed is obtained by mixing 8.8M real samples of CC12M and 11.2M synthetic samples from SynthCI-20M. Mixing real data with synthetic not only outperforms the best real checkpoint (CLIP on CC12M) and SynthCLIP on the same data scale, but it also outperforms our best SynthCLIP model.**
| Network | CC12M | SynthCI | Lin. Prob. | Few-shot | Img Ret. | Text Ret. | IN 0-shot |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| samples ($\times 10^{6})$ | samples ($\times 10^{6})$ |  |  |  |  |  |  |  |
| CLIP | 8.8 | - | 76.7 | 84.9 | 58.9 | 71.7 | 33.6 |  |
| SynthCLIP | - | 20 | 74.6 | 83.4 | 59.0 | 75.7 | 28.0 |  |
| - | 30 | 75.0 | 84.9 | 61.7 | 77.1 | 30.5 |  |  |
| Mixed | 8.8 | 11.2 | 78.7 | 87.0 | 64.7 | 79.6 | 39.1 |  |

**Table 7: Finetuning the OpenAI CLIP. The OpenAI CLIP text encoder is used in Stable Diffusion. To prove that our pipeline is not limited by its performance, we finetune it on SynthCI for a reduced number of steps. This is sufficient to improve performance on almost all tasks, proving that our data creation pipeline allows the creation of novel content.**
| Network | Lin. Prob. | Few-shot | Img Ret. | Text Ret. | IN 0-shot |  |
| --- | --- | --- | --- | --- | --- | --- |
| OpenAI CLIP | 85.7 | 92.4 | 75.6 | 88.1 | 68.6 |  |
| OpenAI CLIP + SynthCI finetuning | 85.8 | 92.6 | 78.1 | 89.8 | 66.7 |  |

## Appendix E Discussion on Stable Diffusion

Stable Diffusion v1.5 uses the OpenAI CLIP textual encoder for prompt encoding, and it is one of the most popular choices for TTI generation. One may argue that this would limit the possibilities of SynthCLIP to the OpenAI CLIP performance. First, let us highlight how only the textual encoder of CLIP is used in Stable Diffusion. CLIP is defined as a pair of encoders mapping text and images to a joint embedding (). While Stable Diffusion uses a pre-trained CLIP text encoder for prompt interpretation, the CLIP visual encoder is not used anywhere in the pipeline. Also, the usage of CLIP for textual encoding is a design choice in Stable Diffusion, but other popular models such as Imagen (Saharia et al., , ) rely on textual encoders like T5-XXL (Raffel et al., , ) trained on text only, hence training on the text-images real datasets of CLIP is not strictly necessary for the SynthCLIP pipeline.

To prove that we are not limited by the pretrained text encoder performance, we fine-tune the OpenAI CLIP (ViT-B/16), pre-trained on 400 million captioned images, on SynthCI. The finetuning is done on 1M samples, for a single optimization step, in order to avoid loss of performance due to catastrophic forgetting. We report results in Table 7, where we improve over the baseline in four tasks out of five, with $\Delta_{\text{MTL}}=$$+0.56\%$. Our improved performance suggests that our approach allows us to extend even the capabilities of the CLIP model used in the text-to-image pipeline, by adding more synthetic data to the training.

## Appendix F Limitations

Here, we discuss limitations. The most evident disadvantage of synthetic CLIP models is the computational effort required to generate the training dataset, which may lead to a high carbon impact. Our generation process currently takes approximately 6.5 days using a 48-A100-80GB GPU cluster, equivalent to 313 GPU days. However, recent advancements in text-to-image and large language models have not only enhanced generation quality but also accelerated inference speeds (Sauer et al., , ; Kwon et al., , ). With continual technological advancements, we anticipate a reduction in the time required for generation, leading to more efficient and scalable end-to-end approaches. Future research should include further experimentation with various language models, text-to-image generators, and caption generation prompts to identify optimal configurations for improvement. Moreover, there may be concerns related to copyright issues and memorization in text-to-image diffusion models (Carlini et al., , ). With the advancements in differential privacy for generative models (Cao et al., , ), these could be solved in the near future. Finally, we acknowledge that although we train our CLIP on synthetic data only, both the LLM and the TTI have been trained on real data. Nevertheless, it is well-known the capacity of generative networks to create new content not included in the dataset (Rombach et al., , ), factor also proved in our experiments.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - I suggest that the authors improve the introduction by explaining the motivations and use-cases of employing a purely synthetic dataset to train CLIP models.
soundness: 2
strengths: - The paper provides a dataset of 30 million captioned images from diverse set of concepts.
- As the concepts are fixed, one can gather a subset of them to train new models with less concerns about NSFW contents leaking into the training data compared to using real images. 
- The models trained with similar dataset scales using synthetic captioned images show similar performance on down-stream tasks.
summary: The paper explores the performance of CLIP-style models trained on purely synthetic image-caption pairs (called SynthCLIP) generated by modern text-to-image diffusion models and LLMs. It studies the scaling trends of such models and also provides a dataset of 30 million captioned images.
weaknesses: - The main weakness of the paper in my opinion is that the paper is not well-motivated. The introduction section does not provide convincing answers to the questions like "why should we use purely synthetic image-caption datasets? why not a hybrid approach? why is the problem significant?"

- Although controlling the concepts that are present in the dataset can be useful, there is no guarantee that the generated images for each concept are 1) faithful to the content and 2) do not contain NSFW content:
1) faithful to content: The described workflow only filters the captions, not the generated images. It is likely that the generated images contain noisy unrelated images. No workarounds in this regard has been proposed in the paper.

2) NSFW content: The latter may happen because the training of models like Stable diffusion have been on unfiltered datasets like LAION. It is possible that some NSFW contents appear with some concepts in these datasets frequently, resulting in generation of NSFW contents inadvertently. 

- Despite that the paper argues that one can use synthetic images from tail classes to augment the real datasets, I think it is not straightforward to do so. Although the idea seems sound, the Stable Diffusion (SD) model has been trained on real images that have the same long-tailed classes. Therefore, the performance of SD on these classes will not be satisfactory.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. I'm curious about the performance on ImageNet variants like ImageNetV2, ImageNet-A, ImageNet-R, and ObjectNet.

2. Will the performance of MLLM understanding tasks be enhanced by employing CLIP trained on SynthCI-30M?
soundness: 3
strengths: 1. The research detailed the pipeline for entirely synthetic creation of image-text pairs and introduced the dataset SynthCI-30M.

2. A comprehensive set of experiments was conducted to elucidate the efficiency and practical value of synthetic data, demonstrating its scalability and effectiveness.
summary: It's interesting to see the emerging trend of training CLIP models using synthetic data. This work introduces SynthCLIP, a CLIP model training on synthetic data comprising both synthetic captions and images. The paper not only proposed the pipeline for creating synthetic data but also release SynthCI-30M, a comprehensive dataset housing 30 million captioned images generated entirely synthetically. This work unveiled the potential of leveraging synthetic data to enhance CLIP model training.
weaknesses: 1. The zero-shot image classification results lack depth to gauge effectiveness comprehensively.

2. The experiments solely compared with CC3M and CC12M. How would results differ if a subset of LAION-400M is employed instead?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It isn’t quite clear how MTL is calculated. What is it averaging over?
2. In an effort to understand differences in synthetic distributions versus natural distributions, how does performance change when using a CC3M concept distribution sample equalized to the real CC3M similar to Table 4? Another experiment to get at some of this would be taking CC3M, for each image prompting the model for a single “concept” then creating a caption and image using the proposed pipeline.
3. To understand the differences in scaling, it would be helpful to know the coefficients of the error with respect to dataset size on a log scale. How do natural and synthetic data coefficients compare?
4. This experiment is less pertinent than the above, but with results on improving the long tail distribution there might be interesting robustness properties as a result of concept representation. How does the real versus natural data compare on effective robustness in a framework like that of [1]?

Overall, the work is well presented but would benefit from a coverage of the first three points, and less importantly the fourth, to round out its presentation. I’d be happy to raise my score if the above are addressed.

[1] Nguyen et al. 2023 "Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP"
soundness: 3
strengths: Each element of the synthetic data pipeline is soundly constructed, and follows community norms for composition. Furthermore, the work details a lot of the smaller design choices (e.g. LLM, prompt, and concept distribution) that contribute to the end to end system. The approach itself is noteworthy as it represents a full departure to synthetic data whereas most existing approaches require one of the modalities to be pre-existing.

Model benchmarks are extensive and representative of image-text model capabilities. They show both good diversity in dataset and task.

Some of the most interesting findings come from the paper's ablations. Figure 4a demonstrates the delta in performance that can be attributed to each different synthetic modality. In addition, it shows a potential failure mode of ignored generation commands and how they may be addressed. Additionally, the study on the concept bank is quite interesting providing support for the hypothesis that some of the difference in performance between natural and synthetic data is from the underlying conceptual distribution not a failure in quality. The experiments towards the mitigation of long-tail effects suggest an interesting direction for improving unseen or undersampled concepts in real world training.
summary: The work proposes a synthetic training protocol for CLIP models extending prior work by creating both generated captions and generated images. In the process, the method demonstrates superior performance to common small scale image-text such as CC12M by curating a dataset of 30M synthetic examples. To better understand how different elements of the pipeline affect performance, they also ablate different choices of language models, differences caused by synthetic data sources, and how concept distribution impacts performance.
weaknesses: One concern with this work is that there is not sufficient evidence that the method might scale. Certain CLIP pretraining augmentations, like M3AE for example, have been shown to work at small scales, but yield no major benefit at larger scales [1]. Understanding that training frontier CLIP models is prohibitively expensive due to batch size needs, it’s sensible that this data is not available but worth keeping in mind.

The most immediate notice is the difference in data efficiency between real and synthetically drawn samples. Combined with the above, for practical applications it is not quite clear when one would adopt this method as it strictly leads to longer training runs and real training data is abundant (LAION-5B [2] and DataComp [3]). The method would benefit from further analysis into what is causing the reduction in performance, though some beginning analysis is done with respect to the concept distribution the gap is still left largely unexplained. 

[1] Weers et al. 2023 "Masked Autoencoding Does Not Help Natural Language Supervision at Scale"
[2] Schuhmann et al. 2022 "LAION-5B: An open large-scale dataset for training next generation image-text models"
[3] Gadre et al. 2023 "DataComp: In search of the next generation of multimodal datasets"

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: - In lines 158-159 the authors state that the captioned are oriented for a single object. How would it effect the performance on tasks that require low-level details understanding? There are many works that try to train on detailed captions to incorporate such an understanding.

- TTI models are currently not good at generating text within images and in understanding relationships between objects. Wouldn't training on generated images result in a model with limited capabilities in such areas?

- Given figure 4, why would we need to generate images and not recaption ones that we can obtain easily by crawling the internet?
soundness: 2
strengths: + The paper is well written and easy to follow. 
+ Utilizing a purely synthetic dataset enables to control the data distribution and to collect data and any scale, without requiring human intervention.
+ Interesting and insightful ablation studies
summary: The paper focuses on training a CLIP-based model using solely synthetic image-text pairs and study the effects of doing so.
SynthCLIP relies on the understanding that curating real multimodal data at scale comes with a cost of quality and alignment between images and their description. To this end, the authors propose to harness the advancement of Text-To-Image models and LLMs to generate a 30M purely synthetic dataset of image-caption pairs. Such a process enables to easily control the distribution of the data and to generate datasets at any scale with no human in the loop.
The authors study the effects of training CLIP with their propose dataset in a various of benchmarks including both vision and vision language tasks and compare it to training with real data.
weaknesses: - Empirical demonstration of the motivation - the authors claim that in real datasets, increasing their scale comes with a cost of quality and the proposed approach mitigates this and enables collecting quality data in any scale. However, empirically proving this requires comparing the performance against much larger real datasets than CC12M. Outperforming a model trained on CC12M with a 30M dataset doesn't showcase the advantage in quality of the synthetic data. Moreover, the model trained with a10M synthetic data has a worse performance compared to the model trained with a similar amount of real data. Thus, I am afraid that the main claim of the paper was not empirically demonstrated.

- Novelty - The proposed framework is based on existing models and approaches. Unfortunately, constructing the concept bank, which could have been an interesting place for novelty, is taken from an existing work.

- The necessity of "human intervention" - real datasets mainly scraped from the internet and the caption is achieved from the alt-text in the html. Thus, curating large datasets is done automatically. Indeed, such datasets are often noisy and there are various methods for filtering them (for example, by a CLIP-score threshold) or bootstrapping [1]. These methods enable collecting huge scale dataset without requiring human intervention. While relying on alt-text for caption often lead to short and oversimplified captions, there are many works that tackle this by proposing automatic recaptioning [2,3]. Thus, one can utilize real images oriented dataset with high quality without human intervention.

- Fairness of comparison in the experimental section - The authors have stated that the training is done for a fixed number of epochs with a fixed batch size for datasets of different size. From my understanding, this leads to a different number of training iterations for datasets at a different scale, impairing the validity of the comparison.

[1] Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
[2] Improving clip training with language rewrites.
[3] FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions

Meta-review: This paper discusses using pure synthetic data for training CLIP models, and shows the performance, scaling property, and analysis compared with real datasets. The paper also introduces a new SynthCI-30M with captions on 30 million images. However, the paper still lacks enough evidence on several key points such as the quality of content of generation images, not enough large-scale experiments and ablations, etc. The motivation of the work also needs to be justified. Therefore, based on the reviews, I recommend rejection of the paper.

Decision: Reject