CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning

## Abstract

Abstract Pretraining robust vision or multimodal foundation models ( e.g . , CLIP) relies on large-scale datasets that may be noisy, potentially misaligned, and have long-tail distributions. Previous works have shown promising results in augmenting datasets by generating synthetic samples. However, they only support domain-specific ad hoc use cases ( e.g . , either image or text only, but not both), and are limited in data diversity due to a lack of fine-grained control over the synthesis process.
In this paper, we design a controllable image-text synthesis pipeline, CtrlSynth, for data-efficient and robust multimodal learning. The key idea is to decompose the visual semantics of an image into basic elements, apply user-specified control policies ( e.g . , remove, add, or replace operations), and recompose them to synthesize images or texts. The decompose and recompose feature in CtrlSynth allows users to control data synthesis in a fine-grained manner by defining customized control policies to manipulate the basic elements. CtrlSynth leverages the capabilities of pretrained foundation models such as large language models or diffusion models to reason and recompose basic elements such that synthetic samples are natural and composed in diverse ways. CtrlSynth is a closed-loop, training-free, and modular framework, making it easy to support different pretrained models. With extensive experiments on 31 datasets spanning different vision and vision-language tasks, we show that CtrlSynth substantially improves zero-shot classification, image-text retrieval, and compositional reasoning performance of CLIP models.

## 1 Introduction

High-quality large-scale datasets have driven the success of large foundational AI models (Radford et al., ; Rombach et al., ; Touvron et al., ). Collecting and annotating datasets at large-scale is challenging and costly. One solution is to crawl data from the web; however, web data is noisy Lai et al. (); Kang et al. (), has long-tail distributions (Udandarao et al., ), and often causes privacy or copyright issues (Schuhmann et al., ). Synthetic data presents a viable and complementary alternative to overcome these challenges, as it allows for precise control over data generation and customization to meet specific requirements. A large body of work has focused on improving the quality of synthetic data for image and text data, from the generation of high-quality images Dunlap et al. (); Islam et al. () to the improvement of synthetic captions Lai et al. (); Fan et al. (). While these works have shown that synthetic data successfully improves model performance for various vision or vision-language tasks, their synthetic pipeline is often ad hoc and tailored to specific purposes such as training better CLIP models or improving domain-specific vision models (*e.g*., DiffuseMix uses diffusion models to augment images and improves accuracy on image classification tasks Islam et al., ). These data synthesis works also lack explicit fine-grained control over the generated texts or images, which are important for tasks with long-tail distribution (*e.g*., augmenting tail class samples) or enforcing safety requirements (*e.g*., mitigating biased or sensitive content generation Schramowski et al., ).

In this work, we aim to systematically control the synthetic pipeline for generating image-text data while accommodating different use cases (*e.g*., improving long-tail task performance, enhancing compositional reasoning of CLIP models, etc.). Our intuition is that large foundation models are already pretrained on a wide range of data and contain general knowledge about concepts, objects, and their relationships. For example, text-to-image models (*e.g*., Rombach et al., ; Podell et al., ) can generate detailed high-quality images based on text instructions. Similarly, large language models (LLMs)  (*e.g*., OpenAI, ; Touvron et al., ) have strong instruction-following capabilities, which can be used to control the text data generation. CtrlSynth leverages these large pretrained models to build a modular and controllable synthetic data generation pipeline. CtrlSynth allows users to apply explicit control instructions to guide data generation for images and texts. Unlike previous data synthesis works that use image-captioning models to directly generate captions given an image (*e.g*., Li et al., ; Lai et al., ), CtrlSynth decomposes image-to-text generation process into two separate steps, providing more fine-grained control to users for synthesizing data. Figure 1 shows an overall architecture of the CtrlSynth pipeline.
For an input image, CtrlSynth first uses a pretrained vision model to extract key objects, attributes, and their relations as visual tags. It then uses a text controller to create text synthesis instructions and guide the LLM to use visual tags to generate high-quality text outputs. Similarly, we devise an image controller that steers how the text prompts (or caption) can be used to guide the diffusion model to generate a desired image. Users can also feed the generated synthetic images into the tagging model again, forming a closed-loop data pipeline. Then users can start with synthetic or original images and texts and further generate more image-text pairs. The text and image controllers are modular, allowing users to control any part of the text or image generation process.

Compared to previous works, CtrlSynth provides three main benefits: (1) Controllable synthesis: CtrlSynth allows users to define policies on the visual tags or texts; enabling granular control over text and image generation; (2) Closed-loop system: CtrlSynth requires no additional training and can synthesize text from images and vice-versa using existing pretrained models. This closed-loop design additionally provides automatic filtering and verification capabilities to discard undesirable synthetic samples without manual or heuristics-based rules. (3) Flexible and scalable: CtrlSynth is modular and allows users to change its components (*e.g*., pretrained models) easily. (4) Extensive empirical results We evaluate the effectiveness of CtrlSynth on different tasks (*e.g*., image classification, image-text retrieval, compositional reasoning, and long-tail tasks), covering 31 datasets for vision and vision-language domains. We observe that CtrlSynth generated data improves the accuracy by (a) 23.4% on retrieval tasks, (b) 5% on the SugarCrepe compositional reasoning benchmark, and (c) 16% $\sim$ 21% for long-tail vision tasks.

Figure: Figure 1: CtrlSynth: A modular, closed-loop, controllable data synthesis system. The oval nodes indicate that the pretrained models and rounded boxes represent text or image data. The text and image controllers are used to guide the data synthesis.
Refer to caption: x1.png

## 2 Related Work

Data-Efficient Vision-Language Representation Learning. Contrastive Language-Image Pretraining (CLIP) (Radford et al., ) has popularized visual representation learning from image-text pairs due to its strong zero-shot transfer capabilities. Many recent works have focused on improving the data efficiency of training CLIP models. SLIP (Mu et al., ) brings self-supervised learning into a multitask learning framework to improve CLIP performance. FLIP (Li et al., ) masks out image patches during CLIP training, improving training efficiency and zero-shot accuracy over baselines. CLIPA (Li et al., ; ) further improves over FLIP ideas and reduces the number of image text tokens by block and syntax masking for CLIP training and it significantly reduces the training costs of CLIP models. LiT (Zhai et al., ) freezes the image encoder in CLIP models and achieves strong zero-shot transfer for CLIP models using much fewer data samples. All these techniques focus on improving the training methods for CLIP models to enable better vision-language representations. CtrlSynth improves data augmentation for CLIP training by synthesizing diverse image text samples. Our method is orthogonal and could potentially benefit from these methods.

Image-text Data Augmentation. Much recent work aims to improve the caption quality of image-text pairs. For example, VeCLIP (Lai et al., ), LaCLIP (Fan et al., ), and ReCap (Li et al., ) leverage LLMs to synthesize new captions that are more informative and contain rich descriptions about the image. The key difference of CtrlSynth is that we provide more diverse and high-quality captions that outperform prior works (we will show in Table 5 and Table 6). This is because CtrlSynth breaks down the image semantics to allow more fine-grained control and recombination using LLM.
Another line of work uses text-to-image models like diffusion models to generate synthetic images and augment downstream vision tasks. ALIA (Dunlap et al., ) uses language to guide the image editing process and provides domain-specific diversity to augment the image samples. DiffuseMix (Islam et al., ) augments image samples using diffusion models to blend original and generated images. EDA (Trabucco et al., ) generates variations of real images using diffusion models to maintain the semantics while augmenting image samples. These semantic image augmentation methods provide strong performance improvements on various vision datasets. Our CtrlSynth instead unifies the image and text synthesis via a closed-loop pipeline, it provides more flexibility and diverse synthetic samples while allowing more fine-grained control over the sample generation process.

## 3 CtrlSynth

CtrlSynth leverages semantic knowledge and reasoning skills of pretrained foundation models (*e.g*., large language and diffusion models) to generate diverse synthetic data samples in a controlled manner. Specifically, CtrlSynth consists of three foundation models: (1) a vision tagging model, (2) a large language model, and (3) a text-to-image model; plus the two text and image controllers. For a given real (
1a1a1a
in Figure 1) or synthetic (
1c1c1c
) input image, a *vision tagging model* (
2a2a2a
) extracts visual tags (*e.g*., objects, attributes, and their relationships) (
1e1e1e
). These tags describe the image’s visual concepts and semantic contexts. The *text controller* (
3a3a3a
) takes the image tags and user-defined control policies as inputs and generates instructions for synthesizing new text. An example control policy is to edit the tags or optionally add the text (
1b1b1b
) associated with the image.
A *large language model* (
2b2b2b
) then follows the instructions and generates the synthetic text (
1d1d1d
). The *image controller* (
3b3b3b
) operates on the given input text
and applies user-defined image control policies to output instructions for image synthesis. An example policy is to specify the style for generating artistic, cinematic, or realistic images.
A *text-to-image model* (
2c2c2c
) takes an image synthesis instruction provided by the image controller as an input and produces a synthetic image as an output (
1c1c1c
).

### 3.1 Key Components

mdfexample1innertopmargin=0.2em,innerbottommargin=0.2em,innerleftmargin=0.2em,innerrightmargin=0.2em,roundcorner=2pt,outerlinewidth=0.1,linecolor=blue!50,hidealllines=true

Vision Tagging Model.
The goal of a vision tagging model (VTM) is to extract the basic visual elements (or tags) of an image, including all objects or entities, attributes (*e.g*., color, shape, and size), and visual relations (*e.g*., interaction between objects).

Figure: Objects and attributes: light candle, patterned rug, white coffee table, sectional sofa Relations: in front of, on top, covered with
Refer to caption: extracted/5929450/fig/sven-brandsma-GZ5cKOgeIB0-unsplash.jpg

An example of extracting visual tags from VTM is shown in Footnote 2. The tagging model can be either a multi-label image classifier (Mehta et al., ) that predicts diverse tags in the image, or a black box system (*e.g*. an API service) that takes the input image and outputs the tags.

VTM, as a key component in CtrlSynth, can be a combination of an advanced captioning model (Xiao et al., ) that generates comprehensive image descriptions and an LLM that extracts the visual tags from the captions to decompose the visual semantics of an image into a set of fine-grained visual concepts. Section A.4 includes more details about this hybrid VTM. These fine-grained visual concepts can be easily modified and recomposed to create new visual contexts. This decompose-recompose feature of vision tags provides a large control space for synthesizing diverse texts.

Existing caption rewriting works (*e.g*., VeCLIP Lai et al. ()) rely on a multimodal captioning model to generate captions that are short sentences containing visual concepts. Image captions can be very descriptive but often only cover the most salient object of the scene, they are coarse-grained in structure (whole sentence or paragraph), and are hard to modify.
Our key distinction is that VTM produces a comprehensive list of metadata information that describes the visual concepts in an image as completely as possible.

Language Model. Large language models (LLMs) have exhibited strong instruction-following capabilities. The goal of an LLM in CtrlSynth is to take an input textual instruction on how to generate a synthetic text that meets the requirements specified in the instruction. CtrlSynth employs the reasoning and composition capability of LLMs to recombine the visual image tags in the task instruction and compose new synthetic texts. The instruction for an LLM consists of three parts (Figure 3): (i) task template that specifies the details of the text synthesis task, (ii) task content that contains the actual visual tags (phrases) and an optional caption paired with the image, and (iii) task constraint that describes the style and formatting of the output text. Users can also apply custom policies over the instructions to guide the text synthesis process.

Figure: Figure 3: An example instruction for LLMs to synthesize texts.

Text-to-Image Model. Text-to-image models generate novel and diverse image samples based on different input text prompts. CtrlSynth applies an image controller to account for the user-specified control policies and accordingly, updates the input text instructions from the previous step (i.e., language model). These updated instructions are then fed to text-to-image models for generating the image as an output. In our experiments, we use StableDiffusion models for text-to-image generation.

Text and Image Controllers. The controller in CtrlSynth is a function that takes an input text and transforms it into a specific text instruction for the LLM or text-to-image model.

The text controller accepts the visual tags of an image and a user-defined policy along with an optional original text as input and produces instructions to control the generation of synthetic text. In CtrlSynth, we study three predefined policies: (a) editing (remove, add, or replace) visual tags, (b) constraining the semantic meaning of a given sentence, and (c) styling the output text. Editing visual tags allows fine-grained control of synthetic visual content, for example, one can remove unwanted objects or attributes so they do not appear in the generated text. Constraining the meaning of synthetic text is useful in generating high-quality captions because many web-crawled captions are noisy. Enforcing the styling of output texts such as outputting into structured formats (*e.g*., JSON) makes the texts easier to use in downstream tasks. In our experiments, we use 10 example text control policies for synthesizing image captions (see Section A.1 for details).

The image controller is similar to the text controller in functionality. It mainly steers image generation via specific prompting. We study two simple control policies to show the controllability and utility of CtrlSynth. The first one involves weighting particular tags in the input prompt (lower or increase individual weights for a given tag) so that the output image has a different focus on the objects or attributes. The second policy applies different styles (*e.g*., cinematic, realistic, or art) to the output images for generating diverse content. Note that the control policies are flexible and can be easily modified for diverse use cases. For example, one can integrate more complex policies such as layout-guided (Lian et al., ) or planning-based (Yang et al., ) image generation.

### 3.2 Image Text Synthesis in CtrlSynth

CtrlSynth is a modular and closed-loop system by design and supports diverse image and text synthesis configurations. In this section, we first introduce different synthesis paths in CtrlSynth and then describe how the closed-loop feature allows CtrlSynth to filter out low-quality samples.

Flexible and diverse synthesis paths. A data synthesis path ($SP$) starts and ends with a data node (rounded box in Figure 1). We define the following synthesis paths:

$SP(1)$: $1a\rightarrow 2a\rightarrow 1e\rightarrow 3a\rightarrow 2b\rightarrow 1d$. This path (Figure 4(a)) means CtrlSynth generates a new text that describes the original image. The synthetic text $1d$ may not align with the semantics in the original image since the LLM can create new combinations of the visual tags and add information that does not exist in the image. Such new information provides useful semantic augmentation over the original image while containing similar visual concepts.

Figure: (a) Synthesis path $SP(1)$
Refer to caption: x2.png

$SP(2)$: $1a\rightarrow 2a\rightarrow 1e\xrightarrow{1b}3a\rightarrow 2b\rightarrow 1d$. This path (Figure 4(b)) is similar to the previous path but a key difference is that it constrains the synthetic text to be faithful(^3^33Or the opposite depending on the user-specified policy) to an original text. We can consider it as using the VTM and LLM to synthesize an improved text over the original one. We will show later in Section 4.5 that text samples generated from this path outperform previous works (Lai et al., ; Fan et al., ) that rewrite noisy captions. We include the example prompts to reflect the control policies in Section A.1.

$SP(3)$: $1a\rightarrow 2a\rightarrow 1e\rightarrow 3a\rightarrow 2b\rightarrow 1d
\rightarrow 3b\rightarrow 2c\rightarrow 1c$. This path (Figure 4(c)) provides both synthetic text ($1d$) and image ($1c$) samples. $1c$ can be an effective image sample that augments the original image ($1a$) or can be paired with ($1d$) to augment the original image-text pair ($1a$ and $1b$).

$SP(4)$: $1b\rightarrow 3b\rightarrow 2c\rightarrow 1c$. This path (Figure 4(d)) bypasses the language model and the original text is directly fed to the image controller and then generates a synthetic image ($1c$). The image sample could be a strong augmentation sample to the original image if the original text has a comprehensive and high-quality description.

Note that CtrlSynth supports more synthesis paths that are not listed above. For example, one can start with original text and use LLM to add creative elements and generate synthetic text and further use it to generate an image, i.e. $1b\rightarrow 3a\rightarrow 2b\rightarrow 1d\rightarrow 3b\rightarrow 2c
\rightarrow 1c$. Another category of examples includes starting with synthetic texts or images and creating more synthetic samples.

Self-filtering for better synthetic data. Synthetic samples often suffer from degraded quality especially when running at large scale. Synthetic systems often rely on heuristics or rule-based filtering techniques to filter out bad-quality samples. Because CtrlSynth pipeline is closed-loop, it implicitly provides self-filtering functionality. To check the quality of the synthetic text, we can detect if the synthetic text ($1d$) contains the visual tags ($1e$), to filter out potentially misaligned or lower quality synthetic text samples, we define that at least some ratio $p_{f}$ of the visual tags exist. For the synthetic image, we run it through the VTM again and output the visual tags, then we do the same check against the starting node text ($1b$ or $1d$). Later in Figure 7(a), we will show that self-filtering improves the synthetic samples.

## 4 Experiments

### 4.1 Setup

Tasks and Datasets. We adopt the CLIP (Radford et al., ) model architecture for multimodal representation learning.
For pretraining CLIP models, we use two public image-text datasets: CC3M (Sharma et al., ) and CC12M (Changpinyo et al., ). To evaluate the representation quality of pretrained CLIP models, we measure the zero-shot performance on classification, retrieval, and compositional reasoning tasks. For image classification, we use 25 common vision datasets, including five ImageNet (Deng et al., ; Recht et al., ) variants and the tasks from the VTAB benchmark (Zhai et al., ). We list the detailed dataset information in Section A.2. We use COCO (Lin et al., ) and Flickr30k (Plummer et al., ) for image-to-text and text-to-image retrieval tasks and report the metrics in recall@1. SugarCrepe (Hsieh et al., ) is a recent benchmark that measures the compositional understanding of vision-language models, we report the zero-shot accuracy numbers. Additionally, to study the effects of CtrlSynth on long-tail tasks, we evaluate the task accuracy of Places-LT and ImageNet-LT datasets (Liu et al., ) by augmenting the tail classes with CtrlSynth synthetic data.

Training and Baselines. Note that CtrlSynth itself does not require any training. We conduct pretraining experiments on CLIP models to evaluate the quality of synthetic data. We use ViT-B/16 (Dosovitskiy et al., ) architecture for the CLIP vision backbone. For a fair comparison, we train all models for the same number of iterations on the original dataset (baseline) and the dataset mixed with CtrlSynth augmented samples.
We use CtrlSynth-cap to denote the original image and synthetic text pair $(1a,1d)$ from synthesis path $SP(1)$. CtrlSynth-img stands for the synthetic image and original text pair $(1b,1c)$ from synthesis path $SP(4)$. CtrlSynth-capimg means the synthetic image and text pair $(1d,1c)$ from synthesis path $SP(3)$. We define CtrlSynth-mix as taking one image-text pair from CtrlSynth-cap and another from CtrlSynth-capimg. We do not take CtrlSynth-img image-text pairs since we found the original texts are noisy and thus a substantial portion of synthetic images are bad quality. We refer CtrlSynth-mix as the default setting unless otherwise stated. We list detailed information in Section A.3.

CtrlSynth Models. For the VTM, we adopt a hybrid approach by default, we combine the tags from a captioning plus tag extraction pipeline and an advanced multi-label image classifier. We use a recent vision foundation model called Florence-large (Xiao et al., ) to generate detailed image descriptions and then extract the objects, attributes, and relations using the Qwen2-7B-Instruct (Yang et al., ) LLM.
Then we use an accurate image classifier, the huge variant of CatLIP (Mehta et al., ), to output multiple high-confidence objects and attributes. We show later in Section 4.5 that this hybrid VTM provides the best visual tags compared with using individual approach alone. For the LLM, we use Mistral-NeMo-instruct model (AI, ) by default due to its strong instruction-following capability. We choose the stable-diffusion-xl-base-1.0 (Podell et al., ) for the text-to-image model by default. We describe the detailed setup in Section A.4. In Section 4.5, we study different pretrained models for each of the three modules in CtrlSynth.

### 4.2 Main Results

**Table 1: Comparison of the zero-shot classification accuracy between the baseline and CtrlSynth. We report top-1 accuracy for 20 commonly used downstream vision datasets, including 12 tasks in the VTAB benchmark (Zhai et al., ) and 8 other ones.**
| Data $\backslash$ Model | CC3M | CC12M |  |  |
| --- | --- | --- | --- | --- |
| CLIP | CtrlSynth | CLIP | CtrlSynth |  |
| CIFAR-10 | 41.5 | 70.3 | 75.4 | 82.6 |
| CIFAR-100 | 14.1 | 34.5 | 47.5 | 53.4 |
| CLEVR Counts | 7.1 | 11.7 | 15.2 | 22.1 |
| CLEVR Distance | 16.1 | 19.8 | 18.6 | 18.0 |
| Caltech-101 | 43.8 | 68.0 | 76.5 | 76.2 |
| Country211 | 0.4 | 0.6 | 1.1 | 1.3 |
| DTD | 11.6 | 17.9 | 23.5 | 29.1 |
| EuroSAT | 12.5 | 15.1 | 25.4 | 27.2 |
| FGVC Aircraft | 1.3 | 0.8 | 0.7 | 1.8 |
| Food-101 | 9.5 | 23.1 | 53.4 | 61.0 |
| GTSRB | 4.6 | 9.7 | 14.5 | 19.1 |
| KITTI | 30.2 | 19.5 | 33.9 | 33.9 |
| Oxford Flowers | 10.8 | 24.8 | 34.5 | 38.9 |
| Oxford-IIIT Pet | 3.1 | 7.9 | 8.0 | 9.4 |
| PatchCamelyon | 50.0 | 48.6 | 52.7 | 50.4 |
| RESISC45 | 17.7 | 27.6 | 36.7 | 39.5 |
| STL-10 | 70.4 | 90.4 | 92.8 | 94.0 |
| SUN397 | 30.7 | 44.3 | 54.1 | 58.1 |
| SVHN | 12.2 | 6.8 | 10.6 | 14.0 |
| Stanford Cars | 0.6 | 0.6 | 2.3 | 2.0 |
| Average | 19.4 | 27.1 (+7.7) | 33.9 | 36.6 (+2.5) |

Image Classification Evaluation. We conduct the zero-shot evaluation for image classification tasks. Table 3 shows the results across 20 commonly used vision datasets and Table 3 shows the results of 6 ImageNet-related datasets. Notably, CtrlSynth outperforms the baseline consistently by 2.5% to 9.4% for the CLIP models trained on the CC3M and CC12M datasets. We observe that CtrlSynth significantly improves the zero-shot performance (by over 7.7%) by augmenting smaller datasets like CC3M, while the performance gains become smaller on larger datasets like CC12M.

Image-Text Retrieval Evaluation. We evaluate the zero-shot image-text retrieval performance for our CtrlSynth and baseline CLIP models and present the recall@1 results in Table 3. CtrlSynth substantially improves the text-to-image and image-to-text retrieval recall by up to 24% and 36% for the Flickr dataset, and overall improves recall by 23.4% on average for CC3M models. CtrlSynth also brings over 9% retrieval gains for CC12M models on average. The improvements show that data samples from CtrlSynth have better coverage of visual concepts.

Compositional Reasoning Results. A key strength in CtrlSynth is the inclusion of visual tags that contain objects, attributes and relations from an image. To understand how the fine-grained visual attributes and relations affect visual reasoning performance, we evaluate CtrlSynth and baseline on the SugarCrepe Hsieh et al. () benchmark which measures the compositional reasoning capability of vision language models. We present the results in Table 4. CtrlSynth improves the baseline CLIP compositional reasoning by a large margin (4.5% for CC3M and 3% for CC12M on average). Note that most of the improvements come from the attribute and relation forms in the Replace and Swap categories, for example, CtrlSynth on CC3M improves the Replace relation accuracy by 4.3% and Swap attribute by 14.8%, indicating CtrlSynth models are robust to the attribute and relation changes.

**Table 4: We evaluate the compositional reasoning accuracy on the SugarCrepe (Hsieh et al., ) benchmark.**
| Data | Model | Add | Replace | Swap | Average |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Attribute | Object | Attribute | Object | Relation | Attribute | Object |  |  |  |
| CC3M | CLIP | 69.2 | 71.0 | 69.3 | 80.3 | 55.2 | 52.6 | 50.6 | 64.0 |
| CtrlSynth | 66.2 | 71.0 | 73.1 | 82.8 | 59.5 | 67.4 | 59.6 | 68.5 (+4.5) |  |
| CC12M | CLIP | 70.7 | 77.8 | 78.7 | 88.4 | 66.7 | 61.7 | 62.0 | 72.3 |
| CtrlSynth | 71.7 | 78.7 | 82.6 | 88.3 | 69.3 | 72.7 | 63.7 | 75.3 (+3.0) |  |

Comparison with Prior Work. CtrlSynth pipeline is flexible and supports synthesizing data from different paths. Previous work like VeCLIP (Lai et al., ) and LaCLIP (Fan et al., ) synthesizing new texts for the images by improving the captions. Though it is impossible to have a completely fair comparison with them(^4^44Factors that prohibit apple-to-apple comparison include training software, variations of CC3M samples due to missing images, exact hardware set up, etc.), the synthetic texts from the synthesis path (2) in CtrlSynth provide similar effects. We present the results on CLIP ViT/B16 models trained on CC3M for the tasks reported in each work. Table 5 shows that CtrlSynth outperforms VeCLIP on most VTAB datasets and improves zero-shot accuracy by 4.8% on average. CtrlSynth also surpasses VeCLIP by 7.9% on the ImageNet 1K dataset. We observe a similar trend when comparing CtrlSynth with LaCLIP in Table 6. Specifically, CtrlSynth achieves an average of 3.4% better accuracy than LaCLIP on 15 common datasets and 2.3% better accuracy on ImageNet 1K.

**Table 5: Comparison of the zero-shot classification accuracy between VeCLIP (Vasu et al., ) and CtrlSynth for CLIP trained on the CC3M. We report top-1 accuracy (%) for the VTAB benchmark (Zhai et al., ) across 9 tasks (6 from natural and 3 from specialized sets). We highlight the best numbers in bold.**
| Model | Natural Sets | Specialized Sets | Average | ImageNet 1K |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Caltech101 | CIFAR100 | SVHN | DTD | OxPet | Flowers102 | EuroSAT | RESISC45 | Camelyon |  |  |  |
| CLIP | 39.50 | 9.83 | 20.89 | 7.42 | 7.44 | 10.40 | 11.94 | 7.93 | 50.65 | 18.45 | 5.46 |
| VeCLIP | 54.30 | 17.74 | 18.74 | 11.23 | 10.09 | 22.75 | 7.35 | 16.54 | 52.52 | 23.48 | 15.98 |
| CtrlSynth | 66.10 | 34.09 | 17.66 | 16.76 | 7.77 | 15.55 | 20.83 | 24.59 | 50.79 | 28.24 | 23.82 |

**Table 6: We report the zero-shot performance on ImageNet 1K and 15 common downstream datasets for both LaCLIP (Fan et al., ) and CtrlSynth for CLIP trained on CC3M. We highlight the best numbers in bold.**
| Model | Food-101 | CIFAR-10 | CIFAR-100 | SUN397 | Cars | Aircraft | DTD | Pets | Caltech-101 | Flowers | STL-10 | EuroSAT | RESISC45 | GTSRB | Country211 | Average | ImageNet |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP | 10.3 | 54.9 | 21.8 | 25.0 | 0.8 | 1.4 | 10.5 | 12.8 | 43.3 | 10.2 | 77.6 | 14.1 | 19.1 | 6.9 | 0.6 | 20.6 | 15.8 |
| LaCLIP | 14.2 | 57.1 | 27.5 | 35.1 | 1.6 | 1.6 | 16.6 | 15.6 | 52.7 | 14.7 | 86.2 | 15.0 | 24.3 | 6.4 | 1.0 | 24.6 | 21.5 |
| CtrlSynth | 17.8 | 69.5 | 34.1 | 44.9 | 0.7 | 1.2 | 16.8 | 7.8 | 66.1 | 15.5 | 88.3 | 20.8 | 24.6 | 10.9 | 0.7 | 28.0 | 23.8 |

### 4.3 Performance on Long-tail Tasks.

Real-world data often have long-tail distributions. Much recent research (Shi et al., ; Liu et al., ) has focused on developing new learning methods for long-tail recognition tasks. Data augmentation remains an important solution, especially when the tail classes only have a few samples. In this section, we evaluate the effectiveness of synthetic samples from CtrlSynth for long-tail tasks.

Setup. We conduct experiments on the ImageNet-LT (Liu et al., ) and Places-LT (Liu et al., ) datasets. ImageNet-LT is a subset of the original ImageNet-2012 (Deng et al., ) and contains 115.8K images from 1000 classes, with 5 to 1280 images per class. Places-LT is even more imbalanced and contains 62.5K images from 365 classes, with 5 to 4980 images per class. The test sets of both datasets are balanced. Following the same setup in (Liu et al., ), we report the overall accuracy as well as the accuracy across the head ($>$100 images), medium (20$\sim$100), and tail ($<$20) classes. We take the same baseline in (Shi et al., ) and fine-tune the classifier head of a pretrained CLIP model (ViT-B/16) for 10 epochs (or the same number of iterations for CtrlSynth). For CtrlSynth synthetic samples, we choose the synthetic path $SP(3)$ to generate synthetic images for the tail classes. We mix the CtrlSynth image samples with the original training set of each dataset. We describe more details in Section A.2.

Key Results. Table 7 shows that CtrlSynth improves the tail class accuracy by 21.3% on ImageNet-LT and by 16.2% on Places-LT. Synthetic samples from CtrlSynth also improve the overall and medium class accuracy by 3$\sim$6%, though slightly decrease the head class accuracy.

**Table 7: Long-tail accuracy on the ImagetNet-LT and Places-LT datasets for the baseline and CtrlSynth models.**
| Model | ImageNet-LT | Places-LT |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Overall | Tail | Medium | Head | Overall | Tail | Medium | Head |  |
| Baseline | 60.8 | 13.8 | 56.7 | 82.6 | 34.9 | 8.2 | 31.3 | 53.7 |
| CtrlSynth | 66.2 (+5.4) | 35.1 (+21.3) | 62.8 (+6.1) | 81.4 | 38.6 (+3.7) | 24.4 (+16.2) | 34.6 (+3.3) | 51.2 |

### 4.4 Analysis

mdf2innertopmargin=0.2em,innerbottommargin=0.2em,innerleftmargin=0.2em,innerrightmargin=0.2em,roundcorner=2pt,hidealllines=true

Figure: Figure 5: Data efficiency comparison between baseline and CtrlSynth for pretraining CLIP models on CC3M. We normalize the iterations by dividing the total iterations with checkpoint steps.
Refer to caption: x6.png

#### Data-Efficiency of CtrlSynth in Training CLIP.

To study the data efficiency of CtrlSynth samples, we plot the top1 zero-shot accuracy of the ImageNet validation set in Figure 5 for the baseline and CtrlSynth CLIP models trained on CC3M. CtrlSynth reaches the 20% accuracy with 40% fewer iterations than the baseline, indicating that using CtrlSynth samples is more data-efficient.

#### Statistics and visualization of CtrlSynth Samples.

In this section, we provide the statistics for the synthetic samples from CtrlSynth. Figure 6 shows examples of CtrlSynth images and texts compared with the original real samples. We observe that the text samples from CtrlSynth are usually longer and contain richer information about the image. On average, CtrlSynth texts have over 60 words while original captions contain 8 words. We plot the histogram of the number of words in Figure 8 at Section A.5.

Figure: (a) *
Refer to caption: extracted/5929450/fig/demo/1.jpg

#### Effects of Self-Filtering.

CtrlSynth provides off-the-shelf self-filtering to control the quality of synthetic samples. We study the effects of applying different filtering thresholds $p_{f}$ for the synthetic text and image. We set the same filtering thresholds for both synthetic text and image samples. Intuitively, a higher threshold filters out more synthetic samples thus providing better quality samples that align with original real samples. On the contrary, a lower threshold keeps relatively less aligned samples but encourages more diverse samples. Figure 7(a) plots the zero-shot accuracy numbers of CLIP model on ImageNet under different threshold settings, we show that thresholds 10%$\sim$30% provide similar accuracy numbers and setting the filtering threshold to 20% provides the best accuracy. Thresholds higher than 50% do not provide accuracy gains, likely because the aligned synthetic samples lack diversity and fail to augment the original samples.

Figure: (a) Accuracy versus filtering thresholds.
Refer to caption: x7.png

#### Mixing Ratios of Synthetic Samples.

To better understand how the synthetic image text samples improve CLIP model training, we study different ratios ($p_{r}$) of mixing CtrlSynth samples with original real ones. During CLIP training, we randomly sample the original sample with probability $0<p_{r}<1$ and our sample with $1-p_{r}$. Figure 7(b) shows that even adding a small portion ($<20$%) of CtrlSynth samples improves the zero-shot accuracy while mixing with 50% provides best accuracy gains. Further higher mixing ratios show diminishing improvements though still better than the baseline that uses all real data.

### 4.5 Ablation Study

In this section, we evaluate the effectiveness of visual tags, the impact of using different pretrained models in the CtrlSynth pipeline, and mixing and filtering effects for CtrlSynth samples. We use the same text and image control policy described in Section 3.2 for all settings. We experiment with CC3M dataset for CLIP pretraining and report the accuracy on the SugarCrepe benchmark, zero-shot accuracy of common downstream vision tasks (same tasks in Table 3), and top1 accuracy on the ImageNet 1k validation set.

**Table 8: Evaluation of using different models, visual tags, and synthetic samples in CtrlSynth. ’-’ denotes the same value from the last row (default setting).**
| Study | Model | Tags | Samples | Common Tasks | ImageNet-1K | SugarCrepe |
| --- | --- | --- | --- | --- | --- | --- |
| Models | Qwen2-7B, SDXL | - | - | 24.7 | 23.5 | 65.1 |
| Qwen2-7B, SD3M | - | - | 26.1 | 23.8 | 65.2 |  |
| Mistral-Nemo, SD3M | - | - | 26.6 | 25.1 | 68.1 |  |
| Tags | - | Obj | - | 26.4 | 24.7 | 64.3 |
| - | Obj+Attr | - | 26.2 | 24.8 | 65.4 |  |
| Samples | - | - | CtrlSynth-cap, SP(1) | 26.2 | 24.5 | 67.2 |
| - | - | CtrlSynth-img, SP(4) | 22.1 | 21.8 | 64.4 |  |
| - | - | CtrlSynth-capimg, SP(3) | 26.5 | 24.8 | 67.5 |  |
| CtrlSynth | Mistral-Nemo, SDXL | Obj+Attr+Rel | CtrlSynth-mix | 27.1 | 25.3 | 68.5 |

Different Pretrained Models. We choose an alternate LLM and a different text-to-image model to understand how different pretrained models affect the quality of synthetic samples. CtrlSynth pipeline is flexible so we can easily swap the pretrained LLM and text-to-image models. Specifically, we use Qwen2-7B (Yang et al., ) for the LLM and Stable Diffusion 3 Medium (Esser et al., ) (SD3M) for the text-to-image model. Comparing the first and last rows in Table 8, we find using a smaller LLM like Qwen2-7B degrades the task performance on all three tasks, indicating that using a strong LLM is key to synthesizing high quality texts. The accuracy boost (+3%) on SugarCrepe benchmark shows the LLM is effective in recombining the visual tags in a compositional way to form diverse synthetic texts.
We also point out that using a more recent diffusion model like SD3M provides similar task performance numbers, this is likely because SD3M has fewer (2B versus 3.5B) parameters compared to SDXL, limiting the image generation capability.

Effectiveness of Visual Tags. We study the effects of using different categories of visual tags, *i.e*., using only objects (Obj), objects plus attributes (Obj+Attr), and all categories including relations (Obj+Attr+Rel). In Table 8, comparing the second and last row, we show attributes marginally improve the CLIP performance on compositional reasoning but not much on zero-shot vision tasks. Importantly, visual relations improves the performance on all three tasks, and significantly improves compositional reasoning performance by over 4%.

CtrlSynth Samples from Different Synthesis Paths. CtrlSynth pipeline supports synthesizing images or texts from different paths, we evaluate their quality by measuring the downstream task accuracy of the CLIP models trained on them. The penultimate and last rows in Section 4.5 show all CtrlSynth samples provides performance gains on downstream tasks, except the CtrlSynth-img samples where they do not improve compositional reasoning performance. CtrlSynth-img samples have the least augmentation benefits and are likely due to the original real texts are noisy and thus the generated images are not of high quality. Notably, mixing with synthetic captions (CtrlSynth-cap, CtrlSynth-capimg, and CtrlSynth-mix) provides meaningful augmentation benefits, highlight the importance of using LLMs to recombine the visual tags.

## 5 Conclusion

Synthetic data emerges as a viable solution to address challenges in curating high-quality samples from noisy, misaligned, and long-tail web data. However, existing data synthesis pipelines are rigid and the generation process is hard to control and thus being tailored for ad hoc use cases.
We develop CtrlSynth, a new image-text synthesis pipeline that allows users to control the data generation in a fine-grained way. CtrlSynth decomposes the semantics of images and texts into basic elements and uses pretrained foundation models to recompose them based on specified control policies. This way, CtrlSynth provides flexible and diverse image-text samples. Synthetic samples from CtrlSynth improve the long-tail task performance by a large margin. They also significantly boost the zero-shot and compositional capability of CLIP models and enable data-efficient multimodal learning.

## Appendix A Appendix

### A.1 Control Policies

#### Text Prompt Templates.

We provide example control policies for text synthesis as predefined prompt templates, the first five templates do not include original text:

- 1.
"Create a detailed and high-quality caption using phrases that represent the entities or objects, their unique attributes, and the visual relationships in the scene depicted. Phrases: {phrases}."
- 2.
"Compose a rich and immersive caption by incorporating a set of phrases that illustrate the entities or objects, their defining attributes, and the interconnections presented within the image. Phrases: {phrases}."
- 3.
"Formulate an articulate and informative caption by using a series of phrases that outline the entities, their attributes, and their visual relationships depicted in an image. Phrases: {phrases}."
- 4.
"Using a set of phrases that highlight the entities, attributes, and their visual associations in an image, craft a detailed and expressive caption. Phrases: {phrases}."
- 5.
"Construct a comprehensive and expressive caption by integrating phrases that detail the entities, their features, and the spatial or thematic relationships in an image. Phrases: {phrases}."

The following five templates include the original text, which is useful for maintaining the original meaning:

- 1.
"Create a comprehensive caption that faithfully represents the objects, attributes, and their relationships contained within the provided sentence and phrases. Given sentence: {caption}. Given phrases: {phrases}. If the original caption specifies particular give phrases, maintain their integrity while using the phrases to enhance the description."
- 2.
"Write a faithful caption by integrating the given phrases with the original sentence. Given sentence: {caption}. Given phrases: {phrases}. Ensure any objects or specific nouns from the original caption are preserved while elaborating on the visual relationships and attributes provided in the phrases to create a more detailed depiction."
- 3.
"Provide a faithful and informative image caption using a given sentence and few phrases. Sentence: {caption}, phrases: {phrases}. Consider the initial sentence as a base for the overall context and ensure that specific objects or nouns such as numbers, car models, animals, etc., are preserved in the new caption. Integrate the given phrases, which describe entities, attributes, or visual relationships, to enrich and elaborate on the original meaning. Maintain fidelity to the original content while enhancing descriptive quality."
- 4.
"Make a detailed caption based on the given phrases and a given sentence. Given phrases: {phrases}. Given sentence: {caption}. The sentence serves as a foundation, while the phrases elaborate on elements depicted in the image, like objects, their characteristics, and interactions. Preserve any pivotal information concerning objects, attributes, and their relations present in the sentence."
- 5.
"Write a new faithful and high-quality caption based on the given phrases and a given sentence. The given sentence is the original caption and the phrases are entities or objects, attributes, and their visual relationships in an image. Given sentence: {caption}. Given phrases: {phrases}. If the sentence contains objects or nouns (e.g. digits, car models, planes, pets, animals, etc.), the new caption should be faithful and keep this information. Otherwise, use the phrases to create the new caption."

#### Image Prompt Templates.

We provide five image prompt templates:

- 1.
"real": "a real photo. {prompt}. 35mm photograph, film, bokeh, professional, 4k, highly detailed",
- 2.
"nocap": "a real photo showing {prompt}. highly detailed"
- 3.
"isometric": "isometric style {prompt} . vibrant, beautiful, crisp, detailed, ultra detailed, intricate"
- 4.
"enhance": "breathtaking {prompt}. award-winning, professional, highly detailed"
- 5.
"quality": "masterpiece, best quality, ultra detailed, {prompt}. intricate details"

### A.2 Datasets Details

#### Evaluation Datasets.

We list the vision datasets for evaluation in Table 9.

**Table 9: Details of evaluation datasets.**
| Dataset | Metric | Classes | Test Set Size |
| --- | --- | --- | --- |
| CIFAR-10 (Krizhevsky, ) | Accuracy | 10 | 10000 |
| CIFAR-100 (Krizhevsky, ) | Accuracy | 100 | 10000 |
| CLEVR Counts | Accuracy | 8 | 15000 |
| CLEVR Distance | Accuracy | 6 | 15000 |
| Caltech-101 (Fei-Fei et al., ) | Mean Per Class Recall | 102 | 6085 |
| Country211 (Radford et al., ) | Accuracy | 211 | 21100 |
| DTD (Cimpoi et al., ) | Accuracy | 47 | 1880 |
| EuroSAT (Helber et al., ) | Accuracy | 10 | 5400 |
| FGVC Aircraft (Maji et al., ) | Mean Per Class Recall | 100 | 3333 |
| Food-101 (Bossard et al., ) | Accuracy | 101 | 25250 |
| GTSRB  (Stallkamp et al., ) | Accuracy | 43 | 12630 |
| KITTI (Geiger et al., ) | Accuracy | 4 | 711 |
| Oxford Flowers-102 (Nilsback & Zisserman, ) | Mean Per Class Recall | 102 | 6149 |
| Oxford-IIIT Pet (Parkhi et al., ) | Mean Per Class Recall | 37 | 3669 |
| PatchCamelyon (Veeling et al., ) | Accuracy | 2 | 32768 |
| RESISC45 (Cheng et al., ) | Accuracy | 45 | 6300 |
| STL-10 (Coates et al., ) | Accuracy | 10 | 8000 |
| SUN397 (Xiao et al., ) | Accuracy | 397 | 108754 |
| SVHN (Netzer et al., ) | Accuracy | 10 | 26032 |
| Stanford Cars (Krause et al., ) | Accuracy | 196 | 8041 |
| ImageNet-1K (Deng et al., ) | Accuracy | 1000 | 50000 |
| ImageNet-V2 (Recht et al., ) | Accuracy | 1000 | 10000 |
| ImageNet-S (Wang et al., ) | Accuracy | 1000 | 50889 |
| ImageNet-A (Hendrycks et al., ) | Accuracy | 200 | 7500 |
| ImageNet-O (Hendrycks et al., ) | Accuracy | 200 | 2000 |
| ImageNet-R (Hendrycks et al., ) | Accuracy | 200 | 30000 |
| Flickr (Plummer et al., ) | Mean Recall@1 | - | 1000 |
| MSCOCO  (Lin et al., ) | Mean Recall@1 | - | 5000 |

#### Long-tail Datasets.

For the tail classes in ImageNet-LT and Places-LT, we generate synthetic images using the “real” style of image prompt template, and we generate 7 samples per tail class so that we roughly double the size of the original real datasets. We obtain 80.4k synthetic samples for ImageNet-LT and 55.2K for Places-LT.

### A.3 Training Details

**Table 10: Training hyper-parameters.**
| Hyperparameter | CC3M | CC12M |
| --- | --- | --- |
| Total iterations | 56,429 | 55,429 |
| Warmup iterations | 2822 | 2771 |
| Image size | 224 | 224 |
| LR scheduler | Cosine | Cosine |
| Max. LR | 0.002 | 0.002 |
| Min. LR | 0.00002 | 0.00002 |
| Optimizer | AdamW | AdamW |
| AdamW $\beta$’s | (0.9, 0.98) | (0.9, 0.98) |
| Weight decay | 0.2 | 0.2 |
| Batch size per GPU | 256 | 256 |
| # A100 GPUs | 8 | 32 |
| A100 GPU Memory | 40 GB | 40 GB |

#### Pretraining Hyper-parameters.

We pretrain the CLIP for the same number of iterations for both the baseline and CtrlSynth. For example, suppose we train for $E$ epochs, if the original dataset has $N$ samples, CtrlSynth has generated $N^{\prime}$ samples ($N^{\prime}<=N$ due to filtering), then the total samples are $E*N$, we train CtrlSynth models for $\frac{E*N}{N+N^{\prime}}$ epochs. This guarantees that the baseline and CtrlSynth CLIP models have seen the same number of data samples.

Table 10 lists the hyper-parameters used for pretraining on CC3M and CC12m. We use AdamW (Loshchilov & Hutter, ) with default $\beta$ values as an optimizer and binary cross-entropy loss as an objective function. We use cosine learning rate schedule (Loshchilov & Hutter, ). We use the CoreNet library (Mehta et al., ; ) for all pretraining experiments. We adapt the LIFT codebase (Shi et al., ) for fine-tuning long-tail tasks, main modifications include adding support for iteration-based training and data loader for multiple datasets.

### A.4 CtrlSynth Inference Details

#### VTM.

We use a hybrid tagging model consisting of two stages. We first run the ViT-Huge variant of CatLIP (Mehta et al., ) for each image and output top20 classes based on the sigmoid score of prediction logits, then we convert the class indices to actual word labels. The vocabulary size of CatLIP is 24320. Most of the vocabulary words are nouns and single-word attributes.
We then run the Florence-large (Xiao et al., ) for each image to extract detailed captions using the task prompt <MORE_DETAILED_CAPTION>. After that, we run Qwen2-7B-Instruct (Yang et al., ) to extract objects, attributes, and relations from the Florence captions. We then merge the objects field with CatLIP-predicted labels. The extraction instruction contains a 2-shot example and we list the prompt template below:

CatLIP is available in CoreNet so we use it directly for inference and we wrap the Florence Transformers (Wolf et al., ) code into the CoreNet inference pipeline for easier integration.

#### LLM.

We use the vLLM engine (Kwon et al., ) for offline inference in Qwen2 and Mistral-Nemo. We use greedy decoding for the generation.

#### Text-to-image Model.

We use the diffusers (von Platen et al., ) library for diffusion model inference. For both SDXL and SD3M models, we use float16 dtype with a guidance scale of 7.0 and set the diffusion steps to 28.

### A.5 More Analysis Details

#### CtrlSynth Samples.

For CC3M, the original dataset has 2.8 million image-caption pairs, CtrlSynth-cap contains 2.6 million captions, CtrlSynth-img contains 2.4 million images, and CtrlSynth-mix contains 5.1 million image-caption pairs. Original CC12M has 11.3 million image-caption samples, CtrlSynth-cap consists of 10.2 million captions, CtrlSynth-img contains 9.5 million images, and CtrlSynth-mix has 19.7 million image-caption pairs.

#### CtrlSynth Synthetic Texts.

We plot the number of words for synthetic texts generated by CtrlSynth and compare them with original real texts in Figure 8.

Figure: Figure 8: Number of words for the original captions and CtrlSynth synthetic texts on CC3M.
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See above.
soundness: 3
strengths: - The pipeline can be thought of as a way to distill knowledge from the language models and stable diffusion models to augment the dataset of CLIP. This is an interesting way to inject new information in synthetic data. 

- The results are good, demonstrating improvements over CLIP while maintaining the amount of data it sees since the fix the number of iterations and just change the proportions of real vs their synthetic data.
summary: The authors propose a controllable image-text generation pipeline that can augment data to improve CLIPs image retrieval, classification, and compositional performance. Specifically, they leverage strong vision models to tag images with objects and attributes, use the knowledge in language models to create new variations of the captions, and use diffusion models to generate images based on the new captions as prompts.
weaknesses: -  They say that the language model takes an instruction on how to generate a caption given the visual tags. They show some examples in Appendix A1. The instructions don't mention any editing, it mostly just says to describe the image better. In that case, do the gains come from some hallucination in the LLM caption that makes varied images?

- Have the authors tried any other variation of editing instructions? Is there any analysis on the kinds of image editing prompted by the text that improve performance more? Are there specific prompts that serve as better negatives when tuning the CLIP contrastive loss?

- There are other works that edit images based on text instructions like instruct pic to pic, magic brush etc. It might have been nice to see to see if editing certain things in images based on the LLM prompts is better than just using SD to generate since SD can often lack accuracy in generating the correct attribute object relation compositions. 

- Nit: There are several works that either generate synthetic images based on the dataset they want to target (https://arxiv.org/pdf/2406.05184), or for cross domain retrieval (https://arxiv.org/pdf/2401.00420). A discussion for comparison could be nice.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Refer to the Weakness.
soundness: 3
strengths: - The paper focuses on noise and misalignment in the large-scale image-text datasets, which is a critical challenge in multimodal learning.
- The paper introduces an innovative approach that emphasizes fine-grained control, utilizing generative models to decompose and refine images and texts at a detailed level. Notably, it is training-free and suited for integration with different pre-trained generative models.
- The experiments presented in the paper show that the proposed method improves downstream performances of multimodal models.
summary: The paper proposes CtrlSynth, a controllable image-text synthesis pipeline for data-efficient multimodal training. Addressing limitations in existing large-scale datasets that are often noisy and misaligned, CtrlSynth enables fine-grained control by decomposing images into basic elements and applying user-specified modifications to synthesize new data. This training-free and flexible pipeline can work with different models and supports closed-loop synthesis (image to text and vice versa). The proposed method also boosts the performance of multimodal model training.
weaknesses: The paper, while contributing valuable ideas, has several notable weaknesses that are significant and need to be addressed. 

### Methodological Weaknesses
- The proposed pipeline shares significant similarities with GenArtist[1] on image editing. The paper does not clearly demonstrate the differences between this work and GenArtist. It is important for the authors to specify these distinctions and highlight the novelty of their approach.  Additionally, a thorough comparison should be incorporated into the experimental section to strengthen the evaluation.
- While fine-grained control is presented as the main contribution, represented by the text and image controllers in the pipeline, the design is inadequate and lacks clarity. The design of the pipeline does not effectively demonstrate how the editing condition is provided to the generative model in a fine-grained manner. The text controller relies solely on prompt concatenation, making the mapping between visual tags and policies unclear and limiting precise control. Additionally, the paper does not address how to maintain image consistency after editing, which is essential for practical use. These shortcomings contribute to potential inconsistencies and an insufficient explanation of how fine-grained control is maintained. The image controller exists the same problem.

### Experimental Limitations
- The datasets used (CC3M and CC12M) are relatively small, with no experiments conducted on larger datasets such as LAION-400M or LAION-5B.
- The paper only tests a limited range of multimodal model structures, lacking experiments on models like BLIP and CLIP of different ViT models.
- The study does not address data-efficiency validation. Existing data-efficiency-focused works, such as SemDeDup[2], Filter-&-Align[3], and Sieve[4], refine or filter datasets for better performance. The paper should include comparisons with these approaches in terms of model performance and the amount of training data.

---
Reference

[1] Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing. arXiv.2407.05600.

[2] Amro Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli and Ari S. Morcos. SemDeDup: Data-efficient learning at web-scale through semantic deduplication. arXiv.2303.09540.

[3] Lei Zhang, Fangxun Shu, Tianyang Liu, Sucheng Ren, Hao Jiang, and Cihang Xie. Filter & Align: Leveraging Human Knowledge to Curate Image-Text Data. arXiv.2312.06726.

[4] Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, and Ari Morcos. Sieve: Multimodal Dataset Pruning Using Image Captioning Models. arXiv.2310.02110.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Can the authors provide details on the overall efficiency of the proposed pipeline? For example, how long does it take to generate 1 million images along with their captions? It would also be good to know the time cost at each component, e.g. vision tagging, caption generation, image generation. A more complete picture of the efficiency in the pipeline would better help to assess the value of this work.
soundness: 3
strengths: 1. The idea is clear and effective: by combining multiple expert models, we can obtain fine-grained image tags, captions, and synthetic images, which together help to create a high-quality synthetic dataset.

2. The modularized pipeline is flexible, as each model can be replaced without affecting the performance of the other components.

3. Experiments are comprehensive. Compared to the baseline CLIP, the improvements from CtrlSynth are evident.
summary: This paper introduces a multimodal data synthesis pipeline called CtrlSynth. Specifically, CtrlSynth includes a vision tagging model to extract key objects, attributes, and relations from an image, which can then optionally be combined with the original text for a language model to generate new image descriptions. Finally, the newly generated image caption is input into a text-to-image model to generate an image. The authors have demonstrated the effectiveness of their pipeline by comparing it with CLIP pretraining data. Overall, the enhanced dataset appears to be superior to the original one.
weaknesses: 1. Practical concerns: By using several models, such as the vision tagging model, LLM, and diffusion model, the proposed method might not be efficient for scaling up to larger datasets, particularly considering the time cost associated with image synthesis.

2. The assumption behind CtrlSynth is based on a fixed number of data samples, where the method helps a model achieve better performance than training on the original dataset. However, given the recent trends in LLM and multimodal LLM research, where pretraining data continues to scale up, the proposed method may not be scalable for very large datasets. While this is a challenge, under the current setting in the paper, CtrlSynth is indeed effective.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - Introduces a new controllable synthesis pipeline (CtrlSynth) that allows fine-grained data manipulation, enabling user-defined policies for image-text synthesis.
- Achieving significant performance improvements across diverse tasks such as zero-shot classification, retrieval, and long-tail recognition is inspiring.
- Clearly explains the methodology with diagrams and examples, easy to understand the synthesis process and its components.
summary: This paper introduces CtrlSynth, a controllable pipeline for generating synthetic image-text data to improve multimodal models. By allowing fine-grained control over data synthesis, CtrlSynth decomposes and recomposes visual semantics using pretrained models, enhancing diversity and alignment of generated samples.
weaknesses: 1. In the ablation experiments, it was observed that the performance improvement brought by CtrlSynth-img alone is minimal. Would it be possible to completely remove the generation of synthetic images and focus resources on improving the diversity and quality of synthetic text? Would this lead to consistent performance improvements across all tasks?

2. The paper mentions that CtrlSynth uses a self-filtering mechanism to improve the quality of synthetic data, but it lacks detailed explanations about the implementation, such as how the alignment threshold for visual tags is selected.

3. The paper does not explain in depth how CtrlSynth fundamentally differs from other caption augmentation methods like VeCLIP and LaCLIP. It is necessary to provide a clearer comparison, clarifying whether the increased diversity brought by the decomposition of visual tags and user control strategies is more important, or whether it is the generation of more fine-grained semantic captions that matters.

4. The experiments may be limited to a few selected models (e.g., Mistral-Nemo and Qwen2-7B). Would using larger LLMs lead to better results?  

5. A drawback of this method is that the data generation pipeline involves multiple different models and is not end-to-end in training, requiring substantial resources and time for building the synthetic data in the early stages.

Meta-review: The submission proposes CtrlSynth, a controllable pipeline to generate synthetic images for representation learning. The broad workings are as follows: 1) Given a real image and optional associated caption, image tags are generated using a vision model; 2) Using an LLM, a new caption can be generated from the original caption and tags, remixed using user instructions; 3) Using the new caption, a text-to-image model is used to generate a synthetic image; 4) This synthetic image can be used to train other vision models or fed back into this pipeline to generate more images.
The authors compare the quality of CLIP models trained on two datasets - 1) A dataset of real images, and 2) A dataset containing a mix of real and synthetic images; and show that CtrlSynth helps improve accuracy on multiple tasks.

The submission received ratings of 3, 6, 5, 6. Some key weaknesses highlighted by the reviewers include:
1) Lack of demonstrations on whether the described image editing workflow is helpful
2) Missing details in the submission including on self-filtering
3) Lack of demonstration showing that synthetic data helps when a larger dataset of real images is available.

The AC would like to note that prior work (which has not been cited) has demonstrated similar results, and reduced the novelty of the current submission:
1) StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners (NeurIPS 2023): They investigate the accuracy of visual representations from simCLR and CLIP trained on synthetic images generated by text-to-image models.
2) Synthetic Data from Diffusion Models Improves ImageNet Classification (TMLR 2023): "augmenting the ImageNet training set with samples from a generative diffusion model can yield substantial improvements in ImageNet classification accuracy ..."

Following the discussion with reviewers, reviewer L3Nm (who gave a rating of 6) was not opposed to rejection given the missing experiments in the submission.

Taking everything into account, especially prior work showing the efficacy of synthetic images, the only major contribution of this submission is the real image -> caption -> modified caption -> synthetic image pipeline, which does not meet the bar for acceptance. The ACs thus recommend rejection.

Decision: Reject