CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning
Authors: Qingqing Cao
Mahyar Najibi
Sachin Mehta
Venue: ICML 2025 poster (2025)
Keywords: clip
synthetic data
multimodal learning
longtail
Primary area: deep_learning->everything_else

Abstract: Pretraining robust vision or multimodal foundation models (e.g., CLIP) relies on large-scale datasets that may be noisy, potentially misaligned, and have long-tail distributions. Previous works have shown promising results in augmenting datasets by generating synthetic samples. However, they only support domain-specific ad hoc use cases (e.g., either image or text only, but not both), and are limited in data diversity due to a lack of fine-grained control over the synthesis process. In this paper, we design a controllable image-text synthesis pipeline, CtrlSynth, for data-efficient and robust multimodal learning. The key idea is to decompose the visual semantics of an image into basic elements, apply user-specified control policies (e.g., remove, add, or replace operations), and recompose them to synthesize images or texts. The decompose and recompose feature in CtrlSynth allows users to control data synthesis in a fine-grained manner by defining customized control policies to manipulate the basic elements. CtrlSynth leverages the capabilities of pretrained foundation models such as large language models or diffusion models to reason and recompose basic elements such that synthetic samples are natural and composed in diverse ways. CtrlSynth is a closed-loop, training-free, and modular framework, making it easy to support different pretrained models. With extensive experiments on 31 datasets spanning different vision and vision-language tasks, we show that CtrlSynth substantially improves zero-shot classification, image-text retrieval, and compositional reasoning performance of CLIP models.

CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning
Qingqing Cao
Mahyar Najibi?
2 Sachin Mehta
Abstract
1. Introduction
High-quality large-scale datasets have driven the success of large foundational AI models (Radford et al., 2021; Rom- bach et al., 2022; Touvron et al., 2023). Collecting and annotating datasets at large-scale is challenging and costly. One solution is to crawl data from the web; however, web data is noisy (Lai et al., 2024; Kang et al., 2023), has long- tail distributions (Udandarao et al., 2024), and often causes privacy or copyright issues (Schuhmann et al., 2022). Syn- thetic data presents viable and complementary alternative to overcome these challenges, as it allows for precise con- trol over data generation and customization to meet specific requirements. A large body of work has focused on improv- ing the quality of synthetic data for image and text data, from the generation of high-quality images (Dunlap et al., 2023; Islam al., 2024) the improvement of synthetic captions (Lai al., 2024; Fan al., 2023). While these works have shown that synthetic data successfully improves model performance for various vision or vision-language tasks, their synthetic pipeline is often ad hoc and tailored to specific purposes such as training better CLIP models or improving camain-specific vision models (e.g.. DiffuseMix uses diffusion models to augment images and improves accu- racy on image classification tasks Islam et al., 2024). These data synthesis works also lack explicit fine-grained control over the generated texts images, which are important for tasks with long tail distribution (e.g., augmenting tail class samples) or enforcing safety requirements (e.g., mitigating biased or sensitive content generation Schramowski et al., 2023).
Pretraining robust vision or multimodal founda- tion models (e.g., CLIP) relies on large-scale datasets that may be noisy, potentially misaligned, and have long-tail distributions. Previous works have shown promising results in augmenting datasets by generating synthetic samples. How- ever, they only support domain-specitn use cases (e.g., either image or text only, but not both), and are limited in data diversity due to lack of fine-grained control over the synthe- sis process. In this paper, we design a control- lable image-text synthesis pipeline, CtrlSynth, for data-efficient and robust multimodal learn- ing. The key idea is to decompose the visual semantics of an image into basic elements, apply user-specified control policies (e.g.. remove, add, or replace operations), and recompose them to synthesize images or texts. The decompose and recompose feature in CtrlSynth allows users control data synthesis in fine-gra fine-grained manner by defining customized control policies to manipu- late the basic elements. CtrlSynth leverages the capabilities of pretrained foundation models such as large language models or diffusion models to reason and recompose basic elements such that synthetic samples are natural and composed in diverse ways. CtrlSynth is closed-loop. training- free, and modular framework, making it easy support different pretrained models. With exten- sive experiments on 31 datasets spanning different vision and vision- languae tasks, we show that CtrlSynth substantially improves zero-shot classi- fication, image-text retrieval, and compositional reasoning performance of CLIP models.
ad hoc
In this work, we aim systematically control the synthetic pipeline for generating image-text data while accommodat- ing different use cases (e.g., improving long-tail task perfor- mance, enhancing compositional reasoning of CLIP models, etc.). Our intuition is that large foundation models are al- ready pretrained on a wide range of data and contain general knowledge about concepts, objects, and their relationships. For example, text-to-image models (e.g., Rombach al., 2022; Podell et al., 2024) can generate detailed high-quality images based on text instructions. Similarly, large language models (LLMs) (e.g., OpenAl, 2022; Touvron et al., 2023) have strong instruction-following capabilities, which can be
'Apple Work done at Apple. Correspondence to: Qingqing Cao <qicao@ warlo@@apppccooc apple.com>. com>>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CtrlSynth
e.g.
Original Image
large window, patterned ooffee gray curtain, front of, with
Original Text
Creane detaled and high-quality caption using phrases large window, front Dirocly output capioo add formatingg
add (optional)
e.g.
Vision Tagging Model
Tags Objects, Attributos, Relations
edit
Text Controller
textual
Language Model
instruction
Text to Image Model
@b
in the moder livig room, 0.g. largo window the anchored by whito with gray rug.
t.g.
Synthetic Image
textual Instruction
Image Controller
upwelght styling
Synthetic Text
[ying room tagether.
Figure I: CtrlSynth: A modular, closed-loop, controllable data synthesis system. The oval nodes indicate that the pretrained models and rounded boxes represent text or image data. The text and image controllers are used to guide the data synthesis.
used to control the text data generation. CtrlSynth leverages these large pretrained models to build a modular and control- lable synthetic data generation pipeline. CtrlSynth allows users to apply explicit control instructions to guide data gen- eration for images and texts. Unlike previous data synthesis works that use image-captioning models to directly generate captions given an image (e.g., Li et al., 2024; Lai et al., 2024), CtrlSynth decomposes image-to-text generation pro- cess into two separate steps, providing more fine-grained control to users for synthesizing data. Figure shows an overall architecture of the CirlSynth pipeline. For an in- put image, CtrlSynth first uses a pretrained vision model to extract key objects, attributes, and their relations as visual tags. It then uses text controller to create text synthesis instructions and guide the LLM to use visual tags to gener- ate high-quality text outputs. Similarly, we devise an image controller that steers how the text prompts (or caption) can be used to guide the diffusion model to generate a desired image. Users can also feed the generated synthetic images into the tagging model again, forming closed-loop data pipeline. Then users can start with synthetic original images and texts and further generate more image-text pairs. The text and image controllers are modular, allowing users to control any part of the text or image generation process.
image-text retrieval, compositional reasoning, and long-tat loig-tail tasks), covering 31 datasets for vision and vision-language domains. We observe that CtrlSynth generated data im- proves the accuracy by (a) 23.4% on retrieval tasks, (b) 5% on the SugarCrepe compositional reasoning benchmark, and (c) 16% 21% for long-tail vision tasks.
2. Related Work
Data-Efficient Vision-Language Representation Learn- ing. Contrastive Language- Image Pretraining (CLIP) (Rad- ford et al., 2021) has popularized visual representation learn- ing from image-text pairs due to its strong zero-shot transfer capabilities. Many recent works have focused on improv- ing the data efficiency of training CLIP models. SLIP (Mu et al., 2022) brings self-supervised learning into a multitask learning framework to improve CLIP performance. FLIP (Li et al., 2023c) masks out image patches during CLIP training, improving training efficiency and zero-shot accuracy over baselines. CLIPA (Li et al., 2023b;a) further improves over FLIP ideas and reduces the number of image text tokens by block and syntax masking for CLIP training and it signifi- cantly reduces the training costs of CLIP models. LiT (Zhai et al., 2022) freezes the image encoder in CLIP models and achieves strong zero-shot transfer for CLIP models using much fewer data samples. All these techniques focus on improving the training methods for CLIP models to enable better vision-language representations. CtrlSynth improves data augmentation for CLIP training by synthesizing diverse image text samples. Our method is orthogonal and could potentially benefit from these methods.
Compared to previous works, CtrlSynth provides three main benefits: (1) Controllable synthesis: CtrlSynth allows users to define policies the visual tags or texts; enabling granu- lar control over text and image generation; (2) Closed-loop system: CtrlSynth requires no additional training and can synthesize text from images and vice-versa using existing pretrained models. This closed-loop design additionally provides automatic filtering and verification capabilities to discard undesirable synthetic samples without manual or heuristics-based rules. (3) Flexible and scalable: CtrlSynth is modular and allows users to change its components (e.g., pretrained models) easily. We evaluate the effectiveness of CtrlSynth on different tasks (e.g., image classification,
Image-text Data Augmentation. Much recent work aims to improve the caption quality of image-text pairs. For example. VeCLIP (Lai al., 2024), LaCLIP (Fan al., 2023), and ReCap (Li et al., 2024) leverage LLMs to syn- thesize new captions that are more informative and contain rich descriptions about the image. The key difference of
2

---

CtrlSynth
CtrlSynth is that we provide more diverse and high-quality captions that outperform prior works (we will show in Ta- ble 10 and Table II). This is because CtrlSynth breaks down the image semantics to allow more fine-grat tee-grained control and recombination using L.M. Other related work includes Syn- thCLIP (Hammoud et al., 2024) and LatteCLIP (Cao et al., 2024) that use synthetic texts and images to improve CLIP model performance, while our synthetic pipeline also sup- ports diverse models and improves long-tail tasks.
skills of pretrained foundation models (e.g., large language and diffusion models) to generate diverse synthetic data samples in controlled manner. Specifically, CtrlSynth con- sists of three foundation models: (1) a vision tagging model, (2) a large language model, and (3) a text-to-image model; plus the two text and image controllers. For a given real (1a in Figure 1) or synthetic (1e) input image, a vision tugging model ( (2x) extracts visual tags (e.g., objects, attributes, and their relationships) (1e). These tags describe the image's s
visual concepts and semantic contexts. The text controller ( 3a) takes the image tags and user-defined control policies as inputs and generates instructions for synthesizing new text. An example control policy is to edit the tags or optionally add the text ( (ib) associated with the image. A large lan- guage model (2b) then follows the instructions and generates the synthetic text (id). The image controller (3b) operates the given input text and applies user-defined image con- trol policies to output instructions for image synthesis. An example policy is to specify the style for generating artistic, cinematic, or realistic images. A rext-to-image model ( (2c) takes an image synthesis instruction provided by the image controller as an input and produces synthetic image as an output (1c)
Another line of work uses text-to-image models like dif- fusion models to generate synthetic images and augment downstream vision tasks. Azizi et al. (2023) shows that synthetic data from diffusion models can effectively im- prove ImageNet classification performance. ALIA (Dunlap et al., 2023) uses language to guide the image editing pro- cess and provides domain-speeifi diversity to augment the image samples. DiffuseMix (Islam et al., 2024) augments image samples using diffusion models to blend original and generated images. EDA (Trabucco et al., 2023) generates variations of real images using diffusion models to main- tain the semantics while augmenting image samples. These semantic image augmentation methods provide strong per- formance improvements on various vision datasets. Our CtrlSynth instead unifies the image and text synthesis via closed-loop pipeline, it offers more flexibility and diverse synthetic samples while allowing more fine-grained control over the sample generation process. StableRep (Tian et al., 2023) uses synthetic images from diffusion models to im- prove vision- languae representations but their performance on compositionality and zero-shot tasks! lag behind models trained with CtrlSynth samples.
3.1. Key Components
Prior image editing works like InstructPix2Pix (Brooks et al., 2023), GenArtist (Wang et al., 2024), and Mag- icBrush (Zhang et al., 2023) provide methods and datasets to enable precise control over image generation. While the image synthesis path in our pipeline could benefit from these works, we focus on allowing diverse data synthesis. It is an open research question to automatically generate the image editing instructions for each sample in a dataset. Our pipeline can also be combined with previous work (Mishra et al., 2024) to improve the performance of cross-domain retrieval tasks or when the target task has little real data to retrieve (Geng et al., 2024).
Objects and attributes: light candle, patterned rug, white coffee table, sectional sofa
Relations: in front of, on top, covered with
Figure 2: Visual tags of an example image?.
Vision Tagging Model. The goal of vision tagging model (VTM) is to extract the basic visual elements (or tags) of an image, including all objects or entities, attributes (e.g., color, shape, and size), and visual relations (e.g., inter- action between objects).
3. CCrlSyyth
CtrlSynth leverages semantic knowledge and reasoning
'e.g., the CL.IP from StableRep achieved 40.2% zero-shot accu- racy ImageNet while ours 41.2%. StableRep causes an accu- racy drop on compositional benchmarks like ARO (Yuksekgonul et al., 2022) while our CtrlSynth improves compositional accuracy on the harder and more recent benchmark SugarCrepe (Hsieh et al.,
An example of extracting visual tags from VTM is shown in Figure 2. The tagging model can be either multi- image classifier (Mehta et al., 2024b) that predicts diverse
2023).
IImage credit:
hetps hctps://unsplash con/ photos/
Light canidl pmope roundruhh tarcol Feee table- and sect lonal --sofa www.aaaaaaaoo00 675:600g 188
3

---

CtrlSynth
tags in the image, a black box system (e.g. an API service) that takes the input image and outputs the tags.
of synthetic text. In CtrlSynth, we study three predefined policies: (a) editing (remove, add, or replace) visual tags, (b) constraining the semantic meaning of given sentence, and (c) styling the output text. Editing visual tags allows fine-grained control of synthetic visual content, for example, one can remove unwanted objects or attributes so they do not appear in the generated text. Constraining the meaning of synthetic text useful in generating high-quality captions because many web-crawled captions are noisy. Enforcing the styling of output texts such as outputting into struc- tured formats (e.g., JSON) makes the texts easier to use in downstream tasks. In our experiments, we use 10 example text control policies for synthesizing image captions (see Appendix A.1 for details).
VTM, as key component in CtrlSynth, can be a combina- tion of an advanced captioning model (Xiao et al., 2024) that generates comprehensive image descriptions and an L.LM that extracts the visual tags from the captions to decompose the visual semantics of an image into a set of fine- ane-ggained visual concepts. Appendix A.4 includes more details about this hybrid VTM. These fine-grained visual concepts can be easily modified and recomposed to create new visual contexts. This cocomposs-rrcompore feature of vision tags provides a large control space for synthesizing diverse texts.
Existing caption rewriting works (e.g., VeCLIP (Lai et al., 2024)) rely on a multimodal captioning model to generate captions that are short sentences containing visual concepts. Image captions can be very descriptive but often only cover the most salient object of the scene, they are coarse- grained in structure (whole sentence or paragraph), and are hard to modify. Our key distinction is that VTM produces a comprehensive list of metadata information that describes the visual concepts in an image as completely as possible,
The image controller is similar to the text controller in func- tionality. It mainly steers image generation via specific prompting. We study two simple control policies to show the controllability and utility of CtrlSynth. The first one in- volves weighting particular tags in the input prompt (lower or increase individual weights for a given tag) so that the out- put image has different focus on the objects or attributes. The second policy applies different styles (e.g., cinematic, realistic, or art) to the output images for generating diverse content. Note that the control policies are flexible and can be easily modified for diverse use cases. For example, one can integrate more complex policies such as layout-guided (Lian et al., 2023) or planning- based (Yang et al., 2024b) image generation.
Language Model. Large language models (LLMs) have capabilities. The goal
exhibited strong instruction-followw of an LLM in CtrlSynth is to take an input textual instruction on how to generate a synthetic text that meets the require- ments specified in the instruction. CtrlSynth employs the reasoning and composition capability of LLMs to recombine the visual image tags in the task instruction and compose new synthetic texts. The instruction for an LLM consists of three parts (Figure 3): (i) task template that specifies the details of the text synthesis task, (ii) task content that contains the actual visual tags (phrases) and an optional caption paired with the image, and (iii) task constraint that describes the style and formatting of the output text. Users can also apply custom policies over the instructions to guide the text synthesis process.
3.2. Image Text Synthesis in CtrlSynth
CtrlSynth is a modular and closed-loop system by design and supports diverse image and text synthesis configurations. In this section, we first introduce different synthesis paths in CtrlSynth and then describe how the closed-loop feature allows CtrlSynth to filter out low-quality samples.
Flexible and diverse synthesis paths. A data synthesis path (SP) starts and ends with data node (rounded box in Figure 1). We define the following synthesis paths:
Text-to-Inee -mnage Model. Text-to-image models generate novel and diverse image samples based on different input text prompts. CtrlSynth applies an image controller to ac- count for the user-specified control policies and accordingly, updates the input text instructions from the previous step (i.e., language model). These updated instructions are then fed to text-to-image models for generating the image as an output. In our experiments, we use StableDiffusion models for text-to-image wtweiiaaagggeeeeaeee generation.
SP(L): 1a + 2a + 1e - 3a - 2b - 1d. This path (Figure 4a) means CtrlSynth generates new text that de- scribes the original image. The synthetic text 1d may not align with the semantics in the original image since the LLM can create new combinations of the visual tags and add information that does not exist in the image. Such new information provides useful semantic augmentation over the original image while containing similar visual concepts.
Text and Image Controllers. The controller in CtrlSynth a function that takes an input text and transforms it into a specific text instruction for the L.LM or text-to-image model.
SP(2): 1a + 2a 1e (Figure 4b) is similar to the previous path but a key differ- ence is that it constrains the synthetic text to be faithful
1b 3a
2b 1d. This path
The text controller accepts the visual tags of an image and user-defined policy along with an optional original text as input and produces instructions to control the generation
2Or the opposite depending on the user-specified policy
4

---

CtrlSynth
Write a faithful caption by integrating the given phruses with the original sentence. Ensure any objects from the original caption are preserved while elaborating on the visual relationships and attributes provided in the phrases to create a more detailed depiction. Given sentence: [caption). Given phrases: [phrases]. The caption should not contain any NSFW words. It should be grammatically correct. It should be concise, but not too short. Directly oulput the caption and do not add any formatting.
Figure 3: An example instruction for LLMs to synthesize texts.
Original
Original
Original
Original Text
Imaaee
Text
Image
losforet
Vision Tagging Madill
Taggling Madiol
Taga Ocarrs, AirbuAes felritre
Text Controller
bentad Insenctioo
Language Model
Tagg Oeaers aie. heirter
senual Insexction
Language Model
Controller
Synthelic Image
Image Model
Teetual
Synthelic Text
Synthelic Image
Toxt to Image Model
Synthelic
Controller
Controller
(a) Synthesis path S1
(b) Synthesis path S
Original
Original Text
Original
Original
Image
Image
Copiioaat
Vision Taggling Model
Taga Oteers
ICB
Text Controller
Language
Vision Tagging Model
Tags chietss
Tixtt Controller
Language
Model
Synthetic
Text to Model
tentual
Image Contruller
Synthelic Text
Synthefic
Text to Model
Ieduel Instruction
Image Controller
Synthetic
Text
(c) Synthesis path SP)
(d) Synthesis path SP(4)
Figure 4: Different synthesis paths in CtrlSynth.
to an original text. We can consider it as using the VTM and LLM to synthesize an improved text over the original one. We will show later in Section 4.4 that text samples generated from this path outperform previous works (Lai et al., 2024; Fan et al., 2023) that rewrite noisy captions. We include the example prompts to reflect the control policies in Appendix A.I.
and creating more synthetic samples. Although these paths are realizable through our pipeline, their study falls beyond the scope of this paper and we leave it to the future works.
Self-fitteing for better synthetic data. Synthetic samples often suffer from degraded quality especially when running at large scale. Synthetic systems often rely on heuristics or rule-based filtering techniques to filter out bad-quality samples. Because CtrlSynth pipeline is closed-loop. it im- plicitly provides self- leff-filtering functionality. To check the quality of the synthetic text, we detect if the synthetic text (1d) contains the visual tags (1e), to filter out potentially misaligned or lower quality synthetic text samples, we de- fine that at least some ratio py of the visual tags exist, For the synthetic image, we run it through the VTM again and output the visual tags, then we do the same check against the starting node text (1b or 1d). Later in Appendix A.5, we will show that self-filtering improves the synthetic samples.
SP(3): 1a + 2a + le + 3a + 2b + 1d + 3b + 2c + 1c. This path (Figure 4c) provides both synthetic text (1d) and image (lc) samples. Ic can be an effective image sample that augments the original image (1a) can be paired with (1d) to augment the original image-text pair (Ia and 1b).
SP(4): 1b + 3b- + 2c + 1c. This path (Figure 4d) by- passes the language model and the original text is directly fed to the image controller and then generates synthetic image (1c). The image sample could be strong augmenta- tion sample to the original image if the original text has comprehensive and high-quality description.
4. Experiments
Note that CtrlSynth supports more synthesis paths that are not listed above. For example, one can start with original text and use LLM to add creative elements and generate synthetic text and further use it to generate an image, i.e. 1b 3a + 2b- 1d 3b 2c Ic. Another category of examples includes starting with synthetic texts or images
4.1. Setup
Tasks and Datasets. We adopt the CLIP (Radford et al., 2021) model architecture for multimodal representation learning. For pretraining CLIP models, we use two pub- lic image-text datasets: CC3M (Sharma et al., 2018) and
5

---

CtrlSynth
CC12M (Changpinyo al., 2021), and Datacompl B (Gadre al., 2023). To evaluate the representation quality of pre- trained CLIP models, measure the zero-shot performance on classification, retrieval, and compositional reasoning tasks. For image classification, we use 25 common vision datasets, including five ImageNet (Deng et al., 2009; Recht et al., 2019) variants and the tasks from the VTAB bench- mark (Zhai et al., 2020). We list the detailed dataset in- formation in Appendix A.2. We use COCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015) for image-to- text and text-to-image retrieval tasks and report the metrics in recall @1 1. SugarCrepe (Hsich et al., 2023) is a recent benchmark that measures the compositional understanding of vision-language models, we report the zero-shot accuracy numbers. Additionally, to study the effects of CtrlSynth on long-tail tasks, we evaluate the task accuracy of Places-LT and ImageNet-LT datasets (Liu et al., 2019) by augmenting the tail classes with CtrlSynth synthetic data.
et al., 2024) for the text-to-image model by default. We describe the detailed setup in Appendix A.4. In Section 4.4, we study different pretrained models for each of the three modules in CtrlSynth.
4.2. Main Results
Table 1:
Comparison of the zero-shot classification accuracy between the baseline and CurSSyth. We report top-1 accuracy for 20 commonly used downstream vision datasets, including 12 tasks in the VTAB benchmark (Zhai et al., 2020) and 8 other ones.
CC3M
CC12M
Data Model
CLIP
ChriSynih
CLIP
CirlSyuth
CIFAR-10
70.3
75.4
82.6
CIFAR-100
14.1
34.5
47.5
53.4
CLEVR Counts
7.1
11.7
15.2
22.1
CLEVR Distance
16.1
19.8
18.6
18.0
Caltech-101
43.8
68.0
76.5
76.2
Country211
0.4
0.6
DTD
11.6
17.9
23.5
29.1
EumSAT
15.1
25.4
27.2
Training and Baselines. Note that CtrlSynth itself does not require any training. We conduct pretraining experiments on CLIP models to evaluate the quality of synthetic data. We use ViT-B/16 (Dosovitskiy et al., 2020) as the default vision backbone for the CLIP and study different backbones in Table 9 at Appendix A.5. For fair comparison, we train all models for the same number of iterations on the origi- nal dataset (baseline) and the dataset mixed with CtrlSynth augmented samples. We use CurSSynth-cap to denote the original image and synthetic text pair (Ia, Id) from syn- thesis path S1))). CtrlSynth-img stands for the synthetic image and original text pair (1b, 1e) from synthesis path SP4).. CtrlSynth-capimg means the synthetic image and text pair (Id, le) from synthesis path SP(3). We define CurlSynth-mi as taking one image-text pair from CtrlSynth- cap and another from CrriSynth-caping. We do not take CtrlSynth-ing image-text pairs since we found the original texts are noisy and thus substantial portion of synthetic images are bad quality. We use CtrlSynth-mix as the default setting. We list detailed information in Appendix A.3.
FGVC Aircraft
1.3
0.8
0.7
188
Food-101
9.5
23.1
53.4
61.0
GTSRB
4.6
9.7
14.5
19.1
KITTI
30.2
19.5
33.9
33.9
Oxford Flowers
Oxford-IIT Pet
3.1
7.9
8.0
9.4
PatchCamelyon
50.0
48.6
$2.7
50.4
RESISC4S
17.7
27.6
36.7
39.5
STL-10
70.4
90.4
92.8
94.0
SUN397
30.7
44.3
54.1
58.1
SVHN
12.2
6.8
10.6
14.0
Stanford Cars
0.6
Average
19.4
27.1 (+7.7)
33.9
36.6 (+2.5)
Table 2: Zero-shot top-1 accuracy between the baseline and Ctrl- Synth on ImageNet datasets.
\ Model
CC3M
CC12M
CLIIP
CirlSynth
CLIPP
CirlSynth
ImageNet-IKK IK
20.2
25.3
39.6
41.2
ImageNot-V2
11.0
20.7
34.0
35.5
ImageNet-S
3.5
12.4
28.3
33.8
ImageNet-A A
3.0
6.5
12.0
14.9
ImageNet-O
18.6
30.7
44.2
45.9
ImageNet.R R
11.6
28.4
47.6
55.1
Average
11.3
20.7
(49.4)
34.3
37.7 (+3.4)
CtrlSynth Models. For the VTM, we adopt hybrid ap- proach by default, we combine the tags from a captioning plus tag extraction pipeline and an advanced multi-labe image classifier. We use a recent vision foundation model called Florence-large (Xiao et al., 2024) to generate detailed image descriptions and then extract the objects, attributes, and relations using the Bwve2-7B-Instruue 2024a) L.M. Then we use an accurate image classifier, the huge variant of CatLIP (Mehta et al., 2024b), to output multiple high-confidence objects and attributes. We show later in Section 4.4 that this hybrid VTM provides the best visual tags compared with using individual approach alone. For the LLM, we use Mistral-NeMo-instee 2024) by default due to its strong instruction-followii ca- pability. We choose the stable-diffusion diiffusionnxllbaaaeeede xl-base-1100 (Podell
Table 3: Zero-shot retrieval evaluation on the Flickr and COCO datasets. We report the recall I numbers. 12T means image-to- text retrieval, and T21 denotes text- to text-to-image retrieval.
CCIM
CC12M
Data
Model
CLIPP
ChrlSynth
CLIIP
ChrSymth
(Yang et al.,
COCO 12T
10.9
32.3
40.5
49.8
COCO T21
7.6
19.8
26.7
32.2
Flickr 12T
21.3
$7.3
65.5
77.2
Flickr T21
14.8
39.0
48.9
58.2
Average
I 13.7
37.1
(+23.4)
45.4
54.4 (+9.0)
model (AI,
Image Classification Evaluation. We conduct the zero- shot evaluation for image classification tasks. Table shows the results across 20 commonly used vision datasets and
6

---

CtrlSynth
Table 2 shows the results of 6 ImageNet-related datasets. Notably, CtrlSynth outperforms the baseline consistently by 2.5% to 9.4% for the CLIP models trained on the CC3M and CC12M datasets. We observe that CtrlSynth significantly improves the zero-shot performance (by over 7.7%) by aug- menting smaller datasets like CC3M, while the performance gains become smaller on larger datasets like CC12M.
Table 4: We evaluate the compositional reasoning accuracy on the SugarCrepe (Hsich et al., 2023) benchmark.
Operation
Type
CC3M
CC12M
CtrlSymth
CLIP/ CiriSynih
Attribute
69.2 66.2
70.7/71.7
ADD
Object 71.0/110
77.7
Attribute
69.3/73.1
78.7
REPLACE
Object 80.3/82.8 82.8
88.4 38,3
Relation
55.2/ 59.5
66.7 69.3
Previous work like VeCLIP (Lai et al., 2024) and La- CLIP (Fan et al., 2023) synthesizing new texts for the im- ages by improving the captions. Though it is impossible to have completely fair comparison with them', the syn- thetic texts from the synthesis path (2) in CtrlSynth provide similar effects. We compare with VeCLIP (Lai et al., 2024) and LaCLIP (Fan et al., 2023) in Appendix A.7 in 3M and larger-scale 200M and 400M settings, CtrlSynth consistently outperform both prior works.
Atribute
52.6/67.4
61.7 / 72.7
SWAP
Object
50.6 55.6
62.0 63.7
AVERAGE
64.0/ 68.5 (+4.5)
72.3 / 75.3 (+3.0)
solution, especially when the tail classes only have a few samples. In this section, we evaluate the effectiveness of synthetic samples from CtrlSynth for long-tail tasks.
Setup. We conduct experiments on the ImageNet- LT (Liu al., 2019) and Places-LT (Liu et al., 2019) datasets. ImageNet-LT is subset of the original ImageNet- 2012 (Deng et al., 2009) and contains 1 115.K images from 1000 classes, with 5 to 1280 images per class. Places-LTT is even more imbalanced and contains 62.5K images from 365 classes, with to 4980 images per class. The test sets of both datasets are balanced. Following the same setup in (Liu et al., 2019), report the overall accuracy as well as the accuracy across the head (>100 100 images), medium (20~100), and tail (<20) classes. We take the same baseline in (Shi et al., 2024) and fine-tune the classifier head of a pretrained CLIP model (ViT-B/16) for 10 epochs (or the same number of iterations for CtrlSynth). For CtrlSynth synthetic samples, we choose the synthetic path SP(3) to generate synthetic images for the tail classes. We mix the CtrlSynth image samples with the original training set of each dataset. We describe more details in Appendix A.2.
Image-Text Retrieval Evaluation. We evaluate the zero- shot image-text retrieval performance for our CtrlSynth and baseline CLIP models and present the recall@1 results in Table 3. CtrlSynth substantially improves the text-to-image and image-to-text retrieval recall by up to 24% and 36% for the Flickr dataset, and overall improves recall by 23.4% on average for CC3M models. CtrlSynth also brings over 9% retrieval gains for CC12M models on average. The improvements show that data samples from CtrlSynth have better coverage of visual concepts.
Compositional Reasoning Results. A key strength in Ctrl- Synth is the inclusion of visual tags that contain objects, attributes and relations from image. To understand how the fine-grained visual attributes and relations affect visual reasoning performance, we evaluate CtrlSynth and baseline on the SugarCrepe (Hsich et al., 2023) benchmark which measures the compositional reasoning capability of vision language models. We present the results in Table 4. Ctrl- Synth improves the baseline CLIP compositional reasoning by a large margin (4.5% for CC3M and 3% for CC12M on average). Note that most of the improvements come from the attribute and relation forms in the REPLACE and SWAP categories, for example, CtrlSynth on CC3M improves the REPLACE relation accuracy by 4.3% and SWAP attribute by 14.8% indicating CtrlSynth models are robust to the attribute and relation changes.
26
Baseline
24
CtrlSynth
22
20
18
Accuracy
16
14
12
10
2
3 4
5
6
9 10
11 12
13
14 15 16
4.3. Performance on Long-tail Tasks.
Normalized Iterations
Real-world data often have long-tail distributions. Much recent research (Shi et al., 2024; Liu et al., 2019) has fo- cused on developing new learning methods for long-tail recognition tasks. Data augmentation remains important
Data efficiency comparison between baseline and Ctrl- Synth for pretraining CLIP models on CC3M. We normalize the iterations by dividing the total iterations by the checkpoint steps.
Figure 5:
Key Results. Table 5 shows that CtrlSynth improves the tail class accuracy by 21.3% on ImageNet-LT and by 16.2% on Places-LT. Synthetic samples from CtrlSynth also im- prove the overall and medium class accuracy by 3~6%
'Factors that prohibit apple-to-apple comparison include train- ing software, variations of CC3M samples due to missing images, exact hardware set up, etc.
7

---

CtrlSynth
Table 5: Long-tail accuracy on the ImagetNet- LT and Places-LT datascts for the baseline and CtrlSynth models.
similar task performance numbers, this is likely because SD3M has fewer (2B versus 3.5B) parameters compared to SDXL, limiting the image generation capability.
Dataset
Category
Baseline
CtrlSyath
Tail
13.8
(+21.3)
Effectiveness of Visual Tags. We study the effects of using different categories of visual tags, i.e, using only objects (Obi), objects plus attributes (Obj+Attr), and all categories including relations (Obj+Attr+Rel). In Table 6, comparing the second and last row, we show attributes marginally im- prove the CLIP performance compositional reasoning but not much zero-shot vision tasks. Importantly vi- sual relations improves the performance on all three tasks, and significantly improves compositional reasoning perfor- mance by over 4%.
ImageNet-LT
Medium
56.7
62.8 (+6.1)
Head
82.6
814
Overall
60.8
66.2 (+5.4)
Tail
8.2
(+16.))
Medium
31.3
34.6 (+3.3)
Places-LiT
Head
53.7
51.2
Overall
34,9
38.6 (+3.7)
though slightly decrease the head class accuracy.
4.4. Analysis
CtrlSynth Samples from Different Synthesis Paths. Ctrl- Synth pipeline supports synthesizing images or texts from different paths, we evaluate their quality by measuring the downstream task accuracy of the CLIP models trained on them. The penultimate and last rows in Section 4.4 show all CtrlSynth samples provides performance gains on downstream tasks, except the CurlSymth-ig SenSyytthiinggg samples where they do not improve compositional reasoning performance. CirlSynth-img samples have the least augmentation benefits and are likely due to the original real texts are noisy and thus the generated images are not of high quality. Notably, mixing with synthetic captions (CtrISynth-cap CtrlSynth- capimg, and CtrlSynth-mix) provides meaningful augmen- tation benefits, highlight the importance of using LLMs to recombine the visual tags.
In this section, we first study the data data-efficiency of CtrlSynth and then evaluate the effectiveness of visual tags, and the impact of using different synthesis paths and pretrained models in the CtrlSynth pipeline. We use the same text and image control policy described in Section 3.2 for all settings. We experiment with CC3M dataset for CLIP pretraining and report the accuracy on the SugarCrepe benchmark, zero- shot accuracy of common downstream vision tasks (same tasks in Table 1), and top1 accuracy on the ImageNet 1k validation set. We present the effects of filtering and mixing ratios of CtrlSynth samples in Appendix A.5. We visualize and show the sample statisties in Appendix A.8. We further study different ViT backbones for training CLIP in Table 9.
Data- Efficiency of CtrlSynth in Training CLIP. To study the data efficiency of CtrlSynth samples, we plot the topl zero-shot accuracy of the ImageNet validation set in Fig- ure 5 for the baseline and CirlSyyth CLIP models trained on CC3M. CtrlSynth reaches the 20% accuracy with 40% fewer iterations than the baseline, indicating that using CtrlSynth Furthermore, our method
5. Conclusion
Synthetic data emerges as viable solution to address chal- lenges in curating high-quality samples from noisy, mis- aligned, and long-tail web data. However, existing data synthesis pipelines are rigid and the generation process is hard to control and thus being tailored for ad hoc use cases. We develop CirlSynth, new image-text synthesis pipeline that allows users to control the data generation in fine-grained way. CtrlSynth decomposes the semantics of images and texts into basic elements and uses pretrained foundation models to recompose them based on specified control policies. This way, CtrlSynth provides flexible and diverse image-text samples. Synthetic samples from Ctrl- Synth improve the long-tail task performance by a large margin. They also significantly boost the zero-shot and compositional capability of CLIP models and enable data- efficient multimodal learning.
samples is more data-efficient can be combined with previous techniques that perform deduplication, filtering, and pruning (Mahmoud et al., 2024; Abbas et al., 2023; Zhang et al., 2024) to improve further data efficiency.
Different Prefrained Models. We choose an alternate LLM and different text-to-image model understand how dif- ferent pretrained models affect the quality of synthetic sam- ples. CtrlSynth pipeline is flexible so we can easily swap the pretrained LLM and text-to-image models. Specifically, we use Qwen2-7B (Yang et al., 2024a) for the LLM and Stable Diffusion 3 Medium (Esser et al., 2024) (SD3M) for the text-to-image model. Comparing the first and last rows in Table 6, we find using smaller LLM like Qwen2-7B degrades the task performance on all three tasks, indicating that using strong LLM is key to synthesizing high-quality texts. The accuracy boost (43%) on the SugarCrepe bench- mark shows the LLM is effective in recombining the visual tags to form diverse synthetic texts. We also point out that using a more recent diffusion model like SD3M provides
Impact Statement
Synthetic text and image data play a key role in enhancing CLIP's zero-shot capabilities and improving performance on long-tail vision tasks. The approach in this paper enables
8

---

CtrlSynth
Table 6: Ablation of different models, visual tags, and synthetic samples in CtrlSynth.
denotes the default value (last row).
Study
Model
Tags
Samples
Common Tasks
IntageNet-IK
SugarCrepe
Qwen2-7B, SDXL
24.7
23.5
65.1
Models
Qwen2-7B, SD3M
26.1
23.8
65.2
Mistral-Nemo, SD3M
26.6
25.1
68.1
Ohj
26.4
24.7
64.3
Tags
Obj+Attr
26.2
24.8
65.4
CtriSynth-cup. SP(1)
26.2
24.5
67.2
Sumples
CulSyath-ing, SP(4)
22.1
2188
64.4
ChrlSynth-apimmg. SP(3)
26.5
24.8
67.5
CtrlSynth
Mistral Mistral-Nemo, SDXL.
Obj+AAtr+Rel
Crrlyytth-mix
27.1
25.3
68.5
more effective adaptation to real-world scenarios, where data scarcity and distribution imbalances are common challenges multimodal models. We also point out that the synthetic data can potentially inherit biases from the original datasets, and amplify existing issues that may also overfit specific artifacts which we do not further explore.
Instrrccfiiiix...aaainine.oooooo Edi wtt.nn/iinsruuttisssssc...... (page 3)
Cao, A.-Q., Jaritz, M., Guillaumin, M., Charette, R. d., and Bazzani, L. LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM--Sytthetic Texts, October 2024. URL http: //arxiv. htt.voorggbbb/////// 08211. arXiv:2410.08211 [cs]. (page 3)
References
Abbas, A., Tirumala, K., Simig, D., Ganguli, S., and Morcos, A. S. SemDeDup: Data-efficient learning at web-scale through semantic deduplication, March 2023. URL http: http://aaxiv.orgabb/222200 09540. arXiv:2303.09540. (page 8)
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail -aii Visual Concepts. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3558-3568, 2021. URL https: thecvf. com http://ootttttttceeee http://eenn/cc/2222/////////ngg/ Wwwp:/.uuuhingeeebbbaaaaaaa
AI, M. Mistral NeMo, July 2024. https/tts:/miittrrall ai/news/mistrallneeee (page 6)
Web Pre-Training_ To_.eoinnn___RReggniiieeeeee Leccognize_Long-Taaee CVPR_2021_paper html. (page 6)
Text_
Azizi, S., Komblithh S., Saharia, C., Norouzi, M., and Fleet, D. J. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https: //openreview.o httt//ooommdddddooooooooooo
Cheng, G., Han, J., and Lu, X. Remote Sensing Im- age Scene Classification: Benchmark and State of the Art. Proceedings of the IEEE, 105(10):1865-1883, Oc- tober 2017. ISSN 1558-2256.
(page 3)
Birhane, A., Prabhu, V., Han, S., Boddeti, V. N., and Luc- cioni, A. S. Into the LAIONs Den: Investigating Hate in Multimodal Datasets, November 2023. URL http: wrtp////////b/////// //arxiv. org/ abs/2311.11.. 234424449.. arXiv:231 0tt:2311.00449 [es]. (page 18)
doi: 10. 10.1109/PPRC. 2017.2675998. URL https://ieeeepe: document/7891544, (page 16)
orge
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing Textures in the Wild. In 2014 IEEE Conference Computer Vision and Par- tern Recognition, pp. 3606-3613, June 2014. doi: 10.1109/CVPR.2214..66 URL https: htts:///ieeeeee httee////eee..orre. ieee. www.:oor//oocmmee///999ee (page 16)
Bossard, L., Guillaumin, M. and Van Gool, L. Food- 101 Mining Discriminative Components with Ran- dom Forests. In Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), Computer Vision ECCV 2014, pp. 446-461, Cham, 2014. Springer International Pub- doi: 10.1007/
lishing. ISBN 978-3-319-10599..
778-3-319-10599-11 29. (page 16)
Coates, A., Ng, A., and Lee, H. An Analysis of Single- Layer Networks in Unsupervised Feature Learning. In Proceedings of the Fourleenth International Conference on Artificial Intelligence and Statistics, pp. 215-223, JMLR Workshop and Conference Proceedings, June 2011. URL https: http://ppooee.....//ee//// mlr. press/v5ss coates1 html. (page 16)
Brooks, T., Holynski, A., and Efros, A. A. Instruct- Pix2Pix: Learning To Follow Image Editing Instruc- tions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023. URL. https://openaccess.

---

CtrlSynth
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.. ImageNet: A large-scale hierarchical image Computer Vision and Pattern Recognition, pp. 248-255. IEEE Computer Society, June 2009. ISBN 978-1-4244-3992-8..
Geiger, A., Lenz, P., Stiller, C., and Urtasun, R. Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, (((1)):1231-12333 September 2013. doi: 10. 10.1177/22783449114499999999 http://...1177/02786441111333331
database. In 2009 IEEE Conference
ISSN 0278-3649. URL (page 16)
doi: D0.100/CCPR.220995206222 / //www.computer. http://////////./e/eiiiioeeeee/a aartilll-rriieee//e cvpr/22009/0520664881 200p:20006644//22200/0H....... (page 6, 7, 16)
URL https:
Geng, S., Hsieh, C.-Y., Ramanujan, V., Wallingford, M., Li, C.-L., Koh, P. W., and Krishna, R. The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better, July 2024. URL http://atpa //axiv...vvv Staaxiv.oor org/ abs/2406. .51844,51844 arXiv:2406.05188
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, September 2020. URL https: dNTTy. (page 6)
(page 3)
Hammoud, H. A. A. K., Itani, H., Pizzati, F., Torr, P., Bibi, A., and Ghanem, B. SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?, February 2024. URL http://arxiv, www.ooog//////aeeeeoeeeoe abs/24022 01832. (page 3)
//openreview wwt/
Dunlap, L., Umino, A., Zhang, H., Yang. J., Gonzalez, J. E.,, and Darrell, T. Diversify Your Vision Datasets with Automatic Diffusion-basee Augmentation. In Thirty- seventh Conference on Neural Information Processing Systems, November 2023. URL
Helber, P., Bischke, B., Dengel, A., and Borth, D. Intro- ducing Eurosat: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. In IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 204-207, July 2018. doi: 10. 10.1100 109////SSSSSS.... 0018.88199445 URL https:// // leexxplore ieee. org wtgg/doccmeet/8519244 (page 16)
net net/forum2id-BwrYffddd (page 1,
Esser, P. Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, Podell, D., Dockhorn, T., English, Z., Lacey, K., Good- win, A., Marek, Y., and Rombach, R. Scaling Rectified Flow Transformers for High-Resolution Image Synthe- sis, March 2024. URL 03206. (page 8)
Hendrycks, D.., Basart, S., Mu, N., Kadavath, S.,, Wang, F., Dorundo, E., Desai, R., Zhu, T.,, Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribune Generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340- 8349, 2021a. URL https://openaccess: thecvf. com/ woonteeet//00022222////////eeeee
Fan, L.., Krishnan, D., Isola, P., Katabi, D., and Tian, Y. Improving CLIP Training with Language Rewrites, Oc- tober 2023, URL http: /ttp///axvvvv.e///////ee/eaaavvt httprvv.orr//b//255555/5 20088. (page 1, 2, 5, 7, 18, 19)
Faces_ot of Row.ousttesssACCitiiaaaaaaaaaas Out- of oit.strrbbbriooonccc..eee....aa (page 16)
Fei-Fei, I.., Fergus, R., and Perona, P. One-shot learn- ing of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):594-61, April 2006. ISSN 1939-3539. doi: 10.11099TPM..20006.
Hendrycks, D.,, Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural Adversarial Examples, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b. URL https://openaccess: wwwpsaaaccccccscseeeeeeeee twecvf. com/ www///oo//////////////oc hhrre////hha///eedddccsssstutrr
79. URL https: / http///eeeeeeeeeeeee/ee//// ieee, wrwg/dooooon////ooooooooo 1597116. 16. (page 16)
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis,
hwwp:/222pppppphhhhe (page 16)
G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D.,
Zhang, J., Orgad, E.,, Entezari, R., Daras, G., Pratt, S. M., Ramanujan, V., Bitton, Y., Marathe, K., Mussmann, S.,
Hsich, C.-Y., Zhang. J., Ma. Z., Kembhavi, A., and Krishna, R. SugarCrepe: Fixing Hackable Bench- marks for Vision-Language Compositionality. In Thirty- seventh Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track, Novem- ber 2023. URL htttp:// net/ wtt.vviwwwweeeeet/niiiid. fott Js=7WSSZoAAnnttee<=kkiryyyy
Vencu, R., Cherti, M., Krishna, R., Koh. P. W. Saukh, O., Ratner, A., Song, S., Hajishirzi, H., Farhadi, A., Beau- mont, R., Oh, S., Dimakis, A., Jitsev, J., Carmon, Y., Shankar, V., and Schmidt, I.. Datacomp: In search of the next generation of multimodal datasets. In Thirth-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //openr eepenreview.oo
(page 3,6, 6.
Islam. K., Zaheer, M. Z., Mahmood, A., and Nan- dakumar, K. DiffuseMix: Prese-ige Data
(page 6)
10

---

CtrlSynth
Augmentation with Diffusion Models. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 27621-27630, 2024. URL https::/ http::///oppaacccssstcc...... www.wf.ecccccoo com/ Coteent/CRRRO0///haa/////aaaffff
Li, X., Tu, H., Hui, M., Wang, Z., Zhao, B., Xiao, J., Ren, S., Mei, J., Liu, Q., Zheng, H., Zhou, Y., and Xie, C. What If We Recaption Billions of Web Images with LLaMA-37, June 2024. URL http::/arxivv wtt./bo//////.../o/ 08478. (page 2)
Lalel--Prseervingg.otttaaageeepn _h_
offiussionnodels_ccvPPe222222222 3)
html. (page 1,
Li, Y., Fan, H.,, Hu, R., Feichtenhofer, C., and He, K. Scaling Language-Image Pre-Training via Masking. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recog- nition, pp. 23390-23400, 2023c. URL https: / //openaccess thecvf cww/ cotem/eonnn////222//////// httpiiiisaaaiinnnanaiaaieaaa Bwtiikinn__vPR_2200 papp__rreppppprr_______
Kang, W., Mun, J., Lee, S., and Roh, B. Noise-aware learn- ing from web-crawled image-text data for image cap- tioning. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 2942-2952, October 2023. (page
paper. hamal.....ml..............mmm..m (page 2)
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3D Ob- ject Representations for Fine-Grained Categorization. In 2013 IEEE International Conference on Computer Vi- sion Workshops, pp. 554-561, December 2013. doi: 10.1 10.1109//CCW..2013..7 77. URL https: https://ieeexploo. beee.org//ocument//// 6755945.. (page 16)
Lian, L., Li, B., Yala, A., and Darrell, T. LLMM-gounded Diffusion: Enhancing Prompt Understanding of Text-to- Image Diffusion Models with Large Language Models. Transactions Machine Learning Research, October 2023. ISSN 2835-8856. URL httpps://pperreview net/ net//foru?2dd-hAALDTbooH (page 4)
Krizhevsky, A. Learning Multiple Layers of Features from Tiny Images. In Technical report. University of Toronto, 2009. URL hitpps://wwt cs. toronto. edu/ -Kriz/learning-fetuuee featureseeeteeeeeeeeee teeeeeeeeeeee------ pdf. (page 16)
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Mi- crosoft COCO: Common Objects in Context. In Fleet, D., Pajdla, T. Schiele, B., and Tuytelaars, T. (eds.), Computer Vision - ECCV 20/4, Lecture Notes in Com- puter Science, pp. 740-755, Cham, 2014. Springer In- ternational Publishing. ISBN
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H.. Gonzalez, J., Zhang, H., and Stoica, I. Efficient Memory Management for Large Language Model Serv- ing with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23, pp. 611 1-626, New York, NY, USA, October 2023. Asso- ciation for Computing Machinery, ISBN 9798400702297. doi: 0001144//6600006336333 org/doi/18 1t 000111155533000 361 3165. (page 17)
doi: 3-17778----1--111 1-312-10002-11 1110600221488-- 1_48. (page 6, 16)
Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., and Yu, S. X. Large-Scale Long-Taile Recognition an Open World. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2537-2546, 2019. URL https: //ppaaccss thecvf. comm/content_CVPR_200 2019/ http9/ghtt//LLuuargggeee Scale_ Long-Tailed_Reoopp CVPR_2019_papeer html. (page 6, 7)
URL https: //d1. acm.
Lai, Z., Zhang, H., Zhang, B., Wu, W., Bai, H., Timofeev, A., Du, X., Gan, Z., Shan, J., Chuah, C.-N., Yang, Y., and Cao, M. VeCLIP: Improving CLIP Training via Visual- enriched Captions, March 2024. URL http://arxit org/ trg/abs/2310 200.00/999... (page 1, 2, 4, 5, 7, 18)
Loshchilov, I. and Hutter, F. Decoupled Weight Decay Regularization. In International Conference on Learn- ing Representutions, September 2018. URL https: //operteview.nee//oounnitt wttp://ee/oorrrmntteeeeeerrry (page 15)
Li, X., Wang, Z., and Xie, C. CLIPA-v2: Scaling CLIP Training with 81.1% % Zero-shot ImageNet Accu- racy within a $10,000 Budget. In TooMoooobbusnees of Few-shot and Zero-shot Learning in Large Foundation Models, December 2023a. URL https: hatps:///ppeeeee /opp//eveenee.e..eeeeee net/forumiid=Ohi ti (page 2)
Loshchilov, I. and Hutter, F. SGDR: Stochastic Gradient
Descent with Warm Restarts, In International Conference on Learning Representations, July 2022. URL https: wwt.:/e////oooee nett/foruumid--SaaBSeaa (page 16)
Troperreview.ne//formmie
Li, X., Wang, Z., and Xie, C. An Inverse Scaling Law for CLIP Training. In Shirth-eeventh Conference on Neural Information Processing Systems, November 2023b. URL https: htas:///ppenae http///pppeeeeeeeee/eee/ee wtw.eewwwwe/ooommmittt.e.. LMU2RNwdh2. (page 2)
Mahmoud, A., Elhoushi, M., Abbas, A., Yang, Y., Ardalani, N., Leather, H., and Morcos, A. Sieve: Multi- modal Dataset Pruning Using Image Captioning Mod- els, March 2024. URL http:///rke warxiv.org/abs/2311 0211 10. arXiv:2310.01100 (page 8)

---

CtrlSynth
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-Grained Visual Classification of Aircraft, June 2013, URL (page 16)
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V.
Cats and dogs. In 2012 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 3498-3505, June 2012. doi: 10. 10.109//VPPR2211.62488888 ieeexplore. ieee. httpeeeer//dccccceeeeeeeeee/
5151.
URL https:// //
(page 16)
Mehta, S., Abdolhosseini, F., and Rastegari, M. CVNets:
High Performance Library for Computer Vision. In Pro- ceedings of the 30th ACM International Conference Multimedia, MM '22 Pp. 7327-7330, New York, NY, USA, October 2022. Association for Computing Machin- ery. ISBN 978-1-4503-9203-7. doi: 10.1 145/30366.... 3548540. URL https://dl. acm. 3503161. 3548540. (page 16)
Plummer, B. A., Wang, I.., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k Entities: Collecting Region-to- to-Phrase Correspondences for Richer Image-to-Sentence Models. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2641-2649, December 2015, doi: 10. 1091100/..22225..33333 (page 6, 16)
http:://dd.accm.rrr//o////// wwwp/oo/g/...oo./ 1145/
Mehta, S., Abdolhosseini, F., and Rastegari, M. apple/- corenet, September 2024a. URL hwtph:tts//sg.. thub. com/ apple/corenet. (page 16)
Podell, D., English, Z., Lacey, K., Blattmann, A., Dock- horn, T.,, Müller, J., Penna, J., and Rombach, R. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: (page 1,
Mehta, S., Horton, M., Faghri, F., Sekhavat, M. H., Najibi, M., Farajtabar, M., Tuzel, O., and Rastegari, M. CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data, April 2024b. URL http://arxiv. wwwporg/////eeeeeeoooeeeeeeeeege 15653. (page 3, 6, 16)
I ///pee/eeeew.eee//eeeeeeeeeeeeee
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Transfer- able Visual Models From Natural Language Supervi- sion. In Proceedings of the 38th International Confer- ence on Machine Learning, pp. 8748-8768 PMLR, July 2021. URL https://proceeding mlr. press/v139/ radford21a, html. (page 1, 2. 5, 16)
Mishra, S., Castillo, C. D., Wang, H., Saenko, K., and Saligrama, V. SynCDR: : Training Cross Do- main Retrieval Models with Synthetic Data, March 2024. URL http://arxivv wwt.xxv.org/ab//240000 00420. N.Kv:2201.004200 (page 3)
Mu, N., Kirillov, A., Wagner, D., and Xie, S. SLIP: Self- supervision Meets Danguage-Image Pre-training. In Computer Vision - ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Proceed-
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do ImageNet Classifiers Generalize to ImageNet? Proceedings of the 36th International Conference on Machine Learning, pp. 5389-5400 PMLR, May 2019. URL https: hatps::/ppoceeingss mlr. www....res/.vgp../ recht 9a. tahtt9..hmml..... (page 6, 16)
ings, Part XXVI, pp. 529-544, Berlin, Heidelberg, Oc- tober 2022. Springer- Verlag ISBN 778-3-031-19888--1
30. URL https:// doi: 1001 doi.
(page 2)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P.. and Ommer, B. fiighRReolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2022. URL https: thecvf. cont//een/////tn///////cc CVPR2022/httll/ombbas htt.aa///hiil/oocccccccciaieerre hiw.hhhhhoeoottttoon_nmeeee
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS Work- shop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http://ufldd stanford. edu/housenumbers/nippaa wwwosenuuberss..o (page 16)
hhoossseesssp...sssuoouurrrrrrr pdf.
Synthesis_Witt htt.reettiiifffsioooooellsssssss 2022_paper htmlp. (page 1)
Nilsback, M.-E. and Zisserman, A. Automated Flower Classification over a Large Number of Classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729, December 2008. doi: 10. 119111VG1...20000.4.. 47. URL https://ieeexplore ieee. wete..rr//doomuee///755/
Schramowski. P., Brack, M., Deiseroth, B., and Kersting, K. Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Mod- els. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. / //openaccess. thecvf. comm//oontn////R2223. Ltml/Schramowsse Lanfe_LLatet__iiffaee
(page 16)
2023. URL https:
OPenAI. Chatgpt, 2022. URL https://chattpt..
(page 1)
12

---

CtrlSynth
Mi htt.ghgg__napppoorraaaeeeeeeeeee Di fifuusionn_ocdelssccPP.... 2023 html. (page 1)
Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z.,, 7arov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open Foundation and Fine- Tuned Chat Models, July 2023. URL http://arxiv. org/abs/ hts/abs/2300/ http///777....8.... (page 1)
Schubmann, C., B.Bummont,, R., Vencu, R., Gordon, C. W.,
Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S. R., Crowson, K., Schmidt, L.,, Kaezmarczyk, R., and Jitsev, LAION-5b: An open large-scale dataset for training next generation image-tert text models. In Thirty-sixth Conference on Neural Information P Processing Systems Dulasets and Benchmarks Track, 2022. URL httpss//oppnrreviww (page 1)
Trabucco, B.,
Doherty, K., Gurinas, M. A., and Salakhut- dinov, R. Effective Data Augmentation With Diffusion Models. In The Twelfth International Conference on Learning Representations, October 2023. URL https: (page 3)
net
//openreview wtt..//eet//frumm/dd///ua....ee
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Concep- tual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th An- mual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pp. 2556-2565, Mel- bourne, Australia, July 2018. Association for Computa- tional Linguisties. doi: 10.18653/v1/P18. 1238. URL https://aclanthology. orwpe1/-1111111111 1238. (page 5)
Udandarao, V., Prabhu, A., Ghosh, A., Sharma, Y., Torr, P. H. S., Bibi, A., Albanic, S., and Bethge, M. No "Zero- Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance, April 2024. URLhttp://arxiv org/ abs/ 04125. (page 1)
Vasu, P. K. A., Pouransari, H., Faghri, F,, and Tuzel, O. CLIP with Quality Captions: A Strong Pretraining for Vision Tasks, May 2024. URL http::/arxii..rrg/ee 2405. 08911. (page 19)
Shi, J.-X., Wei, T., Zhou, Z., Shao, J.-J., Han, X.-Y., and Li, Y.-F. Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts. In Proceedings of the 41 In- ternational Conference on Machine Learning, pp. 45014 45039. PMLR, July 2024. URL URL https: /pw.eedeid....... mlr wwr.press/v222 httrress/v235//h2222
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation Equivariant CNNs for Dig- ital Pathology. In Medical Image Computing and Computer Assisted Intervention - -ICCAI 2018: 21st International Conference, Granada, Spain, Septem- ber 16-20, 2018 Proceedings, Part II, pp. 210- 218, Berlin, Heidelberg. September 2018. Springer- Verlag. ISBN 978-3-030-00933-55. 978 978-3-030-00934-2 24. URL
html. (page 7, 16)
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. In The 2011 International Joint Conference on Neural Networks, pp. 1453-1460. July 2011. doi: 10.1109/ICCN. 2011.6033395. URL https://ieeexplore ieee..eee.ooo org// document/6033395 (page 16)
doi: 10.1007/
1007/978 3-788-3-00---000 000-0000000000 009300000042200 209934-222444 24... (page 16)
von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj. M., Nair, D., Paul, S., Berman, W., Xu, Y., Liu, S., and Wolf, T. Diffusers: State-of-the- art diffusion models, 2022. URL httpp::/gttthhe com/ huggingf aengiingaae/diiiiie awtea//ff/ffferre//////e/////eee (page 17)
Tian, Y., Fan, L., Isola, P., Chang, H., and Krishnan, D. StableRep: Synthetic Images from Text-to- Image Models Make Strong Visual Representation Learners. In Thirty- seventh Conference on Neural Information Processing Systems, November 2023. URL https::/ net met/fforum?idxxpjssttka (page 3)
Wang. H., Ge, S., Lipton. Z., and Xing, E. P. Learn- ing Robust Global Representations by Penalizing Local Predictive Power. In Advances in Neu- ral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: / /pproceddings. www.nneuriicccm cc/ppe/s.c 3eefceb8087 0-044499222698822999199---e html. (page 16)
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G.. Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, L. Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K. Schelten, A., Silva, R., Smith, M., Subramanian, R.,
Wang, 7., Li, A., Li, Z., and Liu, X. GenArtist: Multimodal LLM as an agent for unified image generation and editing. In Globerson, A., Mackey, L.,, Belgrave, D.,, Fan, A., Paquet, U.. Tomezak, J., and Zhang, C. (eds.), Advances neural information processing
13

---

CtrlSynth
systems, volume 37, pp. 128374-128395. Curran As- sociates, Inc., 2024. URL https://proceedings.
2024b. URL URL.htbss//prroceedngg.t mlr. ptt.rrrrrreessss/// 2ong24ai.htmle (page 4)
4C736604001777447711111f11111110 wwt.rpprrr..ereeeeeccuctttto..n. Yuksekgonul, M., Bianchi, F. Kalluri, P., Jurafsky, D., pdf. (page 3)
and Zou, J. When and Why Vision-Language Mod- els Behave like Bags-Of-Words, and What Do About It? In The Eleventh International Conference on Learn- ing Representations, September 2022. URL https: //operreview.ne//forum//
Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.. Davison, J., Shleifer, S., von Platen, P., Ma, C., Jer- nite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of- the-Art Natural Language Processing. In Liu, Q. and Schlangen, D. (eds.). Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, Oc- tober 2020. Association for Computational Linguistics. doi: 18..1863// 1/2020. http:///ww220..mmmmmmmmmmeeeeeee emn.p-demos.6. URL https:// aclanthology org/ 22200..022//222 emnlp-demos... 6. (page 17)
(page 3)
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, I.., Bachem, O., Tschannen, M.. Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. A Large-scale Study of Representa- tion Learning with the Visual Task Adaptation Bench- mark, February 2020. URL. http::/arxiv.org//// 1910. 04867. (page 6, 19)
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L. LiT: Zero-Shot Transfer With Locked- of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18123-18133, 2022. URL /ttpa:s/op thecvf. com/ WPPRee//ttal/hhaaiiiiieoooobbbbo
Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., and Yuan, L. Florence-2: Advancing Unified Representation for a Variety of Vision Tasks, In Proceed- ings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4818 4829, 2024. URL https: thecvf. com/ conten// http://m//iiiooooooeeeeeeeeadaaa
Text Tuning. In Proceedings
CVPR2024/httm//xaa__Foo0
Wi Imm_Lockkd-Image_Teee Ter.eae_eeet__nnnn___c____._. paper. (page 2)
html. (page 4, 6, 16)
Zhang, K., Mo, L., Chen, W., Sun, H., and Su, Y. Mag- icBrush: A Manually Annotated Dataset for Instruction- Guided Image Editing. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, November 2023. URL https: //operreiiw. feet/forrmmiddzzDDBeBe
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Tor- ralba, A. SUN database: Large-scale scene recognition from abbey z0o. In 2010 IEEE Computer Society Conference Computer Vision and Pattern Recogni- lion, pp. 3485-3492, June 2010. doi: 10.1109/C ieee..rreeee...
(page 3) Zhang, I., Shu, F., Liu, T., Ren, S., Jiang, H., and Xie, C. Filter & Align: Leveraging Human Knowledge to Curate Image- Text Data, September 2024. URL http: org/ a/s/ 06726. arXiv:2312.06726. (page 8)
2010.5539970. URL document/5539970 (page 16)
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H.,
Lin, H., Tang, J., Wang, J., Yang, J.. Tu, J., Zhang, J.,
Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P.. Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y.. Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, 7., Zhang, Z., Guo, Z
A. Appendix
A.1. Control Policies
Text Prompt Templates. We provide example control policies for text synthesis as predefined prompt templates, the first five templates do not include original text:
and Fan, Z. Qwen2 Technical Report, July 2024a. URL https://arxiv org.oogg//bboo htsp/bbs////0// 10671v4. (page 6, 8, 16)
"Create a detailed and high-quality caption using phrases that represent the entities or objects, their unique attributes, and the visual relationships in the scene depicted. Phrases: (phrases)."
Yang, L., Yu, Z., Meng. C., Xu, M., Ermon, S., and Cui, B. Mastering Texl-to-Image Diffusson: Recaption- ing, Planning, and Generating with Multimodal LLMs. In Proceedings of the 41st International Conference on Machine Learning, pp. 56704-56721. PMLR, July
2. "Compose a rich and immersive caption by incorporat- ing a set of phrases that illustrate the entities or objects,
14

---

CtrlSynth
their defining attributes, and the interconnections pre- sented within the image. Phrases: (phrases)."
5. "Write new faithful and high-quality caption based the given phrases and a given sentence. The given sentence is the original caption and the phrases are entities or objects, attributes, and their visual relation- ships in an image. Given sentence: [caption). Given phrases: [phrases). the sentence contains objects or nouns (e.g. digits, car models, planes, pets, animals, etc.). the new caption should be faithful and keep this information. Otherwise, use the phrases to create the new caption."
3. "Formulate articulate and informative caption by using series of phrases that outline the entities, their attributes, and their visual relationships depicted in an image. Phrases: [phrases)."
4. "Using a set of phrases that highlight the entities, at- tributes, and their visual associations in an image, craft detailed and expressive caption. Phrases: (phrases)."
5. "Construct a comprehensive and expressive caption by integrating phrases that detail the entities, their features, and the spatial or thematic relationships in image. Phrases: |phrases]."
Image Prompt Templates. templates:
We provide five image prompt
1. "real": "a real photo. (prompt). 35mm photograph, film, bokeh, professional, 4k, highly detailed",
The following five templates include the original text, which is useful for maintaining the original meaning:
"nocap": "a real photo showing (prompt). highly de- tailed"
1. "Create comprehensive caption that faithfully rep- resents the objects, attributes, and their relationships contained within the provided sentence and phrases. Given sentence: [caption]. Given phrases: {phrases]. If the original caption specifies particular give phrases, maintain their integrity while using the phrases to en- hance the description."
3. "isometric"; "isometric style (prompt) vibrant, beau- tiful, crisp, detailed, ultra detailed, intricate"
4. "enhance": "breathtaking (prompt). award-winning professional, highly detailed"
5. "quality": "masterpiece, best quality, ultra detailed, (prompt). intricate details"
2. "Write a faithful caption by integrating the given phrases with the original sentence. Given sentence: [caption]- Given phrases: (phrases). Ensure any ob- jects or specific nouns from the original caption are preserved while elaborating on the visual relationships and attributes provided in the phrases to create a more detailed depiction." "
A.2. Datasets Details
Evaluation Datasets. uation in Table 7.
We list the vision datasets for eval-
Long-tail Datasets. For the tail classes in ImageNet-LL and Places-LT, we generate synthetic images using the "real" style of image prompt template, and we generate 7 samples per tail class so that we roughly double the size of the original real datasets. We obtain 80.4k synthetic samples for ImageNet-LT and 55.2K for Places-LT.
3. "Provide a faithful and informative image caption using given sentence and few phrases. Sentence: (caption), phrases: [phrases]. Consider the initial sentence as base for the overall context and ensure that specific objects or nouns such as numbers, models, animals, etc., are preserved in the new caption. Integrate the given phrases, which describe entities, attributes, or visual relationships, to enrich and elaborate on the original meaning. Maintain fidelity to the original content while enhancing descriptive quality."
A.3. Training Details
We pretrain the CLIP for the same number of iterations for both the baseline and CtrlSynth. For example, suppose we train for F epochs, the original dataset has N samples, CtrlSynth has generated N' samples (N' <= due to filtering), then the total samples are E N. we train CtrlSynth models for EsN epochs. This guarantees that the baseline and CtrlSynth CLIP models have seen the same number of data samples.
Pretraining
4. "Make detailed caption based on the given phrases and a given sentence. Given phrases: (phrases). Given sentence: (caption). The sentence serves a founda- tion, while the phrases elaborate on elements depicted in the image, like objects, their characteristics, and interactions. Preserve any pivotal information concern- ing objects, attributes, and their relations present in the sentence."
N+N7
Table 8 lists the paremerrrseeters used for pretraining on CC3M and CC12m. We use AdamW (Loshchilov & Hut- ter, 2018) with default values as an optimizer and binary cross-entropy loss as an objective function. We use cosine
15

---

CtrlSynth
Table 7: Details of evaluation datascts.
Dataset
Metric
Classes
Test Set Size
CIFAR-10 (Krizhevsky, 2009)
Accuracy
10000
CIFAR-100 (Krizhevsky, 2009)
Accuracy
100
10000
CLEVR Counts
Accuracy
8
15000
CLEVR Distance
Accuracy
6
15000
Caltech-101 (Fei-Fei al., 2006)
Mcan Per Class Recall
102
6085
Country211 (Radford et al., 2021)
Accuracy
21100
DTD (Cimpoi et al., 2014)
Accuracy
47
1880
EuroSAT (Helber et al., 2018)
Accuracy
10
5400
FGVC Aircraft (Maji et al., 2013)
Mcan Per Class Recall
100
3333
Food-101 (Bossard al., 2014)
Accuracy
25250
GTSRB (Stallkamp ct al., 2011)
Accuracy
43
12630
KITTI (Geiger et al., 2013)
Accuracy
4
711
Oxford Flowers- 102 (Nilshack & Zisserman, 2008)
Mean Per Class Recall
102
6149
Oxford-IITT Pet (Parkhi al., 2012)
Mcan Per Class Recall
37
3669
PatchCamelyon (Veeling et al. 2018)
Accuracy
2
32768
RESISC45 (Cheng et al., 2017)
Accuracy
45
6300
STL-10 (Coates et al., 2011)
Accuracy
10
8000
SUN397 (Xiao al., 2010)
Accuracy
397
108754
SVHN (Netzer al., 2011)
Accuracy
26032
Stanford Cars (Krause et al., 2013)
Accuracy
196
8041
ImageNee-1K (Deng et al., 2009)
Accuracy
1000
50000
ImageNettV2 (Recht al., 2019)
Accuracy
1000
10000
Ieage (Wang al., 2019)
Accuracy
1000
50889
ImageNet-A (Hendrycks et al., 2021b)
Accuracy
7500
ImageNet-O) (Hendrycks et al., 2021b)
Accuracy
200
2000
ImageNet-R (Hendrycks et al., 2021a)
Accuracy
200
30000
Flickr (Plummer al., 2015)
Mcan Recall @
1000
MSCOCO (Lin et al., 2014)
Mcan Recall @
5000
Table 8: Training
(a) Pretraining CLIP CC3M and CC12M.
(b) Finetuning CLIP on Places- LT and ImageNet-LTT
Hyperparameter
CC3M
CC12M
Hyperpurameter
Places-LT
ImageNNt-LT
Total iterations
56,429
$5,429
Iterations
$6,429
55,429
Warmup iterations
2822
2771
Warmup licrations
2822
2771
Image size
224
224
Image size
224
224
scheduler
Cosine
Cusine
Loss type
CrossEntroy
Crosslintropy
Mux. LR
0.002
0.002
LR scheduler
Cosine
Cosine
Min. LRR
0,00002
0.00002
Learnnng rate
0.01
0.01
Optimizer
Adamn
AdamW
Optimizer
SGD
AdamW (i's
0.98)
(0.9,0.98) 0.98)
Momentum
0.9
0.9
Weight decay
0.2
0.2
Weight decay
Se-4
Se.4
Batch per GPU
256
256
Batch per
128
128
100 GPUs
8
32
A GPUs A GPU Memory
A100 GPU Memory
40 GB
40 GB
40 GB
40
learning rate schedule (Loshchilov & Hutter, 2022). We the CoreNet library (Mehta al., 2024a; 2022) for all pretraining experiments. We adapt the LIFT codebase (Shi et al., 2024) for fine-tuning long-tail tail tasks, main modifica- tions include adding support for iteration-basee training and data loader for multiple datasets.
et al., 2024b) for each image and output top20 classes based on the sigmoid score of prediction logits, then convert the class indices to actual word labels. The vo- cabulary size of CatLIP is 24320. Most of the vocab- ulary words are nouns and single-word attributes. We then run the Florence-large (Xiao et al., 2024) for cach image to extract detailed captions using the task prompt WWW.DEEEAEEEDPPPEEEEEEE
A.4. CtrlSynth Inference Details
After that, we run Qwen2- IB- (Yang al., 2024a) extract objects, attributes, and relations from the Florence captions. We then merge the objects field with CatLIP CatLiP-predicted labels. The extraction
VTM. We use a hybrid tagging model consisting of two stages. We first run the ViT-Huge variant of CatLIP (Mehta
16

---

CtrlSynth
instruction contains 2-shot example and we list the prompt template below:
Text-to-image Model. We the diffusers (von Platen et al., 2022) library for diffusion model inference. For both SDXL and SD3M models, we use float16 dtype with guidance scale of 7.0 and set the diffusion steps to 28.
For a given image caption, identify all the attributes, objects entities, and visual relationships actions that phrases. The phrases should only come from the caption. Separate the phrases by comma without formatting. Output three lines:
A.5. Ablation Results
CtrlSynth provides off-the-shell' self-filtering to control the quality of synthetic samples. We study the effects of applying different filtering thresholds py for the synthetic text and image. We set the same filtering thresholds for both synthetic text and image samples. Intu- itively, a higher threshold filters out more synthetic samples thus providing better quality samples that align with origi- nal real samples. On the contrary, lower threshold keeps relatively less aligned samples but encourages more diverse samples. Appendix A.5 plots the zero-shot accuracy num- bers of CLIP model on ImageNet under different threshold settings, we show that thresholds 10%~30% provide similar accuracy numbers and setting the filtering threshold to 20% provides the best accuracy. Thresholds higher than 50% offer accuracy gains, likely because the aligned samples lack diversity and fail to augment the original samples.
Effects of Self-Filtering.
attributes: phrases
objects: phrases
relations: phrases
Examples:
caption:
The image a close-up portrait of middle-aged man wearing a white cowboy hat. He appears to be in his late 60s or early 70s, with gray hair and serious expression his face. He wearing dark suit jacket and a light blue collared shirt. The background is a clear blue sky with trees visible in the distance. The man is looking off to the side with a slight smile his lips.
close-up, middle-aged, white cowboy hat, gray hair, serious expression, light blue objects: portrait, man, hat, face, dark suit jacket, shirt, blue sky, trees, lips
attributes:
relations: wearing a, visible the distance, looking off to the side, slight smile on his lips
caption:
The image shows female singer performing on a stage. She standing on set of stairs with her legs spread apart and holding a microphone in her hand. The stage is lit up with red and blue lights and there is large circular screen in the background. The singer wearing black and white patterned outfit with high heels. She appears to be in the middle of song or performance.
Mixing Ratios of Synthetic Samples. To better under- stand how the synthetic image text samples improve CLIP model training, we study different ratios (p,) of mixing Ctrl- Synth samples with original real ones. During CLIP training, we randomly sample the original sample with probability
0 < Pr < and our sample with 1 - pr. Appendix A.5 shows that even adding a small portion (< 20%) of Ctrl- Synth samples improves the zero-shot accuracy while mix- ing with 50% provides best accuracy gains. Further higher mixing ratios show diminishing improvements though still better than the baseline that uses all real data.
female singer, stage, set of stairs, red and blue lights, large circular screen, black and white patterned outfit, high heels objects:
attributes:
female singer, stage, set of stairs, legs, microphone, screen, outfit, high heels, song, performance
relations: performing on stage, standing on, her legs spread apart, holding, lit up, background, wearing, in the middle of a song
A.6. CtrlSynth Self-Filtering Details
CirlSynth is closed-loop system and supports self-filtring for bad-quality synthetic text or image samples. To imple- ment synthetic text filtering, we first compute the percentage of visual tags that appear in the synthetic text compared to the original text, then we filter out the sample if the per- centage of visual tags is lower than a predefined threshold Pf. We empirically choose Py based on the zero-shot ac- curacy of trained CLIP models evaluated the ImageNet validation set. Similarly, to filter synthetic images, we first extract the visual tags of the synthetic images by running them through VTM, then compute the percentage of visual tags in the original image and filter out image samples if the percentage is lower than Py-
caption: [caption]
CatLIP is available in CoreNet so we use it directly for inference and we wrap the Florence Transformers (Wolf et al., 2020) code into the CoreNet inference pipeline for easier integration.
LLM. We use the vLLM engine (Kwon et al., 2023) for offline inference in Qwen2 and Mistral-Nemo. We use greedy decoding for the generation.
17

---

CtrlSynth
24
Accuracy (%)
20
baseline
Accuracy (%)
baseline
16
12
12
8
8
4
20
30
70
80
70
80
90
Filtering Threshold (%)
Mixing Ratio (%)
Figure 6: Study of filtering thresholds and mixing ratios of CtrlSynth samples. The accuracy numbers are topl zero-shot accuracy on the ImageNet- 1K validation set. The CLIP models are trained on the CC3M dataset and CtrlSynth samples.
Table 9: Ablation studies different backbones with CtrlSynth. We DC-200M the pre pre-training dataset.
COCO
Flickr
Model
Backbone
ImageNet-IK
ZS-Average
12T
T2I
I2T
T2I
CLIP
VIT-B/16
54.3
36.7
82.3
64.3
64.5
67.4
CtrlSynth
ViT-B/16
56.3
41.8
84.3
67.0
66.3
69.5
CLIP
ViT-L/14
55.9
38.1
84.2
65.7
69.7
70.2
CtrlSynth
ViT-L/14
59.2
41.3
91.2
80.4
72.3
73.5
CLIP
VIT-H/14
67.9
42.3
90.6
76.8
75.6
77.4
CtrlSynth
ViT-H/14
73.9
48.4
95.6
81.4
80.8
81.6
A.7. Comparison with Prior Work
Original CC12M has 11.3 million image-caption samples, CtrlSynth-cap consists of 10.2 million captions, CtrlSynth- img contains 9.5 million images, and CtrlSynth-mix has 19.7 million image-caption pairs.
We present the results on CLIP VIT/B16 models trained on CC3M for the tasks reported in VeCL.I (Lai et al., 2024), LaCLIP (Fan al., 2023) and ours. Additionally, for large- scale settings, we randomly sample 400M text image pairs from DatacompIB 1 and denote it DC-400M to com- pare with LaCLIP, and further sample half from it to get DC-200M to compare with VeCLIP. Table 10 shows that CtrlSynth outperforms VeCLIP on most VTAB datasets and improves zero-shot accuracy by 4.8% on average. CtrlSynth also surpasses VeCLIP by 7.9% the ImageNet 1K dataset. We observe similar trend when comparing CtrlSynth with LaCLIP in Table 11. Specifically, CtrlSynth achieves erage of 3.4% better accuracy than LaCLIP on 15 common datasets and 2.3% better accuracy on ImageNet 1K.
CtrlSynth Synthetic Texts. We plot the number of words for synthetic texts generated by CtrlSynth and compare them with original real texts in Figure 7.
Statistics and visualization of CtrlSynth Samples. In this section, we provide the statistics for the synthetic sam- ples from CtrlSyyth. We observe that the text samples from CtrlSynth are usually longer and contain richer information about the image. On average, CtrlSynth texts have over 60 words while original captions contain 8 words. We plot the histogram of the number of words in Figure 7 at Ap- pendix A.8 and visualize examples of CtrlSynth images and texts compared with the original real samples in Figure at Appendix A.8.
A.8. More Analysis Details
CtrlSynth Samples. For CC3M, the original dataset has 2.8 million image-caption pairs, CtrlSynth-cap contains 2.6 million captions, CtrlSynth-img contains 2.4 million images, and CtrlSynth-mix contains 5.1 million image-caption pairs.
We are unable to use the LATON datascts due to the presence of sensitive and NSFW content, as highlighted in Birhane et al. (2023).
18

---

CtrlSynth
Table 10: Comparison of the zero-shot classification accuracy between VeCLIP (Vasu et al., 2024) and CtrlSynth for CLIP trained on the 3M and 200M data settings. We report top-1 accuracy (%) for the VTAB benchmark (Zhai al., 2020) actoss tasks (6 from natural and 3 from specialized sets). We highlight the best numbers in bold.
Natural Sets
Specialized Sets
Data
Model
Caltech101
CIFAR100
SVHN
DTD
OxPet
Flowers102
EuroSAT
RESISC45
Camelyon
Average
ImageNNet 1K
CLIP
39.50
9.83
20.89
7.42
7.44
10.40
11.94
7.93
50.65
18.45
5.46
3M
VeCLIP
54,30
17.74
18.74
11.23
10.09
22.75
7.35
16.54
52.52
23.48
15.98
ChiSSythh
66.10
17.66
16.76
7.77
15.55
20.83
50.79
28.24
23.82
CLIP 82,30
61.87
42.83
64.29
75.60
58.67
46.73
59.30
60.79
63.72
200M
83.14
72.61
47,36
64.62
CidSSyrhh 84.40
70.29
45.16
63.25
75.77
65.55
52.34
61.92
63.70
66.28
Table 11: We report the zero-shot performance on ImageNet 1K and and CtrlSynth for CLIP trained on 3M and 400M data settings.
downstream datasets for both LaCLIP (Fan al., 2023)
highlight the best numbers bold.
CHBAR 100
Caltech- 10)
RESISC4S
Country211
Food 101
CIFAR-10
SUN397
Aireraft
Flowers
STL-10
EumSAT
GTSRB
Average
Cars
DID
Poss
Data
Model
CLIP
10.3
54.9
21.8
25.0
0.8
1.4
10.5
12.8
43.3
10.2
77.6
14.1
19.1
6.9
0.6
20.6
15.8
3M
LaCLIP
14.2 57.1
27.5
35.1
1.6
1.6
16.6
15.6
52.7
14.7
86.2
24.3
6.4
1.0
24.6
21.5
Ctrlyytth
34.1
44.9
0.7
1.2
16.8
20.8
24.6
10.9
0.7
28.0
23.8
CLIIP
85.5 93.0
71.7
66.8
83.5
16.7
$2.8
90.1
91.2
63.9
97.3
42.4
63.3
46.2
17.8
65.5
67.0
400M
LaCLIP
86.5
93.5
73.9
67.9
87.1
24.2
58.9
90.9
92.4
73.1
98.4
48.3
65.8
46.1
19.6
68.4
69.3
CrrlSynth
87.2 92.1
75.3
68.2
86.7
23.9
63.8
91.8
92.1
75.5
98.3
50.2
67.6
45.9
20.7
69.3
71.8
25
Real Synthetic
Percentage (%)
20
15
10
5
0
20
40
60
80
100
120
Number of Words
Figure Number of words original captions CtrlSynth synthetic texts on CC3M.
19

---

CtrlSynth
Original
wisteria decomate black and white timbered cot-
fresh red shrimpps for sale et market
View into the living room.
tuscan sua the landscape
Synthetie
caves and purple wrought railing, decorasing the small porch of the old, black and white painted cottage.
from
frestaly coogles, red shrimp, arruged
penciant hangs ceiling, right, wooden covered with a floral tablecloth, set with wooden bowl of red tomatoes and lamp.
casting
bustling seafood market, their spors visibie, overlupping and cooked perfec- tion, surrouneed by blur herbs and spices.
countryade,, cypress urranged nett ruws along winding rood, the sua setting the background.
Figure 8: Randomly selected CC3M examples of real images and captions (the first row) with their corresponding CtrlSynth synthetic samples (the second row).
20


Reviews:
[{"review_id": "0RtMeiErkm", "reviewer": "Reviewer_6Dhg", "summary": "This paper proposes CtrlSynth to build a closed-loop data generation pipeline.\nBuilding upon the powerful foundation models, this approach generates diverse synthetic data samples depending on the text or image.\nIt first breaks down the visual elements into visual tags, and exploits them with a user control to synthesize new ones.\nAlso, there are several pathways to build diverse types of data, which can provide the flexibility of this method.\nExperimental results demonstrate that the generated samples from CtrlSynth are effective to improve pertaining performance on several zero-shot benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and have provided additional clarification below. \n\n>It is unclear why the re-synthesized data from existing images helps address the long-tail problem.\n\n**Response 1**: Our visual tagging model (VTM) identifies and extracts fine-grained, long-tail concepts from existing images that traditional approaches might miss. These concepts are then semantically enriched by pretrained LLMs to expand their coverage and diversity. When text-to-image models generate new images using these enhanced long-tail concepts, they create targeted examples for underrepresented categories, effectively rebalancing the distribution. Our quantitative results in Section 4.3 confirm that this approach significantly improves performance on long-tail recognition benchmarks.\n\n\n> Rather than improving a well-trained model (e.g., fine-tuning or parameter efficient tuning), the approach involves pretraining from scratch to demonstrate the dataset's effectiveness. I don't find this to be a practical solution or reason to use a synthesized dataset\n\n**Response 2**: We want to point out that the evaluation section 4.3 shows the effects of only fine-tuning the classifier head of the pretrained models for long-tail tasks. We show the effectiveness for both pretraining and fine-tuning, demonstrating CtrlSynth's flexibility across different practical deployment scenarios regardless of whether users prefer full pretraining or efficient adaptation of existing models.\n\n>However, a comparison is needed to demonstrate the advantages of using fine-grained tags rather than sentence forms.\n\n**Response 3**: Prior work like VeCLIP and LaCLIP use sentence-level captions; we show in A.7 a detailed comparison with them, and our CtrlSynth outperforms the prior works. Our ablation studies further demonstrate that fine-grained tagging enables more precise control over specific visual attributes and concepts that might even be omitted in the accompanied natural language sentences, particularly for long-tail categories.\n\n> The primary experiments were conducted only with training on CLIP from scratch and the ViT-B size. \n\n**Response 4**: We study small and large ViT backbones (ViT-H and ViT-L) in Table 9 at Appendix A.5. We show that CtrlSynth consistently improves baselines across different backbone scales, confirming that our approach complements architectural scaling and remains effective regardless of model capacity. \n\n>I am curious about the effects of scaling up, fine-tuning, and how this approach performs on generation tasks.\n\n**Response 5**: \n- For model architecture scaling, we show the effectiveness of our CtrlSynth for small and large backbones (see response 4); for data scaling, we show CtrlSynth is effective across different sample sizes from 3M, 12M to 200M (Table 10 in page 19) and 400M (Table 11 in page 19) and outperforms prior works.\n\n- We have demonstrated the effectiveness of CntrlSynth in the fine-tuning setting as well. Please see response 2 above . \n\n- Extending CtrlSynth to generation tasks such as image generation is an important future direction. The main goal of CtrlSynth is to demonstrate the effectiveness and controllability of diverse text-image synthesis across different settings, including image-text datasets and vision longtail datasets.\nThat said, while we believe that discriminative tasks are an important domain in themselves, our data synthesis approach is not limited to these task types. Users can use the synthetic data for both understanding and generation tasks. Due to the resources budget limitation and the scope of this work, we leave exploring CtrlSynth data for training LLMs or generative multimodal models for future work.", "claims_and_evidence": "It is unclear why the re-synthesized data from existing images helps address the long-tail problem. Additionally, rather than improving a well-trained model (e.g., fine-tuning or parameter efficient tuning), the approach involves pretraining from scratch to demonstrate the dataset's effectiveness. I don't find this to be a practical solution or reason to use a synthesized dataset.", "methods_and_evaluation_criteria": "Clear instructions on how to use the image controller are needed.", "theoretical_claims": "I cannot find theoretical claims in this paper.", "experimental_designs_or_analyses": "The tasks are overly focused on discrimination-based learning. Given that such LLMs and large VL models were used to generate the data, I believe it is important to also evaluate generative models, such as image generation and long-text captioning, among others.", "relation_to_broader_scientific_literature": "One of the key reasons for generating new or additional data is to address the scale-up challenge. However, with a model size as small as ViT-B/16 in CLIP, it is necessary to verify whether this dataset can effectively solve such practical issues in other multimodal models.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "This paper presents a dataset generation framework with various pathways for generating both images and text.\nAdditionally, the paper claims that decomposing visual tags is a main contribution. \nHowever, a comparison is needed to demonstrate the advantages of using fine-grained tags rather than sentence forms.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "My main concern is that the primary experiments were conducted only with training on CLIP from scratch and the ViT-B size. I am curious about the effects of scaling up, fine-tuning, and how this approach performs on generation tasks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HKvYwiUJJL", "reviewer": "Reviewer_vHvx", "summary": "The paper introduces CtrlSynth, a closed loop framework to generate synthetic data in both text and images. The core idea of the work is to decompose an image into granular components (objects and relationships) and re-compose them based on user-specified controls. This is facilitated through the use of foundational models such as an image tagging model, image and text generation model(s).  Through this setup, CtrlSynth is able to create synthetic data with diverse \"synthesis\" paths, which enables them to create various forms of multi-modal data. With extensive experiments on different vision and vision-language tasks CtrlSynth substantially improves zero-shot classification, image-text retrieval, and compositional reasoning performance of CLIP models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the effectiveness of our method in the current setting. \n\n>The method lacks any comparison to other relevant tasks, such as text to image generation or text-based image editing. Firstly, since CtrlSynth can generate both images and text as part of its pipeline, comparing {real} images/captions vs {synthetic} images/captions should be made. Furthermore, since the paper claims to be able to perform user-based edits to images, there is no quantitative evidence that supports this claim. This makes me believe that such a set-up only works on CLIP-like models and might not be scalable to other models such as T2I and VLMs.\n\n**Response 1**: \nWe appreciate this thoughtful feedback. To clarify, CtrlSynth's primary contribution is not developing a better text-to-image model, but rather leveraging existing text-to-image models to generate diverse, controllable training data. The user-based control we describe refers to specifying desired attributes for synthetic data generation, not proposing novel image-editing techniques.\nOur pipeline is intentionally designed to be modular, allowing easy integration of any advanced text-to-image or image-editing methods as they become available. This flexibility ensures users maintain fine-grained control over synthetic sample characteristics while benefiting from improvements in generative technology.\nRegarding scalability beyond CLIP models, while our current evaluations focus on vision-language representation learning, the synthetic data generated by CtrlSynth is model-agnostic. We have preliminary explorations suggesting potential benefits for T2I and VLMs, though comprehensive evaluation across these model families would require significant additional resources and falls outside our current scope. We will add this limitation and future direction to the discussion section.\n> Since there are 3 foundational models used in this work, and are essentially treated as black-box, they could a) each have their own modes of failures and b) since their outputs depend on each other, there is the trivial case of compounding errors. Any analysis on the above will help back-up the papers' claim. There is no analysis on the correctness of these fine-grained concepts (i.e., how good is the vision tagging model)\n\n**Response 2**: \nThis is an excellent point about potential error propagation. Our methodology deliberately employs a redundancy-based approach where imperfections in individual components don't critically impact the overall system performance. In practice, we found that even when specific visual tags are missed or text is occasionally hallucinated, the aggregate diversity and quality of the synthetic data remains beneficial for downstream tasks. We conducted additional quality assessments of our Visual Tagging Model, finding 92% precision on a manually annotated subset of 50 images. More importantly, our ablation studies in Section 4.4 empirically demonstrate that the end-to-end system produces data that significantly improves model performance, suggesting that any noise introduced by component imperfections is outweighed by the benefits of the diverse synthetic samples. We will add a brief error analysis section to address these concerns directly.\n\n>There are some other works that the authors failed to mention/discuss.\n\n**Response 3**: Thank you for identifying these oversight gaps in our literature review. We will expand our related work section to include the additional papers you referenced and discuss how our approach relates to and differs from these contributions.", "claims_and_evidence": "1. I am convinced that this method works in generating synthetic (image-text) pairs for CLIP-like models. The authors perform comprehensive experiments and ablations to support this claim across multiple datasets and tasks.\n\n2. I have 1 major concern : \ni) The paper performs all experiments on classification and retrieval tasks, by fine-tuning CLIP on their data and comparing to it baseline models. However, the method lacks any comparison to other relevant tasks such as text to image generation or text-based image editing. Firstly, since CtrlSynth can generate both images and text as part of its pipeline, comparing {real} images/captions vs {synthetic} images/captions should be made. Furthermore, since the paper claims to be able to perform user-based edits to images, there is no quantitative evidence that supports this claim. This makes me believe that such a set-up only works on CLIP like models, and might not be scalable to other models such as T2I and VLM's.\n\n3. The paper could be improved a lot with an error analysis which could help explain the importance of the individual foundation models used in the paper. Since, there are 3 foundational models used in this work, and are esssentially treated as black-box, they could a) each have their own modes of failures and b) since their outputs depend on each other, there is the trivial case of compounding errors. Any analysis on the above will help back-up the papers' claim.", "methods_and_evaluation_criteria": "For the domain of CLIP-like models, the authors perform comprehensive experiments across multiple benchmarks.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "I do not have concerns with the experiment(s) performed in the paper. The design and the ablations provided are sound and just.", "relation_to_broader_scientific_literature": "Synthetic data for CLIP has been largely studied in the last couple of years. The core contribution of this work is developing synthetic data using fine-grained concepts. However, since there is no analysis on the correctness of these fine-grained concepts (i.e. how good is the vision tagging model), it is hard to pin-point the exact gains achieved because of it.", "essential_references_not_discussed": "There are some other works that the authors failed to mention/discuss : \n\n1. https://arxiv.org/pdf/2407.05600\n2. https://arxiv.org/pdf/2402.01832\n3. https://arxiv.org/pdf/2410.08211", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please check above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ei8yBmS4Cu", "reviewer": "Reviewer_xCh4", "summary": "The paper introduces CtrlSynth, a controllable image-text synthesis framework designed to enhance data efficiency and address challenges in training robust vision-language models. By decomposing visual semantics into modular elements (objects, attributes, relations) and enabling fine-grained control over synthetic data generation, CtrlSynth generates high-quality, diverse multimodal samples. It outperforms baselines across 31 datasets, showing significant improvements in zero-shot classification, compositional reasoning, and long-tail task performance.\n\n- **Fine-Grained Control via Modular Visual Tags** Breaks down visual semantics into objects, attributes, and relations, allowing precise manipulation of synthetic data (e.g., augmenting underrepresented classes or mitigating biases). Combines hybrid visual tag extraction (captioning + multi-label classification) to improve robustness, unlike prior domain-specific methods.\n\n- **Closed-Loop Synthesis Without Additional Training** Leverages pre-trained models (e.g., Mistral-NeMo for text, SDXL for images) in a plug-and-play pipeline, avoiding costly retraining. Filters low-quality outputs automatically, ensuring data quality.\n\n- **Data Efficiency and Versatility** Achieves comparable performance with 40% fewer training iterations than baselines (Table 2, Figure 5).\nOutperforms on long-tail and robustness benchmarks (ImageNet-R/A/O) and compositional tasks (SugarCrepe).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We clarify the filtering threshold and computation costs below:\n\n>The paper does not clarify whether the “label existence ratio threshold” (used for filtering visual tags) generalizes across datasets. Experiments focus on common benchmarks (e.g., ImageNet, COCO), but domain-specific tasks might require manual threshold adjustments.\n\n**Response 1**: We empirically validate the generalizability of our filtering threshold across datasets in Figure 6 (Appendix A.5). Our ablation study demonstrates that a consistent threshold value (20%) works effectively across all tested datasets without requiring domain-specific adjustments. While fine-tuning thresholds for specific domains might yield marginal improvements, our experiments show that values between 10-20% provide robust performance across diverse visual domains with minimal manual intervention. \n\n> For example: Training with SDXL (3.5B parameters) and Mistral-NeMo demands significant GPU resources (e.g., 8–32 A100 GPUs, Table 8). Repeated generation of images/texts through sequential LLM and diffusion steps scales compute costs, though no direct comparison to alternative methods is provided.\n\n**Response 2**: We do not train the SDXL and Mistral-NeMo models. Our method is training-free. The computational cost primarily scales with the number of synthetic samples needed. This is particularly efficient for long-tail tasks, where generating targeted synthetic samples for underrepresented classes yields substantial performance gains with minimal computational overhead compared to collecting and annotating real examples.", "claims_and_evidence": "**Limitations**\n- **Adaptability of Preset Label Thresholds** The paper does not clarify whether the “label existence ratio threshold” (used for filtering visual tags) generalizes across datasets. Experiments focus on common benchmarks (e.g., ImageNet, COCO), but domain-specific tasks might require manual threshold adjustments.\n\n- **High Resource Consumption** The pipeline relies on multiple heavy pre-trained models (e.g., LLMs, diffusion models). For example:\nTraining with SDXL (3.5B parameters) and Mistral-NeMo demands significant GPU resources (e.g., 8–32 A100 GPUs, Table 8).\nRepeated generation of images/texts through sequential LLM and diffusion steps scales compute costs, though no direct comparison to alternative methods is provided.", "methods_and_evaluation_criteria": "SEE Claims And Evidence", "theoretical_claims": "SEE Claims And Evidence", "experimental_designs_or_analyses": "SEE Claims And Evidence", "relation_to_broader_scientific_literature": "SEE Summary", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "SEE Claims And Evidence", "other_comments_or_suggestions": "SEE Claims And Evidence", "questions_for_authors": "SEE Claims And Evidence", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eir594PTwt", "reviewer": "Reviewer_n47i", "summary": "This paper proposes CtrlSynth, a image-text synthesis pipeline designed for efficient and robust multimodal learning. Specifically, CtrlSynth decomposes an image's visual semantics into basic elements and recompose them to generate images or texts. With these synthetic data, the performance of CLIP-based model improves on zero-shot classification, image-text retrieval, and compositional reasoning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting that our experiments are comprehensive. We have added a detailed explanation below:\n\n> Did the author try using both CtrlSynth-mix and original image-text pairs?\n\n**Response 1**: Yes, all our reported results for CtrlSynth-mix include both synthetic and original image-text pairs during training. This combined approach consistently yielded the best performance across our experiments. \nWe did not explore pre-training exclusively on synthetic data before fine-tuning on original data, as our focus was evaluating the direct impact of our synthetic data when integrated with standard training procedures. We agree that exploring more sophisticated training strategies (such as curriculum learning with different mixing ratios at various training stages) represents a promising direction for future work that could potentially further enhance the benefits of our synthetic data.\n\n>The image/text controller is somehow unclear how it affect the quality of the synthetic data. Are them useful? There may be a lack of an ablation experiment w/wo image/text controller.\n\n**Response 2**: We provide an ablation study in Table 6 that directly addresses this question by evaluating different controller configurations. The results clearly demonstrate that both controllers significantly contribute to data quality and downstream performance. Specifically, CtrlSynth-cap (which lacks image control) and CtrlSynth-image (which lacks text control) both underperform compared to the full CtrlSynth model with both controllers enabled.  \n\n>Has the author tried to use the original image as a constraint when synthesizing the image? This will keep the synthesized image from deviating too much from the original image and make the synthesized image more realistic.\n\n**Response 3**: This is an excellent suggestion that aligns well with CtrlSynth's modular design philosophy. While we did not implement this specific constraint in the current work, our framework is explicitly designed to accommodate such extensions. Using original images as additional conditioning signals could indeed help preserve certain visual characteristics while introducing targeted variations. Our current implementation demonstrates four distinct synthesis paths to showcase the framework's versatility, but the architecture readily supports incorporating image-anchored generation as suggested. \n\n\n>Did the author try to get more accurate tags obtained from Florence-large + Qwen2-7B-Instruct. For example, for tag1, ask MLLM to assess whether this tag exists in the image. In this way, more accurate tags can be obtained. I am not sure if more accurate tags will help in generating higher quality synthetic data.\n\n**Response 4**: While we did not implement this specific verification loop in our current pipeline, it represents a valuable extension that aligns with our modular design. Our experiments indicate that the current tagging approach achieves sufficient accuracy to significantly improve downstream task performance. Moreover, CtrlSynth's strength lies partly in its ability to generate diverse variations even from imperfect tags. The controllers also apply filtering policies that remove low-confidence tags. Our framework is designed to be component-agnostic, allowing straightforward integration of improved tagging models or verification mechanisms as they become available, without requiring architectural changes to the overall pipeline.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no theoretical proofs in this paper.", "experimental_designs_or_analyses": "Yes, the experimental designs or analyses are reasonable.", "relation_to_broader_scientific_literature": "The key contribution of this paper is a new pipeline to synthesize high-quality pre-training data for multimodal learning, which is related to topic - data augmentation.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1. The experiment is comprehensive, however, did the author try using both CtrlSynth-mix and original image-text pairs? In this way, the CtrlSynth-mix serves as a data augmentation of image-text pairs. There are two ways: (1) Mix CtrlSynth-mix and original image-text pairs for training. (2) Pre-train on noisy image-text pairs and fine-tune on high-quality synthetic data.\n\n2. The image/text controller is somehow unclear how it affect the quality of the synthetic data. Are them useful? There may be a lack of an ablation experiment w/wo image/text controller.", "other_comments_or_suggestions": "In order to obtain more confident conclusions, I suggest also including top-3, top-5 Accuray and Reall@3, Recall@5 in the Table. If there is not enough space, at least put it in the appendix.", "questions_for_authors": "1. Has the author tried to use the original image as a constraint when synthesizing the image? This will keep the synthesized image from deviating too much from the original image and make the synthesized image more realistic.\n\n2. Did the author try to get more accurate tags obtained from Florence-large + Qwen2-7B-Instruct. For example, for tag1, ask MLLM to assess whether this tag exists in the image. In this way, more accurate tags can be obtained. I am not sure if more accurate tags will help in generating higher quality synthetic data.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents CtrlSynth, a modular, closed-loop pipeline for controllable multimodal data synthesis. By decomposing image semantics into visual tags and enabling user-specified recomposition, it produces diverse image-text pairs that improve downstream vision-language tasks. All reviewers leaned positive (all weak accepts), highlighting strong empirical results across 31 datasets the authors did, solid improvements in long-tail tasks, and a well-structured framework/writing. AC also appreciate that the paper is comprehensive, and the experimental setup is extensive.

Review concerns focus on limited evaluation beyond CLIP-like models, somewhat lack of quantitative error analysis for component modules, and concerns over computational cost. The authors respond diligently, offering ablations, controller impact studies, and clarification on scaling and flexibility. While broader generative evaluation would improve impact, the paper’s contribution to controllable synthetic data generation is clear. AC recommends clear accept.
Decision: Accept (poster)