Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning
Authors: Amit Peleg
Naman Deep Singh
Matthias Hein
Venue: NeurIPS 2025 poster ()
Keywords: Multi-modal learning
Compositionality
CLIP
Primary area: deep_learning
TL;DR: A fine-tuning method for Compositionally-aware Learning in CLIP

Abstract: Vision-language models like CLIP have demonstrated remarkable zero-shot capabilities in classification and retrieval.
However, these models often struggle with compositional reasoning - the ability to understand the relationships between concepts. A recent benchmark, SugarCrepe++, reveals that previous 
works on improving compositionality have mainly improved lexical sensitivity but neglected semantic understanding. In addition, downstream retrieval performance often deteriorates, although one would expect that improving compositionality should enhance retrieval. In this work, we introduce CLIC (Compositionally-aware Learning in CLIP), a fine-tuning 
method based on a novel training technique combining multiple images and their associated captions. CLIC improves compositionality across architectures as well as differently pre-trained CLIP models, both in terms of lexical and semantic understanding, and achieves consistent gains in retrieval performance. This even applies to the recent CLIPS, which achieves SOTA retrieval performance. Nevertheless, the short fine-tuning with CLIC leads to an improvement in retrieval and to the best compositional CLIP model on SugarCrepe++.
All our models and code are available at https://clic-compositional-clip.github.io.

Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning
Amit Peleg"
Naman Deep Singh"
Matthias Hein
Tübingen Al Center, University of Tübingen
Abstract
Vision-language models like CLIP have demonstrated remarkable zero-shot capabil- ities in classification and retrieval. However, these models often struggle with com- positional reasoning the ability to understand the relationships between concepts. A recent benchmark, SugarCrepe++ [11], reveals that previous works on improving compositionality have mainly improved lexical sensitivity but neglected semantic understanding. In addition, downstream retrieval performance often deteriorates, al- though one would expect that improving compositionality should enhance retrieval. this work, we introduce CLIC (Compositionalll-awe Learning in CLIP), a fine-tuning method based novel training technique combining multiple images and their associated captions. CLIC improves compositionality across architectures well as differently pre-trained CLIP models, both in terms of lexical and se- mantic understanding. and achieves consistent gains in retrieval performance. This even applies the recent CLIPS [33], which achieves SOTA retrieval performance. Nevertheless, the short fine-tuning with CLIC leads to an improvement in retrieval and to the best compositional CLIP model on SugarCrepe+++, All our models and code are available at https: http://cccccccciiiiicccccccccccl
compos iimpposstiin titatoooollloooo cllip. ip. hap.-itthu thub. io.
Introduction
Recent advances combining multiple modalities via large models have propelled us toward sys- tems with greater capabilities. In the vision-language domain, models like CLIP [45], BLIP [24], FLAAVA [24] and ALIGN [20] have been particularly transformative, by training on image-text paired datasets. These Vision-Language models (VLMs) showcase not only enhanced performance on native tasks like retrieval, captioning, etc., but also enable remarkable zero-shot capabilities image classification [26, 331, segmentation [59]. human-level perception of images [13], and similar other tasks. This suggests that CLIP like models are good associating individual concepts with images. However, several works have shown [56, 52, 11] that CLIP models struggle in understanding how concepts combine form complex meanings. i.e., compositionality.
As initially highlighted in [56], these models tend to learn a "bag of words" representation of the multimodal data, making them incapable of solving simple compositional tests. For instance, when given an image of "a man a red shirt next to gray horse", CLIP models have been shown to choose "a man in a gray shirt next red horse" the more similar of the captions (see [11] for more such examples). To this end, several new benchmarks (WinoGround [52], VALSE [39], Crepe [36], SugarCrepe [18], SugarCrepe++ [11]) have been created
compositionality in VLMs.
While SugarCrepe [18] mainly tests whether a model can distinguish between lexically similar but semantically different texts (P1: "The dog is chasing car" vs. N: "The cat is chasing dog"). it has been pointed out in SugarCrepe++ [11] that this is not sufficient for true understanding. Given an image of dog chasing cat, models should attach higher similarity of the image to both P
Equal contribution. Correspondence: amit peleg @ uni-tttteenn tuebingen.de
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and semantically equivalent but lexically dissimilar text P3 ("The cat is being chased by dog") than the wrong text, N. Surprisingly, it turns out that improving compositionality on Sugar- Crepe++ is non-trivial. As can be seen in Table 1, previous works perform very well on SugarCrepe (e.g., DAC [9], but their performance on SugarCrepe*+ is even worse than the pre-trained model. Since capturing semantic similarity is crucial for the downstream performance of CLIP models, in particular retrieval, we focus the evaluation of compositionality with SugarCrepe++.
DAC-SAM
TSVLC-R
In this paper, we introduce Compositionally- aware Learning in CLIP (CLIC). This tech- nique fine-tunes CLIP models by leveraging al- ready available high-quality captioned datasets like PixelProse [50] or common text-image datasets like Laion [46], which we recaption using CLM [53]. Using novel technique of combining images and captions allows us to generate positives as well as hard-negatives with minimal additional overhead, e.g., we do not require LLM for generating hard-nec negativvss [9]. nor do we need to generate synthetic images [41]. In Figure 1, we show that CLIC consis- tently improves both on SugarCrepe++ and on image-to-text and text-to-image retrieval bench- marks compared to the pre-trained CLIP-ViT ViT- B/32 model [45], whereas existing methods do not achieve consistent gains. To summarize, our contributions are as follows.
Image-Rod
CON-CLIP
Text Ret.
SC++ (Swap)
TripletCLIP
(Replace)
CLIC(Ours)
-18
-15
-12
-9
46
33
12
CLIP (Better
))
Figure 1: Performance of fine-tuning techniques for improving compositionality compared to the pre-trained CLIP ViT-B/32-moder Previous tech- niques neither yield consistent improvements on SugarCrepe++ (SC++) [11] nor for the retrieval tasks (R 5 of COCO). CLIC is the only method which shows enhanced compositionality and re- trieval performance.
1. We propose efficient fine-tuning method, CLIC, with little overhead, which improves composi- tionality and retrieval performance pre-trained CLIP models across different sizes (ViT-B/32, ViT-B/16 and ViT-L/14) and pre-training approaches (CLIP [45], CLIPA [26], CLIPS [33]).
We use CLIC to improve CLIPS [33], a recent version of CLIP trained to enhance retrieval perfor- mance. Our fine-tuned model, CLIPS CLIC, yields SOTA numbers for CLIP-like models, with a +9% average improvement on Image-to- (ITT) set of SugarCrepe++ and even improving on their already SOTA results in text- and image-retrieval, (+133%/2..%%)
3. Finally, we show that fine-tuning the large vision language model in LLaVA-1.5-7b [32] with CLIC vision encoder enhances it's compositional ability as measured by VQAScore [30] while maintaining its ability on tasks like question answering, captioning and chain-of of-thought reasoning.
2 Methods and benchmarks for improving compositionality
Recent studies VLMs have revealed significant limitations in their compositional abilities [56, 18, 52, 36, 42] via specially designed benchmarks. Throughout these works, several types of image-text models have been shown lack compositional reasoning, ranging from those using just contrastive loss (CLIP [45], LaCLIP [12], SigLiP (57]) to the ones using different losses or architectures like CoCa [55], FLAVA [48], CLIPA [32], BLIP [16]. In this section, we introduce our notation for standard contrastive learning and provide details on methods to improve compositionality and the associated benchmarks.
2.1 Contrastive learning and baselines
The contrastive loss the backbone for training VLMs like CLIP. contrastive training, image-text paired data (x,, y:) is embedded into a joint embedding space via the text and vision encoders. Formally, let v²(-) and d(-) define the normalized embeddings of the vision- and text-encoder of the CLIP model. Then, given batch of image-caption pairs, the model is trained using an average of image-to-test text and text-to-image contrastive losses,
1
m
exp((1(Z;;) (()))) + log exp((u(zi)) (()))) 2==1 exp((((<;)) ((y)))) Σn (xp((1((z;)) o(v))))
Lcont
log m
(1)
2m

---

Py: man wearing black hat. water feature in garden.
Py: water feature in garden. man wearing a black hat.
Phhh box has label on the top. firThe image contains a watermark.
Bie water flows over series of stones, Rrehh man is pointing to the box.
FlAA man wearing a black hat. (ee box label top. PiThe man pointing to the box.
MM water Teature in garden. (9 image contains (W water flows over a series of stones.
Positive captions n: Hard negative captions
m:: (1AA man water wearing black hat. feature garden.
Figure 2: Data generation scheme for CLIC. For every image, sample additional image and concatenate the two. This concatenated image is the input to the model alongside five captions: Pı, concatenation of the first sentence from each image. P2 is sentence-shuffled version of P1- Ps and are concatenations of two additional sentences from each caption, and is a hard negative constructed by swapping one word from each sentence of pt.
Since training batches rarely include hard negative samples that challenge the "bag of words" representation, previous methods add hard negatives by slightly modifying captions to change their meaning, encouraging the model to recognize compositional cues. Denoting by y the hard- negatieee augmentation of the caption y/ methods such as NegCLIP [56] and TripletCLIP [41] adapt the image-to-text loss as follows,
1
exp((v((z;)) d(y:)))
Nee
log
2m
ES= exp((((()) o(;;))) + Σド= (1x,),
This loss is different from the image-to-text loss used in SVLC [10] and DAC [9], which introduce an additional loss term to the CLIP loss in Eq. (1), denoted as the Single Negative (S-Neg) loss,
m
exp((u(z)).
Cs.Nog
Σ log
exp((e((x)) ((x))) + exp ((v(a;), ((v))))
77
Compositionality enhancing methods
Using hard negatives to enhance compositionality in VLMs was first introduced in NegCLIP [56]. which employs spaCy [17] swap words and phrases within sentence. Similarly, SVLC [10] generates hard-negatives with two configurations: (i) SVLC-R, which uses a rule based generation
with a lookup table and (ii) SVLCC-+I, which additionally uses an L.LM based masked input completion, where masking is done for specific concepts using spaCy [17].
Dense and Aligned Captions (DAC) [9] uses the same method to create hard negatives, but first generates higher quality captions by passing the image through BLIP-2 [25]. In DAC-LLM, the enhanced captions are used input GPT-Neo-2.7B [14] for detailed and dense captions, and in DAC-SAM, segments of the original image, created using SAM [21], are captioned via BLIP-2.
The model is then fine-tuned using a combination of Lcont (Eq. (1)), Ls-Neg (Eq. (3)) and MIL-loss adapted from [38], which uses multiple negatives for each image.
TripletCLIP [41] uses rewritten captions from LaCLIP [12] and utilizes L.MM to generate hard- negatives by changing concepts such as objects, attributes, and relationships that change the meaning of the original caption while slightly distorting it. Afterward, synthetic image is generated for the hard-negative captions using a text-to-image diffusion model. Finally, training is done by using sum of image-text and text-image based LNog from Eq. (2).
2.3 Compositionality benchmarks
Several benchmarks have been proposed measure the lack of compositionality in VLMss These benchmarks usually add negative caption (semantically different) and measure cross-modal Image-

---

to-Text (ITT) similarities across different captions. Among them is WinoGround [52], which uses two images and two corresponding captions that differ only in word order. The task of the model is to assign to each image the corresponding text and vice versa. However, it has been shown in [8] that models might fail on this benchmark for reasons other than compositionality, such as the ability to locate small and out-of-focu objects. SugarCrepe [18] measures image to text similarity and is built on the MS- ASSCCOO validation set. For each image, it requires the model to identify the correct caption from pair of lexically similar, but semantically different captions (Pi and N).
Recently, SugarCrepe++ [11] extended this idea by introducing P2, an additional positive caption that is semantically similar to Pi, but lexically different from both P1 and N. We note that SugarCrepe++ uses the additional positive caption P2, for the same images, original positive captions, and plausible negative captions from SugarCrepe. To answer correctly, the model needs to assign a higher score to Pi and P3 compared to the score assigned to N. This makes SugarCrepe++ harder and more meaningful than SugarCrepe, as the model can no longer rely solely on lexical differences but has to instead understand the semantic differences between the captions. Therefore, we predominantly focus on SugarCrepe++ in this work.
On top of ITT. SugarCrepe++ also measures the uni-modal Text-to-Text (TOT) scores between the original, the semantically different, and the lexically different captions. The TOT is interesting for measuring the compositionality of the text encoder, but the ITT is arguably more important for the cross-modal downstream tasks for which CLIP models are used. Nonetheless. we report TOT for all experiments conducted in this work. Furthermore, we would also like remark here, that many methods like [56, 49] train on MS-COCO, which may lead to knowledge leakage, as the captions and images in benchmarks like SugarCrepe and SugarCrepe++ come from the same data source.
3 Compositionally-awar learning in CLIP
Large-scale web-crawled datasets are known to be noisy, with many captions not reflecting their corresponding images [27, 3, 59]. This can be detrimental to compositional learning, which relies on
minor modifications that alter semantic meaning. To improve the compositionality. we opt for captions that better align with the images and describe the images in detail. For this, we use approximately 1M sample subsets taken from different data sources, like, RedCaps and CC12M from PixelProse [50], and our Cog VLM [53] recaptioned Laion. Details about the datasets can be found in Appendix B. We train several versions of CLICC using each dataset to demonstrate the generalizability of our approach across data sources. Additionally, since some baselines are trained using MS- MS-COCO, for fair comparison, we train another version on this dataset, the details of which are in Section 4.
3.1 Generation of positives and -ard-negatives: concatenation of images and captions
To overcome the "bag of words" limitation, we aim for the model to distinguish between captions that are lexically similar but semantically different. In order to achieve this, we construct hard negatives that retain the same words in the caption but no longer correctly describe the image. For efficiency and diversity, we create these examples by concatenating pairs of images and swapping words across their two captions, ensuring that the resulting caption does not represent the concatenated images. To help the model learn that different lexical or syntactic sentences can represent the same image, we create multiple positives. For this, we use dense captions that provide varied descriptions of each image. The concatenation enables the number of combinations to grow quadratically with the dataset size, allowing CLIC to generalize better than other methods, see discussion in Appendix B.3. We detail our data preparation strategy based on concatenating random image-caption pairs in the following section. The process is illustrated in Figure 2 and described in Algorithm I in Appendix C. See Appendix B.4 for a discussion on a non-random approach.
During training, in each iteration, we sample a batch of m then randomly select additional m pairs (2,, y1 )mimm and ensure that each selected image Iitm has the same orientation as I; We denote by Concat(:,-) the process of concatenating two images or sentences and by RandomConcat(-> the process of randomly shuffling the order of the two images or sentences before concatenation. We start by concatenating the images in random order to mitigate potential biases and term the new image u, = RandomConcat (zy, m) Then, from the two associated captions y and yitm, we create five new captions for the image u, four positive
pairs, (z,, Vi ) We

---

captions and one negative. As each original caption yi consists of several sentences, in the following. we denote by y the kin sentence of the i caption.
P1: We concatenate the first sentence of each caption and term this caption e = Cy m This choice is based on the observation that the first sentence caption generally provides high-level description of the image, whereas subsequent sentences describe more specific details, as demonstrated in the examples in Appendix B.I.
Pa: We shuffle the order of the two sentences te" Concat(y)+m v.). We do this as both concatenated captions describe the corresponding concatenated image, irrespectively of the order of the sentences; thus, the model should be invariant the order of the sentences.
P3.P4 P4: We then sample (without repetition) two additional sentences from each caption and concatenate them in random order t!" = RandomConcat(s? .u(fm), " and eee RandomConcatyy, Fi+m) for random indices ,,, hi,, k4s This step reinforces the model's ability to distinguish incorrect captions from correct ones, even when the incorrect caption presents a general description of the image while the correct caption describes only a specific part. This different from previous works, such as DAC [9], where multiple positives exist, but each caption faces its own hard-negative. An ablation on the number of positives can be found Table 21.
Next, we leverage the spaCy package [17] generate hard-negatives. We decompose the first two sentences into individual words and identify their respective linguistic categories using spaCy, A table detailing these categories, along with examples, provided in Table in Appendix B, To construct hard-negatives, we randomly select a linguistic category that is common to both sentences and choose two words from that category one from each image. If common category exists, we randomly select word from each image and swap them. Certain categories, such as punctuation, are excluded from this process (see Appendix B.2 for details). To prevent over-fitting to specific benchmarks, we do not impose constraints on the selection of particular categories or word positions. As result, the generated sentence no longer accurately represents the concatenated image unless, by chance, the swapped words share the same meaning.
3.2 Training of CLIC
The concatenated image is fed into the model along with the five captions. We employ a combination of different loss functions, each targeting a specific desired property of the model.
Contrastive loss: The standard CLIP loss (1) extended to handle the four positive captions,
m
(x)(( o(t))) + log ΣS exp((x((,)) ((1
((p((
LCont
log Σn= (xp((pppuuxuu) of o(t))))
(4)
8mm
where t." is the positive caption of an image u, for l E (1,2,3, 4} and E ml.
Hard-negative loss: We compute hard-negative loss between each of the positive captions and the negative caption of the same image, similar to Eq. (3),
1
exp(( o(1"))
Ls Neg
log (xp((((()), o()")) +
4m
o(t))))
where t, is the hard-negative text of u4- We use a separate loss for the hard-negatives to ensure they always influence the training and are not "masked" by other terms in the standard clip loss.
Uni Modal loss: The la-distance between the first positive t.t and its shuffled version e
Lumi 1 Σ - (1
Since the first two positives are simply shuffled versions of each other and the concatenated images are unrelated, there is no inherent difference between the positives (except for biases in the pre-trained model). Thus, we use this loss to teach the model to be invariant to syntactic alterations that do not change the meaning of the texts. This approach differs from the text-text similarity loss used in SVLC, where the loss between two different descriptions of the same image, one of which was generated by language model.
5

---

Table I: Previous methods ITT results SugarCrepe do not generalize to SugarCrepe++. Comparison of methods on SugarCrepe and SugarCrepe++ compositionality benchmarks. While the improvements of previous methods SugarCrepe do not generalize to SugarCrepe*+. all versions of CLIC show strong improvements on SugarCrepe++, and also on WinoGround and SugarCrepe. Best number for each column is in bold, second best underlined. Exact details on fine-tuning data for each method can be found in Appendix D.3.
SugarCrepe++ [11]
WinoGround [52]
SugarCrepe [18]
Replace
Swap
Text
Image
Group
Add
Replace
Swap
Method
TOT
TOT
Score
Score
Score
ITT
ITT
ITT
CLIP [45]
69.5
60.5
45.7
25.9
31.2
11.0
8.7
72.9
80.0
62.7
NegCLIP [56]
70.5
74.1
56.4
44.8
30.2
11.0
8.0
87.6
85.4
76.5
DAC-LLM [9]
53.7
59.6
32.2
18.1
22.5
10.5
4.7
93.7
89.4
74.6
DAC-SAM [9]
52.3
60.2
30.7
18.4
21.2
12.2
7.5
91.5
87.0
73.6
SVLC-R [10]
64.0
69.5
51.4
28.4
25.2
8.5
6.7
85.3
78.9
68.8
SVLC-R+L [10]
61.8
70.0
45.9
26.4
28.0
8.2
5.7
78.4
75.8
65.4
CoN-CLIP [49]
69.0
70.8
48.0
32.5
29.5
10.0
7.2
83.0
79.7
65.3
TripletCLIP [41]
73.5
72.7
43.4
33.2
30.7
11.2
8.2
86.6
88.0
69.6
CLIC-COCO
73.5
79.4
52.9
47.2
31.0
10.0
85.5
83.8
69.3
CLIC-LAION
75.6
61.1
27.9
31.5
11.5
84.5
84.0
73.7
CLIC-CC12M
74.4
62.5
60.6
30.0
31.0
11.8
9.5
89.7
85.3
74.3
CLIC-RedCaps
76.0
57.6
61.5
23.1
32.2
12.2
10.7
86.5
84.8
72.6
The final objective function we use a weighted combination of these three losses:
L = CContCCont + As-Nog Cs-Neg + AuniCuui
(7)
where AConts As-Neg. and AUni are hyperparameters controlling the contribution each loss term. To prevent the model from deviating too much from the pretrained CLIP when training only on concatenated images, we employ the standard clip training every second iteration, using a single image, the first sentence, and the standard clip loss as in Eq. (1).
3.3 Differences between CLIC and other methods
Other approaches for generating hard text negatives differ from ours in computational cost and methodology. In DAC [9] and SVLC [10], hard-negatives tailored explicitly for specific composi- tionality tasks (e.g., replacing specific categories like colors, actions, etc.) are either found through look-up tables or created with computationally expensive LLM-based generation. Similarly, Triplet- CLIP uses an LLM to generate hard-negatives; for details, see Section 2.2. Closest to our work is NegCLIP, which also uses word swapping. However, NegCLIP focuses on specific word classes such as adjectives, adverbs, verbs, and nouns as described in Appendix D. We allow swap of any two random words long as they are from the same class, e.g., prepositions, numbers, and pronouns, as seen in Appendix B.2. Our approach offers two main advantages: cheaper computational cost compared LLLM-based generation and a design that doesn't 't target specific benchmarks, reducing the risk of over-fitting existing benchmarks. Other methods also add hard-negative images, further increasing overall computational complexity. NegCLIP and CoN-CLIP search for hard images to add to the batch, and TripletCLIP generates hard images using diffusion model. In addition, none of the methods use a combination of image pairs and corresponding text descriptions in our method.
Experiments and evaluation
4.1 Experimental setup
Dataset. For CLIC, we fine-tune models either with CogVLM [53] recaptioned Laion images, or PixelProse [50] recaptioned images from RedCaps and CC12M (see Appendix B for details). We stress that this approach keeps training data independent of MS-COCO, which is the basis for

---

Table 2: Downstream retrieval/classifcatiio performance. Most methods degrade zero-shot classfiiaatinnreeriivall performance. CLIC improves retrieval and shows minimal degradation for classification. is not zero-shot MS-COCO Column-best and second-best higlighted.
Fine- Tuning
Classification
Text-Ret.
Image-Ret
Rete-
Method
Data
IMNET
ZS-
COCO
F30k
COCO
F30k
CLIP [45]
63.3
61.4
74.1
95.1
54.6
83.5
NegCLIP"(5))
MS-COCO
60.9
60.7
83.6
96.4
72.2
91.7
DAC-LLM [9]
CC3M
51.1
54.3
63.3
79.3
58.1
88.3
DAC-SAM [9]
CC3M
52.3
53.9
57.3
85.6
58.6
82.2
SVLC-R [10]
CC3M
58,8
58.4
70.4
93.0
61.1
87.0
SVLC-R+L [10]
CC3M
59.7
59.0
68.9
90.4
61.1
87.0
CoN-CLIP"[49]
CC3M.MS-COCO
63.2
61.3
71.4
90.7
55.5
84.5
TripletCLIP [41]
CC3M,CC12M
54.8
52.3
72.3
91.5
56.8
85.1
CLIC-COCO"
MS-COCO
62.7
60.7
82,9
97.1
68.2
90.2
CLIC-LAION
Laion
61.7
61.0
75.9
95.0
60.0
86.7
CLIC-CCI2M
CC12M
62.2
60.6
76.9
95.8
60.8
87.9
CLIC-RedCaps
RedCaps
62.3
60.2
76.0
95.6
59.5
86.3
benchmarks like SugarCrepe. SugarCrepe++, and retrieval evaluations. By that, we ensure CLIC always functions as zero-shot model. However, to fairly compare with the VIT-B/32 model of NegCLIP, which is trained on MS-COCO, we also fine-tune one model on the MS-COCO [29] dataset. In our experiments, denote each model by the respective dataset it was trained on, i.e., CLIC-COCO, CLIC-LAION, CLIC-RedCaps, or CLIC-CC12M.
Models. To be comparable with previous compositionality enhancing methods, we base the majority of our experiments and the ablations on ViT-B/32 models pre-trained by OpenAl [45]. However, to highlight the adaptability and scalability of our approach, also fine-tune other commonly used tre-trained architectures, such as ViT-B/16 and ViT-L/14 from [45]. Up to our knowledge, among methods improving compositionality. only CoN-CLIP [49] provides checkpoints for these architectures. Additionally, we fine-tune more recent VIT-L equivalent VLMs, including CLIPS and CLIPA show that our method can enhance compositionality even in models that already possess compositional abilities and are pre-trained on varying datasets such as Re-DataComp (CLIPS), WIT (CLIP) and DataComp (CLIPA). More model specific details can be found in Appendix D.1.
Training details. all of our experiments for CLIC, we keep the vision encoder frozen and fine-tune only the text encoder at an image resolution of 224 224. An ablation showing that this works better than miee-tnnng the entire model can found in Table 20 in the appendix. Only for experiments associated with LLa VA [32], we fine-tune the complete model at higher resolution of 336 x 336. We defer other training details like LR, loss parameters in Eq. (7), etc, and a discussion on our NegCLIP baselines (denoted everywhere with 1), used whenever checkpoints available Appendix D.
Evaluation. We evaluated the models on the compositionality benchmarks SugarCrepe++ [11], WinoGround [52], and SugarCrepe [18]. Additionally, we assess the effects of fine-tuning downstream tasks such as classification on IMAGENET [6] and on ZS-10, which computes the average score across ten standard classification datasets, detailed in Appendix D.2. In addition, we report image (T + I) and text retrieval (I + T') Recall scores on MS- IS-COOC (Val2017) [29] and Flickr30k [43]. We show Recall @ numbers for retrieval and detailed SugarCrepe and SugarCrepe++ numbers for all the models in Appendix E.
4.2 Comparing to other methods
Given that most publicly available checkpoints of compositionality methods use ViT-B/32 based CLIP, we first focus on this architecture. From Table 1, we observe that most methods perform worse than the pre-trained CLIP on SugarCrepe+++ Notably, only NegCLIP, CON-CLIP, and the CLIC models improve or are on par with the pre-trained baseline. Among these, CLIC-COCO better than NegCLIP and CoN-CLIP (all three were trained on MS-COCO), while CLIC- CLIC-RedCaps
7

---

Table 3: CLIC improves different Large (ViT-L/14) CLIP models. CLIC improves several versions of CLIP, across architectures and pre-training datasets." is not zero-shot for MS-COCO evaluations. WinoGround results for these models are in Table 27. Detailed retrieval, SugarCrepe++, SugarCrepe results and more architectures can be found in Appendix E.
Downstream Evaluations
+
Sugarcrepe
Method
Classification IMNET ZS-
COCO Ret.
Rep.
Swap
Add
Rep.
Swap
+ T T
+
ITT
TOT
ITT
TOT
ITT
ITT
ITT
CLIP [45]
75.5
72.3
79.0
60.0
70.7
58.7
44.7
22.7
74.9
79.5
61.4
NegCLIP
73.9
70.6
77.7
65.9
70.6
63.5
50.5
35.8
83.6
82.8
70.0
CoN-CLIP [49]
75.9
72.4
78.4
63.1
72.2
71.3
46.0
30.5
83.9
81.3
64.2
CLIC-LAION
74.2
71.8
80.1
65,4
79.0
59.1
59.8
26.3
87.7
85,6
71.3
CLIC-RedCaps
75.2
72.1
81.3
63.9
76.0
55.4
55.2
20.0
84.3
83.8
66.1
CLIP-A [26]
79.6
78.3
85.4
70.5
74.8
74.8
45.8
30.8
85.8
82.9
63.8
CLIC-LAION
78.4
78.0
84.8
71.5
78.4
66.3
55.1
30.7
92.9
88.2
71.6
CLIC-RedCaps
78.9
78.3
85.2
71.2
80.2
62.8
61.2
22.8
93.4
85,9
71.5
CLIPS [33]
78.4
76.5
90.4
77.3
79.3
74.2
60.6
51.1
89.4
88.0
79.6
CLIC-LAION
76.7
76.2
90.4
79.4
84.3
71.9
73.0
49.4
96.7
92.4
85.7
CLIC-RedCaps
77.2
76.6
91.7
79.5
84.9
71.4
75.1
50.2
95.3
91.1
84.5
has lower TOT with a higher ITT (10.2% gain on pre-trained model across replace and swap). We also evaluate all models on the WinoGround benchmark, where, apart from CLIC, no other method improves on the pre-trained model, highlighting the effectiveness of CLIC. Moreover, CLIC improves in compositionality regardless of the fine-tuning dataset, showing that CLIC works for different sources of captions and images used for fine-tuning. An interesting finding in Table 1 is that many models perform well ITT in SugarCrepe but struggle on SugarCrepe++ ITT, an extended version including lexically different but semantically similar positives. This stark difference for methods like DAC (achieves 89.4% on SugarCrepe replace ITT but only 53.7% on SugarCrepe++)) suggests these models detect lexical changes rather than truly learning compositional relations. This likely stems from their over reliance on hard- negativvs tuned for previous benchmarks (see Appendix D.4).
Downstream tasks. We evaluate models on standard retrieval benchmarks (MS-COCO, Flickr30k) and classification tasks (IMAGENET, ZS-10) in Table 2. While TripletCLIP performs well SugarCrepe++ and WinoGround, it exhibits significant degradation on IMAGENET and ZS-10. In contrast, our CLIC models show minimal performance degradation on classification tasks, with CLIC- CC12M achieving the highest text retrieval performance among models not trained on MS-COCO while also improving image retrieval over the pre-trained baseline.
4.3 Ablation Study for CLIC
In Table 4 we ablate the components of CLIC in steps (C1-C5) and present the analogous non- concatenated, single image version of CLIC (B1. The CLIC baseline (C1) is fine-tuning on the concatenated images with caption pi and in (C2) we add our single hard negative. These two steps are sufficient to improve on SugarCrepe but not on SugarCrepe++. This suggests that the hard-negative distractor helps the model to detect lexical changes, but not to distinguish between semantically similar sentences and ones with different meaning. Adding multiple positives (C3) and the uni-modal loss (C4) improves significantly on SugarCrepe++ while performance on SugarCrepe slightly reduces. Thus, adding diverse positives and enforcing invariance of semantically identical captions lets the model identify semantically similar captions while being sensitive to lexically similar but semantically dissimilar captions. In the last step (C5) we add the iterates with single images for standard contrastive loss, which further improves compositionality and downstream task performance by eliminating the potential concatenated image biases. The concaancate,ted, single-image variant of CLIC shows similar but smaller improvements in compositionality while being significantly worse in retrieval. Here, the NegCL.Iff variant (similar to B2, but employing LNeg (2) in place of L (3)) was used to compare CLIC to NegCI.IP on a dataset other than MS-COCO, in order to avoid potential test data leakage from MS-COCO during training, discussed in Section 2.3.
signific
8

---

Table 4: Effeets of different components in CLIC. The non-coneat baselines represent single-image versions of CLIC. Rows standard fine-tuning our 1M Laion with either concatenated single image and a single caption. Rows present standard one hard negative training. In the single image case, with either the Neg (2) for NegCLIPi or the Ls-Neg (3) for B2. Finally, rows feature
CLIC (with all the components presented in Section 3) with its analogous single-image version.
Downstream Evaluations
Sugarcrepet+
Sugarcrepe Swap
Method
Classification
COCO Ret.
Rep.
Add
Rep.
Swap ITT
IMNET ZS-10
Text
Image
ITT
TOT
ITT
TOT
ITT
ITT
A1
CLIP [45]
63.3
61.4
74.1
54.6
69.5
60.5
45.7
25.9
72.9
80.0
62.7
NegCLIPt
61.0
60.5
72.0
59.7
67.9
64.7
54.5
36.1
82.4
80.9
70.7
Single image Baselines (Non-Concat)
B1
Fine-tuned
62.1
61.8
76.4
60.4
69.6
69.2
49.8
35.9
83.4
80.7
68.8
+ Single hard-negative
60.6
59.8
70.4
57.9
66.6
63.8
55.0
36.2
81.9
80.5
70.6
B3 + Multi Positives
58.5
59.2
58.2
56.6
70.2
48.5
58.5
23.1
77.7
76.0
64.7
B4 Single-Img Baseline
61.3
60.6
67.3
58.0
74.1
53.9
60.2
27.8
81.0
79.9
69.7
Ablating CLIC (Concat)
Fine-tuned Concat
62.6
61.2
76.5
59.1
69.8
66.7
49.3
35.9
82.7
81.9
68.2
C2
+Siglee hard-negativy
60.9
60.0
73.9
57.8
68.7
63.6
53.7
34.2
84.2
83.8
73.2
+ullii Positives
61.3
59.6
61.3
54.8
74.2
47.4
59.0
19.5
84.2
81.9
70.9
C4 +Uni-Modal Loss
60.9
59.3
59.9
54.5
74.2
50.7
59.1
20.3
83.4
81.6
70.3
C5 +CLIP iterate (CLIC)
61.7
61.0
75.9
60.0
75.6
60.1
61.1
27.9
84.5
84.0
73.7
Table 5: Comparison to SOTA models for SugarCrepe++ and WinoGround. CLIPS fine-tuned with CLIC-RedCaps attains average the strongest compositionally aware model, especially among CLIP variants, as seen both SugarCrepe++ and WinoGround.
SugarCrepe++
WinoGround
Model
Params (M)
Replace
Swap
Average
Score Image
ITT
TOT
ITT
TOT
ITT
TOT
Text
Group
FLAVA [48]
358
74.4
75.0
56.8
51,5
67.3
65.5
32.2
20.5
14.2
BLIP-Base [24]
225
73.8
73.9
54.0
42.1
66.1
61.1
35.8
15.8
13.3
BLIP-Large [24]
470
81.7
59.8
72.9
37.7
13.5
10.5
VIT-H/14 [46]
986
73.8
72.1
48.8
39.1
63.7
58.8
33.5
12.7
10.5
ViT-bigG/14 [46]
2540
76.6
73.2
51.5
40.2
59.7
35.5
15.0
12.0
CLIPS [33]
427
79.3
74.2
60.6
51.1
71.9
64.6
36.2
16.0
12.5
CLIPS CLIC
427
84.9
71.4
75.1
50.2
81.0
62.9
41.7
17.5
15.2
4.4 Generalization to larger architectures and models
Next, we measure the generalization of the method to larger architectures such as ViT-L/14 across models pre-trained on different datasets (CLIPA and CLIPS). Results on ViT-B/16 can be found in Table 24 in the appendix. We fine-tune each of the pre-trained models, CLIP, CLIPA, and CLIPS, on our IM Laion subset and on RedCaps (PixeIProsc) subset using CLIC. As most other compositionality enhancing methods do not explore larger architectures. the only baselines have are CoN -LIP and our NegCLIPi (the details of which can be found in Appendix D). Table 3, we can see that for all types of pre-trained models, CLIC-RedCaps shows consistent improvements the ITT score for SugarCrepe++: +7.4% for CLIP, +9.4% for CLIPA, and +9.2% for CLIPS when averaged across replace and swap. These improvements ViT-B/32 model, suggesting that even for larger architectures, CLIC can attain substantial gains.
similar scale the ones for the
What we find even more remarkable are the improvements for image and text retrieval attained by CLIC. Specifically, for CLIPSS (which has to best of our knowledge SOTA retrieval numbers of CLIP model), CLIC further improves text retrieval by 1.3% and image retrieval by 2.2%

---

MS-COCO. These gains come at tiny fraction ()) of the pre- treiniiinng cost, as we only see 850k (RedCaps) resp. IM (Laion) samples compared to 13B samples seen during pre-training of CLIPS.
Constrasting again SOTA. In Table 5, we pitch our best model in CLIPS + CLIC against the SOTA CLIP like and other models on SugarCrepe++. To the best of our knowledge, CLIPS + CLIC attains the best available ITT numbers on SugarCrepe++ across all architectures. We note that CLIPS in itself already good for ITT on SugarCrepe++ but our short fine-tuning enhances significantly further in terms of compositional understanding. For TOT, among CLIP like models CLIPS + CLIC performs similarly to the best model (CLIPS) in this category. We were unable to evaluate Large for TOT, since the individual BLIP-Large encoders are not publicly available. The WinoGround results for CLIPS CLIC are the best in text and group score, while the image score is the best among CLIP--ike models. As far as we know, CLIPS + CLIC attains SOTA WinoGround scores among CLIP based models.
4.5 CLIC vision encoder in LLaVA
Unlike our previous experiments, we fine-tune the CLIP model with CLIC at 336 x 336 resolution while unfreezing both the vision and text encoders. We then replace the CLIP vision encoder in LLaVA-1,5-7b (Vicuna) [31] with our CLIC version and fine-tune the full LLaVA model following standard projector pre-training (full details in Appendix D.5). As shown Table 6, CLIC enabled LLaVA-1.5 achieves notable compositional reasoning improvements. For SugarCrepet+, we observe gains across most categories with marginal degradation for swap object, while WinoGround scores improve for text and group sets by more than 6% and 3% respectively. Crucially, these gains preserve performance on standard VLM benchmarks (GQA, TexIVQA, SQA-I), with MME (perception) improving from 1441.0 to 1465.3. These results show that even short compositional fine-tuning via CLIC can significantly enhance fine-grained risual-linguise understanding while maintaining general vision-language capabilities, as further illustrated by qualitative examples in Figure 6.
Table 6: Comparing standard LLaVA-1.5-7b (CLIP) with it's CLIC enabled version. Using the VQAScore metric from [30], we evaluate compositionality via SugarCrepe++ (ITT) and WinoGround, while showing results for some benchmarks (VLM-tasks) native to LLa VA. Across compositionality tasks, CLIC enabled LA-11. is almost always better than CLIP based LLaVA while maintaining the same performance nominal VLM tasks. Task-wise best compositional model is highlighted.
SugarCrepe++ (TTT)
WinoGround
VLM-tasks
Model
Replace
Swap
Obj
Att
Rel
Obj
Att
Text
Img
Grp
GQA
TextVQa SQA-I
MME (P)
LLaVA-1.5 (CLIP)
94.3
80.4
73.3
66.9
69.8
36.7 38.2
23.2
60.6
57.1
69.2
1441.0
LLVVA-1.5 (CLIC)
94.3
80.8
73.8 66.5 71.2
43.5 42.0
26.2
62.1
57.7
68.2
1465.3
Conclusion
We introduce CLIC, simple, low-cost, and scalable method for boosting compositionality in vision- language models, as measured by SugarCrepe++. Notably, this improvement comes with a minimal degradation in downstream tasks like zero-shot classification and enhanced text and image retrieval performance. CLIC works across different good caption quality datasets. We further illustrate the generalization of CLIC variedly (dataset/triinng---chee, pre-trained models, architectures. Lastly, we show that CLIC even improves performance on models already exhibiting strong compositional and retrieval abilities like CLIPS and yields SOTA results on SugarCrepe++ as well as image retrieval. From our initial experiments, CLIC powered LLaVA improves in compositional reasoning over the standard version. We believe compositionally aware vision encoders can further help large VLMs like LLaVA on more tasks, detailed study of this is left for future work.
10

---

Acknowledgements
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting AP and NDS. We acknowledge support from the Deutsche Forschungsgemeinschatt
(DFG, German Research Foundation) under Germany's 's Excellence Strategy (EXC number 2064/1, project number 390727645), as well as in the priority program SPP 2298, project number 464101476. We are also thankful for the support of Open Philanthropy and the Center for Safety Compute Cluster. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors. We thank the reviewers for the discussions and suggesting additional experiments.
References
[1] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.7
[2] Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food- 101 mining discriminative components with random forests. In ECCV, 2014. 6
[3] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi- modal models with better captions. In ECCV, 2024. 4
Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in wild. In CVPR, 2014. 6
[5] Gheorghe Comanici. Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626/ 2025. 3
[6] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 7
[7] Jacob Devlin, Ming-Way Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), 2019. 6,7 7
[8] Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald. Why is winoground hard? investigating failures in visuolinguistic compositionalliy. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Prucessing, 2022.
[9] Sivan Doweh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Sircante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and aligned captions (dac) promote compositional reasoning in vl models. In NewrIPS, 2023. 2, 3. 5, 6, 7. 18
[10] Sivan Doveh, Assaf Arbelle, Sivan Harary, Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In CVPR, 2023. 3, 6, 18
[11] Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, and Hassan Sajjad. Sugarcrepe++ dataset: Vision-language model sensitivity to semantic and lexical alterations. In NewrIPS, 2024. 1, 4, 7,8
[12] Lijie Fan, Dilip Krishnan. Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. In NewrIPS, 2023. 3.7
[13] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In NewrIPS, 2023, 1
[14] Leo Gao. Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb datasct of diverse text for language modeling, arXiv preprint ar0iv:2101. 00027, 2020.
[15] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.

---

[16] Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya. Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurlPS, 2020. 2
[17] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, Adriane Boyd, et al. spacy; celtustrial-strnnger natural language processing in python. 2020. 3, 5, 21
[18] Cheng- Yu Hsich, Jicyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. In NeurlPS, 2023. 1, 4, 6, 7,
[19] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 7
[20] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hssunn Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, 2021.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg. Wan-Yes Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2334.022433, 2023. 3
[22] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- Ene-grained categorization. In CVPR Workshops, 2013.
[23] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. 6
[24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip:
Bootstrapping hanuuag-immage pre-training
for unified vision-language understanding and generation. In ICML, 2022. 1,9
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training
with frozen image encoders and large language models. JCML, 2023. 6, 7
[26] Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. NewrlPS, 2023. 1, 8, 15, 17
[27] Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjje Zheng, et al. What if we recaption billions of web images with Ilama-3? arXiv preprint arXiv:2406.8 08478, 2024. 4
[28] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shocybi, and Song Han. Vila: On pro-training for visual language models. In CVPR, 2024. 1
[29] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. ECCV, 2014.
[30] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In ECCV, 2024. 2, 10, 7
[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
10, 1
[32] Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jac Lee, Improved baselines with visual instruction tuning. In CVPR, 2024, 2,7
[33] Yanqing Liu, Xianhang Li, Zeyu Wang, Bingehen Zhao, and Cihang Xie. Clips: An enhanced clip framework for learning with synthetic captions. arXiv preprint arXiv:2411.16828 2024. 1, 1,
15, 17
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
[35] Pan Lu,, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NewrlPS, 2022. 7
[36] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? In CVPR, 2023. 1,2
[37] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aireraft. arXiv preprint 1rx55 2013. 6
12

---

[38] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic. and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020. 3
[39] Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Jacer Calixto, and Albert Gatt. VALSE: A task-independent henchmark for vision and language models centered on linguistic phenomena. In ACI,, 2022.
[40] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar, Cats and dogs. In CVPR, 2012.
[41] Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives, In NeurIPS, 2025. 2, 3, 6, 7, 18
[42] Wujian Peng. Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize diagnose and optimize: Towards fine-grained vision-language understanding. In CVPR, 2024.
[43] Bryan A. Plummer. Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier. and Svetlana Lazebnik. Flickr30k entities: Collecting region-to tegioant--phrase correspondences for richer image-to tonge-ttt-eeeence models. In ICCV, 2015.
[44] PyTorch. Country-211 dataset.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 2, 6, 7, 8, 9, 4, 13, 14, 15, 16, 17, 18
[46] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5h: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022. 2, 9,
[47] Amanprect Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019. 7
[48] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022. 2,
[49] Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. Lear" no" to say" yes" better: Improving vision-language models via negations. In WACV, 2025. 4, 6, 7, 8, 15, 16, 17, 18
[50] Vasu Singla, Kaiyu Yue. Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang. Abhinav Bhatele, Gowthami Somepalli. and Tom Goldstein. From pixels prose: A large dataset of dense image captions. arXiv preprint arXiv:2406.1033 2024. 2, 4. 6, 1, 11
[51] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 2023. 15, 17
[52] Tristan Thrush Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022. 2, 4, 7.8
[53] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. In NeurIPS, 2024. 2, 4,
6, 1, 7, 11
[54] Shukang Yin, Chanyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 1 1(12))wwac403, 2024. 7
[55] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Scyedhosscini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions of Machine Learning Research, 2022. 2
[56] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why models behave like bags-of words, and what to do about it? In ICLR, 2023. 2, 3, 4, 6, 7,
18
13

---

[57] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 2.
[58] Beichen Zhang. Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In ECCV, 2024.
[59] Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language-image pre-training with long captions. In ECCV, 2024. 1,4
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide empirical results in Section 4 and in Appendix E to back up the contributions of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the paper in the limitations section, Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This is an empirical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The algorithm is described Appendix and the hyper-parameters are given in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Code will be released upon acceptance of the paper under MIT license.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/pu
blic/ Sgwidde/Cooeeiiie ssssssssssissss. ssi oooolicyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. cccpubliceecccccccccccccccccccbc wtwp//gg///e/eeeec/ceoccccc.e /aaee//ddduubbii ssi assinnollli inooiicyy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Evaluation is done on standard compositionally benchmarks WinoGround, SugarCrepe, SugarCrepe++ and on standard classification and retrieval benchmarks for CLIP models (IMAGENET, Z0- MS-COCO, Flickr30k). The different parameters are justified in the ablation tables in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars and multiple repetitive experiments are have been added for the baseline experiments of this work. We also demonstrate the generalization of results across different architectures, pertaining strategies, and fine-tuning datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ttttpa://neurppai http://ppp..ppppppc/ttccccccciia cetcddieeeeiiiiienneennenn
Answer: [Yes]
Justification: The paper uses standard datasets, and models (Laion, MS-COCO, RedCaps, CC12M, CLIP, CLIPS, etc.) that are licensed to be used openly for academic purposes. There is no research on human subjects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: The paper deals with method improving compositionality of CLIP models while keeping or even improving standard abilities CLIP models like zero-shot classifica- tion or retrieval tasks. Thus we possible negative social impact of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper finetunes existing models to improve their results. We see no risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
All of the models and the data cc-by-4.0 licenses. We appropriately credit the original authors by citing the corresponding papers and list the models and their checkpoints at Table 22.
Justification:
publicly available under MIT or
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswithcom com datasatss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release new fine-tuned models under the original model licenses available, else under the MIT license with documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We are using spaCy [17] to extract the POS tags of captions as described in Appendix B.2. We use VLM for creating captions, but the same results can be achieved with standard datasets, as corroborated by our experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpr/iees.p./eeeee for what should or should not be described.
/LLM)
21

---

Contents
Appendix A Limitations
Appendix Dataset preparation
Appendix .....lgorithmit details
4. Appendix D ... Experimental details and discussions
5. Appendix Additionna experiments
A
Limitations
Our work considers only fraction of contrastive models, we believe CLIC should work similarly for other models like SigLIP [57], which we were unable to test due to compute and time constraints. Moreover, we were only able to test LLaVA [31], whereas other large VLMs like VILA [28] and CogV VL. [53] might also benefit from CLIC, which we leave for future work.
B Data preparation
In this section, describe the data preparation process in detail. Appendix B.1 showcases randomly selected images and captions from the PixelProse [50] dataset and from the Laion [46] dataset, along with captions generated for these images by CogVLM [53]. This part also presents further details on these datasets. In Appendix B.2, we explain the negatiee generation procedure. In Appendix B.3 we show examples of the limitations of using single image for generating negatives as done in [56]. Appendix B.4 shows ablations regarding other ways of combining images for CLIC.
B.1 Training datasets
In this section, we present our data generation process. To show that CLIC performs well across diverse datasets, we train models from three different sources. We took subset of 1M images from Laion with captions that were generated by CogVLM [53], using the query presented in Figure 3. We also used recently proposed high-quality caption dataset in PixelProse [50], which contains 850k available samples from the RedCaps subset. In order to compare faithfully to our baselines in Table 1 who use CC3M, we also use a 1.2M subset of CC12M that comes from the same data stream CC3M to train ViT-B/32 model. Since our method requires detailed captions, we use the recaptioned version of CC12M from PixelProse. These two datasets come from a different stream of internet data than Laion. A selection of random image-caption pairs from both kinds of datasets provided Figure 4. As can be seen from the examples, the first sentence in each caption usually provides general description of the image, while the other part of the caption often captures more specific details. To clean the data, we removed the starting strings like "This picture depiict//shossddemonstrttee
Then, since the generated captions tend to be longer than those in the standard datasets like Laion and DataComp (the average caption length in our recaptioned dataset is 78 words), we split the captions into individual sentences as done in [33, 59]. This results in an average sentence length of words for captions generated by CogVLM [53]. We retain only images whose captions contain more than sentence, which is more than 99% of the 1M set for Laion. This approach favors captions that provide both general context and specific details about the images. For fair comparison with NegCLIP, also train CLIC with MS-COCO, the details of which in Section 4.
Query: Can you please descrihe this image in long and detailed paragraph? Please keep your deserip- tions factual and terse but complete. DO NOT add any unnecessary speculation about things that are not part of the image. The description should he purely factual, with no subjective speculation.
Figure 3: Query used for re-captioning for the CogVLM model [53].

---

Table 7: List of Universal POS Tags with their Meaning and Examples. Examples are taken from ChatGPT. For further information, refer to
org/ u/ pos//
POS Tag
Meaning
Example Words
ADJ
Adjective
happy, blue, large in, on, under, with quickly, very, well is, was, will, do and, but, or the, a, an, this wow, oh, hey dog, city, John one, two, 100 not, (as in John's) he, she, it, they London, Microsoft
ADP
Adposition (preposition)
ADV
Adverb
AUX
Auxiliary verb
CCONJ
Coordinating conjunction
DET
Determiner
INTJ
Interjection
NOUN
Noun (common & proper)
NUM
Numeral
PART
Particle
PRON
Pronoun
PROPN
Proper noun
PUNCT
Punctuation
!.?
SCONJ
Subordinating conjunction
because, although, if $, %, & run, eat, write jdhfk, asdfg
SYM
Symbol
VERB
Verb
X
Other (foreign words, typos)
B.2 Hard negative creation
Our hard negative creation process utilizes the spaCy package to identify the part-of-speech (POS) of words in sentence. We consider only the roarse-grained POS level, which is universal and not language-specific. The complete list of universal POS tags is provided in Table 7. For each sentence, classify all words, group them by their POS, and determine eligible categories for swapping. valid swap nomination requires at least two words from the same POS category, excluding the following: "AUX", "CCONJ", "DET", "INTJ", "PART", "PUNCT", "SCONJ", "SYM" and "X". Once a valid category is identified, we randomly select it and swap two randomly chosen words within that category. We deliberately did not select specific tags to avoid overfitting current compositionally benchmarks. no category falls into these cases, we swap two words at random.
B.3 In-scene vs across-scene swaps
Creating negatives by swapping words between sentences from different scenes offers several advantages over swapping within single scene:
1. Swapping words within single sentence can lead to negatives due to the changes in word order. For example, the sentence from NegCLIP: "The horse is eating the grass" can become "The grass eating the horse". However, swapping words across sentences from different images greatly increases the diversity of negative examples. Taking the example from the paper (Section 1), concatenating the "The horse is cating the grass" image with an image of dog chasing a cat may lead to the new negative: "The horse is chasing the cat. The dog is cating the grass". Since each image can be paired with each of the other images, this scales the diversity of negatives quadratically and helps the generalization of the method.
Creating negatives from a single scene, like in NegCLIP, may result in a modified sentence that still accurately describes the image. This might happen in many sentences, changing the order of the words does not change the semantic meaning of the sentence. This is different from changing words across scenes, where the concatenated sentence will reflect the concatenated image only if we by chance swap words that have the same meaning (we remind the reader that we do not allow swaps of the same word, i.e., 'man' can not be swapped with 'man'). Here are few examples from the CC12M subset from PixelProse:
"In the top left corner, a person wearing green jacket" switching left and top does not change the meaning.

---

"there is a tuna salad with celery, onion, and mayonnaise" changing onion and celery does not change the meaning.
"Bugs Bunny, Taz, Lola Bunny, and Daffy Duck are standing on either side" - changing Lola and Bugs.
"The Jeep has a black bumper, black wheels" changing bumper and wheels.
"A dining room and living room" - changing dining and living.
"A black and gray backpack" changing gray and black.
B.4 Non-random image concatenation
Combining images at random offers several advantages, such as computational efficiency and reduced risk of overfitting to specific benchmarks. However, examine whether generating more plausible or grammatically correct negatives leads to better results (at the expense of higher computational resources), we construct negatives non-random manner using two strategies:
1. Combining images that share common noun.
2. Swapping words with different semantic meanings and selecting images accordingly.
Common nouns
We begin by extracting all the nouns from the first sentence of each image using spaCy. For each image, we then sample up to five other images (without repetition) that share a common noun and use this to concatenate the images. We then continue usual by swapping random words (excluding the common noun). Although we witnessed a higher Sneg loss (Eq. (5)). indicating that the hard negatives indeed harder, the results the downstream tasks remained largely unchanged (see Table 8, row SAME NOUN).
LLM swaps
To test the impact of -on-mennig or ungrammatical negatives, we made the following modification our method construct negatives that are grammatically correct by leveraging LLM based swaps:
1. We extracted the fine-grate fine-grained Part-of-Speech (POS) tags for each word in the first sentence using spaCy's 's most computationally expensive model, which took around 5 hours. In con- trast, the version used for the random image-caption pairs employed more computationally efficient model and took only few minutes.
2. For each word, we queried Gemini 2.5-flash [5] to generate up to five replacement candidates from the same POS category, such that replacing the word changes the meaning of the caption, while keeping the sentence coherent. Note: these queries to Gemini via API calls took over 24 hours of compute time to find word replacements for the 1M Laion subset. In comparison, fine-tuning a CLIP model with CLIC only takes around 40 minutes.
3. During training, for each caption, we randomly selected a POS category and then randomly chose a word from that category. The second image and word replacement were selected based on the candidates from step 2.
This significantly more expensive variant of CLIC (LLM SWAPS Table 8) results harder negatives (as evidenced by higher loss during training) and leads to improvements in Replace (ITT) for SugarCrepe++ but degrades Swap (ITT) and TOT. While at first surprising, we hypothesize that the lower diversity of negatives compared to our original computationally cheap version of CLIC, with random replacement of words from the same POS category, could be a reason for this.
To check the quality of the negatives, we hand- 75 negative samples from both LLM SWAPS and the random negatives used to train CLIC-LAION. This process yielded 69% meaningful, grammatically correct negatives with distinct semantic meanings from LLM SWAPS, compared to 28% from the CLIC-LAION. We include below coherent and grammatically correct hard negatives examples, and 5 nonsensical or grammatically incorrect ones. While there are some errors, the overall quality is sufficient to prevent the model from relying solely on shortcuts, as many examples remain meaningful and grammatically correct, and the errors relatively subtle.

---

Conrect hard negative examples:
(postcard telegram)
N: a collage of various telegram marketing materials. vintage posteard.
(stripes + swirls)
N: a black satin jacket with white swirls on the collar and cuff'. graphic design element that features a large red heart at the center surrounded by intricate golden stripes and patterns
(short ++ thick)
N: portrait of woman with thick textured hair that has mix of brown and blonde colors. indoor setting possibly a café or a bakery with a man pouring a short brown liquid which appears to be chocolate onto white countertop.
(plain + fancy)
N: a woman standing against fancy background. a scrapbooking paper pack titled 'sunny memories' by plain pants designs.
(corners + hearts)
N: pair of red envelopes with intricate gold embroidery on the hearts. two wooden objects that are shaped like corners.
Nonsensical or grammatically incorrect samples:
(whistle ++ shout)
N: red kettle with shout attached to its spout. a man in dramatic pose seemingly in the middle of punch or whistle
(made + destroyed)
N: a hand-destroyed card with a floral design. a scene of destruction with building that appears to have been damaged or made.
(printed ++ removed)
N: a black bag with white lines and the word 'thule' removed on it. one of the nuts is shown with its cap printed revealing the coiled thread inside.
(setting breaking)
N: this image captures a moment in an outdoor park breaking where two individuals are engaged in playful activity. bear's face that appears to be setting through shattered glass surface.
(appears ++ disappears)
N: it disappears to be motorized treadmill with a digital display on the top. the headline reads "german billionaire appears on matterhorn' and is dated april 11 2018 at 11:35 am edt.
Table 8: Comparing with non-random image concatenation. All methods use LAION data captioned by CLgVLM and the ViT-B/32 architecture. Although training with non-random image concatenation leads to higher Ls,Neg loss (Eq. (5)), the performance of the model remains the same.
Downstream Evaluations
SugarCrepe++
SugarCrepe
Classification
Text Ret.
Image Ret.
Replace
Swap
Add
Rep.
Swap
Method
IMNET ZS-10
COCO F30k
COCO
F30k
ITT
TOT
ITT
TOT
ITT
ITT
ITT
CLIP [45]
63.3
61.4
74.1
95.1
54.6
83.5
69.5
60.5
45.7
25.9
72.9
80.0
62.7
CLIC-LAION
61.7
61.0
75.9
95.0
60.0
86.7
75.6
60.1
61.1
27.9
84.5
84.0
73.7
SAME NOUN
61.7
60.7
75.9
95.0
60.0
86.6
75.0
60.1
61.8
28.5
84.3
83.3
73.0
LLM SWAPS
61.8
61.1
76.1
94.7
59.8
86.7
76.7
59.9
60.7
24.0
83.3
83.9
72.1

---

C
Random order random Algorithmic details
Algorithm Training Procedure with Concatenated Images and Hard Negatives pair (Ti, Fitm). caption pair (Hi, Vi+m)) 2: U4 RandomConcat((r;, Fi++m)
1:
function
3: Extract the sentences from the captions yi vř, Vitm ti
4: Positive Generation
5:
Create t = Co vi+m) % Concatenate first sentences
6:
Create t Concat (v) mm v % Shuffle order
7:
Create tpa RandomConcat(e y, (itm) for random k1, k2
sentence pair
8:
RandomConcat(13" ka v.tm) for random k3, k4 % Random order of random Create eT*
sentence pair
9:
% Hard Negative Generation
Extract linguistic categories for words in y and vi+m using spaCy if common category exist then
11:
Select a random common category CAT (e.g. NOUN, ADI, ...) Randomly select CAT words Wa++ from y Vi4m respectively
13:
else
15:
Randomly select words Wi, Wi+m from yi Ditm respectively end
16:
17: Swap and wi-m in 4 to create eT Swap(14, wa, Wi+m)
18:
return uy t 222 2P P4
19:
end function
Require: Training images I, captions T, hyperparameters Aconts Às-Neg, Auni for each iteration do Sample a batch of images and corresponding captions {x,, Vo 1m if iteration % 2 = 0 then
20:
22:
Concatenated training images
24:
for each image caption pair 24, Vi in the batch do
25:
Ui, 1." p PP P4 =GeneratePoaNeg((r. x++m), (3 3l+m))
26:
end for
27:
Compute loss (Contrastive, Hard Negative, Uni Modal)
28:
L = Aom + As-Neg + AuniCua
29:
else
30:
Standard CLIP training for each image caption pair æ;, Yi in the batch do Extract first sentence of caption y!
31:
32:
33:
end for
34:
Compute standard CLIP contrastive loss
35:
end if
36: Update model parameters
37:
end for
In this section, we present a pseudo-code of our algorithm, as discussed in Section in the main paper. The pseudo-code can be found in Algorithm 1.
D Experimental details and discussions
In this section, we give a detailed description of various training and baseline design choices.
D.I Further training details and discussions
As stated in the main paper, train with alternating steps of CLIP loss (Eq. (1)) and the pro- posed loss (Eq. (7)). This decision made to help the fine-tuned model retain its performance on downstream tasks, evident from the ablations in Table 4.

---

40G Computational resources. All the work was carried out on A100 40G GPUs. The training runs are across 4 GPUs. Running CLIC on smaller architectures like ViT-B/32 and ViT-B/16 took less than 30 minutes for the Laion dataset. Datasets such as RedCaps took longer due to high-resolution images. The larger ViT-L/14 models took around 1 hour per dataset. The evaluation time is negligible, in the order of 20 minutes per model for all datasets
single GPU.
Training parameters and details. In all of the experiments, the loss parameters Eq. (7) are to Acoat = 1/2, Ns-Neg 1/2 and Aua = 1. We train for one epoch on our IM Laion subset and on our 850k PixelProse dataset, while for MS-COCO, we trained for five epochs; ablation on the number of epochs can be found in Table 19. All experiments are conducted at an image resolution of 224 x 224. Our effective batch size 200 X 4, and we use a cosine scheduler, where the warm-up up phase is 20% of the training time. The learning rate (LR) starts at le 7, peaks at 1e 6, and arrives at le 8 at the end. We used the standard Adam W [34] optimizer with beta parameters (0.9, 0.98) and 6 set to le 8 with a weight decay of 0.1.
Training with COCO. Although for most of the experiments, we use our 1M Laion subset or the PixelProse dataset, we also train with MS-COCO to be comparable to the original NegCLIP checkpoint from [56]. Note that due to training on MS-COCO all retrieval numbers on MS-COCO for NegCLIP from [56], and CLIC-COCO are no longer zero-shot. The same is also true to CoN-CLIP, who use COCO images as distractor in their method.
Our NegCLIP baselines. To have a baseline for all models considered in this work, we train our own version of NegCLIP (denoted everywhere with t). It is used only when there are available checkpoints (Table 3, Table 24 and Table 25), and in our ablation study (Table 20). This version of NegCLIP differs from the NegCLIP [56] as follows:
1. We do not use hard negative images that are added to the batch.
2. We the two from the coarse-grained POS level ("ADV" and "ADJ"), two from the fine-grained level ("NN" for singular nouns and "NNS" for plural nouns), and two phrase-level categories (verb phrases and noun phrases).
categories in our method, while the NegClip paper employs six tags:
3. The vision encoder frozen during training.
We use their negative loss calculation (see Eq. (2)), where the hard negatives are part of the standard clip loss.
D.2 Zero-shot datasets
The 10 zero-shot classification datasets we use are a subset from the CLIP CLIP_ _be_chmarck'. Specif- ically, we use Ik images of each of the following datasets: Country-211 [44], Caltech-101 [15]. OxfordPets [40], DTD [4], FGCV Aircrafts [37], StanfordCars [22], Cifar-10, 100 [23], Food-101 [2].
D.3 Dataset from other methods
To improve compositionality, many methods use re-labelled/captioneed versions of standard datasets. The details of these recaptions can be found in Table 9. TripletCIIP uses recpationed data from LaCLIP and DAC uses BLIP2 [25] to create better captions. Methods like CoN-CLIP and SVLC-R+L use the original captions and PaLM-2 and BERT [7] to generate hard-negatives for their base datasets.
D.4 Other methods do not generalize
From the results on SugarCrepe++ and SugarCrepe, sees that other compositionality enhancing methods like SVLC-R, DAC, etc., perform relatively well on SugarCrepe but fail to generalize to SugarCrepe++.. We believe this is due to the way these methods operate. Specifically, most of these methods create hard-negatives aimed at certain attributes such as color, position, or size (for example, modifying P1: "a woman plays black guitar" N: "a woman plays green guitar"). This aligns closely with what benchmarks like ARO, and SugarCrepe evaluate (with SugarCrepe generating more fluent and sensical negatives).
Phttps://gitaa com/LAIOO AI/ AI/CLIP, benchmark

---

Table 9: Recaptioning/negatiee generation for different methods. We show the different datasets and the respective re-captioning model used by compositionally enhancing methods.
Base
Re-caption
Method
Source
Dataset
Method
NegCLIP
[56]
MS-COCO
DAC-LLM
[9]
CC3M
BLIP2 [25]
DAC-SAM
[9]
CC3M
BLIP2 [25]
SVLC-R
[10]
CC3M
SVLC-R+L
[10]
CC3M
BERT [7]
CoN-CLIP
[49]
CC3M
PaLM-2 [1]
TripletCLIP
[41]
CC3M, CC12M
LaCLIP [12]
CLIC-COCO
Ours
MS-COCO
CLIC-LAION
Ours
Laion
CogVLM [53]
CLIC-CCI2M
Ours
CC12M
PixelProse [50]
CLIC-RedCaps
Ours
RedCaps
PixelProse [50]
While such approaches can indeed improve performance on lexical understanding, they fail to capture word replacements that More importantly, we believe these changes do not foster a deeper, more nuanced understanding of compositionality within the model. True compositional reasoning requires distinguishing between examples that differ not only lexically but also syntactically and semantically, for instance, recognizing that N2: "a woman plays plastic guitar" negative, whereas P2: "a guitar is being played by woman" is positive. These cases involve more complex syntactic and semantic variations and move beyond lexical differences.
in the dictionary and fall outside predefined structure.
For SugarCrepe++, in addition to the evaluation in SugarCrepe, an additional lexically different text is introduced, and since these methods do not account for general compositionnalsttuutuuaa variations in their
they fall short on SugarCrepe++.. On the other hand, the random word swapping in CLIC makes our method focus not on the specific attributes but the general composition of different words in the text. This makes our method generalize better and work equally well for both SugarCrepe++ and SugarCrepe.
D.5 Detailing the LLaVA experiment
As LLaVA uses only the vision encoder from CLIP models trained at a higher image resolution (336 x 336 pixels), performed an additional fine-tuning of ViT-L/14 with CLIC, this time also unfreezing the vision encoder. We employed the PixelProse dataset and reduced the effective batch size 40 8, while keeping all other settings identical to those in Appendix D.1.
Subsequently, we fine-tuned LLaVA-1.5 using Vicuna-7b as the LLM. Initially, only the multi-modal projector was fine-tuned, followed by fine-tuning of both the projector and the LLM. This procedure follows the standard setup from the official LLaVA codebase To ensure a fair comparison, we conducted these experiments using both the original CLIP and our CLIC vision encoders. Hence, the two resulting models in Table are both fine-tuned by us, making the subsequent comparison fair.
For the evaluation reported in Table 6, we adopt the VQAScore from Lin et al. [30] for both SugarCrepe++ and WinoGround. For standard VLM benchmarks, we select subset of tasks from the original LLaVA codebase. Specifically, we assess question answering using accuracy on GQA [19] and Tex [47], chain-of-thought reasoning with ScienceQA-Imagee (SQA-I) [35], and perception via MME [54]. For all evaluations we use the default setup from the original LLaVA codebase.
E Additional experiments
In this section, we present more detailed results the experiments Section in the main paper and provide additional experiments.
7

---

Extension of results from the main paper As we only show averaged ITT and TOT numbers across different sets of SugarCrepe++ and SugarCrepe in the main paper, the full versions for ViT-B/32 models can be found in Table 17 and Table 18 resp. Similarly, for ViT-B/16 (omitted in the main part) and ViT-I/14, detailed results are shown in Table 24 and Table 25, respectively. Detailed compositionality results for the newer CLIP versions in CLIPA, EVA02-CLIP and CLIPS with their respective better versions achieved via CLIC can be found Table 26. In Table 23, we give dataset wise accuracy of all the large models for the ZS-10 setting. WinoGround results for large model can be found in Table 27. The pre-trained checkpoints used for fine-tuning with CLIC are listed in Table 22.
Improvements across architectures In Table in the main part of the paper, we show how CLIC improves compositionality and retrieval for differently pre-trained models. We visualize these improvements in Figure 5. Fine-tuning with CLIC yields improvements on all models for ITT on SugarCrepe++ and SugarCrepe. The improvements in case of CLIP are as high as 7% and as much as 9% for CLIPS.
Ablating training steps In Table 19, we show how the proposed CLIC changes with more training steps. Here, we use MS-COCO for training which means epochs corresponds 2.5 Laion epochs of our dataset. From the table, we see increasing gains in SugarCrepe++ numbers when increasing the number of training step. However, this comes with marginal degradation in zero-shot classification, which goes down from 63.2% to 62.7% on IMAGENET as we move from 2 to 5 epochs. The same trend holds for our 1M set of Laion, where training for more than 1 epochs led to decay on IMAGENET and ZS-10 10 numbers with marginal gains on SugarCrepe+++ Hence, for our runs in the main part of the paper with Laion (CLIC-LAION) we only train for epoch.
Ablating freezing of model components In Table 20, we show how the proposed CLIC (frozen vision encoder) yields, on average, the best compositionality without sacrificing downstream performance. From the table, it is clear that freezing just the text encoder leads to the least amount of improvement on compositional benchmarks. Fine-tuning the whole model works better (in terms of retrieval) but the gains on SugarCrepe++ are smaller than freezing the vision encoder and it also degrades zero-shot classification performance. From these results, we infer that the most improving configuration is freezing the vision encoder, which is the final setting for CLICC This also highlights that substantial gains can be made by improving the text encoder alone.
Ablating number of additional positives In Table 21, we show the impact of adding additional positives for each concatenated image (p3 and P4 from Section 3.1). Notably, increasing the number of positives has opposing effects on the TOT and ITT scores. In addition, after two positives, the effects on ITT in SugarCrepe++ become minimal, so we chose to add only two positives.
Error bars To show the resilience to randomness of CLIC, we conduct three runs for two versions of CLIC presented in Table and Table 2 (CLIC-LAION and CIJICRedCaps). We report the mean and standard deviation in Table 10 and Table 11, respectively. The standard deviation is small, particularly when considering that the results in the paper are rounded to one decimal place.
Table 10: Small standard deviation across runs. Results are reported as mean + std over three independent runs on compositionality benchmarks with the ViT-B/32 architecture
SugarCrepe++ [11]
WinoGround [52]
SugarCrepe [18]
Replace
Swap
Text
Image
Group
Add
Replace
Swap
Method
ITT
TOT
ITT
TOT
Score
Score
Score
ITT
ITT
ITT
CLIP
69.5
60.5
45.7
25.9
31.2
11.0
8.7
72.9
80.0
62.7
CLIC-LAION
75.34 + 60.10 + 61.55 + 27.86 + 31.75 + 11.83 + 9.33 +
84.56 + 83.84 + 73.79 +
0.18
0.11
0.38
0.08
0.20
0.31
0.31
0.09
0.14
0.12
CLIC-RedCaps
76.18 + 57.82 + 61.60 + 23.34 + 32.00 + 11.67 + 10.08 + 86.27 + 84.80 + 72.54 +
0.16
0.17
0.07
0.19
0.61
0.12
0.24
0.15
0.04
0.04
Long captions retrieval To evaluate the influence of exploiting superficial artifacts such as potential abrupt topic changes, generated by our concatenation scheme, we compared the retrieval performance on longer captions (up to 200 tokens) between the original model, other compositionaliy-enhanne methods, and our fine-tuned models (CLIC) on the ViT-B/32 architecture. This setup is adapted
8

---

Table 11: Small standard deviation across runs. Results are reported as mean + std over three independent runs on downstream tasks with the ViT-B/32 architecture.
Classification
1 T
T
Method
IMNET
ZS10
COCO
F30K
COCO
F30K
CLIP
63.3
61.4
74.1
95.1
54.6
83.5
CLIC-LAION
61.72 = 0.02
60.89
0.06
75.92 0.17
95.10 = 0.08
60.32 + 0.64
86.70 = 0.03
CLIC-RedCaps
62.41
60.22 0.03
76.00 0.09
95.67 0.05
59.42 1 0.04
86.27 = 0.02
from long- retgrvetrieval previously done in [58]. We truncate the caption to the nearest punctuation (within context-length) as CLIP has context length of 77. It is important to note that these longer captions describe a single scene but consist of multiple sentences. This design choice helps us determine our CLIC models have inadvertently overfit to abrupt topic changes or even to concatenation with punctuation. As can be seen in Table 12, only CLIC and TripletCLIP outperform the base model for both image and text retrieval, whereas DAC-LLM and SVLC-R+L are worse.
deliberate:
Table 12: CLIC does not overfit to abrupt topic changes. Evaluation of long-caption retrieval task for the VIT-B/32 architecture.
Method
I T
T
CLIP
83.2
79.5
NegCLIP
82.6
80.7
DAC-LLM
67.6
72.5
SVLC-R+L
75.0
74.8
TripletCLIP
84.1
81.5
CLIC-COCO
85.9
81.4
CLIC-LAION 83.9
79.6
CLIC-RedCaps 84.7
79.9
More retrieval results the main part the paper, all results on MS-COCO and Flickr30k retrieval reported with Recall@5, In Table 28, additionally report Recall @1 for all ViT-B/32 models from Table 1. The overall ordering of methods is similar to Recall Similarly, in Table 27, we additionally report Recall @ for all ViT-L/14 models from Table 3.
Visual examples and failure case analysis To illustrate the strengths and limitations of the method and the compositionality task in general, we selected four of the SugarCrepe++ categories: Replace attribute, Replace relation, Swap attribute, and Swap object. For each, we examined first instance in which the pretrained model's prediction changed from incorrect to correct with CLIC-RedCaps, and vice versa. The results show that CLIC has lower absolute cosine similarity values compared to the pretrained model.

---

Table 13: Visual examples illustrating cases where the pretrained model's prediction changed: from incorrect to correct with our CLIC-RedCaps version, and from correct to incorrect in the replace attribute class of SugarCrepe++. / marks correct predictions and X marks incorrect predictions.
P1: person holding up a chocolate doughnut with face drawn on it. Pz: A chocolate doughnut with face drawn on it is being held up by person.
P1: A tan toilet and sink combination in small room.
P2: A small room with a tan toilet and sink combination positioned in close proximity to one another.
N: A person holding up vanilla doughnut with face drawn on it.
N: A white toilet and sink combination in small room.
Algorithm
cos(P)
cos P2)
cos (N)
Algorithm
cos P1)
cos( P2)
cos( (N)
CLIP (X)
0.323
0.306 0.285
0.309
CLIP (V)
0.325
0.341
0.319
CLIC (
0.280
0.263
CLIC (X)
0.260
0.256
0.262
Table 14: Visual examples illustrating cases where the pretrained model's prediction changed: from incorrect to correct with our CLIC-RedCaps version, and from correct to incorrect in the replace relation class of SugarCrepe+++. / marks correct predictions and X marks incorrect predictions.
Pr: A pizza covered lots of greens on top of table.
Pi A zebra is standing in an open field.
Ps: table has a pizza on top of covered in
Pe: On an open field, stands zebra.
lots of greens.
N: A pizza covered lots of greens next to a table.
N: A zebra is running across an open field.
Algorithm
cos(P1)
cos (P2)
cos(N)
Algorithm
cos(P)
cos (os(P2)
cos(N)
CLIP (X)
0.300
0.280
0.293
CLIP (V) CLIC (X)
0.311
0.312
0.298
CLIC (V)
0.276
0.275
0.272
0.283
0.254
0.270
10

---

tree, with Laion re-captioned images
An elderly man wearing glasses and a suit standing behind a lectern with a golden eagle design. He appears to be in a formal setting possibly delivering speech or presentation. The background dimly lit and there are some decorative elements such as flowers visible behind him.
A dimly lit cave under- ground setting. The cave has various clements such as sta- lactites rocks and a chalkboard with some drawings on The atmosphere seems mysterious and adventurous suggesting that the characters might be on quest or exploration.
A bar chart that represents the average sales price of homes in Centerville Georgia over a 12-month period from Jan- uary 2010 to December 2010. The chart shows two sets of data: the average sales price of homes and the average sales price to list price ratio.
PixelProse images
In the foreground, there long bridge with train it. The bridge is made metal and has brown color. The train is red and white. Be- hind the bridge, there is large body of water. The water is a blue- suue-green color. There are some clouds in the sky. The image displays the Pamban Bridge, which is a still active train bridge located in Ramesh- waram, India
A pink and white frangipani flower. The flower is made up of five petals, with the pink petals the outside and the white petals on the inside. The flower is surrounded by green leaves.
A small Bengal cat with brown fur and black stripes. The cat is lying its front paws hanging off the edge of the platform. The is looking at the camera with curious expression.
Figure 4: Random images from Laion and PixelProse, after re-captioning with CogVLM [53] and re-captions from Singla et al. [50]. For each caption, the first sentence and another two randomly sampled sentences are presented. The first sentence often describes the entire image, while the additional sentences highlight specific details. As can be seen, the captions generated by Cog VLM [53] exhibit high quality similar the PixelProse dataset. Note these models still hallucinate (e.g. train on the bridge) sometimes and not all captions are 100% correct.

---

Table 15: Visual examples illustrating cases where the pretrained model's prediction changed: from incorrect correct with CLIC-RedCaps version, and from correct incorrect in the swap attribute class of SugarCrepe++. marks correct predictions and marks incorrect predictions.
P1: One apple and several oranges sit in bowl.
Pi: Two giraffe and zebra are standing in field.
P2: several oranges and an apple are positioned in bowl.
P2: In a field, two giraffes and a zebra are standing.
N: Several apples and one orange sit bowl.
N: giraffe and two zebras are standing field.
Algorithm
cos(P)) P1)
cos( P2)
cos(N)
Algorithm
cos( P1)
cos (os(P)
cos N))
CLIP (x)
0.306
0.315
0.309
CLIP (V)
0.316
0.316
0.315
CLIC (V)
0.276
0.298
0.270
CLIC (X)
0.305
0.290
0.295
Table 16: Visual examples illustrating cases where the pretrained model's prediction changed: from incorrect to correct with CLIC-RedCapp version, and from correct to incorrect in the swap object class of SugarCrepe++. / marks correct predictions and X marks incorrect predictions.
PI: Girls wash motorcycle while men look
Pi: A painting of vase with sunflower on
on.
a table.
Py: The motorcycle being washed by girls
P2: A vase containing sunflower positioned on table in a painting. N: painting of sunflower with a vase on a table.
while men observe.
N: Men wash motorcycle while girls look on.
Algorithm
cos(P:) Pt)
cos( Pz)
cos(N)
Algorithm
cos P1)
cos (2))
cos(N)
CLIP (X)
0.365
0.356
0.358
CLIP (V)
0.378
0.381
0.378
CLIC (V)
0.364
0.371
0.362
CLIC (X)
0.317
0.321
0.322
12

---

Table 17: Extended results for SugarCrepet++ for VIT-B/32 architecture. This table is an extension
of Table 1.
Method
Replace-ohj
Replace-att
Replace-rel
Swap-ohj
Swap-att
AVG
ITT
TOT
ITT
TOT
ITT
TOT
ITT
TOT
ITT
TOT
ITT
TOT
CLIP
86.8
83.7
65.6
59.3
56.2
38.6
46.1
19.2
45.2
32.7
60.0
46.7
NegCLIP
89.6
94.5
69.5
76.4
52.3
51.4
54.7
33.1
58.8
56.7
64.8
62.5
DAC-LLM
65.7
76.8
47.7
59.5
47.6
42.3
31.4
11.4
32,9
24.8
45.1
43.0
DAC-SAM
64.3
75.8
43.9
56.1
48.6
48.6
27.7
11.4
33.6
25.4
43.6
43.5
SVLC-R
82.9
89.5
61.5
67.4
47.6
51.5
49.4
20.4
53.3
36.3
58.9
53.0
SVLC-R+L
80.9
91.6
57.1
67.0
47.3
51.3
42.8
18.4
48.9
34.4
55.4
52.5
CoN-CLIP
88.1
91.5
66.6
69.4
52.3
51.6
44.1
22.4
51.9
42.6
60.6
55.5
TripletCLIP
86.8
92.3
71.7
74.0
61.9
51.7
38.8
24.1
47.9
42.3
61.5
56.9
CLIC-COCO
90.2
97.3
72.3
81.6
58.0
59.3
48.6
33.5
57.2
60.8
65.3
66.5
CLIC-LAION
90.0
84.5
75.8
57.9
61.0
38.0
61.2
22.4
61.0
33.4
69.7
47.2
CLIC-RedCaps
90.1
84.3
75.1
52.4
62.9
36.3
60.8
17.9
62.4
28.2
70.3
34.6
Table 18: Extended results for SugarCrepe for ViT-B/32 architecture. This table is an extension
of Table 1.
Methods
IMAGENET
Add
Replace
Swap
Obj-
Att.
Obj.
Att.
Rel.
Obj.
Att.
CLIP
63.3
77.2
68.6
90.9
80.1
69.1
61.2
64.3
NegCLIP
60.9
88.8
84.7
93.8
87.7
73.9
75.5
76.4
DAC-LLM
51.1
89.6
97.7
94.4
89.3
84.4
75.1
74.2
DAC-SAM
52.3
87.5
95.5
91.2
85.9
83.9
71.8
75.4
SVLC-R
58.8
79.4
91.2
91.3
81.2
64.2
68.6
69.1
SVLC-R+L
59.7
75.8
81.1
88.1
76.8
62.6
64.1
66.7
CoN-CLIP
63.2
87.9
78.0
91.8
81.0
66.3
63.7
67.0
TriplerCLIP
54.7
87.3
85.8
94.4
86.7
82.8
66.5
72.7
CLIC-COCO
62.7
88.8
82.4
93.6
85.9
71.6
65.7
72.8
CLIC-LAION
61.7
81.6
87.4
93.4
86.3
72.4
73.1
74.2
CLIC-RedCaps
62.3
84.2
88.7
93.3
86.9
74.0
71.8
73.3
Table 19: Witnessing the effect of training steps on CLIC. Although with more training steps, SugarCrepe++ numbers improve, it comes at the marginal degradation in downstream classification tasks. Hence, we do not train for more steps. are not zero-shot for MS-COCO evaluations.
Downstream Evaluations
SugarCrepe++
SugarCrepe Add Rep. Swap
Fine- tuning
Classification
Text Ret.
Image Ret.
Replace
Swap
Method
Epochs
IMNET ZS- 10 COCO F30k
COCO F30k
ITT
TOT ITT TOT
ITT
ITT ITT
CLIP [45]
63.3
61.4
74.1
95.1
54.6
83.5
69.5 60.5 45.7 25.9
72.9
80.0
62.7
CLIC-COCO*
2
63.2
60.9
81.7
96.6
66.3
89.8
73.0 78.3 53.0 45.4
85.6
83.3
69.9
CLIC-COCO*
3
62.9
60.9
82.3
97.1
67.5
90.3
73.5
78.7
52.6 45.8
85.2
83.5
70.1
CLIC-COCO"
62.7
60.7
82.9
97.1
68.2
90.2
73.5
79.4
52.9
47.2
69.3
13

---

Table 20: Effect of freezing/unfreezing different model components on CLIC for ViT-B/32. All these runs done with the CLIC-LAION version, and overall CLIC with frozen vision encoder works the best.
Downstream Evaluations
SugarCrepe++
SugarCrepe
Method
Frozen
Classification
Text Ret.
Image Ret.
Repl.
Swap
Add
Repl.
Swap
IMNET ZS-10
COCO F30k
COCO F30k
ITT
TOT
ITT
TOT
ITT
ITT
CLIP [45]
63.3
61.4
74.1
95.1
54.6
83.5
69.5
60.5
45.7
25.9
72.9
62.7
CLIC
None
60.5
60.8
77.7
94.7
62.1
87.4
72.2
60.0
60.0
25.1
84.5
83.6
71.9
CLIC
Text
60.2
60.3
75.8
93.4
61.1
86.5
72.5
60.5
49.8
25.9
76.8
80.1
64.4
CLIC Vision
61.7
61.0
75.9
95.0
60.0
86.7
75.6
60.1
61.1
27.9
84.5
84.0
73.7
Table 21: Effect of adding additional positives on CLIC for ViT-B/32. All experiments use the CLIC-LAION version, with the negative caption (n). The column "Positives" reports addition of more positives (p2, P3. P4, Ps) from Section 3.1. Beyond four positives, changes are marginal, so we chose this configuration.
Downstream Evaluations
SugarCrepe++
SugarCrepe
Method
Positives
Classification IMNET ZS 10
Text Ret.
Image Ret.
Replace
Swap
Add
Replace Swap
COCO
F30k
F3Ck
ITT
TOT
ITT
TOT
ITT
ITT
ITT
CLIP [45]
63.3
61.4
74.1
95.1
54.6
83.5
69.5
60.5
45.7
25.9
72.9
80.0
62.7
CLIC
P1
62.0
61.3
75,5
94.4
60.0
86.4
69.0
67.1
51.8
35.8
85.9
83.1
70.9
CLIC
+2
61.8
61.2
75.5
94.6
59.9
86.5
69.7
68.6
53.3
35.3
85.6
82.8
72.3
CLIC
+ps
61.7
60.8
75.8
94.9
59.8
86.6
73.8
61.7
58.7
30.2
84.5
83.8
72.6
CLIC
44
61.7
61.0
75.9
95.0
60.0
86.7
75.6
60.1
61.1
27.9
84.5
84.0
73.7
CLIC
+Ps
61.7
60.7
76.2
95.1
59.8
86.7
75.6
59.3
61.5
27.0
84.1
83.6
73.0
Table 22: Pre-training/Vvalustee checkpoint locations. Key values for the pre-trained models,
taken from different sources.
Model
Source
Key
ViT-B/32
OpenClip
openai
ViT-B/16
OpenClip
openai
ViT-L/14
OpenClip
openai
NegCLIP
GitHub
DAC-SAM
GitHub
SivanDoveh/DAC
SVLC-R+L
GitHub
SivanDoveh/TSVLC
CoN-CLIP
GitHub
jaisidhhiinghhCol-CLLL
TripletCLIP
GitHub
triplettclip/Ir:pppeeeCLPP
EVA-CLIP
OpenClip
er00__lagge_pacc111.......222
CLIPA
HF-Hub
UCSC-VLAA / hft.mn.ucce..yaaArrrr-----------
CLIPS
HF-Hub
UCSC-VLAA V
14

---

Table 23: Zeroshot image classification. We report the zero-shot image classification performance for different models.
Caltech101
Country211
Food101
Cifar10
Cifar100
PGVS
Flowers
Mean
Cars
DID
Pets
Model
Source
CLIP
[45]
93.6
96.4
76.3
76.5
30.4
54.6
94.9
86.4
80.6
33.1
72.3
CON-CLIP
[49]
93.9
96.0
79.4
76.3
30.4
56.8
93.6
86.8
78.1
33.0
72.4
CLIC-LAION
93.3
96.4
78.3
75.0
34.1
56.5
89.6
86.6
75.1
33.6
71.8
CLIC-RedCaps
ours
92.5
96.3
78.0
76.0
30.8
56.6
91.7
87.3
78.6
32.9
72.1
CLIPA
[26]
94.9
92.7
42.2
95.0
88.5
81.8
78.3
CLIC-LAION
ours
93.9
98.8
88.4
92,9
43.1
67.8
95.7
88.4
80.7
30.8
78.0
ELIC-RedCaps
ours
94.5
98.4
88.0
93.3
41.9
69.5
95.9
89.3
80.6
31.5
78.3
EVA-CLIP
[51]
94.1
99.7
90.6
89.7
37.4
61.8
94.7
88.6
77.0
32.7
76.6
CLIC-LAION
ours
94.1
99.7
90.6
89.7
37.4
61.8
94.7
88.6
77.0
32.7
76.6
CLIC-RedCaps
92.9
99.4
89.9
88.8
33.4
62.6
92.3
89.1
77.4
32.5
75.8
CLIPS
[33]
93.2
98.2
86.9
91.6
39.4
66.2
94.9
86.9
78.2
29.4
76.5
CLIC-LAION
ours
91.8
98.1
87.1
90.6
37.6
64.2
93.8
86.8
78.1
27.2
75.5
CLIC-RedCaps
ours
92.7
98.0
86.8
91.2
37.7
67.1
94.5
87.8
80.8
30.0
76.7
Table 24: Compositionality and downstream results for VIT-B/16. We present detailed set of results for the pre-trained CLIP, our version of NegCLIP, CLIP fine-tuned by CoN-CLIP, CLIC- LAION and CLIC-RedCary (using pre-trained CLIP). For SugarCrepet+, we report only the ITT scores in detail. We note here that models with are not zero-shot for MS- evaluations.
ZS-Class.
Text-Ret.
Image Ret.
Model
Source
IMNET
Avg.
COCO
F30k
COCO
F30k
CLIP
[45]
68.3
65.5
76.2
96.4
85.6
NegCLIP
Ours
64.5
64.0
74.2
94.7
65.8
89.7
CoN-CLIP"
[49]
68.9
69.1
73.9
93.5
60.1
88.1
CLIC-LAION
Ours
66.5
62.3
77.6
96.9
88.9
CLIC-RedCaps
Ours
67.3
64.4
79.0
97.0
62.0
88.0
Compositionality: SugarCrepe++
Model
Source
Rep-obj
Rep-Att
Rep-Rel
Swap-obj
Swap-Att
Avg-ITT
Avg-TOn
CLIP
[45]
89.6
67.5
53.1
39.6
48.3
59.6
45.7
NegCLIP
Ours
88.7
66.0
46.9
47.3
53.7
51.9
CoN-CLIP
[49]
88.1
66.6
52.3
44.1
51.9
60.5
55.5
CLIC-LAION
Ours
91.6
75.9
62.5
55.9
62.0
69.6
45.7
CLIC-RedCaps
Ours
76.0
64.5
55.9
61.1
69.8
38.6
Compositionallty: SugarCrepe
Add-ohj
Add-Att
Rep-Ohj
Rep-Att
Rep-Rel
Swap-Ohj
Swap-Att
CLIP
[45]
78.4
66.8
93.5
81.0
66.6
60.0
65.0
NegCLIP
Ours
80.0
87.3
93.9
82.2
70.1
CoN-CLIP
[49]
87.3
79.6
93.6
81.0
53.3
59.2
65.2
CLIC-LAION
Ours
88.4
91.0
95.6
86.5
75.5
71.0
73.9
CLIC-RedCaps
Ours
87.2
88.6
94.9
85.7
73.3
69.0
71.3
15

---

Table 25: Comparison of Sugarcrepe and Sugarcrepet+ along with Downstream Retrieval/- Classification Performance for ViT-L/14. We present in this table detailed set of results for the pre-trained CLIP, our version of NegCLIPi, CLIP fine-tuned by CoN-CLIP. CLIC-LAION and CLIC-RedCappl (using pre-trained CLIP). For SugarCrepet+, we report only the ITT scores detail.
Compositionality: SugarCrepe*+
Model
Source
Rep-obj
Rep-Att
Rep-Rel
Swap-ohj
Swap-Aut
Avg-ITT
Avg-TOT
CLIP
[45]
90.6
67.5
54.0
43.6
45.6
60.3
50.3
NegCLIP
Ours
90.5
68.9
52.3
46.5
54.5
62.5
52.4
CoN-C CLI
[49]
92.2
68.6
55.6
44.1
47.9
61.7
54.9
CLIC-LAION
Ours
79.6
62.8
59.6
46.1
CLIC-RedCaps
Ours
93.5
75.0
59.5
54.7
55.7
67.7
41.3
Compositionality: SugarCrepe
Add-obj
Add-Aut
Rep-Obj
Rep-Att
Rep-Rel
Swap-Obj
Swap-Att
CLIP
[45]
78.3
71.5
94.1
79.2
60.4
62.3
NegCLIPt
Ours
81.2
86.0
94.3
83.1
71.1
69.4
70.6
CoN-CLIP
[49]
90.2
77.6
95.2
81.7
67.0
65.3
63.1
CLIC-LAION
Ours
86.1
89.3
96.5
86.4
73.8
71.6
CLIC-RedCaps
Ours
85.8
82.8
95.4
82.9
70.0
64.5
67.7
95.0
CLIPP
CLIPA
FVA-
CLJPSS
+
Improvements with CL
87.0
+5.7
+3.1
79.0
+3.4
+9.1
71.0
+0.2
+9.4
+8.5
+3.9
55.0
SugarCrepe
SugarCrepe+
Image-Retz.
Compositionality Denchmarkss
Retrieval Denchmaries
Figure 5: CLIC improves differently pre-trained CLIP models. We show for varied CLIP models, using CLIC (PixelProse) gives consistent improvements both compositionality and downstream evaluation benchmarks. Specifically, to the best of our knowledge, CLIPS +CLIC-RedCaps yields SOTA numbers SugarCrepe++ for CLIP- like models.
16

---

Table 26: Comparison of SugarCrepe and SugarCrepe+4 performance for CLIP style models. We present in this table a detailed set of results for the pre-trained CLIPA, EVA-CLIP. CLIPS, and their respective versions fine-tuned with CLIC. For SugarCrepe++, report only the ITT scores in detail.
Compositionality: SugarCrepe++
Model
Source
Rep-obj
Rep-Att
Rep-Rel Swap-obj Swap-Att
Avg-ITT
Avg-TOT
CLIPA
[26]
94.5
73.6
56.3
41.2
50.4
63.2
57.2
CLIC-LAION
Ours
93.9
79.1
62.1
52.2
57.9
69.1
52.1
CLIC-RedCaps
Ours
95.0
80.6
64.8
60.4
62.2
72.6
46.8
ENA02-CLIP
[5]]
93.2
73.5
59.3
43.7
46.7
63.3
52.7
CLIC-LAION
Ours
93.8
79.7
61.8
55.9
61.1
70.4
45.2
CLIC-RedCaps
Ours
95.2
80.7
63.8
59.2
60.0
71.8
41.9
CLIPS
[33]
95.5
64.9
51.4
69.8
71.9
65.1
CLIC-LAION
Ours
95.5
84.8
72.6
66.1
80.0
79.8
62.3
CLIC-RedCaps
Ours
95.9
85.1
73.7
68.6
81.7
81.0
62.9
Compositionality: SugarCrepe
Add-obj
Add-Att
Rep-Obj
Rep-Att
Rep-Re Rel
Swap-Obj Swap-Att Att
CLIPA
[26]
88.8
82.6
96.8
83.0
68.8
62.4
65.1
CLIC-LAION
Ours
93.5
92.2
97.1
89.3
78.3
72.2
70.9
CLIC-RedCaps
Ours
94.5
92.3
97.0
86.9
73.8
69.8
73.3
EVAU2-CLID
[51]
92.3
82,8
95.9
84.9
71.6
65.3
64.0
CLIC-LAION
Ours
85.5
91.3
96.4
86.5
72.7
70.6
70.5
CedC-RedCaps
Ours
90.7
90.6
96.7
86.7
73,2
70.2
70.3
CLIPS
[33]
92.5
86.3
88.6
78.1
74.3
85.0
CLIC-LAION
Ours
96.6
96.8
98.7
93.0
85.5
82.0
89.3
CLIC-RedCaps
Ours
95.1
95.5
97.9
92.2
83.2
81.2
87.8
Table 27: Retrieval @ and WinoGround for ViT-L/14 models. In addition to retrieval results with Recall Table we report retrieval scores at Recall and WinoGround results. are not zero-shot for MS-COCO. The improvements made by CLIC here are consistent with Recall
Retrieval
WinoGround
Method
Text
Image
COCO
F30k
COCO
F30k
Text
Image
Group
CLIP [45]
56.0
85.1
35.2
64.7
28.7
11.0
8.5
CON-CLIP* [49]
55.8
82.5
37.8
69.3
29.5
11.0
8.0
CLIC-LAION
56.8
85.4
40.2
70.4
30.2
10.7
8.2
CLIC-RedCaps
58.3
86.9
39.0
68.8
32.2
12.2
9.0
CLIPA
64.4
90.4
46.2
73.8
32.7
8.0
7.0
CLIC-LAION
63.0
86.7
46.8
73.6
28.7
8.7
6.7
CLIC-RedCaps
63.2
89.1
46.6
73.9
32.0
11.2
7.2
EVA-CLIP
63.7
90.5
46.8
77.3
32.7
12.5
10.2
CLIC-LAION
63.6
90.5
46.5
77.4
31.7
12.7
8.7
CLIC-RedCaps
63.2
89.8
46.4
76.7
36.0
11.7
8.2
CLIPS
73.6
95.7
54.2
82.6
36.2
16.0
12.5
CLIC-LAION
71.0
95.3
56.6
85.7
37.5
16.0
13.2
CodC-ssdCaps
74.6
96.3
56.6
84.3
41.7
17.5
15.2
17

---

Table 28: @ for ViT-B/32 models. addition to retrieval results with Recall @5 in Table report retrieval scores Recall 1. zero-shot for MS-COCO.
Retrieval
Method
Text
Image
COCO
F30k
COCO
F30k
CLIP [45]
49.5
78.5
30.1
59.1
NegCLIP" [56]
60.8
85.1
45.1
70.9
DAC-LLM [9]
33.0
64.9
36.7
53.2
DAC-SAM [9]
33.0
61.5
33.2
58.4
SVLC-R [10]
45.0
73.5
35.4
64.5
SVLC-R+L [10]
42.8
70.4
35.6
63.7
CoN-CLIP" [49]
48.5
75.7
30.3
59.7
TripletCLIP [41]
46.9
73.7
32.3
60.2
CLIC-COCO"
60.0
84.7
41.1
68.3
CLIC-LAION
52.2
79.9
35.2
62.6
CLIC-RedCaps
52.5
79.4
33.8
62.8
18

---

Prompt: Can you describe this image detail?
Input Image
LLaVA-1.5-7b (CLIP)
LLaVA-1.5-7B (CLIC)
The festures small dog with curly eajoyang sandy beach. dog appears foo people
The features small brown and whise dog running on sandy runs through enjeying
appears
There standing closer side the dog und the other person funber back un the right side. The beach setting is complemented by the presence of backpack, likely belonging the people, placed on the sand near the center
kicking up some dusi. There another persne in the background, possibly observing the dog or engaging in their own activities. In the sorne. there are also two handbags, one near the person in the background and another closer to the
Prompt: Can you describe this image detail?
Input Image
LLaVA-1.5-7b (CLIP)
LLaVA-1.5-7b (CLIC)
In the image, there are two peuple in setting. wommnn standing machine. holding pair dumbbelis, and appears
The image features woman in pink shirt and black paits working our in gym, Sit holding of dumbbells
working They both focused holding pair clambbellls. gym is equipped inclading bench, treadmill, and a car. The bench is loested in the agrkooddddd while the trendmill is pusitioned towards the right side the scene. The car situsted on the right side the imago, possibly indicating that the gym locused parking
litting appears
standingg
focused hend. workout, demonstraing straugth determination,
routines.
varions picces equipment.
backgroundd there is another person who scems to observing the women's is workout A car is visible the for right comer of the image, possibly parked outside iddilionally, there is bench located center of
Prompt: Can you describe this image in detail?
Input Image
LLaVA-1.5-7b (CLIP)
LLaVA-1.5-7b (CLIC)
sitting roodtop. performing handstaadd. The the
image
The image
ing handstaad on roottop. woman sining on nonflcss while man standing on his hands, doing handstand. They are both enjoying their time ouidrur, showcasing their im- preseive balance and strength. In the background. there several parked, adting urban setting the The noeflpp, some others away.
the some, while the woman the right side. They are both baluncing oa their hands. showcasing their impressive strength and flexibility. In the background. cund. there are two cars visible, one the left side and the other the right side the roofup. The
parked distance, providing depth szene.
scatlsted around foreground
Prompt: Can you describe this image in detail?
Input Image
LLaVA-1.5-7b (CLIP)
LLaVA-1.5-7b (CLIC)
The image festunes small dog wearingg white checkered shirt, standing next to car. They are buth positioned in grassy yord, with the dog on the left side and the car on the right side of the scene. In background, Incased
and
The image features a emall brown and white dog standing sext to lagg orange and white car yard. Both animals appear to be looking at each other, possibly engaging in playful inter action. left side background.
plants, right
The dog positiond right..
yard. side yard. providing place for people sit enjoy the ouidocr space.
are two potted plants, side of some and the other the right side. bench can also be seen in the yard. situ- sted towaeds the right side of the seens.
Figure 6: Comparing response quality different LLaVA-1.5-7b versions on random WinoGround validation images. We compare the open-ended prompt (Can you describe this image in detail?) based image captioning of LLaVA with CLIP's vision encoder to that of CLIC. Overall, we believe the responses from CLIC enabled LLaVA to be marginally better than the standard LLaVA, but both models show aspects of ballucinations. Incorrect parts are highlighted and the respective corrections (if available) are also highlighted.
19


Reviews:
[{"review_id": "94c8WhUPeq", "reviewer": "Reviewer_m5ip", "summary": "The paper looks at how VLMs like CLIP struggle with with how images are composed, often confusing semantic concepts and their physical relation to each other. This is a well-documented and measured by several benchmark datasets. The authors propose a new fine-tuning method called CLIC, which trains the model using a variety of altered positive captions and negative captions for image-caption pairs. They show this method of training with negative captions helps models with semantic understanding, not just lexical. The SugarCrepe++ benchmark was used.", "questions": "Could the authors explain why they are not concerned about the negative captions being ungrammatical or with awkward syntax?\nIf they can show they get the same results with grammatical and proper sentences, then my issue of the model learning artifacts has been weakened.\n\nSimilarly, can the authors provide any analysis that rules out the possibility that the model is learning to exploit superficial artifacts like lists or abrupt topic changes?\nMy confidence in the model’s compositional abilities would increase.\n\nDo the authors have suggestions for how their approach could be extended to encourage learning of more integrated and hierarchical compositionality?\nConcrete proposals or pilot experiments in this direction would help demonstrate that the approach is not limited by its current synthetic design, and would positively impact my assessment.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths \n\nThe method of combining random images and negative captions by substituting one word offers a cheap way to improve how VLMs understand the meaning of captions. It is flexiblee nough to be used across different types of data.\n\nBy joining together two random images and their captions, the training data becomes more complex, which pushes the model to handle combinations of concepts instead of just single objects.\n\nThe model achieves high scores on tough benchmarks like Sugarcrepe++, and on some tasks like downstream classification and retrieval, it matches or comes close to the best models available.\n\nWeaknesses\n\nSubstituting a single word to make a negative caption may result in unnatural sentences and since the process might often produce sentences that are not meaningful or not grammatical, the model may learn to spot textual artifacts rather than actual compositional relationships. \n\nAlso, the model might learn to ficus on lists of captions, separated by a period and with abrupt topic change.\n\nReal photos often contain multiple objects and layers of relationships that are more complex than just combining two random scenes. This might be addressed in a followup study.\n\nSugarCrepe++ is the main test for compositionality in this paper, and while the model does well on it, this benchmark doesn’t require understanding of complex, hierarchical relationships. So, there may still be important gaps in real-world performance.\n\nThe paper does not report error bars or statistical significance, so it’s hard to know if the improvements are always reliable, especially when differences are small. The authors address this by saying how it is computationally expensive. However, the differences in Table 1 and 4 are large enough to heurisitally assume there;s likely to be statistical significance. And in Table 2, the differences are small enough to assume statistical insignificance, or at least practical insignificance.", "was_revised": true, "final_justification": "I responded to the authors' rebuttal by saying they have some good ideas for how to address the concerns, and it would be good to see how they turn out. My original score remains unchanged.", "rebuttal": "We appreciate the reviewer's constructive comments and their recognition of our method’s strong performance on challenging benchmarks, as well as its computational efficiency. We address the raised questions/weaknesses in order below. \n\n### Question-1\n\n(Weakness 1) Substituting a single word  …  not grammatical …\n\n(Question 1) Could the authors explain  ... negative captions being ungrammatical...\n\nAnswer-1\n\nThanks for the suggestion, we agree that using only meaningful and grammatically correct negatives could further improve the compositionality of the model. To test the impact of non-meaningful or ungrammatical negatives, we made the following modification to our method to get grammatically correct negatives:\n1) We extracted the fine-grained Part-of-Speech (POS) tags for each word in the first sentence using spaCy’s most computationally expensive model, which took around 5 hours instead of a couple of minutes (in the paper, we prioritized computational efficiency instead).\n2) For each word, we queried Gemini 2.5-flash to generate up to five replacement candidates from the same POS category, such that replacing the word changes the meaning of the caption, while keeping the sentence coherent. **Note**: these queries to Gemini via API calls took over 24 hours of compute time to find word replacements for the 1M Laion subset. In comparison, fine-tuning a CLIP model with CLIC only takes around 40 minutes.\n3) During training, for each caption, we randomly selected a POS category and then randomly chose a word from that category. The second image and word replacement were selected based on the candidates from step 2.\n### Table A: Meaningful and grammatical negatives. All methods use LAION data captioned by CogVLM and the ViT-B-32 architecture.\n| Method | Sampling | Epochs | IMNET | COCO Ret. I→T | COCO Ret. T→I | SC++ Replace (ITT) | SC++ Replace (TOT) | SC++ Swap (ITT) | SC+ Swap (TOT) | Wino Text Score | Wino Image Score| Wino Group Score| SC Add (ITT) | SC Replace (ITT) | SC Swap (ITT) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| CLIP | - | - | 63.3 | 74.1 | 54.6 | 69.5 | 60.5 | 45.7 | 25.9 | 31.2 | 11.0 | 8.7 | 72.9 | 80.0 | 62.7|\n| CLIC | Paper| 1| 61.7 | 75.8| 59.8| 75.3| 60.2| 61.9 |28.0| 31.8| 12.3| 9.5| 84.2| 83.8 | 74.0|\n| CLIC | NR |1| 61.8| 76.1| 59.8| 76.7| 59.9| 60.7| 24.0| 31.3| 10.0|7.8 | 83.3| 83.9 |72.1|\n\nThis significantly more expensive variant of CLIC (NR for Non-Random in Table A) results in harder negatives (as evidenced by a higher loss during training) and leads to improvements in Replace(ITT) for SugarCrepe++ but degrades Swap(ITT), TOT, and Winoground. While at first surprising, we think that the smaller diversity of negatives compared to our original computationally cheap version of CLIC, with random replacement of words from the same POS category, could be the reason. \n\nTo check the quality of the negatives, we labeled 75 negative samples from both NR and our version of the paper. This process yielded 69% meaningful, grammatically correct negatives with distinct semantic meanings from **NR**, compared to 28% from the **Paper version**. Due to character limits, we include below 2 coherent and grammatically correct (_italicized_) hard negatives examples, and 2 nonsensical or grammatically incorrect ones. We will give additional examples in the revised version. While there are some errors, the overall quality is sufficient to prevent the model from relying solely on shortcuts, as many examples remain meaningful and grammatically correct, and the errors are relatively subtle.\n\n(postcard→ telegram)\n_N: a collage of various telegram marketing materials. a vintage postcard._\n\n(stripes→ swirls)\n_N: a black satin jacket with white swirls on the collar and cuffs. a graphic design element that features a large red heart at the center surrounded by intricate golden stripes and patterns._\n\n(whistle→ shout)\nN: a red kettle with a shout attached to its spout. a man in a dramatic pose seemingly in the middle of a punch or a whistle.\n\n(made→ destroyed)\nN: a hand-destroyed card with a floral design. a scene of destruction with a building that appears to have been damaged or made.\n\n\n### Question-2\n\n(Weakness 2) The model... lists captions … abrupt topic change.\n\n(Question 2) Similarly, can the authors provide any analysis….\n\nAnswer-2\n\nTo evaluate the influence of exploiting superficial artifacts such as abrupt topic changes, we compared the retrieval performance on larger captions (up to 200 tokens) between the original model, the other compositionality-enhancing methods, and our finetuned models (CLIC) on the ViT-B-32 architecture. This setup is adapted from long-retrieval as done in [1]. We truncate the caption to the nearest punctuation (within context-length) as CLIP has a context length of 77. It's important to note that these longer captions describe a single scene but consist of multiple sentences. This design choice was deliberate: it helps us determine if our CLIC models have inadvertently overfit to abrupt topic changes or even to concatenation with punctuation. Only CLIC and Triplet-CLIP outperform the base model for both image and text retrieval, whereas DAC-LLM and SVLC-R+L are worse.  \n### Table B: Evaluation of long-caption retrieval task\n| Eval | clip | Negclip | DAC-LLM | SVLC-R+L | Triplet | clic-coco | clic-pixpr | clic-cvlm |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| **I2T** | 0.832 | 0.826 | 0.676 | 0.750| 0.841| 0.859 | 0.847 | 0.839 | \n| **T2I** | 0.795 | 0.807| 0.725 | 0.748 | 0.815 | 0.814 | 0.799 | 0.796 |\n\nWe think that this indicates that there is no problem, but it is easier to show that something works well rather than that there is no potential problem. Please let us know if you have other suggestions on how to test this.\n\n### Question-3\n\n(Weakness 3) Real photos often contain multiple objects and layers of relationships …\n\n(Question 3) Suggestions…hierarchical compositionality? Proposals or pilot experiments …\n\nAnswer-3\n\nWe think it is possible, at least to a certain extent, with a **variant of CLIC** as we outline below in our proof-of-concept: In the paper, we use a random image for concatenation and a random word for swapping. However, similar to the experiments in App. B.4 and the answer to Q1 above, we can instead pair each image with another that contains the same main object (e.g., a cat) but differs in attributes (e.g., resting vs. in motion). This approach combines the two earlier non-random strategies: (1) keeping the noun consistent, and (2) selecting attributes with different meanings. We can then construct modified captions by changing the second reference to the object, while either preserving (Positive) or changing its attributes (Negative). The following examples highlight how our approach can be extended to such scenarios. However, this modification is more challenging, as the combinations are more prone to false positives and require additional preprocessing. E.g., in image A, the cat may indeed have a curled tail; thus, N2 might be an accurate description of the scene. \n\nAn example from Laion-CogVLM dataset (when dropping any extra sentences with non-relevant information):\n\nImage A: \"The cat has a tabby pattern with a mix of brown, black, and white fur, and appears to be resting or lying down.\"\n\nImage B: \"The cat appears to be in motion with its tail curled.\"\n\nWe concatenate the images to generate four variants (two positives and two negatives). Unlike the approach in the paper, we now use a single sentence that describes only one of the two cats in the scene. As a result, the model must correctly associate the described action with either the cat that has a tabby pattern or the one with a curled tail. P1 and P2 as above (Image A and Image B).\n\nN1:\"The cat has a tabby pattern with a mix of brown, black, and white fur, and appears to be in motion.\"\n\nN2:\"The cat appears to be resting with its tail curled.\"\n\nAnother option is to use different objects and attributes (presenting only the negatives):\n\nImage C: A fluffy white dog possibly a Samoyed walking on a road.\n\nN1:The cat has a tabby pattern with a mix of brown black and white fur, walking on a road.\n\nN2:A fluffy white dog possibly a Samoyed, appears to be resting or lying down.\n\n### Question-4\n\n(Weakness 4) Report error bars or statistical significance…\n\nAnswer-4\n\nWe do three runs on two of the datasets presented in Tab. 1 and 2 (CogVLM and RedCaps), and we report the mean and standard deviation in Table C. The standard deviation is small, particularly when considering that the results in the paper are rounded to one decimal place. We include these results in the revised version. \n### Table C: Small variance across 3 different runs of CLIC finetuning for ViT-B-32 for two datasets CogVLM on LAION and RedCaps\n| Benchmark | Metric | Pre-trained | LAION mean (± std) | RedCaps mean (± std) |\n| --- | --- | --- | --- | --- |\n| **SugarCrepe++** | replace ITT| 69.5| 75.34 ± 0.18 | 76.18 ± 0.16|\n| | replace TOT | 60.5 | 60.10 ± 0.11| 57.82 ± 0.17 |\n| | swap ITT | 45.7 | 61.55 ± 0.38 | 61.60 ± 0.07|\n| | swap TOT | 25.9 | 27.86 ± 0.08 | 23.34 ± 0.19|\n| **SugarCrepe** | add ITT | 72.9| 84.56 ± 0.09| 86.27 ± 0.15 |\n| | replace ITT | 80.0| 83.84 ± 0.14| 84.80 ± 0.04|\n| | swap ITT| 62.7| 73.79 ± 0.12| 72.54 ± 0.04|\n| **Wino** |txt score| 31.2| 31.75 ± 0.20| 32.00 ± 0.61 |\n|| img score| 11.0| 11.83 ± 0.31| 11.67 ± 0.12 |\n|| group score| 8.7| 9.33 ± 0.31 | 10.08 ± 0.24 |\n| **COCO** | img ret.| 54.6 | 60.32 ± 0.64 | 59.42 ± 0.04 |\n|| txt ret.| 74.1 | 75.92 ± 0.17| 76.00 ± 0.09 |\n| **Flickr30K** | img ret. | 83.5| 86.70 ± 0.03 | 86.27 ± 0.02 |\n|| txt ret. | 95.1| 95.10 ± 0.08 | 95.67 ± 0.05 |\n| **ImageNet** | acc | 63.3 | 61.72 ± 0.02 | 62.41 ± 0.09 |\n| **ZS10** | acc| 61.4| 60.89 ± 0.06| 60.22 ± 0.03 |\n\n[1] Zhang, Beichen, et al. \"Long-clip: Unlocking the long-text capability of clip.\" ECCV, 2024."}, {"review_id": "3Swc3sJVd1", "reviewer": "Reviewer_2CLt", "summary": "This paper proposes CLIC, a method to train and improve CLIP models' compositionality. The key idea is to randomly concatenate two images, and utilize their associated captions to make alternative positive captions and hard negative captions to train CLIP models to be able to distinguish lexically similar but semantically different captions, while at the same time being able to understand lexically different but semantically similar captions. Experiment results show some improvements on certain datasets, with more improvements on the text encoder.", "questions": "Please see the questions in weaknesses above.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper tackles an important problem on improving CLIP models' compositional understanding capability as recent work has pointed out that CLIP models are not sensitive to finer-grained semantic understanding.\n\n- The proposed method does not require much additional cost in generating hard positive and hard negative captions, whereas previous work may rely on external models in rewriting captions and generating hard negative captions (or images).\n\n- Experiment results show some improvements on the recently introduced SugarCrepe++ dataset (with more gain shown on text-to-text setting). The resultant CLIC trained model also mostly retains standard classification and retrieval accuracy.\n\n\nWeaknesses:\n- The introduced method that concatenates two randomly sampled images appears somewhat adhoc to me. This changes the image aspect ratio and may negatively affect the model's capability on standard images. This can be seen from CLIC models degraded performances on retrieval/classification performance in Table 2, as compared to original CLIP and baselines like NegCLIP (comparing both MS-COCO trained models).\n\n- The hard negative captions generated by CLIC through swapping words from two captions can introduce unrealistic captions as seen in Figure 2 (\"A water wearing a black hat\"). As pointed out in SugarCrepe work, these implausible captions might introduce biases into the resultant model. Thus, on the SugarCrepe dataset where the dataset specifically removed implausible negative captions, CLIC models' performances appear to be generally worse than other baselines. Could the authors discuss further on this issue?\n\n- The hard negatives generated from CLIC are almost like replacing words in a caption with other random words (in CLIC, it is done by sampling from another randomly selected caption). Mechanically, this is in fact not too different from existing methods in generating hard negatives by sampling the replacing words from some dictionary.\n\n\n- While the experiment results show some improvements on SugarCrepe++ dataset, I am concerned that for ITT setting, the improvements mostly comes from training on larger dataset (as CLIC-COCO shows mixed results compared to NegCLIP which is also trained on COCO). On TOT settings, while there are improvements, this might mainly come from the introduction of uni-modal loss, which I do appreciate but is not a significant contribution.", "was_revised": true, "final_justification": "The author responses clarify my concerns on the methodology similarity to existing baselines. Additional experiments with models trained on LAION make the results more convincing without direct train/test distribution match. I am raising my score to 4 after the author rebuttal.", "rebuttal": "We thank the reviewer for their time and valuable comments. We also thank the reviewer for their appreciation of the importance of the topic, and the efficiency+efficacy of the proposed method, CLIC. Below, we address your concerns.\n \n### Question-1\n\n\n(Weakness1) The introduced method that concatenates two randomly sampled images … may negatively affect the model's capability on standard images. This can be seen from CLIC models …, as compared to original CLIP and baselines like NegCLIP\n\n\nAnswer-1\n\nAs stated in the paper (lines 127-129), we believe that **there is a problem in assessing methods that were trained on COCO**, since many current evaluations are based on COCO images and captions (SugarCrepe, SugarCrepe++, COCO retrieval, etc.). Thus, training on COCO can leak test information to the model. We trained on COCO to be directly comparable to NegCLIP, but do not consider this as a proper setting, and thus train all other CLIC models on datasets independent of COCO. As can be seen in the ablation results in Table 10 (the relevant parts also included here for convenience), when training NegCLIP and our CLIC on Laion recaptioned with CogVLM, which is independent of  MS-COCO, CLIC performs better on compositionality ITT as well as downstream tasks. Both methods are then evaluated in a zero-shot setting across compositionality benchmarks as well as on downstream tasks. In addition, as can be seen in Tables 1-3 and other experiments in the appendix, when comparing CLIC with other methods that were not trained on COCO as well as a variant of NegCLIP trained on the same dataset as CLIC, **our method achieves the best SugarCrepe++ results as well as Text-retrieval results**, even though all these benchmarks are using a single image input. So we don’t think there is a problem with the model's capability on standard images.\n\n### Table D: Comparing CLIC to the single image baseline and NegCLIP† when all are trained on Laion recaptioned with CogVLM\n\n| ID  | Method              | IMNET | ZS-10 | COCO Text Ret. | COCO Image Ret. | SC++ Rep. ITT | SC++ Rep. TOT | SC++ Swap ITT | SC++ Swap TOT | SC Add ITT | SC Rep. ITT | SC Swap ITT |\n|-----|---------------------|-------|--------|----------------|-----------------|----------------|----------------|----------------|----------------|--------------|----------------|----------------|\n| A1  | CLIP                | 63.3  | 61.4   | 74.1           | 54.6            | 69.5           | 60.5           | 45.7           | 25.9           | 72.9         | 80.0           | 62.7           |\n|     | **NegCLIP†**        | 61.0  | 60.5   | 72.0           | 59.7            | 67.9           | 64.7           | 54.5           | 36.1           | 82.4         | 80.9           | 70.7           |\n| B4  | Single-Img Baseline | 61.3  | 60.6   | 67.3           | 58.0            | 74.1           | 53.9           | 60.2           | 27.8           | 81.0         | 79.9           | 69.7           |\n| C5  | **CLIC**            | 61.7  | 61.0   | 75.9           | 60.0            | 75.6           | 60.1           | 61.1           | 27.9           | 84.5         | 84.0           | 73.7           |\n\n### Question-2\n\n\n\n(Weakness 3) The hard negatives generated from CLIC are almost …  Mechanically, this is in fact not too different … replacing words from some dictionary.\n\nAnswer-2\n\n\n\nWe politely disagree with this perspective. When you rely on dictionary-based word replacements, the modifications are inherently confined to specific, pre-defined changes. For example: altering P1: \"a woman plays a black guitar\" to N: \"a woman plays a green guitar.\" While this approach can certainly be beneficial for improving performance on lexical understanding (and for existing benchmarks), it doesn't address word replacements that aren't in the dictionary – and perhaps, by extension, aren't in current benchmarks either. More importantly, we believe it doesn't foster a deeper, more nuanced understanding of compositionality within the model. In contrast, true compositional reasoning requires additionally distinguishing between examples like  N2: “a woman plays a plastic guitar” (negative) and P2: “a guitar is being played by a woman” (positive), which involve more complex syntactic and semantic variations. Dictionary-based swaps, though potentially useful for lexical understanding and current benchmark performance, do little to address these challenges, whereas CLIC’s approach, with its variety of negatives that increase quadratically with the size of the dataset (and their varied difficulties starting from non-plausible or grammatically incorrect sentences to hard non-predefined negatives), does. Additionally, at its core, CLIC always swaps two entities that are both present in the scene, whereas dictionary-based methods replace an entity in the scene with one that does not appear in it.\n\n### Question-3\n\n\n(Weakness 2) The hard negative captions generated by CLIC … Could the authors discuss further on this issue?\n\n(Weakness 4) While the experiment results show some improvements on SugarCrepe++ dataset,…, which I do appreciate but is not a significant contribution.\n\nAnswer-3\n\nWe agree with the reviewer’s concern that using unrealistic or non-grammatical hard negatives might provide weaker training signals compared to using only grammatical and realistic negatives. However, it is important to note that the Image-to-Text (ITT) metric in SugarCrepe++ uses the **same images, original positive captions, and plausible negative captions from SugarCrepe and adds to them an additional positive caption (lexically different, but semantically similar), as explained in lines 117-122.** We believe our slightly worse performance on SugarCrepe, but better results on SugarCrepe++, are due to the greater variety of negatives and positives seen by our method. Since SugarCrepe++ includes a second positive caption that is lexically more different than both the first positive and the negative captions, to succeed in this benchmark, methods can not rely on shortcuts such as targeting specific attribute replacements that improve results on SugarCrepe but do not generalize to SugarCrepe++. In fact, as can be seen in Table 1, previous methods such as DAC improve over the baseline by 10.4% on Sugarcrepe, but perform worse by -14.9% on Sugarcrepe++ (based on average ITT scores for the replace and swap tasks). This is detailed in Appendix D.4 and further discussed in our response to the previous question regarding dictionary-based methods.\nOur method outperforms previous methods on SC++, suggesting it is **better compositionally aligned than all previous methods, despite being computationally more efficient**, even though all the samples in the benchmark are still grammatically correct and plausible.\n\n**Regarding potential biases introduced by implausible or non-grammatical negatives**\n\nSugarCrepe and SugarCrepe++ are benchmarks, so it is essential that they use plausible and grammatically correct negatives to avoid shortcuts and ensure reliable evaluation. In contrast, since we use negatives only during training, **mixing plausible and implausible (or ungrammatical) negatives should not introduce biases, since all are treated as incorrect** and the model is trained to treat them as incorrect. To avoid biases, our main concern was to accidentally generate negatives that still accurately describe the scene. However, as discussed in Appendix B.3, **swapping words between captions of different images rarely results in captions that accurately describe the concatenated image** as opposed to swapping words within the caption of a single image. Thus, swapping random words should not introduce biases.\nTo test whether more plausible and grammatically correct negatives improve performance (at higher computational cost), we conducted an additional experiment that considerably reduced the number of non-plausible and non-grammatical negatives. The results were largely similar, as explained in the first response to reviewer m5ip (associated results in Table-A there)."}, {"review_id": "T8qhVlPyJg", "reviewer": "Reviewer_uTvH", "summary": "This paper introduces CLIC: a finetuning method designed to enhance the compositional understanding of vision-language models like CLIP. The core idea is to train on concatenated image pairs and their corresponding captions. The authors generate positive examples with diverse sentence structures and hard negatives by swapping words between captions. The authors demonstrate that CLIC improves performance on compositionality benchmarks like SugarCrepe++ and WinoGround, while maintaining or improving performance on standard retrieval and classification tasks. Experiments are run across different CLIP architectures and pre-training datasets. The method is efficient by fine-tuning only the text encoder.", "questions": "See weaknesses section", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Addresses the known limitation of vision-language models regarding compositional reasoning.\n- Proposes a novel and efficient data generation and fine-tuning strategy using image/caption concatenation and word swapping for hard negatives.\n- Demonstrates effectiveness across various CLIP architectures and pre-training datasets.\n\nWeaknesses:\n- The reliance on SpaCy for hard negative generation, while efficient, might be less nuanced and powerful compared to LLM-based approaches.\n- Would be nice to add visual examples and failure case analysis", "was_revised": true, "final_justification": "Thanks for the rebuttal. I keep my initial score (4) and look forward to see more visual examples and failure case analysis in the final version of the manuscript.", "rebuttal": "We appreciate the reviewer’s recognition of the importance of the topic, as well as their acknowledgment of our method’s strong performance, efficiency, and ability to generalize across different architectures and datasets. We address the raised questions below.\n\n### Question-1\n\n(Weakness 1) The reliance on SpaCy for hard negative generation, while efficient, might be less nuanced and powerful compared to LLM-based approaches.\n\nAnswer-1\n\nWe agree that LLMs have stronger abilities than the lightweight version of spaCy used in our work for **computational efficiency**. However, as demonstrated by the experiments in the paper, CLIC not only offers greater computational efficiency but also outperforms methods that utilize LLMs, such as TripletCLIP and DAC-LLM. Furthermore, the core approach of CLIC, which combines pairs of images with their corresponding captions, can be extended using LLMs, albeit with increased computational cost.\nAn LLM extension involving more challenging image concatenations and word replacements was explored in an experiment addressing Q1 from reviewer m5ip, yielding similar results but at a significantly higher computational cost. Another potential extension of generating negatives from scratch via LLM queries is too computationally demanding, given the resources available to us with current SOTA LLMs.\n### Question-2\n\n\n(Weakness 2) Would be nice to add visual examples and failure case analysis\n\nAnswer-2\n\nThank you for the suggestion. We agree that it will help clarify the strengths and limitations of the method and the compositionality task in general, and we will incorporate this into the final version of our paper. Unfortunately, due to the current rebuttal format, we cannot upload these results."}, {"review_id": "NTMG2O1E04", "reviewer": "Reviewer_PGUp", "summary": "This paper presents CLIC, a fine-tuning strategy aimed at enhancing compositional reasoning in CLIP-based vision-language models. The method combines multiple images and associated captions during training, with the goal of improving both lexical and semantic compositionality while maintaining or improving downstream retrieval performance. Results on SugarCrepe++ and various retrieval tasks are promising.", "questions": "Given the growing prominence of large vision-language models (LVLMs), a discussion—preferably supported by quantitative or qualitative evidence—on whether CLIC-style compositional fine-tuning benefits or transfers to LVLMs (e.g., BLIP-2, LLaVA) would greatly enhance the paper’s relevance to the current landscape.\n\nThe paper addresses an important problem and presents a promising direction, but the limited novelty and open questions around generalization and practical utility suggest that further clarification or extensions would strengthen its impact. I'm leaning towards borderline reject before rebuttal and open to discussion with authors and other reviewers.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper addresses a well-recognized weakness in CLIP models—compositional generalization—and proposes a reasonably efficient solution.\n* Experimental evaluation is comprehensive across multiple CLIP variants, including strong baselines like CLIPS.\n* The paper is generally well writen and easy to follow.\n\nWeaknesses:\n* The core contribution—mixing multiple images and captions during training—can largely be interpreted as a form of compositional data augmentation. While effective, the methodological novelty is relatively incremental, and does not introduce fundamentally new learning principles.\n* The inclusion of multiple loss terms introduces several hyperparameters. These are not thoroughly analyzed, and may affect the generalizability or reproducibility of results. A more systematic ablation or sensitivity study would help clarify this.\n* Since CLIC is applied as a fine-tuning method on top of pre-trained CLIP models, it is crucial to ensure that their strong generalization and zero-shot capabilities are preserved. The paper emphasizes improvement in compositionality and retrieval but does not clearly analyze potential trade-offs in broader zero-shot tasks.", "was_revised": true, "final_justification": "The rebuttal fully address my concerns, I lean towards accept as promised.", "rebuttal": "We appreciate the reviewer’s recognition of the importance of the topic, as well as their acknowledgment of our method’s efficiency, the extensive evaluations, and the clarity of the paper. We address the raised questions below.\n\n\n### Question-1 \n\n(Weakness 1) The core contribution .. form of compositional data augmentation... learning principles \n\nAnswer-1\n\nThank you for the new perspective regarding our work. We agree that CLIC can be seen as a better and effective way of compositional data augmentation. While data augmentation is not a new learning principle, it has been instrumental in numerous significant advancements in deep learning. Thus in our point of view suggesting an effective and computationally lightweight form of compositional data augmentation with a thorough evaluation is at least as good as a new ``learning principle’’ . Additionally, our work also highlights a critical issue: how existing methods have overfitted on benchmarks such as SugarCrepe, compromising their generalizability and real-world utility in terms of degraded zero-shot classification or retrieval performance of the resulting CLIP model. \n\n### Question-2\n\n(Weakness 2) The inclusion of multiple loss terms introduces several hyperparameters. … Systematic ablation\n\nAnswer-2\n\nThe hyper-parameters used to balance the losses in Equation 7 are:\n$\\lambda_{cont}=½$, $\\lambda_{S-Neg}=½$, and $\\lambda_{uni}=1$, as stated in Appendix D.\nWe did not perform a grid search to find the optimal parameters, and it is possible that fine-tuning these hyperparameters (especially with different configurations for each model or dataset) could lead to improved results. However, we used the same hyper-parameters and training settings across all architectures (ViT-B-32, ViT-B-16, ViT-L-14), pretraining datasets and methods (OpenAI, CLIPS, CLIPA, etc.), finetuning datasets (CogVLM, RedCAPS, CC12M, MS-COCO), and different tokenizers (OpenAI, CLIPS). Therefore, we believe that the method is reproducible and generalizable. We would like to highlight that compared to previous work we evaluate on significantly more models and study the dependency on the finetuning dataset. \nIn addition, based on the reviewers’ suggestion, we conducted three runs on two of the datasets presented in Tables 1 and 2 (CogVLM and RedCaps), and we report the mean and standard deviation. As can be seen in the table below, the standard deviation is small, particularly when considering that the results in the paper are rounded to one decimal place. We will include these results in the revised version.\n ### Table E: Small variance across 3 different runs of CLIC finetuning for ViT-B-32 for two datasets CogVLM on LAION and RedCaps\n| Benchmark | Metric | Pre-trained | LAION mean (± std) | RedCaps mean (± std) |\n| --- | --- | --- | --- | --- |\n| **SugarCrepe++** | replace ITT| 69.5| 75.34 ± 0.18 | 76.18 ± 0.16|\n| | replace TOT | 60.5 | 60.10 ± 0.11| 57.82 ± 0.17 |\n| | swap ITT | 45.7 | 61.55 ± 0.38 | 61.60 ± 0.07|\n| | swap TOT | 25.9 | 27.86 ± 0.08 | 23.34 ± 0.19|\n| **SugarCrepe** | add ITT | 72.9| 84.56 ± 0.09| 86.27 ± 0.15 |\n| | replace ITT | 80.0| 83.84 ± 0.14| 84.80 ± 0.04|\n| | swap ITT| 62.7| 73.79 ± 0.12| 72.54 ± 0.04|\n| **Wino** |txt score| 31.2| 31.75 ± 0.20| 32.00 ± 0.61 |\n|| img score| 11.0| 11.83 ± 0.31| 11.67 ± 0.12 |\n|| group score| 8.7| 9.33 ± 0.31 | 10.08 ± 0.24 |\n| **COCO** | img ret.| 54.6 | 60.32 ± 0.64 | 59.42 ± 0.04 |\n|| txt ret.| 74.1 | 75.92 ± 0.17| 76.00 ± 0.09 |\n| **Flickr30K** | img ret. | 83.5| 86.70 ± 0.03 | 86.27 ± 0.02 |\n|| txt ret. | 95.1| 95.10 ± 0.08 | 95.67 ± 0.05 |\n| **ImageNet** | acc | 63.3 | 61.72 ± 0.02 | 62.41 ± 0.09 |\n| **ZS10** | acc| 61.4| 60.89 ± 0.06| 60.22 ± 0.03 |\n\n\n### Question-3\n\n(Weakness 3) Since CLIC is applied as a fine-tuning method …crucial to ensure that their strong generalization and zero-shot capabilities …\n\nAnswer-3\n\nWe fully agree that enhancing compositionality should not degrade the generalizability and Zero-Shot (ZS) capabilities of CLIP models. The common generalizability and ZS-abilities of CLIP models are measured by ZS-classification and retrieval, both of which have been extensively tested in our work across models. We would like to point out that, unlike CLIC, previous compositionality enhancing methods often suffer from severe degradation in this aspect. For instance, in Tables-2 and 3, we show for models of different sizes that CLIC has a minimal reduction, if any, in zero shot classification (across 11 commonly used datasets) but almost always **improves over the baseline in the standard retrieval benchmarks**. We've also extended individual classification performance for several datasets in Table 15 in the appendix for a more in-depth look. Moreover, in Table-B in response to reviewer m5ip, we show that the performance for long-caption retrieval (significantly longer captions than the standard MS-COCO ones) is also preserved by CLIC fine-tuning. We would be happy to add other zero-shot tasks per the reviewer’s suggestions. \n\n### Question-4\n(Question 1) Given the growing prominence of large vision-language models (LVLMs), … a discussion supported by quantitative or qualitative evidence for LVLMs..\n\nAnswer-4\n\nGiven our limited computational resources and time, we couldn't perform full-scale fine-tuning to integrate CLIC models with LVLMs like LLaVA. As we noted in our submission, **CLIC typically fine-tunes only the text-encoder**. However, LLaVA primarily utilizes the CLIP vision encoder, which means CLIC isn't directly compatible **out of the box**. We made sure to highlight this in the limitations section of our paper as well.\nTo address this and evaluate LLaVA-1.5 with CLIC, we took a specific approach:\nWe fine-tuned CLIC on the Laion CogVLM set, crucially **unfreezing the vision encoder** during this process.\t\nThen, we replaced the original CLIP vision encoder in LLaVA-1.5 with our CLIC fine-tuned model.\nAll evaluations were conducted at a resolution of 224x224. We compared the performance of the original OpenAI CLIP LLaVA-1.5-7B with our CLIC-integrated version in the following tables. In Table F, for general captioning (COCO, Flickr30k), question-answering tasks (VQav2, VizWiz, TExtQA), hallucination (POPE), and Chain-of-thought (ScienceQA-Images), our CLIC-adapted LLaVa performance is very similar to the original LLaVA model (CLIP), even **without further fine-tuning of the vision-language projector**.\n\nTable F: General utility of LLaVA with and without CLIC vision encoders\n|Model  |  VQAv2 | VizWiz | TextQA|  COCO  |   FK30   | POPE | SQA-I\n|---------------|---------|---------|---------|---------|---------|---------|-------|\n**CLIP** |  73.7    |    40.0   |   36.4    | 118.9   |   77.4    |   0.844   | 65.1\n**CLIC** |  73.4    |    39.7   |   36.8    |  118.1   |   76.3   |  0.848   | 65.3 \n\nWe further adapted the VQAScore from [1] to evaluate LLaVA on compositionality benchmarks, including WinoGround and SC++. In these evaluations, our CLIC-enabled LLaVA achieved **higher compositionality** compared to the original OpenAI-CLIP LLaVA. It's important to reiterate that this improvement was achieved simply by **replacing the vision encoder in LLaVA with our CLIC fine-tuned model**. We are confident that fine-tuning the entire LLaVA model with the CLIC encoder would yield even greater improvements. However, due to the limitations in computing resources and time, this full fine-tuning is not feasible for us at present. We plan to include a detailed discussion regarding this in the final version of our paper.\n\nTable G: WinoGround evaluation of LLaVA\n| Model         | Text    | Image   | Group   |\n|---------------|---------|---------|---------|\n| **CLIP**     | 31.00%  | 30.00%  | 18.00%  |\n| **CLIC**| 33.00%  | 32.25%  | 18.00%  |\n\nTable H: SugarCrepe++ evaluation of LLaVA\n| Model          | Rep_obj | Rep_att | Rep_rel | Swap_obj | Swap_att |\n|----------------|---------|---------|---------|----------|-----------|\n| **CLIP**      | 80.23%  | 63.96%  | 57.47%  | 47.76%   | 55.11%    |\n| **CLIC** | 79.90%  | 65.74%  | 58.25%  | 45.71%   | 56.16%    |\n\n[1] Zhiqui Lin et al., Evaluating Text-to-Visual Generation with Image-to-Text Generation, ECCV 2024"}]

Author rebuttal:

Meta-review: This paper introduces CLIC, a simple and computationally efficient fine-tuning method to improve the compositional reasoning of vision-language models like CLIP. The core idea is to create challenging training examples by concatenating two image-caption pairs. This process is used to generate a diverse set of positive examples and hard-negative examples. The authors demonstrate that this method significantly improves performance on compositional benchmarks, particularly on the challenging SugarCrepe++ dataset, while maintaining or even enhancing performance on downstream retrieval and classification tasks.

The reviewers unanimously agree that the paper addresses a significant and well-known limitation in vision-language models. There were some concerns in the initial reviews, but authors addressed these concerns. The final review scores are three borderline accepts and an accept. Therefore, we have decided to accept this paper.
Decision: Accept (poster)