TripletCLIP:  Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
Authors: Maitreya Patel
Naga Sai Abhiram kusumba
Sheng Cheng
Changhoon Kim
Tejas Gokhale
Chitta Baral
Yezhou Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Contrastive Learning
Synthetic data
CLIP
Compositionality
TripletCLIP
Primary area: machine_vision
TL;DR: Contrastive synthetic dataset and triplet contrastive learning to imporve CLIP compositionality.

Abstract: Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. This makes the nature of the training data a significant factor in the efficacy of CLIP for downstream tasks. However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP. We show that generating ``hard'' negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution. We introduce a novel contrastive pre-training strategy that leverages these hard negative captions and images in an alternating fashion to train CLIP. We demonstrate that our method, named TripletCLIP, when applied to existing datasets such as CC3M and CC12M, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark on an equal computational budget, as well as improvements in zero-shot image classification and image retrieval. Our code, models, and data are available at: tripletclip.github.io.

Trip Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
Maitreya Patel R
Abhiram Kusumba
Sheng Cheng
Changhoon Kim
Tejas Gokhale 9
Chitta Baral
Yezhou Yang
Arizona State University
University of Maryland, Baltimore County
ripletclip-gituub.io
Abstract
Contrastive Language-Ing Image Pretraining (CLIP) models maximize the mutual in- formation between textual and visual modalities to learn representations. However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP. We show that generating "hard" negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers solution. We introduce novel contrastive pre-training strategy that leverages these hard negative captions and images in alternating fashion train CLIP. We demonstrate that our method, named TripletCLIP, when applied to existing datasets such as CC3M and CC12M, enhances the compositional capabilities of CLIP, resulting in an absolute improve- ment of over 9% on the SugarCrepe benchmark on an equal computational budget, as well as improvements in zero-she shot image classification and image retrieval. Our code, models, and data are available at: tripletclip. www.lip..gtbii......
Introduction
Large-scale vision-language models, such as CLIP [41], have significantly advanced multi-modal learning by employing contrastive learning acquire shared semantic representations from paired datasets. This approach has resulted in improved performance in vision-languagg tasks as well as zero-shot image classification [51] and segmentation [23, 62]. Beyond vision-lange language tasks, the individual components of these models, such as the vision encoder and the language encoder, are integral to several multimodal architectures and generative models such as multimodal large language models (MLLMs) [30, 26] and text-to-image (T21) diffusion models [45, 37, 38). Yet, compositional reasoning remains challenging and multimodal models continue to exhibit naïve "bag of words" behavior, frequently failing distinguish between expressions like "bulb in the grass" and "grass in the bulb" [58, 53]. Addressing this challenge remains critical for enhancing vision-language models and their downstream applications.
Contrastive learning of representations benefits from "hard negative samples" (i.e., points that are difficult to distinguish from an anchor point) [43]. However, at each optimization step for training CLIP, image-text pairs are randomly sampled from the training dataset- - this random sampling seldom exposes the model to highly similar negative pairs. We hypothesize that the limited compositional understanding of CLIP may stem from such issues the optimization objective and sampling from training datasets. A straightforward solution could involve iteratively identifying hard negative pairs for each training iteration. However, due to the noisy captions and the scarcity of such pairs existing datasets, prior work generates hard negative captions form of augmentation using rule-based strategies [58, 61]. For instance, given an image-text pair labeled "a brown horse". an additional
"Corresponding author: maitreya, patel asu.edu. t indicates the equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

negative caption "a blue horse" might be introduced. However in prior work, image data is not subjected to similar hard negative semantic augmentation during training; this is mainly because of the difficulty of making semantic perturbations at the pixel levels compared to sentence perturbation. While the text-only augmentation strategies have improved the models' compositional understanding to certain extent, it raises an intriguing question: could incorporating hard negative augmentation for both text and image modality further enhance the compositional reasoning capabilities of vision-language models?
Motivated by this, in this paper, we introduce a novel, simple, and yet highly effective strategy for integrating hard negative images as well as hard negative text to enhance the compositional understanding of uision-languag models. Recent developments in cext-to-imag diffusion models have opened up possibilities for performing semantic perturbations within images [21]. Existing works have evaluated the impact of creating synthetie data for text-to-image generative models [3. 6]. However, it remains less explored how these generative models can benefit the CL.IP-like models. To tackle this challenge, our approach leverages the in-context learning capabilities of L.MSs to produce realistic, linguistically accurate negative captions [55]. We then employ pre-trained text- to-image diffusion model to create images corresponding to these captions, thereby enriching any given image-text dataset with valuable hard negatives that foster improved reasoning. This resulting TripletData comprises 13M image-text pairs to complement the CC3M and CC12M datasets [5].
We developed TripletCLIP, which incorporates hard negative image-text pairs effectively by using them to optimize a novel triplet contrastive loss function. Extensive experiments on the CC3M and CC12M datasets and various downstream tasks with equal compute budget demonstrate that TripletCLIP significantly enhances compositional reasoning. Notably, TripletCLIP results in more than 9% and 6% absolute improvement on the SugarCrepe benchmark compared to LaCLIP and NegCLIP, respectively. TripletCLIP also improves zero-shot classification and image-text retrieval performance with similar training-time concept diversity. An investigation into the effects of increasing training-time concept diversity revealed that baseline models consistently under-performed in compositional tasks despite an increase in integrated knowledge, while TripletCLIP demonstrated significant improvements. In summary, our key contributions are as follows:
We introduce novel CLIP pre- pre-training strategy that employs hard negative images in conjunction with triplet contrastive learning to enhance compositionality.
TripletCLIP consistently improves across downstream tasks, demonstrating the effectiveness of synthesizing hard negative image-text pairs.
Our extensive ablations the choice of the loss function, modality-specifie pre-training, the increase in concept diversity, and filtering high-quality TripletData provide deeper insights into the utility of hard negative image-text pairs for CLIP pre-training.
Ultimately, we present a promising avenue where synthetic contrastive datasets significantly improve reasoning capabilities, leading the creation and release of the TripletData 13M contrastive image-text dataset.
2
Related Work
Vision-Language Models. Recent advancements, including ALIGN [22] and CLIP [41], have gained significant interest due to their capability to learn transferable semantic representations across multiple modalities through contrastive learning. These models facilitate downstream tasks such as zero-shot classification [51], image-text retrieval [60, 2], visual grounding/reasoning [31], text- to-image generation [52. 37, 38, 24], semantic segmentation [62, 23], and various evaluations [19, 47]. Subsequent research has sought to enhance various aspects of these models, including data efficiency [15], hierarchical representation learning [8], and the quantization of latent spaces for more stable pre-training [7]. LiT [59] employs pre-trained frozen CLIP vision encoder to fine-tune a BERT-liKe text encoder [9], achieving notable improvements in zero-shot transfer performance. Similarly, BLIP-2 [27] combines contrastive pre- pre-training with the next-token prediction for image captioning during training. However, these approaches generally presume the availability of high- quality data. In contrast, TripletCLIP focuses on leveraging the proposed hard negative contrastive dataset and incorporating triplet contrastive pre-training for compositional data. This approach orthogonal to prior works.

---

LyaL (Eq.
Ligeeui (Eq. 2)
heup (Eq. 1)
Exaeerip
Loggeui (Eq.
Visian (Fx)
Text (Fy)
Vision (Fx)
(Fy)
Vision (Fx)
Text (Fv)
Vision (Px)
x
y
y
X
y
x'
[a] CLIP
[b] NegCLIP
(c) TripletCLIP
Figure I:
Comparison of training workflows of CLIP, NegCLIP, and TripletCLIP, (x, v) represents positive a image-text pair, and (x',3') represents the corresponding negative image-text pair.
Data for Contrastive Pre-training. The effectiveness of maximizing mutual information between modalities heavily relies on the quality of extensive, web-scraped datasets that ideally encompass all possible concepts and knowledge. For instance, despite its noise, the LAION dataset [48, 16], which includes more than 5 billion internet images paired with alt-text captions, is primary resource. Studies show that over billion data points necessary to match the performance of the original CLIP model [16, 56]. Recent works like DataComp [16] and MetaCLIP [56] have focused on creating smaller, high-quality datasets by applying stringent filters and ensuring wordnet [33] synset-level concept diversity. Nevertheless, the inherently noisy nature of internet-scraped datasets can degrade model performance. Studies such SynthCLIP [18] demonstrate that tripling the volume of fully
synthetic data is required to equal the efficacy of real data. Other efforts like VeCLIP [25] and LaCLIP [14] enhance dataset quality by using generative language models re-caption existing images, significantly boosting performance.
Compositionality for vision-language. Despite the increased emphasis data quality and modeling techniques, mastering compositionality remains a significant challenge for vision-language models. Benchmarks like ARO [58], VALSE [35], and CREPE [32] have been developed assess models' abilities to handle compositional data. SugarCrepe [20], in particular, offers a large-scale, systematic framework for such evaluations. Previous methods primarily focused identifying hard negatives within existing datasets or generating synthetic negative captions [58, 61, 12, 11, 57, 49]. However, these rule-based generated captions are often unrealistic and linguistically flawed, leading subopti- mal model performance on complex datasets like SugarCrepe. A handful of works focus on finding negative images. [54] propose utilizing the video data. [44] focuses on object-centric image-editing to synthesize the negative images. [39] utilizes the simulation-bain data negative data.
Contrary prior approaches that predominantly add unrealistic negative captions very constrained negative images that are either very synthetic or which centers generating naturally occurring hard negative image-text pairs. We propose novel triplet contrastive learning strategy that effectively utilizes these challenging data pairs. Additionally, while our method is distinct, integrating advancements that refine contrastive learning could potentially boost TripletCIIP's efficacy further.
this work introduces TripletCLIP,
3 Method
This section begins with an overview of the contrastive learning algorithm used by CLIP and NegCLIP. We then describe the synthetic data generation pipeline for generating hard negatives using LLMSs and T2I models and introduce triplet contrastive learning which forms the basis of TripletCLIP. A high-level comparison between prior work and TripletCLIP can be found in Figure 1.
3.1 Preliminaries
The goal for self-supervised contrastive learning [13], when dealing with inputs from single modality, is to use a feature extractor (F)) to encode inputs and their augmentations and minimize

---

deiig
"Heart trimmed white on top of black frame."
"Heart trimmed in red on bottom of silver frame."
"A cartoon illustrations of stressed man taking cover from gunfire."
"A cartoon illustration of peaceful man hicing from rain."
"man breaks free From tackle curing the football game."
"man gets tackled during the basketballl game."
"A solid-colored background with with vintage a futuristic color color scheme." scheme."
"A patterned background
"small peany in the garden."
"large peony in the greenhouse."
"A photo of map with pins or stack of pins."
"A photo of map with marbles or a stack marbles."
Figure 2: Examples image-text pairs from TripletData. In each block, a positive pair from CC3M on the left and corresponding negatives from TripletDataare shown on the right.
the InfoNCE loss [34] between the two encodings. CLIP designed for multimodal settings (for example, vision and language inputs) this entails using encoders (one for each modality).
Let X' and y represent two modalities and = { yi Ji..i.....,MM. be the training dataset, where I; X and E y. The goal is to train two modality modality-specific -spccifcc encoders, Fx and Fy. by minimizing InfoNCE loss between the normalized features extracted from the encoders.
exp((Fx(as) Fy(m))/T) log ΣN exp((Px(x)), Fy(mx))/+>*
(I)
N
where (.) represents cosine similarity and is the trainable temperature parameter. For simplicity, do not show feature normalization in the InfoNCE loss. Similarly, can define the LCL training loss. The combined CLIP total training objective is given as, LOLIP CCL -x-y+ + CCL By minimizing this training loss, both encoders learn representations that maximize the mutual information between two modalities.
L-
y-xxx
NegCLIP introduces synthetic augmentations to generate "hard" negative captions (sl E y") by performing semantic inverting perturbations the reference captions (y/ y). Therefore, the single modality-specific hard negative augmentation-base training loss can be formulated as:
NegCL X----"'
-1 En exp((Fx(m)), Fy(w)))/7) -n exp((Fx(x)) Fo(A))))7 (2)
Σ log
(x))
N
The total loss for NegCLIP for image modality (2) and text modality ()) is given by:
CNegCLIP(Xx))))) ESL + Lx-y---
In Eq. 2, the negative samples are generated only for language modality as is easy to make semantic- level perturbations. Existing methods have not explored performing semantic perturbations in the image modality to create hard negatives. In this work, we demonstrate how hard negatives can be created in the image modality by leveraging the semantic language grounding and photorealism of text-to-image diffusion models. Our novel hard negative generation pipeline and refined training objective seeks to bridge the significant gap identified in literature.

---

3.2 TripletData: Image-text hard negative data augmentations
To generate high-quality hard negative image-text pairs, we follow a two-step procedure. The first stage is to generate hard negative captions from the ground truth positive caption. Second, to generate images corresponding to the hard negative captions as negative images. The AlíText captions from the existing web-scrapped datasets are very noisy, leading the noisy and unreliable generation of hard negatives. Therefore, we build upon the existing work LaCLIP, which first rewrites the captions using LLM from the existing data that are linguistically accurate. Figure 2 illustrates several examples of positive and corresponding negative image-text pairs.
Generating hard negative captions. Existing works perform random swapping, replacing, and adding actions between the nouns, attributes, and relations of the positive caption [58, 61]. This method results in nonsensical and grammatically incorrect artifacts, such as "a person riding on four slope." which impedes the generation of negative images, ultimately leading to diminishing performance on harder benchmarks [20]. Therefore, we utilize the in-context learning ability of LLMs to generate negative captions. The choice of LLM is a trivial task as long as they provide hard negative captions. We find that -
performs reasonably better our goal, and the output is easy to parse. We generate the negative captions in batches to speed up the generation process. Generating the 13M negative captions takes only 3 days 8xRTX A6000, Instead of generating multiple hard negative captions, we find that a single high-quality hard negative caption is enough to improve the performance compared to the traditional NegCLIP style caption generation (see NegCLIP++ results in Table 4). We provide examples of various types of negative captions in the appendix. Specifically, we provide the following prompt to LLM:
You are given the description of an image. You should provide mostly similar description, changing the original one slightly but introducing enough significant differences such that the two descriptions could not possibly be for the same image. Keep the description length the same. Finally, only a few things (such as counting, objects, attributes, and relationships) can be modified to change the image structure significantly. Provide just the updated description. Examples: Input: dog the left of the cat. Output: A dog the right of the cat. Input: A person wearing a red helmet drives motorbike on dirt road. Output: A person in a blue helmet rides a motorbike on a gravel path. Now, do the same for the following captions: Input: (} Output:
Generating hard negative images. Typically, semantic perturbations within images require tools like image editing, which are resource-intensive and cannot be scaled. Remember that we want to provide additional ground truth references for the negative caption. Therefore, we propose to utilize the negative captions from the previous stage to generate the respective reference images directly for pre-training. As the previous stage generates negative captions that are linguistically correct, it becomes easier for image-genan gengeeegnrrttvee models to synthesize the respective images precisely. We utilize pre-trained ext-to-image diffusion models to generate the corresponding images. Specifically, we select SDXL-turbo [46] due to its relatively faster generation speed. After applying various inference time optimizations, can generate 13M negative images within 2 days using 30 v v100 GPUs. We provide various examples of the hard negative image-text pairs in the appendix.
Analyzing difficulty of the hard Triplet- Data. Let's assume we have positive and neg- ative image-text pairs from the TripletData, (I,, y:) and (a%, v(), respectively. If the data is truly hard negative, existing pre-trained mod- els should struggle to find the correct image- text pairs (i.e., co5 v:) > cos(E,, v()). Fol- lowing winoground, we measure the text-score, image-score, and group-score to evaluate the popular pretrained CLIP models. Table 1 shows that even CLIP models trained on billions of data struggle to get near human performance on TripletData, which is less difficult than winoground. Importantly, the goal of generating hard negative samples isn't to add more diversity
Table I: Winoground-style evaluation of pretrained CLIP models on TripletData.
Ing Score
Text Score
Grp Score
B/T-
40.29
68.17
36.53
VII-L/14 ViT-bigG
44.84
69.21
40.91
42,94
77.61
40.98
Siglip so400m
44.24
71.27
26. 10
Humans (on Winoground)
88.50
89.50
85.50
Table 2: Wordnet synset analysis of captions from CC3M and TripletData.
TripleDData (Negative Only)
CC3M
TripletData Intersection
# unique
59094
59616
62741 446M
55969
toial synsets
231M
215M
Pattps://huggingtann http:///wwaaaccco/////aaaaaaaaaa htt.://ttt...ttt--ttatttttttto.. Instruct-v0.2
5

---

hence, the Table 3: Importance of image-text hard negatives. We measure the importance of various modality- specific hard negatives on SugarCrepe, image-text retrieval, and ImageNetIk. We find that Triplet- CLIP results into the most optimal solution. Bold number indicates the best performance.
Models
Negative Captions
Negative Images
SugarCrepe
Retrieval
ImageNetlk
LaCLIP
x
54.09
8.19
3.79
NegImage
56.28
9.20
4.48
++gCLIP++
61.69
8.36
3.84
TripletCLIP
63.49
16.42
7.31
in terms of unique concepts during the training but to add diversity in semantic meanings. Therefore, we measure the unique wordnet synsets in CC3M vs. TripletData. From Table 2, it can be observed that TripletData does not add any new concepts but uses existing concepts to provide negative samples that are semantically different. To summarize, TripletData contains the relatively hard negative image-text pairs that current models find difficult to differentiate.
3.3 TripletCLIP
Prior works have demonstrated the value of hard negative captions for enhancing the compositionality of CLIP models via LNegCLIP as the key training objective (Eq. 2) [58, 61]. However, it remains elu- sive if negative images alone can benefit or not. We conduct modality-specine average performance across the diverse set of benchmarks in Table 3 (we provide more details about experiments in Section 4). Our findings indicate that both "hard" negative captions and images indi- vidually boost performance when compared to LaCLIP. However, this initial empirical experiments to train the CL.IP model on hard negative images (i.e., Neglmage by minimizing (DegCCLi((VV X, X") reveal that negative images alone cannot improve the compositionality significantly (see Table 3). We hypothesize that images contain low-level information, making it difficult to train the model using images as negative examples. Aligning with our initial motivation and building upon this crucial insight, we propose to utilize the negative images to regularize the effect of negative captions and to stabilize the pre- mre-training. Therefore, to utilize these hard negative image-text pairs from the previous stage more effectively, we propose to focus on two triplets (V )") and (X', final triplet contrastive learning training objective is defined as:
ablations, reporting the
Crcr- LNeGCLIP (X,Y, - LNeSCLIP(X', (X,,," ,",
(4)
Intuitively, the second term introduces the additional form of supervision that hard negative images closer to the corresponding negative captions than positive captions. This allows the system to understand that if the positive image does not represent the negative caption "blue horse," then what does this caption entail? Through this strategic alternation of hard negative image-text pairs for the TripletCLIP, we improve compositionality and image-text understanding of the vision-language model (see Table 3). We provide the pseudo-code in the appendix and the code in supplementary materials. This simple yet effective strategy elevates the training of the CLIP, offering a scalable framework improve overall performance.
4 Experiments & Results
4.1 Experiment Setup
We utilize the CC3M and CC12M datasets, which comprise 2.6M and 8.6M image-text pairs, respectively. Following the approach demonstrated by LaCLIP, we use LLM- rewritten captions to replace noisy original captions. For NegCLIP, we introduce four negative captions per positive image-text pair, focusing on semantic inverting perturbations across four categories: attribute, relation, object, and action [61]. This generates approximately 10.4M and 34.4M text-only augmentations for CC3M and CC12M, respectively. To train the TripletCLIP, we create augmentations (TripletData) for both datasets to integrate hard negatives effectively. We produce one augmentation per image-text pair, adding 2.6M and 8.6M image-text augmented pairs for CC3M and CC12M, respectively. Finally, perform all the ablations on the CC3M dataset.
Pretraining Datasets.

---

Table 4: Composition evaluations of the methods on SugarCrepe benchmark. Bold number indicates the best performance and underlined number denotes the second-best performance. represents the results taken from SugarCrepe benchmark.
Replace
Swap
Add
Overall
Methods
Object
Attribute
Relation
Object
Attribute
Object
Attribute
Avg.
LaCLIP
39.44
53.17
51.42
$4.69
49.25
$5.29
55.35
54.09
DataComm CC12M CC3M
LaCLIP
55.96
50.71
50.60
56.98
51.16
NegCLIIP
62.71
58.12
54.48
56.33
51.20
56.26
61.13
57.18
MxgCLIP++(owm)
64.77
66.12
68.93
$5.51
55.41
59.65
64.45
61.69
TripletCLIP (ours)
69.92
69.03
64.72
56.33
57.96
62.61
63.87
63.49
Performincc Gain wiz.f LaCLIP
T0.48 %
18.50 %
13.30%
1,64%
8.71%
7.32%
8.52%
9.40%
LaCLIP
75.06
65.48
58.68
53.47
57.66
67.65
66.76
63.54
NegCLIP
77.84
69.29
63.23
66.53
62.31
67.17
69.65
68.00
NegCL. (ours)
82.99
78.68
75.75
61.63
65.47
70.08
76.01
72.94
TripletCLIP (ours)
83.66
81.22
79.02
64.49
63.66
73.67
75.43
74.45
Performance Gain wish LaCLIP
8.60%
15.75%
20.34%
11.02%
6.00%
8.67%
7.35%
10.91%
small: VIT-R/32' (13M)
smal1:ViTBB32
56,90
56.85
51,99
50.81
50.00
53.93
60.55
54.43
medium: ViT-B/32' (128M)
77.00
69.54
57.68
57.72
57.06
66.73
64, 88
64.37
Lerge:ViT-B/Ik VIT-B/16 (IB)
large:
92.68
79.82
63.94
56.10
57.66
84.34
78.61
73.31
xlarge: VIT-L/14" (13B)
95.52
84.52
69.99
65.04
66.82
91.03
84.97
79.70
Table 5: Zero-shot image-text retrieval and classification results. Bold number indicates the best performance and underlined number denotes the second-best performance.
Retrieval (R@5)
Zero-shot Classification
Methods
Image-to- Text
Text-to-Image
VTAB
ImageNettk
MSCOCO
Flickr30k
MSCOCO
Flickr30k
top-1
top-5
top-1
top-5
LaCLIP
5.06
10.90
5.97
10.84
11.56
34.72
3.79
10.49
CC3M
LaCLIP HN
8.08
16.10
8.64
16.64
12.31
37.14
5.75
15.22
NegCLIP
6.32
13.80
6.61
12.96
12.25
36.38
4.67
12.69
NegCLIP++ (ours)
5.8
11.20
6.19
10.24
11.65
35.47
3.84
10.52
TripletCLIP (ours)
10,38
22.00
11.28
22.00
12.31
41.45
7.32
18.34
Performance Gain
5.32%
11.1%
5,31 %
11.16%
0.75%
6.73%
3.53%
7.85%
CC12M
LaCLIP
25.86
42.70
19.78
36.30
19.08
49.06
19.72
41.39
NegCLIP
30.16
46.60
23.11
41.70
19.12
50.56
20.22
42.63
NegCLIP++ (ours)
26.96
43.90
22.69
42.86
18.48
50,38
19.06
40.91
TripletCLIP (ours)
33,00
$5.90
28.50
52,38
20.81
53.40
23.31
47.33
Performance Gain
7.14%
13.2%
8.72%
16.08%
1.73%
4.34 %
3.59%
5.94%
Baselines. We train LaCLIP, LaCLIP with real hard negatives (LaCLIP+HN), and NegCLIP from scratch to ensure consistency and fairness in comparisons. As NegCLIP's rule-based augmenta- tions closely resemble some compositional benchmarks, so we introduce NegCLIP++ as an improved baseline. NegCLIP++ incorporates hard negative captions generated using LLM from TripletData, enhancing the language comprehension compared to standard NegCLIP.
Implementation Details. Our experiments employ the ViT-B/32 [10] model architecture. To guarantee fair comparisons, we retrain all baseline models using identical hyperparameters. Since the overall training data for NegCLIP and TripletData is more than the baseline datasets, we align the number of iterations across all models to equalize the number of image-text pairs seen during training. similar to the strategy used in DataComp. The batch size is fixed to 1024 with the AdamW optimizer at a maximum learning rate of 0.0005, employing cosine decay. Training durations are set approximately 100k iterations for CC3M and 200k iterations for CC12M. All models are trained on single A 100 (80GB) GPU using bf16 precision. The final training-relatee ablations will cost about 1200 A100 GPU hours. We leave the experiments on increasing the data and model size as future works for the community, as scaling further is not viable in the academic budget.
experiments and
Downstream Datasets. The primary objective of this study is to enhance the compositional ca- pabilities of CLIP models. We mainly evaluate TripletCLIP and the baseline models using the challenging SugarCrepe composition benchmark, with additional performance assessments provided the appendix for older benchmarks. Models are also tested on image-text retrieval tasks for broader evaluation using the Flickr30k [40] and MSCOCO [29] datasets. Zero-shot classification performance

---

Table 6: Ablation on filtering high-quality image-text pairs from TripletData. We evaluate the TripletCLIP after applying the filters to ensure the quality similar to DataComp and compare the baselines on three benchmarks. We find that TripletCLIP results in the most optimal solution. Bold number indicates the best performance. i represents that results are borrowed from DataComp.
Models
Filtering Strategy
Data Size
Augmentations
SugarCrepe
Retrieval
ImageNetlk
No filtoring
12.8
55.61
6.49
2.7
CLIP
CLIP Score
3.8
57.31
9.08
Image-based n CLIP Score
54.75
5.63
3.9
LáCLIP
No filtering (CC3M)
2.6
54.09
8.19
3.79
TripletCLIP
No filtering (CC3M)
2.6
2.6
63.49
16.42
7.31
IripletCLIP++ CLIP Score (from CC12M)
1.4
66.09
T9.85
8.85
is assessed across approximately 18 different datasets. Evaluations adhere to the methodologies outlined in the CLIP-Benchmark or the official benchmark implementations.
4.2 Compositional reasoning
We comprehensively analyze the compositional understanding of models on the SugarCrepe bench- mark, detailed in Table 4. Notably, TripletCLIP consistently outperforms all baseline models across all sub-categories SugarCrepe on both the CC12M/CC3M training datasets. Specifically, TripletCLIP surpasses LaCLIP and NegCLIP by 10.91 %%.44% and 6.45% % on the CC12M and CC3M datasets, respectively. Our enhanced baseline, NegCLIP++, also shows improvement over standard NegCLIP, highlighting the benefits of LLMM-generater negatives. Nevertheless, Triplet- CLIP further advances performance, underscoring the critical role of hard negative image-texi pairs, not just text. Additional comparisons and Winoground [53]) in the appendix reveal TripletCLIP's consistent performance. Table also contrasts TripletCLIP with models trained using the DataComp approach, which involves more parameters and training data, demonstrating that TripletCLIP achieves comparable performance to ViT-B/16 model trained on 1 billion image-text pairs.
older composition benchmarks (Valse [35], Cola [42],
4.3 Zero-shot evaluations
Image-Text Retrieval. In Table summarize the performance of models on text-to-image (T2I) and image-to-text (I2T) retrieval tasks on MSCOCO and Flickr30k datasets, where we report @@5 scores. Remarkably, TripletCLIP significantly outperforms baseline models by an average of 8%/10% and 8 8%11.%%% on 12T and T2I tasks, respectively, on the CC3M and CC12M datasets. Intriguingly, while LaCLIP+HN performs better than NegCLIP, TripletCLIP outstrips both.
Zero-shot Classification. Table also presents the average zero-shot classification performance on 18 standard datasets, including ImageNetlk. TripletCLIP consistently enhances top-1 accuracy by an average of 3% and top-5 accuracy by 5-7% compared to LaCLIP. Like the retrieval performance, LaCLIP+HN exceeds NIIC yet TripletCLIP maintains the highest performance. Dataset- specific results are in the appendix.
4.4 Finetuning performance
In this paper, we focus on pretraining bbased experiments as they allow greater flexibility in learning better representations. To complement this, also performed additional fine-tuning experiments using hyperparameters similar to the baselines (without LoRA) and compared them against various publicly available baselines [44, 50, 12, 11]. As reported Table 7, TripletCLIP improves com- positionality and outperforms nearly all baselines. Furthermore, the observed drop in retrieval and zero-shot classification performance (Table 16) is attributed to limitations the vision encoder, high- lighting the challenges of existing pre-trained vision encoders in capturing semantic representations. This is further demonstrated in Table 8.
hhttps://ggtaiii http://ggttuuu.ccoAAAA// AI/CLIP benchmark

---

Table 7: Finetuning- composition evaluations of the methods on SugarCrepe benchmark. Bold number indicates the best performance and underlined number denotes the second-best perfor- mance.
Replace Attribute
Swap
Add
Overall
Methods
Object
Relation
Object
Attribute
Object
Attribute
Avg.
CLIP
90.92
80.08
69.13
61.22
64.26
77.16
68.64
73.06
CLIP (finetuned)
90,92
79.69
64.01
60.82
64.26
84.67
78.76
74.73
NegCLIP
91.53
83.25
73.97
72.24
67.72
86.95
88
80.59
Baseline [44]
93.22
84.39
67.35
62.04
70.12
88.31
79.48
77.84
CoN-CLIP [50]
63.3
87.29
79.62
75.58
TSVLC (RB) [12]
91.34
81.34
64.15
68.16
69.07
79.49
91.33
77.84
TSVLC (LLM+RB) [12]
88.13
76.78
62.73
64.08
66.67
75.80
81.07
73.61
[1C
94,43
89,48
84.35
75.10
74.17
89.67
97.69
86.41
TripletCLIP (ours)
94.43
85.53
80.94
69.80
69.82
90.40
86.27
82.46
Table 8: Frozen encoder ablation. LiT style fine-tuning ablations on SugarCrepe, image-text retrieval, and ImageNetIk. Bold number indicates the best performance.
Models
Train Text
Train Vision
SugarCrepe
Retrieval
ImageNetlk
LaCLIP
0.6373
0.5345
31.21%
TripletCLIP (ours)
X
0.6227
0.6817
34.25%
LaCLIP
X
0.5886
0.1134
5.51%
TripletCLIP (ours)
X
0.6923
0.2626
12.51%
4.5 Ablations
Can a high-quality filtered dataset improve the performance? Given that negative images in TripletData are generated using SDXL-turbo, these may not always be precise. Inspired by Data- Comp, we employ a pre-trained CLIP-L/14 to filter the image-text pairs, selecting the highest average similarity pairs (positive and negative) individually (i.e., score = (s(x, y:) + s sa /2). The top 1.4M positive image-text pairs and their corresponding negatives from TripletData Table 6 details this comparison against DataComp pre-trained models. Remarkably, TripletCLIP al- ready surpasses baselines without filtered data; however, with the filtered dataset, despite being trained on 50% smaller dataset, TripletCLIP++ shows further performance improvements. This underlines the significant benefits of carefully selected TripletData in enhancing the performance.
selected.
Which modality-speeffic encoder plays the key role in improving compositionality? this open question, we designed an ablation study similar to LiT, freezing either the pre-trained CLIP vision or text encoder while training the opposite modality-specific encoder from scratch. We observe the performance of LaCLIP and TripletCLIP the vision model results no performance gain on the SugarCrepe for TripletCLIP. However, significant improvements are noted when the vision encoder actively trained, suggesting that the vision modality may be the bottleneck in compositionality. Notably, TripletCLIP outperforms LaCLIP in all settings, further demonstrating its robustness different pre-training approaches.
address
CC3M, as shown in Table 8. Freezing
Concept coverage analysis. Improving performance on zero-shot transfer learning tasks such as retrieval involves two key components: adding more concept diversity during training and enhancing image-text alignnen/comppsttionaitt, We create subsets of CC12M data with increasing concept diversity based on unique WordNet synsets. Specifically, we select 3M, 4M, 5M, and 6M subsets for training LaCLIP, while TripletCLIP training involves only half these training data as positive pairs, and the rest corresponding augmentations. Evaluations across SugarCrepe, retrieval tasks, and ImageNetlk (see Figure 3) indicate that TripletCLIP not only enhances SugarCrepe performance even at lower concept coverage levels but also significantly outperforms similar concept coverage in retrieval tasks, matching LaCLIP's performance on zero-shot classification tasks that do require compositionality at all. This bolsters argument that incorporating hard negatives from both modalities markedly improves compositional understanding CLIP, while baseline struggles to do even with more concept diversity.
What Tri TripletData used for large-scale compositional evaluations? evaluated the CC12M tre-trained models 50,000 random subset of the CC3M dataset using Winoground-styllc

---

SugarCrepe
Tnate-Textaae Retrieval
ImageNetlk Performance
0.32
0.30
14.00
0.64
0.26
0.62
10.00
0.20
LaCLiP
TipisCUP
66000
67000
68000 6:000 Concept Coverage
70000
66000
67000
68000 69000 70000
66000
67000
68000 69000 70000
Coverage
Cancept Coverage
Figure 3: Average Results of LaCLIP and TripletCLIP for SugarCrepe Compositions, Image-Text Retrieval, and ImageNetlk over increasing concept diversity.
Table 9: TripletData as large-scale composition evaluation dataset after [53].
Methods
Text-Score Image-Score
Group-Score
CLIP
52.69
29.66
24.64
NegCLIP
54.84
30.42
25.82
NegCLIP++
36.50
30.67
20.11
TripletCLIP (ours)
92.25
66.82
64.30
approach [53]. As shown Table 9, TripletCLIP significantly improves performance compared to the baselines. However, partially attribute this improvement to spurious correlations learned from the data. At the same time, we note that the models have not fully converged, suggesting minimal risk of overfitting to these spurious correlations.
5 Conclusion
In this work, introduce TripletCLIP, novel approach enhancing compositional reasoning in vision-language models through the strategic incorporation of hard negative image-text pairs. Our comprehensive experiments across suite of benchmarks demonstrate that TripletCLIP significantly outperforms existing methodologies such as LaCLIP and NegCLIP, achieving notable gains not only in compositionality but also in zero-shot classification and retrieval tasks well. Further, our ablation studies highlight the critical role of modality-specific training and the careful curation of training data, underscoring the importance of both hard negative image and text components in the learning process. TripletCLIP's effectiveness with a smaller, refined dataset suggests promising direction for future research-maximizae maximizing performance without the need for extensive data collection, thereby reducing computational costs and enhancing model efficiency. To this end, we provide an intriguing application of synthetic datasets via hard negative image-text pairs for vision-language tasks that could be easily extended to improve Multimodal Large Language Models and Text-to-Imme generative models.
Limitations. Due constraints inherent academic settings and limited computational resources, we were unable to scale TripletCLII to handle hundreds of millions of image-text pairs or employ larger models within the scope of this study. Nevertheless, our results indicate promising direction for future research within consistent experimental framework, and we encourage subsequent work to explore scaling both the TripletData and TripletCLIP. Our experimental focus was primarily on the CLIP and LiT methodologies. With additional resources, however, extending our methodologies to more advanced contrastive learning techniques, such as SigLIP, would feasible. In conclusion, our work introduces a compelling strategy for integrating open-ended hard negatives (both text and image) during the pre-training phase, providing a methodology and large-scale data that could benefit variety of research domains.
10

---

Acknowledgments
This work was supported by NSF RI grants #1750082, #2132724, and CPS grant #2038666. We thank the Research Computing (RC) at Arizona State University (ASU) for providing computing resources. The views and opinions of the authors expressed herein do not necessarily state or reflect those of the funding agencies and employers.
References
[1] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint- embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15619-15629, 2023.
[2] Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Composed image retrieval using contrastive learning and task-oriented clip-based features. ACM Transactions Multimedia Computing, Communications and Applications, 20(3):1-24, 2023.
[3] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufci Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3 pdf. 2(3):8, 2023.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-96600 2021.
[5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail tail visual concepts. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3558-3568, June 2021.
[6] Agneet Chatterjee, Gabriela Ben Melech Stan, Estelle Aflalo, Sayak Paul, Dhruba Ghosh, Tejas Gokhale, Ludwig Schmidt, Hannaneh Hajishirzi, Vasudev Lal, Chitta Baral, et al. Getting it right: Improving spatial consistency in text-to-image models. In European Conference Computer Vision, pages 204-222. Springer, 2024.
[7] Yuxiao Chen, Jianbo Yuan, Yu Tian, Shijie Geng, Xinyu Li, Ding Zhou, Dimitris N Metaxas, and Hongxia Yang. Revisiting multimodal representation in contrastive learning: from patch and token embeddings to finite discrete tokens. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 15095-15104, 2023.
[8] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ra- makrishna Vedantam. Hyperbolic image-text representations. In International Conference on Machine Learning, pages 7694-7731. PMLR, 2023.
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
[11] Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante- Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and aligned captions (dac) promote compositional reasoning in vl models. Advances Neural Information Processing Systems, 36, 2024.

---

[12] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky, Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2657-2668, 2023.
[13] Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales. Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Processing Magazine, 39(3):42-62, 2022.
[14] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024.
[15] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Representations, 2023.
[16] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
[17] Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704-677 6719,, 2022.
[18] Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. Synthclip: Are we ready for a fully synthetic clip training? arXiv preprint arXiv:2402.01832, 2024.
[19] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7522 2021.
[20] Cheng-Yu Yu Hsich, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugar- crepe: Fixing hackable benchmarks for vision- compositionality. Advances in Neural Information Processing Systems, 36, 2024.
[21] Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao. Diffusion model-based image editing: A survey, arXiv preprint 24X. 2024.
[22] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904 4916. PMLR, 2021.
[23] Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao, and Humphrey Shi. Learning mask-aware clip representations for zero-shot segmentation. Advances Neural Information Processing Systems, 36:35631 2023.
[24] Changhoon Kim, Kyle Min, and Yezhou Yang. Race: Robust adversarial concept erasure for secure text-to-image diffusion model. arXiv preprint arXiv:2405. 16341, 2024.
[25) Zhengfeng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, al. From scarcity to efficiency: Improving clip training via visual-enriched captions. arXiv preprint arXivv2310.07699 2023.
[26] Hugo Laurençon, Léó Tronchon, Matthieu Cord, and Victor Sanh. What matters when building lasion- models?, 2024.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning pages 19730-19742. PMLR, 2023.
12

---

[28] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.0328// 2023.
[29] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer Eision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014.
[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages Inc., 2023.
Curran Associates,
[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[32] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910-10921, 2023.
[33] George A. Miller. Wordnet: a lexical database for english. Commun, ACM, 38(1 nov 1995.
[34] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: arXi:: 000///80700043 2018
[35] Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, lacer Calixto, and Albert Gatt. Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8253-8288 2022.
[36] Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang. Conceptbed: Evaluating concept learning abilities of text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 44554114622 2024.
[37] Maitreya Patel, Changhoon Kim, Sheng Cheng, Chitta Baral, and Yezhou Yang. Eclipse: A resource-efficient text-to-image prior for image generations. arXiv preprint arXiv:2312.04655, 2023.
[38] Maitreya Patel, Changhoon Kim, Sheng Cheng, Chitta Baral, and Yezhou Yang. Eclipse: A resfurte-efficient text-to-image prior for image generations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9069-9078, 2024.
[39] Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize diagnose and optimize: Towards fine-grained vision-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13279-13288, 2024.
[40] Bryan Plummer, Liwei Wang, Chris Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer smage-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8788688 0763.876.3110 PMLR, 2021.
[42] Arijit Ray, Filip Radenovie, Abhimanyu Dubey, Bryan Plummer, Ranjay Krishna, and Kate Saenko. cola: benchmark for compositional text-to-image retrieval. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[43] Joshua Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint 2XXi::001000452. 2020.
[44] Ugur Sahin, Hang Li. Qadeer Khan, Daniel Cremers, and Volker Tresp. Enhancing multi- modal compositional reasoning of visual language models with generative negative mining, In Proceedings of the IEEE/CVP Winter Conference on Applications of Computer Vision, pages -5663,,3,, 2024.
[45] Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403.120115 2024.
[46] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042 2023.
[47] Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metries with 1isccresccree (ls2). arXiv preprint wrXiv:240..04251, 2024.
[48] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 2022.
[49] Harman Singh, Pengchuan Zhang, Qifan Wang, Mengjiao Wang, Wenhan Xiong, Jingfei Du. and Yu Chen. Coarse-to-fine contrastive learning in image-text-graph space for improved vision- language compositionality. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 869- 893,9893.220 2023.
[50] Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. Learn" no" to say" yes" better: Improving vision-language models via negations. arXiv preprint arXiv:2403.20331 20311 2024.
[51] Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. In Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pages 6088-6100, 2022.
[52] Ming Tao, Bing-Kun Kun Bao, Hao Tang, and Changsheng Xu. Galip: Generative adversarial clips for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14214-14223, 2023.
[53] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238-5248, 2022.
[54] Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang. Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11998-12008, October 2023.
[55] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Supper-naturalinstrutiiosss Generalization via declarative instructions on 1600+ nlp tasks. In 2022 Conference Empirical Methods in Natural Language Processing, EMNLP 2022, 2022.
[56] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint ar 21X:::3309996711 2023.
14

---

[57] Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li. Ziyong Feng, Jia Guo, Jing Yang. and Tongliang Liu. Alip: Adaptive language-inan image pre-training with synthetic caption. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 2922-2931, 2023.
[58] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou, When and why vision-language models behave like bags-of-worde and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
[59] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133, 2022.
[60] Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Wei Chang, Magiclens: Self-supervised image retrieval with open-ended instructions. In Forty-first International Conference on Machine Learning, 2024.
[61] Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross- modal hard negatives to enhance visio-linguistic fine-grained understanding. arXiv preprint 00X.02330.08882. 2023.
[62] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11175-11185, 2023.
15

---

A Broader Impact
In this study, we have demonstrated the potential of utilizing high-quality positive and negative pairs to enhance the compositional understanding of vision-language models like CLIP through the introduction of TripletCLIP. While our findings specific TripletCLIP, the underlying techniques hold promise for broader applications, including enhancing visual understanding in Multimodal Large Language Models (MLLMs) and Eext-to-Image diffusion models. Although our approach yields significant performance improvements, it does require resources to generate large- scale synthetic datasets. We encourage future research to explore the utility of this pre-training strategy within the latent space, which could reduce dependence on large generative models. Initiatives like JEPA [1] have already demonstrated the efficacy focusing on latent space models, which suggests promising avenue for reducing computational overhead. Importantly, our experiments reveal that significant enhancements in model performance are achievable even with substantially smaller data scales. This finding suggests that, when scaled appropriately, our methodology could substantially diminish resource dependencies and enhance the efficiency of pre-training processes for CLIP-like models.
B Pseudocode of TripletCLIP
def forvard(batch. batch_negative): get positive and negative image-text pairs img-pos, txt_pos batch_positive img_neg, txt_neg batch_negative
compute positive image text representations featueeseetuue__pee text_ fext__eeature__po = clip. encppencodeeeele (ex((tt__oo)
(ing.poo))
compute negative image and text representations image_features _eee clip. sdippenncde_imee imaeeeemmeeeeeeeee (img_neg) text_features_nte
siip.enncoe_teee (txt_netxxnegge
positive_txt torch. cat (atttt((tfttteeeeeeeeeeeeeeeeeee negative_txt torch. cat wat.e/atttttttttteeeeeee [[text_features_nng text_features__pos]
fext__estture__neg]
# compute NegCLIP losses
loss_1 1 negclip_lose lolsssp__sssseeeel (image_features www.fffettepppeee positive_txt loss_2 2 negelip_ losa (image_featureee features_negg nng, negative_tzt)
loss loss_1 loss_2
return loss
C Hyperparameters
Table 10: Detailed pre-training ablations.
for CLIP training across various experiments and
Hyperparameters
CC3M
CC12M
LiT
Concept Coverage Ablations
Batch size
1024
1024
1024
1024
Optimizer
AdamW
AdamW
AdamW
AdamW
Learning rate
5 x 10
5 10-4
5 x 10
5 x 10 10-4 0.5 (0.9, 0.999)
Weight decay
0.5 (0.9, 0.999)
0.5
0.5
Adam 3
(0.9, 0.999) 1 x 10
(0.9, 0.999)
Adam
x 10 8
x 10
x 10 $
Total steps
90,000
230,000
90,000
200,000
Learning rate schedule
cosine decay
cosine decay
cosine decay
cosine decay
Table 10 provides a comprehensive overview of the pre-training hyperparameters employed across all baseline models and TripletCLIP. To ensure fair comparisons, standardized the hyperparame-
16

---

Table 11: Composition evaluations of the methods various benchmarks. Bold number indicates the best performance and underlined number denotes the second- performance.
Methods
Valse
Cola
Wineground
SugarCrepe
Overall
Txt21mg
1mg21xt
Group
Tx222mg
Ingg2
Group
LaCLIP LaCLiP+ HN
43,19
28.10
13:33
10.48
24.75
9.25
6,00
54.09
23.65
47.91
20.95
5.54
2.38
23.25
4.25
53.92
20.15
NegCLIP
18.06
21.42
14.76
$.57
21.00
8.50
5.25
57.18
23,09
NegCLIPas
43,65
29.05
13.33
7.14
22.00
5.50
3.25
61.69
23.20
TripletCLIP (oues)
48.36
25.11
[TipleCCLIP++ooou
48.53
31.43
15.71
12.86
22.25
9.50
6.75
66.09
26.64
Performance Gain m.rd. LaCt.P
5.34%
3,33%
2.38%
2.380%
-2.50
0.25%
0.75%
T2.00"
2.09%
LaCLiP
$5.69
20.95
15.71
7.62
26.25
8.00
6.25
67.21
25.96
NegCLIP
58.59
NegCLIP++ (ours)
58.12
33.33
11.43
7.14
23.50
7.75
5.50
73.05
27.48
TriplerC1.PP (ours)
57.57
27.62
19.53
11,43
23.25
6.25
4.25
74L55
28,06
Performunce Rizz. LaCLIP
11.8%
3325
3381%
7.33
2.10%%
Table 12: Dataset-specific zero-shot classification results. Bold number indicates the best perfor-
mance and underlined number denotes the second- best performance.
Ser deren objoct albot--itaae ditaan
Seprite lald anlderaattaii artuntatiiss
caallbott kihel wiherth
Sgm dee-de
Mbodde
Arg.
LaCLiP
IU54
3.79
24.55
EJAS
12.99
11.28
19442
2,83
3.14
5-85
3.55
3,43
26-72
40.15
3.33
5.75
30.MA
6.66
11.56
NegCLIP
12.37
12:93
6.31
2.39
12.35
Neat3LPPe pari
1/2
3364
2861
6.46
22413
10.00
18.91
2.00
3.21
4.85
11.70
3.50
33.M1
-9.34
4.34
5.43
304G
7.20
11.65
1.29
14.39
13:45
15:37
8:75
22.24
10.33
12:31
LacLiP
1.99
19:22
23.23
11:9
19.55
2.01
13.00
2385
13433
24455
51.78
6.34
31.08
9:58
12:15
NeyCLIPP
1.55
20.31
53.45
26.00
18.09
13.80
15.30
2.45
3.13
16.55
11.3
13.55
22.78
30.2
3025
5.37
12,04
10.34
19.12
TripleC
1.24
18.4
31.38
17.69
17.45
850
13.8
15.
38.00
ters across all methodologies. Although larger batch sizes are typically associated with improved performance in contrastive learning, computational constraints necessitated fixing the batch size at 1024 for all experiments. To accommodate this batch size on single A100 GPU, we employed bf16 precision. In terms of computational resources, experiments using the CC3M dataset required approximately 16 GPU hours, while those involving the CC12M dataset utilized up to 56 GPU hours per experiment.
D Detailed Results
D.1 Compositional reasoning
Previously, we reported the results on SugarCrepe, the most challenging dataset, noted for its absence language biases. However, evaluations were also conducted on other benchmarks, such as Valse, Cola, and Winoground. As indicated in Table 11, TripletCLIP achieves overall improvements of 2-3% % compared LaCLIP and NegCLIP. The Valse benchmark, which contains text prompts that heavily favor the perturbations made for NegCLIP, shows strong performance from NegCLIP, while NegCLIP++ encounters difficulties. Interestingly, TripletCLIP faces challenges in maintaining performance on Winoground, and baseline LaCLIP maintains the SOTA, which is counterintuitive to other benchmarks. Nonetheless, TripletCLIP still manages to outperform NegCLIP significantly. These results affirm that TripletCLIP sets standard for state-of-the-an compositional reasoning across diverse benchmarks.
D.2 Dataset-specific zero-shot classification
Table 12 provides fine fne-grained results for the 18 zero-shot classification datasets. It can be observed that TripletCLIP consistently outperforms the baselines, achieving the best average results across these challenging datasets. Although the improvements are marginal, they are in line with expectations. As discussed in Figure 3, TripletCLIP does not introduce new concepts into the training data but focuses augmentations that enhance representation without increasing concept diversity. These
17

---

Table 13: Composition evaluations of the methods various benchmarks. Bold number indicates the best performance and underlined number denotes the second-best performance.
Retrieval
Image-to
MSCOCO
Flickriik
MSCOCO
Flickr3t6
Roi
1,85
R:00
R1:11
R8055
R681
12055
RE61
R:65
R.09
LaCLiP
1.56
5.06
10.90
16.00
1.73
5.97
3.54
10.84
1012 00:00
NogCLiP
13.80
1.50
11:30
1.85
TriplecCLP
Performunce
1,00%5
11.10F5
1385%
LaCUP
10.50
25.55
35 60
21,
42. 70
54.00
7.21
19.23
28.37
15:06
36.30
47.00
LigCLIP
12:32
V0.16
41
24.70
46 80
18.20
8.56
23:11
22.60
18.6
41.70
53.42
NigCLiPs
1094
36.96
16.64
18.70
43.90
$5.00
8.40
23.89
12.15
18.34
42.86
59.78
TripteeCLIP
14.00
33,00
4384
25,00
55,99
65.70
11.38
28.99
39.04
25.28
$2.38
63.33
Performine Gain v. CLIP
4,00
7.54%
5.30%
6.707%
13,2045
11.10%
4.07
8.7245
10.87
10.22%
16,08%
16.33
Table 14: Ablation on choice pre-trained LLM. We train NegCLIP++ (ViT-B/32) on negative captions generated from various LLMs and report SugarCrepe, Flickr30k Retrieval (R @ 5), and ImageNet-topS performances.
Models
SugarCrepe (avg.)
Retrieval (R@5)
ImageNetIk (top-5)
Gemma-2h-it
56.00
12.60
12.09%
Phi-3-minil44k-nstror Miistral-7b-insttuutt-222
61.22
13.02
10.94%
Table 15: Ablation CyCLIP with TripletLoss. To evaluate the compatibility of TripletLoss with the CyCLIP, we train the ViT-B/32 models from search on CC3M with 512 batch size and report SugarCrepe, Flickr30k Retrieval (R 5), and ImageNet-top5 performances.
Models
SugarCrepe (avg.)
Retrieval (R@5)
ImageNetIk (top-5)
LaCLIP
55.11
12.80
12.58%
TripletCLIP
65.71
24.62
19.95%
CyCLiF
54.62
11.77
13.00%
58.64
20.29
19.05%
enhanced representations from TripletCLIP lead average improvements of 1-3% depending on the scenario.
D.3 Detailed image-text retrieval performance
Table 13 presents detailed T2I and 12T retrieval results for the MSCOCO and Flickr30k datasets. We report results at different recall thresholds: R@ 1, R@5, and R@ 10. The data shows that TripletCLIP significantly outperforms the baselines across all recall rates. On average, Triplet- CLIP achieves a performance gain of 7-11% % over LaCLIP. Additionally, TripletCLIP improves performance by 3-6% compared to previous state-of-the baselines.
D.4 Additional ablations
Choice of Pre-trained LLM. We provide further details regarding the selection of LLMs for generating hard negative captions. NegCLIP++ was trained million generated negative captions for CC3M using three different LLMs, and the results are reported in Table 14. We find that Phi-3 achieves the best average performance, while Gemma-2b unexpectedly has a notable negative impact on compositionality. However, we opted to use Misttall77-inntmmet--0..22 as Phi-3 was released after our experiments were completed, preventing its earlier evaluation.
Evaluating the Orthogonality of TripletLoss. integrated with various previously proposed methodologies. To evaluate this, we applied TripletLoss conjunction with CyCLIP [17] and reported the results in Table The most significant perfor- mance improvement is observed with the TripletLooss alone. However, Tripletl oss also enhances the performance over the base CyCLIP, demonstrating its adaptability and orthogonality with respect to other approaches.
discussed in Section TripletCLIP can be
18

---

Table 16: Finetuning-baser evaluations of the methods on Retrieval and ImageNet- 1k bench- marks. Bold number indicates the best performance and underlined number denotes the second-best performance.
Retrieval
ImageNet lk
Methods
Text Retrieval (R@5)
Image Retrieval (R@55)
MSCOCO
Flickr30k
MSCOCO
Flickr30k
top-1
top-5
CLIP
74.9
94.60
55.92
83.38
63.31
88.22
CLIP (finetuned)
68.9
88.40
53.50
81.10
49.95
79.16
NeICLIP
66.00
88.60
53.41
81.12
48.85
78.34
Baseline [44]
81.4
96.0
67.49
89.84
61.40
88.10
TSVLC (RB) [12]
71.70
93.00
62.01
87.12
58.81
85.97
TSVLC (LLM+RB)) [12]
71.82
92.50
62.24
87,46
59.77
87.02
DAC [1]]
54.5
79.60
63.51
87.84
51.02
81.22
TripletCLIP (ours)
55.6
82.60
53.32
80.88
45.92
75.54
0.40
DINOv2
GTE
CLIP
0.35
CLIP
0.25
0.30
Freguency
0.20
Frequency
0.25
0.20
0.15
0.15
0.10
0.10
0.05
0.2
0.4
0.6
0.8
0.2
0.6 Cosine Similarity
0.4
1.0
Cosine Similarity
Figure 4: Positive vs. Negative modality-specific pair-based similarity distribution of pre-trained CLIP VIT-B/32 model w.r.t. the vision and text-only encoders. The left plot is the vision embedding similarities between positive and negative images. The right plot is the text embedding similarities between positive and negative captions. In the ideal scenario, the distribution should be skewed towards 0.0, which indicates that the model can correctly distinguish between the positive and negative data.
E Encoder Representation Distribution Analysis
Remember, this study aims learn the representations that can distinguish between two data points that are very similar but semantically different. Firstly, we take LaCLIP and TripletCLIP models trained on CC12M. We also sampled 50000 positive+negative pairs from CC3M. Then, we measure the vision and text modality-seecific cosine similarities between positive and negative pairs and plot the distribution (see Figure 5). It can be observed that vision representations from TripletCLIP more skewed towards 0.0, suggesting that the vision encoder can distinguish between hard negative samples better than the baseline LaCI.IP. However, in the perform similarly. This aligns with our findings from Table 8 that the vision encoder plays a crucial role in improving the compositionality. and to achieve this, our TripletData is necessary.
of the text modality, both methods
F TripletData Analysis
This section provides qualitative examples of the TripletData and discusses various data analyses.
19

---

0.35
TripietLIIP (CC12M)
LaCLIP (CC12M)
0.30
0.30
Frequency
0.25
Frequency
0.25
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.00
0.00
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
1.0
Cosine Similarity
Cosine Similarity
Figure 5: Positive vs. Negative modality-speciff pair-based similarity distribution of baseline LaCLIP and TripletCLIP. The left plot is the vision embedding similarities between positive and negative images. The right plot is the text embedding similarities between positive and negative captions.
Table 17: Question Generation. Examples of LLM LLL--generated existence-related questions from captions to evaluate the generated images.
Captions
Questions
Is the entity a sheet?
the sheet made fabric?
A smooth, flat sheet of woven fahric (which looks like woven silk) shown in closeup.
Is the fabric woven? Does fabric look like silk?
Is the fabric flat and smooth?
Is there sunset? the sea calm?
The sunset over calm
Is the sunset over the sea?
the sunset happening during the day?
the mughh?
Did the businesswoman finish second?
Did the businesswoman descend the podium?
the businesswonan finished second and descended the podium of the runners up.
Are there runners-upt up?
the podium for the second place?
the businesswoman runner?
Is there kid in the public place?
Is the kid near a scooter? the scooter Is the kid standing still?
Kid standing still scooter public place.
public place?
public place
caption?
the local resident driving?
Is happening in the fall?
local resident drives the scenic wearing red searf.
there drive?
Is the local resident wearing red scart?
the local resident walking?
Difficulty of the data. We further add one more analysis to investigate how difficult our dataset is. First, we take state-of-the-art language-only (GTE (28]) and vision-only (DINO [4]) embedding models and pretrained CLIP ViT-B/32. Later, we measure the modality similarity between positive and negative vision and language data pairs. Figure shows that the similarity distribution of the pretrained CLIP model between positive and negative text pairs follows the distribution of the text-only GTE model. Interestingly. vision distribution is drastically different. DINO can distinguish the positive and negative pairs correctly with high confidence. However, despite the visuals being so different, the pretrained CLIP model struggles to distinguish the different images. This further highlights that Triple eData is indeed challenging for the vision- vision-language models, the ones trained on large-scale datasets.
Evaluations of Generated Images. Even though T2I diffusion models are widely evaluated on various tasks. We perform additional evaluations to measure how accurately generated images follow
20

---

the text prompt. To do this, taking inspiration from [36], we first use LLM to generate binary "yes/no" style questions from the given caption. As shown in Table 17, we create five questions per hard negative caption. Later, we utilized the ViLT model to answer visual questions. Upon this investigation, we find that SDXL--turbo archives on an average of 76% accuracy. other words, the T21 model can correctly generate an image that follows around 3/4th of the text. Additionally, we hypothesize that using an improved T2I model or image editing models to generate "hard" negative examples can further improve composition reasoning.
Qualitative Examples. In Figure 6, we provide additional qualitative examples of the contrastive positive and "hard" negative pairs from the TripletData. Additionally, in Figure 7, we illustrated several examples where the T2I model could not precisely generate images corresponding to the caption. However, we may notice that in most cases, it maintains some of the important aspects. Because of this, despite not being 100% accurate all the time, it can help TripletCLIP improve performance across the evaluation benchmarks.
Hard negative caption only examples:
Raw Caption: dog looking out from a window
Language Rewrite: A dog looking through the window at his owner.
Negative Caption (NegCLIP):
(a) A window looking through the dog at his owner.
(b) A dog looking through the window at his dog.
(c) A dog screams through the window his owner.
TripletData Negative Caption: A cat observing its owner from the window.
2. Raw Caption: person attends the premiere of film
Language Rewrite: A person attends the premiere of film
Negative Caption (NegCLIP):
(a) A premiere attends the person of film
(b) A person attends the festival of film
(c) A person watches the premiere of film
TripletData Negative Caption: A person waits in line for film tickets.
3. Raw Caption: white crocus spring flowers in the forest.
Language Rewrite: white crocus flowers in the forest
Negative Caption (NegCLIP):
(a) white crocus flowers in the sky
TripletData Negative Caption: red orchid flowers in the meadow.
Raw Caption: flag with industry in the background
Language Rewrite: A flag is holding in the background an industrial site.
Negative Caption (NegCIII)):
(a) A background is holding in the flag industrial site.
(b) A flag is holding in the background an earthquake site.
(c) A drone is holding in the background an industrial site.
(d) A flag is visible in the background an industrial site.
TripletData Negative Caption: A flag is flapping in the foreground of a pastoral scene.
Raw Caption: portrait of businessman with cardboard on his head carrying a briefcase and using
an umbrella while standing by.
Language Rewrite: A portrait of a businessman standing by with a briefcase and cardboard on his head carrying an umbrella while looking a blue sky and parked cars the street
Negative Caption (NegCI.I)):
(a) A businessman of a portrait standing by with briefcase and cardboard on his head carrying an umbrella while looking at blue sky and parked cars on the street
(b) A portrait of businessman standing by with briefcase and cardboard on his head carrying an umbrella while looking at grey sky and parked cars on the street
(c) A portrait of a businessman standing by with briefcase and cardboard on his head carrying an umbrella while looking at blue truck and parked cars on the street
(d) A portrait of a businessman standing by with briefcase and cardboard on his head carrying an umbrella while pointing at a blue sky and parked cars on the street
TripletData Negative Caption: A portrait of a businesswoman seated on bench with a tote bag and newspaper her lap holding umbrella while looking red sunset over row houses.
21

---

6. Raw Caption: 158834 is the portion of the bound train.
Language Rewrite: Huge locomotives sit the tracks in front of building.
Negative Caption (NegCIIP):
(a) Huge tracks on the locomotives in front of building.
(b) Three locomotives sit on the tracks in front of a building.
(c) Huge locomotives sit on the tracks in anticipation of building.
(d) Huge locomotives mounted the tracks in front of building.
TripletData Negative Caption: Huge locomotives sit on the tracks in front of bridge.
7. Raw Caption: biological subfamily eating fish seaweed covered shore
Language Rewrite: Two blue whales are eating salmon on beach surrounded by seaweed
Negative Caption (NegCLIP):
(a) Two blue salmon are eating whales on a beach surrounded by seaweed
(b) Two stranded whales eating salmon beach surrounded by seaweed
(C) Two blue whales are eating salmon on a farm surrounded by seaweed
(d) Two blue whales are eating salmon beach covered by seaweed
TripletData Negative Caption: Two blue whales are feeding on herring in bay surrounded
by kelp.
8. Raw Caption: image of an original oil painting on canvas
Language Rewrite: young lady holding painting your face so you can see the detail of the
painting
Negative Caption (NegCLIP):
(a) A young painting holding lady to your face so you can see the detail of the lady
(b) bearded lady holding painting to your face so you can see the detail of the painting
(C) A young lady holding pencil to your face so you can see the detail of the painting
(d) young lady holding painting to your face so you can enjoy the detail of the painting
TripletData Negative Caption: A young lady holding painting away from her face to show
its beauty to the audience.
22

---

"photo of interior of big white house with many windows that was the stage of the filmng a series."
"photo of an interior of big red house with namoow windows that was the stage the filming of movie."
"A bench by the sea shown on empty beach"
"A bench by the lake shown on a sandy beach."
"a thoroughbred bengal cat, as year old one, very powerful animal*
"a siamese cat, six months old, has impreseive strength for size."
AA man's Lattoo covers much of his chest with the positive and negative space playing an important part."
'A woman's tattoo covers much of her back with the positive and negative space plaging important part."
"add a modern finish to your look with these chic courts."
*add vintage touch to your appearance with these charming sneakers."
llights, streets, teemming"
llihhs, squares, quiet,,"
'A simple paper heart pinned pink background
'A detailed paper flower pinned on white background.
'Pignons sitting park
"Sparrows are sitting on lawn."
1A bedroom with king bed and bedside table"
"A bedroom with queen bed and chest of drawers. *
gold necklace"
'A silver necklace with different design."
"The knife sits "The knife sits on on black background"
*<PERSON> for portrait by an artist"
"<PERSON> stands for portrait by an artist.
white background."
Figure Qualitative examples positive and hard negative image-text pairs from TripletData. In each block, left image-text pairs are positive images from CC3M, and right pairs are corresponding negatives from TripletData.
23

---

"A small car at the side of the road where traditional horse-drawn carriage is driven by a woman."
"someone entered building looking cool in a blackTeathe leather jacket she walked."
"Standing next to a red ocean, two children gaze up the half moon with promise to one day swim across the seas"
"A woman is using ruler to measure the distance. between two points outside of a bus." "
"A decorative Easter egg balanced precariously on fluffy bunny's head.*
"An hourglass the shape of rectangular pool."
Figure 7: Examples of T2I failures.
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide our novelty in abstract and introduction and perform experiments
accordingly.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide limitations in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all hyperparameters and plan release the codebase.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: We plan to release the data. However, due to the large scale of the data, we have not released it for review; instead, we have provided the randomly sampled data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
httpps.ccppublic//ee / ssi ssionooiccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See the section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Due to the nature of large-scale pertaining, we cannot repeat each training multiple times calculate the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See section and 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neurrs httpps//ourrps..ccppblcc/eeee csGuidel ines?
Answer: [Yes]
Justification: Read and Agree.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide broader impacts in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We plan to release the data for academic purposes with whomever agrees with terms and conditions.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We provide credits to all works utilized in this study.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "sM1YOKS1Lo", "reviewer": "Reviewer_ZTUH", "summary": "The paper introduces a novel pre-training strategy aimed at enhancing the compositional reasoning capabilities of CLIP models. The authors identify the limitation in current image-text datasets that restricts the compositional understanding of CLIP models and propose a solution that involves generating \"hard\" negative captions and corresponding images. This is achieved through a two-step process: leveraging in-context learning for negative caption generation and utilizing text-to-image generators to create matching negative images. The improvement in modeling effectiveness is more significant with the manufactured negative sample dataset.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "[S1] This paper is well-written, making it easy for readers to follow.\n[S2] The method of the paper is concise and easy to implement, and the experimental results demonstrate its effectiveness, which could encourage more people to apply this method to their own tasks.\n[S3] For multimodal contrastive learning pre-training, incorporating both image negative samples and text negative samples is comprehensive.", "weaknesses": "This research primarily involves leveraging LLMs to generate negative samples of similar image descriptions based on the original descriptions, followed by synthesizing the corresponding images using text-to-image models. It is well-known that using synthetic data can easily lead to model overfitting. Consequently, various techniques are typically employed to enhance data diversity, such as data augmentation. It would be intuitive to explore whether utilizing multiple models (multiple LLMs and text-to-image models) during the synthesis process might yield better results. I would be interested in seeing an analysis related to this issue, which could make this paper comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and consideration given to our paper. We are delighted that our work is recognized as well-written, easily reproducible, and effective. We appreciate your belief in our approach and its potential to inspire similar strategies in downstream tasks.\n\nIn response to your inquiries, please find our clarifications below:\n\n> ## Ablation on Generative Model Choices\n\nWhile we acknowledge that exploring multiple LLMs and T2I models could be beneficial, our focus in the paper was shaped by three primary considerations:\n\n- The semantic manipulation of text by an LLM is paramount, regardless of the specific LLM used. For better comprehensiveness, **we have performed an additional ablation study**, which is detailed below.\n- T2I models must precisely synthesize images from hard negative captions. Hence, we selected SDXL-Turbo, a state-of-the-art and efficient open-source model, **thus not necessitating further ablation on T2I model choices**.\n- Synthesizing negative captions and images at scale demands significant resources, making holistic evaluations challenging.\n\nFor better comprehensiveness, we conducted experiments with different LLMs to generate negative captions, which is a crucial component. The table below presents results from using three LLMs to generate 3M negative captions each for the CC3M dataset. We then trained NegCLIP++ models to assess the effectiveness of these synthetic captions. Notably, Gemma-2 significantly reduces compositionality, while Phi-3 performs best overall. The Phi-3 model was released after the NeurIPS deadline; hence, we use Mistral (second-best choice) in this case.\n\n| Models                      | SugarCrepe | Retrieval (R@5) | ImageNet1k (top-5) |\n|-----------------------------|:------------:|:------------------:|:--------------------:|\n| **Gemma-2b-it**                 | 56.00      | *0.1260*          | **12.09%**             |\n| **Phi-3-mini-4k-instruct**      | *61.22*      | **0.1302**           | *10.94%*             |\n| **Mistral-7b-instruct-v0.2**    | **61.69**      | 0.1072           | 10.52%             |\n\n\n> ## Overfitting Due to Synthetic Data\n\nWe agree that synthetic data can lead to overfitting. As noted in our paper and supported by recent work on fully synthetic CLIP models like SynthCLIP [1], three times more synthetic data is required to match the performance of real data due to limited diversity. Our research was motivated by the question: “What kind of synthetic data can enhance performance?” We found that generating hard negative captions and images is the optimal solution. Preliminary analyses on data scaling trends, shown in Figure 3, demonstrate consistent performance improvements with our approach as we scale the synsets.\nWhile our experiments did not observe overfitting (Figure 3), it remains a potential issue if scaled to billions of hard negative image-text pairs. This topic extends beyond the scope of our current work, and we suggest further exploration by the community in future research.\n\n\nWe hope these clarifications address all your questions and enhance the comprehensiveness of our work. Additionally, we provide a summary of responses to other reviews in our global response for your reference.\n\n---\n\n[1] Hammoud, Hasan Abed Al Kader, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. \"SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?.\" arXiv preprint arXiv:2402.01832 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gpvKjsvgCH", "reviewer": "Reviewer_U1Yz", "summary": "To enhance the compositional capabilities of CLIP, authors propose to generate “hard” negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution.", "questions": "see weakness", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Authors introduce a novel CLIP pre-training strategy that employs hard negative images in conjunction with triplet contrastive learning to enhance compositionality.\n2. TripletCLIP consistently improves across downstream tasks.\n3. A new dataset TripletData is proposed.", "weaknesses": "The novelty and contribution are limited. While [1] builds negative samples from the text perspective, similar to [1], authors primarily build additional negative samples from the image perspective with LLM and generated images. \n\nGiven the existing observations (bag-of-words phenomenon, etc.) and soulutions presented in [1], I think the contribution of the current version is marginal. \n\n\n### Reference\n[1] When and why vision-language models behave like bags-of-words, and what to do about it? ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We respectfully disagree with the claims regarding our work's limited novelty and contributions. Other reviewers have unanimously recognized numerous strengths in our work despite its straightforward nature. Therefore, we would like to reiterate our research's key strengths and contributions.\n\n\n> ## Novelty and Contributions\n\n**Importance:** CLIP models have demonstrated limited performance in terms of compositionality and tend to behave as bags-of-words, as observed by [1]. This work suggested introducing negative mining, particularly negative captions, could enhance model performance. Following this, several studies have focused on improving synthetic hard negative captions [2,3,4,5], while others have aimed at enhancing data quality [6,7,8]. However, with recent advances in image generative models, it remains unclear whether such tools can improve compositionality. Another recent study introduced training CLIP on a fully synthetic dataset [9], but it required three times more data to achieve performance comparable to real data.\n\n**Novelty/Contributions:** To address these limitations and improve image-text understanding, we propose TripletCLIP. TripletCLIP leverages LLMs to generate hard negative captions (similar to [3,4]) and introduces hard negative images focusing on various image semantics. Additionally, we introduce a novel contrastive pretraining loss, TripletLoss, which enhances the usability of our synthetic data generation pipeline. To be specific, our contributions are as follows:\n\n- **TripletCLIP significantly improves baseline performance by an absolute 6-10%** across benchmarks and shows absolute 4-7% improvements over the baseline proposed by [1].\n- **We have released approximately 13M synthetic image-text pairs**, complementing real-world datasets like CC3M and CC12M.\n- **Additional experiments (Table 7) indicate that vision encoders are the primary source of compositionality limitations** in pretrained CLIP models, which were previously unknown. TripletCLIP offers a promising solution to overcome this through careful training on hard negative images, specifically using TripletLoss.\n- **Experiments with increasing concept diversity (Figure 3) further validate our approach**, demonstrating that TripletCLIP consistently improves performance with larger dataset sizes.\n\nWe trust that our response adequately addresses your concerns regarding novelty and encourages you to reevaluate our submission. We also kindly request you to consider the feedback from other reviewers and our responses, contributing to a comprehensive assessment of our work.\n\n---\n\n[1] When and why vision-language models behave like bags-of-words, and what to do about it? ICLR 2023.\n\n[2] Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Fine-grained Understanding, CVPR 2024.\n\n[3] Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models, NeurIPS 2023.\n\n[4] Teaching Structured Vision&Language Concepts to Vision&Language Models, CVPR 2023.\n\n[5] Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality, EMNLP 2023.\n\n[6] DataComp: In search of the next generation of multimodal datasets, NeurIPS 2023.\n\n[7] Demystifying CLIP Data, ICLR 2024.\n\n[8] DreamLIP: Language-Image Pre-training with Long Captions, ArXiv 2024.\n\n[9] SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?, ArXiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8zeMbH4Avg", "reviewer": "Reviewer_f4Gf", "summary": "This paper proposes a TripletCLIP, a pre-training framework aimed at enhancing compositional reasoning task. It computes the NegCLIP objective separately for each, with the image serving as the anchor in each instance. For training TripletCLIP, such triplet data is constructed as follows: Hard negative captions are generated from the original LaCLIP captions using a large language model (LLM) through in-context learning mechanism. Hard negative images are created from text-to-image diffusion models. The experiment section compared various hard negative (HN) based objectives and demonstrated the superior performance of the proposed TripletCLIP mechanism across a wide range of benchmarks: compositionality, zero-shot retrieval, and classification.", "questions": "- Missing citations [L17-19]; image classification [CITATION], segmentation [CITATION], and image-text retrieval [CITATION]\n\n- From [L180], it needs referring to the proper section in the appendix\n\n- From [L227], what is the meaning of the 'real' hard negatives?", "limitations": "Limitations are adequately and honestly addressed in the appendix.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "[Originality and Significance]\nThis paper addresses both model and data aspects in the context of pre-training CLIP. Given the challenges of synthesizing hard negative texts and particularly images at scale, the proposed work for constructing triplet data is likely to be valuable to the research community. On the model side, the proposed hard negative objective in pre-training, though simple, consistently enhances performance across a variety of tasks, including compositionality, retrieval, and classification.\n\n[Quality and Clarity]\nThe experiments are well-designed and meticulously verify the claims made in the paper, demonstrating the necessity of such design choices. Some additional baselines, beyond the default LaCLIP, are included to demonstrate the effectiveness of TripletCLIP. The overall presentation was clear.", "weaknesses": "- [W1] The effectiveness of the TripletCLIP objective is less convincing. Although the paper considered some hard negative (HN)-based baselines such as NegImage and NegCLIP++, it did not compare with HN baselines that jointly consider both HN text and image in the loss calculation, such as [1, 2, 3, 4, 5]. I understand that some works are concurrent, but it would be more comprehensive to see whether the design of the TripletCLIP objective is optimal, either quantitatively or conceptually. In addition, it is necessary to note the 'previous' work that generates hard negative images to enhance compositional reasoning tasks [2, 3]. Accordingly, the sentence from [L136-137] needs to be adjusted. \n\n\n- [W2] Though there are consistent improvements, the absolute level of performance is relatively weak compared to the 'default' CLIP models from the CyCLIP [1] and ALIP [6] papers (in terms of zero-shot imagenet and retrieval tasks). This may be due to the reduced computational resources during pre-training, which limits the significance of the results in the context of pre-training. The only possibility left, and neither the triplet data nor the pre-trained model shows the evidence of benefiting further downstream tasks in applications such as advanced VLMs like LLaVA, or text-to-image generation, as noted in [L70-72]. \n\n\n- [W3] Connected to [W2], if only a single GPU is used for pre-training, it would be an attractive alternative to fine-tune the pre-trained CLIP model with the (subset of) triplet data. It's clear that the proposed methodology remains valid, and it would be interesting to see whether employing triplet data with the proposed hard negative objectives proves superior compared to other compositional reasoning methodologies in the context of fine-tuning [7, 8], specifically on both compositional reasoning and retrieval/classification tasks. If the authors wish to focus solely on the pre-training context, referencing other previous work discussing both pre-training, as well as fine-tuning is missed [9].\n\n\n- [W4] It is suggested to consider including additional compositional benchmarks for evaluation that consist of counterfactual image-text pairs, similar to the Winoground style: EqBen [2], COCO-Counterfactuals [10], MMVP-VLM [11], and SPEC [4], for comprehensiveness. \n\n\nIn summary, the main concerns are the missing comparisons with hard negative (HN) objectives that incorporate both image and text, and the limited significance due to small-scale pre-training. Fine-tuning could be considered as an alternative approach to address these issues. Superiority over previous fine-tuning methodologies can further increase the significance. Meanwhile, I believe the large-scale triplet data constructed in the paper is valuable and could promote further studies in various aspects. \n\n---\n\nReferences\n\n[1] Goel et al., CYCLIP: Cyclic Contrastive Language-Image Pretraining, in NIPS 2022.\n\n[2] Wang et al., Equivariant Similarity for Vision-Language Foundation Models, in ICCV 2023.\n\n[3] Sahin et al., Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining, in WACV 2024.\n\n[4] Peng et al., Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding, in CVPR 2024.\n\n[5] Singh et al., Learn “No” to Say “Yes” Better: Improving Vision-Language Models via Negations, in arXiv preprint 2024.\n\n[6] Yang et al., ALIP: Adaptive Language-Image Pre-training with Synthetic Caption, in ICCV 2023.\n\n[7] Doveh et al., Teaching Structured Vision&Language Concepts to Vision&Language Models, in CVPR 2023.\n\n[8] Doveh et al., Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models, NIPS 2023.\n\n[9] Singh et al., Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality, in EMNLP 2023. \n\n[10] Le et al., COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs, in NIPS Dataset and Benchmark Track, 2023.\n\n[11] Tong et al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, in CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review of TripletCLIP and the time you've dedicated to it. Thank you for the detailed and holistic review of our work. We are delighted that you found it valuable to the research community.\n\n> ## Comparison with Related Works [W1]\n\nFirstly, we want to clarify the key baselines selected in the main paper, reflecting the current state of research at the pretraining stage:\n\n- NegCLIP: SOTA baseline for negative captions based on the latest CVPR’24 work [12].\n- NegCLIP++: LLM-enhanced negative caption similar to [6,7].\n- CLIP + HN: Real-world hard negatives filtered from existing datasets.\n\nWe further divided the related works into two categories:\n\n### Orthogonal Works [1,6-9]\n\nOur contributions are orthogonal, meaning our findings can be combined with theirs for further improvements. For instance, CyCLIP [1] introduces a novel regularizer but does not include hard negatives. [6,7] introduce synthetic captions similar to NegCLIP++. [8] focuses on detailed caption-based NegCLIP fine-tuning. [9] introduces scene graph-based negative captions.\n\nWe retrained CLIP and CyCLIP models with and without TripletLoss using the CC3M training dataset and a batch size of 512. **Adding TripletLoss consistently improved performance over baseline CLIP and CyCLIP**:\n\n| Models | SugarCrepe | Retrieval (R@5) | ImageNet1k (top-5) |\n|--------|:-----:|:------:|:--------:|\n| CLIP | 55.11| 0.1280| 12.58% |\n| TripletCLIP| 65.71| 0.2462| 19.95% |\n| CyCLIP | 54.62| 0.1177| 13.01% |\n| CyCLIP+TripletLoss | 58.64| 0.2029| 19.05% |\n\n### Negative Captions and Images [2-5]\n\nThese works are closer to TripletCLIP. Specifically, [2] utilizes the video dataset to perform hard negative mining and [3] focuses on object-centric caption and image editing for hard negatives, while we focus on free-form captions and image editing that modifies semantics globally. [4] introduces a synthetic data pipeline for benchmarking and proposes a simple extension to NegCLIP pretraining, **aligning with our analysis in Table 3, which shows TripletCLIP’s significance**. [5] focuses on a \"negation\" and utilizes a method similar to [4].\n\nAlthough we could incorporate these negative captions [6-9] along with the TripletCLIP procedure for further improvements, due to time constraints, we perform direct comparisons with finetuning-based experiments and leave the joint optimization as future work. \n\nWe will include this detailed comparisons in the related work to make it more comprehensive in the final draft.\n\n> ## Finetuning Based Experiments [W2 + W3]\n\nOur paper focuses on pre-training, as fine-tuning focuses on post-training solutions, which may not be generalizable. That said, we performed additional finetuning experiments with hyper-parameters similar to [3,5,7] (w/o LoRA) and compared against various baselines [3,5,7,8], _whose public checkpoints are available_. **Our results show that TripletCLIP improves compositionality** and outperforms the almost all baselines. Additionally, the drop in retrieval and zero-shot classification is attributed to the vision encoder (Table 7), indicating the limitations of existing pre-trained vision encoders to represent semantics. This another reason to conduct the pretraining based experiments for comprehensive evaluations.\n\n| Models | SugarCrepe | Retrieval (R@5) | ImageNet1k (top-5) |\n|------|:-----:|:----:|:-------:|\n| CLIP (org) | 73.06 | 0.8899| 88.82% |\n| CLIP (finetuned) | 74.73| 0.8475| 79.16% |\n| NegCLIP| 80.59| 0.8486| 78.34% |\n| Baseline [3] | 77.84| 0.9292| 88.10% |\n| CoN-CLIP [5] | 75.58| -| -|\n| TSVLC (RB) [7]| 77.84| 0.9006| 85.97% |\n| TSVLC (LLM + RB) [7] | 73.61  | 0.8998| 87.02% |\n| DAC [8]| 86.41| 0.8372| 81.22% |\n| **TripletCLIP (ours)**| 82.46| 0.8174| 75.54% |\n\nHere, DAC [8] introduces detailed human-annotated captions with additional loss functions to incorporate multiple sentences of the captions, which is crucial this performance. Our work focuses on synthetic data with short captions. Future work could explore combining both approaches together. \n\n> ## Additional Clarifications on W2\n\nLines 70-72 convey that CLIP models are key components behind VLMs and T2I models. Our method can potentially improve CLIP models, directly impacting downstream tasks. TripletData could inspire future improvements (noted by ZTUH), possibly requiring different pretraining like DPO. We leave this as future work. However, we will make it clear in the camera-ready draft.\n\n> ## Additional Benchmark Evaluations\n\nWe have **already provided additional comparisons on three widely adopted benchmarks in Table 9**. Importantly, the newly suggested benchmarks [2,4,10,11] are not well-adopted by the community. But below are the additional evaluations on COCO-counterfactuals [10] that is very similar to [2]. Interestingly, all models perform consistently with no significant improvements over the baseline, possibly due to the evaluation data's difficulty or noise. Having said that, due to the time constraints associated with rebuttal phase, we could not extend large-scale evaluations on another two benchmarks.\n\n| Model  | Text-Score | Image-Score | Group-Score |\n|------|:-----:|:-----:|:----:|\n| CLIP| 26.1 | 24.77 | 16.73 |\n| NegCLIP| 26.06| 25.55 | 17.56 |\n| NegCLIP++| 27.21| 25.6  | 16.82 |\n| TripletCLIP  | 26.47| 25.37 | 17.92 |\n\n> ## Response to Remaining Minor Questions\n\n- [Q1] We will add the suggested citations in the final draft (if accepted).\n- [Q2] We will clarify the section we are referring to, specifically Figure 2 (main paper) and Figure 6 (appendix).\n- [Q3] By “real” hard negatives, we mean the hard negative image-text mining within the training datasets instead of synthesizing them.\n\nWe trust that our response adequately addresses your concerns and encourages you to reevaluate our submission. We look forward to the discussion.\n\n---\n\n[12] Zhang et. al., “Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding,” CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p3MLjBG518", "reviewer": "Reviewer_m3ja", "summary": "The paper proposes enhancing CLIP's compositional reasoning by generating high-quality negative image-text pairs using LLMs and text-to-image models. This approach improves beyond previous works that considered unrealistic, rule-based captions and unexplored negative images. Experiments show consistent improvements across different tasks.", "questions": "- Have you considered splitting the test set and evaluating the models on the TripletData?\n- Could you clarify how the total of 13M image-text pairs is calculated? The numbers provided (2.6M for CC3M and 8.6M for CC12M) in Section 4.1 add up to 11.2M.\n- Since the negative images are generated from a text-to-image model, there's likely a distribution mismatch from the real positive images. Have you inspected this potential discrepancy?\n- Can you elaborate on the relatively poor performance on the Winoground benchmark compared to other compositional tasks? Are there specific aspects of Winoground that TripletCLIP struggles with?\n- There's a mention of \"MiraData\" in the Appendix - is this a typo for TripletData, or is it referring to something else?", "limitations": "The authors have clearly addressed limitations in their paper.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- **Dataset**: The paper proposes using LLMs and text-to-image models to generate hard negative captions and images, respectively. The generation and filtering methodology is effective. I find that releasing TripletData alone is already valuable for the community.\n- **Comprehensive experiments**: The paper shows extensive evaluation and consistent improvements over baselines on compositional and downstream tasks like SugarCrepe, zero-shot classification, and image-text retrieval.\n- **Clear presentation**: The paper is well-structured and clearly written, explaining the methodology and results effectively.", "weaknesses": "- **Limited scaling**: The authors acknowledged the lack of experiments on increasing data and model size due to academic budget constraints.\n- **Limited study on the selection of LLM and image-to-text model**: The authors acknowledged the limited study on how the selection of the models could affect the quality of the TripletData.\n- **Nit**: Consider using \\log and \\exp in the equations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are encouraged by your review! We thank you for your comprehensive evaluation of our paper. We are grateful that you found TripletData alone to be of significant value and that our experiments are comprehensive.\n\nPlease find the requested clarifications below.\n\n> ## Response to Weaknesses\n\n**[W1] Limited Scaling:** We acknowledge that scaling our experiments further in academic settings is challenging. However, to approximate this, we experimented with increasing the data (i.e., WordNet synsets), as shown in Figure 3. Our findings indicate a consistent upward trend, with TripletCLIP outperforming the baseline CLIP in terms of compositionality while maintaining zero-shot classification performance.\n\n**[W2] Selection of LLMs and T2I Models:** Grounding the selection procedure of LLM to generate negative captions is indeed essential. We performed additional experiments with three LLMs and trained NegCLIP++ to evaluate their behaviors. The results show that Phi-3 is the best overall. However, at the time of submission, Phi-3 wasn’t released, so we selected the current second-best option, Mistral. The selection of T2I models is straightforward, as their goal is to generate images that truthfully align with the text. Therefore, selecting the best open-source T2I model is a logical choice.\n\n| Models                      |    SugarCrepe    |    Retrieval (R@5)    |    ImageNet1k (top-5)    |\n|-----------------------------|:----------------:|:---------------------:|:-----------------------:|\n| **Gemma-2b-it**                 |       56.00      |        _0.1260_       |        **12.09%**       |\n| **Phi-3-mini-4k-instruct**      |      _61.22_     |       **0.1302**      |         _10.94%_        |\n| **Mistral-7b-instruct-v0.2**    |      **61.69**   |        0.1072         |          10.52%         |\n\n\n**[W3]** We will update the equations in the final draft (if accepted). Thank you!\n\n\n> ## Response to Questions\n\n**[Q1] Evaluations on valdiation subset of TripletData:** Thank you for suggesting this interesting analysis. We evaluated the CC12M pre-trained models on a 50,000 random subset of the CC3M dataset and reported the scores below. As expected, TripletCLIP significantly boosts the performance of the baselines. However, we also partially attribute this to the spurious correlation learned from the data (related to Q3).\nAt the same time, we note that models are not fully converged, so there is very little chance of overfitting on this spurious correlation.\n\n| Model           |    Text Score    |    Image Score    |    Group Score    |\n|-----------------|:----------------:|:-----------------:|:-----------------:|\n| **CLIP**            |      52.69       |       29.66       |       24.64       |\n| **NegCLIP**         |      _54.84_       |      30.42      |      _25.82_      |\n| **NegCLIP++**       |      36.50        |      _30.67_    |       20.11       |\n| **TripletCLIP (ours)**     |     **92.25**    |      **66.82**    |      **64.30**     |\n\n**[Q2] Clarification on TripletData size:** We created TripletData on top of LaCLIP, which provides a ~2.9M subset of captions for CC3M and a ~10M subset for CC12M, totaling ~12.9M. However, we observed failures in downloading several images during training, leading to a further decrease in image-text pairs during the training phase. We will clarify this in the final draft (if accepted).\n\n**[Q3] Potential real vs. synthetic distribution mismatch:** Yes, a distribution mismatch exists between real and synthetic images. The generated images are of higher quality than CC3M/12M, potentially adding spurious correlations (partially observed in the Q1 table). However, other generated images represent similar objects with different semantics/relations that are not present in the real data. Despite TripletCLIP learning these biases, it still improves compositionality.\n\n**[Q4] Winoground clarification:** Winoground data contains only 400 instances, which may not be statistically significant for evaluating models trained with lower resources. Even with finetuning on top of the pretrained models, we observed a decrease in performance. Ideally, TripletData should replace the Winoground. We will plan to release the high-quality subset of TripletData as an evaluation set for the community to rely on. Thank you for inspiring this idea. \n\n| Models              |    Text-score    |    Image-score    |    Group-score    |\n|---------------------|:----------------:|:-----------------:|:-----------------:|\n| **CLIP (org)**          |    **0.3125**    |      _0.1100_     |    **0.0875**     |\n| **CLIP (finetuned)**    |      _0.2975_    |      0.0875       |      0.0625       |\n| **NegCLIP**             |      0.2700      |      0.0875       |      _0.0700_       |\n| **TripletCLIP**         |      0.2700      |    **0.1125**     |      _0.0700_       |\n\n\n**[Q5]** Yes, this is a typo. We meant TripletData. Thank you for pointing this out.\n\nWe hope this clarifies all the remaining questions. Additionally, we offer a summary of responses to other reviews for your reference in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the constructive feedback provided by the reviewers. It is gratifying to observe the positive evaluations across various dimensions of our work, as highlighted by the reviewers unanimously.

- The reviewers unanimously recognize our paper as **"well-written and easy to follow"** (Reviewers ZTUH, U1Yz, f4Gf, m3ja).
- Reviewers f4Gf and m3ja highlight our work's **originality and significance** in addressing model and data aspects in pre-training CLIP.
- The thoroughly designed experiments, which verify our claims and demonstrate the **effectiveness of TripletCLIP** compared to baselines, are acknowledged by Reviewers f4Gf and m3ja.
- Reviewer ZTUH finds the **paper concise, reproducible, and effective** enough to **encourage wide adoption** of similar strategies across the tasks.
- The **extensive evaluation and consistent improvements** over baselines in compositional and downstream tasks, such as SugarCrepe, zero-shot classification, and image-text retrieval, are noted by Reviewers f4Gf and m3ja.
- Reviewers f4Gf and m3ja **value** the contribution of **releasing large-scale TripletData** to the community.

We have provided detailed responses to each reviewer individually. Below, we summarize responses to two key questions. Additionally, **we have attached the pdf containing detailed benchmark performance** for all the experiments conducted during the rebuttal.

> ## Summarized Responses to Key Questions

- **Choice of LLMs and T2I models:** We provide additional information on the choice of LLMs for generating hard negative captions. We trained NegCLIP++ on 3M generated negative captions for CC3M using three different LLMs and reported the results. We find that Phi-3 performs the best on average, and Gemma-2b surprisingly affects the compositionality significantly. Moreover, unlike LLMs, whose goal is to follow instructions (which is complicated to evaluate), the T2I model's straightforward goal is to synthesize images faithfully. Therefore, we use SDXL Turbo (SOTA fast model) as the default choice without requiring any ablation.


| Models                      |    SugarCrepe    |    Retrieval (R@5)    |    ImageNet1k (top-5)    |
|-----------------------------|:----------------:|:---------------------:|:-----------------------:|
| **Gemma-2b-it**                 |       56.00      |        _0.1260_       |        **12.09%**       |
| **Phi-3-mini-4k-instruct**      |      _61.22_     |       **0.1302**      |         _10.94%_        |
| **Mistral-7b-instruct-v0.2**    |      **61.69**   |        0.1072         |          10.52%         |


- **Comparison with related works:** Reviewer f4Gf kindly pointed out several related works for comparison. We categorize these related works into two categories: 1) **Orthogonal works:** Our paper makes orthogonal contributions to methods like CyCLIP, meaning our work can be jointly utilized with these works, advocating independent contributions that do not necessitate comparisons. 2) **Relevant works focusing on hard negatives:** Most of the existing works are orthogonal and focus on various ways to improve hard negative captions with minor perturbations in loss functions. While two relevant works focus on generating negative images. 
    - In response to Reviewer f4Gf, we have performed additional finetuning-related experiments and reported the detailed comparisons. We find that additional baselines struggle even to match the compositionality performance of NegCLIP. While **TripletCLIP retains the competitive compositionality performance.** 

Again, we thank the reviewers and AC for their time reviewing our paper and providing detailed feedback. We hope that we have addressed all remaining concerns and questions. We look forward to the rebuttal discussion period.

Meta-review: The reviewers find the proposed method of generating image and text hard negatives to be effective and the resulting generated dataset to be a useful resource for the community. The reviewer also appreciate the extensive and meticulous experimentation and find the experimental results to be convincing. The reviewers had raised some concerns, but the rebuttal successfully addressed most of them and all reviewers recommend acceptance. The authors are recommended to improve the final paper version by following the reviewer recommendations.
Decision: Accept (poster)