TripletCLIP:  Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives

Authors: maitreya  patel
abhiram  kusumba
sheng  cheng
changhoon  kim
tejas  gokhale
chitta  baral
yezhou  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 22:38:33

Abstract: Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. This makes the nature of the training data a significant factor in the efficacy of CLIP for downstream tasks. However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP. We show that generating ``hard'' negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution. We introduce a novel contrastive pre-training strategy that leverages these hard negative captions and images in an alternating fashion to train CLIP. We demonstrate that our method, named TripletCLIP, when applied to existing datasets such as CC3M and CC12M, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark on an equal computational budget, as well as improvements in zero-shot image classification and image retrieval. Our code, models, and data are available at: tripletclip.github.io.

Main paper: 
 introduction: 
 Large-scale vision-language models, such as CLIP [41] , have significantly advanced multi-modal learning by employing contrastive learning to acquire shared semantic representations from paired datasets. This approach has resulted in improved performance in vision-language tasks as well as zero-shot image classification [51] and segmentation [23, 62] . Beyond vision-language tasks, the individual components of these models, such as the vision encoder and the language encoder, are integral to several multimodal architectures and generative models such as multimodal large language models (MLLMs) [30, 26] and text-to-image (T2I) diffusion models [45, 37, 38 ]. Yet, compositional reasoning remains challenging and multimodal models continue to exhibit naïve "bag of words" behavior, frequently failing to distinguish between expressions like "bulb in the grass" and "grass in the bulb" [58, 53] . Addressing this challenge remains critical for enhancing vision-language models and their downstream applications.Contrastive learning of representations benefits from "hard negative samples" (i.e., points that are difficult to distinguish from an anchor point) [43] . However, at each optimization step for training CLIP, image-text pairs are randomly sampled from the training dataset -this random sampling seldom exposes the model to highly similar negative pairs. We hypothesize that the limited compositional understanding of CLIP may stem from such issues in the optimization objective and sampling from training datasets. A straightforward solution could involve iteratively identifying hard negative pairs for each training iteration. However, due to the noisy captions and the scarcity of such pairs in existing datasets, prior work generates hard negative captions as a form of augmentation using rule-based strategies [58, 61] . For instance, given an image-text pair labeled "a brown horse", an additional negative caption "a blue horse" might be introduced. However in prior work, image data is not subjected to similar hard negative semantic augmentation during training; this is mainly because of the difficulty of making semantic perturbations at the pixel levels compared to sentence perturbation. While the text-only augmentation strategies have improved the models' compositional understanding to a certain extent, it raises an intriguing question: could incorporating hard negative augmentation for both text and image modality further enhance the compositional reasoning capabilities of vision-language models? Motivated by this, in this paper, we introduce a novel, simple, and yet highly effective strategy for integrating hard negative images as well as hard negative text to enhance the compositional understanding of vision-language models. Recent developments in text-to-image diffusion models have opened up possibilities for performing semantic perturbations within images [21] . Existing works have evaluated the impact of creating synthetic data for text-to-image generative models [3, 6] . However, it remains less explored how these generative models can benefit the CLIP-like models. To tackle this challenge, our approach leverages the in-context learning capabilities of LLMs to produce realistic, linguistically accurate negative captions [55] . We then employ a pre-trained textto-image diffusion model to create images corresponding to these captions, thereby enriching any given image-text dataset with valuable hard negatives that foster improved reasoning. This resulting TripletData comprises 13M image-text pairs to complement the CC3M and CC12M datasets [5] .We developed TripletCLIP, which incorporates hard negative image-text pairs effectively by using them to optimize a novel triplet contrastive loss function. Extensive experiments on the CC3M and CC12M datasets and various downstream tasks with an equal compute budget demonstrate that TripletCLIP significantly enhances compositional reasoning. Notably, TripletCLIP results in more than 9% and 6% absolute improvement on the SugarCrepe benchmark compared to LaCLIP and NegCLIP, respectively. TripletCLIP also improves zero-shot classification and image-text retrieval performance with similar training-time concept diversity. An investigation into the effects of increasing training-time concept diversity revealed that baseline models consistently under-performed in compositional tasks despite an increase in integrated knowledge, while TripletCLIP demonstrated significant improvements. In summary, our key contributions are as follows:• We introduce a novel CLIP pre-training strategy that employs hard negative images in conjunction with triplet contrastive learning to enhance compositionality. • TripletCLIP consistently improves across downstream tasks, demonstrating the effectiveness of synthesizing hard negative image-text pairs. • Our extensive ablations on the choice of the loss function, modality-specific pre-training, the increase in concept diversity, and filtering high-quality TripletData provide deeper insights into the utility of hard negative image-text pairs for CLIP pre-training. • Ultimately, we present a promising avenue where synthetic contrastive datasets significantly improve reasoning capabilities, leading to the creation and release of the TripletData -a 13M contrastive image-text dataset. 
related work: 
 Vision-Language Models. Recent advancements, including ALIGN [22] and CLIP [41] , have gained significant interest due to their capability to learn transferable semantic representations across multiple modalities through contrastive learning. These models facilitate downstream tasks such as zero-shot classification [51] , image-text retrieval [60, 2] , visual grounding/reasoning [31] , textto-image generation [52, 37, 38, 24] , semantic segmentation [62, 23] , and various evaluations [19, 47] . Subsequent research has sought to enhance various aspects of these models, including data efficiency [15] , hierarchical representation learning [8] , and the quantization of latent spaces for more stable pre-training [7] . LiT [59] employs a pre-trained frozen CLIP vision encoder to fine-tune a BERT-like text encoder [9] , achieving notable improvements in zero-shot transfer performance. Similarly, BLIP-2 [27] combines contrastive pre-training with the next-token prediction for image captioning during training. However, these approaches generally presume the availability of highquality data. In contrast, TripletCLIP focuses on leveraging the proposed hard negative contrastive dataset and incorporating triplet contrastive pre-training for compositional data. This approach is orthogonal to prior works. Data for Contrastive Pre-training. The effectiveness of maximizing mutual information between modalities heavily relies on the quality of extensive, web-scraped datasets that ideally encompass all possible concepts and knowledge. For instance, despite its noise, the LAION dataset [48, 16] , which includes more than 5 billion internet images paired with alt-text captions, is a primary resource. Studies show that over 1 billion data points are necessary to match the performance of the original CLIP model [16, 56] . Recent works like DataComp [16] and MetaCLIP [56] have focused on creating smaller, high-quality datasets by applying stringent filters and ensuring wordnet [33] synset-level concept diversity. Nevertheless, the inherently noisy nature of internet-scraped datasets can degrade model performance. Studies such as SynthCLIP [18] demonstrate that tripling the volume of fully synthetic data is required to equal the efficacy of real data. Other efforts like VeCLIP [25] and LaCLIP [14] enhance dataset quality by using generative language models to re-caption existing images, significantly boosting performance.Compositionality for vision-language. Despite the increased emphasis on data quality and modeling techniques, mastering compositionality remains a significant challenge for vision-language models. Benchmarks like ARO [58] , VALSE [35] , and CREPE [32] have been developed to assess models' abilities to handle compositional data. SugarCrepe [20] , in particular, offers a large-scale, systematic framework for such evaluations. Previous methods primarily focused on identifying hard negatives within existing datasets or generating synthetic negative captions [58, 61, 12, 11, 57, 49] . However, these rule-based generated captions are often unrealistic and linguistically flawed, leading to suboptimal model performance on complex datasets like SugarCrepe. A handful of works focus on finding negative images. [54] propose utilizing the video data. [44] focuses on object-centric image-editing to synthesize the negative images. [39] utilizes the simulation-based data negative data.Contrary to prior approaches that predominantly add unrealistic negative captions or very constrained negative images that are either very synthetic or object-focused, this work introduces TripletCLIP, which centers on generating naturally occurring hard negative image-text pairs. We propose a novel triplet contrastive learning strategy that effectively utilizes these challenging data pairs. Additionally, while our method is distinct, integrating advancements that refine contrastive learning could potentially boost TripletCLIP's efficacy further. 
method: 
 This section begins with an overview of the contrastive learning algorithm used by CLIP and NegCLIP.We then describe the synthetic data generation pipeline for generating hard negatives using LLMs and T2I models and introduce triplet contrastive learning which forms the basis of TripletCLIP. A high-level comparison between prior work and TripletCLIP can be found in Figure 1 .The goal for self-supervised contrastive learning [13] , when dealing with inputs from a single modality, is to use a feature extractor (F ) to encode inputs and their augmentations and minimize the InfoNCE loss [34] between the two encodings. CLIP is designed for multimodal settings (for example, vision and language inputs) -this entails using two encoders (one for each modality).Let X and Y represent two modalities and D = {(x i , y i )} i=1,...,M , be the training dataset, where x i ∈ X and y i ∈ Y. The goal is to train two modality-specific encoders, F X and F Y , by minimizing the InfoNCE loss between the normalized features extracted from the encoders.EQUATIONwhere ⟨•⟩ represents cosine similarity and τ is the trainable temperature parameter. For simplicity, we do not show feature normalization in the InfoNCE loss. Similarly, we can define the L CL Y→X training loss. The combined CLIP total training objective is given as, L CLIP = L CL X →Y + L CL Y→X . By minimizing this training loss, both encoders learn representations that maximize the mutual information between two modalities.NegCLIP introduces synthetic augmentations to generate "hard" negative captions (y ′ i ∈ Y ′ ) by performing semantic inverting perturbations to the reference captions (y i ∈ Y). Therefore, the single modality-specific hard negative augmentation-based training loss can be formulated as:EQUATIONThe total loss for NegCLIP for image modality (X ) and text modality (Y) is given by:EQUATIONIn Eq. 2, the negative samples are generated only for language modality as it is easy to make semanticlevel perturbations. Existing methods have not explored performing semantic perturbations in the image modality to create hard negatives. In this work, we demonstrate how hard negatives can be created in the image modality by leveraging the semantic language grounding and photorealism of text-to-image diffusion models. Our novel hard negative generation pipeline and refined training objective seeks to bridge the significant gap identified in literature.To generate high-quality hard negative image-text pairs, we follow a two-step procedure. The first stage is to generate hard negative captions from the ground truth positive caption. Second, to generate images corresponding to the hard negative captions as negative images. The AltText captions from the existing web-scrapped datasets are very noisy, leading to the noisy and unreliable generation of hard negatives. Therefore, we build upon the existing work LaCLIP, which first rewrites the captions using LLM from the existing data that are linguistically accurate. Figure 2 illustrates several examples of positive and corresponding negative image-text pairs.Generating hard negative captions. Existing works perform random swapping, replacing, and adding actions between the nouns, attributes, and relations of the positive caption [58, 61] . This method results in nonsensical and grammatically incorrect artifacts, such as "a person riding on four slope," which impedes the generation of negative images, ultimately leading to diminishing performance on harder benchmarks [20] . Therefore, we utilize the in-context learning ability of LLMs to generate negative captions. The choice of LLM is a trivial task as long as they provide hard negative captions. We find that Mistral-7B-Instruct-v0.22 performs reasonably better on our goal, and the output is easy to parse. We generate the negative captions in batches to speed up the generation process. Generating the 13M negative captions takes only 3 days on 8xRTX A6000. Instead of generating multiple hard negative captions, we find that a single high-quality hard negative caption is enough to improve the performance compared to the traditional NegCLIP style caption generation (see NegCLIP++ results in Table 4 ). We provide examples of various types of negative captions in the appendix. Specifically, we provide the following prompt to LLM:You are given the description of an image. You should provide a mostly similar description, changing the original one slightly but introducing enough significant differences such that the two descriptions could not possibly be for the same image. Keep the description length the same. Finally, only a few things (such as counting, objects, attributes, and relationships) can be modified to change the image structure significantly. Provide just the updated description. Examples: Input: A dog to the left of the cat. Output: A dog to the right of the cat. Input: A person wearing a red helmet drives a motorbike on a dirt road. Output: A person in a blue helmet rides a motorbike on a gravel path. Now, do the same for the following captions: Input: {} Output:Generating hard negative images. Typically, semantic perturbations within images require tools like image editing, which are resource-intensive and cannot be scaled. Remember that we want to provide additional ground truth references for the negative caption. Therefore, we propose to utilize the negative captions from the previous stage to generate the respective reference images directly for pre-training. As the previous stage generates negative captions that are linguistically correct, it becomes easier for image-generative models to synthesize the respective images precisely. We utilize pre-trained text-to-image diffusion models to generate the corresponding images. Specifically, we select SDXL-turbo [46] due to its relatively faster generation speed. After applying various inference time optimizations, we can generate 13M negative images within 2 days using 30 v100 GPUs. We provide various examples of the hard negative image-text pairs in the appendix. Analyzing difficulty of the hard Triplet-Data. Let's assume we have positive and negative image-text pairs from the TripletData , (x i , y i ) and (x ′ i , y ′ i ), respectively. If the data is truly hard negative, existing pre-trained models should struggle to find the correct imagetext pairs (i.e., cos(x i , y i ) > cos(x i , y ′ i )). Following winoground, we measure the text-score, image-score, and group-score to evaluate the popular pretrained CLIP models. Table 1 shows that even CLIP models trained on billions of data struggle to get near human performance on TripletData, which is less difficult than winoground. Importantly, the goal of generating hard negative samples isn't to add more diversity Table 3 : Importance of image-text hard negatives. We measure the importance of various modalityspecific hard negatives on SugarCrepe, image-text retrieval, and ImageNet1k. We find that Triplet-CLIP results into the most optimal solution. Bold number indicates the best performance.Negative Prior works have demonstrated the value of hard negative captions for enhancing the compositionality of CLIP models via L N egCLIP as the key training objective (Eq. 2) [58, 61] . However, it remains elusive if negative images alone can benefit or not. We conduct modality-specific ablations, reporting the average performance across the diverse set of benchmarks in Table 3 (we provide more details about experiments in Section 4). Our findings indicate that both "hard" negative captions and images individually boost performance when compared to LaCLIP. However, this initial empirical experiments to train the CLIP model on hard negative images (i.e., NegImage) by minimizing L N egCLIP (Y, X , X ′ ) reveal that negative images alone cannot improve the compositionality significantly (see Table 3 ). We hypothesize that images contain low-level information, making it difficult to train the model using images as negative examples. Aligning with our initial motivation and building upon this crucial insight, we propose to utilize the negative images to regularize the effect of negative captions and to stabilize the pre-training. Therefore, to utilize these hard negative image-text pairs from the previous stage more effectively, we propose to focus on two triplets (X , Y, Y ′ ) and (X ′ , Y ′ , Y), hence, the final triplet contrastive learning training objective is defined as:EQUATIONIntuitively, the second term introduces the additional form of supervision that hard negative images are closer to the corresponding negative captions than positive captions. This allows the system to understand that if the positive image does not represent the negative caption "blue horse," then what does this caption entail? Through this strategic alternation of hard negative image-text pairs for the TripletCLIP, we improve compositionality and image-text understanding of the vision-language model (see Table 3 ). We provide the pseudo-code in the appendix and the code in supplementary materials. This simple yet effective strategy elevates the training of the CLIP, offering a scalable framework to improve overall performance. 
experiment setup: 
 Pretraining Datasets. We utilize the CC3M and CC12M datasets, which comprise 2.6M and 8.6M image-text pairs, respectively. Following the approach demonstrated by LaCLIP, we use LLMrewritten captions to replace noisy original captions. For NegCLIP, we introduce four negative captions per positive image-text pair, focusing on semantic inverting perturbations across four categories: attribute, relation, object, and action [61] . This generates approximately 10.4M and 34.4M text-only augmentations for CC3M and CC12M, respectively. To train the TripletCLIP, we create augmentations (TripletData) for both datasets to integrate hard negatives effectively. We produce one augmentation per image-text pair, adding 2.6M and 8.6M image-text augmented pairs for CC3M and CC12M, respectively. Finally, we perform all the ablations on the CC3M dataset. Baselines. We train LaCLIP, LaCLIP with real hard negatives (LaCLIP+HN), and NegCLIP from scratch to ensure consistency and fairness in our comparisons. As NegCLIP's rule-based augmentations closely resemble some compositional benchmarks, so we introduce NegCLIP++ as an improved baseline. NegCLIP++ incorporates hard negative captions generated using LLM from TripletData, enhancing the language comprehension compared to standard NegCLIP.Implementation Details. Our experiments employ the ViT-B/32 [10] model architecture. To guarantee fair comparisons, we retrain all baseline models using identical hyperparameters. Since the overall training data for NegCLIP and TripletData is more than the baseline datasets, we align the number of iterations across all models to equalize the number of image-text pairs seen during training, similar to the strategy used in DataComp. The batch size is fixed to 1024 with the AdamW optimizer at a maximum learning rate of 0.0005, employing cosine decay. Training durations are set at approximately 100k iterations for CC3M and 200k iterations for CC12M. All models are trained on a single A100 (80GB) GPU using bf16 precision. The final training-related experiments and ablations will cost about 1200 A100 GPU hours. We leave the experiments on increasing the data and model size as future works for the community, as scaling further is not viable in the academic budget.Downstream Datasets. The primary objective of this study is to enhance the compositional capabilities of CLIP models. We mainly evaluate TripletCLIP and the baseline models using the challenging SugarCrepe composition benchmark, with additional performance assessments provided in the appendix for older benchmarks. Models are also tested on image-text retrieval tasks for broader evaluation using the Flickr30k [40] and MSCOCO [29] datasets. Zero-shot classification performance is assessed across approximately 18 different datasets. Evaluations adhere to the methodologies outlined in the CLIP-Benchmark3 or the official benchmark implementations.We comprehensively analyze the compositional understanding of models on the SugarCrepe benchmark, as detailed in Table 4 . Notably, TripletCLIP consistently outperforms all baseline models across all sub-categories of SugarCrepe on both the CC12M/CC3M training datasets. Specifically, TripletCLIP surpasses LaCLIP and NegCLIP by 10.91%/9.4% and 6.45%/6.31% on the CC12M and CC3M datasets, respectively. Our enhanced baseline, NegCLIP++, also shows improvement over standard NegCLIP, highlighting the benefits of LLM-generated negatives. Nevertheless, Triplet-CLIP further advances performance, underscoring the critical role of hard negative image-text pairs, not just text. Additional comparisons on older composition benchmarks (Valse [35] , Cola [42] ,and Winoground [53] ) in the appendix reveal TripletCLIP's consistent performance. Table 4 also contrasts TripletCLIP with models trained using the DataComp approach, which involves more parameters and training data, demonstrating that TripletCLIP achieves comparable performance to a ViT-B/16 model trained on 1 billion image-text pairs.Image-Text Retrieval. In Table 5 , we summarize the performance of models on text-to-image (T2I) and image-to-text (I2T) retrieval tasks on MSCOCO and Flickr30k datasets, where we report R@5 scores. Remarkably, TripletCLIP significantly outperforms baseline models by an average of 8%/10% and 8%/12.5% on I2T and T2I tasks, respectively, on the CC3M and CC12M datasets. Intriguingly, while LaCLIP+HN performs better than NegCLIP, TripletCLIP outstrips both.Zero-shot Classification. Table 5 also presents the average zero-shot classification performance on 18 standard datasets, including ImageNet1k. TripletCLIP consistently enhances top-1 accuracy by an average of 3% and top-5 accuracy by 5-7% compared to LaCLIP. Like the retrieval performance, LaCLIP+HN exceeds NegCLIP, yet TripletCLIP maintains the highest performance. Datasetspecific results are in the appendix.In this paper, we focus on pretraining-based experiments as they allow greater flexibility in learning better representations. To complement this, we also performed additional fine-tuning experiments using hyperparameters similar to the baselines (without LoRA) and compared them against various publicly available baselines [44, 50, 12, 11] . As reported in Table 7 , TripletCLIP improves compositionality and outperforms nearly all baselines. Furthermore, the observed drop in retrieval and zero-shot classification performance (Table 16 ) is attributed to limitations in the vision encoder, highlighting the challenges of existing pre-trained vision encoders in capturing semantic representations. This is further demonstrated in Table 8 . Can a high-quality filtered dataset improve the performance? Given that negative images in TripletData are generated using SDXL-turbo, these may not always be precise. Inspired by Data-Comp, we employ a pre-trained CLIP-L/14 to filter the image-text pairs, selecting the highest average similarity pairs (positive and negative) individually (i.e., score = (s(x i , y i ) + s(x ′ i , y ′ i ))/2). The top 1.4M positive image-text pairs and their corresponding negatives from TripletData are selected. Table 6 details this comparison against DataComp pre-trained models. Remarkably, TripletCLIP already surpasses baselines without filtered data; however, with the filtered dataset, despite being trained on 50% smaller dataset, TripletCLIP++ shows further performance improvements. This underlines the significant benefits of carefully selected TripletData in enhancing the performance.To address this open question, we designed an ablation study similar to LiT, freezing either the pre-trained CLIP vision or text encoder while training the opposite modality-specific encoder from scratch. We observe the performance of LaCLIP and TripletCLIP on CC3M, as shown in Table 8 . Freezing the vision model results in no performance gain on the SugarCrepe for TripletCLIP. However, significant improvements are noted when the vision encoder is actively trained, suggesting that the vision modality may be the bottleneck in compositionality. Notably, TripletCLIP outperforms LaCLIP in all settings, further demonstrating its robustness to different pre-training approaches.Concept coverage analysis. Improving performance on zero-shot transfer learning tasks such as retrieval involves two key components: adding more concept diversity during training and enhancing image-text alignment/compositionality. We create subsets of CC12M data with increasing concept diversity based on unique WordNet synsets. Specifically, we select 3M, 4M, 5M, and 6M subsets for training LaCLIP, while TripletCLIP training involves only half of these training data as positive pairs, and the rest are corresponding augmentations. Evaluations across SugarCrepe, retrieval tasks, and ImageNet1k (see Figure 3 ) indicate that TripletCLIP not only enhances SugarCrepe performance even at lower concept coverage levels but also significantly outperforms similar concept coverage in retrieval tasks, matching LaCLIP's performance on zero-shot classification tasks that do not require compositionality at all. This bolsters our argument that incorporating hard negatives from both modalities markedly improves compositional understanding in CLIP, while baseline struggles to do so even with more concept diversity.TripletData is used for large-scale compositional evaluations? We evaluated the CC12M pre-trained models on a 50,000 random subset of the CC3M dataset using a Winoground-style approach [53] . As shown in Table 9 , TripletCLIP significantly improves performance compared to the baselines. However, we partially attribute this improvement to spurious correlations learned from the data. At the same time, we note that the models have not fully converged, suggesting minimal risk of overfitting to these spurious correlations. 
conclusion: 
 In this work, we introduce TripletCLIP, a novel approach to enhancing compositional reasoning in vision-language models through the strategic incorporation of hard negative image-text pairs. Our comprehensive experiments across a suite of benchmarks demonstrate that TripletCLIP significantly outperforms existing methodologies such as LaCLIP and NegCLIP, achieving notable gains not only in compositionality but also in zero-shot classification and retrieval tasks as well. Further, our ablation studies highlight the critical role of modality-specific training and the careful curation of training data, underscoring the importance of both hard negative image and text components in the learning process.TripletCLIP's effectiveness with a smaller, refined dataset suggests a promising direction for future research-maximizing performance without the need for extensive data collection, thereby reducing computational costs and enhancing model efficiency. To this end, we provide an intriguing application of synthetic datasets via hard negative image-text pairs for vision-language tasks that could be easily extended to improve Multimodal Large Language Models and Text-to-Image generative models.Limitations. Due to constraints inherent in academic settings and limited computational resources, we were unable to scale TripletCLIP to handle hundreds of millions of image-text pairs or employ larger models within the scope of this study. Nevertheless, our results indicate a promising direction for future research within a consistent experimental framework, and we encourage subsequent work to explore scaling both the TripletData and TripletCLIP. Our experimental focus was primarily on the CLIP and LiT methodologies. With additional resources, however, extending our methodologies to more advanced contrastive learning techniques, such as SigLIP, would be feasible. In conclusion, our work introduces a compelling strategy for integrating open-ended hard negatives (both text and image) during the pre-training phase, providing a methodology and large-scale data that could benefit a variety of research domains. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: Read and Agree.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We provide broader impacts in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We plan to release the data for academic purposes with whomever agrees with terms and conditions.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We provide credits to all works utilized in this study.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this study, we have demonstrated the potential of utilizing high-quality positive and negative pairs to enhance the compositional understanding of vision-language models like CLIP through the introduction of TripletCLIP. While our findings are specific to TripletCLIP, the underlying techniques hold promise for broader applications, including enhancing visual understanding in Multimodal Large Language Models (MLLMs) and Text-to-Image diffusion models. Although our approach yields significant performance improvements, it does require resources to generate largescale synthetic datasets. We encourage future research to explore the utility of this pre-training strategy within the latent space, which could reduce dependence on large generative models. Initiatives like JEPA [1] have already demonstrated the efficacy of focusing on latent space models, which suggests a promising avenue for reducing computational overhead. Importantly, our experiments reveal that significant enhancements in model performance are achievable even with substantially smaller data scales. This finding suggests that, when scaled appropriately, our methodology could substantially diminish resource dependencies and enhance the efficiency of pre-training processes for CLIP-like models. ters across all methodologies. Although larger batch sizes are typically associated with improved performance in contrastive learning, computational constraints necessitated fixing the batch size at 1024 for all experiments. To accommodate this batch size on a single A100 GPU, we employed bf16 precision. In terms of computational resources, experiments using the CC3M dataset required approximately 16 GPU hours, while those involving the CC12M dataset utilized up to 56 GPU hours per experiment.Previously, we reported the results on SugarCrepe, the most challenging dataset, noted for its absence of language biases. However, evaluations were also conducted on other benchmarks, such as Valse, Cola, and Winoground. As indicated in Table 11 , TripletCLIP achieves overall improvements of 2-3% compared to LaCLIP and NegCLIP. The Valse benchmark, which contains text prompts that heavily favor the perturbations made for NegCLIP, shows a strong performance from NegCLIP, while NegCLIP++ encounters difficulties. Interestingly, TripletCLIP faces challenges in maintaining performance on Winoground, and baseline LaCLIP maintains the SOTA, which is counterintuitive to other benchmarks. Nonetheless, TripletCLIP still manages to outperform NegCLIP significantly. These results affirm that TripletCLIP sets a new standard for state-of-the-art compositional reasoning across diverse benchmarks.Table 12 provides fine-grained results for the 18 zero-shot classification datasets. It can be observed that TripletCLIP consistently outperforms the baselines, achieving the best average results across these challenging datasets. Although the improvements are marginal, they are in line with expectations. As discussed in Figure 3 , TripletCLIP does not introduce new concepts into the training data but focuses on augmentations that enhance representation without increasing concept diversity. These Table 13 : Composition evaluations of the methods on various benchmarks. Bold number indicates the best performance and underlined number denotes the second-best performance. enhanced representations from TripletCLIP lead to average improvements of 1-3% depending on the scenario.Text-to-Image Retrieval Image-to-Text Retrieval MSCOCO Flickr30k MSCOCO Flickr30k R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@Table 13 presents detailed T2I and I2T retrieval results for the MSCOCO and Flickr30k datasets.We report results at different recall thresholds: R@1, R@5, and R@10. The data shows that TripletCLIP significantly outperforms the baselines across all recall rates. On average, Triplet-CLIP achieves a performance gain of 7-11% over LaCLIP. Additionally, TripletCLIP improves performance by 3-6% compared to previous state-of-the-art baselines.Choice of Pre-trained LLM. We provide further details regarding the selection of LLMs for generating hard negative captions. NegCLIP++ was trained on 3 million generated negative captions for CC3M using three different LLMs, and the results are reported in Table 14 . We find that Phi-3 achieves the best average performance, while Gemma-2b unexpectedly has a notable negative impact on compositionality. However, we opted to use Mistral-7b-instruct-v0.2, as Phi-3 was released after our experiments were completed, preventing its earlier evaluation.Evaluating the Orthogonality of TripletLoss. As discussed in Section 2, TripletCLIP can be integrated with various previously proposed methodologies. To evaluate this, we applied TripletLoss in conjunction with CyCLIP [17] and reported the results in Table 15 . The most significant performance improvement is observed with the TripletLoss alone. However, TripletLoss also enhances the performance over the base CyCLIP, demonstrating its adaptability and orthogonality with respect to other approaches. The left plot is the vision embedding similarities between positive and negative images. The right plot is the text embedding similarities between positive and negative captions. In the ideal scenario, the distribution should be skewed towards 0.0, which indicates that the model can correctly distinguish between the positive and negative data.Remember, this study aims to learn the representations that can distinguish between two data points that are very similar but semantically different. Firstly, we take LaCLIP and TripletCLIP models trained on CC12M. We also sampled 50000 positive+negative pairs from CC3M. Then, we measure the vision and text modality-specific cosine similarities between positive and negative pairs and plot the distribution (see Figure 5 ). It can be observed that vision representations from TripletCLIP are more skewed towards 0.0, suggesting that the vision encoder can distinguish between hard negative samples better than the baseline LaCLIP. However, in the case of the text modality, both methods perform similarly. This aligns with our findings from Table 8 that the vision encoder plays a crucial role in improving the compositionality, and to achieve this, our TripletData is necessary.This section provides qualitative examples of the TripletData and discusses various data analyses. "someone entered a building looking cool in a black leather jacket as she walked.""Standing next to a red ocean, two children gaze up at the half moon with a promise to one day swim across the seas" "A small car at the side of the road where a traditional horse-drawn carriage is driven by a woman.""A woman is using a ruler to measure the distance between two points outside of a bus.""A decorative Easter egg is balanced precariously on a fluffy bunny's head.""An hourglass in the shape of a rectangular pool." Answer: [Yes] Justification: We plan to release the data. However, due to the large scale of the data, we have not released it for review; instead, we have provided the randomly sampled data. Guidelines: Justification: See section 3 and 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2https://github.com/LAION-AI/CLIP_benchmark 


Decision: The reviewers find the proposed method of generating image and text hard negatives to be effective and the resulting generated dataset to be a useful resource for the community. The reviewer also appreciate the extensive and meticulous experimentation and find the experimental results to be convincing. The reviewers had raised some concerns, but the rebuttal successfully addressed most of them and all reviewers recommend acceptance. The authors are recommended to improve the final paper version by following the reviewer recommendations.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel pre-training strategy aimed at enhancing the compositional reasoning capabilities of CLIP models. The authors identify the limitation in current image-text datasets that restricts the compositional understanding of CLIP models and propose a solution that involves generating "hard" negative captions and corresponding images. This is achieved through a two-step process: leveraging in-context learning for negative caption generation and utilizing text-to-image generators to create matching negative images. The improvement in modeling effectiveness is more significant with the manufactured negative sample dataset.
strength_weakness: strength_weakness: This research primarily involves leveraging LLMs to generate negative samples of similar image descriptions based on the original descriptions, followed by synthesizing the corresponding images using text-to-image models. It is well-known that using synthetic data can easily lead to model overfitting. Consequently, various techniques are typically employed to enhance data diversity, such as data augmentation. It would be intuitive to explore whether utilizing multiple models (multiple LLMs and text-to-image models) during the synthesis process might yield better results. I would be interested in seeing an analysis related to this issue, which could make this paper comprehensive.
questions: questions: See Weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: To enhance the compositional capabilities of CLIP, authors propose to generate “hard” negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution.
strength_weakness: strength_weakness: The novelty and contribution are limited. While [1] builds negative samples from the text perspective, similar to [1], authors primarily build additional negative samples from the image perspective with LLM and generated images. 

Given the existing observations (bag-of-words phenomenon, etc.) and soulutions presented in [1], I think the contribution of the current version is marginal. 


### Reference
[1] When and why vision-language models behave like bags-of-words, and what to do about it? ICLR 2023
questions: questions: see weakness
limitations: limitations: n/a
paper_summary: paper_summary: This paper proposes a TripletCLIP, a pre-training framework aimed at enhancing compositional reasoning task. It computes the NegCLIP objective separately for each, with the image serving as the anchor in each instance. For training TripletCLIP, such triplet data is constructed as follows: Hard negative captions are generated from the original LaCLIP captions using a large language model (LLM) through in-context learning mechanism. Hard negative images are created from text-to-image diffusion models. The experiment section compared various hard negative (HN) based objectives and demonstrated the superior performance of the proposed TripletCLIP mechanism across a wide range of benchmarks: compositionality, zero-shot retrieval, and classification.
strength_weakness: strength_weakness: - [W1] The effectiveness of the TripletCLIP objective is less convincing. Although the paper considered some hard negative (HN)-based baselines such as NegImage and NegCLIP++, it did not compare with HN baselines that jointly consider both HN text and image in the loss calculation, such as [1, 2, 3, 4, 5]. I understand that some works are concurrent, but it would be more comprehensive to see whether the design of the TripletCLIP objective is optimal, either quantitatively or conceptually. In addition, it is necessary to note the 'previous' work that generates hard negative images to enhance compositional reasoning tasks [2, 3]. Accordingly, the sentence from [L136-137] needs to be adjusted. 


- [W2] Though there are consistent improvements, the absolute level of performance is relatively weak compared to the 'default' CLIP models from the CyCLIP [1] and ALIP [6] papers (in terms of zero-shot imagenet and retrieval tasks). This may be due to the reduced computational resources during pre-training, which limits the significance of the results in the context of pre-training. The only possibility left, and neither the triplet data nor the pre-trained model shows the evidence of benefiting further downstream tasks in applications such as advanced VLMs like LLaVA, or text-to-image generation, as noted in [L70-72]. 


- [W3] Connected to [W2], if only a single GPU is used for pre-training, it would be an attractive alternative to fine-tune the pre-trained CLIP model with the (subset of) triplet data. It's clear that the proposed methodology remains valid, and it would be interesting to see whether employing triplet data with the proposed hard negative objectives proves superior compared to other compositional reasoning methodologies in the context of fine-tuning [7, 8], specifically on both compositional reasoning and retrieval/classification tasks. If the authors wish to focus solely on the pre-training context, referencing other previous work discussing both pre-training, as well as fine-tuning is missed [9].


- [W4] It is suggested to consider including additional compositional benchmarks for evaluation that consist of counterfactual image-text pairs, similar to the Winoground style: EqBen [2], COCO-Counterfactuals [10], MMVP-VLM [11], and SPEC [4], for comprehensiveness. 


In summary, the main concerns are the missing comparisons with hard negative (HN) objectives that incorporate both image and text, and the limited significance due to small-scale pre-training. Fine-tuning could be considered as an alternative approach to address these issues. Superiority over previous fine-tuning methodologies can further increase the significance. Meanwhile, I believe the large-scale triplet data constructed in the paper is valuable and could promote further studies in various aspects. 

---

References

[1] Goel et al., CYCLIP: Cyclic Contrastive Language-Image Pretraining, in NIPS 2022.

[2] Wang et al., Equivariant Similarity for Vision-Language Foundation Models, in ICCV 2023.

[3] Sahin et al., Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining, in WACV 2024.

[4] Peng et al., Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding, in CVPR 2024.

[5] Singh et al., Learn “No” to Say “Yes” Better: Improving Vision-Language Models via Negations, in arXiv preprint 2024.

[6] Yang et al., ALIP: Adaptive Language-Image Pre-training with Synthetic Caption, in ICCV 2023.

[7] Doveh et al., Teaching Structured Vision&Language Concepts to Vision&Language Models, in CVPR 2023.

[8] Doveh et al., Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models, NIPS 2023.

[9] Singh et al., Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality, in EMNLP 2023. 

[10] Le et al., COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs, in NIPS Dataset and Benchmark Track, 2023.

[11] Tong et al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, in CVPR 2024.
questions: questions: - Missing citations [L17-19]; image classification [CITATION], segmentation [CITATION], and image-text retrieval [CITATION]

- From [L180], it needs referring to the proper section in the appendix

- From [L227], what is the meaning of the 'real' hard negatives?
limitations: limitations: Limitations are adequately and honestly addressed in the appendix.
paper_summary: paper_summary: The paper proposes enhancing CLIP's compositional reasoning by generating high-quality negative image-text pairs using LLMs and text-to-image models. This approach improves beyond previous works that considered unrealistic, rule-based captions and unexplored negative images. Experiments show consistent improvements across different tasks.
strength_weakness: strength_weakness: - **Limited scaling**: The authors acknowledged the lack of experiments on increasing data and model size due to academic budget constraints.
- **Limited study on the selection of LLM and image-to-text model**: The authors acknowledged the limited study on how the selection of the models could affect the quality of the TripletData.
- **Nit**: Consider using \log and \exp in the equations.
questions: questions: - Have you considered splitting the test set and evaluating the models on the TripletData?
- Could you clarify how the total of 13M image-text pairs is calculated? The numbers provided (2.6M for CC3M and 8.6M for CC12M) in Section 4.1 add up to 11.2M.
- Since the negative images are generated from a text-to-image model, there's likely a distribution mismatch from the real positive images. Have you inspected this potential discrepancy?
- Can you elaborate on the relatively poor performance on the Winoground benchmark compared to other compositional tasks? Are there specific aspects of Winoground that TripletCLIP struggles with?
- There's a mention of "MiraData" in the Appendix - is this a typo for TripletData, or is it referring to something else?
limitations: limitations: The authors have clearly addressed limitations in their paper.

Comments: title: Reply to Reviewer U1Yz
Reply to Reviewer m3ja
Reply to Reviewer f4Gf
Reply to Reviewer ZTUH
Please read the author rebuttal, other reviews and respond to the authors NOW!
comment: Thank you for raising the score. While incorporating hard negative scores might seem obvious, much of the existing literature overlooks the significance of negative images and how to utilize them effectively. Through this work, we aim to offer a more comprehensive perspective on compositionality and demonstrate how recent generative models can aid in this, even in their simplest forms.
Thank you for all your time and attention given to our work. We are pleased to see that all concerns have been effectively addressed.

Yes, we are planning to release the all versions of the TripletData (CC3M, CC12M, and High-quality filtered version -- 2.8M).
Thank you for raising the score! We are pleased to see that all concerns have been effectively addressed.

Yes, we will add all the new experiments in the camera-ready version as they will make our work more impactful. We are also planning to release the code, data, and checkpoints.
We are pleased to see that all concerns have been effectively addressed.
Dear Reviewers,

Thanks to those of you who already responded to the authors acknowledging the rebuttal and asking follow-up questions if any.

Those who have not responded yet, please do the following ASAP: thoroughly read the rebuttal, the other reviews and respond to the authors about whether all your questions / concerns have been addressed or not. If not, please elaborate on which questions / concerns are still not addressed so that the authors have fair chance of addressing them before the author-reviewer discussion period ends in ~41 hours from now (August 13th, 11:59pm AoE).

Your AC
Thank you for the detailed response. I appreciate additional experiments that you provided. I have no further concerns. Overall, I’m looking forward to the dataset that you will provide.
Thanks for authors' explanation. Although the methodology is simple and the results are actually not surprising to me as it is pretty intuitive that constructing such “hard” negative would lead to improvements, the additional analytical experiments and analysis could be interesting. I would raise my score to 5.
Thanks for the response. I have no more concerns.
Thank you for the detailed response. I have no remaining concerns and will adjust the score to 6.

I believe that including additional experiments with CyCLIP and fine-tuning comparisons will provide valuable references for future research.
It would be great if that part of the codebase were released.