Title: Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-modal learning

Compositionality

CLIP
Abstract: Vision-language models like CLIP have demonstrated remarkable zero-shot capabilities in classification and retrieval.
However, these models often struggle with compositional reasoning - the ability to understand the relationships between concepts. A recent benchmark, SugarCrepe++, reveals that previous 
works on improving compositionality have mainly improved lexical sensitivity but neglected semantic understanding. In addition, downstream retrieval performance often deteriorates, although one would expect that improving compositionality should enhance retrieval. In this work, we introduce CLIC (Compositionally-aware Learning in CLIP), a fine-tuning 
method based on a novel training technique combining multiple images and their associated captions. CLIC improves compositionality across architectures as well as differently pre-trained CLIP models, both in terms of lexical and semantic understanding, and achieves consistent gains in retrieval performance. This even applies to the recent CLIPS, which achieves SOTA retrieval performance. Nevertheless, the short fine-tuning with CLIC leads to an improvement in retrieval and to the best compositional CLIP model on SugarCrepe++.
All our models and code are available at https://clic-compositional-clip.github.io.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents CLIC, a fine-tuning strategy aimed at enhancing compositional reasoning in CLIP-based vision-language models. The method combines multiple images and associated captions during training, with the goal of improving both lexical and semantic compositionality while maintaining or improving downstream retrieval performance. Results on SugarCrepe++ and various retrieval tasks are promising.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper addresses a well-recognized weakness in CLIP models—compositional generalization—and proposes a reasonably efficient solution.\n* Experimental evaluation is comprehensive across multiple CLIP variants, including strong baselines like CLIPS.\n* The paper is generally well writen and easy to follow.\n\nWeaknesses:\n* The core contribution—mixing multiple images and captions during training—can largely be interpreted as a form of compositional data augmentation. While effective, the methodological novelty is relatively incremental, and does not introduce fundamentally new learning principles.\n* The inclusion of multiple loss terms introduces several hyperparameters. These are not thoroughly analyzed, and may affect the generalizability or reproducibility of results. A more systematic ablation or sensitivity study would help clarify this.\n* Since CLIC is applied as a fine-tuning method on top of pre-trained CLIP models, it is crucial to ensure that their strong generalization and zero-shot capabilities are preserved. The paper emphasizes improvement in compositionality and retrieval but does not clearly analyze potential trade-offs in broader zero-shot tasks.\n\nQUESTIONS:\nGiven the growing prominence of large vision-language models (LVLMs), a discussion—preferably supported by quantitative or qualitative evidence—on whether CLIC-style compositional fine-tuning benefits or transfers to LVLMs (e.g., BLIP-2, LLaVA) would greatly enhance the paper’s relevance to the current landscape.\n\nThe paper addresses an important problem and presents a promising direction, but the limited novelty and open questions around generalization and practical utility suggest that further clarification or extensions would strengthen its impact. I'm leaning towards borderline reject before rebuttal and open to discussion with authors and other reviewers.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

comments: SUMMARY:
This paper introduces CLIC: a finetuning method designed to enhance the compositional understanding of vision-language models like CLIP. The core idea is to train on concatenated image pairs and their corresponding captions. The authors generate positive examples with diverse sentence structures and hard negatives by swapping words between captions. The authors demonstrate that CLIC improves performance on compositionality benchmarks like SugarCrepe++ and WinoGround, while maintaining or improving performance on standard retrieval and classification tasks. Experiments are run across different CLIP architectures and pre-training datasets. The method is efficient by fine-tuning only the text encoder.

STRENGTHS AND WEAKNESSES:
Strengths:
- Addresses the known limitation of vision-language models regarding compositional reasoning.
- Proposes a novel and efficient data generation and fine-tuning strategy using image/caption concatenation and word swapping for hard negatives.
- Demonstrates effectiveness across various CLIP architectures and pre-training datasets.

Weaknesses:
- The reliance on SpaCy for hard negative generation, while efficient, might be less nuanced and powerful compared to LLM-based approaches.
- Would be nice to add visual examples and failure case analysis

QUESTIONS:
See weaknesses section

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CLIC, a method to train and improve CLIP models' compositionality. The key idea is to randomly concatenate two images, and utilize their associated captions to make alternative positive captions and hard negative captions to train CLIP models to be able to distinguish lexically similar but semantically different captions, while at the same time being able to understand lexically different but semantically similar captions. Experiment results show some improvements on certain datasets, with more improvements on the text encoder.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles an important problem on improving CLIP models' compositional understanding capability as recent work has pointed out that CLIP models are not sensitive to finer-grained semantic understanding.\n\n- The proposed method does not require much additional cost in generating hard positive and hard negative captions, whereas previous work may rely on external models in rewriting captions and generating hard negative captions (or images).\n\n- Experiment results show some improvements on the recently introduced SugarCrepe++ dataset (with more gain shown on text-to-text setting). The resultant CLIC trained model also mostly retains standard classification and retrieval accuracy.\n\n\nWeaknesses:\n- The introduced method that concatenates two randomly sampled images appears somewhat adhoc to me. This changes the image aspect ratio and may negatively affect the model's capability on standard images. This can be seen from CLIC models degraded performances on retrieval/classification performance in Table 2, as compared to original CLIP and baselines like NegCLIP (comparing both MS-COCO trained models).\n\n- The hard negative captions generated by CLIC through swapping words from two captions can introduce unrealistic captions as seen in Figure 2 (\"A water wearing a black hat\"). As pointed out in SugarCrepe work, these implausible captions might introduce biases into the resultant model. Thus, on the SugarCrepe dataset where the dataset specifically removed implausible negative captions, CLIC models' performances appear to be generally worse than other baselines. Could the authors discuss further on this issue?\n\n- The hard negatives generated from CLIC are almost like replacing words in a caption with other random words (in CLIC, it is done by sampling from another randomly selected caption). Mechanically, this is in fact not too different from existing methods in generating hard negatives by sampling the replacing words from some dictionary.\n\n\n- While the experiment results show some improvements on SugarCrepe++ dataset, I am concerned that for ITT setting, the improvements mostly comes from training on larger dataset (as CLIC-COCO shows mixed results compared to NegCLIP which is also trained on COCO). On TOT settings, while there are improvements, this might mainly come from the introduction of uni-modal loss, which I do appreciate but is not a significant contribution.\n\nQUESTIONS:\nPlease see the questions in weaknesses above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper looks at how VLMs like CLIP struggle with with how images are composed, often confusing semantic concepts and their physical relation to each other. This is a well-documented and measured by several benchmark datasets. The authors propose a new fine-tuning method called CLIC, which trains the model using a variety of altered positive captions and negative captions for image-caption pairs. They show this method of training with negative captions helps models with semantic understanding, not just lexical. The SugarCrepe++ benchmark was used.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n\nThe method of combining random images and negative captions by substituting one word offers a cheap way to improve how VLMs understand the meaning of captions. It is flexiblee nough to be used across different types of data.\n\nBy joining together two random images and their captions, the training data becomes more complex, which pushes the model to handle combinations of concepts instead of just single objects.\n\nThe model achieves high scores on tough benchmarks like Sugarcrepe++, and on some tasks like downstream classification and retrieval, it matches or comes close to the best models available.\n\nWeaknesses\n\nSubstituting a single word to make a negative caption may result in unnatural sentences and since the process might often produce sentences that are not meaningful or not grammatical, the model may learn to spot textual artifacts rather than actual compositional relationships. \n\nAlso, the model might learn to ficus on lists of captions, separated by a period and with abrupt topic change.\n\nReal photos often contain multiple objects and layers of relationships that are more complex than just combining two random scenes. This might be addressed in a followup study.\n\nSugarCrepe++ is the main test for compositionality in this paper, and while the model does well on it, this benchmark doesn’t require understanding of complex, hierarchical relationships. So, there may still be important gaps in real-world performance.\n\nThe paper does not report error bars or statistical significance, so it’s hard to know if the improvements are always reliable, especially when differences are small. The authors address this by saying how it is computationally expensive. However, the differences in Table 1 and 4 are large enough to heurisitally assume there;s likely to be statistical significance. And in Table 2, the differences are small enough to assume statistical insignificance, or at least practical insignificance.\n\nQUESTIONS:\nCould the authors explain why they are not concerned about the negative captions being ungrammatical or with awkward syntax?\nIf they can show they get the same results with grammatical and proper sentences, then my issue of the model learning artifacts has been weakened.\n\nSimilarly, can the authors provide any analysis that rules out the possibility that the model is learning to exploit superficial artifacts like lists or abrupt topic changes?\nMy confidence in the model’s compositional abilities would increase.\n\nDo the authors have suggestions for how their approach could be extended to encourage learning of more integrated and hierarchical compositionality?\nConcrete proposals or pilot experiments in this direction would help demonstrate that the approach is not limited by its current synthetic design, and would positively impact my assessment.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}