Title: Object-centric binding in Contrastive Language-Image Pretraining
Venue: NeurIPS 2025
Accepted: yes
Keywords: object-centric representation

binding problem

object-centric language alignment
Abstract: Recent advances in vision language models (VLM) have been driven by contrastive models such as CLIP, which learn to associate visual information with their corresponding text descriptions. However, these models have limitations in understanding complex compositional scenes involving multiple objects and their spatial relationships. To address these challenges, we propose a novel approach that diverges from commonly used strategies that rely on the design of finegrained hard-negative augmentations. Instead, our work focuses on integrating inductive biases into the pretraining of CLIP-like models to improve their compositional understanding. To that end, we introduce a binding module that connects a scene graph, derived from a text description, with a slot-structured image representation, facilitating a structured similarity assessment between the two modalities. We also leverage relationships as text-conditioned visual constraints, thereby capturing the intricate interactions between objects and their contextual relationships more effectively. Our resulting model not only enhances the performance of CLIP-based models in multi-object compositional understanding but also paves the way towards more accurate and sample-efficient image-text matching of complex scenes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents OC-CLIP, a CLIP-based vision-language model enhanced with object-centric inductive biases for compositional understanding. The proposed method integrates a binding module (inverted cross-attention) with a structured similarity score guided by scene graphs. Experimental results on multiple benchmarks (SugarCrepe, GQA-spatial, COCO-spatial) show significant gains over baseline models, highlighting the benefits of slot-based image representation and relational modeling.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* A clearly articulated motivation around the binding problem in CLIP.\n* Introduction of object-centric inductive biases via slot-attention-like mechanism and structured similarity score.\n* Strong results on targeted compositional benchmarks.\n\nWeaknesses:\n* The text-only Vera baseline from the SugarCrepe paper is not reported or discussed, despite the fact that the ARO benchmark is explicitly designed to be partially solvable by language priors. This omission significantly weakens the claims regarding model improvements on ARO.\n* The paper emphasizes that it avoids hard-negative captions during training. However, hard-negative scene graphs derived from text are used instead. These serve an equivalent function, so the distinction seems more rhetorical than substantive.\n* OC-CLIP is fine-tuned on COCO, GQA, and VG, which are also the sources of the evaluation benchmarks. Meanwhile, some baselines are evaluated under stricter zero-shot settings. Without proper controls, it is unclear whether the observed gains stem from architectural improvements or data distribution alignment.\n\nQUESTIONS:\nAt this stage, due to the above concerns—particularly the lack of fair and comprehensive comparisons, the potentially misleading claims about hard-negative usage, and the unclear generalization performance—I am unable to recommend acceptance. That said, I find the direction promising and technically sound, and would be open to increasing my score if the authors can address these issues in rebuttal.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-08"}

comments: SUMMARY:
This paper proposes Object-Centric CLIP (OC-CLIP), a novel method that enhances compositional scene understanding in CLIP-like models by incorporating object-centric inductive biases. Instead of relying on hard-negative samples, OC-CLIP uses a binding module to align a slot-based image representation with a scene graph extracted from text, capturing object attributes and spatial relationships more effectively. The model achieves significant improvements on compositional benchmarks (e.g., +16.5% on SugarCrepe, +89% on COCO-spatial) and shows strong sample efficiency and scalability. However, it depends on a parser for scene graph extraction and requires further scaling to match larger CLIP variants.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Introducing the binding module to enhance the compositional understanding capabilities of CLIP-like models is intuitively making sense and technically sound.
2. The proposed algorithm is easy to integrate into the existing CLIP-like models' training pipeline.

Weaknesses:
1. The proposed algorithm appears unable to capture directional spatial relations. Take the relationship “A red apple to the left of a blue car” (Line 142) as an example. I don't think the proposed structured similarity score is able to learn the spatial relationship beween the apple and the car. Because in Eq.2, the relationship embedding is learnt from the fs([r, Ss]) + fo([r, So]), where the two terms are simply added. This operation makes "A to the left B" and "B to the left A" exactly the same, because A + B = B + A.

QUESTIONS:
Please address the question listed under the Weaknesses section. I will consider raising my final score if these concerns are adequately addressed.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting contrastive models like CLIP excel at recognizing individual objects but have limitations in understanding spatial relationships in multi-object scenes and correctly binding object attributes, a problem known as the \"binding problem\". This paper introduces Object-Centric CLIP (OC-CLIP), a new pretraining method aimed at improving the compositional understanding of Vision-Language Models (VLMs) for complex scenes. OC-CLIP boosts performance in real-world benchmarks for attribute binding and spatial relationship understanding, and achieves notable improvements in zero-shot classification and compositional understanding tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. The binding module utilizes an inverted cross-attention mechanism and introduces default query tokens to process visual information not explicitly mentioned in the text, a design that is unique.\n2. The proposed structured similarity score combines object and relationship scoring. It is also supplemented by a relational loss to enforce the learning of non-symmetric relationships, providing a novel and more refined approach to image-text alignment.\n3. Research has demonstrated that enhancing compositional understanding through model architectural design (inductive bias) can be more effective than solely relying on large-scale hard-negative augmentation. This holds significant practical implications for resource-constrained or rapid-deployment scenarios.\n\nWeaknesses:\n1. The OC-CLIP method relies on a parser to extract object-centric attributes and spatial relationships from text descriptions. Does it quantify the impact on OC-CLIP's downstream performance if the parser produces low-quality, incomplete, or erroneous scene graphs.\n2. Although the paper demonstrates OC-CLIP's scaling potential at the 15M scale, the model still needs to be further scaled to at least the 400M scale to be fully comparable to all CLIP variants.\n3. OC-CLIP introduces a notable computational overhead due to the binding module requiring additional cross-attention operations. Despite the paper implementing mitigation strategies, such as using a smaller text encoder and operating in a reduced embedding space for the binding module, there is still a significant 2.2x overhead for the base architecture.\n4. The core of the binding module is to generate \"slots\". However, the paper does not extensively discuss how the number of slots or their granularity affects performance. A fixed number of slots might not adapt well to scenes with a dynamic number of objects, and the choice of slot granularity could also influence the model's understanding capabilities. The paper does not provide insights into how to select or dynamically adjust the number of slots.\n\nQUESTIONS:\n1. Despite the implemented mitigation strategies, such as using a smaller text encoder and operating the binding module in a reduced embedding space to lower computational overhead, the base architecture still exhibits a significant 2.2x overhead. Aside from the strategies mentioned in the paper, are there any other potential optimization methods that could further reduce this computational cost?\n2. The paper discusses error modes of different parsing methods in Appendix A.4 and qualitatively analyzes the advantages of LLM-based parsers in relation understanding. To what extent would OC-CLIP's downstream performance be impacted if the parser generates low-quality scene graphs, e.g., incomplete or containing erroneous objects/relations? Could this impact be mitigated through the model's inherent robustness or specific training strategies?\n3. The paper does not extensively discuss how the number of slots or their granularity affects performance. In designing the binding module, how is an appropriate number of slots chosen to adapt to a dynamic number of objects in a scene? Has there been exploration into mechanisms for dynamically allocating the number of slots, or the impact of combining different granularities on the model's understanding capabilities? What types of visual information do the \"default query tokens\" introduced in the binding module learn during training, and do they offer insights into the model's \"background understanding\" or \"non-core object recognition\" abilities?\n4. Considering how OC-CLIP enhances CLIP's compositional understanding through object-centric inductive biases and a structured similarity score, when applying OC-CLIP's core mechanisms to large Vision-Language Models, could this effectively improve their visual comprehension and significantly mitigate visual hallucination, specifically attribute hallucination and relation hallucination? More precisely, how might OC-CLIP's slot-structured representations and scene graph constraints help VLMs more accurately bind object attributes and understand spatial relationships, thereby reducing the generation of factually incorrect image descriptions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The authors introduce Object-Centric CLIP (OC-CLIP), which enhances CLIP with object-centric inductive biases to improve compositional understanding which cannot be solved by CLIP. The proposed model not only enhances the performance of CLIP-based models in multi-object compositional understanding, but also paves the way towards more accurate and sample-efficient image-text matching of complex scenes.

STRENGTHS AND WEAKNESSES:
Strengths:
1.It tries to deal with the complex compositional scenes involving multiple objects and their spatial relationship which cannot be solved by CLIP.
2.It can work with noisy data with zero-shot.

Weakness:
1.These modules do not seem to be a good plug in to current vLLM (not end-to-end), and they may decrease the training and inference speed.
2.The performance does not seem to be good in some tasks on Table 1.
3.The parsing procedure at the beginning may limit the performance.

QUESTIONS:
1.How to combine the proposed method with current vLLM without decreasing the training and inference speed as it is not end-to-end.
2.How to explain the performances of the proposed method is not so good on some tasks.
3.Figure 2(b) and c do not seem to be straightforward and easy to understand. It misleads me to think of correlation.
4.How will the parser impact the performance?

LIMITATIONS:
1.How to combine the proposed method with current VLLM is not discussed, especially the performance of training and inference.
2.How to further improve the performance of the method as in some tasks it is not so good.
3.This work has not discussed the impact of parser on the whole workflow, which may lead to a disaster.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-07