OC-CLIP : Object-centric binding in Contrastive Language-Image Pretraining

Authors: Rim Assouel, Pietro Astolfi, Florian Bordes, Michal Drozdzal, Adriana Romero-Soriano
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: object-centric representations, object binding, CLIP, contrastive learning, compositional image-to-text retrieval

Abstract:
Recent advancements in vision-language models (VLMs) have been driven by contrastive models like CLIP which learn to associate visual information with their corresponding text descriptions. However, these models have limitations in understanding complex compositional scenes involving multiple objects and their spatial relationships. To address these challenges, we propose a novel approach that diverges from traditional data-centric methods of enhancing model performance with hard negatives examples. Our work instead focuses on integrating sufficient inductive biases into pre-trained CLIP-like models to improve their compositional understanding without using additional data annotations. We introduce a binding module that connects a scene graph of the text with an induced graph-like representation of the image, facilitating a structured similarity assessment. We also leverage relationships as text-conditioned visual constraints, thereby capturing the intricate interactions between objects and their contextual relationships more effectively. Our resulting model (OC-CLIP) not only enhances the performance of CLIP in multi-object compositional understanding but also paves the way for more accurate and efficient image-text matching in complex scenes.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729065479011

thread_type:
official_review

<thread_object>
creator: reviewer_hgdj

<content>
summary:
Most existing vision and language models (VLMs) face challenges when dealing with compositional reasoning in images and texts. To enhance the compositional reasoning capabilities of VLMs, previous research primarily refined these models through the construction of specialized datasets, such as introducing hard negative examples via textual modifications. This paper presents an innovative approach aimed at overcoming the limitations of existing VLMs when handling complex compositional scenes involving multiple objects and their spatial relationships. The work introduces a novel framework focused on integrating sufficient inductive biases into pre-trained CLIP-like models, providing additional guidance through the use of scene graphs rather than relying on traditional hard negative augmentation strategies. Experimental results indicate that such a design optimizes the performance of CLIP models across multiple benchmark datasets.

strengths:
The idea presented in this paper of using scene graphs to help VLMs improve their compositional reasoning capabilities is novel and interesting, and it appears to be an effective approach.

This paper proposes a new structured similarity score based on contrastive loss, consisting of an object scoring function and a relationship scoring function, which seems capable of effectively evaluating the relationships between nodes in the scene graph.

The organization of this paper is clear, the problem definition is explicit.

The proposed method has achieved effective improvements on certain datasets.

weaknesses:
This paper lacks convincing evidence in several key areas, specifically:

The experimental comparisons are clearly insufficient. Mainstream datasets in this field, such as the CoCo and Flickr datasets from the ARO dataset, the VL-Checklist dataset, and the sDCI dataset, have not been included in the comparative experiments. Additionally, mainstream methods such as DAC, sDCI, and SVLC have not been compared, making the experimental results significantly unconvincing.

VL-Checklist: "Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations." arXiv preprint arXiv:2207.00221 (2022).
sDCI: "A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
DAC: "Dense and aligned captions (dac) promote compositional reasoning in vl models." Advances in Neural Information Processing Systems 36 (2024).
SVLC: "Teaching structured vision & language concepts to vision & language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Ablation studies are missing. There is no analysis of the ablation of the loss function or the selection of object and relationship extraction models.

The accuracy of the scene graph heavily relies on the relationship extraction model. However, models like Llama3 also face issues with misalignment in compositional reasoning. This paper does not provide any method to ensure the accuracy of the scene graph.

There is no analysis of whether CLIP retains its original image understanding capabilities after fine-tuning with the scene graph.

The proposed loss function in the paper includes multiple hyperparameters, but there is no experimental or theoretical analysis provided.

questions:
Experimental Comparisons:
1. Why not include comparisons with mainstream datasets like CoCo, Flickr (ARO dataset), VL-Checklist, and sDCI?

2. Why are mainstream methods such as DAC, sDCI, and SVLC not included in your experiments?

Ablation Studies:
3. Provide an analysis of the ablation of the loss function.

4. Include an ablation study on the selection of object and relationship extraction models.

Scene Graph Accuracy:
5. How do you address the issue of misalignment in compositional reasoning, as seen in models like Llama3?

6. What methods do you propose to ensure the accuracy of the scene graph?

CLIP Fine-Tuning:
7. Analyze whether CLIP retains its original image understanding capabilities after fine-tuning with the scene graph.

Loss Function Hyperparameters:
8. Provide experimental or theoretical analysis to justify their inclusion and effectiveness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730246591758

thread_type:
official_review

<thread_object>
creator: reviewer_uy5m

<content>
summary:
The paper proposes augmenting CLIP contrastive finetuning with a "binding module" supervised by objects and relations data extracted from high-quality human-annotated data via an LLM. The proposed architecture of the binding module explicitly matches objects and relations to learned attention slots applied to visual encoder output. Noise register tokens are also used in the matching. Experiments were conducted on either synthetic data (train and test generated by the authors) or training on high-quality real VG / COCO / GQA annotated data. Testing of the real data-trained models is primarily done on ARO, SugarCrepe, and GQA (which are derived from the same datasets used for training). Additionally, WinoGround performance improvements are reported. Seems some important baselines are missing.

strengths:
* compositional reasoning is still an interesting problem, yet as the field has mostly shifted to decoder LMMs it is unclear how long the encoder-only models can still be compared just to themselves. Decoder LMMs have much higher compositional reasoning performance.
* designing and training explicit alignment modules for objects and relations is interesting
* performance improvements reported

weaknesses:
1. synthetic data experiments seem limited and less convincing. 
2. in the real data experiments, the model is finetuned on high-quality human annotated data from the same datasets out of which the evaluation benchmarks (ARO, SugarCrepe, COCO) are constructed. It seems that the method performs supervised training as opposed to hard-negative based techniques mentioned which train on unconstrained internet data (LAION, CC, etc)
3. seems that important baselines are missing as well as additional compositional reasoning benchmarks to compare to those baselines.
e.g. https://paperswithcode.com/sota/visual-reasoning-on-winoground - for many higher reported results on WinoGround; more hard negative mining methods (DAC = https://arxiv.org/pdf/2305.19595 being one example); etc
4. no zero-shot evaluation of the fine-tuned model is done - the model is fine-tuned for 100 epochs on supervised data explicitly focusing objects and relations there - one might expect significant degradation on zero-shot benchmarks like ELEVATER used in DAC above. It is important to show the proposed technique does not solve some compositional reasoning issues on expense of the general applicability of the model (zero-shot performance)
5. Ablation of the proposed components and losses is completely missing, but should be a significant part of the paper informing the reader on the relative contribution of components, insights related to them, and so on.
6. More minor:
a. the "rel" loss is very similar to the hard negative losses employed by others, a bit counter the (a bit unjustified) reasoning of the authors that hard negative based approaches are detrimental (if anything they require much less supervision than the proposed approach) - also rel loss is not ablated (well nothing is ablated except the synthetic dataset)
b. reliance on LLM parsing is not ablated, what if it introduces noise how to deal with it? Parsing real captions (eg if one would try to apply this on unconstrained data like LAION or CC) seems a much harder task compared to negative augmentation used in other works (contrary to the reasoning of the authors in the intro)
c. in eq. 3 - what prevents the collapse of f_s and f_o to just extracting the "r" part from the concat? not ablated / analyzed...

questions:
please see weaknesses, lots of questions there
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648354929

thread_type:
official_review

<thread_object>
creator: reviewer_ar8u

<content>
summary:
This paper roposes a approach that diverges from commonly used strategies relying on the design of hard-negative augmentations. This work instead focuses on integrating sufficient inductive biases into pre-trained CLIP-like models to improve their compositional understanding without using additional data annotations.

strengths:
-The proposed method is simple and intuitive.

-The experimental performance on several datasets demonstrate large improvements in terms of accuracy.

-Interesting idea to generate object/relation tags by utilizing LLM to help scene graph undetstanding

weaknesses:
-Using LLM for parsing and training with object + relation classification loss, are a bit too straightforward to be claimed as a contribution

-One possible way to make LLM contributions more reasonable is to compare multiple different LLM/cues and analyze the results/performance.

-Extract the computational cost of labels with LLM. LLM is known for its large scale and requires a significant amount of computing resources. I would like to know how much computation is required to process the visual genomic data used in the paper.

questions:
-Can some MLLMs (e.g., internVL, llava) be regarded as text parser? Can MLLMs (use text and image) parse better relation and objects than LLM from text

-Tagalign[2] also use a LLM-based as text parser. Suggest comparing with this paper.

[2] Tagalign: Improving vision-language alignment with multi-tag classification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715813512

thread_type:
official_review

<thread_object>
creator: reviewer_kw33

<content>
summary:
The paper introduces OC-CLIP, a novel approach aimed at enhancing the compositional understanding capabilities of CLIP-like models. 
The key innovation is the integration of a binding module that connects a scene graph derived from text descriptions with a slot-structured image representation. 
The proposed structured similarity score between the two modalities seems to capture interactions between objects and their contextual relationships more effectively.

strengths:
(1) The introduction of the binding module is well-motivated and technically sound.
(2) The structured similarity score, composed of an object scoring function and a relationship scoring function, seems a reasonable method for modeling the scene graph and the visual slots relationships.

weaknesses:
(1) The binding module and structured similarity score introduce additional complexity to the CLIP model. The authors should provide an analysis of the trade-offs of computational overhead and performance gains.

(2) The authors evaluate OC-CLIP on several standard benchmarks of compositional understanding and spatial relationship understanding. However, as a fundamental multi-modal representation model, additional downstream experiments on tasks such as zero-shot image classification, video action recognition, geo-localization, text-to-image retrieval etc. should strengthen the paper.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732303025168

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal Answer - Part 1

comment:
We thank all the reviewers for taking the time to read and review our paper. We believe these reviews helped us **considerably improve our work**. We answer each review separately but this comment is to address and comment on some recurring points of the majority of the reviewers.

We would first thank all the reviewers for acknowledging our work to be well-motivated by the importance of improving compositional reasoning (Rev. **kw33** and **uY5M**) and our newly proposed binding module to be simple and reasonable yet novel, interesting, and sound (Rev. **kw33**, **ar8U**, **uY5M**, **HgdJ**). Moreover, we are glad to see that Rev. **ar8U**, **uY5M**, **HgdJ** highlighted the empirical effectiveness of our method.

Importantly we made the **following main changes** in the paper  ( and highlighted them in blue in our draft) :

- We trained CLIP and OC-CLIP from scratch on CC12M in addition to training the OC-CLIP’s binding module on top of a pre-trained CLIP on the COCO dataset. \[Addressing concerns of Reviewers **kw33**,**uY5M** and **HgdJ** about general understanding capacity of OC-CLIP\]. And show performance gains of OC-CLIP (**\+9.2% on ImageNet**).  
- We included results on additional benchmarks: ARO/VL-Checklists and the ELEVATER datasets \[As asked by Reviewer  **HgdJ**\], and show our method is competitive in all of them  
- We included new baselines: DAC, DCI and CLIP-SVLC \[As asked by Reviewers  **HgdJ, uY5M** \] and show that they still fall behind OC-CLIP in splits requiring precise attribute-binding despite their large-scale finetuning.   
- We introduced additional ablations and explored different scene parsers, architecture components and loss terms. \[ As suggested  by Reviewers **HgdJ,uY5M, ar8u**  \] and show the importance of OC-CLIP design choices.  
- We presented a computational analysis of OC-CLIP \[ As asked by Reviewer **kw33**\], evaluating computational overhead vs performance gains.


We give further details below :

**Additional compositional Baselines/Experiments**

1. Added Baselines

In our study, we focused on comparing OC-CLIP to  hard-negative-based baselines that generate hard negatives **in-distribution**, specifically from the COCO/VG/Flickr domain. This is why we included methods such as NegCLIP, CE-CLIP, and CounterCurate, which also focus on generating hard negatives within these specific domains.

However, we acknowledge the importance of broader comparisons and have now included additional methods such as DAC,  DCI and CLIP-SVLC  to our analysis (see Table 2 and Table 5).. These methods are different in that they incorporate both hard negative mining on a **larger scale** and **dense captioning/recaptioning**, which is not the primary focus of our work. Nonetheless, their inclusion provides a more comprehensive comparison and allows us to better position our approach within the broader landscape of compositional reasoning benchmarks.

Notably, we see that on the sugarcrepe/ARO splits that assess for precise object-attribute binding (swap attribute/ARO-A) all these methods still fall behind : 

|  | OC-CLIP | DAC-LLM | DAC-SAM | DCI | CLIP-SVLC |
| :---- | :---- | :---- | :---- | :---- | :---- |
| Swap-Att | 88.5 | 74.1 | 75.3 | \\- | \\- |
| ARO-A | 82.0 | 73.9 | 70.5 | 67.6 | 73.0 |

Which confirms the results we presented with the synthetic experiments that the hard-negative approach, even at scale,  is not sufficient to solve the bag-of-words behavior of CLIP and justifies the inclusion of more structured inductive biases within the architecture of the model like we do for OC-CLIP

2. Added Benchmarks

Our goal is to evaluate CLIP-like models compositional understanding in **plausible and grammatically correct cases**, the SugarCrepe benchmark  has identified exploitable  textual biases in previous mainstream procedurally-generated hard negatives benchmarks like the COCO and Flickr set of ARO and VL-checklist. Specifically the \[1\] shows that procedurally generated hard negatives are either highly grammatically incorrect and can be identified by a blind model or by a good language model. SugarCrepe proposes to follow the same fine-grained taxonomy on attributes, objects, relationships  as VL-checklist but ensures that the hard-negative are not distinguishable by a blind model. However, as requested by the reviewer we **added the results on the full ARO suite and VL-Checklist  in Table 5**. We also added DAC, DCI, SVLC as suggested as baselines in sugarcrepe and ARO/VL-checklist.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper focuses on improving the compositional understanding of pre-trained CLIP-like models without using additional data annotations. To achieve this, the paper introduces a binding module relying on a scene graph of the text to facilitate structured similarity assessment. The idea of the paper is well-explained and discussed with ablations/experiments/comparisons. The introduced binding module might incur extra costs for inference, and due to the nature of the design, the original capabilities of the CLIP model might be lost. Moreover, the paper still lacks enough discussion about the direct relationship between the quality of the compositional data (either generated by LLMs or other tools) and the final evaluation. Based on these weaknesses pointed out by the reviewers, I would not recommend accepting the paper.