Object-centric binding in Contrastive Language-Image Pretraining
Authors: Rim Assouel
Pietro Astolfi
Florian Bordes
Michal Drozdzal
Adriana Romero-Soriano
Venue: NeurIPS 2025 poster ()
Keywords: object-centric representation
binding problem
object-centric language alignment
Primary area: deep_learning
TL;DR: object-centric inductive biases to a address the binding problem  in CLIP-like pretraining

Abstract: Recent advances in vision language models (VLM) have been driven by contrastive models such as CLIP, which learn to associate visual information with their corresponding text descriptions. However, these models have limitations in understanding complex compositional scenes involving multiple objects and their spatial relationships. To address these challenges, we propose a novel approach that diverges from commonly used strategies that rely on the design of finegrained hard-negative augmentations. Instead, our work focuses on integrating inductive biases into the pretraining of CLIP-like models to improve their compositional understanding. To that end, we introduce a binding module that connects a scene graph, derived from a text description, with a slot-structured image representation, facilitating a structured similarity assessment between the two modalities. We also leverage relationships as text-conditioned visual constraints, thereby capturing the intricate interactions between objects and their contextual relationships more effectively. Our resulting model not only enhances the performance of CLIP-based models in multi-object compositional understanding but also paves the way towards more accurate and sample-efficient image-text matching of complex scenes.

Object-centric Binding in Contrastive Language-Image Pretraining
Rim Assouel 1.2.3
Florian Bordes'
Pietro Astolfi
Michal Drozdzal-
Adriana Romero-Soriano- 2,24,5
FAIR at Meta
Mila Québec Al Institute
3 Université Montréal
MeeGill University
$Canada CIFAR Al Chair
Abstract
Recent advances in vision language models (VLM) have been driven by contrastive models such as CLIP, which learn to associate visual information with their corresponding text descriptions. However, these models have limitations in understanding complex compositional scenes involving multiple objects and their spatial relationships. To address these challenges, propose novel approach that diverges from commonly used strategies, which rely on the design of finegrained hard-negen -ngattiv augmentations. Instead, our work focuses integrating inductive biases into CLIPP-lik models to improve their compositional understanding. To that end, introduce binding module that connects scene graph, derived from a text description, with slot-structured image representation, facilitating structured similarity assessment between the two modalities. We also leverage relationships as text-conditioned visual constraints, thereby capturing the intricate interactions between objects and their contextual relationships more effectively. Our resulting model not only enhances the performance of CLIP-based models in multi-object compositional understanding but also paves the way towards more accurate and sample-efficient image-text matching of complex scenes.
1 Introduction
Recent advancements modii-modal representation learning have primarily been enabled by the intro- duction of CLIP [Radford et al., 2021). CLIP learns aligned image-tett representations from Internet- scale data. Despite its success, CLIP exhibits limitations in understanding complex scenes composed of multiple objects (Kamath et al. 2023 Yuksekgonul et al. 2023a Doveh et al. 2023b Paiss et al.) 2023]. For instance, while capable of recognizing individual objects, CLIP struggles es with interpreting spatial relationships among objects in the scene(e.g.. "the cat is to the left of the mat" "the cat is to the right of the mat") and adequately associating objects with their corresponding attributes (e.g., "a red square and blue circle" "a blue square and red circle"). The process of acquiring this compositional understanding of the world known as the binding problem in the literature, and may be decomposed into segregation, representation. and composition problems [Greff et al. 2020b).
Efforts to improve the compositional understanding of CLIP-like models have largely relied on leveraging hard negutive exampley" either in the text space Kalantidis et al. 2020 Yuksekgonul
al. 2023b] Zhang et al. a [2024b] Doveh et al. [2023b Paiss et 2023 to improve sensitivity
order of words and subtle textual differences or the image space Awal et al. [2024] Le et al. 2023
Zhang et al. 2024a] - to improve sensitivity to subtle visual differences. Although these methods have
somewhat improved CLIP-like models' performance
compositionality benchmarks (Par
calabescu et al. 2022 Zhao et al. 2022 Yuksekgonul et al. 2023b Hsich et al. 2023b), they do not
HHard negativess are additional samples that either contain subtle visual changes in the image and/or subtle linguistic/smmantic difference the caption and are sampled as negatives in the same batch.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

explicitly address the binding problem as they focus mainly on enhancing the model's 's representation capabilities with additional data, hindering their generalization to unseen scene compositions.
Yet, the literature on object-centric representation learning (Eslami et al.) 2016 Greff et al. 2020a
Locatello et al., 2020. Wu et al., 2023 Seitzer et al. 2023] has long focused on devising methods to address the segregation and representation problems as way to facilitate the subsequent compositional processing of images. This has led to the development of inductive biases segregate different objects in scene into distinct representational slots, which have been shown to naturally scale to an increasing number of visual objects and relations [Locatello et al., 2020 Webb et al., 2023 Mondal et al., 2024 Elsayed et al.] 2022]. To the best of our knowledge, advances in object-centric representation learning are yet be explored in the vision-language domain.
Therefore, in this paper, we focus on enhancing the compositional scene understanding of CLIP-like models by leveraging advances from object-centric representation learning. In particular, we propose to endow CLIP--assd vision-language architectures with segregation and composition capabilities. Our core idea is adapt the slot-centric representation paradigm for CLIP architectures and dynamically align each representational slot with the object entities mentioned in the text. To do so, we design a binding module that connects a scene graph, derived from the textual description, with slot-structured image representation. We utilize the scene graph's 's relationships as constraints to effectively capture the complex interactions among the visual entities represented as slots. Our enhanced model, which we refer to as Object- Centric CLIP (OC-CLIP), not only boosts CLIP's performance in understanding multi-object compositional scenes but also improves the sample efficiency of the model when trained from scratch.
Our contributions are summarized follows:
We introduce OC-CLIP, a pretraining method which endows CLIP-based architectures with inductive biases to address the binding problem.
We evaluate the sample efficiency of our approach against methods leveraging hard negative augmentations in a controlled 3D environment and show the overall efficiency of OC-CLIP compared to both text and image based a hard-negative augmentations.
We demonstrate that OC-CLIP significantly enhances the binding of object-centric attributes and spatial relationships across a representative set of challenging real-world compositional image-text matching benchmarks. Notably, we report an increase of 16.5% accuracy in the challenging swap-attribute split of SugarCrepe compared to OpenCLIP [Iharcco et al. 2021 finetuned in-domain, and go from random chance to more than 89% on COCO-spatial and on%
GQA-spatial from the Whatsup benchmark Kamathh et al.) 2023)..
We show the scaling potential of OC-CLIP when trained from scratch noisy data [Changpinyo et al. 2021 Sharma et al. 2018). We report an increase of 12.7 % accuracy in zero-shot ImageNet classification compared OpenCLIP.
2 Related Work
Contrastive Pretraining of VLMs. Vision-language models (VLMs) have made substantial strides in both the vision and multi-modal domains [Bordes et al.] 2024). Modern VLMs are pretrained on vast, diverse and oftentimes noisy multi-modal datasets [Changpinyo et al. 2021 Schuhmann et al. 2022 Ilharco et al. 2021 Zeng et al. [2022), and have shown substantial improvements when applied various zero-shot tasks. CLIP [Radford et al. 2021 presented a contrastive learning approach used for pretraining, which involves training the model to differentiate between similar and dissimilar image-text pairs. This approach encourages the model to learn a shared representation space for images and text, where semantically similar pairs are close together and dissimilar pairs are far apart. Following CLIP's lead, image-text contrastive learning has become a prevalent strategy for VLM
pretraining [iu et al., 2023 Cai et al. 2024 Liu et al. 2024a Dai et al. 2023 Zhai et al. 2022b Chen et al., 2022 Beyer et al. 2024 Fini et al. 2023]. Contrastive vision- pretraining spans numerous downstream applications, including zero-shot image classification (Zhai et al., 2022a Radford et al., 2021 Metzen et al., 2024 Gao et a [202]] text-to-image generation [Podell et al.
2023 Abdal et al.] 202 Ramesh et al. 2022 /aharia et al. 2022), as well as assessing text-image alignment Moens et al. 2021| [ho et 2023) In this work, we are particularly interested in the ability of CLIP-based models to evaluate compositional text-image alignment.

---

Compositional Understanding Benchmarks. Several benchmarks have been developed to assess the compositional understanding of VLMs. In this work, we focus on benchmarks structured as cross-modal retrieval tasks where the model needs to distinguish between correct and incorrect text descriptions given an image, and evaluations are based on accuracy metrics. The majority of these benchmarks (Zhao et al. 2022, Yuksekgonul et al.] 2023a Parcalabescu et al. 2022] rely on the rule-based construction of negative captions and the generation of their associated image counter-factuals (Zhang et al., 2024a Awal et al. 2024). Yet, many of these benchmarks may be solved by leveraging the language prior exclusively [Goyal et al. 2017 Lin et al. [024]] hence disregarding the information from the visual input. To address this, benchmarks such as SugarCrepe [Hsich et al., 2023a] leverage large language models to generate plausible and linguistically correct hard negatives, and show that previously introduced text-based hard negative strategies are not always effective (Yuksekgonul et al. 2023b) - e.g., when considering attribute and object swaps between textual descriptions. Other benchmarks focus assessing the VLMMs' spatial understanding |Kamath et al.] [023] Yuksekgonul et al.) [023b] Zhang et al.] 2024a], and propose finetune CL.iP-based models data containing high proportion of spatial relationships since these relationships tend to be under-represented in commonly used pretraining datasets. Interestingly, Kamath al. (0223) show that even when finetuning with in-domain data containing an over-representation of spatial relationships, state-of-the-ar models still exhibit a close to random chance performance. In this work, we test the hypothesis that spatial relationship failures are due to the lack of composition in the similarity score computation used to train CLIP-like models.
Object-centric Binding Inductive Biases. CLIP has been shown Yuksekgonul et al., [023a] to be pushed to learn disentangled, mag-of-words-sti representations from the contrastive loss and the easily distinguishable negatives typically used for pretraining. Although the learned representations might be effective for objects presented in isolation, they struggle with scenes containing multiple objects [Tang et al. 2023]. For example, consider simple scene with a green apple and a yellow banana. In this case, the model must maintain and correctly link the attributes ("green"" "yellow") to the objects ("apple", "banana"), without mixing the concepts - e.g. "yellow apple" or "green banana". This exemplifies the importance of devising robust mechanisms within the CLIP architecture and/or training to accurately handle multiple objects, while preventing feature interferences. In this work. we focus on equipping CLIP with object-centric binding inductive biases and take inspiration from the architectures proposed in the unsupervised object-centric visual representation learning literature [Locatello et al. 2020. Wu et al. 2023 Seitzer et al. 2023 Assouel et al.. 2022 Many recent image-only approaches follow a simple inductive bias introduced by slot attention [Locatello] et al. 2020], where an image encoded as a set of input tokens is soft partitioned into K slots. In particular, attention maps are computed via an inverted cross attention mechanism (Wu et al.). where the softmax is applied along the query dimension in order to induce competition between the slots to explain different groups of input tokens. In this work, we extend these inductive biases to define text-conditioned visual slots from the input image.
3 Method
Our goal is to enhance CLIP-based architectures with object-centric binding and composition capabilities. Our method starts by extracting representations of distinct open-ended objects and relationships in textual description, as well as representations of patches in an image. Next, a binding module matches the text representation of objects to the relevant image patches, producing a slot-centric representation of the image. Finally, a structured similarity score compares the slot-centric representation with the textual representations of different objects, and leverages the extracted relationships as constraints applied to the visual slots. Our key contributions lie in the design of the binding module" and the proposal of the structured similarity score, which we detail in sections [.1 and 3.2 respectively. Figure presents an overview of the proposed approach. Our approach relies on scene- graph representation of the text modality. We assume the parser given and orthogonal to our approach and discuss the choice of the parsing method in Appendix A.4
Notation. We denote as x image of shape Rxx and x = [x], ...,.... N Fx)(x) Nxd its patch-level encoding, where Es is an image encoder typically a pre-trained ViT [Dosovitskiy ct al. 2020] - N is the number of patches and d the dimensionality of the patch embeddings. We denote as the text description, or caption, associated with x. We extract a scene graph. For example,
"Code for the binding module is given in the Appendix Fig 16
3

---

Scene-graph
R
relation scoring
Eg s sS
dinan on!
Text Encoder
qully
object scoring
"sop sign'
Tintal pole!
S
Eoeniee(N ,s)
Parser
Invorn Coos SAIL
Sar
Structured
similarity score
stop
sein att-attn
sign put
Image Encoder
Binding
relations
upside down
Module
N: nodes
S: visual slots
default slots
Figure 1: Object- Centric CLIP (OC-CLIP) overview. OC-CLIP begins with scene parsing, where we utilize a text parser (e.g,, Llama3-based to extract objects and relations from the input caption. The extracted text objects and relations are then fed into a text encoder, which generates distinct text embeddings for both nodes and relations. In parallel, the corresponding image is processed by an image encoder produce patch-level image embeddings. These image embeddings are then combined with the text entity embeddings and passed through a binding module, which outputs visual token slots embeddings. Both modality are aligned in new space using structured similarity score that matches nodes embeddings to visual slots and models relational constraints between them.
the scene graph of "A red apple to the left of a blue car" will be represented with the set nodes ("red apple", "blue car") and the set of edges {(tto the left of", "red apple", "blue car")). In practice, we represent V as a matrix of node features N, where each row contains the embedding of node in the graph. Moreover, represent each s and of in the relationship tuples indices referencing the nodes (rows) in N.
3.1 Binding Module
Our first contribution resides in the binding module. The idea is that when comparing the content of a caption and an image we do not want the features of different objects to interfere with each other but rather keep them separate at a representational level. The role of the binding module is thus to extract a slot-centric representation of an image where the content of the slots are pushed to represent the nodes of the associated scene graph.
To do so, we implement the binding module using a inverted cross-attention layer [W et al.). where the queries are the nodes from our scene graph and the keys and values are the image patches. We normalize the attention coefficients over the queries' dimension in order introduce competition between queries to explain different parts of the visual input. We follow common practice and set the attention's 's softmax temperature to vD,, with D being the dimensionality of the dot-product operation. Applying the softmax along the queries' dimension pushes all the candidate keys to be softly matched to at least one query. However, captions mostly describe specific parts of the image, and rarely capture all the visual information. Since we want only the relevant visual information to be captured by the queries, we add set of default query tokens, stored in matrix Quefault- which participate in the competitive attention mechanism with the goal of absorbing the visual information not captured in the caption. These default query tokens are dropped in the subsequent computation steps of our model (akin to registers ViT backbones [Darcet et al. 2024)). We find the default query tokens crucial to stabilize the training our model.
The binding module computations are formalized as follows:
Q WGN,
K, V W W.X,
Q' = [Q: Qdefault]:
Q' K
dim='²) v,
Attn(Qu Q' K, V) softmax
VD
S, Saeanlh (t, K, V).
(1)
Here, W,. W., and W, are the linear projection weight matrices for the queries, keys, and values, respectively, are the visual slots, Saefaltt are the visual slots from default query tokens, which are discarded for subsequent steps, and [.] denotes the concatenation operation.

---

Thus, the output of this binding module are the visual slots S. Intuitively, these slots are pushed represent the visual objects, or entities, that correspond to the nodes of the scene graph. Their object-centric learning is driven by the structured similarity that we detail in the next section.
3.2 Structured similarity score
Our second contribution resides in the introduction of a structured similarity score, whose goal is to promote the constraints imposed by the scene graph on the learnable visual slots. Our proposed struc- tured similarity score composed of an object scoring function and relationship scoring function. The object scoring function assesses the presence of each node in the scene graph (objects present in the caption). We model this function as the sum of the cosine similarity between each textual node representation N' and its assigned visual slot S'. The relationship scoring function encourages the relational constraints imposed by each edge in the scene graph and is defined as a learnable function
fa of the relationship embedding r', and the visual slot representations S and S corresponding to the subject and object of the relationship, respectively. We derive the overall structured similarity score over the visual slots from an image and a graph G ([N")==1..a {(r', s o')))......))
such that: S 9) Ei
cosine + 11-++--aa... E.st. folr' s _s'') where and B are learned pa- aM+BBP
rameters controlling the strength of each score. A and P are the number of nodes and relationships in the scene graph G, respectively.
We define f follows:
(r, S") cosine (r, fer, + fo((r,
where I.] denotes the concatenation of two vectors and f. and fo are MLPs that reduce the dimen- sionality of their inputs. Note that we model the relationship scoring function so that it keeps the same scale the object scoring function and can take the order of the relationship into account.
3.3 Training
The model trained using the following loss Lite + Lret where Lite is the image-text contrastive loss defined to minimize the distance between image and scene graph representations from paired text-image data while maximizing the distance between image and scene graph representations from unpaired text-image data as:
exp E/= 2 (x
+ log exp S(x,
Lite =
Σ log im1
2j=1 exps(x.,9))
where B is the number of elements in the batch. Note that the S is the structured similarity score defined in Eq. 3.2. Lrel is the loss that pushes the model learn a non-symmetric relationship scores:
B
((x.
Crel Σ log exp Sxx..G)) + exp (x). + ex..5(x..0))
where & and 9 are altered scene graphs In G, swap the order of the subject and the object of relationship, whereas in G, we randomly chose the relationship's 's subject and object from the nodes in the scene graph. We ablate the main components of OC-CLIP in Appendix A.2
4 Results
We evaluate OC-CLIP's inductive biases in different settings:
Addressing CLIP's binding problem. We show the efficiency of OC-CLIP in addressing the binding problem compared to finegrained hard-negative based augmentation on a synthetic Cataaet.(Secttonn 4.1
Compositional understanding. We showcase OC-CLIP's compositional understanding. in domain, on real-world object-centric attribute binding and spatial relationship understanding benchmarks (Section 4.2)
5

---

Scaling on noisy data. We show that OC-C consistently outperforms a CLIP-based model in both zero-shot single object classification and zero-shot compositional understanding multi- objeit text retrieval, when training both models fully from scratch on larger-scale and noisy dataset (Section
4.1 Addressing CLIP's bag-of-words behavior
In this section, aim to assess the efficiency and effectiveness of leveragingffinegraineed hard- negatives (swap attributes negatives) OC-CLIP and CLIP-like models in addressing the binding problem. To do enumerate all possible object-attribute conjunctions and systematically evaluate the potential of CLIP-like models and OC-CLIP in addressing simple swap-attribute retrieval tasks under varying hard-negative sample sizes.
synthetic dataset with closed-set vocabulary, from which we can
Dataset. We consider controlled 3D environment based PUG [Bordes et al. 2023 and build a dataset composed of a single textured animal, or pairs of animals, in different backgrounds. We use combination of 4 textures, 20 animal classes, and 5 different backgrounds e.g., see example in Figure 2a We follow prior benchmarks (Hsich et al., [2023a] and perform a text-retrieval task between the correct caption and the associated negative caption. We give additional details about the subsets compositions Appendix
Baseline and OC-CLIP training. We train models on data splits from our synthetic data, while considering an increasing proportion of hard-negative samples. We consider a CLIP model initialized with OpenCLIP weights Ilbarco et al. [2021]. We also initialize OC-CLIP's text and vision backbones with OpenCLIP weights, but train OC-CLIP's binding module from scratch.
Results. Our results, presented in Figures 2b and 2c show that simply adding more hard-negatives to OpenCLIP's training plateaus and not sample-efficiene as the swap-attribute binding performance always underperforms OC-CLIP trained on less data without any targeted bard-negatives in simple object-atrribut binding task. On seen object pairs, with 70% of the possible pairs and 70% of their corresponding wwap-attribute hard Sard-negatives CLIP plateaus at 81 % compared to OC-CLIP which solves the task at 97% on the same training data size and no swap attributes hard-negatives. We hypothesize that the root cause of this issue lies in the representation format used in CLIP's original formulation, which relies on a single vector capture complex semantic relationships. Our proposed method introduces inductive biases that allow the model to learn more structured representations, avoiding superposition of features Greff et al. 2020b and effectively mitigating the bag-of-words behavior.
OpentLPP9:300
Opencurs
Model: Wratebo
OpentLIVV:70
70
85
93
Sppr
86
73
15
9us-
53
Elephant Fish White Elephant and Red Fish
Ouz
Gurs
of Hord Negatives
of Hard Nagetivaa
(a) Synthetic data example
(b) Scen object pairs
Unseen object pairs
Figure 2: Efficiency and effectiveness of OC-CLIP: Analysis on synthetic data. Performance of the finetuned OpenCLII and OC-CLIP models on a binary classification task between a caption and its corresponding hard-negative given a synthetic image, as shown in (a). Performance is shown as a function of the percentage of animal pairs (y-axis) seen during training and the proportion of hard- negatives used in the training data (x-axis). Results shown for (a) seen and (b) unseen object pairs.
4.2 Compositional Understanding
In this section, we verify that the observations made in the controlled environment presented in Section 4.1 also transfer to real-word datasets, thereby assessing the real-world compositional understanding of OC-CLIP.
Datasets. We train OC-CLIP and finetune OpenCIII in-domain on set of datasets relevant
for real-world compositional understanding. The training text descriptions representing positive samples are taken from COCO [Lin et al. 2014), Visual-Genome (VG) [Krishna et al. 2017

---

and GQA [Hudson and Manning 2019). The latter annotates images coming from Visual Genome [Krishna et al. [017] with objects and both spatial and non-spatial relationships, and thus contains a high representation of spatial prepositions. We evaluate the different models on the most challenging benchmarks representative of compositional understanding. ensuring that we validate both their attribute binding and spatial relationship understanding capabilities. In particular, we use SugarCrepe [Hsich et al., 2023b] and ARO-A (Yuksekgonul et al. [023a] for attribute binding and ARO-Relation (ARO-R) Yuksekkonuul et al. 2023a]. COCO-spatial and GQA-spatial (Kamath ct al. 2023] for spatial relationship understanding. Although Hsich et al. [202bb] showed that other benchmarks such as VL-Checklist [Zhao et al. 2023 COCO-Order and Flickr-Order splits of
ARO [Yuksekgonul et al., 2023a| were easily hackable because the negatives are not semantically
conrect, we include the results on those benchmarks for reference in Appendix (A.5 [A.5]
Training. As in section 4.1 we initialize the text and vision backbones of OC-CLIP with pre-trained model weights, and train the binding module of OC-CLIP from scratch. In particular, we initialize the text backbone with OpenCLIP weights [lhharco et al.) 202T] and consider two different vision backbones, OpenCLIP (ViT-B-16) [Ilharco et al., [02 I] and V2 (ViT-B-14) [Oquab et al. 22224). to show the flexibility of our binding module and learned structured similarity score. We noticed that taking the patches from earlier layers in OpenCLIP helps the training and ablate it in Appendix A.2 We use a batch size of 128 and a learning rate of 2 100 to train OC-CLIP for 100 epochs. We use a batch size of 256 following previous finetuming approaches (Kamath et al. 2023 Yuksekgonul et al., (023b) - and a learning rate of 4 10 6 for 20 epochs to finetune the OpenCLIP bascline. We run all the models for 3 seeds and report the mean performance along with their standard deviation. Note that since OC-CLIP's binding module is trained from scratch, OC-CLIP's learned aasion-language-liignan space does not rely on the rision-languae backbone (in fact, we drop the CLS token). Therefore, the new representation space learned by OC-CLIP can only be expected to generalize within the vocabulary it has been trained on. F
alignment captured by the CLS token of OpenCLIPP's
Baselines. We report the performance of a representative set of strong baselines which we separate in two groups: the first group of baselines are models trained contrastively and finetuned in-domain and recaptioning- -aaedd methods,
(on COCO/VG) and the second group are hard sarrdneeative--bat further divided into small scale and large scale. For the first group, include OpenCLIP referredd to as OpenCLIP-FT -, BLIP [Li et al. 2023a], and XVLM [Zeng et al., 2022]. BLIP is augmented with an image-text matching loss and XVLM uses bounding boxes assist the object-centric binding. Note that these two baselines are also equipped with a language modeling objective which may help identify unplausible captions. For the second group, we select methods that augment the dataset with rule-based text hard-negatives (NegCLIP (Yuksekgonul et al. 2023b)) ) segguugge-mddll-baseee hard-negatives (CE-CLIP et al. [2020) and CLIP-SVLC (Doveh et al.) 2023b)). D. and image-&-langune Hannguage-mode--aare also include dense recaptioning baselines such as DAC (Doveh al. 2023a] for reference.
hard-negatives (CLIP-CC (Zhang et al. 2024a]). We
Attribute Binding Results. We evaluate the attribute binding capabilities of OC-CLIP and baselines on SugarCrepe [Hsic et al. 2023b) and ARO-A YYuksekgonul et al. 20236] benchmarks. We report the results in Table 1 When comparing OpenCL.IPrr to OC-CLIP (ours - both models), we observe notable performance boosts on the hard SugarCrepe's swap-attribute, and swap-object. In particular, OC-CLIPP.14 achieves improvements of + +16.5% on the hard swap attribute split, +20.4% on the swap object split, and a smaller +4.15 % on the replacement relationship split. Moreover, both OC-CLIP models perform similarly to OpenCLIPFr on the remaining SugarCrepe splits. This is to be expected since the remaining splits do not require precise binding distinguish between positive and negative representation. We additionally compare OC-CLIP to finetuned versions of CLIP that rely on in domain hard-negatives (NegCLIP, CE-CLIP, CC-CLIP, MosaiCLIP) and with dense recaptioning (DAC-LLM and DAC-SAM), In particular DAC finetunes OpenCLIP with ~ 3M VLM -Le--eenettted dense captions (along with their corresponding hard negatives) that significantly increase the vocabulary coverage compared to methods that only finetune in domain (e.g., on COCO). Interestingly, OC-CLIP still outperforms them on both swap-attribute and swap-object, showing improvements of +13.6% and +8.4% over the second best performing method, respectively. Those results confirm the behavior that we observed
captions and may therefore be solved with a
'Important note: Although OC- CCIC is compared in domain against post-training baselines of CLIP, our approach is pretraining strategy and can only be compared in domain. We discuss this point further in the Appendix(A.]
7

---

in Section 4.1 and the inefficiency of hard-negative methods in solving the binding problem of CLIP-like models, even at the scale of DAC finetuning.
ADD
ARO
MODEL
ORJECT
ATTRIBUTE
OBJECT
ATTRIBUTE
OBJECT
ATTRIBUTE
RELATION
ATTRIBUTION
Zero-shot
OPEN CLIP
68.2
66.2
82.7
80.3
93.8
82.8
67.3
63.2
do-domiin At baxelines
BLIP
66.2
76.2
96.5
81.9
68.35
88.0
XVLM
64.9
73.9
95.2
87.7
77.4
86.8
63.1
72.4
93.4
83.1
95.4
87.0
75.5
60.0
Hord-Negative small scale
NEGCLIP
75.2
75.4
88.8
82.8
92.7
85.9
76.5
71
CF-CLIP
72.8
77
92.4
93.4
93.1
88.8
79
76.4
CC-CLIP
68.6
73.6
86.7
90.3
95.9
87.9
76.2
CLIP-SVLC
73.0
MOSAICLIP
78.0
Hord-NegativeDDnss Captioning large scale
DAC-LLM
75.1
74.1
89.7
97.7
89.3
84.4
73.9
DAC-SAM
71.8
75.3
87.5
95.5
91.2
83.9
83.9
70.5
Ours OC-CLIP a:st
76.5
85.6
86.5
85.7
91.3
88.6
75.4
82.5
OC-CLIP 8-16
76.6 #0.0
87.5 =0.5
91.1+0.4
83.8 +1.0
94.6
=0.4
87.9 +0.1
76.0
=0.4
83.2+0.8
OC-CLIP 2.14
83.5
88.9
92.8
84.8
=0.4
89.2
79.6
84.0+0.
Table 1: Attribute binding: Performance SugarCrepe and ARO-A. Both OpenCLIP-FT and OC-CLIP are initialized with the same OpenCLIP checkpoints. OC-CLIP is trained with two ViT base backbones with different resolutions: OpenCLIP's backbone (B-16) and Dinov2 (B-14).OC-CLA) bidning module is always trained from scratch.
Relationship Understanding Results. We evaluate the spatial relationship understanding capabilities of OC-CLIP and baselines on COCO-spatial, GQA-spatial, and ARO-Relation (ARO-R). Note that IRO-Relation contains both spatial and non-spatial relations but about half of the test examples consists of left/right relationships understanding. We report the results in Table improvements of both OC-CLIP models over the baseline models and across the 3 datasets. In particular, the best OC-CLIP model outperforms OpenCLIP-FT by +44.1 1% on COCO-spatial, 443.tial,+43.69) +43.6% on GQA-spatial, and +34.8% on ARO-R. When compared to contrastive VLMs finetuned with in-domain data (XVLM, BLIP), OC-CLIP models exhibit superior performance, with improvements between + +10% and +27% over the strongest contrastive finetuned VLM. Finally, when compared to baselines leveraging hard-negatives (NegCLIP), OC-CLIP remains the highest performer. Additional results on the ARO benchmark are reported in Table 9 of the Appendix.
show consistent
4.3 Training OC-CLIP from scratch
MODEL
COCO ACCO-PPATIAL
GQA SPATIAL
ARO-R
XVLM
73.6
67
73.4
In this section, we aim to assess the potential of OC-CLIP when trained fully from scratch from scene-graphs obtained from large scale non-human-curated captioning dataset.
BLIP
56.4
$2.6
NEGCLIP
46.4
46.7
80.2
OPENCLIPeT
45.6 #0.2
50.1
OC- OC-CLIP (4-0)
86.3
90.0
84.3
OC-CLIP
Datasets. We train both ViT-B-16 Open- CLIP model and OC-CLIP fully from scratch on increasingly large dataset sizes using CC3M (Sharma et al.) 2018), CC12M (Chang- pinyo et al. 2021] and the combination of both datasets. We evaluate all models
Table 2: Spatial relationship understanding: Performance on COCO-spatial, GQA-spatial from the Whats'up Benchmark and ARO-R. We finetune both OpenCLIP (OpenCLIPFT here) and OC-CLIP in-domain on COCO, Visual Genome, and GQA data. Both models are initialized with the same OpenCLIP checkpoints.
ImageNet (Deng et al., 22099 zero-shot classi- fication in this section, and report results the ELEVATER suite [Li et al. 2022) in Appendix (Table 6). We also evaluate zero-shot compositional understanding of the models on the challenging sbje-t and attt-btte splits of SugarCrepe, and on Winoground (Thrush et al. (002).
Baseline and OC-CLIP training. Both CLIP and OC-CLIP architectures are trained fully from scratch for 15, or 25 epochs, using a batch size of 4096, a learning rate 10-3 2k steps of learning rate warm-up, and cosine decay. As recommended by Mu et al. 2021 AdamW

---

optimizer with 0.5 of weight decay and B3 set to 0.98. We use a ViT-B-16 16 backbone for both models. Since OC-CLIP's text bakcbone only needs encode single objects and relationships, we use smaller text bakchone with a context length of 20 and only 6 layers instead of 12 Note that we do not tune the hyper-parameters in this experiment. We further discuss those design choices in Appendix A.3
Results. We start by verifying the sample efficiency of OC-CLIP using ImageNet [Dngg al.] 2009] zero-shot
Parged154)
classification performance
in Figure 3a We show that OC-CLIP shows better sample-efiicinny than the baseline trained on the same data, while using a smaller text backbone. We then evaluate OC-CLIP on zero-shot classifica- tion and compositional understanding in Figure (bb] Interestingly. OC-CL.I shows performance gains in general zero-shot classifica- tion (+12.8% on ImageNet, when trained from scratch on CC3M+CC12M) while also showcasing substantial improvements in zero-shot compositional understanding. For example, OC-CLIP exhibits a notable +12.7% and +6.6% in SugarCrepe's ' swap- aatributriut and swap-object splits, respectively. This experiment shows that the structured training of OC-CLIP is also effective when scaling to noisy image-caption dataset and, therefore. does not solely rely on high-quality human captions. We additionally report extensive zero-shot downstream classification performance on the ELEVATER (Li al. [022] suite and discuss the computation trade-off of our approach in Appendix A.3 We leave further scaling for future work.
Imagetec
3
Vodal
Dc CUP
QJP
Lom
15m
10
15
20
25
Eoochs
Seap
(a) Sample efficiency
(b) Zero-shot accuracy
Figure 3: Scaling the training noisy data. CLIP and OC-CLIP are trained from scratch on varying sizes data (3M, 12M and 15M) for varying number of epochs. OC-CLIP shows (b) better zero-shot compositional understanding performance on SugarCrepe's swap-attribute and swap-object, and on Winoground (I = Image score and T = Text score), as well as (a) better sample efficiency shown on zero-shot ImageNet classification.
5 Conclusion and limitations
Conclusion. this paper, we proposed Object-Centric CLIP (OC-CLIP), a pretraining method to en- hance the compositional scene understanding of CLIP-like models by leveraging advances from object- centric representation learning. Our approach adapts the slot-centric representation paradigm CLIP and dynamically aligns each representational slot with the objects mentioned in the text description. This is achieved by the introduction of binding module and structured similarity score that allows to train OC-CLIP in contrastive way. We evaluated our approach against finegrained hard-nee negative augmentation strategies and demonstrated that OC-CLIP significantly enhances the binding of object- centric attributes and spatial relationships across representative set of challenging real-world com- positional image-text matching wrechmarks.Finnlly.. we show the scaling potential of OC-CLIP to be trained from scratch on noisy dataset [Changpinyo et al. 202 Sharma et al. 2018). Notably, re- port performance gains in zero-shot classification (+12.8) % in ImageNet while maintaining substan- tial improvements in zero-shot SugarCrepe swap-attribute (+12.7%) and swap-object (+6.6%) splits.
Limitations and Future Work. Our OC-CLIP model has several limitations and avenues for future work. Notably, our approach relies parser to extract object-centric attributes and spatial relationships from text descriptions. While we have chosen an L.LM-base parser, which discussed in Appendix A.4 studying the different biases of L-M--bsed parser families could be interesting. Additionally, while we show the scaling potential of OC-CLIP 15M scale (A.1 A.3), the model needs further scaling to be fully comparable to all the CLIP variants, trained at least at 400M scale.

---

References
Rameen Abdal, Peihao Zhu, John Femiani, Niloy J. Mitra, and Peter Wonka. Clip2stylogan: Unsuper- vised extraction of stylegan edit directions, 2021. URL htttp:::/aaxi...rg////o//// wtgp/b//////////// 2212. 21111052199111111111111111
Rim Assouel, Lluis Castrejon, Aaron Courville, Nicolas Ballas, and Yoshua Bengio. VIM: Variational independent modules for video prediction. In Bernhard Schölkopf, Caroline Uhler, and Kun Zhang, editors, Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pages 70-89. PMLR, 11-13 Apr 2022. URL
https://prroceeddnngs mlr pttpss//vl/aassseeeeee
html
Rabiul Awal, Saba Ahmadi, Le Zhang, and Aishwary Agrawal. Vismin: Visual minimal-change understanding, 2024. URL htttp:://aaxiiii wwwp:/wwww..rgg//////..../
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver. Pinelopi Papalampidi. Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai Paligemma: A versatile 3b vlm for transfer, 2024. URL pattps://arie org/ abs abs/2407.m 077247077776666
Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vineent, and Ari Morcos. Pug: Photorealistic and semantically controllable synthetic data for representation learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36. pages 45020-45054 Curran As- sociates, Inc., 2023. URL UtL//tttps:/pprrccediigg http://pccceciiig....urrro...cpp #11-8885220000044444777666777--3
www.s//pppeeeeeeeeeaeeaaaa /023/ 2023/ and and_Bennhmarkss.... pdf
Florian Bordes, Richard Yuanzhe Pang, Amurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk. Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. An introduction to vision-language modeling. arXiv preprint arXiv:2405. 7247,, 2024.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Making large multimodal models understand arbitrary visual prompts. In CVPR 2024, 2024.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre- pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0677 2022.
Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. Fine-grained image captioning with clip reward, 2023. URL httt:httts///aaat //arxvvv//axvv....// .rg. wrg..bs/////./.....o 2205
13115
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. h/tptttts::///////// org/ /205 06500
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2024. UtR.htttss://arxiv.orrgg.
16588
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint 2rXiv:2010.11929, 2020.
10

---

Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonie Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, al. Dense and aligned captions (dac) promote compositional reasoning in vl models. Advances in Neural Information Processing Systems, 36,713777,6,,,,,, 2023a.
Sivan Doveh, Assaf Arbelle, Sivan Harary, Rameswar Panda, Roci Herzig, Eli Schwartz, Donghyun Kim, Raja Giryes, Rogerio Feris, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision&language concepts to vision&languuae models, 2023b. (ttpssss:://axxt 2211.11733
abs/
Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 28940-28954, Curran Associates, neurips.
Inc., 2022. URL
wwt.fffffleeee///eeeeee paapee//ppaar/22///1leel pdf
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models, 2016. https:/t::///rxxitt haattp::/arr.v..or//aa//////// 08575
Enrico Fini, Pietro Astolfi, Adriana Romero-Soriano, Jakob Verbeck, and Michal Drozdzal. Improved baselines for vision-language pre-training. Transactions on Machine Learning Research, 2023,
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters, 2021. URL https://arxiv. org.iiii.orr/aaaoe abwp.//11..g/////1..o... 04544
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904- 6913, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation rettssiohptttrrreteetttttoo learning with itcrative variational inference, 2020a. URL hRtpps:/://aaa wwwpaawwwww...g//////////.0
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks, 2020b. https:// wttpe/ii.org/abs//////ea 05208
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom
embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Cheng-Yr Yu Hsich, Jicyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. In NeurlPS 2023, 2023a.
Cheng- Yu Hsich, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality, 2023b. URL https://arxiv org/ atd//b0b/330..... 2306. 14610
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL https:://oo:: http::///601.0rg/1055522////// www..o/..............o 5143773
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzzepfel, and Diane Larlus. Hard negative mixing for contrastive learning, 2020. https:ss: wwwpaaawwvvv..iaaaaaaaa wwg/abs/////eeeoooee/..ooeeaeeee ttabs/2010..m 01028
11

---

Amita Kamath, Jack Hessel, and Kai- Wei Chang. What's "up" with vision-language models? investigating their struggle with spatial reasoning, 2023. URL /ttps///s/ar /ttgg////////o000 19785
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
Tiep Le. Vasudev Lal, and Phillip Howard. Coco-counterfactuall:: Automatically constructed counterfactual examples for image-text pairs. In NeurIPS 2023, 2023.
Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang. Jyoti Aneja, Jianwei Yang. Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng Gao. Elevater: A benchmark and toolkit for evaluating language-augmented visual models, 2022. URL https://arxiv atrg/bbs/204
08790
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-ima -inuag pre- training with frozen image encoders and large language models. arXiv preprint arX1v:2301.12597. 2023a.
Zhuang Li, Yuyang Chai, Terry Yue Zhuo, Lizhen Qu, Gholamreza Haffari, Fei Li, Donghong Ji, and Quan Hung Tran. Factual: A benchmark for faithful and consistent textual scene graph parsing, 2023b. URL httphhtps:////t /topps://rrdtee http///rxvvvv.. org/ abs/ 17497
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer, 2014. doi: 10.1007/ 188-3-119-10602- 18. URL https//p:///do wtt//oo//////. 1007 319- 10602- 1_48
Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models. arXiv preprint arXiv:2306.01879, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS 2023, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://
1lava-vl.gite laa.aarl.iittuub.o//bllg... /ttp.////oooo000000000000------0
Qinying Liu, Wei Wu, Kecheng Zheng. Zhan Tong, Jiawei Liu, Yu Liu, Wei Chen, Zilei Wang, and Yujun Shen. Tagalign: Improving iision-llnguage alignment with multi-tag classification, 2024b. URL
14149
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020. htttp:////aa org/ abs. 15055
Jan Hendrik Metzen, Piyapat Saranrittichai, and Chaithanya Kumar Mummadi. Autoclip: Auto- tuning zero classifiers for vision-language models, 2024. URL http:::/s// org. abs/ 2309.1644
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors. CLIPScore: A Reference-free Evaluation Metric for Image Captioning, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 1/255 emnlp-main.595. URL
abstract visual reasoning. 2024. URL wrgp/yy1..rg/..0..... http:/22000...mmmmmma........
Shanka Subhra Mondal, Jonathan D. Cohen, and Taylor W. Webb. Slot abstractors: Toward scalable 03458
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training, 2021. https:/s:///ar wrg/ abaaabbs///11 22t//2122.111 12750
12

---

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Wei Wen Li, Ishan Mísra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. URL htt.hhtts/// http::://aa......g//aaaaaaa///ae /ttp///awww..rgg//////.....
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In ICCV 2023, 2023.
Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, lacer Calixto, and Albert Gatt. Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), page 8253-8280. Association for Computational Linguistics, (16653///20222.allooogggg
2022. doi: acl acl-long.550 567
URL http: http://px.ooioo/////////o/////// 10. 18663//1/22/22
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. 2023. URL htl.:/tppp///axxx..o//////////
01952
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 11477, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022. URL
httppp/:/axxxv.r/g//// wwwrriivvor//aaee abs 2210 08402
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann SimonnGabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning, 2023. URL htttps://arx: org/abs/2209. 14860
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2556-2565 Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v 1/188-138. URL wwtp//1-11////33001
Yingtian Tang, Yutaro Yamada, Yoyo Minzhi Zhang, and IIker Yildirim. When are lemons purple? the concept association bias of vision- vision- language models. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023. 5SCLPiG1vE
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Can- dace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2022. 2022.
13

---

Taylor Webb, Shanka Subhra Mondal, and Jonathan D Cohen. Systematic visual reasoning through object-centric relational abstraction. In A. Oh, T. Naumann. A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 72030-72043. Curran Associates, Inc., 2023. URL https://prooceedings wwwrreinnnnnnnnn............r..e wwtpsssr.cppppppppeepeep samper_ffllst www.fillesppaaeee pttps/ppppr/////////// pdf
Yi-Fu Wu, Klaus Greff, Google Deepmind, Gamaleldin F. Elsayed, Michael C. Mozer, Thomas Kipf, and Sjoerd van Steenkiste. Inverted-attentioon transformmrssssss can learn object representations: Insights from slot attention. UtL.ihttpes/pppp..eeaaaaaccooooo
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffuusio: Object-centric generative modeling with diffusion models, 2023. URL ftttps://axxiv.org/att
11281
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision, 2022. URL https
7/axxiv.or/ wtwp://g/ab////o.aa 11094
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why lision-language models behave like bags-of-words, and what to do about it?, 2023a. URL
https://arxiv org/ /tt.s///2...c 2210001990000066
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference Learning Representations, 2023b. URL httpppss///ppeeeeeeee...eee net/ffoummidd KRLUvxhBuaX
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts, 2022. URL htttps///aaxiii/
abs//11s..1111/1//////// 08276
Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pages 18123-18133, 2022a.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning, 2022b. URL https
//axxiv. org/ wwt.aaaaaaa. 07991
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training 2023. URL httltst/////::ti..co htgp://aaxivv....aaaa wttp:///.gga/bb////////e// 2233/ 15343
Jianrui Zhang, Mu Cai, Tengyang Xie, and Yong Jae Lee. Countercurate: Enhancing physical
and semantic visio-linguistic compositional reasoning via counterfactual examples, 2024a. URL
https://arxiv.or abs//42...22222 13254
Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross- modal hard negatives to enhance lisiuliinciisii compositional understanding, 2024b. URL
https::/arrie org/ awtpa//....... 08832
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.002211 2022.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vi-checklist: Evaluating pre-trained vision languaae models with objects, attributes and relations, 2023. URL https: org/ atw.//00 200220222222222 00221
Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Iterated learning improves compositionality in large laniune models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13785-13795, 2024.
14

---

A
Appendix
Impact Statement. This paper presents work whose goal to advance the field of machine learning. specifically the multi-modal architecture of CLIP. CLIP itself has broad applications and impact thus our proposed technique shall be considered when such models are used.
A.1 OC-CLIP as Pretraining alternative to CLIP
Our method is not intended be post-training method -but rather pre-training strategy showcasing the benefits object-centric inductive biases in the context of contrastive pretraining. When using binding module in the in-domain setting (Section 4.2), do not expect maintain CLIP's zero-shot performance. As our binding and relationship modules are trained from scratch, they do not rely the previously aligned CLS tokens (in fact, drop the vision CLS token completely). As a result, our model can only be expected to work well within the vocabulary domain it is exposed to when tested for its zero-shot performance.
Given that we only inherit the vision tokens from the CLIP vision backbone, we believe that it's still worthwhile to check whether training is detrimental to the inherited vision tokens. We probe this by computing linear evaluations of the pre-trained CLIP-based vision backbone vs the vision backbone after our OC-CLIP training strategy on COCO/VG. To do patch-tokens that serve as input to the binding module and train a linear probe on commonly used image classification datasets. We report those results in Table 4 and observe that our training strategy did not significantly impact the generalizability of the backbone despite only training on COCO/VG. In order further motivate our approach additional baselines trained from scratch on CC3M/CC12M specifically designed to address the lack of compositional generalization in CLIP models. Specifically, compare NegCLIP (Yuksekgonul et al. 2023b], which augments CLIP training with rule-based hard negatives and IL- IL-CLIP (Zheng al. 2024). novel iterated learning method for contrastive learning, which improves compositional understanding, Our results in Table show the effectiveness of our proposed inductive biases as an effective alternative to CLIP training.
average-pool the
good pretraining alternative, we also compare to
Training Data
Model
SC-add
SC-replace
SC-swap
Wino (T21)
Imagenet
OpenCLIP-3M
61.9
64.3
52.9
8.1
13.7
NegCLIP
59.3
59.2
60.1
11.8
11.8
CC3M
IL-CLIP
66.1
67.0
54.5
13.3
14.2
OC-CLIP
74.6
73.0
64.7
12
19.6
OpenCLIP-12M
67.5
70.0
60.2
7.2
31.4
NegCLIP
65.0
70.2
67.2
7.3
28.9
CC12M
IL-CLIP
73.8
73.0
62.9
10.1
32.8
OC-CLIP
81.7
79.8
72.9
13.0
42.4
Table 3: Training from scratch CC3M/CC12M comparison of OC-CLIP with OpenCLIP,
IL-CLIP, and NegCLIP. Comparison is done on SugarCrepe (Hsieh et al., [003b) swap splits,
Winoground (Thrush al. 2022
T21 score and Imagenet (Deng al. | 2009
UCF101 Frames
Food-IP Food-101 101
CIFAR-10 10
CIFAR-1O1 100
FER-201 2013
EuroSAT
RESISC45
Country211
ImageNet
Aireraft
STL-10
GTSRB
Caltech
MNIST
Cars
DTDD
Pets
OpenCLIP87..1 87.1 95.8 83.3 30.1 80.3 82.8
65.6
95.9 93.9 88.3 20.1 79.5 91.2 99,2 73.I
OC-CLIP 87.0 96.0 83.7 64.0
30.0 65.6
65.6
96.9
96.0
88.0
20.0
91.1 99.3 73.2
Table 4: Linear average pooling of CLIP vs OC-CLIP. CLIP is pretrained on Laion400M and OC-
CLIP is trained COCO/VG.
15

---

A.2 Ablations
In Table we ablate the key design choices of our model. Specifically, we investigate two key components of the model: the use of competitive (inverted) cross-attention and the local graph contrastive loss. On the one hand, results show that removing the competitive rros--atenntion mechanism greatly affects fine-grained attribute binding (decreasing from 89.0 to 85.9). On the other hand, removing the local graph contrastive loss significantly impacts downstream relational understanding, with accuracy decreasing from 80.5 to 72.8. Adding attention layers helps relational understanding (boosting performance from 77.6 to 80.5), while adding more default tokens does not necessarily help with attribute binding. These findings highlight the importance of the main design choices behind OC-CLIP.
Loc Loss
COMP. X-ATT
ATTN LAY
DEFAULT
REI.
ATT
80.5
89.0
79.2
87.6
72.8
87.7
78.3
85.9
Table 5: Ablation of OC-CLIP's main components. Fine-grained accuracy on attribute binding and relational splits of SugarCrepe
We then further discuss some important design choice of OC-CLIP. Notably
The similarity score coefficients a and 3 that control the weight of the objects and relations in the global graph-image similarity score.
Binding module inductive biases and their impact on compositional understanding perfor-
mance.
Local Loss impact on downstream compositional understanding of relationships.
Layer selection with OpenCLIP backbone.
Similarity Score OC-CLIP's structured global similarity score is combination of the object and relationship components respectively weighted by two learnt parameters and balancing the different contributions. We let the model learn those parameters throughout the training. However, during preliminary experiments we tested different combinations of initial coefficient within the [1.5, 1, 0.5, 0.1] grid and noticed that the model was always converging to a ş ~ 3 without any difference in the downstream compositional performance. We thus fix the initial coefficients to 1.5 and / 0.5 and treat them as parameters.
Default Token and Competitive Cross Attention In the binding module we propose to use an inductive biases to encourage the query tokens to attend to different groups of patches. In order to do so we use a competitive attention mechanism, the so called inverted cross attention common to many object-centric image encoder architecture [Locatello et al. [020 Wu et al.]. We found that the use of inverted cross attention impacts slightly the fine-grained attribute binning performance (see ATT performance in Table -Comp Att model does not any inverted cross attention and is rather implemented with a regular cross attention mechanism, the softmax being done along the keys dimensions.). The finegrained attribute understanding (ATT) is affected by the absence of competitive attention between query slots going from 89.0% to 85.9% accuracy.
Local Graph Contrastive Loss In designing the structured similarity score of OC-CLIP the relational component is formulated as the following cosine similarity fo S* cosine(r, ((r, + (rr S In theory both f[r I*)) and fo([r, S*]) can collapse to ig- nore the subject object visual representation. In order to prevent such collapse we propose add a local graph contrastive loss that shares similarity with hard-negative based learning. We enforce the model model with a higher similarity the graph composed of the same nodes but with either swapped object and subject indices or shuffle objects and subjects indices within the local graph. In both of those cases the relation component the structured similarity score becomes (for a single relation graph)
swap G: cosine fa ((r, S*1) + (rr
(5)
16

---

Figure 4: VIT features layer ablation.
swap G: ff ((r
(6)
shuffle (
fo([r, (()
This prevents the model from collapsing because ground-truth G' is distinguishable from G and G only the visual representations are not ignored in the relationships components. As shown in Table S removing the local loss effectively impacts downstream relational understanding SugarCrepe with a REL accuracy decreasing from 80.5 to 72.8 hence showing the effectiveness the local graph contrastive loss.
Scoring dimensionality Our structured similarity score allows the text encoder focus on encoding information about individual objects and their relationships, rather than the entire scene configuration. To achieve this, we experimented with different dimensionality for both the object scoring bottleneck and the relationship scoring bottleneck. Specifically, each of these scores is designed as cosine distance between text representation and a visual component (as described in Section 3.2), with each operating at bottleneck dimension of darj and dret- In contrast, OpenCLIP represents both the scene caption and the visual representation at a shared dimension of = 512. We expect that our it requires less capacity to encode
model can operate effectively single objects and relationships. We present an ablation study of these two dimensions in Figure and notice that our model is quite robust when we operate lower dimensional space (eg. 128). We use this results to scale our experiments and train the model from scratch with a smaller text encoder explained in the next section A.3 corresponding experiments shown in Section 4.3
much lower dimensionality,
Layer Selection Previous work focusing on dense segmentation tasks Xu et al. 2022 show that taking features from earlier layer in CLIP's VIT help with fine fine-grained tasks. Here we ablate OC- CLIP's version using the OpeNCLIP pretrained backbone when inserting the binding module after layer L & for k E [0..4]. 0 being the last layer. Results merged Sugarcrepe splits are given in Figure 4 The object related queries seem to decrease as a function where as the replace rel split increasing with k. To that end, actually insert our binding module after layer L 2, where is the index the last transformer layer of the VIT.
Cloject Score dim
120
0.9
0.88
Object dim
522 235 128
0.79
0.81
0.83
378
0.96
0.96
0.9E
320
0.88
Q.87
128
0.79
0.77
0.77
Objject Score
Object Score
532 235
0.88
0.9
0.88
0.8
0.82
0.84
532 238
0.96
0.97
0.96
532 238
0.68
0.89
0.89
0.8
0.77
0.78
Object
0.88
0.89
0.8
0.8
0.96
0.95
0.88
0.9
512
78
0.78
128
255
512
128
256
512
129
256
512
128
256
512
128
256
512
Relacion
dim
dim
Relation
Relation Score
(a) Swap All
(b) Swap Obj
(c) Replace Obj
(d) Replace
(e) Replace Rel
Figure 5: Score dimensionality ablations In this ablations we keep the initialization seed fixed and vary the dimensionality of the relation score dilt (x-axis) and object score doty(y-axinn and report the performance on the swap and replace splits of sugarcrepe.
A.3 Experiments CC3M/12M.
In the compositional understanding experiments we compare our approach with data-centric finetuning methods that do not add any additional parameters. These methods are expected to retain some of
17

---

UCF 101 Frames
HaefuMemes
Food: 101
CIFAR-10
CIFAR-IIR
SUNDOT
Airroit
FER
STL-10
EwoSAT
RESISCAS
GTSRB
KITTI
Comnty211
PCAM
CLEVE
MNIST
SST2
ImageNer
CTIP
12.8
44.9 19.9
27.9 1.2
133
9.7
12.
13
76.1
188
13.5
699
24.9
0.6
44.3
17.8
11.1
S1.
7.8
47.4
13.9
(3-)LIP(3m)
15.3 57.1 24,8
38.1
1.2
1.6
133
163
9.9
82.0
16.2
220
30,8
0.7
55.6
22.5
12.5
52.0
11.6
50.1
19,2
CLIP (12m)
42.7 63.2 30.2
42.7
15.5
3.1
14.5
526
13.1
85.7
12.3
28.5
8.
34.9
3.9
54,4
33.9
11.8
514
112
51,9
30.5
OC-CLIP (12m)
54.1 14.5 44,6
51.4
21,
3.7
19.4
664
74
91.9
318
40.6
811
41.
5.9
56
45.7
12.7
49.
9.7 50.2 42.3
CLIP (15m)
43.4
72.3 33.8
44.2
15.8
2.3
140
538
89.1
30.0
28.0
3.3
37.4
12.5
50.6
10.2 50.1
32.2
OC-CLIP 54.5
82.3
46.6
54.1
20.2
3.7
22.1
892
922
94,2
26.8
44.4
9.4
29.9
5.9
52.9
47.3
14.5
51.3
9.0
49,9
45.0
Table 6: Zero-shot evaluation of CLIP vs OC-CLIP. Trained on varying size of data (cc3m, cc12m, merged 15m) for 25 epochs.
the general capabilities of the initial backbone. In contrast, our binding and relationship modules trained from scratch, which means may not generalize as well to unseen data and can only be expected to work well within the vocabulary domain it has been exposed to (eg. COCO/VG/GQA in our experiments setting). However an interesting question would be to asses whether such inductive biases and structured similarity object might have some sclaing potential on noisy and non human curated datasets such as CC12M [Changpinyo et al. 2021]. To answer that question we propose to train both CLIP and OC-CLIP architectures from scratch on combinations of CC3M, CC12M and CC3M+ 12M and compare both of their general understanding and compositional downstream performance. In addition to the zero-shot evaluation, we also provide computational analysis of the binding module to gain insights into its behavior and limitations.
Training Details In order to show the potential of OC-CLIP to learn from scene-graph obtained from a non human-curated captioning dataset we train both ViT-B-16 OpenCI.P model and OC-CLIP from scratch on CC3M [Sharma et al. 2018), CC12M [Changpinyo et al. 2021] and the merge of both (15M). We did not tune the hyperparameters and used the same hyperparameters as suggested in [Mu et al.] (021). Both models are trained for 5, 15, and 25 epochs, using batch size of 4096, learning rate lE - 3, 2k steps learning rate warmup and a cosine decay after. As recommended by Mu et al. [021] we used AdamW optimizer with 0.5 of weight decay and 32 set to 0,98, We report extensive zero-shot downstreeam classification performance on the ELEVATER [Li et al. 2022] suite in Table 6 We did not use any templates and use raw labels instead. Both models were trained using 4x8 V100 GPUS with a local batch size of 128. OC-CLIP shows performance gains in both zero-shot classification (a notable +12.7% in ImageNet) when trained on the same setting. These experiments show that the structured training of OC-CLIP can scale to automatic alt-text captioning dataset. We leave further scaling for future work the main focus of our work is to emphasize the binding problem that arises when using a vector-based representation and a set of inductive biases as a way of operating on a more structured representation (eg. scene graph).
Computational analysis of OC-CLIP In OC-CLIP the visual and text modalities representations no longer independent (as opposed to CLIP). A image representation is the results of some text-conditioned mechanism operated by the binding module. It essentially extracts relevant visual slots that constitutes the nodes of the scene graph coming from the caption. As result, there is some notable computational overhead introduced by the additional cross-attention operations of the binding module. In particular:
1. The text encoder needs to encode the N nodes and R relations of the scene graph as opposed to a single sentence encoding in CLIP.
2. For each Image-Graph pair, The N text nodes cross-attends to Nim patches of the ViT in order to extract the structured visual slots.
When training OC-CLIP from scratch we propose to mitigate those two overheads respectively by:
1. Using a smaller embedding width (256 vs 512) and number of layers (6 vs 12) in the text encoder. Indeed OC-CLIP only need to encode information about objects and relationships and we expect such encoding to require much less capacity than an encoder that needs to encode whole caption composed of multiple objects and relations between them.
18

---

2. We operate a reduced embedding space 256 for the binding module and thus first project the ViT-B-16 patches from 768 to a 256 embedding space before computing the nodes to patch cross attention logits.
3. We use the SigLip (Zhai et al. 2023 loss to make efficient use of batch chunking and gradient checkpointing.
We only perform experiments with a B- architecture for the VIT but perform the computational analysis fro both B and L backbones. We report the results in [ note that there is a significant overhead with a base architecture 2.2x but since the binding module perform the same number of operations matter what the ViT is show that when scaling the ViT backbone, the binding module is not the bottleneck anymore and the computational overhead is reduced (1.3x).
Model
VIT Backbone
Text (w.l.ct))
Binding Module GFLOPs
Text GFLOPS
Vision GFLOPs
Total GFLOPs
OCCCLIP
B
(256, 6, 20)
12( 12("num workers)
180
Ik
2.2x
CLIP
B
(512, 12. 77)
186
1k
lx
OC-CLIP
(256, 6, 20)
12 workers)
180
4.9k
lx
CLIP
L (512, 77)
186
4.9k
1.3x
Table Computational Comparison of CLIP and OC-CLIP. Calculations are made for a local batch size (per GPU) of 64. We give the Total GFLOPs based on global batch size of 8192 (=128 num workers). When scaling the VIT backbone the computational overhead of the binding module remains fixed and is not the main bottleneck anymore.
A.4 Scene Graph Parsing Discussion
Comparison of different parsing methods Although the parsing method is not the core of our contribution we provide here couple of qualitative and quantitative comparisons to motivate the choice of using an LLM to perform the parsing of the captions despite the pre-processing computational overhead it entails. We identify 3 families of parsing method that operate text-only input and provide insights on their respective:
Automatic parsing methods: method based on hand-crafted rules about the semantics in order to extract tags and more complex dependency graphs. TagAlign also compares to nltk and justifies the choice of going to Ilm-based method. We consider a representative of those automatic parsing methods based on spacy [Honnibal and Montani 2017].
Finetuned factual scene graph parser trained in a supervised way to extract scene graph. We consider a representative of them, state-of fate--o-the-aasd fi-the-art art factual scene graph parser based on T5 model [Li al.) 2023b] trained to extract fine- ime-grained scene graph information about the objects and relations in an input caption.
LLMebased, here we choose Ilama3-88 8b as a representative and leave the extensive analysisof the bias/cues of different IIm families of model for future work.
We identified failures modes of automatic parsing and finetuned that are relevant to compositional understanding of clip-like models and justify the use of an llm-basen parsing method and summarize them in Table We show on one hand that automatic parsing methods are prone to oversinplification missing relations and mistaking scene graph parser seems to be prone to relation classification error and important atteibute binding error when the different objects mentioned in caption share the same label tag.
attribute modifiers with object. On the other hand supervised
A brows cal bring
Liggnn beone
Retuiies: (os, D, DDvningiiicaton omei
Ufzzoo bown
Rekutees os, 1.1)
(Retator dasiiiiarion emer)
Ulppann
Relation:: (bviggos ns. 1
tntt the ring
Diycctc mas, the a (Whing NOS)
(of. 1. (Mileiny
Okyeas: mon dog
(orthe D.
Objecs man. dog
(on
Miee and rod
Relaioea: [1..1) (ia 2.
Rolaiona [sea, I]
Rdatiaus
Table 8: Comparison of parsing errors made by different parsers.
We additionally train OC-CLIP on COCO captions parsed by those different parsing models and compare the downstream compositional understanding performance in Figure 6 Coherent with the qualitative analysis the choice of the parsing family mostly impact relational understanding. We observe for the SugarCrepe swap object (replace rel resp.) decrease of 9.3% (resp. 14.1%) for spacy
19

---

Swap AZI
Swap 00
Redlace ACI
Replace Rai
0.7
0.75
6..00
845
046
046
8.03
buool
LL bood
2.0
bood Harting Herbod
ULM UMMt
0.00
koo!
046
13 buoot
hool
Panking Harred
Farsing Hattod
Parsing Partoc
Figure 6: Downstream Compositional Understanding of OC-CLIP when trained on different parsing of COCO-Captions.
and 3.4% (resp. 6.3%) for a supervised T5 model as compared to OC-CLIP on scene graphs extracted by llama3-8b, Close to our work, TagAlign(Liu et al., (202bb also quantitatively and qualitatively analyze the objects tags than can be extracted with an nitk-based and Ilm-based parser and show that training CLIP with an additional object and attribute tag classification loss with tags coming from an IIm results in better downstream zero-shot semantic segemntation.
Limitations of LLM-based parsing for OC-CLIP We also acknowledge that using and LLM as a parser may also have some limitations and evaluating the impact of the downstream performance of different LLMs or VLMs is an interesting question. In particular, Ilm-based parsing might not extract accurate scene graphs, especially when the dependency between the objects in captions is rather complex or ambiguous. And informing the parser in prompt with visual information might be an interesting direction. However the exact instanciation of the LLM-based parser used is orthogonal to our contribution and we leave this analysis for future work.
Scene Graph Parsing cost We performed the parsing by serving instances of Llama3. GPUs. Each datasets is then chunked in N process that do not require any GPUs and send requests to the served LLM parsers through vlim to maximize the throughput of the parallelized requests. For reference we parsed the COCO datasets (~ 500k captions) parallelizing 10 instances of the parser, and with 128 chunks in 3.5 hours and Visual-Genome (~~ 200k captions) with 8 instances, 64 chunks in 1.7hours. The parsing time can further be optimized by serving more instances, using more performant GPUs (A100, H100 etc..), serving each instance in parallel in more GPUs to maximized the number of requests that can be processed per second. For the cc3m and ec cc12m, in order to accelerate the parsing, we kept the LLM parser local using ollama on v100 GPUs. CC3M was chunked into 500 and CC12M into 1000 smaller chunks, we launched the jobs sequentially. CC12M took about 3days to parse but could likewise be accelerated using faster GPUS.
v100
A.5 Additional Compositional Understanding results
Our main goal is to evaluate CLIP-like models compositional understanding in plausible and gram- matically correct cases. Hsieh et al. 20236) have identified exploitable textual biases in previous mainstream procedurally- generated hard negatives benchmarks like the COCO and Flickr set of ARO and VL-checklist. Specifically they show that procedurally generated hard negatives are either highly grammatically incorrect and can be identified by blind model or by good language model that can measure the plausibility of the caption. The SugarCrepe thus designed to pfollow the same fine-grained taxonomy on attributes, objects, relationships as VL-checklist but ensures that the hard-negative are not distinguishable by a blind model. The main results of our paper thus focus on this benchmark. We however also give the performance of our model on the full ARO suite and VL-Checklist in Table for reference.
A.6 PUG Dataset
In this section we describe in more details the content of the synthetic experiments, give more context on the motivation along with additional results.
https://iithuh.om//llmmmmpppiip https///ollama.co/
20

---

MODEL
VL-CHECKLIST
OBJECT
RELATION
ATTRIBUTE
ATTRIBUTION
RELATION
COCO-ORDER
FLICKR-ORDER
BLIP
82.2
70.5
75.2
63.2
60.0
47.9
60.2
XVLM
85.8
70.4
75.1
86.8
73.4
cond-neguiive Methods
CVLC
85.0
68.9.7
72.0
73.0
80.6
84.7
91.7
NEOCLIP
84.1
63.5
70.9
71
81
86
CE-CLIP
71.8
72.6
76.4
83,0
MOSAICLIP
88.5
77.0
75.5
77
82.6
Donve cupioming+Hont-Neaahi
DAC-LI M
66.5
56.8
57.4
63.8
60.1
50.2
61.6
DAC-LLMMa
87.3
86.4
73.3
73.9
81.3
94.5
95.7
DAC-SAMoN
DCI
80.7
70.1
68.7
67.6
76.2
88.6
91.3
DCT.n
88.4
61.3
70.4
62.0
57.3
39.4
44.6
Table 9: Results (%) VL--Chekllist and ARO Benchmark.
Motivation The rise of data-centric hard negative methods were motivated by the bag-of-words behaviour Yuuksekgouul et al., 2023b) of CLIP noticed "simple swap-attribute" retrieval tasks. Hard-negative methods propose to mitigate this behaviour by finetuning CLIP-like models data points with minimal changes but semantically different meanings. However we experimentally observed that all the methods fail increase performance specifically in swap attribute kind of splits. In order to further isolate the root cause, we propose a series of synthetic experiments that compare covering more hard-negative data points with OC-CLIP varying proportion of training samples and hard-negative samples. By restricting the environment to closed-set vocabulary of backgrounds, attributes, and object classes, we can enumerate all possible hard-negatives, allowing us to systematically evaluate the effectiveness of different approaches. Our results show that simply adding more hard-negatives plateaus and not sample-effcient, as the swap attribute binding performance always underperforms OC-CLIP trained on less data without any hard-negativel in simple object-atrribute binding task [] However, when combined with OC-CLIP inductive bias. hard-ne hard-negatives complementarily improve downstream performance. This suggests that our model, OC-CLIP, is more sample-efficies approach addressing the bag-of-words behavior of CLIP models. We hypothesize that the root cause of this issue thus lies in the representation format used CLIP's original formulation, which relies on a single vector to capture complex semantic relationships. Our proposed method introduces inductive biases that allow the model to learn more structured representations, avoiding superposition of features [Greff et al. 2020b) and effectively mitigating the bag-of- bag-of-words behavior. Through these synthetic experiments, demonstrate the effectiveness of our approach and provide insights into the -eaple--fffciency limitations of existing data-centric methods.
Dataset splits The synthetic experiments we propose are based on the controlled 3D environment PUG [Bordes et al. 2023). We operate in a 3D envrionment with pairs single textured animals in different backgrounds. The factors of variation are
5 Backgrounds: desert, arena, ocean floor, city, circus
20 Animals goldfish, caribou, elephant, camel, penguin, zebra, bear, crocodile, armadillo, cat, gecko, crow, gianttortoise, rhinoceros, dolphin, lion, orca, pig, rabbit, squirrel
4 textures red, white, asphalt, grass
2 spatial constraints for pairs: left/right, above/under
The different splits We then construct splits that aim at evaluating separately attribute binding and spatial relationships understanding. In all the different splits, we include images with single animals all the possible background-texture htwp:ouudd--ett-----n--n-ee animal conjunctions.
Attribute Binding Splits The attribute binding training and testing splits are constructed as follows (1) list all possible pairs of animals,(2) We randomly and i.i.d. select a percentage Nuran of pairs to include in the train split, (3) For each training pair we select pair of assigned attribute (for example if cat and caribou are the train split we will assign red cat and white to caribou and will remove all the other aitribute-animal conjunction from the training. This is done
21

---

such that we can control for the replace attribute hard negative presence. (4) For each pair in the training set we separate the corresponding hard negative examples with the same bag of words but swapped attributes (referred to as Seen Pairs in Figure 7 and the same pair but different bag of
words ( referred to as Different Bag-of-words in 7 (5) finally we also isolate unseen pairs of animals. We also include the accuracy on the training pairs that do not have their corresponding hard negatives in the test set).
Vilue wodii
Pairs
(h) Tain Pairs
Scen
(d) Different Bag-of-wonds
(c) Unseen Pains
plus cipper
Single Object
Train Pairs
(h) Seen
Diffcrent Hag-of-words
Paics
Figure 7: Attribute Binding on PUG Additional Results Performance of the finetuned OpenCLIP and OC-CLIP models on a binary classification task between caption and its corresponding hard-negative. We do that for captions that mention Pairs of animals (top row) like the example in Figure (a) and for captions that mention single animal (bottom row) like the example in Figure (b).To assess the models' performance, we compute the accuracy across two dimensions. The first one is percentage of animal pairs (y-axis) seen during training (animals like elephants and fish could be seen either alone or with other animals but never together). The second dimension (x-axis) is the number of hard-neer negatives used in the training data. For instance, whether we have the combination "red elephant" and "white fish" in the training data while we only have "white elephant" and "red fish" the test data.
Spatial Relation understanding Splits For these splits we do not assign specific pairs of attributes to train/test split but rather consider pairs of animals and their order with respect to the spatial relationship tested and systematically include all the possible attributes assignment to those pairs. We then construct the different splits by restricting the number of pairs and their spatial configuration.
Hard Negative Samples For both tasks the hard negative samples we consider are align with the test tasks taxonomy. For attribute binding we always test the model's ability to distinguish between eg. a red cal and a white caribou and a white cal and a red caribou. Hence we consider as a hard negative sample any image that corresponds to the swapped attribute version of training pairs. To augment the dataset with hard negative, we sample i.i.d. a percentage % .Nhard of the training pairs and include in their corresponding hard negatives in the train set. Similarly for the spatial relationship understanding task, we test the model's 's ability to distinguish between eg. a red cat to the left of white caribou and white caribou the left of a red cat. Hence we consider as a hard negative sample any image that corresponds to the swapped order with respect to the relationship tested of the animal pairs seen during training.
A.6.1 Spatial Relation Understanding
In this section, we aim to evaluate the spatial relationship understanding capabilities of the models. To do so, we conduct controlled experiments using data splits where not all pairs of animals are seen during training. The relations considered in these experiments are "left/right" and "above/below", Hence, the task is to choose between the original caption of the form "X left of Y" and the caption with the swapped order "Y left of X We consider the following generalization axes:
Unseen object order: This axis tests the generalization when swapping the order of objects in a relationship. For example, "elephant to the left of fish" may be used for training, while "clephant to the right of fish" is used for evaluation
22

---

Unseen object pairs: This axis test for unseen pairs of animals in seen relationships.
We follow the experimental setup of section ??, and finetune OpenCLIP and OC-CLIP while considering the effect of adding different % of hard negative images and/or different % of object pairs to the training data.
We test both models on image-text retrieval tasks and report the results in Figure 8 Figure 8 b) shows the results for the unseen object order generalization, whereas Figure ( presents the results for the unseen object pairs. As shown in Figure b), OC-CLIP outperforms OpenCLIP in all data regimes considered, with improvements between 6% and 18%. Similarly, shown in Figure [(c), OC-CLIP improves upon OpenCLIP in all data regimes, yielding absolute improvements between 5% and 20%.
OpenCUP-5000
85
70
OpanCUIP0300
Medee Training Pale
Modell Training PARP
73
78
50
SpenCu
73
85
White Elephant right
Oun-/670
86
95
96
66
Dum-5570
91
05
97
0%
20%
73%
0%
20%
50%
70%
Elephant the left
Negatives
Hard Negatives
(a) Example image
(b) Unseen object order
(c) Unseen object pairs
Figure 8: Spatial Relationship Understanding. We finetune OpenCLIP and train OC-CLIP's binding module splits containing different % of animals pairs (y-axis) and different % of hard-negative image in the training split (x- axis). We test the models on images with either unseen order (b) unseen pairs (c) during training. The testing done against the swapped order of the ground truth caption as shown in the visual example (a).
A.7 Parsing
For the parsing of the training and testing data we used Ilama-3-70b Instruct model with the following prompt:
23

---

Parsing Prompt
Given caption, your task is to parse into its constituent noun phrases and relationships. The noun phrases should represent independent visual objects mentioned in the caption without semantic oversimplification. For each caption, output the parsed noun phrases (c.g,, entities) and relationships in ISON format, placing the dictionary between [ANS] and [/ANS] brackets. In the relationships, use indices specify the subject and object of the relationship mentioned in the caption. The indices of the subject and object should be integers. Here
few examples:
Caption: A large brown box with a green toy in it
Output:
[ANS]
"entities"; "large brown box", "green toy"
rellationshis";;
"relationship" "; "in",
" "subject"; 1, "object";
[/ANS]
[...] More examples
PAY ATTENTION the following:
Relationships MUST relate two different entities in the caption and NOT be unary. For example, in the caption 'red suitcases stacked upon each other', 'stacked upon each other' is not considered relationship.
Do not forget any relationships.
Relationships MUST directed. 'and' is not a relationship.
Pay attention to spatial relationships like 'behind', "left of", 'with', 'below", 'next to', etc. "and' is not relationship.
Check the right dependencies when the relationships not direct. In the caption template with it, it refers to X. Pay attention to co-references.
Now, parse the following caption into its constituting entities and relationships. You MUST place the answer between [ANS] and [/ANS] delimiters.
Caption:
To showcase the effectiveness of this parser, are showcasing below the most and least common entities and relations that are find by this parser across MS-COCO and CC12M. In Figure 9 we can that the most common entities people for the COCO dataset as well as for CC12M in Figure We also plot the least common entities in Figure 10 Finally, we also compute the number of tokens that is require for modelling the entities and relations. In Figure 14] display the frequency of the number tokens use to encode the relations and entities on COCO. Using just 10 tokens, we encode most of the entities and relations, thus do not need to have text encoder that take into input 77 tokens but can use a much smaller one instead. In Figure [5] we show a similar plot but normalized. Even dataset with noisy captions like CC12M, most entities and relations can be encoded with less than 20 tokens.
II
A.8 Datasets
Training Data For the compositional experiments we train both OpenCLIP and OC-CLIP on aggregated data form COCO-Captions (COCO) [Lin et al. 2014), Visual Genome (VG) [Krishna] et al. 2017 and GQA [Hudson and Manning 2019 All these datasets cover the same 1 11kk images from COCO but focus on different kind annotations. COCO provide global scene annotation, Visual Genome emphasizes specific region descriptions and general relationships and GQA annotates

---

-10
Top 30 Emitiee for COCO
1.5
frequency
0.5
mas
table
fold
skandoond
dogg
water
houdh
hod
halhnoom
UrANN
lising room
tollet
wellooed
pluze
5000
stkk
-10
Top
frequency
0.5
holdiap
saring on
near
weating
with sittngg on top of in fromt of
danding play ing sopped
sinting
exting
scttigg
booking
under
hehind
flling
lapiing on
Juy
sunding
Figure 9: Plot the most common entities and relations that were extracted by our LLM-based parser for the COCO datasets.
both objects and spatial relationships. Both Visual Genome and GQA have annotated scene graph that we do not need parse train OC-CLIP. For OpenCLIP, sample 2 region annotations from VG to from caption following this template A photo of (Region /) and Reeiion 2). Similarly to get the captions from GQA, if there relationship model a caption following this template photo of [Subject) {Rel) {Object). If only objects are mentionned sample up to 3 objects and give the model caption following this template A photo of [Obj7), (Obj)), Obj3).
follow Kamath al. [2023 and give the
A.9 Training Details and Hyperparameters
In table I0 we detail the hyperparameters of the OC-CLIP architecture for results in real-world compositional understanding (section 4.2).
Optimization Details In order to train OC-CLIP we followed prior work and use Adam Optimizer with 3n and 3g set to 0.9 and 0.95 and a weight decay of 0.2. We used different learning rate for the pretrained backbones and for our modules that we train from scratch learning rate of 2c for binding and the scoring modules, learning rate of -5 for the text Transformer backbone, and smaller rate of - for the ViT backbone. We also used warmup schedule for both of the text (1k steps) and the vision (5k steps) backbones followed by cosine decay. We train the model for total of 100 epochs.
25

---

1.2
Least
1.1
frequescy
abandoaod
delapidaeed
light colired counentop
reffigeratoo
plasii poraalipahc compatee
sppoabemm Soy
right
the heating misorrycl cles
Clive
while polm piloe
chicken bergor
white hand
fumny focel charnster
Toroel Bog
nel pouch
Envn swmmm Unain
minhow Image
gress DOW truck
axpholl
grspp
funky looking purple
candatted school
chouolate sprinkid
sbot
Japaneee
Dinnyy
1.2
Leust 30 Relation for COCO
111
fregiency
0.9
side
focling horeel
being mixad
toging to dome watte with
sctup against
one hand on botoom on use by:
porched lighh up in wakingg
pooning cull onto
deieniing
care for
finibeed
bowwn
drinking milk from
resching head into
squashin
walking single file in comere with
at each otee
fesling
wrapped
gathe
nying close
jumps
olter oe propuring
dangling
Figure 10: Plot of the least common entities and relations that were extracted by our LLM-based parser for the COCO datasets.
A.10
Binding Module Code
See Figure l6
26

---

-10
Entiies CCT2M
Word frequecyy
[person
baildnng
White backgrondd
hod
womin
mun
beach
vesttrr ilustratioo
honse:
ropalty froe illustration
kitchen
portrait
hono
flowess
110
Top 30 Relation for CC.EM
frequency
0.5
9016
On
2nd
located
weuting
has
Inclados
boldnng
incholing
festuring
is
fn
noxt
Canures
shosing
againe
sitting
Figure 11: Plot of the most common entities and relations that were extracted by our LLM-based parser for the CC12M dataset.
Hyperpprametee/Prrameeee Init
Architecture
Value
Binding Module
-Image Patches Processing
Linear
768 256
-Self-Attention #Layers/#Heads
2/4
Self-Attention MLP ratio/act
2/nn, //LU
Keys K, Values V Normalization Keys/Values
Linear
256, 256
LayerNorm
256
Grouping Module
Cross-Attention #HHeds
Queries
Linear LayerNorm
256
Normalization Queries
256
Num Default Tokens Qdefnutt
nn. Par
Scoring Functions
Object Scoring Function
cosine sim
Relation Scoring subject fa
MLP(128 + 256, 128)
layers layers
Relation Scoring object fo
MLP(128 256, 128)
Coef ent init (learned parameter)
1.5
Coef rel init (learned parameter)
0.5
Table 10: Table of hyperparameters for OC-CLIP architecture
27

---

370 0mc Empurny
Figure 12: Distribution of the number of tokens require for modeling the entities and relations COCO (we do not need more than 8 tokens to capture 99% of the entities in COCO). Since we need less token, leverage smaller text encoder extract the entities and relations.
Teeesho ovrr Devitete beites CC1M
Teesiraeancyy Bariiasee Aestiara
232
Figure 13: Distribution of the number of tokens require for modeling the entities and relations CC12M (we do not need more than 14 tokens to capture 99% of the entities in CC12M). Since we need less token, leverage smaller text encoder extract the entities and relations.
28

---

Frequency Distribution Entities COCO
100
99% Frequency (8 takens]
Cumulative Percertage
40
20
Number Tokens
Frequency Distribution Relations COCO
100
90% Fraquency (8 takens)
80
Cumulative Percentage
60
40
20
Number Tokens
Figure 14: Distribution of the number tokens require for modeling the entities and relations on COCO (we do not need more than 8 tokens to capture 99% of the entities in COCO). Since need less token, we can leverage a smaller text encoder to extract the entities and relations.
29

---

Token Frequency Distribution for Entities CC12M
100
9-00-00
80
Cumulative Percentage
40
20
99% Frequency (14 tokens)
Number Tokens
Token Frequency Distribution for Relations CC12M
100
000
80
Cumulative Percenaage
40
20
99% Frequency (17 takens)
20
40
50
60
Number Tokens
Figure 15: Distribution of the number tokens require for modeling the entities and relations on CC12M (we do not need more than 14 tokens to capture 99% of the entities in CC12M). Since need less token, we can leverage a smaller text encoder to extract the entities and relations.
30

---

class AssignAttention Kndull)
def init
self, dim, gkv skv_bias-False, (k_scale-None) super ()__initt.......
self.scale qk_scale or dimx
self.q_proy proj = nn nn. um.Sequential (nn. Linear (dim, dim, laasgkv_biass
self.k_proj nn. Se..eenuniilll Lineiiear (dim, dim, stas-gkv_bias
self.v_proo nn. Sequenntiaal (nn. Linear (dim, dim, bias-qkv_bian
def forward (self, query, key=None, value=None): #before cross attention projections
self.g_pron (query)
k celf.k_proj (key)
v self.v_proj (value)
#scaled dot product
attn (q @ k.transpose 1-2, -1)) self.scale #softmax across query dim
attn_dim = -2
attn F. c.coftmax (attn, dim-attn_dim) + 1c-8
fattn normoalization attn attn / (attn. wwt.:sssmmmmmll. keepdin=True) output torch. boa attn, v)
return output
class BindingModule (mn. Module) Mon.ul
def
dim, num_patches, num_default_tokens
() ()peit))__nit__
holf.im_pror nn Sequentiaaa (nn. Linear nn.GELU nn, Linear (dim, dim)
dim),
self.pos__mbeeddnng = nn Parametter randn (nun_patcchss, dim)
salf.img_processeo
nn Seguentnttaal ( ResidualAttnBBochh (dim, 4)
ResidualAttnBlcck (dim, 4))
self.default_tokee = nn. Parameter(
torch. (1, sum_default_tokents dim) )
self.to_kg_groups nn Sequentiaa (an Linear (dim, dim))
self.dim dim
delt..un_detault_ttkee
num__ee gun_detault_tokenn
self.k_norm nn (aayeeroommdiin)
self.v_norm = nn LayerNorm (dim)
aslf.assign_slcss AssignAttent (dim)
def encode_patches (self, patches)
patches self.im im cpatc
patches patches self. pos_embeddingg patches self. hal.fiim__proeesoo (patches) K_ing, V_img torch.split(
self,to_kod helf.to_kg_groups (patches), (pattcchee self. dim=-1)
K_ing, V_img self. s (off.kknoom(K_inng self.v_nome (V_img)
return K_img, V_img
def group (self, query_tokens, K_img, V_img) #adding default tokens query_tokens- torch, cat llkeery_tokens, default_ ), 1)
iquery_tokens, fremove default tokens out out :-self. momoff..mm_effullttttoeeee return out
out - self. www..aassign_soll (ourreressssssseerrccockeesss Kiimg, V Vmggg
Figure 16: Code for the Binding Module
31

---

-10
Top 30 Entiises for CC3M
Froquency
Whit backgroumd
city
pecniene
bosch
fooball player
stege
pontakk
musiaal instrmett
river
mode
ca
pop unbi
politiion
flgg
Top 30 Relanion For CC3M
snennds
agains
over
for
and
anendd
bolding
plaping
sitting on
along
behind
ot
untives
performs
helskedd
Figure 17: Top entities and relation for CC3M
32

---

-10
Top Fntinies CC12M
pperoos
posvin
baiking
whet Bnckgmonn Fimmodd
bod
man
beach
sirl
wedeer I1la striiin
hoose
ropalty fre seeellleee attaiiiiii
kiishen
purtrit
hoons
Blwerss
30 Relaion for CC12M
Frequency
0.5
and
located in
by
wearing
has
incindes
bnkinng
noar
in mest to
showing
againo
siiin on
Figure 18: Top entities and relation for CC12M
33

---

-104
Tup 30 Emities for CoCO
1.5
Frequescy
folk
skarebour
dog
water
besch
hol
huikdig
Food
Litege coom
surlband
pizza
S00W
sink
of
Top Relation for COCO
Frequency
Laying down
bakdnng
attingg on
has
near
wearing
sittnng top of
playing
sitting in
extin
under
Rylng
laying on
top
in foont
dealing
szunding
sitting
looking
topped
sanding
Figure 19: Top entities and relation for COCO
34

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable "[No] ", it is perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" "we were unable find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. we acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendíx. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The 3 parts of our Results section each verify the claims listed in the introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
35

---

Justification:
We discuss limitations of our method (scale, computational overhead, parsing) extensively our appendices. Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No thcoritical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all the training parameters, dataset preprocessing, and pseudo code of our binding module to reproduce the results.
Guidelines:
36

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Pseudo code is given, we plan releasing the code later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides/CodeSuuiin
httpe///CoooSSubbiisissoooooooio for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //npss.cc/pubicc/ggg... guides/ CodeSubmiss ismmiissis issi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
37

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All the details given in Appendix A.9
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
38

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We added impact statement in the Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
39

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspppppssttteeeeee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We plan to release the code later.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
40

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
41


Reviews:
[{"review_id": "EVvkVr5INj", "reviewer": "Reviewer_Hp4d", "summary": "The authors introduce Object-Centric CLIP (OC-CLIP), which enhances CLIP with object-centric inductive biases to improve compositional understanding which cannot be solved by CLIP. The proposed model not only enhances the performance of CLIP-based models in multi-object compositional understanding, but also paves the way towards more accurate and sample-efficient image-text matching of complex scenes.", "questions": "1.How to combine the proposed method with current vLLM without decreasing the training and inference speed as it is not end-to-end.\n2.How to explain the performances of the proposed method is not so good on some tasks.\n3.Figure 2(b) and c do not seem to be straightforward and easy to understand. It misleads me to think of correlation.\n4.How will the parser impact the performance?", "limitations": "1.How to combine the proposed method with current VLLM is not discussed, especially the performance of training and inference.\n2.How to further improve the performance of the method as in some tasks it is not so good.\n3.This work has not discussed the impact of parser on the whole workflow, which may lead to a disaster.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.It tries to deal with the complex compositional scenes involving multiple objects and their spatial relationship which cannot be solved by CLIP.\n2.It can work with noisy data with zero-shot.\n\nWeakness:\n1.These modules do not seem to be a good plug in to current vLLM (not end-to-end), and they may decrease the training and inference speed.\n2.The performance does not seem to be good in some tasks on Table 1.\n3.The parsing procedure at the beginning may limit the performance.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and appreciate the opportunity to clarify the limitations mentioned. Below are our responses to the specific points raised : \n\n**Q1-a :Compatibility with vLLM:**\n\nvLLM is a framework that optimizes inference and training of llm-based models. We do in fact use vLLM to optimize the scene graph parsing efficiency (Appendix A.4).\nHowever vLLM is generally not considered useful for training and inference of embedding models like CLIP. This is because the primary optimization vLLM implements (e.g. chunked prefill, automatic prefix caching, etc..) focuses on generation efficiency.\n \nWe would like however to clairify that our model is an embedding model and is in fact end-to-end. It can therefore benefits from any additional opitimizations applicable to CLIP-like models.\n\nWe also would like to also emphasize that the **scene graph parsing step is not part of the model but a data preprocessing step done offline** as specified in l136. The model itself **is end-to-end** and can benefit from further wrapping and optimizations the same way as CLIP. We will update the main Figure to clarify that aspect.\n\n**Q1-b: Computational Efficiency:**\n\n  Our module introduces a trade-off in computational efficiency, which we discuss in detail in the \"Computational Analysis\" section of Appendix A.3 (l752). We quantitatively measure the computational overhead of our modules and demonstrate that for larger-scale ViT models (e.g., ViT-L), the binding module does not become a bottleneck, as it remains fixed when scaling the ViT backbone. To mitigate overheads when training our model from scratch, we propose:\n\n* Using a smaller text encoder (fewer layers and reduced width), since the OC-CLIP text encoder only needs to encode information about single objects and relationships.  \n* Operating the binding module on a reduced embedding space (256 vs. 512 for the original CLIP).\n\n**Q2 -Drop in Performance on some Tasks:**\n\nThe binding module's relatively low training vocabulary size (eg. COCO domain), due to being trained from scratch, is the primary reason it does not outperform (and is not expected to)  all baselines on \"replace\" and \"add\" tasks of Table 1\\. We anticipate this performance gap would close if trained on the original OpenCLIP pretraining set (Laion400m), as indicated by results in Section 4.3 at the 15m scale.\n\nHowever we want to highlight that on tasks that require finegrained object and attribute binding (spatial, and swap splits) our model outperform all the other baselines, even when they are trained on order of magnitude more data (eg. DAC finetuned on CC3M augmented with hard negatives).\n\nSpecifically, the \"add obj\" (91.1% vs. 93.4% for CLIP counterpart) and \"replace obj\" (94.6% vs. 95.4% for its CLIP counterpart) splits in SugarCrepe show slightly lower performance for our model. While both models achieve high accuracy, this difference stems from two factors:\n\n1. **Vocabulary Generalization:** Our binding module and multimodal alignment scores are trained from scratch, limiting their generalization to the vocabulary they were exposed to during training.  \n2. **Out-of-Distribution Negatives:** The \"replace\" and \"add\" negatives in SugarCrepe are generated using an LLM (GPT-4). This process does not guarantee that the objects/attributes introduced are within the distribution of our training data (COCO/VG). These LLM-generated negatives may include objects/attributes outside the COCO domain but within OpenCLIP's broader pretraining domain, leading to a slight, albeit minimal, drop in our model's accuracy. Training on the original OpenCLIP pretraining set (Laion400m) is expected to mitigate this issue.\n\n**Q3 - Lack of Parser Impact Discussion:**\n\n We would like to clarify that  **we already extensively discuss the impact of scene graph parsing in Section A.4** of the Appendix (l777) as referred in line 136  and line 382 of the main text, where we:\n\n* Compare different parsing methods (automatic, fine-tuned lightweight parser, and LLM-based parser) and describe their qualitative and quantitative performance impacts, along with the error types each can introduce.  \n* Comment on the limitations of LLM-based parsing, especially when ambiguous parsing requires visual grounding, providing interesting directions for future work.  \n* Discuss and reference the scene graph parsing cost in the preprocessing step.\n\n**L3 -Clarity of section 4.1 figures** \n\nFigure 2(b) and 2(c) aim at comparing CLIP and OC-CLIP sample efficiency in a controlled setting where we vary the training data size (y axis) and the amount of hard negatives (x-axis). It shows that without any hard negatives and only 30% of the possible animal pairs combinations OC-CLIP outperforms CLIP even when trained on 70% of the colored animals pairs combination and 70% of all the possible swap hard negatives. We will update the figure titles of part 4.1 to emphasize that the numbers correspond to performance accuracy and not correlation. We apologize for the confusion\\!\n\nWe hope this clarifies our approach and addresses the reviewer's concerns. We are happy to answer any additional questions the reviewer might have."}, {"review_id": "eAXSRkP8Nl", "reviewer": "Reviewer_ArR9", "summary": "Existing contrastive models like CLIP excel at recognizing individual objects but have limitations in understanding spatial relationships in multi-object scenes and correctly binding object attributes, a problem known as the \"binding problem\". This paper introduces Object-Centric CLIP (OC-CLIP), a new pretraining method aimed at improving the compositional understanding of Vision-Language Models (VLMs) for complex scenes. OC-CLIP boosts performance in real-world benchmarks for attribute binding and spatial relationship understanding, and achieves notable improvements in zero-shot classification and compositional understanding tasks.", "questions": "1. Despite the implemented mitigation strategies, such as using a smaller text encoder and operating the binding module in a reduced embedding space to lower computational overhead, the base architecture still exhibits a significant 2.2x overhead. Aside from the strategies mentioned in the paper, are there any other potential optimization methods that could further reduce this computational cost?\n2. The paper discusses error modes of different parsing methods in Appendix A.4 and qualitatively analyzes the advantages of LLM-based parsers in relation understanding. To what extent would OC-CLIP's downstream performance be impacted if the parser generates low-quality scene graphs, e.g., incomplete or containing erroneous objects/relations? Could this impact be mitigated through the model's inherent robustness or specific training strategies?\n3. The paper does not extensively discuss how the number of slots or their granularity affects performance. In designing the binding module, how is an appropriate number of slots chosen to adapt to a dynamic number of objects in a scene? Has there been exploration into mechanisms for dynamically allocating the number of slots, or the impact of combining different granularities on the model's understanding capabilities? What types of visual information do the \"default query tokens\" introduced in the binding module learn during training, and do they offer insights into the model's \"background understanding\" or \"non-core object recognition\" abilities?\n4. Considering how OC-CLIP enhances CLIP's compositional understanding through object-centric inductive biases and a structured similarity score, when applying OC-CLIP's core mechanisms to large Vision-Language Models, could this effectively improve their visual comprehension and significantly mitigate visual hallucination, specifically attribute hallucination and relation hallucination? More precisely, how might OC-CLIP's slot-structured representations and scene graph constraints help VLMs more accurately bind object attributes and understand spatial relationships, thereby reducing the generation of factually incorrect image descriptions?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. The binding module utilizes an inverted cross-attention mechanism and introduces default query tokens to process visual information not explicitly mentioned in the text, a design that is unique.\n2. The proposed structured similarity score combines object and relationship scoring. It is also supplemented by a relational loss to enforce the learning of non-symmetric relationships, providing a novel and more refined approach to image-text alignment.\n3. Research has demonstrated that enhancing compositional understanding through model architectural design (inductive bias) can be more effective than solely relying on large-scale hard-negative augmentation. This holds significant practical implications for resource-constrained or rapid-deployment scenarios.\n\nWeaknesses:\n1. The OC-CLIP method relies on a parser to extract object-centric attributes and spatial relationships from text descriptions. Does it quantify the impact on OC-CLIP's downstream performance if the parser produces low-quality, incomplete, or erroneous scene graphs.\n2. Although the paper demonstrates OC-CLIP's scaling potential at the 15M scale, the model still needs to be further scaled to at least the 400M scale to be fully comparable to all CLIP variants.\n3. OC-CLIP introduces a notable computational overhead due to the binding module requiring additional cross-attention operations. Despite the paper implementing mitigation strategies, such as using a smaller text encoder and operating in a reduced embedding space for the binding module, there is still a significant 2.2x overhead for the base architecture.\n4. The core of the binding module is to generate \"slots\". However, the paper does not extensively discuss how the number of slots or their granularity affects performance. A fixed number of slots might not adapt well to scenes with a dynamic number of objects, and the choice of slot granularity could also influence the model's understanding capabilities. The paper does not provide insights into how to select or dynamically adjust the number of slots.", "was_revised": true, "final_justification": "After careful consideration of their responses and alignment with fellow reviewers, I will maintain my original rating.", "rebuttal": "We thank the reviewer for their insightful feedback and supporting our work\\! We take the opportunity to answer the listed questions below : \n\n**Q1 \\- Additional strategies to further optimize the computational cost**\n\nSince the bottleneck of our module for the binding module is the number of KQ comparison in the binding module (where each query node from the scene needs the be compared to each key candidate from the visual backbone), any method that reduces the number of candidate keys could drastically improve the computational efficiency during training. In that spirit combining our binding module with token merging methods such as \\[1\\] could constitute a good avenue for further computational cost improvement. \n\nWe would also like to emphasize the fact that this 2.2x overhead is present at training time only (because each element of the batch needs to perform the binding KQ attention with all the other elements of the batch to compute the CLIP loss), At inference time the overhead is minor. \n\nMoreover we would like to emphasize that while our model introduces a computational overhead, our training from scratch results in section 4.3 shows better sample efficiency on downstream zero-shot performance when compared to CLIP. For that reason we believe the computational overhead introduced by the binding module to be a reasonable trade off for better general performance and compositional understanding.\n\n\\[1\\] Token Merging: Your ViT but Faster\n\n**Q2 \\- Discussion on Robustness of the Parser**\n\nThe quality of the parser definitely constitutes a bottleneck in terms of performance. We also comment on the quantitative impact of the parser quality in Figure 6 where the parser ability to extract good relationship structure greatly impacts performance on the relational tasks (12 to 15% performance drop when using a Spacy-based parser vs an llm-based parser).\n\nWe anticipate that the following mitigations strategies could greatly improve the robustness the parsing : \n\n-  Finetuning an llm-based parser on noisy captions to extract good scene graphs would be straightforward (although costly),  \n- Including more noisy and hard parsing examples given in the context prompt of the llm-based parser,  \n- Allowing the llm-based parser to filter out bad data points by prompting it to output an empty graph (thus ignored) when unsure about the scene graph parsing .\n\nWe believe those directions constitute great avenues for future extensive research on the parsing method itself and are orthogonal to our approach.\n\n**Q3a \\- Number of slots analysis**\n\nWe would like to clarify that our approach in fact handles a dynamic number of slots as opposed to fixed (slot attention like) methods. The extraction of the slots in our binding module are conditioned  on the input scene graph, such that the number of visual slots will be equal to the number of query nodes in the input graph (corresponding to the number of parsed textual objects) . We believe this design is key to the inherently ill-defined notion of an “object” in a visual scene that can be understood at different levels of granularities (as emphasized by the reviewer). In our case the visual patches are grouped conditioned on the objects (=query nodes) mentioned in the input graph. The number of resulting slots will be equal to the number of nodes in the scene graph.\n\n To give a more concrete example, let us consider an image containing several colored chairs. For that same image  different scene graphs will lead to different slots decomposition depending on the granularity of caption:  \n\n* (1) “a group of chairs” will be parsed as a single node graph “a group of chairs” that will group together the visual information coming from all the chairs into one slot \n\n* (2)“a red chair and a blue chair” will be decomposed into a graph with  2 separate query nodes ( ‘red chair’ and ‘blue chair)’ and each one of the nodes will group the visual patches that correspond to a specific chair into a separate slot, resulting in 2 slots. \n\nWe believe this conditioning mechanism is key to the dynamic handling of plausible scene decomposition and hope we have clarified this point. \n\n**Q3b \\- Default Query Token role**\n\nSince the perceptual grouping is done using inverted cross attention ( eg. by applying the softmax along the query dimension), this inductive bias pushes all the visual key candidates to be softly matched along at least one query (as explained by \\[2\\]). However, natural captions are far from being an extensive description of the visual content and can arbitrarily focus on a single part of the image only. Since we only want the relevant visual information ( as captured by the input scene graph) to be extracted from the candidate patches, we need to re-route the “un-necessary” visual information towards other tokens, that we coin default query tokens. Those tokens participate in the competitive grouping attention but are not taken into account when computing the score. Their role is thus to absorb any remaining visual information that is not mentioned in the input scene graph.\n\n\\[2\\] Object-centric learning with Slot Attention\n\n**Q4 \\- Extension to MLLMs general framework**\n\nWhile a direct extension to the OC-CLIP to a more general MLLM framework is not straightforward, a good avenue for future work could be to study the impact on the representation learnt by the vision backbone of OC-CLIP when trained from scratch (on larger 400m or 2b scale, like its CLIP counterpart) on a downstream Llava-style MLLM (using OC-CLIP backbone stitched to an LLM by an MLP connector). Indeed, without any specific patch tokens constraints, CLIP-like training does not force any locality on the visual patches representation, which might in turn lead to poor feature binding at the visual representation level of the MLLM. Techniques like masked image modeling, or local iBot losses (as used in Siglip2 and Dinov2) aim to improve the locality and density of the visual patches.Therefore, it could be an interesting future work to study how OC-CLIP local graph matching constraints might similarly impact the locality of the visual patches that can be used by a downstream MLLM. As this question requires full scale training from scratch, we leave that for future work.\n\nWe hope we have addressed the reviewer concerns and are happy to answer any additional questions the reviewer might have."}, {"review_id": "5x9zaz1qLZ", "reviewer": "Reviewer_9eeE", "summary": "This paper proposes Object-Centric CLIP (OC-CLIP), a novel method that enhances compositional scene understanding in CLIP-like models by incorporating object-centric inductive biases. Instead of relying on hard-negative samples, OC-CLIP uses a binding module to align a slot-based image representation with a scene graph extracted from text, capturing object attributes and spatial relationships more effectively. The model achieves significant improvements on compositional benchmarks (e.g., +16.5% on SugarCrepe, +89% on COCO-spatial) and shows strong sample efficiency and scalability. However, it depends on a parser for scene graph extraction and requires further scaling to match larger CLIP variants.", "questions": "Please address the question listed under the Weaknesses section. I will consider raising my final score if these concerns are adequately addressed.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Introducing the binding module to enhance the compositional understanding capabilities of CLIP-like models is intuitively making sense and technically sound.\n2. The proposed algorithm is easy to integrate into the existing CLIP-like models' training pipeline.\n\nWeaknesses:\n1. The proposed algorithm appears unable to capture directional spatial relations. Take the relationship “A red apple to the left of a blue car” (Line 142) as an example. I don't think the proposed structured similarity score is able to learn the spatial relationship beween the apple and the car. Because in Eq.2, the relationship embedding is learnt from the fs([r, Ss]) + fo([r, So]), where the two terms are simply added. This operation makes \"A to the left B\" and \"B to the left A\" exactly the same, because A + B = B + A.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' response and acknowledge that it addresses my main concerns. Therefore, I have decided to raise my score from 3 to 4.", "rebuttal": "We thank the reviewer for their feedback and appreciate the opportunity to clarify the limitations mentioned. We understand the importance of accurately capturing directional spatial relationships in our proposed algorithm. We would like to clarify how our structured similarity score is in fact designed to handle such directional nuances.\n\n### **Explanation of Directional Handling**\n\nIn our model, the functions ( $f_s$ ) and ( $f_o$ ) are designed on purpose with different learnt parameters ( $\\\\theta_s$, and $\\\\theta_o$) to capture the directionality of relationships and break the symmetry of the addition. This means that the embeddings for the subject and object are processed differently, allowing the model to distinguish between \"A left of B\" and \"B left of A.\"\n\n**Simplified Example**\n\nTo illustrate this, consider a simplified example where the slots S are 2d embeddings containing x,y coords of a visual object. \n\nIn that case  $f_s$ and  $f_o$  can easily model a simple ‘left of’ relationships by extracting the x coord with different signs (+ for $\\\\theta_s$ and - for $\\\\theta_s$)  such that: \n\n$f_s(A, left) = f_{\\\\theta_s}(A, left) \\= x_A$\n\n$f_s(B, left) = f_{\\\\theta_s}(B, left) \\= x_B$\n\n$f_o(A, left) = f_{\\\\theta_o}(A, left) = -x_A$\n\n$f_o(b, left) = f_{\\\\theta_o}(B, left) = -x_B$\n\nScore ( ‘A left of B’ )  $= f_s(A, left) +f_o(B, left)  = x_A - x_B$\n\nScore(‘B left of A’) $=  f_s(B, left) \\+f_o(A, left)  = x_B - x_A = $ - Score ( ‘A left of B’ )\n\nHence capturing the directionality of the spatial relationship ‘left of’.\n\nThis approach is easily generalizable to higher dimensions. In a multi-dimensional space, the learnt functions ( $f_s$ ) and ( $f_o$ ) can be extended to operate on vector embeddings, allowing the model to capture complex spatial relationships by considering all relevant dimensions. The distinct parameterization ensures that directional relationships are accurately modeled regardless of dimensionality.\n\n**Learning Directionality with local loss ( $\\\\mathcal{L}_{\\\\text{rel}}$ )**\n\nTo further enhance the model's ability to learn directional relationships, we employ a loss function (  $\\\\mathcal{L}_{\\\\text{rel}}$ ) (as described in Eq. 3.2 and Line 197). This involves swapping the subject and object indices to create a local negative example, effectively enabling our module to learn a non-symmetric relation score. We have ablated this local loss in Appendix 3.2 to demonstrate its impact.\n\n**Empirical Validation**\n\nOur results on the GQA-spatial and VG-spatial from the What’s up benchmark (Table 2), which are designed to exclusively test for spatial relationship understanding (eg. ‘A chair to the left of  the table’), further supports the model's capability. We achieve accuracy improvements from random chance to over 90% when compared to CLIP, indicating the model's effectiveness in learning non-symmetric relation scores.\n\n**Conclusion**\n\nBy employing distinct parameter sets for $f_{\\\\theta_s}$  and $f_{\\\\theta_o}$, and leveraging ( $\\\\mathcal{L}_{\\\\text{rel}}$ ), our model is capable of learning and distinguishing directional spatial relationships. This design choice ensures that the model does not treat \"A left of B\" and \"B left of A\" as equivalent, thus addressing the concern raised. We will emphasize this asymmetrical parametrization in the main method description to avoid any confusion.\n\nWe hope this explanation clarifies how our score module can handle directional spatial relationships. We are open to answer any additional questions the reviewer might have."}, {"review_id": "WKSM9WAzu5", "reviewer": "Reviewer_7dWz", "summary": "This paper presents OC-CLIP, a CLIP-based vision-language model enhanced with object-centric inductive biases for compositional understanding. The proposed method integrates a binding module (inverted cross-attention) with a structured similarity score guided by scene graphs. Experimental results on multiple benchmarks (SugarCrepe, GQA-spatial, COCO-spatial) show significant gains over baseline models, highlighting the benefits of slot-based image representation and relational modeling.", "questions": "At this stage, due to the above concerns—particularly the lack of fair and comprehensive comparisons, the potentially misleading claims about hard-negative usage, and the unclear generalization performance—I am unable to recommend acceptance. That said, I find the direction promising and technically sound, and would be open to increasing my score if the authors can address these issues in rebuttal.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* A clearly articulated motivation around the binding problem in CLIP.\n* Introduction of object-centric inductive biases via slot-attention-like mechanism and structured similarity score.\n* Strong results on targeted compositional benchmarks.\n\nWeaknesses:\n* The text-only Vera baseline from the SugarCrepe paper is not reported or discussed, despite the fact that the ARO benchmark is explicitly designed to be partially solvable by language priors. This omission significantly weakens the claims regarding model improvements on ARO.\n* The paper emphasizes that it avoids hard-negative captions during training. However, hard-negative scene graphs derived from text are used instead. These serve an equivalent function, so the distinction seems more rhetorical than substantive.\n* OC-CLIP is fine-tuned on COCO, GQA, and VG, which are also the sources of the evaluation benchmarks. Meanwhile, some baselines are evaluated under stricter zero-shot settings. Without proper controls, it is unclear whether the observed gains stem from architectural improvements or data distribution alignment.", "was_revised": true, "final_justification": "The rebuttal fully address my concerns, I lean towards accept as promised.", "rebuttal": "We thank the reviewer for their insightful feedback and take the opportunity to address the listed concerns below :\n\n**Q1 \\- Missing Text-only Vera baseline** \n\nWe acknowledge the importance of including the Vera baseline in our analysis and will incorporate it into our updated manuscript for both Table 1 and Table 2\\. Our results demonstrate that our model's enhancements are robust and extend beyond the capabilities of text-only baselines, which strengthens our contribution.\n\n|  | ARO-R | ARO-A | SC  \\- swap att | SC  \\- swap obj |\n| :---- | :---- | :---- | :---- | :---- |\n| Vera | 61.7 | 82.6 | 49.4 | 49.2 |\n| OpenCLIP-ft | 50.1 | 60.0 | 63.1 | 72.4 |\n| NegCLIP | 80.2 | 71.0 | 75.4 | 75.2 |\n| OC-CLIP | **84.9** | **84.0** | **88.9** | **83.5** |\n\nWe would like to emphasize that the **Vera text-only was initially used by the Sugarcrepe paper as a motivation to develop their benchmark** and to challenge models beyond mere plausibility and grammar correctness  In fact, the Vera baseline has a random chance performance in all the Sugarcrepe splits.\n\n**We will update our results tables with the Vera scores** for the sake of completeness of baselines coverage. We thank the reviewer for this suggestion\\!\n\n**Q2 \\- Hard-negative usage clarification**\n\nWe appreciate the reviewer's concern regarding our statement about not using hard negatives during training. Upon reflection, we believe this statement may have been misleading as our proposed method does utilize a form (although not finegrained) of negatives derived from the scene graph structure.\n\nTherefore, we'd like to contrast the negatives used in OC-CLIP with the hard-negatives leveraged by other approaches. Our method employs \"negative\" scene graphs, as defined in Eq. 5, which are derived from the scene graph at a coarse level (***manipulating relationship directions only, needed to learn a non symmetric relation score***). However, other hard-negative methods operate at a finer-grained level, generating new negatives by ***altering specific attributes or parts of speech tags*** in the ground-truth sentence (including both nodes and edges content as well as relationship directions). **Our approach does not employ such fine-grained modifications.**\n\nWe in fact refer to other methods as employing “**finegrained hard negatives**” as specified  in lines 6, 204, 214 and 373\\. \n\nWe believe this distinction in granularity is essential to consider when comparing our approach to other hard-negative-based methods, especially in our controlled experiments (Section 4.1) focused on attribute-binding where fine-grained hard negatives consist of swapping the attributes of two objects in the scene. In those controlled experiments, no hard negatives are used for OC-CLIP (the controlled scene graphs do not have any edges).\n\nIn order to avoid any confusion, we will update our manuscript with a **dedicated paragraph** clearly stating the difference between the graph-based relationship negatives (as defined in Eq 5\\) used in OC-CLIP and the fine-grained negatives used by other approaches. We will clarify the claim accordingly.\n\n**Q3 \\- In-d vs architectural improvements** \n\nWe would like to **emphasize that the OpenCLIP-ft baseline is finetuned on the same data mixture as ours** ( COCO, VG, GQA). Therefore, we believe this comparison **isolates the effectiveness of the proposed modules vs the use of data from a similar distribution** to ARO/SugarCrepe. When compared to this baseline, OC-CLIP shows improvements of \\+15.1% in the hard swap-attribute split and \\+23.2% on ARO-A.. \n\nMoreover the controlled experiments in section 4.1 are specifically designed to further isolate our model contribution effectiveness by comparing both OC-CLIP and CLIP trained on a 3D  simulated domain with fixed vocabulary size. \n\nThe results of section 4.3 further confirms the source of improvements coming from our architectural biases. In this setting we train both CLIP and OC-CLIP from scratch on cc3m and cc12m and show that not only has better sample-efficiency but outperforms CLIP by a large margin in zero-shot Imagenet classification and compositional understanding (eg, SugarCrepe).\n\n We will update our manuscript in the baselines description paragraph (l273) accordingly to avoid any confusion and clearly state that we finetune the  Openclip-FT baseline on the same data as OC-CLIP.\n\nRegarding the other baselines, we would like to emphasize that all of them except DAC are finetuned using data  in the COCO/VG domain. The exception DAC recaptions 3m images coming from CC3M dataset along with dense and finegrained hard negatives.\n\nWe hope this clarifies our approach and addresses the reviewer's concerns. We are happy to answer any additional questions the reviewer might have."}]

Author rebuttal:

Meta-review: The paper proposes an approach to integrate inductive biases into the pretraining of CLIP-like models to improve their compositional understanding. This is done via a binding module that introduces scene graph information into the training of the model. The evaluation focuses on multi-object compositional understanding and shows good results.

The paper was considered by four reviewers. The final ratings were: BR (reviewer did not answer rebuttal) - A - BA - BA
Reviewers mainly highlighted the paper's approach towards a compositional understanding of CLIP style models as well as the demonstrated scaling behaviour. Most questions and weaknesses could be resolved during the rebuttal.

The AC follows the majority of the reviewer voting and recommends accepting the paper. The AC would encourage the authors to integrate the findings of the rebuttal in the CR version of the paper.
Decision: Accept (poster)