CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally
Authors: Darina Koishigarina
Arnas Uselis
Seong Joon Oh
Venue: ICLR 2026 Poster (2026)
Keywords: vision-language models
CLIP
compositionality
binding
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: CLIP binds concepts, but only uni-modally.

Abstract: CLIP (Contrastive Language-Image Pretraining) has become a popular choice for various downstream tasks. However, recent studies have questioned its ability to represent compositional concepts effectively. These works suggest that CLIP often acts like a bag-of-words (BoW) model, interpreting images and text as sets of individual concepts without grasping the structural relationships. In particular, CLIP struggles to correctly bind attributes to their corresponding objects when multiple objects are present in an image or text. In this work, we investigate why CLIP exhibits this BoW-like behavior. Our key finding is that CLIP does not lack binding information. Through linear probing, robustness tests with increasing object counts, and conjunctive search experiments, we show that attribute–object bindings are already encoded within CLIP’s text and image embeddings. The weakness lies in the cross-modal alignment, which fails to preserve this information. We show it can be accessed cross-modally with a simple linear transformation to text embeddings. This improves CLIP’s attribute-object binding performance and confirms that the information was already encoded unimodally. In practice, this means CLIP-based systems can be enhanced with a lightweight linear layer trained on existing embeddings, avoiding costly encoder retraining.

Published as a conference paper at ICLR 2026
CLIP BEHAVES LIKE A BAG-OF-WORDS MODEL CROSS-MODALLY BUT NOT UNI-MODALLY
Darina Koishigarina: Arnas Uselis & Seong Joon Oh Tübingen Al Center, University of Tübingen
ABSTRACT
CLIP (Contrastive Language-Image Pretraining) has become a popular choice for various downstream tasks. However, recent studies have questioned its ability to represent compositional concepts effectively. These works suggest that CLIP of- ten acts like a bag-of-words (BoW) model, interpreting images and text of individual concepts without grasping the structural relationships. In partie- ular, CL.IP struggles to correctly bind attributes to their corresponding objects when multiple objects are present in an image or text. In this work, we inves- tigate why CLIP exhibits this BoW- behavior. Our key finding is that CLIP does not lack binding information. Through linear probing, robustness tests with increasing object counts, and conjunctive search experiments, we show that at- trijute-object bindings are already encoded within CLIP's text and image em- beddings. The weakness lies in the cross-modal alignment, which fails to pre- serve this information. We show it can be accessed cross- moos-mmdally with sim- ple linear transformation to text embeddings. This improves CLiP's attribute- object binding performance and confirms that the information was already en- coded unimodally. In practice, this means CLiP-based systems can be enhanced with lightweight linear layer trained on existing embeddings. avoiding costly encoder retraining. The code is available at https: /tpp:://githbu com/ kdariina/ CL.PP-not wow-anww-----oo-eeeeeenne undmoda 11y
INTRODUCTION
Vision-language models (VLMs) like Contrastive Language-Imag Pretraining (CLIP) (Radford et al. (2021) have achieved widespread adoption due to their shared embedding space for text and image modalities, enabling strong performance on downstream tasks. However, fundamental lim- itation has emerged: CLIP often struggles with compositionality (Thrush et al. 2022). specifically the ability to bind attributes to corresponding objects in complex scenes ( (Tang et al. 2023] Lewis
al. 2024 Yuksekgonul et al., 2023). Compositionality is essential for VL.Ms, it allows models to generalize effectively by combining simpler concepts and understanding their relations.
Recent studies (Yuksekgonul al. 2023) show that CLIP frequently behaves like bag-of-words (BoW) model, failing to bind attributes to corresponding objects. For instance, given an image of "an orange square and blue triangle" as in Fig. [ CLIP often matches the image blue square and an orange triangle". It is often unable to distinguish the structural difference. We refer to this phenomenon as BoW ness, indicating the model's treatment of each data point as an unordered set of concepts. The BeWne significantly limits CLIP's compositional understanding. Previous research has evaluated this limitation by jointly considering the image and text embeddings. However, there has been little investigation into the source of the inability. In particular, we do not know whether the BoWness arises (I) from a lack of attribute-object binding information in the individual text and image embeddings or (2) from a mere lack of cross-modal alignment.
caption "a
Distinguishing these two cases is crucial for accurate diagnosis and effective improvement. Cur- rent evaluations typically measure binding in the cross-modal space, so poor performance could be misattributed to missing knowledge rather than misalignment. If the limitation lies in the en- coders, retraining is required; if it is in cross-modal alignment, lightweight adjustment may suf- fice. Clarifying the source can guide the design of downstream VLMs and adapters, ensuring that improvements target the true limitation and that pre-trained models are used effectively.
"Corresponding author; da Biri na51 0.:511220gmmm 0120gmai com

---

Published as a conference paper at ICLR 2026
Known: CUP is BoW cross-modally.
Our result 1: image and text embeddings are not BoW uni-modally.
Fimage
fimaae
blue square and an orange triangle*
'An orange square and a blue triangle
"A blue square and an orange triangle"
S
*
Srees
Our result 2: Modalltee can be aligned Innerrt.
square and x
orange triangle"
"An orange square and blue triangle"
froe
*An orange square v
Misaligned
① Aligned
and blue triangle"
Figure I: CLIP is not BoW uni-modally. (1) has been reported that CLIP behaves like a BoW model with weak attribute-obye -ojjcct binding. (2) We discover that embeddings of individual image and text modalities already contain the obbjit binding information; this suggests the BoWness stems from the lack of alignment across the modalities. (3) simple linear transformation of the text modality mitigates the BoWness of CLIP.
In this work, investigate the cause of CLIP's BoWness by testing whether attribute-object bind- ing is present in individual image and text embeddings. We train linear probes to extract attribute information for specific objects two-object scenes and show that attributes can be linearly sepa- rated in both modalities. This signal remains robust as the number of objects increases, especially for text embeddings. For image embeddings, a conjunctive search experiment confirms that CLIP captures feature bindings, not just BoW collection of features. Together, these results demonstrate that the embeddings already contain the right attribute-object binding.
Building on this insight, we hypothesize that CLIP's BoW-like behavior stems from superficial cross-modal misalignment. Although binding information present in each modality, the model not sufficiently encouraged to use it, leading to mismatched binding signals between text and image embeddings. To validate this, we apply a simple linear transformation A to one modality. We train A using negative samples created by permuting attribute-object pairs in text captions from image- caption datasets. Empirically, this approach yields significant gains cross-modal moddll attribute- object binding on ARO, SugarCrepe, and COCO, indicating that CLIP's encoders were already capable of representing correct bindings. This finding has a practical implication: updating existing CLIP vector databases does not require retraining of the encoders or re-extracting of features.
2 RELATED WORK
Limitations of CLIP's encoders. Numerous studies have highlighted weaknesses in both CLIP's vision and text encoders. CLIP's visual encoder tends to prioritize high-level understanding, of- ten missing finer details crucial for distinguishing objects (Tong et al., 2024b). Meanwhile, its text encoder struggles with tasks involving negations, spatial and numerical reasoning, and nuanced at- tribute distinctions (Tong et 2004a] Kamath et al.] 2023a) These limitations affect performance in downstream applications (Parashar al. 2024 Tong al. 2024b a). Several efforts aim inter- pret CLIP's 's representations to understand these limitations better (Esfandiarpoor et al. 2024 Yun al. 2023 Bhalla et 20024). In contrast, our work specifically focuses on attribute-object binding in scenarios with multiple objects, providing a targeted analysis of its compositional capabilities.
One line of work studies the modality gap, the separation between image and text embeddings. This gap is often viewed as source of misalignment. However, (chrodi et al. (2024) shows that it arises from the information imbalance between images and text, which limits alignment. Varying the modality gap can affect performance, fairness, and downstream behavior (Schrodi et al., 2024 Liang et al. 2022). We examine whether improving binding changes the modality gap.
Compositional reasoning and alignment. Compositional reasoning, critical for understanding complex scenes, has been studied extensively neural networks (Hupkes et al. 2020 Greff et al., 2020). Prior work on CLIP investigates its ability to handle novel attribute-object combinations (Abaas et al. 2024, Bao et al. 2023) and attributes its failures to weak compositional reasoning Lewis et al. 2024 T Tang al. (023). Some employ controlled setups with two objects and distinct
attributes (Lewis et al. 2024, Tang et al. (023), providing initial insights into the binding problem.
Other works specifically study attribute-object binding failures in text-to-image generation models
2

---

Published as a conference paper at ICLR 2026
Trusca al. 2024 Zarei al. 2024). In work, ask whether CLIP's embedding space inherently limits binding, and evaluate its capacity to represent binding within and across modalities.
Benchmarks for compositionality. A growing number of benchmarks evaluate compositionality in VLMs, often using hard negatives or fine-grained distractors. These include VL-CheckList (Zhao
al. 2022). CREPE (Ma et al., 2023). COLA (Ray et al. 2024). ARO (Yuksekgonul et al. 2023),
SrgerCrepe (Hsieh et a 2024), Winoground (Thrush et al. 2022). These benchmarks vary in
focus: some test fine-grained distinctions like VisMin (Awal al. 2024), others use hard posi-
tive pairs for reasoning (Kamath et al. 2024), and some target specific challenges like counting in CountBench (Paiss et al. (003)). negation NegBench (Alhamoud al. (025) or spatial reasoning in What Wp (Kamath et al., 2023b). In contrast, our work specifically focuses on attribute-object binding. Other compositional tasks, such as spatial reasoning or negation, are outside the scope of our study. Synthetic benchmarks like PUG (Bordes et al. (024) and CLEVR (Johnson et al.) 2017) provide controlled environments to test attribute-object binding with targeted scenarios often miss- ing in real-world datasets. Our work extends these efforts by contributing a synthetic, controlled dataset with greater variation, designed to evaluate attribute-object binding.
3 THE BINDING PROBLEM
Ideally, VLMs like CLIP need to capture the compositional structure of real-world scenes. A neces- sary condition for compositional understanding is the ability accurately bind attributes corre- sponding objects in scenes with multiple objects. Prior research suggests that CLIP'ss attribute-object binding is often arbitrary to the degree that the model can effectively thought of as a bag-of-word. (BoW) extractor that treats objects and attributes in image and text an unordered collection of con- cepts, completely ignoring the order and structure therein (Yuksekgonul et al. 2023).
In this work, we define Boess of vision-language model the general tendency in models to treat inputs (image or text) as unordered sets of concepts. In contrast, we say that a model has a binding ability when it can link attributes correctly to the corresponding objects.
3.1 PRELIMINARIES
A CLIP model has two encoders: fimage I - R. for images and fixt T - RR" for texts, where D is the dimensionality of the encoders. For an image x E I and text sequence x T, CLIP embeds both inputs independently into shared vision-language space. We are interested in the behavior and information content in embeddings freu(x")) and (x!1)).
We consider paired image-text dataset = ((]), of samples, where each sample consists of a text sequence x!M T and a corresponding image x; provides positive pairs, where attributes are correctly associated with objects in the text captions, matching the corresponding images. We refer to negative pairs as synthetic pairs where the text cap- tions in positive pairs are modified such that the binding is artificially broken through a permutation. For example, given positive pair with the caption "red cube and blue sphere", we create negative pair by keeping the image intact and modifying the caption to "blue cube and red sphere".
img E I. Such dataset typically
The creation and usage of negative pairs have been established strategy in the assessment of CLIP's compositional abilities, as seen in previous benchmarks (Yuksekgonul et al.) 2023 Thrush et al. 2022; Hsich et al.) 2024). Some researchers have considered incorporating the negative pairs in training or fine-tuning to equip models with improved compositionality (Yuksekgonul et al. 2023 Patel et al. 2024). Our work, likewise, employs negative pairs for assessing and improving CLIP
3.2 DATASETS
To evaluate CLIP's cross-modal and unimodal binding abilities, synthetic datasets, as shown below. Further dataset details are in the Appendix A.
mix of real-world and
While real-world benchmarks offer insights into cross-modal alignment, they too complex for systematically evaluating uni-modas attribute-ojject binding. Cross-modal binding only requires image-to-text matching, but assessing binding within single modality requires controlled set of
3

---

Published as a conference paper at ICLR 2026
Table 1: Datasets used in our evaluation. weal-world benchmarks are used for cross-modee binding, while synthetic datasets provide controlled settings for uni-mat moi-modal binding.
Dataset
Type
Description / Purpose
ARO Poksekgunul
2018
Real
Tests compositionality with relationships, attributes, and order. Comparison against fluent and sensical hard negatives.
SugarCreg (Loch et al.
X021
Real
COCO Lin al 2014
Real
Large dataset for recognition, segmentation, and captioning.
CC3M (Shama et 2018
Real
Three million web image-caption pairs for image captioning.
CLEVR Johnson
Synthetic
Simple shapes and colors; controlled environment for binding.
PUG:SPAR [Bndes et al. 2024
Synthetic
Animals with controlled attributes; limited by positional bias.
PUG:SPARE (ours)
Synthetic
Extension of PUG:SPAR with positional bias removed.
PUG:SPAR
Proposed dataset PUG:SPARE
"blue elephant and red zebra
"blue elephant and green zebra
"green zebra and blue elephant"
"yellow elephant and cyan zebra
"cyan zebra and yellow elephant
Figure 2: Examples from PUG:SPAR and PUG:SPARE. In PUG:SPAR, attributes correlate with object
positions: objects on the left are linked to "blue" or "grass" and objects on the right are "red" or "stone". Our dataset PUG:SPARE de-correlates the potential shortcut.
attributes and objects. Real-world datasets lack this control, so we use synthetic datasets with an exact set of objects and attributes, allowing targeted assessments of compositional behavior.
Note that in PUG:SPAR, attributes are tied to positions, which can potentially create shortcuts for models. PUG:SPARE removes this bias by randomizing attributes across positions (see Fig. 2
We use ARO, SugarCrepe, COCO, and CC3M in $ and CLEVR, PUG:SPAR, PUG:SPARE in
5
3.3 CLIP IS A BAG-OF-WORDS CROSS-MODALLY
In this section, we explain how previous approaches demonstrated the bag-of-words nature of CLIP. We reproduce prior results and confirm that CLIP is BoW cross-modally.
Previous approach demonstrating Bo Wness. A common approach to demonstrate CLIP's BoW- ness behavior is by comparing the embeddings of an image with permutations of its caption (Yuk- sekgonul et al. 2023). For example, given an image of an orange square and blue triangle, possible captions could be "an orange square and blue triangle" and "an orange triangle and a blue square". The task is to identify the correct caption from these options, where one has correct color-object associations and the other has swapped associations. CLIP's prediction is based on which caption embedding has a higher cosine similarity with the image embedding.
Ideally, CLIP should exhibit cross-modal binding, or an accurate association of attribute-object pairs across modalities. However, CLIP has been reported to cross- BoW, treating the concepts in inputs similarities rank input pairs, as shown below.
unordered collection. These opposing behaviors are reflected in how cosine
Cross-modally BoW:
fimnge furt("rragge square and blue triangle") Stmage
Swer ("blue square and orange triangle")
Cross-modal binding:
fingee
funt(" orange square and blue triangle") ") >
fimare
f (bblue square and orange triangle")
Replicating BoWness results. Using this approach on the datasets discussed previously, our results confirm prior findings. Specifically, when choosing between the two options (correct and permuted), we observe 0.56 accuracy on CLEVR, 0.51 on PUG:SPAR, and 0.50 on PUG:SPARE, indicating that CLIP's performance is virtually at the level of random guessing. These results strongly suggest that CLIP cannot distinguish between correct and permuted attribute-object bindings. CLIP indeed a bag-of- words model.
4

---

Published as a conference paper at ICLR 2026
(4)
(b)
Imtege
dimensional embedkking
Pradicted attiibute
Cylindar Pay
GLIP image encoder
Cube oolor
red
d-dimensional emberdding
Predicted
Cube
attribute
"Buu sylnder and red cube
CLIP text encoder
Cube color
Figure 3: Uni-modal attribute-object binding. we a linear probe per object distinguish its color within image and text modality separately. (b) linear probes establish decision boundaries in CLIP's representation space that differentiate between various attribute-object associations.
CLIP BINDS CONCEPTS UNIMODALLY
Prior works evaluating CLIP's BoW tendencies have relied on assessments that combine both text and image modalities. This approach has a key limitation: it does not separate the encoding of attribute-object binding within each modality from the cross-modal matching step. As a result, it mains unclear whether CLIP's BoW behavior stems from limitations in the embeddings themselves or from issues in cross-modal alignment. To address this, we examine the uni-modal binding, refer- ring to CLIP's ability to encode attribute-object relationships independently within each modality. By evaluating text and image embeddings separately, we aim to clarify whether each modality alone captures sufficient binding information, shedding light on the roots of CLIP's BoW behavior.
4.1 LINEAR PROBING UNI-MODAL BINDING
To evaluate if CLIP encodes binding information within each modality, we use linear probing (Alain 2016). By training linear classifiers for the information we care about on top of frozen representa- tions, assess the existence of information attribute-object binding information in each modality; we train lincar classifiers separating attributes for each object present in the image text inputs. We focus scenes with two objects, the simplest non-trivial case, where at least two binding configurations are possible.
seek in the representation. In this case, seek the
Given a dataset D {xx img > attributes the dataset. For each object O, we train two separate classifiers. for images and for text. These classifiers are trained with embeddings extracted from the respective CLIP encoders, which remain frozen throughout the training. We denote object-specific probes as image-prebe, and text text-probe,, each predicting the attribute a E A for the object o:
; we define O as the set of all objects and A as the set of all
image-probe, fimage img +
text-probe,
f 1
(1)
For example, as illustrated in Fig. B(a), given an image or text describing "blue cylinder and red cube" extract corresponding embeddings using the CLIP encoders. We then train linear clas- sifier to recognize the attribute (c.g. a = ='red") of specific object (c.g. =ccube"". This process allows us isolate and probe attribute recognition for each object individually within each modal- ity. This differs from cross-modal retrieval in benchmarks such as ARO and SugarCrepe, where one modality is queried against the other (image-to-texr disentangle whether errors arise from the embeddings themselves or from their alignment.
text-to-image). In such cases, is difficult to
We use synthetic datasets here because they provide the necessary coverage of objects and attributes to train and test probes systematically. In CLEVR and PUG:SPARE, the linear probes classify among 8 possible attribute classes. In PUG:SPAR, they classify among 4 classes. The datasets split into train, validation, and test sets such that each attribute-object binding pair is unique to a single split. This prevents reliance on memorized bindings and ensures evaluation on unseen compositions. To quantify binding information in each modality, report the test accuracy of linear probes, averaged across all objects. This corresponds to how linearly separable the attributes in CLIP's representations, illustrated in Fig. b).
To contextualize the information content in pre-trained CLIP representations, we provide accuracies of random baseline (1/lA)) and CLIP encoders fine-tuned for the attribute prediction task. They
5

---

Published as a conference paper at ICLR 2026
provide the reference points for no binding information and maximal binding information in the encoders, respectively. See Appendix A.3 for further details on linear probing.
Results. Table 2 presents the classification accuracies for the linear probes on text and image embeddings for CLEVR, PUG:SPAR, and PUG:SPARE. The probes trained on pre-trained CLIP embeddings achieve ac- curacies far beyond the random baseline for both im- age (e.g. 0.96 on image test set compared to 0.12 for CLEVR) and text (e.g. 1.00 on text vs to 0.12 for ran- dom) modalities. The accuracies are close to the max- imal information bound given by the fine-tuned CLIPP
Table 2: CLIP is not BoW uni-modally. Lin- ear probe accuracies on CLIP's image and text embeddings show that binding information is preserved within cach modality. Random and finettuned encoders provide minimal and maxi- mal reference points, respectively.
Image
Dataset
Encoder
Train
Test
Train
Test
Random
0.12
0.12
0.12
0.12
Sanity check: BoW models lack discriminative sig- nal. We confirm that a BoW representation lacks the structure necessary for binding. We train randomly ini- tialized CLIP encoders under a BoW constraint, where they recognize all attributes but do not associate them with objects. Both encoders are trained to predict the presence of each attribute using soft label cross- entropy loss.
CLEVR
CLIP
1.00
0.96
1.00
CLIP (I)
1.00
0.99
1.00
1,00
Random
0.25
0.25
0,25
0.25
PUG:SPAR
CLIP
1.00
0.99
1.00
0.99
CLIP
1.00
0.98
1.00
Random
0.12
0.12
0.12
0.12
SUG:S
CLIP
0.99
0.95
1,00
1,00
CLIP (I)
1.00
1.00
1.00
1,00
Linear probing on the resulting embeddings yields significantly original CLIP (0.66 for BoW image embeddings, 0.85 for BoW text embeddings vs. 0.96 for original CLIP embeddings). This confirms that BoW models lack the structure needed for binding. High- dimensional embeddings alone do not guarantee that binding can be recovered with a linear probe.
accuracy compared to the
Takeaway $4.1; CLIP embeddings encode attribute-object binding in a linearly separable way. This separability does not hold cross-modally, suggesting the problem lies in alignment.
4.2 UNI-MODAL BINDING WITH MORE OBJECTS
Text modality
To evaluate the robustness of CLIP's uni-modal attribute-object binding, we extend the experiments in $4.1 by varying the num-
Accuracy
1.0
0.7
ber of objects per scene. We increase the object count within
0.4
Random
the CLEVR dataset and measure the accuracy of linear classifiers
0.1
2
7
9 10
trained identify opject-specific attributes in the representations.
Num. of objects
Image modality
Fig. shows the attribute probing accuracy as function of ob- ject count for text and image modalities. The text modality main- tains high accuracy, consistently above 0.8 across all counts, indi- cating stable uni-modal binding. Image embeddings decline grad- ually, from about 0.9 with two objects to around 0.6 with many objects. Still, performance stays well above chance, showing that binding signals persist even in cluttered scenes. The discrepancy in performance for the two modalities may stem from how they repre- sent objects. In text, objects are expressed through separate tokens, which keep them distinct. In images, pixels from different objects overlap, so the encoder must disentangle them, making binding harder as scenes grow crowded.
Accuracy
1.0
Train
0.7
0.4
Test
Random
0.1
2
10
Num. objects
Figure 4: Image and text em- beddings encode multiple ob- jects. Average lincar probing ac- curacy on CLEVR the number of objects increases.
Takeaway 44. CLIP's text embeddings preserve binding even with many objects. Image em- beddings, while challenged by complexity, still retain binding signals above chance.
4.3 CONJUNCTIVE SEARCH
In this section, we show that CLIP's visual embeddings encode binding using a visual search exper- iment. This provides further evidence that CLIP is not BoW model within the image modality.
We adapt the conjunctive search experiment from Campbell et al.] (2024), originally designed to study binding in multimodal language models. Unlike their cross-moda. evaluation, we focus on

---

Published as a conference paper at ICLR 2026
CLIP's visual encoder to examine its uni-modal binding. Notably, their experiments LLaVA-1.5 (Luu et al., 2023b), which uses a pre- trained CLIP ViT-L/14 as its image encoder, showed poor per- formance. Again, this raises the question of whether the limitation stems from the visual embeddings or the interaction between the two modalities.
Method. The conjunctive search task requires identifying target object among distractors, where the target shares one feature (e.g., color or shape) with each distractor type. As illustrated in Fig. an image contains many green spheres and red cubes. The task is to determine whether red sphere is present. The red sphere shares color with the red cubes and shape with the green spheres. It has no distinguishing features from other objects except for the unique binding of the features. Half of the images include the red sphere (incongruent case), while the other half do not (congruent case).
To test CLIP, we trained binary linear classifiers on its visual embeddings to predict whether an image contained an incongruent object. Performance is evaluated on test set as the number of distractor objects increases. We also conducted zero-shot classification using captions and included randomly initialized CLIP model for comparison. More details are in Appendix A.4
Results. The results (Fig. 5 right) show that pre-trained CLIP embeddings enable accurate binary classification of incongru- ent objects, regardless of cluttered scenes (accuracy exceeds 0.80 for up to 35 objects in both settings). In contrast, zero-shot classification and the randomly initialized model stay the random baseline level, showing no meaningful signal for binding.
Congruent
Incongruent
Proting Zaro-shnt
Randem Probing
Random Zera-shot
1.0
Accuraty
0.9
Lettere
0.8
20
1.0
Accurayy
CLEVA
0.0
These results suggest that, while CLIP fails to align captions to images in zero- shot settings, its visual embeddings are not purely BoW. If they were, the em- beddings would reflect frequent concepts ("green", "sphere", "red", and "cube") without significant change when a red sphere is added. However, the linear clas- sifier's success in distinguishing incongruent cases indicates that CLIP's embeddings encode nu- anced binding between attribute "red" and object "sphere", enabling the recognition.
10 20 4045
Num. of objects
Figure CLIP embeddings identify objects with shared features but unique bindings. Left: Examples from the let- ters and CLEVR settings (Campbell et al. (024). Incongru- ent examples contain an object sharing feature (color shape) with others but with different binding. Congruent amples lack this extra object. Right: Linear and zero-shot classification accuracies on congruent and incongruent cases.
CLIP's visual embeddings support conjunctive search, allowing detection of the object defned only by a unique attribute-object binding, even in cluttered scenes.
Takeaway 443
We conclude that CLIP is already aware of attribute-object bindings in individual modalities. We hypothesize that CLIP's BoWness stems from the poor cross-moda alignment that takes place after the encoding. We verify the hypothesis in the next section.
5 IMPROVING CROSS-MODAN BINDING
We observed that CLIP is not bag-of-words (BoW) model uni-modally. We thus narrow down the root cause of the previously observed cross-modal BoWness to a poor cross-modal alignment in the representation space. In this section, verify this by proposing simple alignment strategy that recovers the biribbute-ojecttt binding information across the modalities. Specifically, we apply a linear transformation one modality's 's embeddings (e.g., text) ensure cosine similarities retrieve pairs with correct binding first. We refer to this method as LABCLIP.
5.1 METHOD
Our linear probing experiments revealed that both vision and text spaces encode attribute-object in- formation in linearly separable manner. This suggests that each modality organizes information in a way that could simplify alignment. If the internal structures of the two spaces are linearly sep- arable, linear transformation could map these to shared alignment, improving cross-modal cor-
7

---

Published as a conference paper at ICLR 2026
respondence. CLIP's contrastive loss encourages global alignment but does not focus aligning attribute-object binding. The original alignment mechanism does not fully use the binding informa- tion, which is linearly accessible within each modality. To address this, we propose learning linear transformation between the text and image spaces using pre-trained CLIP embeddings.
Linear Attribute Binding CLIP (LABCLIP) trains linear transformation to better align the text and image embeddings. Instead of the standard image-text matching in CLIP.
(fimage( ""),,, fieat )x,
LABCLIP applies a matrix E DxD on the text embeddings before the inner product:
(3) Applying the transformation to text or image embeddings is mathematically equivalent. We apply it to text embeddings for practical reasons. Keeping the visual encoder unchanged preserves its utility in downstream tasks, where the original CL.IP image embeddings are often used. Additionally, negative texts are casier to obtain than negative images. Experiments aligning the image space confirm that the results remain consistent (see Table in the Appendix).
(Fimage (x!m8), A fiex (x"
Training. We train the linear transformation matrix contrastively, initialized from the identity matrix, with no further constraints. CLIP's weights remain frozen. We include negative text samples within the batch, similar to YYuksekgonul et al. 2023). Negative samples are created by permuting the concepts in the original captions: we transform "red cube and blue sphere" into "blue eube and red sphere" without changing the image. Such samples can be generated without extra annotation structured in straightforward "attributeohjeec"
costs. The captions in COCO and CC3M format, making it challenging to isolate and swap attributes and objects directly. Because of this, we use the NegCLIP strategy from (Yuksekgonul et al.) 2023), which shuffles nouns and adjectives to create negative samples. In training, including negative text samples results in a B x 2B batch, where the transformation is used minimize the similarity with mismatched attribute-object object pairs.
See Appendix A.5 for the details of the experimental setup and Appendix f.6 additional analyses of the effect of LABCLIP on the CLIP's s shared embedding space.
LABCLIP does not update CLIP's parameters. It intends to leverage existing binding signals and aligns them through linear transformation. This approach is efficient to train and backward compatible with existing CLIP-based vector database systems.
Takeaway $5.
5.2 RESULTS
Table 3 provides the cross-modal bind- ing results on CLEVR, PUG:SPAR, and PUG:SPARE. The first column is the ac- curacy a model in matching an image to the correct caption the permuted cap- tion. Recall@1 measures the model's abil- ity to retrieve the correct caption from all possible captions in the dataset. For ref- erence point, we also provide results with a fine-tuned CLIP model (CLIP-FT).
Table LABCLIP recovers cross-modal binding. We measure the ability to rank attribute-ojeed pairs correctly. "Frozen?" indicates whether CL.IP's encoders are up- dated. "#Params" reports the number of learnable parame- ters. While baseline CLIP performs near random, LABCLIP shows near upper-bound performance of fine-tuned CLIP.
Accuracy
Recall@
Dataset
Model
Frozen?
Parrams
Train
Test
Random
0.50
0.01
0.01
CLEVR
LABCLIP CLIP-FT
CLIP
589.8K
0.49
1.00
0.58
0.95
0.25
1.00
0.36
0.93
428.2M
1.00
0.99
0.97
Without training, the original CLIP results are at the random chance level, as dis- cussed in Section 3.3 CLIP cannot differ- entiate between correct and permuted cap- tions cross-modally. Fine-tuning with neg- ative samples enables near-perfect scores, providing an upper bound on the possible attribute-object binding. We observe that LABCLIP improves performance compared to CLIP. On CLEVR, for example, it achieves an accuracy of 0.95, significantly greater than CLIP's 0.58.
Random
0.50
0.50
0.00
0.00
PUG:SPAR
LABCLIP
CLIIP
589.8K 8K
0.52
1.00
0.97
0.08
0.98
0.09
0.91
CLIP-FT
428.2M
1.00
1.00
1.00
0.99
Random
0.50
0.50
0.00
0.00
PUG:SPARE
LABCLIP
CLIP
V
589.8K
0.98
0.94
0.06
0.95
0.90
0.06
CLIP-FT
428.2M
1.00
1.00
1.00
1.00
Note that our CLEVR results differ from the two-object experiment in Lewis et al. (2024). Their evaluation tests both binding and compositional generalization, while we focus only on attribute- object binding. See Appendix
A.5 A.5 for details the differences.
8

---

Published as a conference paper at ICLR 2026
Table 4: LABCLIP enhances compositional reasoning on real-world benchmarks. We compare the perfor- mance to baseline CLIP and fine-tuned CLIP with negative examples (NegCLIP). CLIP and NegCLIP update all parameters of the VIT-B/32 backbone (15IM learnable parameters). LABCLIP adds only lightweight 512 x 512 linear layer (262K learnable parameters) on top of the frozen CLIP encoders.
ARO
SugurCrepe
cocO
Model
Dataset
Frozen?
VG-A
VG-R
Flickr PRC
COCO
Add
Replace
Swap
Recall@
CLIP
0.63
0.63
0.60
0.48
NegCLIP
COCO
0.71
0.81
0.91
0.86
0.87
0.85
0.75
0.41
LABCLIP COCO
0.69
0.82
0.84
0.81
0.81
0.82
0.74
0.41
LARCLIP
CC3M
0.68
0.78
0.83
0.73
0.83
0.80
0.68
0.34
Table shows the results on the real-world benchmarks, ARO (Yuksekgonul et al. 2023) and Crepe (Hsich et al. 2024), when trained on COCO (Lin al. (2014) and CC3M (Sharma et al. 2018). LABCLIP significantly outperforms the standard CLIP, indicating better under- standing of attributes, relations, and word order. LABCLIP trained on CC3M performs slightly below the COCO-trained version on some compositional benchmarks, which is expected given the closer match between COCO and the ARO and SugarCrepe datasets (Oh et al., (024). Importantly, LABCLJP-CC3M still shows strong gains over CLIP, suggesting that the performance improvements stem from improved object-attribute binding, not from dataset similarity.
We present these results alongside NegCLIP, fine-tuned CLIP model trained with negative im- age-text pairs from COCO (Yuksekgonul et al. 2023). NegCLIP achieves strong results, but LAB- CLIP is not intended to outperform demonstrate that since CLIP embeddings are not BoW uni-modally and samples with permuted bindings are linearly separable, this structure can be leveraged through linear transformation.
other state-of-the-an methods. Instead, LABCLIP's results
Despite its simplicity, LABCLIP matches NegCLIP on compositional benchmarks, while offering practical benefits. Rather than retraining vision text encoders to improve binding and re-extracting all features, LABCLIP only requires training a lightweight linear layer on top of existing text em- beddings. This means that it can work directly with existing CLIP vector databases. This offers backward compatibility with deployed CLIP systems, post hoc modularity without altering the pre- training pipeline, and far greater efficiency (training over 100x faster than NegCLIP on CLEVR).
We also test LABCLIP on spatial reasoning dataset (Appendix A.5). LABCLIP matches CLIP's spatial performance and improves when trained with spatial data.
Downstream performance. We report downstream results in Table I1 in the Appendix. LABCLIP performs slightly worse than CLIP on single-object classification tasks (CIFAR, ImageNet), sug- gesting tradeoff between coarse object-level signals and attribute-ojeect binding. One possible explanation that binding supervision shapes the linear transformation toward a binding-speccfic structure rather than broad object discrimination.
Our results suggest that better cross-modal binding can be achieved by linearly transforming one of the embedding spaces without requiring extensive computations, complex methodologies, any change to CLIP parameters. This further corroborates our previous findings that all the ingredients and information for attribute-object binding are already present in the pre-trained CLIP models.
Takeaway $5.2: simple linear transformation recovers available binding information, thereby improving cross-modal binding both on synthetic and real-world benchmarks.
5.3 THE EFFECT OF ALIGNMENT ON LINEAR PROBING
Training CLIP enables the image and text encoders to learn attribute-ohject binding within their own modalities, as shown in our uni-modal binding experiments. However, this binding is not aligned across modalities. According to (Yuksekgonul et al. 2023), a bag-of-words strategy suffices for CLIP's high performance. Moreover, the VLMs tend to be biased towards objects rather than attributes because objects appear more often in captions (Schrodi et al. 2024). We hypothesize that because CLIP's training objective does not depend on attribute-ojeect binding, there is no incentive to align binding signals across modalities. While the encoders learn attribute-obeect binding, they fail to align these signals effectively.

---

Published as a conference paper at ICLR 2026
Method. To investigate this misalignment, we compare the coefficients of linear probes for image and text embeddings before and after alignment. We first compute image embeddings fimage (x""*) and text embeddings fieu(x (x**). Then, we obtain aligned text embeddings A (x) by multiplying the alignment matrix with the text embeddings. We normalize all the embeddings and learn object- specific probes as described in $4.1, We define Wimg, Wxt. W Waligned-txt E vectors of object-specifie linear probe coefficients for image embeddings, original text embeddings, and aligned text embeddings, respectively. We then compute the probe similarities:
as concatenated
(Wing:
Wid)
(Wings Waligned-oo)
cos- S in Wing, Wxx)
cos-simi sim( W ing: Waligned-oxt)
Wing
Weall
|| Wing
W Waligned-axi
Results. The results are illustrated in Fig. 6 Before align- ment, the cosine similarity of probe weights low (0.20, 0.18, and 0.09 for CLEVR, PUG:SPAR, and PUG:SPARE, respec- tively), indicating differing encoding of binding. After align- ing text embeddings with image embeddings, the similarities increase significantly to 0.75, 0.78, and 0.65. This increase confirms that the alignment process effectively adjusts the text embeddings to match the attribute-object binding encoded in the image embeddings, whereas CLIP's original alignment mechanism does not naturally support cross modal binding.
Similarity
1.0
0.75
0.78
0.65
0.5
0.20
0.18
0.09
CLEVR PUG:SPAR PUG:SPARE
Before
After alignment Alignment enhances
Figure 6:
similarity between image and text probes. Cosine similarities probe coefficients before and after alignment show an increase, confirming align- ment of hinding across modalities.
Takeaway $5.3 The simple linear transformation on text embeddings aligns the attribute-object binding signals the image and text modalities.
6 CONCLUSION
In this study, we investigated the reasons for CLIP's bag-of-words behavior, focusing on at- tribute-object binding. We showed that binding information is already present CLIP's text and image embeddings: it is linearly separable, remains robust the number of objects increases, and supports conjunctive search. The poor performance, therefore, stems not from missing information but from misalignment. To validate this, introduced LABCLIP, simple linear transformation applied to text embeddings. LABCLIP recovers the unimodal binding signals during cross-moda matching, enhancing compositional understanding. It also offers practical advantages: requires no changes to CLIP encoders, making it efficient to train, modular as a post hoc method, and backward compatible with existing systems. Our work motivates further exploration into the properties of CLIP embeddings uni uni-modally and into alignment strategies that enhance compositional reasoning.
7 ACKNOWLEDGMENTS
We thank the anonymous reviewers for their helpful feedback. This work was supported by the Tübingen AI Center. Arnas Uselis was supported by the International Max Planck Research School for Intelligent Systems (IMPRS-IS).
REFERENCES
Reza Abbasi, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah. Deciphering the role of representation disentanglement. Investigating compositional generalization in clip models. arXiv preprint arXiv:2407.05897, 2024.
Guillaume Alain. Understanding intermediate layers using linear classifier probes. arXiv preprint
1rXiv:1610.01644, 2016.
Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip HS Torr, Yoon Kim, and Marzych Ghassemi. Vision-language models do not understand negation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29612-29622, 2025.
Rabiul Awal, Saba Ahmadi, Le Zhang, and Aishwarya Agrawal. Vismin: Visual minimal-chang understanding. arXiv preprint arXiv:2407.16772 2024.
10

---

Published as a conference paper at ICLR 2026
Wentao Bao, Lichang Chen, Heng Huang. and Yu Kong. Prompting language informed distribution for compositional zero-shot learning. arXiv preprint arXiv:2305.14422 2023.
language-informed
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P Calmon, and Himabindu Lakkaraju. Inter- preting clip with sparse linear concept embeddings (splice). arXiv preprint arXiv:2402.10376, 2024.
Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari Mor- Cos. Pug: Photorealistic and semantically controllable synthetic data for representation learning. NIPS, 36, 2024.
Declan Campbell, Sunayana Rane, Tyler Giallanza, Nicolò De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven M Frankland, Thomas I. Griffiths, Jonathan Cohen, et al. Understanding the limits of vision language models through the lens of the binding problem. arXiv preprint arXiv:2411.00238, 2024.
Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL URLhttpp: http:/t///// www www.bleeeeerrr..e.
Reza Esfandiarpoor, Cristina Menghini, and Stephen H Bach. If clip could talk: Understand- ing vision-language model representations through their preferred concept descriptions. arXiv preprint arXiv:2403. 2024.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208. 2020.
Cheng-Yr Yu Hsich, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarerepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural informa- fion processing systems, 36, 2024.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pp. 2901-2910, 2017.
Amita Kamath, Jack Hessel, and Lai-Wei Chang. Text encoders bottleneck compositionality in con- trastive vision- vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4933-4944, 2023a.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? inves- tigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9161-9175, 2023b.
Amita Kamath, Cheng- Yu Hsieh, Kai-Wei Chang, and Ranjay Krishna. The hard positive truth about lision-language compositionality. In ECCV, 2024.
Martha Lewis, Nihal Nayak, Peilin Yu, Jack Merullo, Qinan Yu, Stephen Bach, and Ellie Pavlick.
Does clip bind concepts? probing compositionality in large image models. In Findings of the Association for Computational Linguistics: EACL 2024, pp. 1487-1500, 2024.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems, 15:17712-1762 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Associ- ation for Computational Linguistics, 11:635-651, 2023a.

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b.
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe:
Can vision-language foundation models reason compositionally? In CVPR, pp. 10910-10921, 2023.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint /002. 2018.
Youngtack Oh, Jae Won Cho. Dong-Jin Jin Kim, In So Kweon, and Junmo Kim. Preserving multi- modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19060-190787 Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: (12.665//1/222.mmmp..... 1062. URL htwp:////o..caacoooooooooooo
emnlp main. 1062/
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran 7ada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3170-3180, 2023.
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caver- lee, and Shu Kong. The neglected tails laionnlanguage models. In CVPR, pp. 12988-12997 2024.
Maitreya Patel, Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, and Yezhou Yang. Tripletclip: Improving compositional reasoning of clip via synthetic vision- language negatives. arXiv preprint PrXiv:2411.02544 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763 PMLR, 2021.
Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan Plummer, Ranjay Krishna, and Kate Saenko. Cola: A benchmark for compositional text-to-image retrieval. NIPS, 36, 2024.
Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision- language representation learning. arXiv preprint arXiv:2404.07983, 2024.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2556-2565, 2018
Yingtian Tang, Yutaro Yamada, Yoyo Zhang, and llker Yildirim. When are lemons purple? the concept association bias of rision-language models. In Proceedings of the 2023 Conference Empirical Methods Natural Language Processing, pp. 14333-14349, 2023.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 5238-5244 2022.
Shengbang Tong, Erik Jones, and Jacob Steinhardt. Mass failures of multimodal systems with language models. NIPS, 36, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xic. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, pp. 9578-, 2024b.
Maria Mihaela Trusca, Wolf Nuyts, Jonathan Thomm, Robert Honig, Thomas Hofmann, Tinne Tuytelaars, and Mrric-Francine Moens. Object-attribute binding in text-to-image generation: Evaluation and control. arXiv preprint arXiv:2404 13766, 2024.
12

---

Published as a conference paper at ICLR 2026
Arnas Uselis, Andrea Dittadi, and Seong Joon Oh. Does data scaling lead to visual compositional generalization? In Forty-second International Conference on Machine Learning. 2025. URL
httpps://epeneevee wet.eewwweeeeeeeeee
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In ICLR, 2023.
Tian Yun, Usha Bhalla, Ellie Pavlick, and Chen Sun. Do vision-language pretrained models learn composable primitive concepts? Transactions on Machine Learning Research, 2023.
Arman Zarei, Keivan Rezaci, Samyadeep Basu, Mehrdad Saberi, Mazda Moayeri, Priyatham Kat- takinda, and Soheil Feizi. Improving compositional attribute binding in text-to-image generative models via enhanced text embeddings. arXiv preprint arXiv:2406.07844, 2024.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations, arXiv preprint arXiv:2207.00221, 2022.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DATASETS
In this section, provide details for the datasets introduced in Section 3.2 A summary of the key dataset characteristics for the CLEVR, PUG:SPAR, and PUG:SPARE is presented in Table
Table 5: Specifications for the datasets used to test attribute-object binding in controlled setting. For CLEVR, the number of attribute-object combinations only reflects the two-object case. For PUG:SPAR, the numbers represent the filtered dataset used in our experiments.
CLEVR
PUG:SPAR
PUG:SPARE
#images #attribute-ojects combinations
5000
19840
88704
192
1984
3696
#objects
3
32
12
#attributes
8
#backgrounds
1
10
positions
random
(left, right)
fleft, right) x ffront, back, equal)
A.1.1 CLEVR
Generation.
Following the CLEVR dataset introduced in Johnson et al. (201)) we generate
new images using the 3D modeling software Blender Community (2018). The dataset tains images with M colored objects and corresponding captions. The set objects is = {cube, sphere, cylinder). and the attributes are selected from the set of eight colors: A (blue, red, purple, cyan, gray, brown, green, yellow).
N
Objects
Attributes
(blus, brown. cyen,
random combination objects
(gray cube, green sphere, cylinder)
Random scene constructor
X
ing
x
"gray cube and green sphere cylinder
Figure 7: CLEVR dataset generation process. From of objects and attributes, randomly generated combinations of objects per scene. Each combination was rendered with Blender, caption was gener- ated by concatenating attributes and objects to match the image.
The data generation process is shown in Fig. 7 We randomly sample objects and attributes to create combinations of M objects of various attributes. For example, attribute-object combination of objects is blue cubes, green sphere, purple sphere, and blue cylinder. For scenes with two objects, enforce that the objects
distinct. This results in 192 unique combinations of two objects: 3 choices for the first object, 2 for the second, 8 color options per object (colors can repeat), divided by ignore object order: 3 8 0.5 192
Using the original CLEVR image generation pipeline, we construct a scene in Blender based on the sampled combinations. Objects are placed randomly on neutral background with variations in material and size. The rendered images have dimensions of 320 x 240. For captions, we concatenate attributes and objects in the format: "aj O₁ and a2 02 and ... and am oM" where aj E and 0j O for j E 1...,..
We generate = 5000 samples for each M -object configuration. For the experiments in Sections and
M 2. For testing uni-modal binding with an increasing number of objects in Section WC consider M ranging from 2 to 10. Sample images and captions are shown Fig.
Train/test split. For each M-object setting, we divide the dataset into training, validation, and test sets in a 90/10/10 ratio based on attribute-object combinations. For instance, if there are 192 attribute-object combinations for the two-object setting, 19 combinations are assigned to the valida- tion set, 19 to the test set, and the remaining combinations to the training set. This ensures that the same combination does appear in both the training and test sets.
14

---

Published as a conference paper at ICLR 2026
M 3
M =
M =
M 6
"green sphere and gray cube and blue eylinder"
"red cube green cube and purple cylinder and brown sphere"
"cyan sphere green sphere and gray cube and brown cube and yellow cylinder"
"gray and purple cube and green sphere and yellow cylinder and blue cylinder and blue cylinder"
M
M 10
"elllo cylinder and brown cube and brown cube and blue cube and oyan cube and yellow cube red sphere*
"blue cube and blue cylinder and gray sylinder and brown cube and purple sphere and green sphere and green cylinder and yellow cube"
"red cylinder and red cube and yellow cube and purple cube and blue sphere and purple oylinder brown sphere and purple cube and purple cylinder"
"gray sphere and yellow cube and green cube and gray sphere and brown cylinder and green cylinder and gray cube and cyan cylinder and green cylinder and gray sphere"
Figure 8: Scene complexity increases with more objects. We show examples from the CLEVR with seenes containing different numbers of objects to highlight how image and text complexity changes with object count.
A.1.2 PUG:SPAR
Description. PUG:SPAR is a synthetically generated dataset Unreal Engine Bordes et al. (2024) featuring animal figures various backgrounds. The animals can have their natural colors at- tributes such as red, blue, grass, and stone. For our project, which tests attribute-object relationships in controlled environment,
filter the dataset include scenes with two animals and annotated attributes. This leaves us with images of two objects either in a blue/red or grass/stone attribute setting, with animal the left and the other the right. However, the attributes positions: the left objects are always blue or grass, and the right objects are always red or stone. 1984 attribute-object combinations and total of 19840 images.
fixed to
This results 32 31
As discussed in Section 3.2 the fixed relationship between attributes and positions could lead to shorteut strategy for the linear classifier: first identifying the blue/red or grass/stone setting, and then determining if the target object is on the left or right. To address this, we create PUG:SPARE, dataset with an extended set of attributes that are independent of object positions.
Train/test split. Similar to the CLEVR dataset, we split the dataset into training, validations, and test sets based on attribute-object combinations 90/10/10 ratio.
A.1.3 PUG:SPARE
Generation. Similar PUG:SPAR, we generate photorealistic images of animals on differ- ent backgrounds. Our dataset includes 12 possible animal objects and 8 possible colors for these animals. The objects appear in different environments, creating varied backgrounds and lighting conditions. The relative positions between the two objects also change: the left object in front and the right object in the back, the left object the back and the right object in front, and both objects at the same distance. Animals and attributes do not repeat. For example, "red zebra and blue lion" valid. However, "red zebra and red lion" or "red zebra and blue zebra" are not. We generate all possible two-object combinations with these conditions. This results in 12 choices for the first object, choices for the second object, 8 colors for the first object, 7 colors for the second object, divided by ignore the order: 12 8 x 0.5 3696 combinations.
15

---

Published as a conference paper at ICLR 2026
"red crow and blue
"green dear stag and yellow deer doe
"brown wolf and purple crocodile
"cyan alephant and gray hippopotamus
"blue lion and yellow lloness
"brown rhinoceros and cyan zebra'
fox
"red crow
"green and deer
crocodile purple walf"
"cyan gray elephant"
"olue loness and yellow lion"
and
cyan thinaceros"
Figure 9: PUG:SPARE dataset examples. PUG:SPARE offers all possible configurations of two objects from the set of objects, attributes, 4 backgrounds, and 3 position configurations (front/back, back/front, the same distance). This allows comprehensive testing attribute-ohject binding.
ARO
SugarCrepe
VG-Attribution
Add
elderly woman girf 'the litle women and the elderty gir!
noodles!
ta plate filled with broccoli, noodles, and carrots"'
VG-Relation
Replace
'the cup the right of the plate" 'the plate the right of the cup'
'a zebra standing field" ta zeirra is running across an open field'
COCO-PRCFFlikk30kk.PP
Swap
'a man with a red helmet small moped on a read 'man helmet with emall moped dirt road on' 'wtt man a red helmet small on moped on a road dirt' 'moped on email a man with red helmet road"
moped
train with yollow running 'a yellow train with while front running on raile*
Figure 10: Examples from compositional benchmarks ARO and SugarCrepe. These datasets are
signed to test VLMs' ability to accurately bind attributes and objects in complex, real world scenarios. The images illustrate varied compositions of objects, attributes, and their relationships, challenging model's com- positional understanding.
The images for these combinations are rendered in Unreal Engine. The image dimensions are 512 x 512. Captions follow the form "aj 0j and 02 02", where aj A and o, E O. The examples are shown in Fig.
Train/test split. Similar to the CLEVR dataset, we divide the dataset into train, validation, and test splits based on the attribute-object combinations.
A.1.4 COCO, ARO, SUGARCREPE
We evaluate the cross-modal binding performance of method solld datasets by training on COCO Lin et al. (2014). Following the protocol in Yuksekgonul et al. (2023). we apply the Karpathy splits to divide the dataset into train, validation, and test sets.
The trained models are then assessed on compositional benchmarks, ARO Yuksekgonul et al. (2023
and SugarCrepe Hsieh al. (2024). Examples from these benchmarks are shown in Fig. 10
16

---

Published as a conference paper at ICLR 2026
(a) CLEVR
CLEVR
(c) PUG:SPAR
PUG:SPARE
yellow sphere' yellow cube'
yellow sphere and red cube' 'red sphere and yellow cube'
"blue elephant and red lion' 'red elephant and blue lion'
"blue elephant and red lion' 'red elephant and blue lion'
Figure 11: Illustrating BoWness. The figure presents examples from CLEVR, PUG:SPAR, and PUG:SPARE datasets, used demonstrate BoWness. The highlighted example red shows case where an image com- pared to captions describing only single object, which can lead to inaccurate assessments incomplete scene information.
A.2 DISCUSSION ON ESTABLISHING BOWNESS
Previous studies evaluating CLIP's Bag-of-Word. (BoW) behavior have typically combined both text and image modalities, following the standard zero-shot learning approach. These assessments generally fall into two categories. The first approach, as demonstrated Lewis et al. (2024): [Tang al. (023), involves comparing images to captions that describe only single object the scene. For example, given an image with yellow sphere and a red cube, the text prompts might be: "a photo of (yellow sphere, yellow cube, red sphere, blue cube, purple cylinder)' (Fig. a)). We argue that focusing on single-object descriptions does not fully capture CLIP's capability for compositional reasoning in multi-object contexts. Since CLIP is trained to match an image the caption that best aligns with its content, 'yellow cube' would be better match than 'yellow sphere' in this example, as it provides more accurate representation of the concepts the scene. Limiting the evaluation to single-object descriptions, therefore, does not fairly test its ability to understand attribute-objed associations.
The second approach, seen in studies like Yuksekgonul et al. (2023), ) provides more robust evalua- tion by comparing query image to permutations more complete description, effectively testing BoWness. We adopt this methodology for experiments in Section 3.3 Specifically, measure CLIP's accuracy in choosing between correct and permuted captions. For CLEVR, the correct caption is 'yellow sphere and red cube', compare it to its permutation, 'red sphere and yellow cube' (Fig. (T))... Similarly, for the PUG:SPAR and PUG:SPARE datasets, we test the model's 's abil- ity to distinguish between captions like 'blue elephant and red lion' versus 'red elephant and blue lion' (Fig. ())d)).
A.3 DETAILS FOR UNDAL BINDING
In this section, we provide details about the experi- ments conducted in Section
Table 6: CLIP is not BoW uni-modally. The table shows accuracies of linear probes classifying attributes for each target object, averaged across objects. Results extend Table 2 with ViT-B/32 and ViT-B/16 backhones.
Since each linear probe is object-specific, we filter the dataset to include only examples containing the target object. These examples are split into training, validation, and test sets with 90/10/10 ratio based on the attribute-object combinations that include the target object.
Image
Text
Datnset
Facoder
Train
Test
Train
Test
CLEVR
VIT-B/32
1.0
0.91
0.99
VIT-B/16
1.0
0.90
0.99
We use OpenAl's CLIP model for all experiments Radford et al. (2021). The main results presented in Table 2 are based on the ViT-L/14 model. while additional results with the ViT-B/32 and ViT-B/16 models are shown T During linear probing,
PUG:SPAR
B/3-
1.0
0.97
1.0
0.98
VIT.R/16
PUG:SPARE
VIT-B/32
0.96
0.90
0.99
VIT-B/16
0.96
0.99
the model weights frozen. For upper-bound results derived with fine-tuning, unfreeze the
17

---

Published as a conference paper at ICLR 2026
Table The results LABCLIP-SB real-world benchmarks. LABCLIP-S does contain hard negative samples in batches. Its performance is lower than LABCLIP-HIE DHNBBPP-BBBBBB on ARO and similar on average for SugarCrepe and COCO retrieval.
ARO
SugarCrepe
COCO
Model
Backbone
VG-A
VG-R
Flickr PRC
COCO PRC
Add
Replace
Swap
Recall
LABCLIP-SE SB LABCLIP-SH
VIT-B/3
0.64
0.59
0.32
0.83
0.69
0.41
VIT-B/16
0.60
0.57
0.41
0.32
0.84
0.84
0.67
0.44
LABCLIP SB
VIT-L/14
0.62
0.50
0.44
0.31
0.85
0.84
0.64
0.46
relevant image or text encoder weights and allow them to update during training. All models are trained on a single Nvidia A100 GPU.
We do not normalize image or text embeddings before passing them the linear classifiers. The linear classifiers are implemented in Py PyTorch using cross-entrope loss. Accuracy is measured by predicting attributes and averaging the results across all object-specific classifiers. In the two-object case, each object has only one possible color, and the task is to predict single color per object. In the multi-object case, where multiple instances of the target object may appear, prediction considered correct only if all colors for the given target object are accurate.
We use both manual and random searches for hyperparameter tuning. A batch size of 32 consistently performs well across all datasets. For CLEVR, linear probing without fine-tuning requires learning rates of 10.1, 0.01, 0.001} and training for 1000 to 5000 epochs for images or 200 to 1000 epochs for text. For fine-tuning, we maintain a batch size of 32, with a learning rate of 0.1, and train for 50 to 200 epochs when not fine-tuning. For fine-tuning, the learning rates range between 0.001 and 0.01, with 5 50 epochs. Both training regimes utilize the SGD optimizer. The optimal configurations are selected based on validation set accuracy.
reduce the number of epochs to range of 5 to 20. For the PUG datasets,
Details on training BoW model. We simulate a BoW model by training CLIP encoders to recog- nize all attributes in the input while ignoring binding objects.
We attach a linear layer to CLIP's image or text encoder that maps to attribute classes, similar to linear probes. We then reinitialize the CLIP encoders randomly and train the model to predict all attributes in the input with soft label cross-entropy loss. The soft labels correspond to the normalized count of attributes the input. This ensures that the model behaves as BoW because is tasked to predict attributes without having to link them to specific objects. We use these newly trained CLIP embeddings and apply linear probing to evaluate the presence attribute-ojeect binding informa- tion.
As explained in the main text, such BoW model does not achieve high linear probing accuracy. On CLEVR, the average test accuracies of the linear probes are 0.66 for images and 0.85 for text, signif- icantly worse than the probing performance on the actual CLIP embeddings (0.96). This reinforces the idea that BoW models do not contain features that useful for binding.
A.4 CONJUNCTIVE SEARCH DETAILS
We trained separate linear classifiers on CLIP's visual embeddings for various object cases (5, 10, 15, 50 objects). The visual embeddings were extracted using the ViT-L/14 encoder. The classifiers predicted whether an image contained red sphere (or red "L" in the letters setting). Training was performed on 800 images per case, with evaluation conducted zero-shot classification with captions. For the incongruent case, the captions were of the form "This image contains red sphere and green spheres and red cubes".. For the congruent case, the caption was "This image contains green spheres and red cubes".
test set of 100 images. In addition, we
A.5 CROSS-MODAL BINDING DETAILS
In this section, we provide additional details about LABCLIP, discussed in Section process of LABCLIP is illustrated in Algorithm
The training
18

---

Published as a conference paper at ICLR 2026
Algorithm Training algorithm for Lincar Attribute Binding CLIP (LABCLIP)
1: Initialize transformation matrix A R PxD
Precompute i, S/imugo (x: m))) and L; (x!"(x for (CLIP encoders frozen)
3:
for epoch = to Nepucts do
4:
for each batch {ii, t,) do tr, At:
6:
Compute positive scores 8i,i = (it, ti,pa))
7:
if negative sampling then
8:
Generate negatives: tj A fiad (permuttt))(
Compute negative scores St, (I., (g,0o)) for 7
10:
Effective batch size: B x 2B
11:
Compute Lingssn-ax CB((C (s,,): {885,))
Compute List-w-ime (E((s {8ja))
13:
L Limg-so-txt + Las-to-ing Update A to minimize L
14:
15:
Return Iearned transformation matrix A
Table 8: LABCLIP enhances cross-modal binding. These results extend the findings shown Table for VIT B/32 and ViT-B/16 backbones.
Model
Backbone
Accuracy
Recall
Train
Train
CLEVR
CLIP
VIT-B/32
0.52
0.49
0.33
0.34
LABCLIP-SB
VIT-B/32
0.99
0.85
0.99
0.83
LABCLIP-HNB
VIT-B/32
0.99
0.83
0.99
0.81
CLIP
VIT-B/16
0.50
0.54
0.31
0.38
LABCLIP-SB
VIT-B/16
LABCLIP-NNB
VIT-B/16
1.00
0.93
1.00
0.92
VIT:1114
LABCLIP-SB
VIT:/114
1.00
0.95
1.00
0.94
LABCLIP-NNB
VIT-L/14
1.00
PUG:SPAR
CLIP
V/3
0.51
0.51
0.02
0,02
LABCLIP-SB
LABCLIP-HNB
VIT-B/32
0.99
0.98
0.93
0.84
CLIP VIT-B/16
0.52
0.53
0.04
0.04
LABCLIP-SB
VIT-B/16
0.99
0.97
0.94
0.83
CLIP
VIT-L114
0.52
0.53
0.08
0.09
LABCLIP-SB
VIT:114
1.00
0.97
LABCLI--NNB
VIT-L14
1.00
0.97
0.98
0.91
PUG:SPARE
CLIP
VIT-B/32
0.51
0.51
0.01
0.01
LABCLIP-SP
VIT-B/32
0.91
0.89
0.73
0.69
LABCLIP-NNB
VIT-B/32
0.95
0.93
0.77
0.73
VIT:B/16
LABCLIP-SB
VIT-B/16
0.91
0.87
0.84
0.80
LABCLIP-HNB
VIT-B/16
0.92
CLIP
VIT-L/14
0.50
0,50
0.06
0.06
VITL/14
LARCLIP-ANR
VIT:/14
0.98
0.94
0.95
0.90
We use OpenAl's CLIP for all experiments, specifically the L/14 model for CLEVR, PUG:SPAR, and PUG:SPARE, shown in Table B Additional results for the B/32 and B/16 models are in Table 8 All models were trained on a single Nvidia A100 GPU.
The LABCLIP method involves adding additional linear layer the text encoder while keeping the CLIP weights frozen. For the upper bound results with fine-tuned CLIP, we unfreeze both en- coder weights. The additional linear layer has the same dimension the network output, equivalent
19

---

Published as a conference paper at ICLR 2026
to multiplying by a D x D matrix. We initialize this matrix as identity matrix, which corresponds to the case when no transformation is applied. The matrix is trained without further constraints.
Table 9: Aligning im- age space instead of text space in LABCLIP yields similar results. Test accuracies across all datasets. Performance improves for synthetic datasets and remains simi- lar for real-world datasets, confirming invariance the projection's s location.
We use the same contrastive loss as in the original CLIP. The temperature parameter in the contrastive loss is a lcarnable parameter, initialized to 0. Two approaches are explored: the Standard Batch (SB) contains only the corresponding images and text in each batch, while the Hard Negative Batch (HNB) also includes hard negative captions. We use the approach with Hard Negative Batch as the default version of LABCLIP. We report results for LABCLIP- in Tables
and 7
For synthetic datasets, we obtain hard negatives by swapping attributes. For COCO, we utilize the attribute and noun shuffling method introduced in NegCLIP to create negative captions Yuksekgonul et al. (2023). For example, in COCO-PRC example shown in Fig. [0] when the correct caption is 'a man with red helmet on small moped on dirt road', possible negative caption is dirt with a small road on a red moped on a helmet man'. There are no negative images in our approach.
Dataset
Accuracy
CLEVR
0.98
PUG
0.99
PUG:SPARE
0.96
VG-A
0.68
VG-R
0.2
We employ a combination of manual and random searches for hyperpa- rameter tuning, with the batch size ranging from 32 to 2048, epochs from to 50, and learning rates between 0.0001 and 0.01. We use the Adam optimizer for LABCLIP but SGD when mne-ttning on the CLEVR and PUG datasets. The optimal hyperparameters are selected based on the final epoch performance on the validation set.
Flick-PRC
0.84
COCO-PRC
081
0.82
Replace
0.82
Swap
0.72
COCO R@1
0.42
Aligning the image space. We chose to apply the alignment matrix to the text embeddings. How- ever, applying the projection the text or image embeddings is conceptually equivalent. Let u E IR be image and text embeddings. Our learned matrix A E DxD is applied on v be- fore inner product: sim u (Av). Note the equivalence: sim (ATw) u) v. which signifies the application of the linear matrix A on the image embeddings, Additional experiments with align- ing the image space confirms the invariance of the results to where the projection is applied (See Table D. The results improve for synthetic datasets and remain similar for real-world datasets.
Comparison to CLEVR by Lewis et al. (2024). Our two-object CLEVR dataset was inspired by the two-object setup in (Lewis et al. (024) as it allows for clear control of attributes and objects. However, the main difference between the works is the division of compositions into train, valida- tion, and test splits. In (Lewis et al.) 2024), some attribute-object pairs (for example, brown cube) are held out, so no scene containing a brown cube appears in training. In contrast, we split by two- object combinations: "brown cube and blue sphere" may appear in training, while "brown cube and red cylinder" does not. Holding out attribute-object pairs reduces training diversity, and prior work (Uselis al. 2025) shows that such diversity is important for compositional generalization.
Training LABCLIP on the dataset from (Lewis et al. 2024) yields similar results (our LABCLIP 1.29% vs their fine-tuned CLIP 0.25%) showing the difficulty of compositional generalization. To isolate binding, we shuffle the splits so every attribute-object pair appears in training. The evaluation still tests binding across scenes because position, material, and orientation vary. LABCLIP's perfor- mance, therefore, reflects whether CLIP's embeddings contain reliable attribute-object signal that holds across these variations. In this setting, LABCLIP reaches 94.3% on the two-object dataset, indicating that the embeddings contain a strong, recoverable binding signal.
Spatial reasoning. We evaluate spatial reasoning to test whether improved attribute-ohject binding transfers to spatial reasoning. Spa- tial relations describe how ob- jects relate to each other or to the scene, while attribute-object binding concerns how an object is linked to its attributes. These are distinct challenges. Spatial rea-
Table 10: LABCLIP trained only on binding matches CLIP's spa- tial reasoning and improves on spatial tasks when given spatial supervision. We evaluate LABCLIP trained on COCO and CC3M with attribute-object hard negatives, and LABCLIP trained with spa- tial hard negatives on half of each dataset, on the Uha
Model
Training data
What'sUp
COCO-Spatial
CQA-Spatial
CLIP
0.31
0.47
0.47
LABCLIP LABCLIP LABCLIP
COCO CC3M
0.31
0.48
0.46
0.31
0.51
0.45
Spatial data
0.54
0.64
0.55
soning is difficult for VLMs because spatial data is rare in training corpora (ammtth et al. 2023b)
20

---

Published as a conference paper at ICLR 2026
and spatial descriptions can be ambiguous (Liu al. 2023a), which makes cross-modal alignment harder.
We evaluate LABCLIP the What's sUp benchmark (Kamath et al. 2023b) in Table 10 LABCLIP trained only on attribute-object binding achieves performance comparable to CLIP (LABCLIP- CC3M 0.31 CLIP 0.31 What'sUp, 0.51 vs 0.47 COCO-Spatial, 0.45 0.47 GQA- Spatial), showing that matches CLIP's level of spatial reasoning. To test whether the method can extract spatial relations with supervision, train LABCLIP half of each dataset and evaluate on the remaining half. LABCLIP-spatial improves across all datasets (0.54, 0.64, 0.55), although the scores remain below human- level accuracy, consistent with Kamath al. 2023b).
Performance on downstream tasks. We show the zero-shot and probing perfor- mance for downstream tasks before and after LABCLIP TI
Table 11:
LABCLIP preserves probing performance but reduces zero-shot accuracy single-object datasets. Since LABCLIP does not modify image embeddings, prob- ing results remain unchanged.
LABCLIP, both when trained on COCO and CC3M, shows lower accuracy on CI- FAR10, CIFAR 100, and ImageNet in the zero-shot setting than CLIP. This is ex- pected, as LABCLIP is trained with hard negative captions that shuffle nouns and adjectives, making it more effective for compositional retrieval tasks. contrast, CIFARIO, CIFARION and ImageNet rely on single-object captions, which LAB- CLIP is not specifically optimized for. The stronger performance of NegCLIP in these cases may stem from its use of hard nega- tive images or fine-tuning. Note that LABCLIP trained on COCO outperforms LABCLIP trained on CC3M on the retrieval tasks. This expected because dataset overlap between training and testing can boost retrieval performance (Oh et al. (024). Since LABCLIP does not modify CLIP's original image representations, probing results remain unchanged.
CLIP
NegCLIP
LABCLIP (COCO)
LABCLIP (CC3M)
Zero-shot
CIFAR
0.90
0.89
0.86
0.89
CIFAR 100
0.65
0.63
0.57
0.63
ImageNet
0.56
0.53
0.46
0.51
Flickr30k
0.59
0.71
Flickr30k 12T
0.78
0.85
0,78
0,72
COCO T21
0.30
0.45
0.41
0.34
COCO 12T
0.50
0.59
0.55
0.46
Probing
CIFAR
0.95
0.94
0.95
0.95
CIFAR 100
0.80
0.79
0.80
0.80
ImugeNet
0.75
0.72
0.75
0.75
A.6 ADDITIONAL. ANALYSES
A.6.1 REPRESENTATIONAL SIMILARITIES AND ALIGNMENT
We analyze cosine similarity distributions between positive and negative pairs before and after ap- plying the alignment matrix A to observe the impact of our alignment transformation. For a text sequence xt',, the aligned text representation fix where A is the alignment transforma- tion applied to the original CLIP text embedding (x
Text-to-text similarity. First, we consider cosine similarities between positive and negative text representations before alignment, (flea(x!"") f fwel (permuute (xmrmntttex""))))))))))) )) and after alignment,
fxen(x)"), A. A (permute(xx)) )))
The distributions are depicted in Fig. 12
Probability
CLEVR
0.1
COCO
Similarity ay
0.22
We observe that, before aligmment, the similarities are higher, indicating that image embeddings may be incorrectly
0.0
0.8
CLEVR
COCO
matched with permuted text embeddings.
Probability
PUG:5PARE
0.2
PUG:SPAR
Similarity
After alignment, positive and negative text pairs become more dissimilar, potentially
0.0
0.0
making it casier to distinguish between permuted text pairs.
Similarity
Similarity Atter allonment
PUG:SPARE
PJG:SPAX
Betore
Positive
Negative
NVAfter alignment
Image-to-text similarity. We analyze cross-man moosl similarities by comparing im- age embeddings to both positive and neg- ative text embeddings (Fig. [2). The solid bars represent similarities between
Figure 12: LABCLIP reduces the similarity between neg- ative pairs. Left: distributions of cosine similarities between original and permuted captions before and after alignment. Right: mean similarities between positive and negative im- age text pairs before and after alignment.
21

---

Published as a conference paper at ICLR 2026
the image and text embeddings for positive pairs (fimag (x; (x Frex ) and negative pairs
(mimage (x."") (ermuttetttcxx before alignment.
The results indicate distinction between positive and negative pairs before alignment, as the solid bars for both are at the same height. However, after alignment (dashed bars), the similarity to positive text embeddings (Fimage (x, (x: fieu (Ax,") is notably higher than to permuted text t=1- This demonstrates that alignment enables better differen- tiation, allowing the model to match images with the correct text rather than the permuted text.
(fimae (xx"")),, fieal (permute(x] (x)
A.6.2 IMPLICATIONS MODALITY GAP
key challenge in VLMs like CLIP the modality gap, discrepancy between vision and text embeddings Liang et al. 2022). Previous studies [Schrodi et al. (2024) suggest that reducing this gap improves interaction between modalities. Motivated by this, we measure the Euclidean distance between mean embeddings and from the vision and text before and after alignment. Specifically, define x == N Σ fimage (x:) and y N E (ys) where fimage and fiext denote the encoders, and N' is the sample size. We then compute ||x - -yl12 for original embeddings and 1xx-Ag Ay|l2 for aligned embeddings, where A is matrix trained with LABCLIP.
Our experiments reveal that the modal- ity gap decreases across the COCO. PUG:SPAR, and PUG:SPARE datasets af- ter alignment, while CLEVR shows a slight increase (Fig. 13 left). We also pro- vide qualitative illustration of the modal- ity gap (Fig. 13 right) using UMAP Mclnnes et al. (2018) on the COCO test set. In this visualization, the aligned text representations (green) move closer to the image representations (blue), indicating a reduced modality gap after alignment. These results suggest that alignment effectiveness may vary across datasets, with our approach successfully enhancing cross- modsl compatibility most cases.
Modality gap
1.0
0.5
ginal
0.0
Original [|x Vllz
Aligned ||x Ayilz Modality gap COCO
Figure 13: Modality gap decreases after alignment. Left: modality gap between image and text representations before and after alignment. Right: UMAP visualization of COCO test set representations with text representations before and after alignment.
A.7 THE USAGE LLMs
In accordance with ICLR 2026 policy, we disclose that large language models were used to assist in text editing and polishing writing. All rescarch ideas, experiments, and analyses were conducted by the authors.
22


Reviews:
[{"review_id": "Vfm9TUxkNo", "reviewer": "Reviewer_dUnx", "summary": "This paper investigates the root cause of CLIP's well-documented \"bag-of-words\" (BoW) behavior in compositional reasoning tasks, where it fails to correctly bind attributes to their corresponding objects in multi-object scenes. The authors make a key distinction between uni-modal and cross-modal binding. Through a series of linear probing, robustness, and conjunctive search experiments, they convincingly demonstrate that CLIP's individual image and text encoders do encode attribute-object binding information in a linearly separable manner. The core problem, they argue, lies not in a lack of information but in a cross-modal alignment failure that prevents this information from being effectively utilized during retrieval. To validate this hypothesis, they propose LABCLIP, a simple method that applies a lightweight linear transformation to the text embeddings. This approach significantly improves performance on compositional benchmarks (ARO, SugarCrepe, COCO) without retraining the frozen CLIP encoders, offering a practical and efficient solution for enhancing existing CLIP-based systems.", "questions": "My major concerns are outlined in the \"Weaknesses\" part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The use of multiple, complementary experimental paradigms (linear probing, robustness to object count, conjunctive search) provides multi-faceted evidence for the uni-modal binding claim.\n- The paper is well-written and clear. The core insight is presented early and reinforced throughout. Figures 1, 3, and 5 are particularly effective in illustrating the key concepts.", "weaknesses": "1. The conclusion that “the problem lies in cross-modal alignment” remains descriptive. The paper lacks a deeper theoretical or mathematical explanation of why the alignment loss fails to preserve binding (e.g., analyzing the contrastive objective’s geometry or the modality gap quantitatively). \n2. The paper strongly asserts that the problem is only alignment. However, linear probing showing high accuracy only proves the information is linearly recoverable, not that it is represented in a way that is semantically robust or generalizable. High-dimensional embeddings can sometimes allow linear separation of concepts without the model having a true understanding of their binding.\n3. The work focuses exclusively on attribute-object binding. While this is a fundamental primitive, true compositionality also involves relations, quantifiers, and negation. The paper does not demonstrate that the alignment fix would generalize to these more complex structures, which may require non-linear solutions.\n4. The method for generating hard negatives on COCO relies on the heuristic from NegCLIP, which shuffles adjectives and nouns. This can create grammatically or semantically implausible captions (e.g., \"a helmet man\"). This risks teaching the model spurious correlations based on surface-level word co-occurrence rather than true structural binding, potentially inflating the reported gains.\n5. The paper convincingly shows a linear layer is sufficient, but it does not compare against other lightweight, non-linear adapter methods (e.g., LoRA, small MLPs). Such a comparison would strengthen the claim that linearity is indeed the optimal and simplest solution. The argument that \"linear is enough\" is practical but not theoretically definitive.\n6. As shown in Table 10 (Appendix), LABCLIP reduces zero-shot accuracy on standard single-object classification datasets (CIFAR, ImageNet). While expected, this trade-off is significant for practitioners and should be discussed more prominently in the main text, as it highlights the specialization of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RTtwApk9NL", "reviewer": "Reviewer_84zT", "summary": "Existing work has questioned CLIP’s ability to bind concepts, suggesting it behaves like a bag-of-words representation. In this paper, the authors find that CLIP can bind attributes to their objects, but only unimodally. They present linear-probing experiments, robustness tests with increasing object counts, and conjunctive-search experiments to show that CLIP’s text and visual representations bind concepts independently. Finally, they train a lightweight layer to patch existing CLIP embeddings and teach the model to bind concepts.", "questions": "Do you suspect that the BOWness arises from the contrastive loss or the dual encoder in CLIP? It would be insightful to the reader if you could include this discussion in the paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The paper is well written. The introduction clearly motivates an important problem and explains how prior work does not fully investigate the source of bag-of-wordness. The related-work section is thorough and effectively positions the paper within the literature.\n\nLarge-scale experiments: The paper shows that concept binding can be improved with a simple linear probe on large-scale datasets such as ARO and SugarCrepe.", "weaknesses": "**Differences in results in prior work.**\nWhile the paper’s results are encouraging, I would like to understand the differences from existing works that show that CLIP, even after fine-tuning, does not bind concepts [a, b].\n\nIn [a], the authors claim that the representations are not expressive enough to bind concepts. They introduce the Concept Binding Benchmark and show that fine-tuning CLIP and linear probes (the type-logical model in their experiments) do not generalize to held-out compositions that require binding. However, the results in this manuscript suggest that binding can be remedied with a linear probe.\n\nFurthermore, the results in this paper contradict those of Kamath et al. (2024) [b]. In Table 2 (Section 4.3), [b] fine-tune CLIP and evaluate performance on held-out compositions for spatial reasoning (which can be viewed as binding objects to their relations).\n\nIt would be helpful if the authors could explain why the experiments in this paper contradict prior work. The paper would be strengthened by running evaluations on datasets [a] and [b] and demonstrating improved binding.\n\n\n**Minor fixes**\n\nLine 309: use \\citet for Campbell et al., 2024. \n\nFigure 5: Could you draw a box to highlight the incongruency? \n\n**References**\n\n[a] Does CLIP Bind Concepts? Probing Compositionality in Large Image Models. Findings of EACL 2024. \n\n[b] What's \"up\" with vision-language models? Investigating their struggle with spatial reasoning. EMNLP 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ecWyh0Ruof", "reviewer": "Reviewer_kJCN", "summary": "CLIP models often has Bag-of-Words (BoW) property where it treats images and texts as sets of individual concepts and struggles to correctly bind each object and its corresponding attribute pairs. This paper questions the reasons of such BoW property with a series of controlled experiments and proposes a simple and efficient fix to this problem. Specifically, several controlled experiments with linear probing on individual visual and text embeddings from object-attribute binding datasets indicate that CLIP models already encode attribute-object relationships in each modality. In addition, it is shown that such attribute-object concepts are linearly separable on the CLIP's representation space. Based on the results, the problem lies on the cross-modal alignment, and a simple solution is proposed by linearly projecting one embedding during the alignment training. In this way, the performance recovers close to the fully fine-tuned CLIP models while achieving several practical advantages such as computational efficiency and backward compatibility of already-deployed systems.", "questions": "Regarding the weaknesses stated above, it is expected to show that the proposed findings and methods can generalize beyond the simple attribute-object binding setting to more complex compositional reasoning tasks, such as Winoground, where the oracle method NegCLIP also show limited performance. In addition, it would be important to show how the proposed LABCLIP method affect the general downstream tasks such as zero-shot image classification and image-text retrieval.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "This paper investigates the reason behind the CLIP models' BoW property which is often unexplored from the previous compositional reasoning literature. The controlled experiments and analyses made could be a valuable reference for guiding a better solution for the following works. From the methodological perspective, LABCLIP for appending a linear projection matrix on either image or text embedding is effective to remedy the CLIP models' attribute-object binding capabilities, while it does not need the full fine-tuning of encoders parameters. This can benefit both on the training and inference procedure.", "weaknesses": "One major concern lies in the scope of the work with respective to compositional reasoning tasks in several aspects. While the paper mainly focused on a relatively simple settings, attribute-object binding, but it is not yet discussed whether the observations and solutions made in the paper can generalize to a more complex and realistic compositional reasoning benchmarks such as Winoground. In addition, since the oracle method to the proposed solution is NegCLIP, which is trained on both original and hard negative captions, the proposed solution might share the NegCLIP's limitations such as lower performance on challenging compositional reasoning benchmarks such as Winoground and original CLIP's downstream tasks including zero-shot image classification and image-text retrieval. Overall, the method appears to focus on a specific area of compositional reasoning and requires further justifications to a complex and realistic settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lshnH2aSNw", "reviewer": "Reviewer_CZn4", "summary": "This paper explores the Bag-of-Words aspect of CLIP embeddings, namely that compositional information is limited and representations are not sensitive to compositional orderings in a cross-modal way, but that this information is present in the individual modalities. The work goes on to explore various ideas aimed at uncovering what is at the core of this problem and how to fix the BoW-nature of standard CLIP embeddings in cross-modal compositional contexts.\n\nIn summary, I loved reading this paper and the simple explanation and guided tour of the hypothesis, easy-to-understand experiments that are nice and conceptually clear. I definitely think this is highly relevant to the community. However, passages such as the one in the conclusion (\"We investigated the reasons behind CLIP's bag-of-words behavior\") sort of ring a little bit hollow because I feel like a solid step was taken towards localising the nature of the issue, but I am not so sure a lot of solid ground was covered to explain the more interesting question - why this occurs. We know that a linear transformation among the spaces to align the embeddings goes some way to solving this issue, but what drives this and how simple linear transformations actually \"recover compositional information\" from a BoW kind of treatment is still a bit of a mystery to me. Having said that, given the space allowed for submissions, this is a great submission and full of detail and interesting ideas.", "questions": "1. Have you considered the cases that extend beyond image-text modalities? \n\n2. What potential pitfalls did you consider in using a synthetically generated benchmark?\n\n3. Why do you think PUG:SPARE in CLIP (ft) achieved higher than PUG:SPAR in the same case (i.e. 1.00 > 0.98) when the PUG:SPARE model was supposed to be more difficult. This is not a hugely important question for the validity of the experiments, but I did find myself having this question when considering the results and I wasn't confident in my hypothesis. Perhaps the authors can share some of their thoughts on this.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is quite a well-written paper. It's informative and also understands the audience needs some examples, so the example with the orange square and blue triangle to highlight the problem in a clear example is very welcome to give a concrete example of the issue at hand (and how the permuted text also has a high match with the inconsistent image). The questions that lie at the crux of the issue are put forward in a simple, clear and easy-to-understand manner. This is refreshing to see and greatly appreciated, making it easy to follow how the setup of the experiments answers the questions that truly get to the heart of what the paper claims to be about. \n\nThere is clear attention to detail and awareness of the often-unavoidable quagmire of confounds that render any experimental results across a large dataset of real images to be impossible to declare to be fully associated with a list of simple claims. This then pushed the authors to work in a controlled setting that they are able to manipulate clearly and cleverly in order to show that conclusions and differences only arise due to the re-ordering of the components in the text encoder. This experimental awareness is a solid positive for this paper, also in the recognition that both real and synthetic datasets can provide complementary results and a holistic approach of the results is often necessary.\n\nAnother strength is the consideration and attention to splits in the linear classifier case. This work correctly identifies previous approaches being problematic and not evaluating where the lack of alignment comes from. Additionally, the presence of summary takeaway points greatly enables the reader to follow the main points being raised and to follow the authors' claims and hypotheses with regard to the presented results.", "weaknesses": "The weaknesses here are only very minor and should be treated as small tips to increase the readability or accessibility of the paper.\n\n1. When listing the compositional datasets, it would be helpful to include the names of those benchmarks rather than just the generic theme and the citation, as not everyone will be aware of the citation but might be familiar with a name of a specific dataset and this would only increase the mental links you are raising in those lines of the paper. I know later in Table 1 this is clarified, but you might facilitate reading by including them in that section, too.\n\n2. When discussing the matching of permuted text encoders, it means something different to be picking the correct match out of a small number of possibilities than it is to picking out the correct match among a large potential batch of potential pairs, due to the fact that there is a higher chance to randomly select the correct one. I found myself curious of this when reading through section 3 and felt like additional information would be welcome in this part of the paper, even if just in brackets it mentions how many options were being selected from in the batch. This would then stop me from wondering if the batches were small enough that random correct selection might be a possibility to explain some results.\n\n3. The paper's goals aimed to find the cause of the BoW phenomenon and did a great job of locating it, but the abstract and conclusion lead one to believe that more of a causative finding has been presented. I am impressed by the paper and the results found, but after finishing the paper, I'm not sure I've been enlightened too much as to what is causing this phenomenon. The authors have certainly made a scientific advance by locating where the issue lies, but without tying it to other relevant concepts like the modality gap (which the authors show increases in some cases in one of the appendices for a dataset) then I'm left even more confused as to what brought about the cross-modal lack of alignment (I had previously suspected it was more tightly connected to and explained by the modality gap issue). Additionally, I feel some of that work and potential explanatory notes should be made in the relevant literature section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)