COMPOSITIONALITY IN CONTRASTIVE
VISION-LANGUAGE MODELS: A SURVEY OF
METHODS AND BENCHMARKS

Maryam Abdolali, Fardin Ayar, Roozbeh Tavanaei, Saba Fathi, Arshia

Momtazi, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi

► To cite this version:

Maryam Abdolali, Fardin Ayar, Roozbeh Tavanaei, Saba Fathi, Arshia Momtazi, et al.. COMPOSITIONALITY IN CONTRASTIVE VISION-LANGUAGE MODELS: A SURVEY OF METHODS AND BENCHMARKS. 2026. hal-05484159

HAL Id: hal-05484159

https://hal.science/hal-05484159v1

Preprint submitted on 29 Jan 2026

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# COMPOSITIONALITY IN CONTRASTIVE VISION–LANGUAGE MODELS: A SURVEY OF METHODS AND BENCHMARKS

A PREPRINT

Maryam Abdolali<fnref n="1" />* Fardin Ayar<fnref n="2" /> Roozbeh Tavanaei<fnref n="1" /> Saba Fathi<fnref n="2" />

Arshia Momtazi<fnref n="1" /> Ehsan Javanmardi<fnref n="3" /> Manabu Tsukada<fnref n="3" /> Mahdi Javanmardi<fnref n="2" />†

## ABSTRACT

Contrastive vision language models (CVLMs), such as CLIP, have achieved impressive results in zero-shot tasks. Despite their progress, CVLMs continue to struggle with compositional reasoning—the ability to correctly interpret novel combinations of familiar concepts. In vision–language learning, compositionality involves several core abilities: linking attributes to objects, reasoning about spatial relations between entities, reasoning about temporal or event relations, understanding logical operators such as negation and quantifiers, and handling role-sensitive interpretation. Research has shown that a common failure mode in CLIP is a bag-of-words behavior, where images and captions are treated as unordered sets of concepts. This undermines attribute-object binding, relational reasoning, and sensitivity to word order or role assignment. More broadly, these models often exploit shortcut signals, such as word co-occurrence statistics or dataset biases, rather than learning systematic, compositional rules that generalize to unseen combinations. This survey reviews recent efforts to strengthen compositionality in CLIP-style vision–language models. We propose a taxonomy with four primary method families and a complementary line of work on diagnostic probes. Data-centric approaches reshape the training signal via rule-based edits, synthetic data generation, graph-based construction, and targeted sampling to expose models to diverse compositional structures. Objective-centric approaches revise learning criteria so that optimization explicitly rewards correct compositional bindings rather than shortcut cues. Architecture-centric approaches modify model architectures to enable finer-grained visual grounding and similarity refinement. Inference-time approaches that are training-free, enhance frozen models by decomposing captions, grounding parts in visual regions, and fusing local signals into the global image–text scores to enforce compositional constraints. Finally, diagnostic analysis identifies where and why compositional failures occur, attributing errors to specific components and evaluating the reliability of existing benchmarks. Our goal is to provide a clear and organized overview of this rapidly growing field. The taxonomy offers newcomers an accessible entry point while serving as a reference for experts; we highlight key open challenges alongside promising directions for future research.

**Keywords** Compositionality · Vision-Language Models · CLIP · Multimodal Learning · Fine-Tuning

“The meaning of the whole is a function of the meaning of its parts.” – MJ Cresswell [1]

## 1 Introduction

The ability to understand and compose novel combinations of known concepts is a defining feature of human cognition and a central requirement for artificial intelligence. CLIP [2] and related Contrastive Vision Language Models (CVLMs)

<footnote n="1">¹K. N. Toosi University of Technology, Tehran, Iran</footnote>

<footnote n="2">²Amirkabir University of Technology, Tehran, Iran</footnote>

<footnote n="3">³The University of Tokyo, Tokyo, Japan</footnote>

<footnote>*Corresponding author maryam.abdolali@kntu.ac.ir</footnote>

<footnote>†Emails: ¹{roozbeh.tavanaei, arshia.momtazi}@email.kntu.ac.ir; ²{fardin.ayar, sabafathi, mjavan}@aut.ac.ir; ³{ejavanmardi, mtsukada}@g.ecc.u-tokyo.ac.jp.</footnote>
learn joint image-text representations from large-scale datasets and excel at open-vocabulary classification and retrieval, yet they often struggle with basic compositional reasoning. For example, CLIP often behaves in a *bag-of-words* (BoW) manner [3], treating an image and its caption as unordered sets of concepts; as a result, it can fail to bind attributes to the correct objects or distinguish subjects from objects in a relation. More broadly, cross-modal matching can collapse fine-grained structure, so models miss *who did what to whom* (relational reasoning) and *which attributes belong to which objects* (attribute-object binding) [3]. Similarly, strong concept-association biases can cause over-reliance on typical combinations (e.g., “purple eggplant”) [4] and misclassification of novel compositions [5]. Such issues limit downstream applications ranging from visual question answering [6] to image editing [7].

**Beyond simple compositions:** Although early works focused on attribute-object binding and word order sensitivity [8], more recent efforts address a broader set of skills: counting and quantification [9], handling negation and logical relations [10, 11], reasoning about 2D/3D spatial relations and orientation [12], tracking temporal and causal dynamics in sequential image data (e.g., frame-based temporal visual question answering) [13], and generalizing systematically to unseen combinations [5]. In addition, evaluations further reveal limitations that synthetic benchmarks may overlook: even state-of-the-art models struggle on vision-heavy retrieval tasks when linguistic priors provide no advantage. For instance, matching a caption like "a mug with its handle on the left" to the correct image requires genuine visual grounding-localizing the object, identifying its parts, and reasoning about spatial relationships-rather than exploiting text-based statistical patterns [14]. Such retrieval queries serve as effective diagnostic probes, exposing whether models rely on genuine visual understanding or superficial linguistic correlations.

**Motivation for compositional understanding:** The need for robust compositionality goes far beyond academic curiosity: many safety-critical and socially impactful applications rely on the ability to parse and combine multiple concepts reliably under diverse and often ambiguous conditions. Consider an autonomous vehicle approaching an intersection: the system must correctly interpret a structured description such as “a child riding a scooter behind a parked truck” and distinguish it from “a truck riding behind a child,” since failing to bind subjects, objects, and spatial relations correctly can lead to catastrophic decisions. Similar requirements arise in robotics and human-computer interaction, where agents must follow instructions such as “pick up the red mug next to the green book” or “avoid obstacles while carrying the fragile package through the narrow doorway.” Such tasks demand fine-grained compositional reasoning, including accurate attribute-object binding, relational interpretation, and spatial grounding, rather than mere global scene recognition. Healthcare and assistive technologies have similar demands: for example, a surgical robot must locate “the artery between the two muscles” and perform basic numerical reasoning, such as counting clamps accurately. Together, these examples highlight how compositional understanding enables the safe deployment of VLMs in real-world settings. Yet, as we will review, current CVLMs often treat images and captions as unordered bags of concepts and fail to generalize to novel combinations or rare compositions.

The importance of addressing compositional failures extends beyond standalone CVLMs to the growing class of instruction-tuned multimodal LLMs that rely on them [15]. Models such as LLaVA [16] integrate CLIP-style vision encoders with large language models, enabling multimodal dialogue and interactive reasoning, but they inherit the same compositional weaknesses from their visual encoders. This dependency means that failures in attribute binding or spatial understanding within CVLMs propagate directly to the language model’s interpretation [17], limiting overall reasoning capabilities regardless of the LLM’s own strengths. Moreover, CLIP-like encoders are also widely used as text encoders in modern image and video generation systems [18], so compositional errors in these representations can lead to misaligned or unsafe generated content even when the underlying generative model is powerful [19, 20]. As a result, improving compositional understanding in CVLMs is essential for reliable performance in instruction-tuned and conversational multimodal systems, with implications for safety-critical applications such as autonomous driving, robotics, and interactive AI.

**What has been tried?** A growing line of work tackles these limits along five complementary axes (summarized in Table 1):

* **Data-centric** methods change what we train on and how we sample data; including rule/template perturbations that create hard negatives/positives [3, 21], synthetic and counterfactual generation [22, 23, 24], and scene-graph construction [25, 26]. These approaches aim to eliminate near-misses that break shortcuts while adding diverse but controlled similar data that preserve meaning. The goal is simple: make it costly for the model to ignore who/what/where by showing minimal edits that flip the truth.

* **Objective-centric** methods change what the model is optimized to do; including extending standard CLIP’s *Information Noise-Contrastive Estimation* (InfoNCE) loss with composition-breaking distractors and counterfactual candidates [9, 27, 28], adding margin/ranking and multi-positive terms that separate near-misses while supervising paraphrase sets [23, 26], introducing geometry-aware objectives that encode hierarchical
structure in the embedding space [29], and distilling knowledge from a teacher so fine-grained structure is learned without losing global zero-shot ability [30, 31]. These losses reward correct binding and penalize close but wrong variants.

* **Architecture-centric** methods restructure model architectures by incorporating similarity refinement mechanisms via lightweight alignment heads [32, 33], modular compositional components including phrase-region alignment modules [34] and graph-based structures [35, 25, 36], and representation factorization through disentangling layers [37]. These modifications address the global cosine similarity bottleneck by introducing explicit mechanisms for representing entities, attributes, and relations, enabling fine-grained visual grounding and structured cross-modal reasoning.

* **Inference-time methods** improve compositional matching without retraining by decomposing text, grounding constituents in visual regions, and reweighting global scores [34, 38]. These approaches use attention maps or external parsers with grounding models to inject region-specific signals, addressing the cosine bottleneck while preserving backbone capabilities.

* **Finally, Evaluation & diagnostic methods** focus on three questions: where CVLMs fail, why they fail, and how to measure and fix those failures. In particular, they localize failure modes via controlled text/image perturbations [39, 40, 41]; they identify where the errors come from [33, 42, 43]: the text encoder, the image encoder, or their alignment; and investigate benchmarks to determine whether they truly measure multimodal compositional skill or merely language shortcuts [44, 45]. These methods make progress easier to quantify and reveal which design choices (data, loss, model, or inference) actually help.

**This survey.** The main contributions of this paper are summarized as follows:

1. **Taxonomy:** We propose a new taxonomy to classify compositional studies in CVLM, dividing existing approaches into five categories: (i) Data-centric, (ii) objective-centric, (iii) architecture-centric, (iv) inference-time methods, and (v) diagnostic studies.

2. **Detailed overview:** We provide a comprehensive overview and analysis of these methods.

3. **Evaluation:** We organize compositional benchmarks into a unified taxonomy based on the data source (real or synthetic), targeted skills (attributes, relations, order, counting, negation, temporal reasoning), evaluation direction (image→text, text→image, or both), and, when applicable, how negative examples are constructed.

4. **Open gaps:** We point out several key open problems, including: improving compositional understanding without hurting zero-shot performance, handling negation and numbers reliably, and cutting down language-only shortcuts; and suggest practical next steps.

Section 2 formalizes our notion of compositionality and reviews common vision-language model architectures. Section 3 presents our taxonomy, organizing methods into data-centric, objective-centric, architecture-centric, and inference-time families, with representative techniques for each category, and includes diagnostic studies that probe failure modes and benchmark reliability. Section 4 surveys existing benchmarks and evaluation methodologies for assessing compositional reasoning. Section 5 presents empirical comparisons across methods. Section 6 discusses current limitations and outlines promising directions for future research.

# 2 Background

## 2.1 Vision-Language Models and CLIP

Contrastive vision-language models (CVLMs) learn a joint embedding space where images and text can be directly compared through similarity measures. The core principle is that semantically aligned image-caption pairs should be close together in this space, while semantically mismatched pairs should be separated. The seminal CLIP model established this paradigm through large-scale training on hundreds of millions of image-caption pairs collected from web data. CLIP employs a dual-encoder architecture: a vision encoder (ResNet or Vision Transformer) encodes visual inputs, while a Transformer-based text encoder encodes text. Both encoders project their respective modalities into a shared latent space, and training is performed using a symmetric contrastive objective (InfoNCE) that maximizes similarity for positive pairs while minimizing similarity for negative pairs.

**CLIP (InfoNCE) baseline.** Given a minibatch $\{(x_i, y_i)\}_{i=1}^B$ of images and captions, a visual encoder $f_\theta$ and a text encoder $g_\phi$ produce embeddings $\tilde{v}_i = f_\theta(x_i) \in \mathbb{R}^d$ and $t_i = g_\phi(y_i) \in \mathbb{R}^d$. After $\ell_2$-normalization,

$$v_i = \frac{\tilde{v}_i}{\|\tilde{v}_i\|_2}, \quad t_i = \frac{\tilde{t}_i}{\|\tilde{t}_i\|_2}, \quad \|v_i\|_2 = \|t_i\|_2 = 1.$$
With cosine similarity $s(x, y) = v^\top t$ and temperature $\tau > 0$, define $S \in \mathbb{R}^{B \times B}$ by $S_{ij} = s(x_i, y_j)/\tau$. The symmetric InfoNCE losses are

$$ \mathcal{L}_{\text{InfoNCE}}^{I \to T} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(S_{ii})}{\sum_{j=1}^{B} \exp(S_{ij})}, \qquad \mathcal{L}_{\text{InfoNCE}}^{T \to I} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(S_{ii})}{\sum_{j=1}^{B} \exp(S_{ji})}, $$

and the overall objective is:

$$ \mathcal{L}_{\text{CLIP}} = \frac{1}{2} (\mathcal{L}_{\text{InfoNCE}}^{I \to T} + \mathcal{L}_{\text{InfoNCE}}^{T \to I}). $$

At inference, cosine similarities between normalized embeddings support efficient retrieval and open-vocabulary classification.

In text-to-image (T2I) retrieval, a text query $y$ is encoded as $t = g_\phi(y)$, and each gallery image $x_j$ is encoded as $v_j = f_\theta(x_j)$. Relevance is scored by the cosine similarities $s_j = v_j^\top t$, and images are ranked in descending order of $s_j$. The converse image-to-text retrieval (I2T) follows the same principle by fixing an image embedding $v$ and ranking candidate captions $\{t_k\}$ using $v^\top t_k$. In Open-vocabulary (zero-shot) classification, class labels are expressed as prompts, for example $y_c = $"a photo of a {class}."$ for each class $c$. Given an image $x$ with embedding $v = f_\theta(x)$, each prompt is encoded as $t_c = g_\phi(y_c)$, and class scores are computed as $z_c = v^\top t_c/\tau$. Predicted probabilities follow a softmax,

$$ p(c | x) = \frac{\exp(z_c)}{\sum_{c'} \exp(z_{c'})}. $$

Subsequent work has significantly extended the functionality of CVLMs beyond basic contrastive learning. Following CLIP's introduction, ALIGN [46] demonstrated the effectiveness of scaling contrastive learning to larger datasets, achieving improved performance through massive web-scale training. BLIP [47] and BLIP-2 [48] augmented dual-encoder contrastive learning with cross-attention and generative objectives, thereby enabling not only retrieval but also captioning and question answering. The MS-CLIP [49] approach investigated parameter sharing across modalities during contrastive pre-training, achieving improved zero-shot classification performance. Extensions to video understanding include CLIP4Clip [50], which adapted CLIP for end-to-end video clip retrieval, and X-CLIP [51], which introduced cross-frame attention mechanisms for general video recognition.

## 2.2 Compositionality in CVLMs

Compositionality, as used here, refers to a broad spectrum of capabilities outlined below and illustrated in Figure 1:

* **Attribute-object binding** [33]: linking properties (color, size, shape, texture) to the correct object, not just anywhere in the scene.

- **Why it is hard**: Models frequently rely on canonical attribute-object associations (e.g., "lemons are yellow") learned from training data rather than visual evidence, and in crowded scenes with multiple objects, attribute signals become entangled across different objects.

- **Typical failure**: Models assign higher similarity scores to incorrect captions (e.g., "yellow lemon and purple eggplant") than to correct ones (e.g., "purple lemon and yellow eggplant") when matching against images that clearly exhibit the correct attribute-object bindings (Fig. 1 (a)).

* **Spatial and relational reasoning** [33]: understanding where things are and who does what to whom (on/under, left/right, in front/behind; holding, pushing, riding).

- **Why it is hard**: Training captions rarely specify precise geometric layouts for objects, so models learn to rely on frequent co-occurrences and salience (big/central objects) rather than explicit geometry. This encourages layout-insensitive matching, so small preposition or role flips that should reverse the label often do not change the score.

- **Typical failure**: Models fail to distinguish between captions with reversed spatial relations (e.g., assigning similar similarity scores to "the cup is to the right of the plates" and "the cup is to the left of the plates" for the same image) or incorrectly match captions with swapped subject-object roles (e.g., "a child holding a kite" vs. "a kite holding a child") (Fig 1 (b)).

* **Negation and logical operators** [10]: noticing what is absent (not/without) and handling simple logic ("A and not B", "either A or B").

- **Why it is hard**: Visual evidence for negation is indirect, requiring models to confirm the absence of properties rather than their presence, which is inherently more challenging than detecting positive visual cues.
<figure><img src="image_2.png" /><figcaption>Figure 1: **Compositionality failure cases in CVLMs.** Each panel pairs an image with two captions. Correct and wrong captions are shown in blue and orange colors, respectively. CLIP similarity scores is indicated underneath each case. (a) *Attribute-object binding*: swapping colors between objects. (b) *2D relations*: left vs. right of the plates. (c) *Negation*: "car not blue" vs. "car not red." (d) *Counting*: three vs. four instances. (e) *Temporal*: putting on vs. taking off a jacket. (f) *Systematicity*: unseen compositions (e.g., "rotten papillon") built from seen parts ("rotten"+"papillon").</figcaption></figure>

- **Typical failure:** Models assign high similarity scores to negated captions (e.g., "a car *not blue*") even when the image clearly exhibits the negated property (e.g., a blue car), or match captions describing absence (e.g., "a mug *without a handle*") when the visual evidence contradicts the negation (Fig 1 (c)).

* **Counting and quantification** [9]: getting the exact count of objects when the caption explicitly names a number ("three," "four") and picking an appropriate range when the caption is vague ("a couple," "several," "many").

- **Why it is hard:** Web-scale training data rarely supervise counts explicitly. Captions rarely name exact numbers, often mention "some" or only the salient object, and provide no boxes/masks to tie instances to words. Crops, duplicates, and occlusions further break any one-to-one mapping (an image with four mugs might be captioned "a mug" so models learn category presence, not cardinality).

- **Typical failure:** Models rank captions with incorrect cardinality (e.g., "four dogs") above correct ones (e.g., "three dogs") when performing image-text retrieval, despite clear visual evidence of the actual count (Fig 1 (d)).

* **Temporal and causal understanding** [13]: reasoning about *order* (before/after) and *duration* (how long).

- **Why it is hard:** Temporal reasoning requires tracking state changes and causal dependencies across multiple frames or time steps, rather than reasoning about static single-frame representations.

- **Typical failure:** Models fail to distinguish between temporally reversed actions, assigning similar similarity scores to captions describing opposite temporal sequences (e.g., "putting on a jacket" vs. "taking off a jacket") when matching against video frames or image sequences (Fig 1 (e)).

* **Systematicity and productivity** [5, 52]: taking pieces the model already understands (objects, attributes, relations) and combining them in new ways (systematicity); and coping as descriptions grow longer or more complex (productivity).

- **Why it is hard:** Models tend to memorize frequent attribute-object or relation-object co-occurrences from training data and fail to systematically recombine known components into novel compositions; increasing caption length introduces additional complexity that degrades compositional reasoning.

- **Typical failure:** Models achieve high similarity scores for seen attribute-object combinations (e.g., "rotten leaf" and "beautiful papillon") but fail to generalize to novel compositions (e.g., "rotten papillon"), or exhibit degraded retrieval performance as caption complexity increases with additional clauses (Fig 1 (f)).
Compositionality in Vision-Language Models

<figure><img src="image_3.png" /><figcaption>Figure 2: Overview of our taxonomy for improving compositionality in CLIP-style contrastive vision–language models. We group approaches into data-centric (§3), objective-centric (§4), architecture-centric (§5), inference-time (§6), and diagnostic studies (§7), along with their main subcategories.</figcaption></figure>

## 2.3 Why do compositional failures occur in CVLMs?

There are two main root causes. First, the contrastive training objective encourages global similarity rather than local alignment. Without fine-grained supervision, the model may match images and captions based on shared concepts (e.g., “dog”, “chain”) and ignore the structure relating them (e.g., “on” or “under”). Cosine similarity on global embeddings discards much of the image–text alignment structure (e.g., attribute–object links and subject–object roles). LABCLIP [33] demonstrates that a simple linear transformation of text embeddings before computing cosine similarity can recover attribute-object binding and mitigate BoW behaviour. Second, web-scale data often contain strong correlations (e.g., “eggplant” often appears with “purple”); these biases push the model toward usual associations and prevent it from generalizing to new combinations [4]. In fact, the training data rarely cover complex reasoning tasks such as counting, negation or temporal order; as a result, CVLMs fail on tasks that require reasoning beyond co-occurrence patterns[10, 9].

# 3 Taxonomy

In this section, we survey how the community has addressed these challenges. Figure 2 provides an overview of our taxonomy and the section structure used throughout the survey. We organize prior work into the categories in Table 1, listing representative papers and their key innovations. We then examine each category in depth, highlighting central methods and outlining the core ideas behind them.

## 3.1 Data-Centric

Data-centric methods modify the training data. They introduce two complementary pair types: (i) *hard negatives/near-miss* pairs that look almost correct but are *false* for the image (e.g., flipping one crucial attribute: “a **yellow** fire hydrant ...” for an original photo with a **red** hydrant; see Fig. 3); and (ii) *hard positives/semantics-preserving* variants that remain *true* (e.g., a faithful paraphrase or a different image depicting the same situation). Hard positives encourage *invariance* to non-essential, meaning-preserving changes, while hard negatives enforce sensitivity to the key details. They aim to push the model to focus on the crucial visual details instead of relying on wording or priors.

In practice, such pairs are created via:
<table><caption>Table 1: Taxonomy of methods for improving compositionality in CLIP-style VLMs.</caption><thead><tr><th>Category</th><th>Subcategory</th><th>Representative methods (year)</th><th>What they change / add</th></tr></thead><tbody><tr><td rowspan="4">Data-centric</td><td>Rule / Template-based</td><td>NegCLIP (2022); SVLC (2023); AHNPL (2025); NAS (2025); SimCLIP (2024); CLIC (2025);</td><td>Generate rule-/template-based edits (e.g., subject-object flips, antonym replacements, and attribute swaps) to create near-miss variants that serve as hard negatives (often with paired hard positives), pushing the model to bind attributes correctly and respect relations/order.</td></tr><tr><td>Synthetic-based</td><td>DAC (2023); LaCLIP (2023); SyViC (2023); TripletCLIP (2024); SPARCL (2025); HP+HN (2024);</td><td>Generate synthetic pairs via LLM edits, diffusion, or 3D simulation to control compositions and near-misses.</td></tr><tr><td>Graph-based</td><td>StructureCLIP (2023); MosaICLIP (2023)</td><td>Scene-graph edits for meaningful negatives/positives</td></tr><tr><td>InfoNCE Extensions</td><td>NegCLIP col-masked (2022); DAC(2023); MosaIClip (2023); CountBench (2023); TripletClip (2024); Navero (2024)</td><td>Expand denominators with composition-breaking captions; binary/multi-choice heads to include hard-positives.</td></tr><tr><td rowspan="3">Objective-centric</td><td>Margin-based</td><td>StructureCLIP hinge (2023); CE-CLIP (2023); SPARCL adaptive (2025); GCS (2025)</td><td>Hinge/adaptive margins for near-miss separation; intra/inter-set logistic objectives.</td></tr><tr><td>Geometry-aware & Distillation</td><td>MERU (2024); HyCoCLIP (2025); SDS-CLIP (2024); IL-CLIP (2024); DeGLA (2025)</td><td>Hyperbolic (Lorentz) embedding w/ entailment; Distill from diffusion priors and EMA global path.</td></tr><tr><td>Similarity refinement</td><td>SmartCLIP (2025); LABCLIP (2025)</td><td>Caption-conditioned / linear pre-cosine alignment.</td></tr><tr><td rowspan="2">Architecture-centric</td><td>Modular architecture</td><td>ComAlign (2024); StructureCLIP (2023); SGVL/CORA (2024); OC-CLIP (2024); COGT (2024)</td><td>Inject explicit compositional components: decompose similarity into entity-relation parts; add scene-graph tokens/heads; object-slot binding with cross-attention; structured/causally ordered decoders that replace or refine global similarity with multi-level scores.</td></tr><tr><td>Representation factorization</td><td>GIPCOL (2023); DCDA (2025)</td><td>Graph/adapters to disentangle attr-obj-comp.</td></tr><tr><td>Inference-time</td><td>Train-free region-grounded</td><td>ComCLIP (2024); GCLIP (2025)</td><td>Detector/dense-caption-guided local reweighting at test time.</td></tr><tr><td rowspan="3">Diagnostic studies</td><td>Failure detections</td><td>CLIP Under the Microscope (2025); Compound-noun diagnostics (2024); Role/relational binding tests (2024); C-OoD factorization/soft-rank (2024); Domain-mix generalization (2025)</td><td>Controlled perturbations to reveal biases & shortcomings in text and vision encoders</td></tr><tr><td>Encoders vs. alignment</td><td>LABCLIP probes (2025); Text-recoverability probes (2023); “Bad Eyes?” superposition (2025); Game-theoretic sensitivity (2024)</td><td>They aim to pinpoint where compositional failures come from (text encoder, image encoder, or cross-modal alignment)</td></tr><tr><td>Benchmark investigation</td><td>Role of language-only baselines (2023); “Good Benchmarks?” shortcut analysis (2025)</td><td>Disentangle language priors from genuine vision–language reasoning by using language-only baselines and diagnostics that expose when benchmark gains come from text shortcuts rather than multimodal understanding</td></tr></tbody></table>
1. Rule/Template edits to captions (e.g., type-consistent swaps) [3, 21];
2. Synthetic or counterfactual generation with LLMs and diffusion-based editors [23, 22, 24]; and
3. Graph-driven construction or edits using scene graphs over objects, attributes, and relations [25, 26].

Refer to Figure 4 for representative examples of each category. Below, we review the main data-centric approaches.

<figure><img src="image_4.png" /><figcaption>Figure 3: Hard negatives and hard positives for compositional training. **Left:** original image–caption pair. **Middle:** a hard negative pair created by a minimal, type-consistent edit that flips the truth value (red→yellow) while holding all else fixed, discouraging shortcut cues. **Right:** a hard positive pair produced by a meaning-preserving variant (e.g., a paraphrase of the caption or another image describing the same caption), teaching invariances and counteracting oversensitivity from training only on hard negatives.</figcaption></figure>

### 3.1.1 Rule/Template-Based

The most direct way to improve a VLM’s compositional understanding is to create challenging contrasts with simple, deterministic edits to existing captions. These rule/template methods apply type-consistent operations (e.g., swapping nouns with nouns or attributes with attributes) and are often guided by Part-of-Speech (POS) or phrase spans from tools like spaCy [53, 54, 21]. Specifically, the edits include: (i) within-sentence noun–noun swaps to change entity relationships [53]; (ii) POS-guided mask-and-fill where masked nouns, verbs, or adjectives are refilled by a masked language model (such as BERT or RoBERTa) to generate close-in-form but contradictory captions [53]; (iii) rule-based attribute replacements drawn from predefined lists (e.g., color, size, material, spatial relations) so the negative differs by exactly one word [21]; and (iv) category-consistent word swaps across paired sentences using spaCy-identified linguistic categories [54]. These edits are cheap to scale and easy to apply, producing tightly matched negative variants that directly target attributes, relations, and word–category bindings, thereby pushing models beyond keyword matching toward sensitivity to structure.

In practice, these edits naturally split into two complementary strategies. The *text-only* branch perturbs captions while keeping pixels unchanged, so the model must distinguish the true description from near-miss paraphrases and role or attribute edits. The *image-aware* branch strengthens image–text coupling by adding visual distractors or changing the image such as masking key regions, or perturbing object attributes/positions, making text-only priors less useful.

**Text-only perturbations:** Early work relies on the least expensive strategy: change the caption, not the image, and construct near-miss text alternatives that a bag-of-words matcher should not ignore. This “text-only pressure” strategy crystallizes in **NegCLIP** [10], a fine-tuning approach designed to mitigate the “keyword overlap” shortcut (a correct and an incorrect caption share the same main words) by training on composition-aware hard negatives. It generates textual hard negatives by permuting compositional elements within a caption, such as swapping nouns or verbs, to test attribute, relation, and order understanding (e.g., “The horse is eating the grass and the zebra is drinking the water” becomes either “The zebra is eating the grass and the horse is drinking the water” (noun swapping) or “The horse is drinking the grass and the zebra is eating the water” (verb-phrase swapping)). In addition, as a minor innovation, it mines visual hard negatives by adding an image’s nearest neighbors (based on CLIP similarity) to the training batch, pushing the model to discriminate among closely related images. This forces the model to learn fine-grained differences between similar concepts, improving compositional understanding beyond a simple bag-of-words strategy. However, as the authors noted, this strategy alone does not seem to be enough to efficiently train model to deal with relation and word order.

Rather than relying on a single swap heuristic, **Structured Vision & Language Concepts (SVLC)** [21] takes a broader, automated, text-centric route: it takes existing captions and synthesizes diverse alternatives to strengthen SVLC
<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4: Hard-negative caption generation strategies. Starting from an original caption ("A gray cat sits on top of a plastic chair near a plant"), (1) *rule-based negatives* use POS parsing (spaCy) with type-consistent replacements and masked-LM (BERT) fill; (2) *graph-based negatives* use a scene-graph parser to swap roles (subject/object) or exchange attributes across objects; and (3) *LLM-prompted negatives* (e.g., LLaMA) produce fluent, minimally altered captions via in-context prompts. *Adapted and modified from* [21].</figcaption></figure>

understanding. The method generates three types of alternative text examples from the original captions to create a richer training signal. First, it creates rule-based negatives by swapping words from a predefined list (e.g., changing the attribute "a brown dog" to "a yellow dog"). Second, it generates BERT-based negatives using a mask-and-fill technique: it hides a keyword (e.g., "a cat sits on a [MASK] chair") and lets a masked language model (e.g., BERT) fill in a plausible but different word (e.g., "wooden"). Third, it creates positive analogies by prompting a strong LLM to paraphrase the original sentence, producing a new sentence with different words but the same meaning (e.g., "a woman to the left of a cat" becomes "a cat to the right of a woman"). These examples are used with a multi-part loss (see Section 3.2.1) that forces the model to distinguish the original caption from hard negatives.

**From text to image:** Text-only modifications can bias models toward language cues; to mitigate this, recent work introduces matched perturbations on the visual side. Yet in many studies, images are not subjected to hard-negative semantic augmentation comparable to text, largely because making controlled semantic edits at the pixel level is far
more difficult than perturbing sentences or words. Without relying on external image synthesis tools, a smaller body of
work addresses image-side hard negatives with three strategies: methods that select adversarial samples from existing
image samples [55], methods that modify visual embeddings to reflect hard-negative caption edits [53], and methods
that edit discretized visual tokens [56].

**SimCLIP** [55] improves hard-negative sampling by constructing mini-batches that combine random examples with clusters of similar images. Each batch of size *N* contains a proportion (1 - *p*) of uniformly sampled examples and a proportion *p* organized into *n*<sub>cluster</sub> = *p*/*k* clusters of size *k*. To form each cluster, the method selects a random anchor, identifies its nearest neighbors in embedding space via cosine similarity, and samples *k* - 1 neighbors to group with the anchor. This strategy yields batches rich in both easy contrasts (random pairs) and hard contrasts (within-cluster pairs), thereby exposing the model to challenging positives and negatives during contrastive training.

While SimCLIP *selects* difficult images, **AHNPL** [53] *constructs* visual hard negatives aligned to textual edits without generating images. It proceeds in two steps: (1) *Textual hard negatives*: from a caption $y_{\text{orig}}$, create $y_{\text{neg}}$ by swapping two nouns (altering roles/relations) and by masked-word replacement with RoBERTa (similar to SVLC). (2) *Visual hard negatives in embedding space*: with frozen encoders $f_{\theta}, g_{\theta}$, compute $\tilde{t}_{\text{orig}} = g_{\phi}(y_{\text{orig}})$, $\tilde{t}_{\text{neg}} = g_{\phi}(y_{\text{neg}})$, and $\Delta = \tilde{t}_{\text{neg}} - \tilde{t}_{\text{orig}}$; for an image $x$ with $\tilde{v}_{\text{orig}} = f_{\theta}(x)$, set $\tilde{v}_{\text{neg}} = \tilde{v}_{\text{orig}} + \Delta$ and $v_{\text{neg}} = \tilde{v}_{\text{neg}}/\|\tilde{v}_{\text{neg}}\|_2$. This generates a visual negative semantically matched to the edited text.

While modifying representations in embedding space is efficient, it does not expose token-level structure on the visual side. **Negative Augmented Samples (NAS)** [56] addresses this limitation by quantizing patch embeddings into discrete visual tokens via a learnable *Visual Dictionary (VD)* (≈2048 codes), and subsequently applying *Negative Visual Augmentation (NVA)* directly to those tokens. Specifically, NAS (i) maps continuous patch features to codebook indices and (ii) constructs hard negative images by editing token assignments, without relying on any external image generator. For a given image-caption pair, NAS computes a weighted similarity between each local token and the global text [CLS] and image [CLS] features to identify object-related tokens, then replaces roughly 30% of these tokens with their nearest neighbors in the VD (excluding itself). The resulting token-level counterfactuals remain visually similar while being semantically altered. This generates controlled, repeatable visual hard negatives without external generators.

Finally, **Compositionally Aware Learning in CLIP (CLIC)** [54] shows that strong compositional understanding can be achieved with a lightweight data construction while freezing the image encoder. Its core innovation is a low-overhead method for generating training data. In each step, it creates a new training sample by concatenating two random images to create a composite sample. This composite image is then paired with five generated captions: four hard positives to teach semantic invariance (p1, a concatenation of the first sentence from each caption; p2, a sentence-shuffled version of p1; p3 and p4, concatenations of other sentences in order to reinforce the model’s ability to distinguish incorrect captions from correct ones, even when the incorrect caption describes a general description of the image while the correct caption describes only a specific part.) and one hard negative, which is constructed by swapping a word of the same linguistic type between the original first sentences. This strategy of generating a rich set of contrastive examples using simple concatenation of image pairs and corresponding text descriptions avoids the need for expensive external generative models.

**3.1.2 Synthetic-Based**

Rule/template-based editing changes what already exists; synthetic pipelines go further by *creating* new samples. We categorize methods by how much synthetic supervision they add and where that supervision comes from: (i) text-centric synthetic captions paired with real original images [23, 24, 57, 10]; (ii) 2D diffusion that generates counterfactual images under LLM guidance [22, 58, 59]; and (iii) 3D simulation environments that render synthetic scenes with precise control over object geometry, spatial arrangements, and dynamic interactions [60]. Later categories offer more control (and typically more cost) aiming to remove shortcuts left by earlier categories.

(i) **Text-centric synthesis over real images.** Template edits that replace nouns, attributes, or relations often produce ungrammatical or implausible strings (e.g., "The horse is eating the grass" → "The grass is eating the horse"), weakening hard negatives. To address this limitation, researchers employ *LLM*-driven caption generation while keeping images fixed. In-context prompting lets LLMs create fluent paraphrases and minimal, controlled semantic changes, thereby generating negative (and positive) captions that better reflect the intended edits and provide cleaner supervision for compositional evaluation. A natural entry point is **Dense and Aligned Captions (DAC)** [23], which addresses two fundamental problems with web-crawled image-text pairs: (i) *poor alignment*, where captions only loosely match their images, and (ii) *low density*, where typically short captions neglects salient entities, attributes, and relations. DAC's approach is straightforward: first, improve the base caption using a strong image-grounded captioner (e.g., BLIP2 [48]) to ensure it accurately describes the image. Then, expand this single caption into richer supervision through two complementary strategies. The first strategy, *DAC-LLM*, is text-only: an LLM (e.g., GPT-Neo [61])
takes the improved caption and generates detailed, related sentences. Since the LLM never sees the image, it can add plausible but potentially incorrect details, a trade-off that enriches linguistic diversity while accepting some hallucination risk. The second strategy, *DAC-SAM*, is image-grounded: a segmentation model (e.g., SAM [62]) divides the image into meaningful regions, and each region is independently captioned with BLIP2. This produces a dense "bag" of region-level descriptions, each tightly tied to visual evidence. During fine-tuning, these multiple captions per image are handled with a *Multiple Instance Learning* [63] (MIL) objective, which treats the bag of captions as a set of positive instances for the same image. To preserve the model's generalization and prevent catastrophic forgetting, training uses parameter-efficient updates (e.g., LoRA [64]) rather than full fine-tuning.

Building on this idea, **LaCLIP** [24] tackles a different limitation in CLIP pre-training: multiple augmented views of the *same* image are repeatedly paired with the *same* caption. This asymmetry in training creates two problems:(i) the vision encoder receives little linguistic diversity for a given visual instance, and (ii) the text encoder rereads identical strings each epoch. LaCLIP introduces *symmetric text augmentation* to break this pattern. Specifically, it performs a large-scale, offline *rewriting* of the entire caption corpus using an open-source LLM (e.g., LLaMA [65]) prompted with few-shot in-context examples to generate diverse paraphrases for each original caption. Then, during the standard CLIP training loop, every augmented view of an image samples its caption from a pool containing the original plus its LLM-generated variants. By varying the language, LaCLIP provides richer textual supervision to the image encoder and prevents the text encoder from overfitting to a single phrasing, thus strengthening image-text binding without modifying the backbone.

Having introduced textual *variety*, the next question is how to turn that variety into a supervision signal that teaches *what should* and *should not* change the label. **HP+HN** [57] adopts balanced contrast: train with both hard positives and hard negatives to avoid the oversensitivity that arises when training on negatives alone. Hard positives are generated by prompting LLAMA with two operations: REPLACE (replace one word with a synonym) and SWAP (swap phrases around “and” while preserving meaning). Hard negatives are taken from the CREPE procedure [5] using corresponding REPLACE and SWAP edits that flip meaning. Trained jointly, these samples teach the intended invariances (synonyms, permissible swaps) versus sensitive changes (meaning-altering replacements or swaps).

Both rule-based swaps and synthetic based generations often miss *absence* and *negation*. **Negation CLIP** [10] complements the above by chaining *LLM* and *VLM* filters to build negation-aware supervision from existing images: (i) an LLM proposes plausible but unmentioned objects from the caption alone; (ii) a VLM (e.g., LLaVA [16]) checks the image to remove candidates that are actually present in the image; (iii) the LLM augments the caption with explicit *negated* statements for those objects judged absent. This adds targeted coverage of “not/without” cases, without creating new images.

**(ii) 2D diffusion under LLM control.** Text-only augmentation does improve compositional behavior, but it leaves open a key question: if we also introduce *visual* hard negatives (edits that change exactly one semantic factor in the image) can we further strengthen a model’s compositional reasoning? Diffusion-based pipelines answer this by using an LLM to propose minimal, linguistically plausible edits and a text-to-image editor to generate matched counterfactual images. A practical caveat is that diffusion edits can hallucinate or drift, injecting noise into supervision.

**TripletCLIP** [22] takes the first step in this direction. It moves beyond random swaps by prompting an LLM with a detailed in-context prompt to generate hard-negative captions that make semantic, compositionally meaningful changes while maintaining linguistic plausibility (e.g., from “a dog looking out from a window” to a flipped negative such as “A cat observing its owner from the window”). A pre-trained text-to-image diffusion model (e.g., SDXL-Turbo [18]) then generates hard-negative images matched to these captions, enabling joint hard negatives in both modalities.

**SPARCL** [58] follows the same LLM-to-diffusion pipeline but differs in two ways: it generates *both* a synthetic positive and a synthetic negative that each change a *single* word/phrase (e.g., surfboards → shovels, male → female); and it preserves scene style and layout by injecting features from original image into the post-[EOS] “style” portion of the prompt (while pre-[EOS] tokens control content), with optional AdaIN [66] for added fidelity. **Counterfactual Sets** [59] keeps the LLM-to-diffusion backbone as well but frames generation as set-structured supervision. An LLM parses the original caption into entities, attributes, and relations (e.g., from “a white dog on the left of a black cat” it produces two entities and a spatial relation), then composes counterfactual prompts by adding, deleting, or modifying exactly one element. Similar to previous methods, a diffusion model generates corresponding counterfactual images. Outputs are grouped into sets (original, change-element, delete-element), which provide structured, set-level supervision and difficulty control. This provides *sets of counterfactual samples*, whereas TripletCLIP [22] focuses on per-example triplets without explicit set grouping.

**(iii) 3D simulation.** SyViC [60] takes a fundamentally different approach: instead of editing existing images or captions, it generates synthetic 3D videos and static scenes from scratch using a physics-based simulation platform
called ThreeDWorld (built on Unity). This approach provides precise control over every aspect of the scene: where objects are placed, what materials and colors they have, where cameras are positioned, and how human characters move and interact. For every rendered frame, the system records rich metadata about the scene: the RGB image, depth information, segmentation masks, the 3D positions of all objects, and their attributes (color, size, material, etc.). This metadata is then fed into a grammar-based caption generator, which deterministically produces precise, factually grounded captions describing exactly what is in the scene. Because the system has complete information about object positions, relationships, and dynamics, it can generate captions with reliable spatial relations (e.g., “left of,” “behind”) and temporal descriptions (e.g., “picking up,” “walking toward”). The trade-off is the added complexity of constructing and rendering full 3D scenes, which is heavier than simpler text-only or 2D editing pipelines.

### 3.1.3 Graph-Based

Text captions have an internal structure: they compose *objects*, their *attributes*, and *relations* between objects. Graph-based methods make this structure explicit by mapping a caption *w* to a scene graph *G*(*w*) = ⟨*O*(*w*), *R*(*w*), *E*(*w*), *K*(*w*)⟩, where *O* are object nodes, *R* are relation nodes, *E* ⊆ *O* × *R* × *O* encodes (head, relation, tail) triples, and *K* ⊆ *O* × *A*(*w*) links objects to attribute nodes. Working in this space lets us generate positives and negatives by editing the graph rather than swapping isolated words: flip roles by exchanging head/tail, replace a relation *r* with its inverse edge, reassign an attribute to a different object, or add/delete a single edge, and then linearize the edited graph back to fluent text. This targets the combinatorial structure of scenes (who does what to whom; which attribute belongs to which object) and avoids the ungrammatical or nonsensical artifacts common with naive word swaps. In practice, captions are parsed into scene graphs; controlled structural edits then generate semantics-preserving subgraphs and semantics-violating graphs for training.

This idea leads to two complementary objectives. *StructureCLIP* [25] addresses the shortcomings of ad-hoc word swaps for negative construction (e.g., swapping “black” and “white” in “black and white cows” may barely change meaning yet corrupt supervision). Using a scene-graph parser [67], each caption is converted into relation triples and attribute-object pairs. Negatives for relation triples are formed by swapping the subject and object; negatives for attribute-object pairs are created by exchanging attributes across different objects. To align structured knowledge with standard text features, pairs (e.g., (white, cow)) are canonicalized into triples (cow, is, white). Entities in each triplet are tokenized and embedded (with BERT); head, relation, and tail embeddings are composed into a triple embedding and passed through Transformer layers to produce structured knowledge representations (more details on this in Section 3.3.2). The negative samples generated from scene graphs are topology-aware corruptions that intentionally violate the original scene's structure.

While *StructureCLIP* shows that graph-aware corruptions generate better negatives, it still treats the caption as a single textual positive. In real images, however, the caption typically under-specifies the scene: the image contains more entities and relations than the sentence mentions, and many subsets of the caption's content remain true for the same image. If negatives benefit from local, graph-consistent edits, then *positives* should also benefit from a compositional treatment that acknowledges correct substructures. This motivates moving beyond “one caption = one positive” toward a hierarchy of semantics-preserving subgraphs that capture valid, coarser descriptions of the same image. *MosaiCLIP* [26] starts from precisely this assumption: a caption’s scene graph is a subgraph of the (often richer) true image scene graph. Consequently, any *subgraph* of the caption is a valid, if less detailed, description of the same image. Each caption graph is decomposed into a hierarchy of subgraphs and every *subgraph* is treated as an additional *positive* text description for the paired image. Hard negatives are created using an external vocabulary of objects *N*, attributes *A*, and relations *R* via three operators: (1) *Node swap/replacement* replaces an object or attribute node with a type-consistent element from *N* or *A*; (2) *Edge replacement* substitutes a relation with one sampled from *R*; and (3) *Connecting subgraphs* attaches a randomly sampled distractor subgraph to a valid subgraph, generating implausible composites. These samples reinforce alignment at multiple hierarchies while maintaining pressure from structure-aware negatives.

#### Data-Centric: Key Takeaways

* Training data determines what a model actually learns. Hard negatives help break bag-of-words shortcuts, but *alone* they rarely teach *when* a small perturbation should or should not change the label. Meaning-preserving positives (paraphrases, analogies) are needed alongside negatives to stabilize zero-shot performance and avoid the typical drop from “negatives-only” fine-tuning.

* Compositional gains are stronger when hard negatives span *both* modalities (edited captions and matched visual counterfactuals) so that differences are grounded in vision as well as language.

* In practice, LLM- and diffusion-based generators can generate higher-quality positives and negatives than rule-based edits, but they introduce a risk of hallucination, which must be managed during data construction and training.
## 3.2 Objective-Centric

We organize objective-level innovations into three categories based on the compositional mechanism they target. First, *InfoNCE extensions* modify the core contrastive objective through: (i) *hard-negative contrastive losses* that explicitly penalize compositional near-misses by expanding the denominator or introducing binary/multiple-choice contrasts [10, 23, 9, 27, 28], (ii) *multi-positive aggregation strategies* that handle semantic equivalence when an image admits multiple valid captions [23, 26], and (iii) *local alignment objectives* that shift from global image-text similarity to token-patch correspondences [68, 69, 70]. Second, *margin-based losses* impose explicit score gaps between correct pairs and counterfactuals, with fixed or adaptive margins and set-level extensions [25, 71, 58, 59]. Third, *geometric and distillation-based objectives* either embed representations in hyperbolic space to encode semantic hierarchies [72, 73] or use teacher guidance (external generative models, exponential moving average (EMA), or iterated listeners) to preserve global alignment while enforcing compositional sensitivity [30, 31, 74]. Together, these categories demonstrate that compositional understanding can be enforced not only by *what* data the model sees, but also by *how* the objective rewards correct bindings and penalizes structural violations.

**Remark 1** Some of the approaches covered in the data-centric section are revisited here, but from a different lens. There, we analyzed them by how positives and negatives are built or edited; here, we analyze them by how they change the training objective. These two views are tightly linked: the choice of positive/negative samples and the choice of loss or scoring function directly affect one another. In this section, we therefore group methods by their objective-level changes, while noting their close ties to data design.

<figure><img src="image_7.png" /><figcaption>Figure 5: Similarity matrix layout for training with hard negatives/positives. Rows: the original image $x_1$ and its counterfactual variant $\bar{x}_1$. Columns: the original caption $y_1$, meaning-preserving positives $y^+$ (blue), and corresponding hard negatives $y^-$ (red). Green cells denote target *high* similarities: for $(x_1, y_1)$, $(x_1, \bar{y}_1)$, and for each matched pair $(\bar{x}_1, y_1)$. All other cross-pairs (e.g., $x_1$ with any $y^-$, or $\bar{x}_1$ with $y_1/y^+$) are encouraged to be *low*.</figcaption></figure>

### 3.2.1 InfoNCE Extensions

The core InfoNCE objective can be extended in several ways to improve compositional understanding. We group these extensions into three categories: hard-negative contrastive losses that expand the denominator with composition-breaking captions, multi-positive aggregation strategies that handle semantic equivalence when an image admits multiple valid captions, and local alignment objectives that shift from global to token-patch correspondences.
**(i) Hard-Negative Contrastive Losses.** The simplest way to sharpen CLIP's objective is to change who competes in the denominator. As an early objective for training with compositionally hard negatives, NegCLIP [3] proposed a **Column-Masked InfoNCE** loss, which has since been adopted by methods like TripletCLIP [22]. For each positive image–caption pair, this loss adds composition-breaking captions (e.g., attribute swaps, relation flips) to the contrastive denominator. We compute row-wise and column-wise cross-entropy as in CLIP, except we do not compute the loss for the negative captions column-wise, since a negative caption has no matching image. This forces the model to assign a higher score to the true caption than to near-miss captions that break object–attribute or object–relation bindings. In practice, this is CLIP's InfoNCE augmented with an expanded, carefully chosen set of distractor captions, improving the model's sensitivity to attributes and relations:

$$ \mathcal{L}_{\text{NegCLIP}} = -\frac{1}{B} \sum_{x_i}^{B} \log \frac{\exp(s(x_i, y_i))}{\sum_{j \in B} \exp(s(x_i, y_j)) + \sum_{y^- \in \mathcal{N}_i^{\text{neg}}} \exp(s(x_i, y^-))} $$

Several training objectives then add targeted pressure from hard negatives as a regularization term which is a light modification of CLIP's InfoNCE that compares a correct caption against one or more counterfactual (negative) captions that differ by a single, meaningful edit (e.g., number word or negation). Unlike the NegCLIP loss, these approaches regularize in addition to optimizing the standard InfoNCE objective. In the **binary (one-to-one) contrastive form** [9, 27, 28], (also the neg-Loss in DAC [23]), for each image $x_k$ with its correct caption $y_k$ and a counterfactual $\bar{y}_k$ (e.g., wrong count or negated meaning), define similarities using $s_k^+ = s(x_k, y_k)/\tau$ and $s_k^- = s(x_k, \bar{y}_k)/\tau$. The binary contrastive loss uses a two-way softmax:

$$ \mathcal{L}_{\text{bin}} = -\frac{1}{B} \sum_{k=1}^{B} \log \frac{\exp(s_k^+)}{\exp(s_k^+) + \exp(s_k^-)} $$

Very closely related, the **multiple-choice (one-to-many) form** [10] presents C hard-negatives $\{\bar{y}_{i,1}, \dots, \bar{y}_{i,C}\}$ for each image $x_i$ and requires the correct positive caption $y_i$ to dominate:

$$ \mathcal{L}_{\text{mcq}} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(s(x_i, y_i))}{\exp(s(x_i, y_i)) + \sum_{j=1}^{C} \exp(s(x_i, \bar{y}_{i,j}))} \quad (1) $$

By explicitly contrasting correct captions against near-miss counterfactuals that differ by a single semantic element, these losses force the model to distinguish between correct compositions and incorrect ones, preventing reliance on independent word matching.

**(ii) Multi-Positive Aggregation Strategies.** When an image admits multiple semantically equivalent captions (paraphrases, subgraphs, or valid subtexts), a CLIP-style objective must decide how to aggregate these co-positives. We group common choices by the numerator (what is rewarded) and make the corresponding denominator explicit to show what competes in the softmax. Let $\mathcal{B}$ denote the current minibatch of images and texts, and let $\mathcal{P}_i \subseteq \mathcal{B}$ be the set of co-positives (valid captions) for image $x_i$.

**(A) Single sampled positive (InfoNCE unchanged).** Equivalence is handled implicitly by sampling one valid caption per image and applying standard symmetric InfoNCE [24, 75, 76]:

$$ \mathcal{L}_{\text{sample}}^{I \to T} = -\frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \log \frac{\exp(s(x_i, \tilde{y}_i)/\tau)}{\sum_{y \in \mathcal{B}} \exp(s(x_i, y)/\tau)}, \quad \tilde{y}_i \sim \text{Uniform}(\mathcal{P}_i). $$

This approach treats multiple valid captions as equivalent by randomly selecting one per image each training step. The numerator rewards the similarity between image $x_i$ and the randomly chosen caption $\tilde{y}_i$, while the denominator includes all captions in the batch as competitors. Over many training steps, different valid captions are sampled, but at any single step, only one positive is supervised. This maintains CLIP's standard loss function but provides weak supervision for the equivalence between valid captions.

**(B) Average of logs over all positives (every positive supervised).** Equivalence is made explicit by supervising each positive and averaging their log-softmax terms; the denominator typically spans all texts in the batch [26]:

$$ \mathcal{L}_{\text{avg-log}}^{I \to T} = -\frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \frac{1}{|\mathcal{P}_i|} \sum_{y \in \mathcal{P}_i} \log \frac{\exp(s(x_i, y)/\tau)}{\sum_{y' \in \mathcal{B}} \exp(s(x_i, y')/\tau)} $$
Unlike (A), this method explicitly supervises every valid caption for each image. The inner sum iterates over all co-positives $\mathcal{P}_i$, computing a separate log-softmax term for each one. These terms are then averaged (the $\frac{1}{|\mathcal{P}_i|}$ factor). The numerator rewards each valid caption individually, while the denominator includes all batch captions as competitors. This ensures the model learns to assign high similarity scores to all valid phrasings, not just one randomly selected variant.

**(C) Log of sum over positives (bag-of-positives).** Here the numerator first pools positive scores and then applies a single log. A multiple instance learning style variant contrasts pooled positives against a bag-aware denominator along with counterfactuals $\mathcal{C}_i$ for each image $x_i$ [23]:

$$ \mathcal{L}_{\text{MIL}}^{I \to T} = -\frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \log \frac{\sum_{y \in \mathcal{P}_i} \exp(s(x_i, y)/\tau)}{\sum_{j \in \mathcal{B}} \sum_{y \in \mathcal{P}_j} \exp(s(x_i, y)/\tau) + \sum_{c \in \mathcal{C}_i} \exp(s(x_i, c)/\tau)}. $$

This approach treats the set of valid captions as a unified "bag" of positives. The numerator sums the similarity scores for all co-positives before applying the log, effectively rewarding the model if any valid caption has high similarity. The denominator is more complex: it includes all captions from all images in the batch (the double sum over $j \in \mathcal{B}$ and $y \in \mathcal{P}_j$), plus any counterfactual negatives $\mathcal{C}_i$ specific to image $x_i$. This bag-aware denominator ensures that the pooled positive scores compete against both other images' captions and explicit hard negatives, providing stronger compositional supervision.

Each option biases learning differently: (A) keeps standard CLIP by randomly picking one positive; (B) pushes the model to score all valid phrasings high; (C) rewards the whole set of valid captions at once.

**(iii) Local Alignment Objectives.** Global contrastive training aligns entire images with entire captions, which can miss fine structure because many captions share similar global semantics even when local details differ. To overcome global similarity, **Fine-grained Selective Calibrated CLIP (FSC-CLIP)** [68] fine-tunes CLIP with a loss that looks at token-patch pairs, following [69]. For each word in the caption, the model checks which image patches it should align with and learns from hard negatives generated via rule-based caption edits. Given image patch features $V = \{v_p\}_{p=1}^P$ and text token features $T = \{t_w\}_{w=1}^W$, it constructs a token-patch similarity map $s_{w,p} = \cos(t_w, v_p)$ for every word $w$ and patch $p$. Each row is min-max normalized to produce nonnegative attention/similarity weights $a_{w,p} = (s_{w,p} - \min_k s_{w,k}) / (\max_k s_{w,k} - \min_k s_{w,k})$. These weights induce a text-aligned visual representation for each word via a weighted average $\hat{v}_w = (\sum_p a_{w,p} v_p) / (\sum_p a_{w,p})$. Local image-text agreement is then quantified by comparing each $\hat{v}_w$ to its corresponding token $t_w$ with cosine similarity:

$$ S_l(I, T) = \sum_{w=1}^{W} \exp(\cos(\hat{v}_w, t_w) / \tau). $$

This local similarity $S_l$ measures how well individual words align with their corresponding image regions, in contrast to global similarity which measures how well the entire image matches. In addition to the standard CLIP loss $\mathcal{L}_{\text{CLIP}}$, FSC-CLIP uses both global and local hard negative losses. The global hard negative (GHN) loss, $\mathcal{L}_{\text{neg}}^g$, uses standard CLIP's whole-image-to-whole-caption similarity (as in Eq. (1)) to contrast the reference caption against hard negatives. The local hard negative (LHN) loss, $\mathcal{L}_{\text{neg}}^l$, has the same form, except it replaces the global similarity with the token-level similarity $S_l$, forcing the model to distinguish correct word-to-patch alignments from incorrect ones.

To prevent the hard negative losses from degrading the pretrained model's capabilities, FSC-CLIP applies Selective Calibrated Regularization (SCR) to both $\mathcal{L}_{\text{neg}}^g$ and $\mathcal{L}_{\text{neg}}^l$. SCR combines two techniques: (i) focal loss, which reduces the weight of easy examples (where the model is already confident) to focus training on harder cases, and (ii) label smoothing, which softens the hard negative labels by assigning them small positive margins rather than treating them as completely incorrect. This prevents the model from overfitting to the hard negatives.

Another way to enforce fine-grained alignment is to ensure that when two text phrases refer to different objects in the same image, their corresponding image regions do not overlap [70]. This approach uses grounding heatmaps: for each text phrase, the model produces a 2D map (of size $h \times w$) where each pixel value indicates how strongly that image location is associated with the phrase. Higher values mean stronger association. Given a training batch $\mathcal{B} = \{(\mathbf{v}^i, \mathbf{t}_0^i, \mathbf{t}_1^i)\}_{i=1}^n$ where each image $\mathbf{v}^i$ is paired with two text phrases $(\mathbf{t}_0^i, \mathbf{t}_1^i)$ that refer to distinct objects, let $H_0, H_1 \in \mathbb{R}_{\ge 0}^{h \times w}$ be the grounding heatmaps for the two phrases. The complementary uniqueness loss penalizes overlap between these heatmaps:

$$ \mathcal{L}_{\text{CU}} = \mathbb{E}_{(v, t_0, t_1) \sim \mathcal{B}} \left[ \frac{1}{w \cdot h} \sum H_0 \odot H_1 \right], \qquad (2) $$
where ⊙ denotes element-wise multiplication (Hadamard product). This loss is minimized when the heatmaps have high values in different locations, that is, when $H_0$ is high where $H_1$ is low, and vice versa.

### 3.2.2 Margin-Based Losses

While methods in the previous section reweight competition and aggregation, margin-based objectives [25, 71] impose explicit score gaps between truth and counterfactuals. Let $s(x, y)$ be the similarity, $N_i$ the counterfactual set for $(x_i, y_i)$, and $[·]_+ = \max(0, \cdot)$, hinge loss is defined as:

$$ \mathcal{L}_{\text{hinge}} = \frac{1}{B} \sum_{i=1}^{B} \frac{1}{|N_i|} \sum_{y^- \in N_i} [m + s(x_i, y^-) - s(x_i, y_i)]_+ \quad (3) $$

This loss assures that role reversals and relation flips must sit at least $m$ below the true pair. Because hard negatives vary in quality, SPARCL [58] uses a per-example adaptive margin: the margin is large if the positive and negative are already well separated, and small if they are close (i.e., truly hard), then combines this ranking with a contrastive term to focus on informative pairs and avoid overfitting to noisy ones.

Counterfactual supervision [59] organizes samples into counterfactual sets built around each real image-text pair $(x^0, y^0)$. Sets are constructed as follows. Given a dense caption (e.g., “Black dog on the right of yellow cat.”), an LLM first extracts entities, attributes, and spatial relations, then generates counterfactual captions via three operations: adding a non-existent object (e.g., introducing “a beautiful white cat with a smile on the right of the black dog”), modifying an existing object (e.g., changing the caption to “a black cat on the left of a black dog”), and deleting an existing object (e.g., generating “a yellow cat sitting on green lawn”). A block-based diffusion pipeline then generates corresponding counterfactual images that match each modified caption, ensuring that each counterfactual variant is a properly matched image-text pair. This creates a structured set where each pair $(x_i, y_i)$ is either the original, a change-element variant, or a delete-element variant, with images and captions matched within each pair. Within a set, labels are assigned to all image-text pair combinations $l_{ij} \in \{+1, -1\}$: $l_{ij} = +1$ (positive pairs) when image $x_i$ is correctly matched with its corresponding caption $y_j$ (e.g., original image with original caption, change-element image with change-element caption); $l_{ij} = -1$ (negative pairs) when an image from one variant is mismatched with a caption from a different variant (e.g., original image with change-element caption). The intra-set and inter-set soft-margin objectives are defined:

$$ \mathcal{L}_{\text{intra}} = - \sum_{i=1}^{m} \sum_{j=1}^{m} \log \left( \frac{1}{1 + \exp(-\tau s(x_i, y_j) + b)} \right), $$

and, across $n$ original pairs $(x_i^0, y_i^0)$, by treating $y_j^0$ from $j \neq i$ as negatives results in:

$$ \mathcal{L}_{\text{inter}} = - \sum_{i=1}^{n} \sum_{\substack{j=1 \\ j \neq i}}^{n} \log \left( \frac{1}{1 + \exp(\tau s(x_i^0, y_j^0) - b)} \right). $$

Both $\mathcal{L}_{\text{intra}}$ and $\mathcal{L}_{\text{inter}}$ are soft-margin objectives. Writing $z_{ij} = l_{ij}(\tau s(x_i, y_j) - b)$ with $l_{ij} \in \{+1, -1\}$, the per-pair loss is the logistic

$$ \ell_{ij} = \log(1 + \exp(-z_{ij})). $$

Because $\ell_{ij}$ is strictly decreasing in $z_{ij}$, it is minimized by making $z_{ij}$ large. For a positive pair ($l_{ij} = +1$), this means increasing $\tau s(x_i, y_j) - b$ and for a negative ($l_{ij} = -1$), it means decreasing $\tau s(x_i, y_j) - b$. Thus $b/\tau$ acts as an effective margin (threshold) in similarity space: positives are encouraged to satisfy $s(x_i, y_j) > b/\tau$, while negatives are encouraged to satisfy $s(x_i, y_j) < b/\tau$. Increasing $b$ (offset) widens the required gap; increasing $\tau$ (scale) steepens the transition, making the loss closer to a hard-margin hinge. In this sense, the set-wise objective enforces margin constraints within each counterfactual set (via $\mathcal{L}_{\text{intra}}$) and between sets (via $\mathcal{L}_{\text{inter}}$), with the log-loss providing a smooth surrogate for classic hinge/triplet margins.

### 3.2.3 Geometric and Distillation-based Objectives

Recent research suggests that concepts are not all equally related. For example, ‘cat’ is more general than ‘cat and dog playing’. This hierarchy can be captured more naturally by arranging concepts in geometric spaces that reflect their relationships, instead of treating them as points that are all equally far apart. However, in the standard Euclidean setup, image and text embeddings are $\ell_2$-normalized and compared with a uniform metric that treats all points alike, making it hard to keep generic concepts (e.g., “cat”) broadly connected while leaving specific ones (e.g., “cat and dog playing”) only selectively close. Hyperbolic spaces, by contrast, expand volume exponentially with radius, acting as continuous
relaxations of trees: placing generic concepts near the origin and specific ones farther away generates neighborhoods where distinct specifics are far apart yet all remain close to a common generic ancestor. Desai et al. [72] introduce a framework, namely **MERU**<fnref n="3" />, for embedding image-text representations in a high-dimensional hyperbolic space instead of the commonly used Euclidean space with normalized vectors. MERU maps image and text embeddings into the Lorentz (hyperboloid) model [77] and learns the curvature. The training is similar to CLIP-style contrastive learning but makes two design choices: (i) transfer encoder features to the hyperboloid and measure similarity via *negative Lorentzian distance*, and (ii) add an *entailment* objective that imposes a partial order in which *text entails image*<fnref n="4" />. MERU uses hyperbolic entailment cones whose *aperture narrows with radius*: broader concepts lie nearer the origin and *geometrically contain* their more specific counterparts farther out. Experiments show that hyperbolic representations lead to more efficient and interpretable embeddings than Euclidean counterparts, performing competitively or better across retrieval and classification while enabling compact embeddings suitable for resource-constrained settings. Hyperbolic Compositional CLIP (HyCoCLIP) [73] generalizes MERU's hyperbolic alignment from global image-caption pairs to a compositional setting covering parts and wholes across modalities. It embeds whole images, localized regions (boxes), full captions, and noun phrases in a single hyperbolic space, enabling consistent comparison among heterogeneous units.

Recent objectives could be analyzed through the lens of *teacher-informed fine-tuning*: a teacher (that could be external, iterated, or EMA) guides fine-tuning so the student preserves global alignment while gaining sensitivity to attributes, parts, and relations. Based on a generative prior, Score-Distillation Sampling-CLIP (SDS-CLIP) [30] regularizes CLIP's contrastive loss with a score-distillation term computed from a *frozen* text to image diffusion model (teacher). Let $f_\theta(x)$ be the student image encoder for image $x$, $h_w$ a small projection head to the diffusion's UNet input space, and $\epsilon_\omega(\cdot, t, y)$ the teacher UNet's noise predictor at step $t$ for caption $y$. The SDS regularizer penalizes the teacher's denoising MSE on student-induced latents:

$$ \mathcal{L}_{\text{SDS}} = \mathbb{E}_{t, \epsilon \sim \mathcal{N}(0, I)} \| \epsilon_{\omega}(h_w(f_{\theta}(x)), t, y) - \epsilon \|_{2}^{2}. \quad (4) $$

With the diffusion model frozen; typically only $\theta$ (CLIP's LayerNorm parameters) and $h_w$ (projection head's parameters) are updated. Using this *distillation*, CLIP is nudged toward the teacher's supervision for the *same* $(x, y)$ pairs, transferring the generative diffusion's fine-grained structure (attributes/relations) into the CLIP's representation.

Decoupled Global-Local Alignment (DeGLA) [31] is based on self-distillation strategy. It separates training into a *global* path that preserves CLIP's zero-shot geometry and a *local* path that targets compositional distinctions using hard negatives. It maintains an EMA teacher with parameters $\theta$. Let $(v_i, t_i)$ be the student's normalized embeddings and $(v_i^{\text{EMA}}, t_i^{\text{EMA}})$ the teacher's. The self-distillation loss is

$$ \mathcal{L}_{\text{SD}} = \|v_i - v_i^{\text{EMA}}\|_2^2 + \|t_i - t_i^{\text{EMA}}\|_2^2. \quad (5) $$

Using LLM-generated hard negatives, DeGLA adds two contrastive losses:

1. **Image-Grounded Contrast Loss (IGC):** pull each image embedding toward its positive caption and push it away from hard-negative captions $\{y_{i,1}, \dots, y_{i,c}\}$ using (1) on $(x_i, y_i)$.

2. **Text-Grounded Contrast Loss (TGC):** strengthen the text encoder's discrimination by contrasting $(y_i, y_i^{\text{EMA}})$ against EMA-derived hard-negative captions $\{y_{i,1}^{\text{EMA}}, \dots, y_{i,c}^{\text{EMA}}\}$, also using (1).

The overall objective combines all three with CLIP's original InfoNCE loss function. In fact, the EMA constraint acts as a “safety rail” for global zero-shot ability, while IGC/TGC inject fine-grained attribute/relation discrimination.

IL-CLIP [74] keeps the standard CLIP InfoNCE but imposes a *shared, sparse codebook* bottleneck: both modalities form representations as sparse mixtures of code vectors. Training proceeds in *generations*: periodically reset the language agent (new listener), briefly *distill* it under the previous generation with the codebook frozen, then resume joint contrastive learning with the codebook unfrozen. This iterated teacher→student pressure functions as *implicit self-distillation*, pushing the interface toward simpler, more compositional codes without introducing a separate auxiliary loss.

<footnote n="3">³Meru is a mountain that symbolizes the center of all physical, metaphysical, and spiritual universes in Eastern religions like Hinduism and Buddhism. The method is named MERU because the origin of the hyperboloid entails everything and plays a more vital role than in Euclidean spaces.</footnote>

<footnote n="4">⁴In MERU's hyperbolic setup, “text entails image” encodes an asymmetric relation: if a caption is true description of an image, then the image embedding must lie inside the caption's entailment cone in hyperbolic space. Captions are treated as more general (nearer the origin, with wider cones) and images as more specific (farther out). A general caption like “a dog” has a broad cone that should contain many dog images; a specific caption like “a brown dog chasing a frisbee” sits farther from the origin with a narrower cone that contains only matching images. Training combines contrastive alignment with an entailment regularizer that penalizes violations of this containment, thereby creating a hierarchical notion of similarity into the representation.</footnote>
## Objective-Centric: Key Takeaways

* Compositional objectives work by controlling who competes and how big the win must be: adding composition-breaking captions to the denominator (with no paired positives) makes near-miss edits easily lose.
* When an image has many correct captions, there are two options with distinct biases: (A) average over all positives that supervise each paraphrase so the model learns to score every valid wording high (more robust to phrasing); (B) bag of positives that sum positive scores first and only require the group to beat negatives, letting the model lean on whichever paraphrase is easiest (sharper gradient, faster, but can neglect rare phrasings).
* Hyperbolic embeddings reveal that geometry is not just a representational choice but an inductive bias: placing generic concepts near the origin and specific ones farther out allows the same model to respect hierarchical relations across captions, boxes and phrases. **But note that: without high-quality hard examples from data-centric methods, these objectives have little to act on; with them, they amplify the gains.**

## 3.3 Architecture-Centric

While data-centric and objective-centric approaches address compositional understanding through specialized training signals, a fundamental limitation remains: the standard dual-encoder architecture itself lacks the inductive biases necessary for compositional reasoning [33]. The dual encoder CLIP architecture compresses entire images and texts into single global vectors, creating a “bag-of-words” representation that collapses spatial structure, weakens attribute-object binding, and fails to capture relational dependencies [25]. This architectural bottleneck, that is, the global cosine similarity between image and text embeddings, becomes the primary obstacle to compositional understanding, regardless of how sophisticated the training data becomes.

We reveal a clear progression from simple similarity refinements to complex structural modifications, showing how each approach incrementally injects compositional biases that the original architecture lacks. The methods fall into three complementary categories that tackle different aspects of the same fundamental problem: (1) **similarity refinement** that modifies how similarity is computed through light-weight alignment heads [32, 33]; (2) **modular architecture** that introduces explicit compositional components [78, 25, 79, 36]; and (3) **representation factorization** that disentangles embeddings into compositional primitives [80]. Together, these approaches demonstrate that architectural modifications can systematically address CLIP's compositional limitations by replacing global co-occurrence patterns with structured, locality-aware reasoning mechanisms.

### 3.3.1 Similarity Refinement

The most straightforward way to address CLIP's limitations begins with noting that global cosine similarity collapses structure by pooling relevant and irrelevant evidence alike [32]. As a result, when aligning images with text, details that matter (e.g., a mug handle) become mixed with background clutter, degrading the model's ability to ground specific textual references [78]. Alignment head methods resolve this by directing similarity computations through the appropriate evidence so that only relevant image-text correspondences influence the final score. These methods employ two complementary strategies: (i) *selective readout* [32] that gates which embedding dimensions matter for specific captions; and (ii) *alignment head* [33] that transforms embeddings to reduce modality gaps. While these approaches differ in their technical implementation, they share a common insight: the global cosine bottleneck can be overcome by making similarity computation more selective and structured.

**SmartCLIP** [32] introduces a caption-conditioned gating mechanism that selectively activates only the parts of an image embedding relevant to a given description. As illustrated in Fig. 6, a caption *y* with embedding *t* is passed through a lightweight mask network that produces a sparse binary gate $m(t) \in \{0, 1\}^d$ using the *straight-through estimator* [81]. The left side of the figure shows that different captions for the same image induce different gating patterns, activating only the coordinates corresponding to concepts mentioned in the caption. The right side depicts how this gate participates in the training procedure: SmartCLIP applies the symmetric InfoNCE objective to the masked similarity $sim(v \odot m(t), t)$, using both an image-anchored loss (fix an image, vary captions) and a text-anchored loss (fix a caption, vary images). Positive pairs are encouraged to align, negative pairs are pushed apart, and an additional $\ell_1$ penalty encourages sparsity in *m*. This training setup mitigates information misalignment and entangled representations, enabling fine-grained and concept-aware visual-textual correspondence.

A second line of work targets the modality gap between image and text representations. **LABCLIP** [33] performs a systematic probing analysis to reveal the source of CLIP's compositional failures. Its core contribution is training linear probes on top of CLIP's frozen uni-modal encoders to extract object-specific attribute information. Across synthetic benchmarks such as CLEVR [82] and PUG variants [83], these probes achieve near-perfect accuracy ($\approx$ 0.95–1.00), showing that CLIP's image and text encoders already linearly encode the relevant attribute-object bindings. The probing
<figure><img src="image_8.png" /><figcaption>Figure 6: **Selective readout in SmartCLIP.** Left: different captions produce different sparse gates over image features. Right: these gated embeddings drive positive/negative separation under SmartCLIP's contrastive training. Figure from [32]</figcaption></figure>

results reveal a consistent pattern: CLIP performs strongly on uni-modal compositional tasks but fails on cross-modal ones, such as matching images to captions with swapped attributes. This indicates that compositional structure is present in the representations but becomes inaccessible during cross-modal alignment, where cosine similarity collapses fine-grained binding cues. LABCLIP hypothesizes that a lightweight transformation can mitigate this misalignment.

Accordingly, LABCLIP trains a simple linear map $W \in \mathbb{R}^{D \times D}$ applied to text embeddings before similarity computation, $\text{sim}(Wt, v)$, while keeping both encoders frozen. Training uses contrastive supervision with synthetic negatives created by permuting attribute-object associations in captions; for COCO, LABCLIP follows NegCLIP [10] and constructs such negatives by shuffling nouns and adjectives.

### 3.3.2 Modular Architecture

While alignment head methods refine how similarity is computed, they still operate within CLIP's fundamental dual-encoder framework. A more radical approach involves injecting explicit structural components, nodes, edges, slots, or causal dependencies that force the model to reason about compositional relationships rather than relying on global co-occurrence patterns. These modular approaches differ fundamentally from alignment heads by introducing new architectural elements that encode compositional structure directly into the model's architecture.

The modular component approach can be organized along three complementary dimensions: (i) *structured similarity computation* that replaces global cosine similarity with multi-component or slot-based matching [78, 35]; (ii) *explicit graph supervision* that uses scene graphs to guide training while preserving the base architecture [25, 79]; and (iii) *structured decoders* that replace global similarity with factorized, causally ordered likelihoods [36]. Each approach represents a different trade-off between architectural complexity and compositional capability, with more sophisticated methods achieving better compositional understanding at the cost of increased computational overhead.

(i) **Structured similarity computation** replaces global cosine similarity with multi-component or slot-based matching that decomposes both image and text into structured components (entities, relations, slots) and computes similarity at multiple levels rather than collapsing everything into a single global vector.

**ComAlign** [78] introduces fine-grained, structure-aware alignment by decomposing both modalities into global, entity, and relation components, as illustrated in Fig. 7. Each caption is parsed into textual entities and relations, while an object detector proposes visual entity boxes and relation regions. These matched components are encoded by the frozen base VLM and concatenated into an ordered sequence for each modality:

$$[g, e_{1..N}, r_{1..M}],$$

where *g* denotes the global feature, *e*<sub>*i*</sub> the entity components, and *r*<sub>*j*</sub> the relation components.

As shown in the Fig. 7, two shallow transformers refine these component features and enable similarity computation at multiple levels. For entities and relations, ComAlign constructs cross-component cosine-similarity matrices and applies the FILIP-style procedure [84]: taking row-wise maximum followed by averaging to produce entity-level and
<figure><img src="image_9.png" /><figcaption>Figure 7: ComAlign performs structured, component-level alignment by constructing entity- and relation-level similarity matrices (left) and matching visual components to textual entities and relations (right). Adapted from [78].</figcaption></figure>

relation-level scores. The global term is computed via a standard dot product between global embeddings. These three scores form a multi-level contrastive objective, and at inference, ComAlign fuses the base model's global similarity with the entity- and relation-level terms using hyperparameters $\alpha_1$ and $\alpha_2$.

**OC-CLIP** [35] augments CLIP with an object-centric binding module that injects explicit entity and relation structure into the alignment stage. A caption is first parsed into a scene graph

$$G = (\{N_i\}_{i=1}^M, \{(r_i, s_i, o_i)\}_{i=1}^P),$$

where $N_i$ denotes the textual embedding of the $i$-th object (a node), and each triple $(r_i, s_i, o_i)$ contains a relation embedding $r_i$ together with the indices of its subject and object nodes ($s_i$ and $o_i$).

On the image side, OC-CLIP produces a set of visual slots $\{S_i\}$ designed to correspond to the textual objects. The slots are obtained via inverted cross-attention, using object nodes $N_i$ as queries and ViT patch embeddings as keys and values, so that each slot $S_i$ pools evidence relevant to a specific object. Extra “default” queries absorb any visual content not mentioned in the caption.

OC-CLIP then replaces CLIP's global cosine similarity with a structured score that combines object-level and relation-level terms. For each object, it computes a cosine match between its textual embedding $N_i$ and its visual slot $S_i$. For each relation $(r_i, s_i, o_i)$, OC-CLIP mixes the relation vector with its subject and object slots via lightweight MLPs $f_s$ and $f_o$, producing relation-aware features that are compared by cosine. The final similarity is defined as

$$S(x, G) = \frac{\alpha \sum_{i=1}^{M} \cos(N_i, S_i) + \beta \sum_{i=1}^{P} \cos(r_i, f_s([r_i, S_{s_i}]) + f_o([r_i, S_{o_i}]))}{\alpha M + \beta P},$$

where $\alpha$ and $\beta$ weight object- and relation-level contributions, and $[\cdot, \cdot]$ denotes concatenation. This structured similarity replaces the global cosine score while keeping both image and text encoders frozen.

Training follows a CLIP-style symmetric InfoNCE objective, using $S(x, G)$ in place of the global score. An auxiliary relation-loss further supervises the binding of subjects, objects, and relations by penalizing swapped or randomized graph structures.

(ii) **Explicit graph supervision** involves using scene graphs as guiding information while preserving CLIP's core architecture. Two complementary strategies demonstrate how graph structure can guide training without requiring major
architectural changes: data-side supervision that creates harder, structure-aware training examples, and model-side supervision that adds lightweight prediction heads.

**Structure-CLIP** [25] addresses CLIP's compositional failures by injecting scene-graph knowledge through two complementary components, illustrated in Fig. 8. The first component (Fig. 8a) creates semantic hard negatives by transforming captions with role/attribute flips that remain linguistically plausible (e.g., "astronaut rides horse" → "horse rides astronaut"), forcing the model to attend to graph structure during matching rather than relying on co-occurrence patterns.

The second component (Fig. 8b) introduces a Knowledge-Enhanced Encoder (KEE) that augments CLIP's text representations with explicit scene-graph triplets. Each caption is parsed into *knowledge triplets* $(h, r, t)$, whose BERT embeddings are composed into a role-aware representation,

$$e_{\text{triple}} = w_h + w_r - w_t,$$

where $w_h$, $w_r$, and $w_t$ denote head, relation, and tail embeddings, respectively. A lightweight Transformer further processes these triplet embeddings, and the resulting knowledge features are fused with CLIP's text embedding via a fixed weighting. This dual mechanism, that is, semantic hard negatives plus role-preserving textual augmentation, encourages CLIP to maintain correct compositional roles and improves robustness to role swaps and attribute confusions. For more information on the data-centric perspective of Structure-CLIP, see section 3.1.3.

**SGVL** [79] takes a more direct approach by adding lightweight graph prediction heads to pretrained VLMs. This parameter-efficient finetuning strategy injects explicit structure from small scene-graph dataset while largely preserving zero-shot behavior, demonstrating that compositional understanding can be enhanced through targeted architectural additions. The methodological innovation operates on both textual and visual side. On the *textual* side, scene graphs are transformed into fine-grained captions and paired with *graph-based negatives* produced by rule-driven perturbations (relation reversal, attribute swaps). These hard negatives are optimized with a dedicated preference objective that drives positives above their graph-derived negatives, combined with standard contrastive learning over LAION [85] pairs.

On the *visual* side, the image encoder is augmented with *Adaptive Scene-Graph Tokens*, two learned token sets for objects and relations that attend jointly with patch and CLS tokens but are processed by *token-specific* transformer parameters (separate $Q/K/V$ and MLPs for SG vs. patch tokens). These tokens are trained to predict SG elements (open-vocabulary object/attribute phrases and relation triplets, plus boxes) by matching token predictions to ground-truth graph nodes/edges via DETR-style [86] bipartite assignment, supervising class predictions through the VLM's own text encoder embeddings.

The entire adaptation remains parameter-efficient via LoRA on trainable matrices, keeping base weights frozen. Empirically, applying SGVL to CLIP/BLIP/BLIP2 yields consistent gains on compositional benchmarks with only mild zero-shot degradation, showing that limited SG supervision can systematically empower VLMs with stronger structure sensitivity without large-scale annotation or full model retraining.

(iii) **Structured decoders** replace global similarity with factorized, causally ordered likelihoods that respect linguistic dependencies. **Causally-Ordered Generative Training (COGT)** [36] treats a caption as a dependency tree: heads (e.g., nouns, verbs) generate their modifiers (adjectives, prepositional phrases, objects). In this tree structure, each word is a node connected by directed edges showing grammatical dependencies—for example, in “a red car”, the noun “car” is the head with “red” as its dependent modifier, while in “the cat sits on the mat”, “sits” is the root verb with “cat” as subject and “on the mat” as a prepositional phrase modifier.

**COGT** reframes CLIP as a syntactically guided generation model by adding a decoder that predicts caption tokens under a dependency-tree factorization. The decoder follows a standard transformer design but replaces left-to-right masking with masks derived from the dependency tree, so each token attends only to its syntactic parents. Given an image $x$, the visual encoder is run once to extract patch-level features $V$, providing fine-grained evidence for each linguistic component. A caption is parsed into a dependency tree, and each token $w_j$ is generated only from its parents and the image features, enforcing conditional independences: modifiers depend on their head nouns, predicates on their arguments, and siblings become independent given their parent. Because nodes at the same depth are independent given their ancestors, the decoder can process multiple tokens in parallel. At inference, the same factorization yields an image-conditioned caption likelihood

$$p(\text{caption} | \text{image}) = \prod_{j} p(w_j | \text{parents}(w_j), V),$$

which serves as a structured alternative to cosine similarity for retrieval and scoring.
<figure><img src="image_10.png" /><figcaption>Figure 8: Structure-CLIP integrates scene-graph knowledge through (a) graph-guided hard negative sampling and (b) a Knowledge-Enhanced Encoder (KEE) that injects role-preserving triplet embeddings into the text encoder. Figure from [25].</figcaption></figure>

### 3.3.3 Representation Factorization

The final category of architectural approaches targets a specific failure mode of CLIP: its tendency to encode attributes, objects, and relations as entangled mixtures rather than separable compositional factors. While alignment heads and modular components modify how similarity is computed, representation disentanglement methods instead reshape the underlying embeddings so that compositional primitives occupy distinct, manipulable subspaces. Recent work employs asymmetric projection heads that pull apart otherwise conflated concepts and graph-guided adapters that impose explicit structure on how attributes and relations attach to objects [37, 80], enabling more robust generalization to unseen attribute-object and relation-object combinations.

**Remark 2 (Compositional ZSCL vs. Open-Vocabulary Tasks)** *The disentanglement-based approaches discussed in this section operate within the Compositional Zero-Shot Learning (ZSCL) framework, where both objects and attributes are drawn from a fixed, closed vocabulary, and evaluation focuses on recognising held-out attribute-object compositions. This setting offers relatively clean supervision and strong structural priors, allowing methods to focus on learning systematic attribute and object factors.*

In contrast, open-vocabulary compositional tasks, examined throughout the rest of this survey, such as image-text retrieval, grounded captioning, and structured visual reasoning, require models to interpret free-form natural language, including relational predicates, multiword attributes, and other unbounded linguistic expressions. Consequently, these methods must address a substantially broader and less constrained compositional generalisation problem. To maintain a clear scope, we review only two representative ZSCL methods here rather than attempting to cover the full breadth of work in this area.

**Gipcol** [37] targets attribute-object compositional zero-shot learning (CZSL) by injecting dataset-level compositional structure into frozen CLIP text prompts through graph-based message passing. The method learns a soft prefix $\Theta = \{\theta_1, \dots, \theta_k\}$ and refines attribute/object tokens via a graph encoder with parameters $\Phi$. For a composition $c_i = (a_i, o_i)$ with attribute $a_i$ and object $o_i$, the class text becomes:

$$ \underbrace{[\text{SOS}, \theta_1, \dots, \theta_k,}_{\text{learned prefix } \Theta} \hat{a}_i, \hat{o}_i, \dots, \text{graph-refined tokens} \quad , \text{[EOS]}], $$

where $\hat{a}_i, \hat{o}_i$ are outputs of message passing on a graph whose nodes are attributes, objects, and seen compositions; edges connect $(c \leftrightarrow a)$, $(c \leftrightarrow o)$, and optionally $(a \leftrightarrow o)$. Only element nodes populate prompts to preserve zero-shot
constraints, while CLIP encoders remain frozen and Θ and Φ are optimized using a regularized cross-entropy loss on image-text similarity logits. This approach demonstrates that compositional factorization can be achieved through graph-guided prompt refinement, enabling better generalization to unseen attribute-object combinations.

**DCDA** [80] shares with GIPCOL the use of a compositional graph and GNN-based message passing, but differs in both motivation and function: whereas GIPCOL refines prompt representations, DCDA targets *cross-composition primitive disentanglement* by enforcing consistency of attribute and object features across diverse contexts.

Motivated by the observation that an attribute can display strong feature divergence when paired with different objects (for example, the appearance of *red* in *red tomato* versus *red car*, or *cooked* in *cooked tomato* versus *cooked steak*) DCDA aggregates evidence from multiple primitive-sharing compositions to learn more stable and transferable primitive representations.

To support this, DCDA constructs a tripartite compositional graph linking attributes, objects, and seen compositions, from which it derives primitive-sharing relationships that guide two lightweight adapters inserted into a frozen CLIP model. The L-Adapter (text side) initializes each attribute, object, and composition with a CLIP prompt embedding and applies a GNN over the graph to propagate semantic information, yielding more disentangled textual primitives. The V-Adapter (vision side) tackles visual entanglement by applying cross-attention among images that share primitives, supported by a primitive relevance-guided sampling strategy that selects and weights auxiliary images based on attribute-object co-occurrence statistics. Together, these components encourage consistent and discriminative primitive features, enabling improved generalization to unseen attribute-object compositions.

## Architecture-Centric: Key Insights

* The dual-encoder architecture compresses entire images and texts into single global vectors, creating a bag-of-words representation that collapses spatial structure, weakens attribute-object binding, and fails to capture relational dependencies, making global cosine similarity the primary obstacle to compositional understanding.
* Methods progress from similarity refinements (lightweight alignment heads) to modular architectures (explicit compositional components) to representation factorization (disentangling embeddings into compositional primitives), with each approach incrementally injecting compositional biases that the original architecture lacks.
* The most effective approaches introduce dedicated components that handle specific compositional tasks while preserving efficiency and zero-shot capabilities, replacing global co-occurrence patterns with structured, locality-aware reasoning mechanisms.
* More sophisticated methods represent a trade-off between architectural complexity and compositional capability, achieving better compositional understanding at the cost of increased computational overhead, and some approaches may require fine-tuning that risks degrading zero-shot performance on standard benchmarks.

## 3.4 Inference-Time

Inference-time methods operate on frozen VLMs (e.g., CLIP) without any fine-tuning, applying a reasoning procedure only at inference time. These approaches work by breaking down captions into simpler units (entities and relations), identifying where these units appear in images, and combining the resulting local matches to modify the global image-text score. Despite requiring external components and additional processing, these methods provide plug-and-play improvements in compositional matching and retrieval.

Compositional CLIP (ComCLIP) [34] follows this recipe: At inference, it splits the caption into subject/verb/object phrases and uses spatial attention evaluated at multiple scales to identify candidate regions for each phrase. It then forms sub-images focused on these regions, encodes both sub-images and phrases with the frozen CLIP encoders, measures phrase-region similarities, and fuses the resulting local signals back into the global image representation with attention-style reweighting before computing the final image-text score. *Concretely, it finds the relevant image parts by calling a GRiT-based dense captioning/region-proposal module that yields phrase-aligned boxes for subject/verb/object, which are converted into sub-images* [87]. By steering similarity toward “the right regions for the right words,” ComCLIP achieves substantial improvements on composition-sensitive benchmarks while adding minimal overhead.

Grounding CLIP (GCLIP) [38] is likewise training-free and zero-shot but is explicitly detector-guided. Given a caption, a two-stage LLM procedure (GPT-3.5) first extracts entity/attribute phrases and then identifies SVO relations via targeted prompts. These phrases are used to drive Grounding DINO, which localizes subjects and objects and forms relation regions by taking the union of subject/object boxes. Masked—rather than cropped—sub-images for subjects, objects, and relations are embedded with the frozen CLIP vision encoder; phrases and the full caption are also embedded with the frozen CLIP text encoders. Component-wise alignment uses normalized cosine similarities as linear fusion weights that additively correct the baseline global image embedding, and the final image-caption score is computed between this
enriched image representation and the full-caption text. Using normalized weights avoids independence assumptions between phrases, and efficiency is maintained by re-ranking only the top-k CLIP candidates.

## Inference-time: Key takeaways

* Inference-time methods enhance thefrozen VLM without any fine-tuning by decomposing captions into simpler units (entities and relations), localizing them in the image, and fusing local signals back into the global image-text score, yielding plug-and-play gains with no architectural changes.
* These methods introduce dependencies on external components (dense captioning modules, grounding detectors, LLM parsers) and add computational overhead from multiple encodings and re-ranking, representing workarounds that address the cosine bottleneck without fixing the underlying architectural limitation.

## 3.5 Diagnostic Studies

Understanding where CLIP-style VLMs fail, why they fail, and how faithfully benchmarks measure true multimodal understanding has become a central research question. We organize the literature as follows. First, controlled perturbation studies identify where specific failures occur across increasingly complex compositional tasks (Sec. 3.5.1). These observations motivate analysis of why failures arise: whether in the text encoder, the vision encoder, or the alignment mechanism (Sec. 3.5.2). Finally, recent work questions whether benchmarks themselves are reliable, revealing that many can be solved by language-only shortcuts without looking at images (Sec. 3.5.3).

### 3.5.1 Where do VLMs fail?

Recent works employ series of controlled perturbations (such as changing word order, resizing an object) across many challenging cases. By observing when these small, targeted changes cause CLIP to flip its ranking or pick the wrong match, they can systematically expose where the model relies on the wrong cues (e.g., first-mention or large-object bias) and where it fails at true composition and relations. We organize the findings along an increasing level of compositional complexity: from positional and salience biases, to part-whole composition in compound nouns, to multi-object attribute binding and relational reasoning, to compositional out-of-distribution generalization, and finally to cross-domain compositional transfer.

**Word-order and object-size biases.** [39] construct a carefully controlled test for multi-object scenes to separate errors arising in the *text encoder* from those in the *image encoder*. Specifically, they apply meaning-preserving transformations: *shuffling word-order* in captions and *object resizing* in images. If these meaning-preserving changes cause the model to rank different captions higher or lower, it indicates the model is relying on shortcut cues rather than true semantic understanding. The study identifies two clear biases: (i) a **first-mention bias** in the text encoder, where the objects named first in the caption are given extra weight, so reordering words can change the ranking; and (ii) a **large-object bias** in the image encoder, where bigger objects are favored, so enlarging an object (without changing what it is) can push its caption to the top.

**Compound nouns: binding parts into wholes.** Moving beyond positional biases, researchers ask whether models truly bind parts into whole concepts.<fnref n="5" /> [40] introduce a text-to-image retrieval benchmark of ~400 common compound nouns⁶ For each compound noun, the benchmark includes two negative images: one showing the first constituent noun (e.g., a *lab*) and another showing the second constituent noun (e.g., a *coat*), but not the compound concept itself (e.g., a *lab coat*). The main result is that **models often choose images containing the parts rather than the compound concept itself**. For example, the query “lab coat” may be confused with images showing a lab or a coat instead of the protective garment. This indicates bag-of-words matching rather than true part-whole composition.

**Multi-object binding and relational reasoning.** Lewis et al. [41] show that while CLIP performs well on single-object adjective-noun compositions (e.g., “red cube”), its performance degrades significantly when *role binding* is required, specifically when distinguishing scenes with *two objects* and *explicit spatial relations* (e.g., “cube behind sphere” vs. “sphere behind cube”). To diagnose this, they compare CLIP with *syntax-insensitive* composition baselines: elementwise ADD, MULT, and circular convolution (CONV). These operations are order insensitive and therefore cannot encode role structure. Predictably, they perform near chance on relational splits, because commutativity produces

<footnote n="5">⁵Here, “binding” means combining the meanings of individual words or visual elements into a single unified concept rather than treating them separately.

⁶Compound nouns are multi-word expressions that form a single concept, e.g., *lab coat*, *toothbrush*, *snowman*.</footnote>
indistinguishable representations for role-sensitive descriptions:

$$ \mathrm{vec}(\mathrm{cube}) + \mathrm{vec}(\mathrm{behind}) + \mathrm{vec}(\mathrm{sphere}) \approx \mathrm{vec}(\mathrm{sphere}) + \mathrm{vec}(\mathrm{behind}) + \mathrm{vec}(\mathrm{cube}). $$

The key insight is that **order-insensitive vector composition is insufficient for role binding**: without explicit structural mechanisms to encode who is behind whom, models cannot reliably assign attributes and relations to the correct objects. This finding suggests that future VLM pretraining must incorporate inductive biases for binding and word order to achieve robust compositional generalization.

**Compositional generalization to unseen attribute-object pairs.** The preceding studies identify failures on *seen* compositions; [52] extend the analysis to *compositional out-of-distribution (C-OoD) generalization*, asking why some CLIP models zero-shot generalize to novel attribute-object pairs (e.g., “striped toaster”) while others fail. Using controlled splits that hold out specific combinations while ensuring each attribute and object appears in training (but not together), they find that cross-model variation in C-OoD accuracy is explained by *representational factorization*. Specifically, models whose text and image embeddings more cleanly separate attributes from objects (higher *disentanglement*) achieve superior generalization, whereas models with entangled representations struggle. This entanglement is measured by *soft-rank*, which quantifies the intrinsic dimensionality of the compositional embedding space. For a set of attribute-object pairs, construct an embedding matrix *M* (separately for text and image encoders), compute its singular values {$\sigma_i$}$_{i=1}^d$, and define

$$ \mathrm{soft-rank}(M) = \frac{1}{d} |\{i : \sigma_i > \tau\}|, $$

where *d* is the embedding dimension and *τ* is a small threshold (e.g., 0.1). Intuitively, soft-rank counts the number of significant singular directions required to span the compositional space. *Lower* soft-rank indicates a compact, factorized subspace well-suited to compositional recombination; *A higher* soft-rank indicates a highly entangled space where novel combinations cannot be synthesized from the parts.

Empirically, *lower soft-rank correlates with higher C-OoD accuracy* for both text and image encoders, while *higher soft-rank predicts failure on novel pairs*. This finding formalizes the intuition that compositional generalization requires approximately *additive* subspaces: when attribute and object representations are separable, novel pairings can be formed via linear combination. Alternatively, when representations are entangled and non-additive, the model cannot reliably synthesize unseen combinations, resulting in systematic errors even when each part of the combination has been observed individually.

**Cross-domain compositional generalization.** The previous studies focus on semantic composition within a fixed distribution; [88] investigate compositional generalization across visual *domains* (e.g., photographs, paintings, sketches, clipart). They address two questions via controlled domain-mixture experiments: (i) *Unseen-domain generalization*: Can CLIP transfer to a domain entirely excluded from training if it was trained on a diverse mixture of other domains? For example, training on *Real*, *Painting*, *Clipart*, *Infograph*, and *Quickdraw*, but exclude *Sketch*; then testing on *Sketch*. (ii) *Unseen class-domain composition*: If only a subset of classes appears in a target domain during training, can CLIP recombine held-out classes with that domain at test time? For instance, include *Sketch* during training but only for {dogs, cats}; keep other classes (e.g., cars) exclusively in *Real*; then test on cars in Sketch.

Two key findings emerge. First, **domain diversity is critical for compositional transfer**: broader coverage of visual styles and layouts (not merely more images) substantially improves both leave-one-domain-out generalization and class-domain recombination. Second, and perhaps counterintuitively, **narrow partial inclusion can degrade generalization**: adding a small slice of the target domain (few classes or restricted viewpoints) can hurt compositional performance, sometimes producing results *worse* than complete exclusion of that domain. Hence, if a domain is included, it should be covered broadly across classes and viewpoints; otherwise, incomplete coverage may induce misleading correlations that harm compositional reasoning.

### 3.5.2 Why do these failures arise: within the encoders or at the alignment stage?

Having documented where VLMs fail, we now ask *why*: do these errors originate in the text encoder, the vision encoder, or the alignment mechanism? Recent studies show that all three components contribute, but to different degrees and in different ways.

**Alignment stage failures.** LABCLIP [33] asks whether compositional errors come from *alignment across modalities* or from the *individual encoders*. Using simple linear probes on frozen CLIP encoders, it shows that attribute-object bindings *are present* in both the text and image representations. The failure, therefore, does not come from an inability to represent compositional structure; rather, it arises at the *alignment stage*, where cosine similarity in the joint embedding space *collapses permutations*, that is, embeddings for “red square” and “square red” become
indistinguishable. LABCLIP fixes this by learning a lightweight linear transformation on text embeddings using
hard negatives formed by permuting attributes and objects. This post hoc step strengthens binding with negligible
computational overhead, confirming that the alignment objective is a key bottleneck.

**Text encoder bottlenecks.** [42] ask whether the *text encoder itself* loses compositional information before alignment. Their method: freeze a text encoder $g_{\phi}$, train a decoder to reconstruct the input caption from its pooled text embedding $g_{\phi}(y)$, and check whether compositional properties (spatial relations, counts, negation, attribute-object bindings) can be recovered. The rationale is that if a property cannot be recovered from the text embedding, a VLM using that embedding is unlikely to handle it correctly.

The authors construct *CompPrompts*, a controlled set of templated captions with various levels of compositional complexity (from single objects to multi-object interactions), and train text-only decoders to reconstruct the input from $g_{\phi}(y)$. They also introduce *ControlledImCaps* (image-caption pairs) to test whether text-side recoverability predicts downstream multimodal performance. Four key findings emerge. First, *limited compositional recoverability*: as captions increase in compositional complexity, reconstruction accuracy degrades sharply. Performance degrades significantly as complexity increases. When captions contain two objects instead of one, average reconstruction accuracy drops by 49% (CLIP ViT-B/32 shows an even steeper 71% drop). Similarly, adding a second attribute causes a 35% drop. Negation and counting are especially challenging, and encoders struggle when word order is semantically critical. Second, *architecture matters*: RoBERTa-CLIP substantially improves recoverability over CLIP ViT-B/32, particularly for negation, whereas modest scaling of CLIP's text encoder (ViT-L/14 vs. ViT-B/32) gives smaller gains on spatial and temporal relations. Third, *hard negatives can help*: training with composition-aware hard negatives (NegCLIP) significantly raises recoverability, suggesting that contrastive learning signals from hard-negatives can partially mitigate information loss. Fourth, *link to multimodal performance*: text-only recoverability correlates with vision-language matching on ControlledImCaps. *Low recoverability* predicts downstream failures, confirming that the text encoder is often a bottleneck. However, high recoverability is *necessary but not sufficient*; residual errors may originate from vision features or cross-modal alignment.

**Vision encoder: feature superposition.** On the vision side, [43] investigate whether *superposition* (when individual neurons respond to multiple unrelated visual features) contributes to compositional failures.<fnref n="7" /> The authors analyze a frozen CLIP ViT-L/14 using two probes. First, *Grad-CAM attribution maps* for compositional prompts (e.g., “green square”) reveal which image regions the model treats as evidence. Ideally, the heatmap should concentrate on the object that jointly exhibits both specified properties (greenness and squareness). Second, *neuron-level feature-response analysis* on a controlled dataset of shapes, colors, and positions: for each neuron, they construct a feature-response distribution (measuring activation strength for *red*, *square*, *left*, etc.) and compute its Shannon entropy. *Low* entropy indicates selectivity (specialization); *high* entropy indicates polysemanticity (superposition).

Empirically, polysemantic neurons are common, and Grad-CAM maps often highlight *separate* regions for different parts of a description (for example, color on one object and shape on another) instead of the single object that has both. Quantitatively, more superposition is linked to reduced feature separability in the embedding space. Together, these results show that superposition in the vision encoder makes binding harder, adding to the text side and alignment stage issues noted above.

**Asymmetric sensitivity and modality mismatch.** Finally, [89] employ *game-theoretic sensitivity metrics* (based on the *Harsanyi dividend* [90]) to probe how the text encoder, the image encoder, and their matching function respond to controlled perturbations of objects, attributes, and relations. Their analysis uncovers two phenomena. First, **asymmetric sensitivity**: text encoders are generally highly sensitive to the right interaction changes (e.g., swapping object roles or altering attributes/relations), whereas image encoders are weaker and often the bottleneck; Second, **modality mismatch**: text and image encoders emphasize different compositional factors, so their sensitivities do not align, degrading cross-modal matching. In particular, the text encoder is sensitive to interactions such as object–relation swaps, whereas the image encoder tends to focus on individual objects rather than their relations, leading to misaligned representations. The authors suggest that strengthening image encoders and aligning the two modalities’ compositional knowledge are promising directions.

Together, these studies show that compositional failures in CLIP-style VLMs arise from multiple sources: the text encoder loses compositional information during pooling; the vision encoder exhibits feature superposition that mixes attributes that should be separate; and the alignment mechanism collapses permutations via order-insensitive similarity. No single component is solely responsible; rather, failures across components jointly produce systematic weaknesses in compositional understanding.

<footnote n="7">⁷Superposition occurs when a single neuron activates for different features (e.g., *red*, *square*, and *left position*) rather than specializing in one. This can hurt compositional understanding by mixing features that should be bound to different objects.</footnote>
### 3.5.3 Do benchmarks measure what we think?

The studies above assume that benchmark performance reflects genuine compositional reasoning. Recent work challenges this assumption, showing that many benchmarks can be solved by *language-only shortcuts* without looking at images.

**Language-only baselines expose linguistic bias.** Wu et al. [44] present a framework to test whether VLM performance comes from real multimodal reasoning or from language priors that can be exploited without images. They evaluate *language-only models* on ARO (Attribute, Relation, Order) [3], a common compositional benchmark. These language-only models never see images and instead select captions based purely on linguistic plausibility (e.g., choosing the caption with lowest perplexity). Surprisingly, these “blind” baselines often *beat* full VLMs, showing that the benchmark has strong linguistic bias.

To separate items that truly need vision, the authors define two metrics: (i) **Hard Test Accuracy (HTA)**, which evaluates only the examples where language models fail, so genuine vision–language understanding is required; and (ii) **Linguistic Gap = Overall Accuracy - HTA**, which measures how much performance comes from language shortcuts rather than real multimodal reasoning. Applying this to fine tuned models (e.g., NegCLIP) shows that their gains *mostly come from exploiting dataset biases* instead of improving compositional understanding. The results imply that current benchmarks overestimate true compositional ability, fine tuning often amplifies bias rather than fixing it, and high accuracy does not guarantee genuine multimodal reasoning.

**Four types of benchmark shortcuts.** Building on this critique, [45] systematically examines compositionality benchmarks and exposes four classes of language shortcuts. Their method uses two blind baselines: a text-only language model (LLaMA2) [91] and a “blindfolded” VLM called **BlindCap** [92]. BlindCap uses a standard vision-language architecture but is made “blind” at test time by feeding only captions (no images). The authors train two variants: one trained on image-text pairs but tested without images (BlindCap I+T), and another trained only on text captions, ignoring images during training (BlindCap T). This allows BlindCap to capture each dataset’s language style without visual information. They score captions using both *unnormalized log-likelihood* (to find length biases) and *length-normalized log-likelihood* (to find plausibility biases).

The analysis finds four systematic problems between positive and negative captions. (i) **Plausibility Bias:** Hard negatives are often linguistically implausible, allowing rejection without visual evidence. For example, from ARO, “The boy is playing in the park” (log-likelihood -3.24) versus “The park is playing in the boy” (log-likelihood -5.94)—the implausible negative can be detected by language models alone. (ii) **Length/Likelihood Bias:** Systematic differences in token length allow blind models to distinguish positives from negatives. From SugarCREPE, a positive caption “A person walking down a street past a teddy bear sign” versus a negative adding “paved with colorful tiles” (26 more tokens): the length difference signals correctness without visual information. (iii) **Source Bias:** Over-reliance on COCO creates dataset-specific priors. 10 of 17 benchmarks use COCO images directly, and tuning with COCO yields consistent gains (e.g., SugarCREPE-Replace: 68.22% → 75.71%, COCO-Counterfactuals: 46.67% → 66.67%), suggesting models learn COCO-specific patterns rather than general understanding. (iv) **Synthetic vs. Real Bias:** Generated negative images contain detectable artifacts. From CounterCurate, synthetic images are easily identifiable as artificial, allowing rejection based on generation artifacts rather than compositional correctness.

Critically, across multiple widely used benchmarks, the blind baselines *match or exceed* full VLMs, showing that many compositionality tests can be solved with text alone. These findings show that progress on current benchmarks may reflect *dataset hacking* rather than real advances in compositional reasoning, highlighting the need for better evaluation methods.

## 4 Datasets and Benchmarks

Evaluating compositional understanding in vision-language models presents a fundamental challenge: traditional retrieval metrics fail to capture whether models genuinely understand how visual and linguistic elements combine. Models may achieve high accuracy by exploiting shortcut cues (e.g., matching "dog" and "ball" independently) rather than comprehending "a brown dog catching a red ball." This limitation has motivated specialized evaluation methodologies that systematically investigate compositional reasoning capabilities.

These benchmarks vary along critical dimensions: data sources (real-world versus synthetic), compositional primitives (attributes, relations, ordering, counting, negation, temporal dynamics), tasks (image-to-text, text-to-image, or both), and strategies for generating challenging test cases. Table 2 summarizes the foundational datasets that serve as the underlying data resources from which benchmark instances are constructed, highlighting their modality, scale, and annotation characteristics. In contrast, Table 3 provides an overview of benchmark-specific evaluation methodologies,
<table><caption>Table 2: Overview of benchmark datasets for evaluating compositionality in CLIP and related models.</caption><thead><tr><th>Dataset</th><th>Year</th><th>Modality</th><th>Scale</th><th>Annotations</th><th>Typical Tasks</th><th>License</th></tr></thead><tbody><tr><td>VG (Visual Genome)</td><td>2016</td><td>Image+Text</td><td>108k images, 5.4M region descriptions</td><td>Objects, attributes, relations</td><td>VQA, scene graph, grounding</td><td>CC BY 4.0</td></tr><tr><td>GQA</td><td>2019</td><td>Image+Text</td><td>113k images, 22M QA pairs</td><td>Scene graphs, QAs</td><td>VQA, reasoning</td><td>CC BY 4.0</td></tr><tr><td>COCO</td><td>2014</td><td>Image+Text</td><td>330k images, 5 captions each</td><td>Objects, captions, segmentation</td><td>Captioning, detection, segmentation</td><td>CC BY 4.0</td></tr><tr><td>Flickr30k</td><td>2014</td><td>Image+Text</td><td>31k images, 5 captions each</td><td>Captions</td><td>Image-text retrieval, captioning</td><td>Research only</td></tr><tr><td>CC-12M</td><td>2021</td><td>Image+Text</td><td>12M pairs</td><td>Weak captions</td><td>Pretraining, retrieval</td><td>Research only</td></tr><tr><td>YFCC-15M</td><td>2016</td><td>Image+Text</td><td>15M pairs</td><td>Weak captions, metadata</td><td>Pretraining, retrieval</td><td>CC BY</td></tr><tr><td>LAION-400M</td><td>2021</td><td>Image+Text</td><td>400M pairs</td><td>Weak captions (CLIP-filtered)</td><td>Pretraining, retrieval</td><td>Open (varies)</td></tr><tr><td>VOC2007</td><td>2007</td><td>Image</td><td>10,582 images</td><td>Bounding boxes, class labels</td><td>Detection, classification</td><td>Research only</td></tr><tr><td>MSR-VTT</td><td>2016</td><td>Video+Text</td><td>10k videos (200k clips)</td><td>Captions</td><td>Video retrieval, captioning</td><td>Research only</td></tr><tr><td>VaTeX</td><td>2019</td><td>Video+Text</td><td>41k videos, 82.5k captions (English+Chinese)</td><td>Parallel multifilingual captions</td><td>Video captioning, retrieval, translation</td><td>CC BY-NC-SA 4.0</td></tr><tr><td>TEMPO</td><td>2022</td><td>Video+Text</td><td>1,000 videos, 6k QA pairs</td><td>Temporal relations, ordering</td><td>Video QA, temporal reasoning</td><td>Research only</td></tr><tr><td>CheXpert</td><td>2019</td><td>Image+Text</td><td>224k chest X-rays, 65k patients</td><td>Pathology labels (14 classes)</td><td>Medical classification, detection</td><td>Research only</td></tr><tr><td>CLEVR</td><td>2017</td><td>Synthetic Image+Text</td><td>100k images, 1M QAs</td><td>Full scene graphs, QA</td><td>VQA, reasoning, compositionality</td><td>MIT</td></tr><tr><td>SUPER CLEVR</td><td>2021</td><td>Synthetic Image+Text</td><td>2M images, 10M QAs</td><td>Objects, attributes, QA</td><td>Classification, color analysis</td><td>MIT</td></tr><tr><td>Natural-Color Dataset (NCD)</td><td>2018</td><td>Image</td><td>~1.3k images</td><td>Fruit/vegetable labels</td><td>Classification</td><td>Research only</td></tr><tr><td>Fruit360</td><td>2018</td><td>Image</td><td>90k images, 131 classes</td><td>Class labels</td><td>Video grounding, retrieval</td><td>MIT</td></tr><tr><td>DiDeMo</td><td>2017</td><td>Video+Text</td><td>10k videos, 40k descriptions</td><td>Temporal grounding annotations</td><td>Action recognition, detection</td><td>Research only</td></tr><tr><td>ActivityNet</td><td>2015</td><td>Video+Text</td><td>20k videos, 648 hours</td><td>Temporal action labels</td><td></td><td>Research only</td></tr></tbody></table>

<table><caption>Table 3: Overview of compositionality benchmarks for vision–language models. The benchmarks vary across multiple dimensions, including task type (image retrieval or text retrieval), modality (image or video), compositional primitives (attributes, relations, objects, order, counting, negation, temporal, spatial), and data type (real or synthetic).</caption><thead><tr><th>Benchmark</th><th>Size</th><th>Data Source</th><th>Task</th><th>Primitives</th><th>Modality</th><th>Image Types</th><th>Text Types</th></tr></thead><tbody><tr><td>Winoground</td><td>400</td><td>Getty Images</td><td>Both</td><td>AttRelObjOrd</td><td>Image</td><td>R</td><td>R/S</td></tr><tr><td>ARO</td><td>58,685</td><td>VG, GQA, COCO, Flickr</td><td>I→T</td><td>AttRelObjOrd</td><td>Image</td><td>R</td><td>R/S</td></tr><tr><td>CREPE</td><td>42,406</td><td>CC-12M, YFCC-15M, LAION-400M</td><td>I→T</td><td>AttRelObj</td><td>Image</td><td>R</td><td>R/S</td></tr><tr><td>SugarCrepe</td><td>7,512</td><td>COCO</td><td>I→T</td><td>AttRelObj</td><td>Image</td><td>R</td><td>R/S</td></tr><tr><td>BIVLC</td><td>2,933</td><td>COCO + SDXL-DPO synth</td><td>Both</td><td>AttRelObj.Neg</td><td>Image/Video</td><td>R/S</td><td>R/S</td></tr><tr><td>NegBench</td><td>79k</td><td>COCO, VOC2007, MSR-VTT, CheXpert, SD synth</td><td>Both</td><td>AttRelObj.Neg</td><td>Image</td><td>R/S</td><td>R/S</td></tr><tr><td>CountBench</td><td>540</td><td>LAION-400M</td><td>I→T</td><td>Count</td><td>Image</td><td>R</td><td>R</td></tr><tr><td>TemporalVQA</td><td>720</td><td>Pixabay, Pexels, Flickr, YouTube</td><td>V→T</td><td>TempRelObj</td><td>Image/Video</td><td>R</td><td>R</td></tr><tr><td>Spatial457</td><td>1k</td><td>CLEVR-style synthetic scenes</td><td>I→T</td><td>AttRelObj.3D</td><td>Image</td><td>S</td><td>R</td></tr><tr><td>CAB</td><td>494</td><td>NCD + synthetic</td><td>I→T</td><td>AttObj</td><td>Image</td><td>R/S</td><td>R</td></tr><tr><td>NEMO</td><td>900</td><td>Fruit360 + DALL-E synth</td><td>I→T</td><td>AttObj</td><td>Image</td><td>R/S</td><td>R</td></tr><tr><td>AARO</td><td>758k</td><td>MSRVTT, DiDeMo, ActivityNet</td><td>V→T</td><td>Act, AttRelObj</td><td>Video</td><td>R</td><td>R/S</td></tr><tr><td>VideoCon</td><td>570</td><td>MSR-VTT, VaTeX, TEMPO</td><td>V→T</td><td>Act, AttRelObjCount</td><td>Video</td><td>R</td><td>R/S</td></tr></tbody></table>
detailing how these datasets are structured into tasks, which compositional primitives are targeted, and the types of visual-linguistic reasoning each benchmark is designed to assess.

We trace the evolution of compositional evaluation through increasingly sophisticated methodologies, emphasizing how each evaluation approach addresses specific gaps revealed by previous benchmarks, collectively building toward a comprehensive evaluation framework.

## Foundational Evaluation: Word Order and Relational Binding

The earliest benchmarks addressed a fundamental question: can VLMs distinguish between captions containing identical words but differing in arrangement? This tests genuine compositional reasoning versus mere word detection. Winoground [8] pioneered this evaluation by testing whether models can correctly pair two images with two captions sharing the same word set but different orderings (e.g., Figure 9 (a)). The benchmark assesses attribute-object binding, spatial layout, and relational structures through three metrics: **Text score** (caption selection given an image), **Image score** (image selection given a caption), and **Group score** (requiring both conditions simultaneously). Evaluation relies on three complementary metrics. Let ($Y_0, I_0$) and ($Y_1, I_1$) denote two image-caption pairs where captions $Y_0$ and $Y_1$ contain the same words with different orderings. Formally:

1. **Text score:** tests caption selection given an image. Define

$$f(Y_0, I_0, Y_1, I_1) = \begin{cases} 1 & \text{if } s(Y_0, I_0) > s(Y_1, I_0) \text{ and } s(Y_1, I_1) > s(Y_0, I_1), \\ 0 & \text{otherwise.} \end{cases}$$

2. **Image score:** tests image selection given a caption. Define

$$g(Y_0, I_0, Y_1, I_1) = \begin{cases} 1 & \text{if } s(Y_0, I_0) > s(Y_0, I_1) \text{ and } s(Y_1, I_1) > s(Y_1, I_0), \\ 0 & \text{otherwise.} \end{cases}$$

3. **Group score:** requires both conditions to hold simultaneously. Define

$$h(Y_0, I_0, Y_1, I_1) = \begin{cases} 1 & \text{if } f(Y_0, I_0, Y_1, I_1) = 1 \text{ and } g(Y_0, I_0, Y_1, I_1) = 1, \\ 0 & \text{otherwise.} \end{cases}$$

CLIP (ViT-B/32) achieves 30.75, 10.50, and 8.00 respectively, while humans score 89.50, 88.50, and 85.50, establishing a critical baseline for compositional failures.

While Winoground revealed fundamental weaknesses, it did not systematically decompose the sources of failure. **Attribution, Relation, and Order (ARO)** [10] addresses this limitation by explicitly separating three distinct capabilities: relational order, attribute-object binding, and word-order sensitivity (e.g., Figure 9 (b)). Building on Visual Genome [93] and GQA [94], ARO creates **Visual Genome Relation** (evaluating correct versus reversed relations) and **Visual Genome Attribution** (evaluating attribute-object binding). Additional stress tests systematically perturb COCO [95] and Flickr30k [96] captions through noun-adjective shuffling. Results reveal that models perform near or below chance on relational and attributive understanding, and that high retrieval performance on standard datasets does not require compositional understanding: randomly shuffled captions still yield strong performance, exposing reliance on shortcut cues.

## Systematic Compositional Generalization

While Winoground and ARO reveal specific failures, they do not measure systematicity (understanding novel combinations of seen concepts) or productivity (comprehending expressions of unbounded complexity). Compositional Representation Evaluation (CREPE) [5] addresses this gap by representing scenes as structured scene graphs, defining atoms (singular concepts) and compounds (primitive compositions)<fnref n="8" />, and testing systematic recombination (e.g., Figure 9 (c)). Evaluation is performed on CC-12M [97], YFCC-15M [98], and LAION-400M [85]. CREPE introduces two evaluation criteria: *Systematicity*, measured through compositional splits (Seen Compounds, Unseen Compounds), and *Productivity*, evaluated across caption complexity levels ($n = 4, 5, ..., 12$ atoms). Hard negatives include atomic replacements, compound-level substitutions, word-order permutations, and negations. Results show performance

<footnote n="8">⁸a singular concept (“atom”) is a single visual or linguistic primitive, e.g., one object (“dog”), one attribute (“red”), or one relation (“left of”). A primitive composition (“compound”) is the smallest binding of those atoms that expresses a concrete scene region, such as an attribute-object pair (“red dog”) or a relation triple (“dog left of ball”)</footnote>
declines from SC to UC splits, with the largest drop in LAION-400M-trained models, and decreasing performance as
caption complexity increases.

However, CREPE's rule-based hard-negative generation introduced critical methodological limitations: nonsensical or ungrammatical negatives that text-based models could exploit without genuine vision-language reasoning [99]. **SugarCrepe** [99] addresses this by ensuring hard negatives are semantically plausible and grammatically fluent, employing LLM-based generation with human validation and adversarial refinement. Built on COCO [95], it introduces seven fine-grained hard-negative types (object/attribute/relation replacements, swaps, additions). Results indicate that large pretrained CLIP models achieve near-human performance on object-based categories, demonstrating partial compositional generalization; however, they exhibit substantial deficiencies on other categories—most notably on SWAP hard negatives and examples involving attribute and relational perturbations—where performance remains far below human levels.

While SugarCrepe refined hard-negative quality, it evaluated only image-to-text retrieval, leaving text-to-image align-
ment underexplored. **BIVLC** [100] addresses this asymmetry by pairing each hard-negative caption with a corresponding
synthetic hard-negative image (generated with SDXL-DPO [101] and human-filtered), creating instances with two
images, two captions, and four retrieval queries (two image-to-text, two text-to-image). The final set contains 2,933
instances across REPLACE/SWAP/ADD categories. Experiments reveal that VLMs perform notably worse on text-to-
image than image-to-text retrieval, and that contrastive training with synthetic pairs improves results but leaves a large
gap to human performance.

**Specialized Compositional Primitives: Negation and Counting**

The benchmarks above assess compositional understanding broadly, but specialized phenomena require targeted eval-
uation. Negation requires reasoning about *absence* rather than presence, while counting demands precise numerical
grounding beyond object detection. **Negation Benchmark (NegBench)** [10] systematically evaluates negation under-
standing across multiple levels: distinguishing affirmative from negative statements, processing hybrid conditions, and
reasoning across static images, video, and medical imaging (e.g., Figure 9 (d)). Adapting COCO [95], VOC2007 [102],
MSR-VTT [103], and CheXpert [104], NegBench introduces two evaluation settings: **Retrieval-Neg** (retrieving images
satisfying both affirmative and negative conditions) and **MCQ-Neg** (selecting correct descriptions from alternatives
using Affirmation, Negation, and Hybrid templates). Results reveal severe challenges: CLIP models show significant
drops with negated queries, MCQ-Neg performance is only slightly above random, and medical settings show the most
severe failures (BioMedCLIP and CONCH drop 15.8% and 33.2% respectively on CheXpert MCQ-Neg). Fine-tuning
on large-scale synthetic negated captions improves performance substantially (10% recall boost, 28% MCQ-Neg
accuracy improvement).

Complementing negation, **CountBench** [9] targets numerical quantification, requiring precise enumeration rather than coarse cardinality estimation. The benchmark contains 540 images from LAION-400M [85], each depicting 2-10 instances of a particular object with explicit count captions (e.g., Figure 9 (e)). CLIP-B/32 achieves only 31.67% accuracy, revealing substantial weaknesses. A counting-contrastive loss improves performance to 75.93% while maintaining standard benchmark results.

**Temporal and Spatial Compositional Reasoning**

Previous benchmarks operate within static images, but genuine compositional understanding must extend to dynamic and
spatially structured environments. **Temporal Visual Question Answering (TemporalVQA)** [13] extends evaluation
into the temporal domain, testing event sequences and elapsed time through two tasks: **Temporal Order Understanding**
(identifying which of two sequential images occurred first, with a "consistent accuracy" metric requiring same predictions
for reversed pairs) and **Time-lapse Estimation** (estimating elapsed time between sequential images across six temporal
categories) (e.g., Figure 9 (f)). Results reveal substantial weaknesses: even GPT-4o struggles, often relying on positional
heuristics rather than genuine temporal reasoning.

While TemporalVQA targets temporal composition, **Spatial457** [12] provides systematic evaluation of spatial reasoning, requiring precise geometric understanding of 3D positions, orientations, and relative configurations. The benchmark defines four core capabilities: (1) multi-object reasoning; (2) 2D location reasoning; (3) 3D location reasoning incorporating depth; and (4) 3D orientation reasoning testing poses, facing directions, and alignments (e.g., Figure 9(g)). Built on CLEVR-style [82] controlled synthetic scenes with 6D annotations, Spatial457 evaluates across five difficulty levels. Results show consistent performance declines as complexity increases, with GPT-4o and GeminiPro-1.5 performing particularly poorly on 3D orientation tasks and exhibiting systematic spatial biases.
## Multi-object Attribute Binding and Canonical Associations

Returning to static images but introducing a different compositional challenge, we now examine benchmarks that target a subtle but pervasive failure mode: the inability to correctly bind attributes to specific objects when multiple objects co-occur in a scene. While earlier benchmarks like ARO tested attribute-object binding through caption swaps, the following benchmarks isolate this problem more directly by requiring models to answer queries about specific objects in multi-object scenes, revealing how models fall back on canonical object-attribute associations rather than grounding attributes in visual evidence.

**Concept-Attribute Binding (CAB)** [4] provides a controlled investigation of CLIP's tendency to misattribute attributes when multiple objects appear in an image. Using 494 composite images with two distinct objects from the Natural-Color Dataset [105], the benchmark poses a 5-way color classification task where prompts specify which object's color to identify (e.g., "In this picture, the color of the lemon is [mask]"). Figure 9 (h) shows an illustrative case. CLIP performs nearly perfectly on single-object classification but falls below chance when two objects are present. Errors are systematic: when asked about object A, CLIP often predicts the color of object B, revealing that it lacks a robust object-centric binding mechanism and instead treats image-text inputs as a "bag of concepts." Although this binding failure becomes less systematic when using synthetic images of fruits with unnatural colors (e.g., a blue tomato)—suggesting that the model is biased toward natural color distributions—the overall error rate remains high. This shows that the issue is not merely a reliance on familiar object-attribute associations but reflects a more fundamental weakness in CLIP's ability to bind attributes to the correct objects.

**NEMO** [106] extends this evaluation to multimodal large language models across multiple query formats (open-ended identification, multiple-choice recognition, unsolvable queries). Using 45 fruit categories, it pairs 450 natural images with 450 synthetic counterparts where objects are rendered in novel colors. Results confirm that while models succeed on natural objects, accuracy drops sharply for attribute-modified ones, demonstrating that object recognition deteriorates when canonical attribute associations are violated.

## Video-based Compositional Understanding

While previous benchmarks focus on static images (except TemporalVQA's event-ordering), video presents unique compositional challenges: models must compose objects, attributes, spatial relations, actions, temporal dynamics, and event sequences. **Action, Attribute, Relation and Object (AARO)** [28] adapts the hard-negative evaluation paradigm to video, testing four compositional skills by generating negative captions where only one aspect is altered (e.g., Figure 9 (i)). Negative captions are generated using rule-based and LLM-based strategies, with the latter yielding more natural and challenging negatives. Evaluation uses Video-Text Matching (VTM) accuracy on MSRVTT [103], DiDeMo [107], and ActivityNet [108]. The authors introduce NAVERO, a training framework integrating negative-augmented data to improve compositional understanding while preserving competitive performance on standard retrieval benchmarks.

While AARO targets compositional primitives through minimal caption edits, **Video-Language Alignment via Contrast Captions (VideoCon)** [109] evaluates robustness under a broader taxonomy of semantically plausible perturbations: object replacements, attribute swaps, relation shifts, counting errors, hallucinations, and event order changes. Constructed from MSR-VTT [103], VaTeX [110], and TEMPO [111], VideoCon also provides natural-language explanations for misalignments. Evaluations show vanilla models are highly sensitive to such perturbations, but finetuning on VideoCon improves robustness.

# 5 Results and Comparisons

Tables 4 and 5 summarize the performance of the major representative approaches covered in this survey on three important benchmarks: **ARO**, **VL-Checklist**, and **SugarCrepe**. Comparing these approaches across different compositional challenges reveals interesting insights about which strategies are most effective for different aspects of vision-language understanding. On the **VL-Checklist** benchmark, baseline models show moderate performance, with relations consistently being the weakest area across all models. Data-centric approaches dominate this benchmark, with NAVERO achieving the highest scores across all three categories, demonstrating that large-scale data augmentation and negative sampling are highly effective. DAC-LLM and DAC-SAM also exceed 88% across most categories, showing that synthetic data generation significantly improves compositional understanding.

On ARO benchmark, baseline models perform poorly on COCO-order tasks, achieving near-chance performance, while showing better but still limited results on Flickr-order and attributions. Data-centric models show significant improvements, with NAS<sub>(NTVA)-COCO</sub> achieving top scores in attributions and relations, while CLIP-SVLC leads in Flickr-order; NegCLIP and DAC-LLM demonstrate strong generalization across most categories. Architecture-centric models
<figure><img src="image_11.png" /><figcaption>Figure 9: Examples from benchmarks designed to evaluate different aspects of vision-language models.</figcaption></figure>

demonstrate that modular architectures can reduce order bias, as seen in OC-CLIP and LABCLIPHNB$_{-ViT-B/32}$, though with more variable results.

For SugarCrepe, the baseline models show a clear performance hierarchy: Replace tasks are easiest, followed by Add, with Swap being most challenging; this pattern persists across most models, indicating that Swap tasks require more sophisticated compositional reasoning. Architecture-centric models show high variability, with OC-CLIP$_{B-14}$ achieving the highest Swap performance and strong results across all tasks, while data-objective strategies, as seen in SPARCL and TripletCLIP, perform strongly on challenging tasks. The improvement of models over baselines, especially in Swap tasks, highlights the importance of training strategies for compositional reasoning.

Several additional patterns emerge from cross-benchmark analysis. Dataset-specific challenges are evident: baseline models perform differently on COCO-Order (47.9%) and Flickr-Order (60.2%), showing a 12-point gap that suggests ordering difficulty varies across data sources. While BLIP outperforms CLIP on Visual Genome attribution and relation tasks, it performs much worse on ordering benchmarks, dropping from 47.9%/60.2% (CLIP) to 13.6%/13.7% (BLIP) on COCO-Order and Flickr-Order. Method-specific inconsistencies reveal limitations: NAS(NVA)COCO achieves 89.8% on ARO Attribution but only 67.8% on Relations, indicating that some techniques improve certain compositional aspects while leaving others unaddressed. Counterintuitive results appear: SVLC (RB) outperforms SVLC (LLM+RB) on SugarCrepe Swap tasks, suggesting that simpler rule-based approaches can sometimes outperform more sophisticated LLM-enhanced methods. Cross-benchmark inconsistencies highlight evaluation gaps: COGT-CLIP achieves strong ARO performance (90.26% Attribution) but underperforms on VL-Checklist Objects (78.91%), indicating that benchmark-specific optimizations may not generalize.

# 6 Discussion and Open Challenges

Despite significant progress in improving compositional reasoning, several important challenges remain. This section examines several key issues that the field must address. First, while synthetic data generated by large language models or diffusion models can improve training coverage, it also brings new problems such as errors and biases. Second, many evaluation benchmarks contain hidden flaws that make models appear better than they actually are, while failing to test important skills like negation, counting, and temporal reasoning. Third, when models are fine-tuned to improve compositionality, they often forget previously learned abilities, reducing their performance on general tasks.
<table><caption>Table 4: (%) on VL-Checklist and ARO benchmarks.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Category</th><th colspan="3">VL-Checklist</th><th colspan="4">ARO</th></tr><tr><th>Object</th><th>Relation</th><th>Attribute</th><th>Attribution</th><th>Relation</th><th>COCO-order</th><th>Flickr-order</th></tr></thead><tbody><tr><td>CLIP</td><td>-</td><td>81.58</td><td>63.05</td><td>67.60</td><td>63.18</td><td>59.98</td><td>47.9</td><td>60.2</td></tr><tr><td>BLIP.2</td><td>-</td><td>84.2</td><td>70.7</td><td>80.1</td><td>71.2</td><td>41.2</td><td>13.6</td><td>13.7</td></tr><tr><td>DAC-LLM</td><td>Data-centric</td><td>88.2</td><td>89.5</td><td>73.3</td><td>81.3</td><td>94.5</td><td>95.7</td><td>-</td></tr><tr><td>DAC-SAM</td><td>Data-centric</td><td>88.5</td><td>89.7</td><td>75.8</td><td>70.5</td><td>77.2</td><td>91.2</td><td>93.9</td></tr><tr><td>MosaICLIP<sub>FineTun.CC3M</sub></td><td>Data-centric</td><td>86.4</td><td>71.9</td><td>73.7</td><td>78.0</td><td>83.7</td><td>-</td><td>-</td></tr><tr><td>NAVERO</td><td>Data-centric</td><td>90.88</td><td>84.11</td><td>81.29</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>NAS<sub>(NTA)COCO</sub></td><td>Data-centric</td><td>-</td><td>-</td><td>-</td><td>91.7</td><td>93.1</td><td>-</td><td>-</td></tr><tr><td>NAS<sub>(NVA)COCO</sub></td><td>Data-centric</td><td>-</td><td>-</td><td>-</td><td>89.8</td><td>67.8</td><td>-</td><td>-</td></tr><tr><td>NAS<sub>(NTVA)COCO</sub></td><td>Data-centric</td><td>-</td><td>-</td><td>-</td><td>93.4</td><td>93.2</td><td>-</td><td>-</td></tr><tr><td>NegCLIP</td><td>Data-centric</td><td>84.1</td><td>63.5</td><td>70.9</td><td>71.0</td><td>81.0</td><td>86.0</td><td>91.0</td></tr><tr><td>AHNPL</td><td>Data-centric</td><td>-</td><td>-</td><td>-</td><td>77.0</td><td>83.8</td><td>-</td><td>-</td></tr><tr><td>StructureCLIP</td><td>Data/Objective/Architecture-centric</td><td>-</td><td>-</td><td>-</td><td>82.3</td><td>84.7</td><td>-</td><td>-</td></tr><tr><td>SVLC<sub>LLM+RB</sub></td><td>Data/Objective-centric</td><td>85.09</td><td>78.82</td><td>73.90</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Counterfactual Sets<sub>generate-10k</sub></td><td>Data/Objective-centric</td><td>83.4</td><td>74.5</td><td>71.9</td><td>67.7</td><td>82.2</td><td>93.9</td><td>92.5</td></tr><tr><td>Counterfactual Sets<sub>generate-300k</sub></td><td>Data/Objective-centric</td><td>89.1</td><td>85.6</td><td>81.4</td><td>73.4</td><td>85.7</td><td>94.6</td><td>95.6</td></tr><tr><td>SPARCL</td><td>Data/Objective-centric</td><td>90.76</td><td>72.99</td><td>73.72</td><td>74.19</td><td>80.10</td><td>-</td><td>-</td></tr><tr><td>DeGLA</td><td>Objective-centric</td><td>-</td><td>-</td><td>-</td><td>74.3</td><td>81.6</td><td>93.8</td><td>94.7</td></tr><tr><td>CE-CLIP</td><td>Data-centric</td><td>84.6</td><td>71.8</td><td>72.6</td><td>76.4</td><td>83.0</td><td>-</td><td>-</td></tr><tr><td>OC-CLIP</td><td>Architecture-centric</td><td>84.5</td><td>73.9</td><td>73.7</td><td>82.0</td><td>86.5</td><td>94.2</td><td>84.8</td></tr><tr><td>COGT-CLIP</td><td>Architecture-centric</td><td>78.91</td><td>89.37</td><td>86.07</td><td>90.26</td><td>87.56</td><td>-</td><td>-</td></tr><tr><td>ComAlign<sub>CLIP-ViT-B/32</sub></td><td>Architecture-centric</td><td>-</td><td>-</td><td>-</td><td>66.60</td><td>61.95</td><td>-</td><td>-</td></tr><tr><td>ComAlign<sub>NegCLIP-ViT-B/32</sub></td><td>Architecture-centric</td><td>-</td><td>-</td><td>-</td><td>71.79</td><td>79.49</td><td>-</td><td>-</td></tr><tr><td>LABCLIP<sub>SB-ViT-B/32</sub></td><td>Architecture-centric</td><td>-</td><td>-</td><td>-</td><td>64</td><td>59</td><td>32</td><td>42</td></tr><tr><td>LABCLIP<sub>HNB-ViT-B/32</sub></td><td>Architecture-centric</td><td>-</td><td>-</td><td>-</td><td>69</td><td>82</td><td>81</td><td>84</td></tr><tr><td>ComCLIP</td><td>Architecture-centric, Inference-time / Post-hoc</td><td>79.0</td><td>69.30</td><td>69.90</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

<table><caption>Table 5: Performance on SugarCrepe.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">Category</th><th colspan="2">Swap</th><th colspan="2">Add</th><th colspan="3">Replace</th></tr><tr><th>Object</th><th>Attribute</th><th>Object</th><th>Attribute</th><th>Object</th><th>Attribute</th><th>Relation</th></tr></thead><tbody><tr><td>CLIP (ViT-L-14)</td><td>-</td><td>60.16</td><td>62.31</td><td>78.32</td><td>71.53</td><td>94.07</td><td>79.19</td><td>65.15</td></tr><tr><td>BLIP</td><td>-</td><td>66.2</td><td>76.2</td><td>-</td><td>-</td><td>96.5</td><td>81.9</td><td>68.35</td></tr><tr><td>NegCLIP</td><td>Data-centric</td><td>75.2</td><td>75.4</td><td>88.8</td><td>82.8</td><td>92.7</td><td>85.9</td><td>76.5</td></tr><tr><td>AHNPL</td><td>Data-centric</td><td>76.3</td><td>75.8</td><td>97.2</td><td>94.5</td><td>93.2</td><td>88.9</td><td>80.1</td></tr><tr><td>SVLC (RB) [22]</td><td>Data-centric</td><td>69.07</td><td>69.07</td><td>79.49</td><td>91.33</td><td>91.34</td><td>81.34</td><td>64.15</td></tr><tr><td>SVLC (LLM+RB) [22]</td><td>Data-centric</td><td>64.08</td><td>66.67</td><td>75.80</td><td>81.07</td><td>88.13</td><td>76.78</td><td>62.73</td></tr><tr><td>SPARCL</td><td>Data/Objective-centric</td><td>78.77</td><td>81.38</td><td>92.43</td><td>93.49</td><td>95.82</td><td>88.95</td><td>78.94</td></tr><tr><td>TripletCLIP</td><td>Data/Objective-centric</td><td>69.80</td><td>69.82</td><td>90.40</td><td>86.27</td><td>94.43</td><td>85.53</td><td>80.94</td></tr><tr><td>DeGLA</td><td>Objective-centric</td><td>81.6</td><td>82.1</td><td>93.8</td><td>95.7</td><td>94.5</td><td>92.6</td><td>84.2</td></tr><tr><td>CE-CLIP</td><td>Objective-centric</td><td>72.8</td><td>72.4</td><td>93.4</td><td>93.1</td><td>93.1</td><td>88.8</td><td>79.0</td></tr><tr><td>OC-CLIP<sub>B-16</sub></td><td>Architecture-centric</td><td>76.6 ± 0.6</td><td>87.5 ± 0.5</td><td>91.1 ± 0.4</td><td>83.8 ± 1.0</td><td>94.6 ± 0.4</td><td>87.9 ± 0.1</td><td>76.0 ± 0.4</td></tr><tr><td>OC-CLIP<sub>B-14</sub></td><td>Architecture-centric</td><td>83.5 ± 0.2</td><td>88.9 ± 0.6</td><td>92.8 ± 0.1</td><td>84.8 ± 0.1</td><td>95.9 ± 0.4</td><td>89.2 ± 0.1</td><td>79.6 ± 0.3</td></tr></tbody></table>

Finally, there is a noticeable disconnect between strong results on benchmarks and reliable performance in real-world applications. The following subsections explore each of these challenges in detail and suggest practical ways to improve data, evaluation methods, and training approaches.

**Synthetic Data Quality and Hallucination Risk** A growing body of work finds that synthetic negatives/positives made with LLMs or diffusion are often higher quality than hand-written templates: LLM-crafted hard negatives are rated more fluent and plausible than templated ones in SugarCrepe [99]; large-scale caption rewrites in LaCLIP broaden phrasing and lexical coverage beyond fixed rules [24]; SVLC-style sequence generation expands beyond simple swaps [21]; and diffusion-based editors in TripletCLIP/BiVLC generate visual counterfactuals that templates cannot reach [22, 100]. However, these gains come with new risks:

**Hallucinations and factual errors.** As explicitly noted in LaCLIP[24], LLMs are trained on noisy web data and their rewrites may contain factual errors or hallucinations, so models trained/fine-tuned on these augmented captions can inherit those flaws.

**Precision of negative images.** Negative-image sample generation often leverages diffusion-based image synthesis (e.g., SDXL-Turbo) [112] to produce “hard” negatives for vision-language contrastive training. The approach was popularized by TripletCLIP[22], which explicitly synthesizes both negative images and captions for training [22]. However, several follow-on papers caution that such synthetic images are “not always precise” and thus require filtering (e.g., with CLIP-similarity thresholds) to ensure useful supervision. For example, the TripletCLIP pipeline includes an ablation step on “filtering high-quality image-text pairs from TripletData” as part of its training recipe [22]. Other related works also emphasize this filtering necessity. [58] note that “efficiently generating large amounts of images with precise variations remains challenging due to the inherent limitations of generative models,” and incorporate
mechanisms to downweight or discard low-quality synthetic samples. Similarly, [113] proposes automatic filters that remove low-quality generated images using similarity to real images to maintain training signal quality.

***Hidden shortcut learning.*** Gurung et al.[114] show that hard negatives alone are insufficient for CLIP to learn robust object-attribute binding: scaling batches (implicitly adding more negatives) or explicitly adding synthetic negatives does not resolve binding failures unless specific data properties are satisfied, and observed gains often reflect shortcuts rather than genuine binding (e.g., lexical or distributional cues). Complementarily, [99] demonstrates that earlier compositional benchmarks were hackable and overstated gains from data augmentation or “compositional” training; once these biases are removed via LLM-crafted, adversarially refined negatives, many reported improvements become small. These findings suggest that synthetic negatives may teach models to exploit dataset artifacts (fluency/style cues, template patterns) rather than compositional reasoning, especially when negative generation is noisy (caption edits, hallucinated content, or misaligned image–text pairs).

***Biases and Evaluation Validity in Benchmarks*** Benchmark quality remains a central bottleneck. [99, 44] reveal that many prior datasets use non-plausible or non-fluent template-based negatives; blind language-only models can outperform vision-conditioned models on these tests. Because many “hard negatives” are detectable from wording alone, that is, they are implausible, ungrammatical, or templated in a way that reveals the correct answer without needing the image. Hence, the success comes from linguistic artifact detection, not visual grounding, so the benchmark rewards shortcut reasoning rather than genuine image–text understanding. Although [99] mitigates plausibility/fluidity biases using LLM (ChatGPT in this case), its authors concede that they address only two human-interpretable biases; more less obvious artifacts may persist. There is also a monoculture of evaluation: most research focuses on CLIP-style dual-encoder retrieval models. Generative VLMs and instruction-tuned models remain largely untested, and evaluation protocols vary widely, making head-to-head comparisons difficult. Future benchmarks should provide text-to-image as well as image-to-text retrieval tasks, cover multiple languages and richer relation categories (BiVLC [100] and SugarCrepe [99] are English-only and reuse SugarCrepe’s categories), include hidden “blindfolded” baselines to verify that vision is truly used, and standardize metrics to enable fair cross-model comparisons.

***Subtle Benchmark Biases That Skew Compositionality Results*** Beyond plausibility/fluency artifacts, several under-recognized biases can systematically skew measured progress. In particular, suffix (prompt-end) insensitivity can undermine attribute binding: recent fine-grained analyses of CLIP report that the text encoder prioritizes first-mentioned objects, with substantial drops when token order is altered (even under semantics-preserving rephrasings) [39] implicating de-emphasis of tail information in multi-object captions. Accordingly, decreases in multi-object binding accuracy may partly reflect length and ordering sensitivity rather than true binding failures. Second, positional bias (left↔right and top↔bottom) can contaminate temporal/ordering judgments: as shown in TemporalVQA [13], two-image setups often pick up layout cues (e.g., “earlier = left/top”), so naïve 50/50 counterbalancing is insufficient when models key on position. Evaluations should randomize horizontal vs. vertical layouts and index order, and report both-direction accuracies plus the consistency drop under swap permutations (i.e., accuracy only when the answer is invariant to swapping positions). Third, *cardinality imbalance* in counting benchmarks performance drops at higher counts (e.g., ≥ 7) [9]. Reporting curves by count avoids inflated results that hide failures on rare cases.

***Stability–Plasticity Trade-off in Training*** Targeted fine-tuning to boost compositional skills often comes with *catastrophic forgetting*: zero-shot recognition and domain transfer degrade as the model overfits to the fine-tuning data. Evidence of this trade-off appears across data-centric compositional works; e.g., “negatives-only” fine-tuning can hurt generalization[57], SVLC reports small but consistent drops in zero-shot accuracy after structure-aware supervision[21]. Mitigations explored in the literature include pairing *hard positives* with *hard negatives* to teach invariances and reduce oversensitivity[57]; parameter-efficient updates (adapters/LoRA) [64] to preserve the backbone’s zero-shot behavior while injecting compositional cues[59, 23]. Although continual learning for CLIP-style VLMs has begun to receive attention; e.g., [115] and [116] propose adapter-based or probabilistic strategies to mitigate catastrophic forgetting in class/task-incremental settings; these methods do not target compositionality. Continual learning for compositional skills (rather than categories/domains) remains essentially unexplored. Future research should investigate regularization and continual learning strategies that add compositional skills without erasing existing knowledge and evaluate them across diverse tasks.

***Ongoing Gaps in Negation, Counting, and Generalization*** Even with advanced augmentation, models still struggle with some fundamentals, in particular, negation (“not a dog”) and counting (“exactly three cars”) remain unsolved: models often ignore negation and confuse adjacent numbers. Similarly, systematic generalization; that is, understanding unseen object–attribute pairs; still causes performance to drop sharply. Future work should develop explicit logical supervision for negation and quantifiers, rather than memorizing co-occurrence statistics.

***Underexplored Temporal and Multimodal Reasoning*** Most compositional benchmarks use static images and short captions. Temporal reasoning remains largely unexplored: models can answer questions about single frames but struggle with event order, duration, or cause–effect in videos. Video benchmarks require grounding actions and tracking objects
over time, yet there is a lack of datasets that scale from seconds to minutes and maintain visual grounding. Additionally, embodied or robotic applications demand compositional reasoning under viewpoint changes and sequential instruction; current models often fail on multi-step tasks despite good performance on web benchmarks [117].

**Head-to-Head Comparisons** We frequently noted the absence of head-to-head comparisons between proposed methods on the same benchmarks. The literature is disjointed and inconsistent: different works evaluate on different subsets of benchmarks and use different training data. Without unified evaluation, it is hard to know whether gains come from the method itself or from differences in data, model size, or tuning. Future research should adopt shared baselines for each compositionality skill, ablate each component, and publish compositional benchmark results to facilitate fair comparisons.

Overall, despite steady benchmark gains, transfer to real-world decision making remains fragile. In robotics and interactive settings, systems must combine attributes, relations, and counts under changing viewpoints. Pipelines built on VLMs still fail on multi-step, composition-heavy instructions despite impressive web-benchmark progress [117]. Embodied tasks require maintaining compositional understanding across temporal sequences, and generalizing to novel viewpoints. Designing training data, control objectives, and evaluation protocols that reliably transfer compositional skills to safe, real-world action remains a central open challenge.

# 7 Conclusion

This survey has examined the growing body of work on improving compositional understanding in CVLMs. We have organized the field into five main approaches: data-centric methods that reshape training examples, objective-centric methods that modify learning criteria, architecture-centric methods that restructure model design, inference-time methods that enhance frozen models, and diagnostic studies that investigate failure modes. This review highlights a few common patterns. Compositionality is not a single skill but a collection of related abilities, including: binding attributes to objects, understanding spatial and temporal relations, handling negation and counting, and generalizing to novel combinations. Despite steady progress on benchmarks, fundamental challenges remain. Models still struggle with negation, precise counting, and systematic generalization to unseen compositions. Moreover, improvements on benchmarks do not always translate to reliable performance in real-world applications.

Looking forward, several directions stand out as particularly promising. The field needs better evaluation methods that verify genuine visual grounding rather than linguistic shortcuts. Training strategies must balance compositional improvements with maintaining zero-shot capabilities, avoiding catastrophic forgetting. Future work should extend beyond static images to temporal reasoning in videos and spatial understanding in 3D environments. Most importantly, the gap between benchmark performance and real-world reliability must be addressed through more rigorous testing in embodied and interactive settings. As VLMs become increasingly central to applications ranging from autonomous systems to assistive technologies, robust compositional understanding will be essential for safe and effective deployment. The methods and insights surveyed here provide a foundation for building systems that can truly understand how visual and linguistic elements combine, not just detect their co-occurrence.

# References

[1] Max J Cresswell. *Logics and languages*. Routledge, 2016.

[2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In *International conference on machine learning*, pages 8748–8763. PmLR, 2021.

[3] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In *The Eleventh International Conference on Learning Representations*, 2023. URL https://openreview.net/forum?id=KRLUvxh8uaX.

[4] Yutaro Yamada, Yingtian Tang, Yoyo Zhang, and Ilker Yildirim. When are lemons purple? the concept association bias of vision-language models. *arXiv preprint arXiv:2212.12043*, 2022.

[5] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. @ crepe: Can vision-language foundation models reason compositionally? In *CVPR*, 2023.

[6] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 4971–4980, 2018.

[7] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. *ACM transactions on Graphics (TOG)*, 42(4):1–10, 2023.
[8] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 5238–5248, 2022.

[9] Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 3170–3180, 2023.

[10] Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip Torr, Yoon Kim, and Marzyeh Ghassemi. Vision-language models do not understand negation. *CoRR*, abs/2501.09425, January 2025. URL https://doi.org/10.48550/arXiv.2501.09425.

[11] Paola Cascante-Bonilla, Yu Hou, Yang Trista Cao, Hal Daumé III, and Rachel Rudinger. Natural language inference improves compositionality in vision-language models. *CoRR*, 2024.

[12] Xingrui Wang, Wufei Ma, Tiezheng Zhang, Celso M de Melo, Jieneng Chen, and Alan Yuille. Spatial457: A diagnostic benchmark for 6d spatial reasoning of large multimodal models. In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 24669–24679, 2025.

[13] Mohamed Fazli Imam, Chenyang Lyu, and Alham Fikri Aji. Can multimodal llms do visual temporal understanding and reasoning? the answer is no! *arXiv preprint arXiv:2501.10674*, 2025.

[14] Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision-language models on natural adversarial samples. In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024. URL https://openreview.net/forum?id=Dx88A9Zgnv.

[15] Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models. In *Advances in neural information processing systems*, 2024.

[16] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. *Advances in neural information processing systems*, 36:34892–34916, 2023.

[17] Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-level visual knowledge. In *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.

[18] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10684–10695, 2022.

[19] Do Huu Dat, Nam Hyeon-Woo, Po-Yuan Mao, and Tae-Hyun Oh. Vsc: Visual search compositional text-to-image diffusion model. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 19153–19162, 2025.

[20] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. *Advances in Neural Information Processing Systems*, 36:3536–3559, 2023.

[21] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 2657–2668, 2023.

[22] Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives. *Advances in neural information processing systems*, 37:32731–32760, 2024.

[23] Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and aligned captions (dac) promote compositional reasoning in vl models. *Advances in Neural Information Processing Systems*, 36:76137–76150, 2023.

[24] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving CLIP training with language rewrites. In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023. URL https://openreview.net/forum?id=SVjDiiVySh.

[25] Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, et al. Structure-clip: Towards scene graph knowledge to enhance multi-modal structured representations. In AAAI, 2024.
[26] Harman Singh, Pengchuan Zhang, Qifan Wang, Mengjiao Wang, Wenhan Xiong, Jingfei Du, and Yu Chen. Coarse-to-fine contrastive learning in image-text-graph space for improved vision-language compositionality. In *The 2023 Conference on Empirical Methods in Natural Language Processing*, 2023. URL https://openreview.net/forum?id=vgaJRhYVje.

[27] Ruisu Zhang, Yicong Chen, and Kangwook Lee. Improving CLIP counting accuracy via parameter-efficient fine-tuning. *Transactions on Machine Learning Research*, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=IZrt6hB2sI.

[28] Chaofan Tao, Gukyeong Kwon, Varad Gunjal, Hao Yang, Zhaowei Cai, Yonatan Dukler, Ashwin Swaminathan, R. Manmatha, Colin Jon Taylor, and Stefano Soatto. NAVERO: Unlocking fine-grained semantics for video-language compositionality. *arXiv preprint arXiv:2408.09511*, 2024. URL https://arxiv.org/abs/2408.09511.

[29] Avik Pal, Max van Spengler, Guido Maria D'Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision-language models. In *The Thirteenth International Conference on Learning Representations*, 2025. URL https://openreview.net/forum?id=3i13Gev2hV.

[30] Samyadeep Basu, Shell Xu Hu, Maziar Sanjabi, Daniela Massiceti, and Soheil Feizi. Distilling knowledge from text-to-image generative models improves visio-linguistic reasoning in clip. *arXiv preprint arXiv:2307.09233*, 2023.

[31] Xiaoxing Hu, Kaicheng Yang, Jun Wang, Haoran Xu, Ziyong Feng, and Yupei Wang. Decoupled global-local alignment for improving compositional understanding. *arXiv preprint arXiv:2504.16801*, 2025.

[32] Shaoan Xie, Lingjing Lingjing, Yujia Zheng, Yu Yao, Zeyu Tang, Eric P Xing, Guangyi Chen, and Kun Zhang. Smartclip: Modular vision-language alignment with identification guarantees. In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 29780–29790, 2025.

[33] Darina Koishigarina, Arnas Uselis, and Seong Joon Oh. Clip behaves like a bag-of-words model cross-modally but not uni-modally. *arXiv preprint arXiv:2502.03566*, 2025.

[34] Kenan Jiang, Xuehai He, Ruize Xu, and Xin Eric Wang. Comclip: Training-free compositional image and text matching. CoRR, abs/2211.13854, 2022. URL https://doi.org/10.48550/arXiv.2211.13854.

[35] Rim Assouel, Pietro Astolfi, Florian Bordes, Michal Drozdzal, and Adriana Romero-Soriano. OC-CLIP: Object-centric binding in contrastive language-image pretraining, 2025. URL https://openreview.net/forum?id=a84AD957m9.

[36] Fiorenzo Parascandolo, Nicholas Moratelli, Enver Sangineto, Lorenzo Baraldi, and Rita Cucchiara. Causal graphical models for vision-language compositional understanding. In *The Thirteenth International Conference on Learning Representations*, 2025. URL https://openreview.net/forum?id=haJHr4UsQX.

[37] Guangyue Xu, Joyce Chai, and Parisa Kordjamshidi. Gipcol: Graph-injected soft prompting for compositional zero-shot learning. In WACV, 2024.

[38] Madhukar Reddy Vongala, Saurabh Srivastava, and Jana Kosecka. Compositional image-text matching and retrieval by grounding entities. In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 241–250, 2025.

[39] Reza Abbasi, Ali Nazari, Aminreza Sefid, Mohammadali Banayeeanzade, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah. Clip under the microscope: A fine-grained analysis of multi-object representation. In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 9308–9317, 2025.

[40] Sonal Kumar, Sreyan Ghosh, Sakshi Singh, Utkarsh Tyagi, and Dinesh Manocha. Do vision-language models understand compound nouns? CoRR, 2024.

[41] Martha Lewis, Nihal V Nayak, Peilin Yu, Jack Merullo, Qinan Yu, Stephen H Bach, and Ellie Pavlick. Does clip bind concepts? probing compositionality in large image models. In *EACL (Findings)*, 2024.

[42] Amita Kamath, Jack Hessel, and Kai-Wei Chang. Text encoders bottleneck compositionality in contrastive vision-language models. In *The 2023 Conference on Empirical Methods in Natural Language Processing*, 2023. URL https://openreview.net/forum?id=hgF8In32gL.

[43] Ashwath Vaithinathan Aravindan, Abha Jha, and Mihir Kulkarni. Do vlms have bad eyes? diagnosing compositional failures via mechanistic interpretability. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 704–712, 2025.

[44] Chenwei Wu, Li Erran Li, Stefano Ermon, Patrick Haffner, Rong Ge, and Zaiwei Zhang. The role of linguistic priors in measuring compositional generalization of vision-language models. In *Proceedings on*, pages 118–126. PMLR, 2023.
[45] Vishaal Udandarao, Mehdi Cherti, Shyamgopal Karthik, Jenia Jitsev, Samuel Albanie, and Matthias Bethge. A good crepe needs more than just sugar: Investigating biases in compositional vision-language benchmarks. *arXiv preprint arXiv:2506.08227*, 2025.

[46] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In *International conference on machine learning*, pages 4904–4916. PMLR, 2021.

[47] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In *International conference on machine learning*, pages 12888–12900. PMLR, 2022.

[48] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In *International conference on machine learning*, pages 19730–19742. PMLR, 2023.

[49] Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Learning visual representation from modality-shared contrastive language-image pre-training. In *European Conference on Computer Vision*, pages 69–87. Springer, 2022.

[50] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. *arXiv preprint arXiv:2104.08860*, 2021.

[51] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In *European conference on computer vision*, pages 1–18. Springer, 2022.

[52] Reza Abbasi, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah. Deciphering the role of representation disentanglement: Investigating compositional generalization in clip models. In *European Conference on Computer Vision*, pages 35–50. Springer, 2024.

[53] Xin Huang, Ruibin Li, Tong Jia, Wei Zheng, and Ya Wang. Visual perturbation and adaptive hard negative contrastive learning for compositional reasoning in vision-language models. *arXiv preprint arXiv:2505.15576*, 2025.

[54] Amit Peleg, Naman Deep Singh, and Matthias Hein. Advancing compositional awareness in clip with efficient fine-tuning. *arXiv preprint arXiv:2505.24424*, 2025.

[55] Shixuan Liu, Yiwei Lyu, Honglak Lee, and Todd C Hollon. An empirical study of CLIP fine-tuning with similarity clusters. In *NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability*, 2024. URL https://openreview.net/forum?id=NmNmlAEBA1.

[56] Yeyuan Wang, Dehong Gao, Lei Yi, Linbo Jin, Jinxia Zhang, Libin Yang, and Xiaoyan Cai. Enhancing fine-grained vision-language pretraining with negative augmented samples. *arXiv e-prints*, pages arXiv-2412, 2024.

[57] Amita Kamath, Cheng-Yu Hsieh, Kai-Wei Chang, and Ranjay Krishna. The hard positive truth about vision-language compositionality. In *European Conference on Computer Vision*, pages 37–54. Springer, 2024.

[58] Haoxin Li and Boyang Li. Enhancing vision-language compositional understanding with multimodal synthetic data. *CoRR*, 2025.

[59] Zexi Jia, Chuanwei Huang, Hongyan Fei, Yeshuang Zhu, Zhiqiang Yuan, Ying Deng, Jiapei Zhang, Jinchao Zhang, and Jie Zhou. A visual leap in clip compositionality reasoning through generation of counterfactual sets. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 23498–23507, 2025.

[60] Paola Cascante-Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gul Varol, Aude Oliva, Vicente Ordonez, Rogerio Feris, et al. Going beyond nouns with vision & language models using synthetic data. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 20155–20165, 2023.

[61] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata, 58(2), 2021.

[62] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 4015–4026, 2023.

[63] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 9879–9889, 2020.
[64] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. *ICLR*, 1(2):3, 2022.

[65] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. *arXiv preprint arXiv:2302.13971*, 2023.

[66] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In *Proceedings of the IEEE international conference on computer vision*, pages 1501–1510, 2017.

[67] Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei-Ying Ma. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 6609–6618, 2019.

[68] Youngtaek Oh, Jae-Won Cho, Dong-Jin Kim, In So Kweon, and Junmo Kim. Preserving multi-modal capabilities of pre-trained vlms for improving vision-linguistic compositionality. *CoRR*, abs/2410.05210, 2024. URL https://doi.org/10.48550/arXiv.2410.05210.

[69] Ioana Bica, Anastasija Ilic, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, and Jovana Mitrovic. Improving fine-grained understanding in image-text pre-training. In *Forty-first International Conference on Machine Learning*, 2024. URL https://openreview.net/forum?id=5nxIRQ8GNa.

[70] Yunan Zeng, Yan Huang, Jinjin Zhang, Zequn Jie, Zhenhua Chai, and Liang Wang. Investigating compositional challenges in vision-language models for visual grounding. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 14141–14151, 2024.

[71] Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13774–13784, 2024.

[72] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam. Hyperbolic image-text representations. In *ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls*, 2023. URL https://openreview.net/forum?id=AkUs5xCcDH.

[73] Avik Pal, Max van Spengler, Guido Maria D'Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision-language models. In *The Thirteenth International Conference on Learning Representations*.

[74] Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Iterated learning improves compositionality in large vision-language models. In *CVPR*, 2024.

[75] Jihoon Kwon, Kyle Min, and Jy-yong Sohn. Enhancing compositional reasoning in clip via reconstruction and alignment of text descriptions. *arXiv preprint arXiv:2510.16540*, 2025.

[76] Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 26700–26709, 2024.

[77] Marc Law, Renjie Liao, Jake Snell, and Richard Zemel. Lorentzian distance learning for hyperbolic representations. In *International Conference on Machine Learning*, pages 3672–3681. PMLR, 2019.

[78] Ali Abdollah, Amirmohammad Izadi, Armin Saghafian, Reza Vahidimajd, Mohammad Mozafari, Amirreza Mirzaei, Mohammadmahdi Samiei, and Mahdieh Soleymani Baghshah. Comalign: Compositional alignment in vision-language models. *arXiv preprint arXiv:2409.08206*, 2024.

[79] Khoi Pham, Chuong Huynh, Ser-Nam Lim, and Abhinav Shrivastava. Composing object relations and attributes for image-text matching. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 14354–14363, 2024.

[80] Yuxia Geng, Runkai Zhu, Jiaoyan Chen, Jintai Chen, Xiang Chen, Zhuo Chen, Shuofei Qiao, Yuxiang Wang, Xiaoliang Xu, and Sheng-Jun Huang. Graph-guided cross-composition feature disentanglement for compositional zero-shot learning. In *Findings of the Association for Computational Linguistics: ACL 2025*, pages 2678–2690, 2025.

[81] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. *arXiv preprint arXiv:1308.3432*, 2013.

[82] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2901–2910, 2017.
[83] Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari Morcos. Pug: Photorealistic and semantically controllable synthetic data for representation learning. *Advances in Neural Information Processing Systems*, 36:45020–45054, 2023.

[84] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training. In *International Conference on Learning Representations*, 2022. URL https://openreview.net/forum?id=cpDhcsEDC2.

[85] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.

[86] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In *European conference on computer vision*, pages 213–229. Springer, 2020.

[87] Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. In *ECCV (80)*, 2024.

[88] Elias Kempf, Simon Schrodi, Max Argus, and Thomas Brox. When and how does CLIP enable domain and compositional generalization? In *Forty-second International Conference on Machine Learning*, 2025. URL https://openreview.net/forum?id=Lktwi30g63.

[89] Jin Wang, Shichao Dong, Yapeng Zhu, kelu Yao, Weidong Zhao, Chao Li, and Ping Luo. Diagnosing the compositional knowledge of vision language models from a game-theoretic view. In *Forty-first International Conference on Machine Learning*, 2024. URL https://openreview.net/forum?id=PzjDsfYwLC.

[90] John Charles Harsanyi. *A bargaining model for the cooperative n-person game*. Stanford University, 1959.

[91] Meta GenAI. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

[92] Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. arXiv preprint arXiv:2306.07915, 2023.

[93] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. *International journal of computer vision*, 123(1):32–73, 2017.

[94] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 6700–6709, 2019.

[95] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In *European conference on computer vision*, pages 740–755. Springer, 2014.

[96] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. *Transactions of the association for computational linguistics*, 2:67–78, 2014.

[97] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 3558–3568, 2021.

[98] Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai, and Jiankang Deng. Rwkv-clip: A robust vision-language representation learner. CoRR, 2024.

[99] Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023. URL https://openreview.net/forum?id=Jsc7WSCZd4.

[100] Imanol Miranda, Ander Salaberria, Eneko Agirre, and Gorka Azkune. BiVLC: Extending vision-language compositionality evaluation with text-to-image retrieval. In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024. URL https://openreview.net/forum?id=WMzQIP7000.

[101] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. CoRR, abs/2311.12908, 2023. URL https://doi.org/10.48550/arXiv.2311.12908.
[102] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. *International journal of computer vision*, 88(2):303–338, 2010.

[103] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 5288–5296, 2016.

[104] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In *33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Annual Conference on Innovative Applications of Artificial Intelligence, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019*, pages 590–597. AAAI press, 2019.

[105] Saeed Anwar, Muhammad Tahir, Chongyi Li, Ajmal Mian, Fahad Shahbaz Khan, and Abdul Wahab Muzaffar. Image colorization: A survey and dataset. *arXiv preprint arXiv:2008.10774*, 2020.

[106] Jiaxuan Li, Junwen Mo, Duc Minh Vo, Akihiro Sugimoto, and Hideki Nakayama. Nemo: Can multimodal llms identify attribute-modified objects? *CoRR*, abs/2411.17794, 2024. URL https://doi.org/10.48550/arXiv.2411.17794.

[107] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In *Proceedings of the IEEE international conference on computer vision*, pages 5803–5812, 2017.

[108] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 961–970, 2015.

[109] Hritik Bansal, Yonatan Bitton, Idan Szpektor, Kai-Wei Chang, and Aditya Grover. Videocon: Robust video-language alignment via contrast captions. In *ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models*, 2024. URL https://openreview.net/forum?id=0sAI7ROD82.

[110] Xin Wang, J Wu, J Chen, L Fi, YF Wang, and WY Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and. *Language Research*, 2020.

[111] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with temporal language. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 1380–1390, 2018.

[112] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In *European Conference on Computer Vision*, pages 87–103. Springer, 2024.

[113] Brandon Abreu Smith, Miguel Farinha, Siobhan Mackenzie Hall, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain. Balancing the picture: Debiasing vision-language datasets with synthetic contrast sets, 2024. URL https://openreview.net/forum?id=FwdnG0xR02.

[114] Bijay Gurung, David T Hoffmann, and Thomas Brox. Clip won't learn object-attribute binding from natural data and here is why. *arXiv e-prints*, pages arXiv:2507, 2025.

[115] Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for vision-language models. *Advances in neural information processing systems*, 37:129146–129186, 2024.

[116] Songlin Dong, Chenhao Ding, Jiangyang Li, Jizhou Han, Qiang Wang, Yuhang He, and Yihong Gong. Beyond clip generalization: Against forward&backward forgetting adapter for continual learning of vision-language models. *arXiv preprint arXiv:2505.07690*, 2025.

[117] Sanjay Haresh, Daniel Dijkman, Apratim Bhattacharyya, and Roland Memisevic. Clevrskills: Compositional language and visual reasoning in robotics. In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024. URL https://openreview.net/forum?id=64sZtFS0h6.