Causal Graphical Models for Vision-Language Compositional Understanding
Authors: Fiorenzo Parascandolo
Nicholas Moratelli
Enver Sangineto
Lorenzo Baraldi
Rita Cucchiara
Venue: ICLR 2025 Poster (2025)
Keywords: Compositionality
Vision-Language Models
Causal Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent work has empirically shown that Vision-Language Models (VLMs) struggle
to fully understand the compositional properties of the human language, usually
modeling an image caption as a “bag of words”. As a result, they perform
poorly on compositional tasks, which require a deeper understanding of the different
entities of a sentence (subject, verb, etc.) jointly with their mutual relationships
in order to be solved. In this paper, we model the dependency relations
among textual and visual tokens using a Causal Graphical Model (CGM), built using
a dependency parser, and we train a decoder conditioned by the VLM visual
encoder. Differently from standard autoregressive or parallel predictions, our decoder’s
generative process is partially-ordered following the CGM structure. This
structure encourages the decoder to learn only the main causal dependencies in
a sentence discarding spurious correlations. Using extensive experiments on five
compositional benchmarks, we show that our method significantly outperforms
all the state-of-the-art compositional approaches by a large margin, and it also improves
over methods trained using much larger datasets. 
Our model weights and code are publicly available.

Published as a conference paper at ICLR 2025
CAUSAL GRAPHICAL MODELS FOR VISION- LANGUAGE COMPOSITIONAL UNDERSTANDING
Fiorenzo Parascandolo, Nicholas Moratelli, Enver Sangineto, Lorenzo Baraldi & Rita Cucchiara Almagel.ab aab, Dipartimento Ingegneria "Enzo Ferrari" University of Modena and Reggio Emilia name. sur www.uuueuuuuuuuuuuuuu moree. it
ABSTRACT
Recent work has empirically shown that Vision-Language Models (VLMs) strug- gle to fully understand the compositional properties of the human language, usu- ally modeling an image caption as a "bag of words". As a result, they perform poorly on compositional tasks, which require a deeper understanding of the dif- ferent entities of sentence (subject, verb, etc.) jointly with their mutual rela- tionships in order to solved. this paper, we model the dependency relations among textual and visual tokens using Causal Graphical Model (CGM), built us- ing a dependency parser, and we train a decoder conditioned by the VLM visual encoder. Differently from standard autoregressive parallel predictions, coder's generative process is partially-ordered following the CGM structure. This structure encourages the decoder learn only the main causal dependencies in a sentence discarding spurious correlations. Using extensive experiments on five compositional benchmarks, we show that our method significantly outperforms all the state-of-the-ar compositional approaches by large margin, and it also im- proves over methods trained using much larger datasets. Our model weights and code are publicly available
de-
INTRODUCTION
Vision-Language Models (VLMs) have shown impressive results in different tasks such as, for in- stance. zero-shot classification, image-text retrieval, vision-question answering, image-captioning and many others (Radford al. 2021] Li al. 2023b [Singh et al.] 202T] Liu al.. 2023). How- ever, despite this success, most VLMS still struggle in understanding the compositional nature of the human language. For instance, Yuksekgonul et al. (202)) empirically showed that common VLMs usually do not consider the order and the syntattic/semantic relations of words in a sen- tence, which is treated as a bag of words, where "the horse is cating the grass" and "the grass is eating the horse" can easily be confused. Jointly with Yuksekgonul et al. (2023), many other authors have recently proposed different compositional benchmarks which confirm the poor performance of common VLMs when tested against compositional tasks (Hsich et al.) 2024 Zhao et al. 2022 Burapacheep et al., 2024: Peng et al. 2024h. One of the probable reasons of this bag-of-words behavior is the contrastive loss used CLIP (Radford et al., (02T) (and in other VLMs), which compares single vector representing the textual encoder's 's output with single vector representing the visual encoder's output, sacrificing textual and visual details (Yuksekgonul al.) [023] Kamath et al. 2023: Basu et al., 2024). Another reason is the low quality of the captions used for VLM pre-training, which are usually noisy or do not describe the details of the image and the interactions among its objects (Doveh et al. 2023a).
Most of the compositional methods that have been recently proposed to alleviate this problem focus on creating annotations with a richer compositional structure, used to fine-tune VLM (Yuksekgonul
al. 2023: Doveh et al., 2023a] Cascante- Bonilla et al., (023). For instance, NegCI.IP (Yuksek- gonul et al.) 2023) creates hard negatives, in which the original caption is modified swapping the positions of some words, and these hard negatives are used jointly with common negatives to fine-
tune CLIP using the standard contrastive loss. However, the automatic creation of hard negatives is
https://gi thub. com/ waaaammmmala...... ab/Co11

---

Published as a conference paper at ICLR 2025
itself noisy, leading to captions which often do not have correct syntactic/semantic meaning (this problem is inherited by some compositional benchmarks, see Sec. 4.. In (Tschannen et al.) 2023), a
VLM is pre-trained from scratch using captioning strategy and a huge private dataset. Specifically,
the authors propose both Cap, where the pre-training strategy is a standard AutoRegressive (AR)
next-token prediction, and CapPa, where the AR training is mixed with a parallel training (Bolelli
et al., 20188. in which all the textual tokens are simultaneously predicted. Tschannen et al. (2023)
show that both Cap and CapPa achieve excellent results on compositional tasks, and argue that generative training encourages the VLM to focus on fine- fine-grained descriptions of the visual content.
In this paper, inspired by Cap and CapPa, we propose a VLM adaptation approach for compositional reasoning which is based on a decoder trained with a captioning strategy. However, differently from the standard fully-sequential AR and the parallel predictions used in (Tschannen et al. 2023 we propose a partially ordered, semi-paralle AR prediction strategy which is guided by the dependency relations of a Causal Graphical Model (CGM) (Schölkopf et al., 202T). In more detail, we use an off- sffeth dependency parser (Dozat& & Manning 2016), which creates syntactic tree from given textual sentence. Specifically, given a caption, a dependency parser automatically builds a De- pendency Tree (DT), in which each node is associated with caption word and each edge represents a syntactic dependency relation between two words (Fig. [T. The DT. jointly with the visual features extracted from the image using frozen visual encoder, are used build a CGM, which describes the dependency relations among image patches and textual tokens. Our token prediction strategy based on the dependency relations contained in this CGM. The rationale behind this approach is illustrated in Fig. | using the caption "A brown bird has a small yellow head", For instance, in the resulting DT. the adjective "brown" depends on the noun "bird". However, using a standard AR approach, where the token prediction order follows the English grammar, the captioning model should predict "brown" before knowing that this adjective refers to "bird", which is a quite ambigu- ous task, since many objects may be brown in the image. Conversely, when our model predicts the adjective ("brown")) knows the noun ("bird"") it refers to, thus the word generation can be specific to the entities, the attributes and the relations contained in the input image. Generally speaking, factorize the joint distribution of all the caption words following the disentangled factorization of a CGM (Schölkopf et al. 2021), and our semi-parallel AR model predicts token conditioned only on the tokens on which it depends. For instance, in the example of Fig. T "small" and "yellow" predicted in parallel and they conditionally independent given "head", thus no statistical de- pendence is learned between these two words. The advantage of this strategy is that the decoder can focus on learning only the main causal dependency relations, ignoring possible spurious asso- ciations (Pearl & Vermal 1995) induced by the sequential order of the words in natural language sentence. Moreover, we use the same prediction strategy also at inference time, when we compute the likelihood of a candidate caption. In this case too, the use of the CGM makes the likelihood estimation independent of spurious associations due to the sequential order of the words.
We validate our method using different VLMs (CLIP, XVLM (Zeng et al. (022) and InstructBLIP (Dai et al. 2023)). Using extensive experiments with five compositional datasets, we show that our approach largely outperforms all previous works, setting new state of the art in all the evaluated benchmarks, and that it also improves on Cap and CapPa, despite being trained on much less data.
RELATED WORK
Compositional Methods. Most of the compositional methods are based on creating annotated train- ing samples which force the VLM to acquire compositional knowledge. For instance, (Yuksekgonul
let al. 2023: Zhang et al.) 2024 Huang et al. 2024 Buettner & Kovashka, 2024, Momeni et al.
2023
Doveh et al. 2023b Singh et al. 2023 Oh al. 2024 Yellinek et al. 2023 Herzig et al.,
2023
use either a rule- based method or a Large Language Model (LLM) to create hard negatives
(Sec. [. which typically consist replacing or swapping the position of some words in the ground-
truth caption associated with a training image. In (Cascante-Bonill et al.) (023). dense captions are constructed using synthetic videos created with a 3D physics-based simulator, while (Singh et al., 2024) use real videos. In DAC (Doveh et al. 2023a), dense captions are created by combining the results of either an LLM (GPT-NEO-2) 2 2.7B) or segmentation network (SAM (Kirillov et al. 2023)) with a captioner (BLIP-2 ( et al. 2023b)). SAM is also used in (Sahin et al. al.] 2024) jointly with Stable Diffusion (Rombach et al. 2022) to generate hard negative images. Moreover, Stable Diffu-
sion is used in (Li et al. 2023a Clark & Jaini
2023 Krojer al. 2023) an alternative VLM. The
2

---

Published as a conference paper at ICLR 2025
Figure 1: Dependency relations between words in sentence. On the left, the DT (T) extracted from the caption shown above using dependency parser (Dozat & Manning 2016). On the right, the corresponding CGM (g). To improve readability, in we use different colors for different variable types and we omit the causal dependencies between visual (7) and textual (W) variables.
main idea is that the noise prediction error of the Diffusion Model (DM) (Ho al. 2020). obtained by feeding Stable Diffusion with a corrupted version of the test image and a given caption, can be used an estimate of the image-caption similarity. Finally, Stable Diffusion is used in (Basu al., 2024) as an additional regularization loss to fine-tune CLIP.
Apart from DM-based methods, most of the compositional approaches are based on fine-tuning or adapting CLIP. For instance, Dhsannce,Zhhang el al. (2024) use an hinge loss with a curriculun-learning based adaptive margin, while Doveh et (2023a) use a Multiple Instance Learning loss. Curriculum learning is used also in (Singh et al. 2023), while Zheng et al. (2024) iteratively retrain CLIP and represent an image using a sparse combination of codebook codes. Oh al. (2024) propose local hard negative loss to fine-tune CLIIP which is based on a dense alignment betweeen patch embed- dings and textual token embeddings. Waznn et al. (2024) use a dependency parser (see below) to extract triplets (subject, verb, object) from caption. Subjects and objects are represented as em- bedding vectors using the CLIP textual encoder, while verbs are represented by matrices that are multiplied with either the subject or the verb to change their meaning. In (Li et al., 2024a), CLIP is embedded in a larger VLM, which includes a detection network and an LLM. The LLM comu- nicates with the detection network using special tokens. A few methods use generative pre-trained VLMs, and their results usually show a large improvement with respect to encoder-based VLMs when applied to compositional tasks, most likely because the next-token prediction pre-training en- courages the VLM to learn the natural language compositional characteristics. For instance, Herzig al. (2023) use "Adaptiie Scene Graph Tokens" to adapt both CLIP and BL.IP-2 (Li al. 2023b) to predict scene graph information, and they show that the BLIP-2 based results are much higher than those based on CLIP. Wan et al. (2024) use LLaVA (Liu et al.) 2023) and a classifier-free guidance strategy, in which they compare the VLM prediction on two images: the original test image and modified version where the main objects are masked-out. BLIP is used also by (Lin et al., 2024), who focus mitigating the linguistic bias the VLM pre-training dataset. Finally, Tschannen cct al. (2023) propose two VLMs. called Cap and CapPa (see Sec. []] both trained generatively. Cap standard AR captioner, while CapPa is trained using combination of 25% AR next-token pre- diction and 75% fully-parallel token prediction. Inspired by the success of generative pre-training, in this paper we propose decoder trained using a semi-parallel prediction strategy, where the order in which future tokens are predicted depends on a CGM, and we show that it can be applied to both encoder-only and generative pre-trained VLMs, significatively boosting the results of both.
Causal Graphical Models. CGMs are used in causal learning to represent the causal relations among a set of variables (Schölkopf al.) 2021 Perry et al., (022). These relations are supposed to be known and are represented by the edges connecting each variable (node) in the graph with the variables on which it depends ("parents"). The joint distribution of all the variables of the CGM computed using the disentangled factorization (Schölkopf et al. [021 (eery et al., 2022). given by the product of all the conditional distributions of each variable with respect to its parents (App. A). The main advantage of this factorization is that, since it is assumed to be causally sufficient, the model does not need to learn other vatialv conditional distributions, in this way reducing the number of training samples necessary to learn the joint distribution (Schölkopf et al.] (021). As far as we know, the only work using CGMs for vision-language compositional tasks is (Jiang et al, 2024), where Independent Causal Mechanisms (ICMs) (Parascandolo et al. 2018 Goyal et al. 2021
3

---

Published as a conference paper at ICLR 2025
Schölkopf et al.,
2021) describe the relations between the subject, the object and the action of an image. However, the method proposed in Tiang et al., (024) is radically different from our proposal, being each ICM simply computed as the CLIP similarity between word and sub-image.
Syntactic Trees. In Dependency Grammars, dependency relations are syntactic and semantic con- nections between words in a sentence, where one word (called "head") governs or determines the grammatical behavior of the "dependent" word (Nivre) 2005). Given a sentence, these dependen- cies are organized in Dependency Tree (DT), which can be automatically extracted using a parser (Honnibal et al.
2020 Zhang et al.) 2020 Dozat & Manning 2016). In (Yang & Wan 2022) a
Language Model (LM) is trained to predict whether the future tokens in the sequence are head or dependent of previously observed token, and this prediction replaces the standard Maximum Like- lihood Estimation objective. Similarly,
(Deguchi et al., 2019) the dependency relations extracted by an external parser are learned by the LM and used to modulate the Transformer (Vaswani et al., 2017) attention maps. DT can also be used to compute a syntactic distance between words, which in turn can be used, e.g., as an additional loss (Du et al. 2020 or to modulate the attention maps (Hou et al, 2022). In the vision-language domain, DT's are used in (Song et al. 2022) to convert a textual question into template for CLIP fine-tuning, and in (Li et (0044b to replace and swap words in a sentence and fine-tune a VLM using the correction of the modified sequence as a pretext task. Finally, constituency parsers group words that belong sub-phrase. Constituency trees are used, e.g., in (Yellinek et al. (203) to generate sub-phrase spe- cific hard negative captions or in (Zhang et al. | (202)) to extended the contrastive loss by maximizing the similarity of words in the same sub- sub-phrase. Differently from previous work, we use a DT to extract syntactic and semantic dependencies between the words of sentence, and we interpret these dependencies as causal relations that guide the construction of our CGM.
specific grammatical category in
3 METHOD
Given an image-caption pair (X, C),, our goal to define a set of conditional distributions over the random variables associated with the image features and the caption words. For this purpose, as
anticipated in Sec. [ and 2 we use an off-the-shelf dependency parser (Dozat & Manning 2016)
which, for a specific C [wi....,
returns a DT T (Fig. [}. where each node corresponds
to a word and each edge (t, connects the "dependent" word wy with its "head" w; (Sec. 2}. T contains the syntactic and semantic dependencies between the words in C (Nivre) 2005). and make this dependency explicit by connecting each word to all the words it transitively depends on in the tree. Specifically, define CGM 9 by associating each word my with random variable Wj, corresponding to node of G. Moreover, we connect the node corresponding to W3 with all the variables corresponding to the ancestors of Wi in T (Fig. D]. Formally, if Wi ..., Wig are the ancestors of Wy in T, then we assume a causal dependence between the corresponding variables: W;, + w..... W., - W,. Furthermore, the parser labels each word in T with a syntactic type using a prefixed vocabulary V (Silveira et al. (0144] Zhang et al. 2020).. For instance, if type(w,)) nsubj E V, it means that w, is a noun and it plays the role of the subject in the sentence. Intuitively, we can think of these syntactic types as categorical syntactic features extracted from C, which we formally describe using n random variables S,..., Sn. where each S, ranges over V. In G, we assume that each W, depends on its corresponding syntactic variable Sy: S₁ + W,.
Finally, we extract a set of features from X using the VLM visual encoder E: = {(X) {z1: ...... Zm3 (details in Sec. 3.I. and, similarly to the textual case, we associate random variable Zy to each feature 2 E Z. In G, we assume that W, depends on all the visual variables: Z, - W;, Wi....? Zm + Wj. Using the above assumptions, we define the parents (Schölkopf al. 2021 of
W; as: (W, = ( . Wa,, Sj. Zi,..., } and we model the conditional joint distribution of the textual variables given the visual and the syntactic variables as:
n
P(W1. I..,.... W,|S Sn Zi..., Zm)
II P(W, (A(,), j=1
(1)
where the right side of Eq. ①1) is obtained using the disentangled factorization of CGMs (Schölkopi
et al. 2021 Perry et al. 2022 and assuming that Si,.., S, and Zi, ..., Zm are independent of each
Note that, for each C in the training/testing set, T needs be extracted only once and can be done offline.
4

---

Published as a conference paper at ICLR 2025
1-10
Hb
15
Coooddr LeW
Figure 2: schematic illustration of decoder.
other (see App. A for more details). In Sec. disentangled factorization both at training and at inference time.
3.1
we show how a VLM can adapted to predict this
Discussion. Tschannen et al. (002)) formulate the joint distribution of the words in C' using the standard AR prediction strategy commonly adopted by image captioning methods (Sec. T
P(W1.- Wn [1, 7m) II P(W, W 21, ....... 7m)
The advantage of our formulation (Eq. over Eq. 22 is that, in our case, the model needs to learn only the inter-variable conditional distributions indicated by the dependency parser, reducing the risk of overfitting on the training data (Schölkopf et al. 2021). Specifically, the dependency parser helps discarding those spurious associations (Pearl 2009 Pearl Verma 1995) contained in Eq. (2) which depend the sequence of words in C' but do not correspond to strict seman- tic/syntactic relation (c.g.. "small" and "yellow" in the example the dependency relations extracted by dependency parser as causal relations because they directly model the (linguistic) influence of the "head" variable with respect to the generation of the "depen- dent" variable. For instance, the probability values an adjective are directly influenced by the noun it refers to, because the adjective describes an attribute of that noun, thus the corresponding conditional probability is not spurious association (more details App.
Sec. [ In contrast, we interpret
While the causal dependency relations in C' may not be exhaustively described by G and there may other relations between words in C, we follow (Goyal & Bengio 2020), and we assume that, in a symbolic domain like the natural language, the joint distribution over the words of sentence should be sparse. This also in line with very recent work which shows that sparse attention in Transformers helps the network focus on the most relevant context and improves its performance removing noise (Leviathan et al. 2024 Ye et al.) 2024). Thus, we prefer sparseness to completeness and we assume that the word dependencies extracted by dependency parser are causally sufficient (Sec.F 2 with more details in App. A). Finally, Tschannen et al. (2023 propose also parallel prediction strategy, which corresponds to:
n
P(W,,... W. Z1, .... Zm) II P(W, Zm). 5=1
In Eq. (3). each W, is assumed to be conditionally independent from all the other textual variables given the visual variables. The empirical results reported (Tschannen et al., 2023) do not show clear winner between the AR and the parallel prediction, and the authors use mixture of the two strategies in training their VLM (Sec. 2b. Conversely, in our experiments (Sec. 4.1 and 4.2p show that proposed disentangled factorization (Eq. (I)) is better trade-off between the conditional independence of Eq. (3) and the standard image captioning factorization of Eq. (2). and
it also improves the mixed strategy adopted in (Tschannen et al.! 2023).
5

---

Published as a conference paper at ICLR 2025
3.1 USING A DECODER FOR CAUSAL PREDICTION
In this section, we show how textual tokens can be generated using our CGM. Note that our goal is not image captioning. but we use our method, which we call Causally-Ordered Generative Train- ing (COGT), for vision-language compositional understanding. Since CLIP is the most commonly adopted backbone by previous works on compositionality (Sec. 2. in the following we use CLIP as an example VLM, and in Sec. [.2 we show additional results obtained with other VLMs.
We freeze the CLIP visual encoder (E) and, from a given image X, we extract a set of features from the last (L) and the penultimate (L - 1) layer of &: 2 = { {#(co)) * ..., 2p= 2 L-1
[[cs]] 2 2/
For the 1-th layer of the encoder, #<cLs: is the embedding vector of the class token (Dosovitskiy et al. 2021], J. while z/..., z are the embedding vectors of the patch tokens. Using a grid of p patch tokens, have
2p +22 2. We use the embedding vectors of the penultimate layer jointly with the last layer features to help the model reasoning about smaller resolution objects. Indeed, previous work (Ghiasi et al., [202 Wysoczańska et al. 2024) showed that there is usually decrease in the amount of spatial information represented in the last layer of CLIP. Moreover, we use a mapping network M (Fig. 2 to reduce the dimensionality of the visual features to match the decoder embedding size. M is composed of a linear layer, preceded and followed by LayerNorm, and all features in Z are obtained as output of M. The parameters of M are learned jointly with our decoder (see below) and M is shared by all features in E and both layers of & (T. and I, 1).
We replace the CLIP textual encoder with our decoder D, a relatively small network, composed of only three blocks with ~64M total parameters, which is the module we use to adapt CLIP to solve compositional tasks. Fig. 2 shows the architecture of DD which takes as input 2n tokens. The first sequence of tokens are masked tokens, while the others are visible tokens, and we represent each wy with both a masked and a visible token. Specifically, to condition D with respect to the event S3 t (t E V) in Eq. (
masked tokens specific for each syntactic type & in V. In more detail, V is composed of the 45 standard syntactic categories defined in (Silveira et al. 2014) (see App. F.. We associate each category t with masked token MSK,- Then, for each word wy E C. type (w;) = L, then the masked token used for wj is MSK, This is simply implemented using lookup table of masked token embeddings, composed of 45 different initial embedding vectors (learned using standard backpropagation) and which extends the (single) masked token used in common masked-token prediction tasks (Kenton & Toutanoval 2019). The other n tokens are visible, standard textual tokens, one for each wy C. In this way, wy is represented both as visible token and as a masked token of type f. In a given layer of D. these two tokens are respectively represented by the masked-toke token embedding vector m, and the visible-token embedding vector vy.
Each block of D is composed of two layers. In the first layer, we compute the self-attention of each masked embedding m, with itself, jointly with the attention of m, with all the visible embeddings "i....., US2. where PA(W,) = (Wi..... Waz: Sy, Zz, 3. Note that there is no attention be- tween m,, and m2 with j1 # j2- In the same layer, we compute the self-attention of each visible embedding v, with itself, jointly with the attention of v with ...,. Us, (Fig. 2). Note that there is no information leak, since mj. later used for the final prediction, has no direct or indirect access to Uj. We call this Dependency Guided Attention to differentiate it from the standard self-attention (Fig. 2). In the second layer of each block of D. both the masked (m,) and the visible (v;) embed- dings pay attention to the visual features in Z using cross-attention, in this way implementing the dependence between W, and Zi...., Zyn Finally, after the last block of we discard the visible- token embeddings and we fed each masked-token final embedding to a linear layer computing a posterior distribution over the vocabulary of textual terms.
D is trained from scratch using as the only objective the maximization of the log-likelihood of the disentangled factorization:
C = log II P(W,)PA(W)) = Σ log ((W)))(())))
(4)
Inference Most of the compositional tasks are modeled as image-to-text retrieval tasks. In case of COGT, given a testing image X, we compute the log- likelihoddood of all the candidate testing captions and we select the highest scoring sentence. Note that computing 2 is independent of the specific

---

Published as a conference paper at ICLR 2025
caption C, and it can be done once for each image in the dataset. The log- lill-kieihodd is computed using Eq. and a semi parallel AR prediction strategy which follows the partial order induced by the DT. Specifically, using the dependency parser we extract T from candidate caption C. Then we proceed using a level order traversal of T, in which, starting from the root, we predict in parallel all the tokens of given level of the tree and then move to the next level.
EXPERIMENTS
In our evaluation we use four common compositional benchmarks: ARO (Yuksekgonul et al. 2023); SugarCrepe (Hsieh et al. 2024), VL-CheckList (Zhao et al. (2022) and ColorSwap (Burapacheep) et al., 2024), and an additional benchmark FG-OVD (Bianchi et al. 2024) which we propose in this paper. Most of them are composed of different tasks and datasets, and we report both the task- specific and the average accuracy across all tasks. Following (Zhang et al., 2024), we do not use COCO Order and Flickr Order (two of the ARO tasks) because it has been previously showed that "blind" LM, with no access to the image, can achieve about 99% accuracy on these tasks (Tschannen) et al. 2023). The reason of this is due the grammatical and semantic errors introduced in the negatives when swapping or replacing caption words (see Sec. D.. For instance, the sentence "with man is wearing ears the glasses pierced orange hat and" (Flickr Order) can be easily detected as false by an LM without any visual knowledge. In App. C.2 we show additional results with Winoground (Thrush et al., 2022). which, however, often not used by other methods based on CLIP (Zhang et al. 2024) because requires the VLM to be able to detect out-of-focus objects in low-resolution images (Diwan al., 2022). In contrast, we propose to use FG-OVD (Bianchi al.) 2024), benchmark originally proposed to evaluate the ability of open-vocabulay object detectors to discern fine-gra grained object properties. In FG-OVD, negative captions are created starting from the object-specific captions by replacing attributes referring to the object's color, material, texture, etc. We crop the objects' bounding boxes which we use jointly with positive and negative captions and image-to-text retrieval task (more details in App. B
Table I:
Comparison between different generative training strategies. The value +x reported in the i-th row, column Average, refers to the average improvement across all datasets with respect to the method in row
ARO
SugarCrepe
VL-Checklist
ColerSwap
PG-OVD
Avg
Model Pully-Parallel
Relation Attrübute
Avg
Replace Swap Avg
Attribute Object Relation
Arg
Arg
83.66
67.55
74.7
57.42
76.98
35.89
76.96
84.76
67.10.ta7
Sequenial-AR
84.86
77.87
81.37
8363
81,50
88.02
36.82
75.45
91.11
84.45
46.24
69.28.111
COGT
87.56
90.26
38.91
98.26
57.10
83.14 39.50
86.07
78.91
89.37
84.78
61.33
$1.48
75.20.543
4.1 ABLATIONS
In the experiments of this section we follow a widely adopted protocol, first proposed in YYuksek- gonul al. 2023), in which the VLM backbone is CLIP and the only training dataset is COCO (Lin et al. 20144. However, we do nol use the hard negatives of (Yuksekgonul et al., 2023) for training because of their frequent semantic and syntactic errors (see above). In Tab. we compare to each other the different word prediction strategies described in Sec. B usiin CLIP as the VLM. Specifi- cally, we indicate with Sequential-AR the replacement of our decoder with a standard AR decoder (cross-attention over Z and standard causal attention over the past words of the caption). trained using the common image captioning objective defined in Eq. (2). Similarly to COGT, we freeze the CLIP encoder and we use the visual features (Z) extracted from both the last and the penultimate layer of & (Sec. 3.I. We use a same-size decoder, which takes only visible words as input (details in App. Dp. Sequential-AR can be considered as re- implementation of Cap (Tschannen et al. 2023). trained on COCO and with a frozen visual encoder, which can directly be compared with the CGM-based strategy of COGT. Similarly, we indicate with Sarll--arllllll our re-implementation of the parallel prediction strategy proposed in (Tschannen et al., (023), using a decoder which takes as input only masked tokens (only cross-attention over Z with frozen visual encoder), trained us- ing Eq Finally, in Mixed we use the aqquential-parally mixed strategy adopted in CapPa, in
'There are no publicly available network weights for (Tschannen al.
2023
7

---

Published as a conference paper at ICLR 2025
Table Empirical contribution of different components of COGT.
ABO
SugarCCrepe
VL-Checklntt
Calertnop FG-OVD Ang
Parser
Sask-Specilic Lagers Relation
Aftributc
Aug
Add
Repliaee
Swap
Ang
Attributtc
Objjcc Relation
Aag
$5.66
59.34
$7.01
98.15
Bi 04
80.30
$7.60
$6.99
71.66
$7.00
K3.92
56.53
43.34
71.36
Deep Buttre
86.56
89.10
$7.83
M8.11
85:90
$1.49
58.00
57.11
76.30
57.75
14.35
01:33
44:74
71.34
Doep Buttine RoftERts
84.75
96.16
98
86
37
97.83
K3.29
76.24
50.8
64.29
50.00
46:90
71.91
Doep Bullice BoBERTa
96:32
78.34
45.00
64.79
75.20
which, following (Tschannen al. 2023), use 75% of the training samples with a parallel pre- diction (Eq. (3) and 25% of the samples with an AR prediction (Eq. 2). Architectural details are provided in App. D The results show that COGT outperforms all the other prediction strategies in all the datasets, often with a significant margin. For instance, COGT achieves an average accuracy improvement of +1.777 points across all datasets with respect to Fully-Parallel, which arguably shows that the conditional independence assumption in Eq. (3) is too strong. Overall, these results confirm that an off-the-shelf dependency parser provides a priori knowledge which can be exploited to model the conditional dependencies between words in a sentence.
We further investigate the role of the dependency parser in Tab. [ Specifically, the column Purser refers to the adopted dependency parser, where compare 3 different methods: Deep Biaffine (Dozat& & Manning 2016). CRFPar (Zhang et al. 2020) and Deep Biaffine + RoBERTa (Dozat & the parsers as black boxes, without any training or fine-tuning, and the differences in the corresponding rows of Tab. [ are based only in the use of a different external parser for COGT. Tab. [ shows that the best results correspond to the use of Deep Biaffine + RoBBRTa (Dozat & Manning 2016), which is aligned with the higher accuracy of this parser compared the other two according to the linguistic leaderboard Penn Tree Bank (Mareus et al. 1993]. Note also that, according to this widely adopted parser ranking (Marcus et al. 1993). there are higher performing parsers (e.g., (Mrini et al., 2019)). however their code not publill available or it is not easy to use. Thus, we opted for Deep Biaffine RoBERTa (used in all the other experiments of this paper). However, the results in Tab. show that, using better parser, COGT can most likely achieve even better results.
Manning, (016). Note that
The Mask-Specific column in Tab. 2 indicates the use of a dedicated masked token for each of the 45 syntactic categories of V (Sec. 3.1). which is compared with generic BERT-like masked token Kenton& & Toutanova] 2019). In the latter case, we use the same masked token initial embedding vector for all the masked tokens fed to D (replicated times), thus dropping any conditioning on
Sj in Eq. ( The results in Tab. 2 show that this corresponds to a - 2.69 point drop in accuracy averaged across all five datasets.
Finally, the Layers column in Tab. 2 indicates the number of layers of CLIP we use to extract the visual features Z: Layers 1 means only the last layer (L); Layers = 2 means that we use also the penultimate layer (Sec. 3.1). When the last layer only is used, the average accuracy drop is -4.75, showing the importance of using lower-level features in compositionality tasks where the VLM needs to consider small, non-foreground objects.
In App. C.I we provide the complete version of Tab. with all the possible combinations between the values of Parser, Mask-Specific and Layers, which confirm the results shown here.
4.2 MAIN EXPERIMENTS
Setting. In this section we compare COGT with state-of-the-art compositional methods. Since different works are based on different VLMs and use different training data, to make the comparison as fair as possible, we split our evaluation based on both the VLM backbone and the used training
set. Specifically, in Tab. 3 we group the approaches based on CLIP (Radford al.] 2021) and in Tab. those which adopt different VLM, while Tab. Tab.55]i dedicated to methods based on VLMs pre-trained using a language based decoder. In the first category, our approach is indicated by COGT-CLIP. In the second group, use XVLM (Zeng et al.) 2022 as our backbone (COGT- XVLM). Finally, we use InstructBLIP (Dai et al. 2023) for the VLM category with a language- based decoder
(more details in App. D. COGT-CLIP, COGT-XVLM and
COGT-InstrucBLLIP are trained on COCO only (~ 100K training samples, see Sec. 4.I.. Moreover, following (Zhang et al. 2024), we present additional results training on combination of three
8

---

Published as a conference paper at ICLR 2025
datasets: COCO, CC3M (Sharma al. 2018), and Visual Genome (Krishna et al. 2017). ) and call the corresponding methods as COGT-CLIP+, COGT-XVLM+ and this case, we use decoder D with four blocks. Note that we use only ~ 50K samples from Visual Genome because we removed those training data which overlap with ARO and VI.-Checklist. On the other hand, CC3M (~3.3M training samples) is much larger but also noisier dataset, since its captions are obtained from the Alt-text HTML attribute associated with web images, and we use it also to indirectly evaluate the robustness of COGT to noisy textual descriptions (see App. C.I)) For each compared baseline, the results shown in the tables refer to the values reported in the original article (when available) or to our reproduction using the (possibly available) public code. The results on FG-OVD are averaged over all tasks and we report in App. C.3 the task-specific values.
In
Table 3: Comparison with compositional methods based on CLIP. For each baseline, we report the values published in the original article. In case a given dataset was not used by that baseline, but public code is available, we report the results obtained by our reproduction. With x, x and x we indicate the first, the second and the third best result, respectively.
SugarCryy Cyppe
Vi-Cheekllet
Celortioap FG-OVD Arg
Model
Relution Attribute Avg
AM Reglace Snop Arg Atrritnte Objjct Metation Arg
ITT
Arg
Zenoabee
CL.P Rtalionl et
$9.00
50 50
99.76
70.33 N605
67.93
$2.83
64.19
71,65
35.57
$5.54
CDCO
C1.PP Pine:Turdd Nikeeeed
61.00
65:00
54 (0)
NayCLIP Vilozbenn (1140
81.00
71.00
76 00) $1.29
85.36
25.30
82.65
72.24
$7.00
71.99
76.87
35.87
41.69
62.57
CECCUP
$3.00
76.40
79 70 92.90
87.00
74.90
84.94
72.00
$4,50
71.00
76,30
18.67
41,97
60.31
1004
85.00
83.50"
84 "0.30
GNM Sabis oi Ir (003)
65.00
65.00
55 00
92.85
86.5
66.7)
76.83
70.15
35.91
64.10
73.36
13.00
38.29
$3.40
Pausihy mor sure hurtiar Kosseita N00
M11T
07:54
16 51
85.37
76.51
35.13
76.17
17.67
57.86
$7.56
58.91
98.26 87.30"
53.14
89.50
56.02
75.91
59.37
84.78
61.33
51.4
75.20
Franiie desseet than COCO
CE-CIIP
Pherrnal al 2PM
81
77.10
K0 95
94
(0
89.30
774
76,70
$6.10
14.20
74:35
IL CLIP
2eg PiNT
73.50
73,00
62.90
69:90
sy-kyt aa (oaiineionillaa MAX
69:00
63.65
56 93
16/M
(LBSSSIIC
71:50
66-40
DWCSAM
77.16
70:50
73
92.8% 87
W6.18
71.06 37
75
$5.50
301300
H4.70
16.33
24.36
61.31
DACALM
81.28
73:51
77 (l)
05 K5"
18.00
44
77.34
47. 317
56.00
83.06
18.33
69.00"
02.95
COOT-CLIP
96.01
93.34
87.05
94.21 89:89
34.91
89.31
$1.06
09.96
94.83
CLIP based methods. Tab. 3 shows that COGT -LDGT-CLIPLA largely outperforms all the other approaches trained only on COCO and it also outperforms all the methods trained on datasets larger than COCO. For instance, using the average across all the datasets, COGT-CLIP outperforms the second best result in Tab. g (DAC-LLM (Doveh et al.- 2023a)) by a remarkable 12.27 points. Note that DAC-LLM was trained on CC3M, a dataset an order of magnitude larger than COCO, and using high-quality LLM-based annotations (Sec. D. Moreover, even considering the average computed on the individual datasets, COGT-CLIP outperforms all the other methods in Tab. 3 (including those trained on datasets larger than COCO). We believe that these results show that CGM-based training strategy can better generalize by leveraging available training data, most likely because we remove spurious variable associations from the learning objective (Sec. 3). Moreover, COGT- CLIP+ achieves even better results, with an average across all benchmarks that is almost 22 points more than the second best result (DAC-LLM).
Table 4: Comparison with methods based on other VL.Ms. Similar to Tab. 3 the baseline results are either taken from the original paper or reproduced using the public code.
ARD
SrgerCrege
VL.-Checkist
Colortwap FG-OND
Avg
Model
Relation Attrilnat Ang Add Replace Soap
Attrilate Objert Relation Ang
FFT
Zem-the
XVLM Zzop et 2022
73.40
85.50
30.10
75,10"
$5.00
70.40
76.30
Tuthioa as CoCCuasdy
CE-XVLM Zhane ot 200d
73.90"
89.30"
$1.60"
74.50
$6.90
79.70"
78.90
HantNNgg pimmmmme kemyee et al 101S
$2.90
67.50
19.95
COGT XVLM
$7.64
92.30
39.97
89.17
84.37
N0.73
85.57
80049
58.74
$5.0
69007
50.12
17.10
Trainss distanes larger than
XX0T-XVIMM-
96.39
94.18
98.90
M6.9
91.18
$6.1"
$9.33
$2.83
66.30
Other VLMs. The results in Tab. are confirmed by those reported in Tab, 4 where COGT-XVLM largely outperforms the other methods, included CE-XVLM (Zhang et al. 2024), which uses our same VLM (XVLM) and the same training data (COCO). COGT-XVLM+ further improves these results and it also outperforms COGT-CLIPA This is probably because the XVLM encoder can better represent scall-scale objects than the CLIP encoder, and these objects are often referenced in the captions of these compositional benchmarks (Yuksekgonul et al. 2023).

---

Published as a conference paper at ICLR 2025
Table 5: Comparison with methods based on encoder-decoder VLM architectures pre-trained with a textual token prediction task. Lin et al. (2024) show additional results with a set using dataset- specific pross-valddttion data, which we do not report, however, make the comparison fair to other methods that do not have access to benchmark data.
SugarCrrge
Vl-Checklls
Colortiaa
FG-OVD
Midel
Relation
Aftrilutt
ArE
Add Reploce Swap Arg Altiitutc Objject Reiation Arx
11T
Axg
$9.00
BLIP Larer 20123
41.20
71.30
55.25
77.50
$4.00
70.00
77.50
Iscms
69-10
90.81
6010
65.5
65.41
16.17
80.5
40
76,
51.55*
45,50
55.20
$1:30
71.30
86.30
QP144
Opean p014
91.08
93.37
$6.51 90.55
LL.VA-155:13
$7.50
LLLVA-1 6-340
[ir
$1.25
SILSS
BLIP-ViaaK in
1011
10A6
99.10'
05.90
92.20
91.001
00.30
91.00
91.73
76.70'
92.00
10.00
K2.52
11 ml 2711
90-10
14.30
11. 90
02 71
05.00
$1.24
23.11
75.90
90.10
10
66.55
Cap TTshateer No1
$5.00
17.71
08.04
001.1
81:00
Capty
CO0T-IsswwBBLIP-
Table 6: Comparison of CLIP-based models using image classification tasks and linear probing.
Medel
CIRARIO CIPAR100 ImageNet11 (tog 3) ImageNett1 (top
CLIP 1331
NeyCLIP Velket Vonie
94.0
70.0
72.0
CECLIP
92.8
78.0
92.6
CE-CLIP (20gg o1 2014
91.8
78.1
92.7
COGT-CLIP
96.7
549
744
93.3
COGT-CL.P-
75,3
9388
Langunge based VLMs. In Tab. 5 compare each other VLMs pre-trained using decoder and a generative word prediction task. The compositional skills of these methods are gen- erally much higher than the other VLMs, which indirectly confirms that word-prediction training helps the VLM to understand the compositional nature of the human language (Sec. (2). However, direct comparison with VLMs such CLIP, XVLM Stable Diffusion (Krojer al. 2023) is
difficult, since each of these backbones has been treitrained on datasets with a huge difference in size. For instance, Cap and CapPa (Tschannen al. (023) were pre- pre-trained with a private dataset composed of 1B image/Alt-text pairs (T'chhnnenn et al., (023), which is different orders of magni- tude larger than the dataset used to pre-train XVLM (~ 16M training samples (Zeng et al., [022]) Despite that, COGT-XVLM+ (Tab. 4 outperforms both Cap and CapPa and all the other methods Tab. Similarly, COGT-InstructLIIPP significantly outperforms the zero-shot accuracy of In- structB and, jointly with COGT-XVLM+ establishes new state of the art each of these compositional datasets.
4.3 DOWNSTREAM TASKS
Doveh et al. 2003b show that most compositional methods usually deteriorate the VLM skills on non-compositional, standard tasks. We analyze this aspect using the protocol adopted by Yuksekk- gonul et al. 2023] Zhang et al. 2024)), which is based on linear probing the fine-tuned CLIP visual encoder on CIFARIC. IO, CIFARI00 and ImageNet. Since in COGT is frozen, we use jointly with our mapping network M and, specifically, the feature (ccs))- The results shown in Tab. show that COGT not only does it not deteriorate CLIP's features but it can even improve them.
CONCLUSION
In this paper we presented COGT, a compositional method based on semi- semi-parallel generative train- ing. Specifically, we exploit the priori knowledge of an off-the-shelf dependency parser to define a set of causal relations between the words sentence. These relations, collectively represented using CGM, are used to make sparser the joint probability distribution of the textual variables by removing possible spurious inter-variable associations. As a result, COGT can better exploit the training data and reduce the risk of overfitting, Using extensive experiments, we showed that COGT is much more effective than standard AR or fully-parallel generative predictions and it largely out- performs all previous compositional works, including methods trained with much larger datasets.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the PNRR project "Italian Strengthening of Esfri RI Resilience (IT- SERR)", funded by the European Union NextGenerationEUU (CUP B53C22001770006), by the EU Horizon project ELIAS (No. 101120237, and by the PNRR project "A Static and Dynamic Database for Historic Urban Contexts Accessibility", funded by the European Union NextGener- ationEU (CUP ES3D22001050000),
REFERENCES
Samyadeep Basu, Maziar Sanjabi, Daniela Massiceti, Shell Xu Hu, and Soheil Feizi. Augmenting CLIP with improved visio-linguistic reasoning. In arXiv:2307009233, 2023.
Samyadeep Basu, Shell Xu Hu, Maziar Sanjabi, Daniela Massiceti, and Soheil Feizi. Distilling knowledge from text-to-image generative models improves visio-linguistic reasoning in CLIP. In arXiv:2307.09233, 2024.
Lorenzo Bianchi, Fabio Carrara, Nicola Messina, Claudio Gennaro, and Fabrizio Falchi, The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained under- standing. In CVPR, 2024.
Federico Bolelli, Lorenzo Baraldi, Federico Pollastri, and Costantino Grana. A hierarchical quasi- recurrent approach video captioning. In 2018 IEEE International Conference on Image Pro- cessing, Applications and Systems (IPAS), 2018.
Kyle Buettner and Adriana Kovashka. Investigating the role attribute context in langual models for object recognition and detection. In WACV, 2024.
Jirayu Burapacheep. Ishan Gaur, Agam Bhatia, and Tristan Thrush. Colorswap: A color and word order dataset for multimodal evaluation. In arXiv:2442.04444 2024.
Paola Biccante-Bonila,, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gül Varol, Aude Oliva, Vicente Ordonez, Rogerio Feris, and Leonid Karlinsky. Going beyond nouns with vision & language models using synthetic data. In ICCV, 2023.
Danqi Chen and Christopher Manning. A fast and accurate dependency parser using neural net- works. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero-shot classifiers. In 0rXiv:2303.15233 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstruciBLIP: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, 2023.
Hiroyuki Deguchi, Akihiro Tamura, and Takashi Ninomiya. Dependency-basse self-attention for transformer NMT. In Recent Advances in Natural Language Processing (RANLP), 2019.
Stefan Diaconescu. generative dependency grammar. PRICAI 2002: Trends in Artificial Intel- ligence, 2002.
Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald. Why is winoground hard? investigating failures in visuolinguistic compositionality. In Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante- Boniila Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, Shimon Ullman, and Leonid Kar- linsky. Dense and aligned captions (DAC) promote compositional reasoning in VL models. In NeurIPS, 2023a.
11

---

Published as a conference paper at ICLR 2025
Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In CVPR, 2023b.
Timothy Dozat and Christopher D. Manning, Deep biaffine attention for neural dependency parsing. In arXiv:1611.0173: 2016.
Wenyu Du, Zhouhan Lin, Yikang Shen, Timothy J. O'Donnell, Yoshua Bengio, and Yue Zhang. Exploiting syntactic structure for better language modeling: A syntactic distance approach. In ACL, 2020.
Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, An- drew Gordon Wilson, and Tom Goldstein. What do vision transformers learn? A visual ex- ploration. In arXiv:2212.06627, 2022.
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, 478, 2020.
Anirudh Goyal, Alex Lamb, Jordan Hofimann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In ICLR, 2021.
Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson. Incorporating structured representations into pretrained vision language models using scene graphs. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2023.
Jonathan Ho, Ajay Jain, and Picter Abbeel, Denoising diffusion probabilistic models. In NeurIPS, 2020.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python, 2020.
Shengyuan Hou, Jushi Kai, Haotian Xue, Bingyu Zhu, Bo Yuan, Longtao Huang, Xinbing Wang, and Zhouhan Lin. -onax- localized self-attention by constituency syntactic distance. In arXiv:2210.11177 11759 2022.
Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. SugarCrepe: fixing hackable benchmarks for vision-language compositionality. In NeurlPS, 2024.
Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, and Wen Zhang. Structure- CLIP: Towards scene graph knowledge to enhance modal structured representations. In AAAI, 2024.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, 2021. URI. htttps://do.
Renodo.514R77
Kenan Jiang, Xuehai He, Ruize Xu, and Xin Eric Wang. ComCLIP: training-free compositional image and text matching. arXiv:2211.1385. 2024.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. Text encoders bottleneck compositionality in contrastive vision-language models. In Conference on Empirical Methods Natural Language Processing (EMNLP), 2023.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. BERT:
Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL, 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yer Van-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In 2344622304.02643, 2023.
12

---

Published as a conference paper at ICLR 2025
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 1 12::32-73,2011 2017.
Benno Krojer, Elinor Poole- Dayee, Vikram Voleti, Christopher Pal, and Siva Reddy, Are diffusion models vision-and- and-languae reasoners? In NeurIPS, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Selective attention improves transformer. In arXiv:2410.02703, 2024.
Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffu- sion model is secretly a zero-shot classifier. In ICCV, 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023c.
Junyan Li, Delin Chen, Yining Hong. Zhenfang Chen, Peihao Chen, Yikang Shen, and Chuang Gan. CoVLM: Composing visual entities and relationships in large language models via communica- tive decoding. In ICLR, 2024a.
Rongjie Li, Yu Wu, and Xuming He. Learning by correction: Efficient tuning task for zero-shot
generative vision-language reasoning. In CVPR, 2024b.
Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang. Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, Yuyin Zhou, and Cihang Xie. What if recaption billions of web images with Ilama-3? In X4ii:2406008478 2024c.
Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Mingyang Chen, Ze Ma, Shiyi Wang. Hao-Shu Fang, and Cewu Lu. Hake: Human activity knowledge engine. In arXiv: arXiv:1904.000 1904.06539, 2019.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models. In ICML, 2024.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In ECCV, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. In NeurIPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, and world knowledge, 2024. URL https: 1o/blog/2020 00-30- 1lava next/
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. CREPE:
Can vision- language foundation models reason compositionally? In CVPR, 2023.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus English: The Penn Treebank. Computational Linguistics, 19(2)313-330, 1993.
Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In arXiv:3304.067000 2023.
13

---

Published as a conference paper at ICLR 2025
Khalil Mrini, Franck Dernnncourt Quan Tran, Trung Bui, Walter Chang, and Ndapa Nakashole.
Rethinking self-attentton: Towards interpretability in neural parsing, In arXiv:1911.03377 2019.
Joakim Nivre. Dependency grammar and dependency parsing, 2005. URL https://api.
semanticscholas ott.://o........oooooooooooaaaaa
Tomasz Obrêbski and Filip Gralinski. Some notes on generative capacity of dependency grammar. In Workshop On Recent Advances In Dependency Grammar, 2004.
Youngtack Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, and Junmo Kim. Preserving multi- modal capabilities of pre-trained VLMs for improving vision-linguistic compositionality. In arXiv:2410.05210, 2024.
OpenAl. Gpt-4v(ision) system card, 2023. URL CorpusID:26321800
Giambattista Parascandolo, Mateo Rojas-Camulla, Niki Kilbertus, and Bernhard Schölkopf. Learning independent causal mechanisms. In ICML, 2018.
Judea Pearl. Causal inference in statistics: An overview. Stutistics Surveys, 1:96-146, 2009.
Judea Pearl and Thomas S. Verma. A theory of inferred causation. In Logic, Methodology and Philosophy of Science IX, volume 134, pp. 789-811. Elsevier, 1995.
Wujian Peng. Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize, diagnose, and optimize: Towards fine-grained vision-language understanding. In CVPR, 2024.
Ronan Perry, Julius von Kügelgen, and Bernhard Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. In NeurIPS, 2022.
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivas- tava. Learning to predict visual attributes in the wild. In CVPR, 2021.
Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. Grounded situation recognition. ECCV, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Ugur Sahin, Hang Li, Qadeer Khan, Daniel Cremers, and Volker Tresp. Enhancing multimodal compositional reasoning of visual language models with generative negative mining. In WACV, 2024.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109:612 634, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
Natalia Silveira, Timothy Dozat,
de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference Language Resources and Evaluation (LREC), 2014.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mar- cus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. CVPR, 2021.
Darshan Singh, Zeeshan Khan, and Makarand Tapaswi. FIGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos. In 2401:24010006663 2024.
14

---

Published as a conference paper at ICLR 2025
Hanman Singh, Pengchuan Zhang, Qifan Wang. Mengjiao Wang, Wenhan Xiong, Jingfei Du, and Yu Chen. Coarse-to-fine contrastive learning in image-text graph space for improved vision- language compositionality. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2023.
Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, and Furu Wei. CLIP Models are few-shot learners: Empirical studies on VQA and visual entailment. ACL, 2022.
Joshua B. Tenenbaum, Thomas L. Griffiths, and Sourabh Niyogi. Intuitive theories as grammars for causal inference. In Causal Leurning: Psychology, Philosophy, and Computation. Oxford University Press, 2007.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. CVPR, 2022.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, 2024.
Michael Tschanmen, Manoj Kumar, Andreas Peter Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners scalable vision learners too. In NeurlPS, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. Contrastive region guidance: Improving grounding in vision-language models without training. In arXii:2403.002335 2024.
Hadi Wazni, Kin Ian Lo, and Mehrnoosh Sadrzadeh. VerbCLIP:
Improving verb understanding in vision-language models with compositional structures. In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), 2024.
Monika Wysoczaúska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursuc, Tomasz Trzcinski, and Patrick Pérez. CLIP-DINOiser: Teaching CLIP few DINO tricks for open-vocabulary semantic segmentation. In arXiv:2312.12333 2024.
Zhixian Yang and Xiaojun Wan. Dependency- mixture language models. In ACL, 2022.
Tianzhu Ye, Li Dong, Yuging Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. In arXiv:2410.05255, 2024.
Nir Yellinek, Leonid Karlinsky, and Raja Giryes. 3VL: using trees teach vision & language models compositional concepts. In arXiw:2312.17345. 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Y. Zou. When and why vision-language models hehave like bags-of-words, and what to do about it? In ICLR, 2023.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In ICML, 2022.
Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding. In CVPR. 2024.
Shuai Zhang, Wang Lijie, Xinyan Xiao, and Hua Wu. Syntax-guider contrastive learning for pre- trained language model. In Findings of the Association for Computational Linguistics, 2022.
Yu Zhang, Zhenghua Li, and Min Zhang. Efficient second-order TreeCRF for neural dependency parsing. In ACL, 2020.
Tiancheng Zhao, Tiangi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision- lasoun-languag models with objects, attributes and relations. In 2t0Xi::2077.02211111 2022.
15

---

Published as a conference paper at ICLR 2025
Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Iterated learning im- proves compositionality in large vision-language models. In CVPR, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
16

---

Published as a conference paper at ICLR 2025
A CAUSAL GRAFICAL MODELS
A Causal Graphical Model (CGM) over random variables X [X,, X,) is defined (Perry et al., 2022 as MMG,, Px), where: (1) G is a directed acyelic graph with vertices X and edges X; Xj iff X; is a direct cause of X;; (2) Px is the joint distribution of X which follows the
disentangled (or causal) factorization (Schölkopf et al. 2021
Perry et al. 2022):
n P(X1 Xn) II P(X))))(X))).
(5)
where (X,),) the set of parents (direct causes) of Xj in G. The difference between a CGM and Directed Graphical Model is that the former assumes that PA(X,) are direct causes of X,. Al- though a formal proof that a statistical dependence is also a causal relation using only observational data is notoriously difficult (Pearl 2009). in this paper we assume that the linguistic dependencies NNirre 2005) extracted by a dependency parser have a causal nature because they represent strict linguistic association between the "head" and its "'deeedent".. Specifically, Dependency Grammars
(Sec. 2 can be considered as (probabilistic) generative grammars (Nivre 2005 Chen & Manning
2014 Obrèbski & Gralinski] 20044 Diaconescu, 2002). in which dependence between "head" word and its "dependent" word can be extracted using context-free generative rules. We interpret these rules causal mechanisms (Schölkopf al. (222)), which describe the causal influence of generating specific "dependent" word given the value of the 'head" word. We leave as future work the possibility of replacing the dependency grammars used in this paper with other grammars such as, for instance. the causal grammars proposed in (Tenenbaum et al. 2007). as well as the possible introduction of counterfactual reasoning (Pearl 2009) in our framework.
In Sec. 3 Eq. I is obtained using Eq. (5) the definition of conditional distribution and the as- sumption that S1, v... Sn and ZI...., Zm are independent of each other. Finally, the cardinality of (W1.... W W in Eq. (2 is, on average, 2. while, assuming balanced DT T, the cardinality of (W.,.... W. (W),,) is, on average. O(log(n)) The consequence of this that the condi- tional distributions learned at training time (Eq. (1)) and used at inference time (Eq. (4) are sparser than those learned by standard AR model (Sec.
B THE FG-OVD DATASET
In this section, we describe the compositional benchmark based on the FG-OVD dataset which we propose in this paper and which was used in Sec. The FG-OVD dataset was originally proposed to evaluate the fineggrined discriminative capabilities of open- detectors in object detection tasks. Each image usually contains multiple objects, where each object is associated with both bounding box and a corresponding caption. We use the bounding box to crop the image and the corresponding caption as the true caption. The cropped images are resized to a resolution of 224 224. Then, each object image is associated with several false captions (on average, ten), selected based on the original FG-OVD Trivial, Easy, Medium, and Hard tasks (Bianchi et al., 2024). Specifically, in the Trivial task, negative captions are randomly sampled from unrelated objects (of different images), offering basic challenge for retrieval. The Easy, Medium, and Hard tasks progressively increase the difficulty by generating negative captions starting from the true caption. For instance, in the Easy task, three attributes of the true caption are replaced with three unrelated attributes. In the Medium and the Hard task, two and one attributes are replaced, respectively. The rationale is that the less the true sentence is modified, the harder is to distinguish the false from the true captions (Bianchi et al.) 2024). Specifically, as fewer attributes are replaced, the distinction between correct and incorrect captions becomes more subtle, requiring the model to capture increasingly fine-grand grained, compositional details in the image-text pairs. Tab, II reports the number of testing images for each task, while different examples shown in App. E.
17

---

Published as a conference paper at ICLR 2025
Table Extended version of Tab.
ARO
SugarCrepp Add Rwpp Arg
VL-Chheklist
ColorSwap FG-OVD
Parser
Mask Specific AclationAttrihau Arg
Attribate Objectt Relation
ITT
Deep Biatline
78
46
78.56
63.36
Deep Hiaffine
$2.34
56.42
38.00
68.59
Desp Biafline
36.10
85.51
85.31
98.48
85.37
80.59
14
$7.60
27,97
91.51
85.69
60.33
46.46
73.39
Deep Blatfine
86.56
89:10
87.83
98.11
85.80
81.49
46
87.02
78.30
87.75
84.35
61.33
44.74
73.34
CRE
X
$6.46
$1.36
87.83
98.45
$4.15
78.65
0765587100
$7.51
77.04
90.15
84.90
14,00
45.83
63.93
CRFPur
$6.85
68.33
87.59
98.10
85.00
81.34
47
86, 66.80
78.49
88.80
84,70
42.33
42.27
69.07
CRFPar
85.53
98.74
81.84
90.30
59.33
44.08
72.65
CRFPPar
56.99
77.68
56.33
71.76
Deep Hiatfine RoBERIa
36.82
85.31
79.11
28.33
Deep Dialline RoBERTa
36.82
89.67
88.25
98.26
85.56
82.33
$4.41
78.94
89.54
84.30
45,00
45.63
70.45
Deep Blaffine KoBERTa
84.75
86. 16
85.46
98.86
84.57
80.25
87.88
83.79
78.24
90.84
84.29
58.00
46.99
72.51
Deep Hiaffine RoBERIa
87.36
90.26
88.91
98.26
87.10
83.14
09.44899.5
36.07
78.91
89.37
84.78
61.33
51.48
75.20
C ADDITIONAL EXPERIMENTS
C.I ADDITIONAL. ABLATIONS
In Tab. show extension of Tab. containing all the possible combinations of the compo- nents analyzed in Tab. 2 For instance, this table shows the importance of using two visual feature layers in some of the datasets, Indeed, when both layers are used, the model can leverage not only high-level, abstract visual features but also more detailed, lower-level information. This is partic- ularly important in tasks which require attention object details, where the additional layer helps to capture more nuanced representation of the input. For instance, in datasets like ColorSwap, this deeper feature extraction leads to drastic improvements (observed across all parsers), which is probably due to the better representation of the color/texture appearance in the lower level features. On the other hand, the use -pec--s tokens also plays significant role. Although the im- provement magnitude varies depending on the dataset and the task, the overall trend indicates that using mask-specific tokens contributes positively to the accuracy.
Comparing the results of COGT-CLIP with COGT-CLIP+ and COGT-XVLM with COGT-XVLM+ (Tab.3aand Tab.E.. the mean improvement of the larger-training versions with respect to the COCO- only trained models is about 9 points averaged across all the datasets. This shows that COGT can benefit from larger training and, indirectly, that the noisier captions in CC3M can be effectively parsed by our parser. Finally, a recent trend in VLM fine-tuning and/or pre-trate pre-training adopts LLMs to create or re-write the textual annotations (Li et al. 2024cc, Doveh et al. (003a), which can in principle help the dependency parser with very noisy captions. We leave this as future work.
C.2 WINOGROUND AND THE MULTIMODAL VISUAL PATTERN (MMVP) BENCHMARKS
In Tab. we show additional results obtained using the Winoground TTrush et al. 2022) and the MMVP (Tong et al. 2024) benchmarks. MMVP is relatively small dataset, with 9 tasks but only 15 samples per task (135 total samples), thus performance measured on this benchmark has a limited statistical significance. Nevertheless, also in this dataset COGT- DOGT-InstruetBLIPP largely outperforms the second best tested model (DAC-LLM) by 4.45 points. On the other hand, on Winoground, despite improvement of +7.5 points with respect InstructBI.IP zero- shot, it is largely outperformed by InstructBLIPP VIA Score (Lin er al.) 2025). InstructBLIP--OO could eventually be merged with COGT, but we leave this as a future work.
C.3 FG-OVD
Tab. 10 reports the FG-OVD task- task-specific accuracy of the methods compared Tab. 3 In the Zero-shot category, CLIP (Radford et al. (021) shows a solid performance on all but the Hard task (21.35 points accuracy). The same applies to those methods based on CLIP, such as NegCLIP Yuukekkonuul et al. 2023), GNM (Sahin et al. 2024), Plausible Adj. Neg (Buettner & Kovashka 2024). and CE-CLIP (Zhang et al. 2024). which show a noticeable drop in performance as the difficulty increases. Among these models, Plausible Adj. Neg (Buettner & Kovashka 2024 stands out with a relatively strong performance, especially on the Medium (43.13) and Easy (45.88) tasks. On the other hand, COGT-CLIP and COGT-XVLM demonstrate significant improvement in per-
18

---

Published as a conference paper at ICLR 2025
Table 8: Model comparison on MMVP and Winoground.
MMVP
Winoground
Group Score Group Score
CLIP [Radford al. 2021
15.55
8.00
NegCLTP Yuksekgonul efal.
2023
26.67
8.00
CE-CLIP Zhang et al 2024
19.25
5.00
DAC-SAM Dovel
20723a
23,70
6.25
DAC-LLM Duveh
202.30
14"
3.25
Plausible Adj. Neg [Buctiner Kovashka,
2024
21.48
6.75
InstructBLIP (Dai et ATA
14,81
4,75
InstructBL.IP- VQA Score [Li at
2025
28,50
COGT-CLIP
20.74
6.65
COGT-CLIP+
26.67
9.75
COGT-XVLM
17.77
7.25
COGT-XVLM+
28. 88
10.50*
InsttuiittuBBBLIIPP
7.75
KOGGT+IsttcctBL.PP
32.59
12.25
formance across all difficulty levels. particularly on the Hard task, where COGT-CLIP achieves score 33.82, which is greater than all other methods, including models trained on larger datasets, such as DAC-SAM and DAC-LLM (Doveh et al., 2023a). These results highlight the strength of COGT when dealing with fine- fine-grained compositional tasks. Finally, similarly to the results shown in Sec. 4.2, COGT-CLIP+ COGT-XVI.M and COGT-InstretBI.IP-. further increase the advantage over other approaches.
C.4 COMPUTATIONAL EFFICIENCY
Tab. 9 shows the training and inference times for COGT and CLIP-based models. COGT-CLIP and COGT-CLIP4 require respectively 8 and 72 hours to train on a single RTX A5000 GPU with a batch size of 128 using the datasets of Sec. 4.2 For comparison, we use DAC-SAM and DAC-LLM (Dvehh et al. 2023a), which are the only models we know with publicly available training times that can be directly compared COGT-CLIP+: as they are trained on a similar dataset of ~3.3M samples. In particular, both DAC-SAM and DAC-LLM complete their training in 12 hours on six V100 GPUs with batch size 32. However, this training time does not include the computationally intensive dense annotation generation pipeline (Sec. 2). which involves BLIP2 and SAM or GPT- Neo-2.7B (for DAC-SAM and DAC-LLM, rispectively). Similarly, the training times for COGT- CLIP and COGT-CLIP+ do not include the DT generations, which however involves relatively quick preprocessing step (one DT per caption), taking approximately minutes for COCO and hours for the combined CC3M, COCO, and Visual Genome datasets. Moreover, computational costs of COGT and CLIP terms of memory usage and inference time. COGT- CLIP. COGT-CLIP+, and CLIP require 0.73 GB, 0.88 GB, and 1.16 GB of memory, respectively. where the difference with respect to CLIP is mainly due to the fact that COGT does not use the CLIP textual encoder. Finally, for both COGT-CLIP and COGT-CLIP+, the inference times reported in Tab. iinclude an additional 0.01 seconds required for generating the DT of the testing caption using the Deep Biaffine + RoBERTa parser, and COGT-CLIP+ slower than COGT-CLIP because of its larger (four blocks) decoder. All times are computed using an RTX A5000 GPU with batch size of 32.
evaluate the
D IMPLEMENTATION DETAILS
D.I ARCHITECTURES
In COGT-CLIP and in COGT-XVLM we use VIT-B/32 CLIP (Ilharco et al., 2021) and the Swin- Transformer XVLM (Zeng et al. 2022) as the visual encoder, respectively. both the last and the penultimate layer features of the encoder (Sec. 3.1 In COGT-InstrccBLLPP we use the output of the InstructBLIP Q-Former (Dai et al. InstructBLIP needs textual description of the task (called "instruction" (Dai et al. (023))), COGT- InstructBLIP is trained using the prompts suggested in (Dai et al. 2023) for captioning tasks. At inference time, both the zero-shot results of InstructBI.IP and those of COGT-InstruttBI.IP
both cases, we use
2023 as the visual encoder. Since
19

---

Published as a conference paper at ICLR 2025
Table 9: training and inference times comparison.
Training
Model
Training Time (hrs)
Batch Size
GPU Setup
COGTECLIP 8
128
RTX A5000 (Single GPU)
COGT-CLIP+
72
128
RTX A5000 (Single GPU)
DAC-SAM
12
32
V100 (Six GPUs)
DAC-LLM
12
32
V100 (Six GPUs)
Inference
Model
Inference Time
Batch Size
GPU Sctup
COGT-CLIP
0.07
32
RIX A5000 (Single GPU)
COGT-CLIP+
A5000 (Single
CLIP
0.06
32
RTX A5000 (Single GPU)
Table 10: FG-OVD: task specific results.
Model
Hard
Medium
Easy
Trivial
Zero-shot
CLIP Radford et al. 2021
21.35
48.75 33,25
51.73
47.33
InstrucTBLIP (Piin (XLL Dai et 2023
22.23
31.87
19.85
26.80
Training COCO only
NogCLIP Yuksekeonuul
2023
18.39
36.96
41.95
69.49
41.69
(Sahin ar 2014
64.48
Plausibis Adi Noa Buetneer Kovashka 2024
21.35
43.13
45. 88
69,59
44.98
CE-CLIP (hhnn et 2024
21.86
Fully Paraliel
25.22
47.41
54.04
40.72
41.84
Mixed
Sequential-AS ARR
30.18
$4.01
$7.04
41.73
46.24
COGT-CLIP
COGT-XVLM
32.69
$8.52
60.05
49:22
50.12
COGT-InstrnetBL.
33,90
$9.91
61.12
50.15
51.26
Training on dotavets larver than COCO
DAC-SAM (Doveh 2023a
DAC-LLM (Doveh at 20238
25.29
$2,36
56.89
63.89
49.60
COGT: CLIP+
$5.40
$1.50
85.29
57.65
69.96
COGT-XVLM+
58.78
83,86
87.45
66.82
74.22
JoGTT-InstrccBBLPPP
53,59
81.37"
84.52
63.36
70.72
obtained using the prompt "Write a description for the photo.". The above considerations apply also to the COGT-X+ models.
Independently of the VLM encoder, the features Z are obtained using a mapping network M on top of the corresponding frozen visual encoder (Sec. 3.Ip Our decoder consists of 3 blocks (respectively, blocks in case of COGT-X+), each composed of a multi-head Dependency Guided Attention (Sec. 3.1) and cross-attention layer. Each attention layer is composed of attention heads, with embedding size equal to 512, while we use 12 heads and embedding size equal to 768 in the COGT- X+ models. We apply a dropout rate of 0.1 the residual connections, the attention weights, and the embeddings.
The differences in the number of trainable parameters among the different baselines in Tab. only due to the size of the embedding dictionary. Fully-Paralle has an embedding dictionary con- sisting of only one MSK token, resulting in a total of 13 million trainable parameters, of which only 512 are dedicated to represent the MSK token. The total number of trainable parameters for Mixed, Sequential-AK and COGT approximately 64 million (M included). Among these, Sequential-AR uses embedding dictionary that matches the size of the CLIP VIT-B/32 textual encoder, while Mixed introduces an additional MSK token for parallel processing. In contrast, COGT employs 45 extra MSK tokens, each representing specific dependency relation extracted by the parser, resulting in a negligible increase in the total parameter count of only 0.04%. All the decoders in Tab. [ are composed of three blocks which differ only in their attention masks, and they all alternate textual- token embedding attention layer with a cross cross-attentinn attention layer with the features Z (extracted from the last and the penultimate layer of the CLIP visual encoder, Sec. 3.1
20

---

Published as a conference paper at ICLR 2025
Tc
NSUBJ
DOBJ
fc
NSUBJ
DOBI
head
AMOD
DE
AMDD
AMOO
DET
AMGD
DET
AMOD
AMOD
brown
smail
yellow
brrow
yellow
COMP
all
Figure 3: Dependency tree modifications due to the sub-word tokenization. On the left, the original dependency tree T represents the sentence "A brown bird with small yellow head", and the output of word-level dependency parser, where "small" is single node. On the right, the modified tree. T, which accounts for sub-word tokenization by splitting the word 'small' into two nodes: "sm" and "all". A new syntactic relation, called comp, introduced between these sub- sub-word nodes.
D.2 TOKENIZATION
The output of dependency parser is tree which each node corresponds to a caption word. In contrast, the COGT decoder uses standard sub-word tokenization, splitting words into smaller tokens which brings to a smaller embedding dictionary. This discrepancy leads to cases where a word of the dependency tree is split into multiple tokens by the COGT decoder's tokenizer. We modify the dependency tree to handle this mismatch: a word wy is split into sub-tokens wi, and node for Wi, and new edge between Wia and Wj, associated with dedicated relation called comp (Fig. 3). Note that wy has been removed. As result, we have that: PA(W,,) PA(W,) and (ATW;;) (A
Wja by the COGT decoder's tokenizer, then create
D.3 EXPERIMENTS
We use the architectures described above across all the experiments (e.g., the same number of blocks, learnable parameters, etc.). Specifically, we freeze the weights of the visual encoder and we train only our textual decoder. We train in mixed precision (FP16) with batch size set to 128 on GPU RTX A5000 with 24GB of VRAM for epochs. Following Yuksekgonul et al. (2023), we select the best checkpoint using the validation set provided in (Yuksekgonul et al. (023). In all the datasets and in all the experiments, we use the Adam optimizer with an initial learning rate set to 5 x 10-4. Finally, we apply Cosine Annealing Learning Rate Scheduler with 50 warmup steps.
E DATASETS AND TASKS
We provide details about the FG-OVD dataset and App. B and we briefly summarize here the main characteristics of the other benchmarks. Tab. shows the main statistics of each dataset and App. E. we show a few images illustrating the benchmark typically tasks.
ARO YYukskkgonul et al. 2023) is a VLM benchmark for compositional reasoning and word-order sensitivity. It composed of two main tasks: Visual Genome Relation and Visual Genome Attri- bution. In the Visual Genome Relation task, the goal is to evaluate the models' ability to correctly interpret the relationships between objects. On the other hand, Visual Genome Attribution focuses on evaluating the ability to associate the correct attribute with the correct object. As mentioned in Sec. we do not use COCO Order and Flickr Order because different authors recently found that grammatical errors in the generated captions of these datasets lead to tasks which be solved
purely relying on an LLM language prior (Zhang et al. 2024 Tschannen et al., 2023 2024
Lin et al.
21

---

Published as a conference paper at ICLR 2025
Table 11: Main statisties of the benchmarks.
Dataset
Split
Number of testing samples
Avg. caption length (n. of words)
ARO
Relation
23,937
8.1
ARO
Attribution
28,748
7.1
SugarCrepe
Add
2,754
12.9
SugarCrepe
Replace
3,846
11.5
SugarCrepe
Swap
911
13.5
VL-Checklist
Attribute
118,253
2,4
VL-Checklist
Object
389,357
3.2
VL-Chceklist
Relation
75,641
3.5
ColorSwap
300
8.8
FG-OVD
Hard
3,545
10.4
FG-OVD
Medium
2,968
11.3
FG-OVD
Easy
1,299
16.2
FG-OVD
Trivial
3,545
9.7
SugarCrepe (Hsich et al. 2024) a dataset developed to evaluate how well VLMs can understand and process complex compositional tasks by presenting them with carefully designed hard negative examples. Drawing inspiration from datasets like CREPE (Ma et al.] 2023). VLL-CheckList (Zhao al., [022), and ARO (Yuksekgonul al., 2023), SugarCrepe focuses
atomic concepts and their compositions, such as objects, attributes, and relations. The dataset is split into three tasks: "Replace", "Swap" and "Add". In "Replace", an atomic concept in the original text is replaced with new, mismatched concept. A replacement can involve an object, an attribute or a relation. In "Swap"" the negative caption created by exchanging two atomic concepts of the same category without introducing new elements. In "Add", new concept is added to the original caption, leading misalignment with the visual scene content.
VI.-Checklist (Zhao et al. 2022) is benchmark composed of four datasets: Visual Genome Kr- ishna et al. 2017). SWiG (Pratt et al. 2020), VAW (Pham et al. 202I), and HAKE (Li al. 2019). Each image is associated with two descriptions: true and false caption. The true descriptions originate from the original image-text pairs in the datasets, while false ones are generated by modifying single word in the true description, altering its overall meaning. These false descrip- tions are organized into three main types: objects, attributes, and relations.
ColorSwap (Burapacheep al. 2024) evaluates the ability of multimodal models to accurately associate objects with their corresponding colors. Each sample contains a caption-image pair along with "color-swapped" pair. The two captions in each sample use the same text, but the relation between colors and objects are inverted.
E. QUALITATIVE RESULTS
we show some qualitative results which we compare COGT-CLIP with the second best approach in Tab. (DACLLMM) We use these figures also to illustrate the tasks of the different benchmarks, with special emphasis on FG-OVD, proposed in this paper.
In Fig. Fig. 12
22

---

Published as a conference paper at ICLR 2025
dog pants.
X
The train the
The pants to the left of the
CoGT
DAC LUM
The man is behind the building
The aki is to the right of the wall. X
X
The building is behind the man.
The wall is to the right of the sió.
CoGT
DAC-LLM
X
CoGT
DAC-LIM
Figure 4: Qualitative results on sample images of the ARO Relation test split. We compare our approach with DAC-LLM which is the second best approach according to the results reported in Tab.
Three todel laying under the covers.
sitting on grassy next to other mnucss.
X
Three leddy bears laying canopy bed under the covers.
X rusly red truck sitting on grassy fiekd next other trucks.
COGT
DAC
X
cDGl
DMC-LUM
Someone holding out their hand with a pair of scissors it. holding their hand with a rusty pair of scissors on it.
group of girafles drink water the wilderness
group girafies drink crystal clear water in the wilderness.
COGT
DAC-LUM
X
CoGT
Figure 5: Qualitative results on sample images of SugarCrepe,
Standing foothbruah.
Looking at
cow.
x Crouching toothbrush.
Looking down
coGt
DAC-LUM
CDGT
DAC-LIM
X
Standing gireffe.
Walking man.
Running giraffe.
Cooking man.
CoGT
DAC-LIM
COGT
DAC-LLM
X
Figure 6: Qualitative results on sample images of VL-CheckList..
rustic, brown cabin surrounee by a peaceful, green forest
Watermelons with red pulp nixed with green applle. Watermelons with green pulp mixed with red apples.
X A peaceful, green cabin surounded
by a rustic, brown forest
CoGT
DAC-IIM
A gray dinossur is riding blue bicycle.
A brown cat is sltting in front of while
X A blue dinosauris gray bicycle.
X white cat sitting in front of brown chair.
DAC-LIM
COGT
DAC LLM
Figure 7: Qualitative results on sample images of ColorSwap.
23

---

Published as a conference paper at ICLR 2025
while bowl.
black plastic battle with white label.
dark pink tissue paper.
body light green
X kattle with a meta lic body green and dark blue color.
DAC-LUM
X
handbag
leather.
A cooking pan with grey wood rim.
white A light brown bench made of plastic.
ce plate
COGT
green (automobile).
A brown astic broom with red and green shaft.
black laptop compuler with text pattern on it.
plastic, screen
light blue, green, purple glass
A blue towel with body made velvet.
A mouse made dark green plastic.
A blue wood pencil with striped while, red and brown body.
A wallet with dark brown and white color, made of leather.
X A dark yellow napkin made of paper.
X A pink plastic broom with red and dark grey shaft.
X A light yellow soap dispenser made of plastic.
black watch made of fabric
black strap.
COCT
X
X A bowl with dark plastic rim.
available white
Figure 8:
Qualitative results sample images of the FG-OVD Trivial task. The image the
bottom shows a failure of COGT.
white bowl.
car with dark red exterior. grey clock case made of wool, with light grey hands.
microwave with dark grey and white plastic exterior.
ceramic light yellow mug with light blue text.
X
fabric fan with black blade and grey canopy.
black leather trash can body.
grey spoon
orange
COGT
pink and dark yellow shoe with light blue outsole.
chair
arme.
X A while newspaper with dark purple text written on paper.
Figure 9: Qualitative results on sample images of the FG-OVD Trivial task where COGT is success- ful while DAC-LLM fails.
ceramic with brown mouth brown neck light grey and red body.
X ceramic vase with a purple mouth, while neck, a dark blue and red body. light grey red bodl. mouth, light yellow rattan vase with a brown mouth, light pink neck. light yellow and red body.
blue mouth. light blue wood vase with a dark pink mouth, a dark grey neck. a light grey and red body.
light grey
body.
DAC-LIM
with brown mouth, black
yellow red body.
A velvet vase with light orange mouth, brown neck, a dark brown and red body. dark yellow neck, light grey yellow body. wool vase with a brown mouth, black neck, a dark yellow and red body.
COGT
crochet vase with brown mouth, black neck, a light gray and dark brown body.
Figure 10: Qualitative results sample images of the FG-OVD Easy task.
A transparent glass bowl with a striped paffem on the rün.
X translucent glass bowl with a dolled patlem the rim. opaque glass bowl with checkered pattem on the rim. translucent glass bowl with a perforated patlem on the rim.
transiucent glass
with checkered pattern the rim. opaque glass bowl with a woven pollem on the rim. transiucent leather bowl
DAC-LIM
X
striped pattem transparent wool bowl with checkered pattern on the rim. transiucent
rim.
pattem the rim. transparent metal bowl with checkered pattem on the rim.
with
striped pattem the
Figure 11: Qualitative results on sample images of the FG-OVD Medium task.
24

---

Published as a conference paper at ICLR 2025
laptop computer grey featuring logo metal.
lsptop computer with red metal back, Teaturing white made metal. laplap computer with grey metal back, featuring a white logo made glass.
leptop computer with dark
metal back, Teaturing while made metal.
laplap computer with grey metal back, featuring a white logo made plastic.
laptap computer with grey metal back, featuring white logo made crochet
laplop computer with grey crochet back, festuring a white logo made metal.
laptop computer with pink metal back, featuring white made metal. leptop computer with grey leather back, feeturing white logo made of metal.
COST
laptop computer with grey metal back, featuring dark purple logo made metal.
X laptop computer with grey metal back. featuring white logo made of stone.
Figure 12: Qualitative results on sample images of the FG-OVD Hard task.
25

---

Published as a conference paper at ICLR 2025
F SYNTACTIC CATEGORIES
We report in Tab. 12 the 45 syntactic categories defined V (Sec. B
(Silveira et al. 2014) and which form our
Table 12: List of the syntactic categories defined in Silveira et al. 2014).
Syntactic Categories
acomp advel
advmod
amod
appos
aux
auxpass
ccomp
conj
cop
csubj
csubjpass
dep
det
discourse
dobj
expl
goeswith
iobj
mark
mwe
neg
nn
mpadvmod
nsubj
nsubjpass
num
number
parataxis
pcomp
pobj
poss
possessive
preconj
predet
prep
prt
punct
quantmod
remod
roet
tmod
xcomp
26


Reviews:
[{"review_id": "QeOGeytIF5", "reviewer": "Reviewer_tN1p", "summary": "This paper models the dependency relations among textual and visual tokens using a causal graphical model (CGM) built using a dependency parser. The proposed dependency-constraint decoding follows the CGM structure which encourages a focus on the main causal dependencies. On several vision-language compositional benchmarks, the proposed method archives significant performance gains over baselines.", "questions": "1. How is the dependency corpus by Silveira et al. representative in web-scale data? Do the authors think future work should extend it or it is enough for future applications?\n2. How does the dependency parser handle cross-boundary entities? For example, there might exist data that refers to a previous entity that cannot be interpreted without the full context.\n3. How could the dependency-constrained decoding extend to generation tasks (i.e., evaluated on non-retrieval task format)? Would the constrained decoding harm the language modeling quality?\n4. What could be potential ways to address the challenge that CLIP backbone cannot detect out-of-focus objects for compositional reasoning?\n5. When scaling up, how do the author handle falsely annotated noisy data in the web-scale data pool?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The author propose a causal graphical model with dependency-constraint decoding method to incorporate compositional knowledge into VLM. Using relationship representations could effectively model the compositionally of the data and resolve the long-existing issue for sequential modeling VLM that the model treats data as bag-of-words.\n2. Experiments on several vision-language compositional benchmarks show that the proposed method is effective in improving the compositional performance. Specifically, the proposed method outperforms all other baselines trained on COCO only by a large margin.", "weaknesses": "1. The training objective might not be robust to mislabeled data, thus posing challenging to strict data cleaning when future work consider scaling up.\n2. The author did not compare the efficiency in terms of training nor inference of the proposed method with other baselines.\n3. The evaluation focuses on retrieval, where the model is tasked to find the correct caption for a given image and vice versa. It remains unclear how to improve the compositional generation ability, which is of a greater need in real-world application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cn78dNj4qG", "reviewer": "Reviewer_nWro", "summary": "This paper introduces a method to improve vision-language compositional understanding by leveraging a dependency parser to construct a Dependency Tree (DT) for each sentence, which is then converted into a Causal Graphical Model (CGM). This CGM guides caption generation in a semi-parallel autoregressive approach, where each word is predicted based on its syntactic dependencies rather than a purely sequential order. They use CLIP as the vision encoder to extract visual features and train a decoder from scratch using the COCO dataset (additional experiments using X-VLM as the vision encoder are shown). This model is then used to validate their finetuning method -- Causally-Ordered Generative Training (COGT) -- on ARO, SugarCrepe, VL-Checklist, ColorSwap, and adapt FG-OVD to evaluate object attributes.", "questions": "- Is it possible to use better performant backbones (e.g., LLaVA, InstructBLIP, BLIP2) and finetune the model rather than train an independent module?\n\n- What is the difference between Deep Biaffine (alone) vs Deep Biaffine + RoBERTa? Is this related to the discrepancy of the dependency parser between word-level vs sub-word tokenization to build the dependency tree?\n\n- The failure case shown in Figure 5 (Supp Material) opens an interesting question: for long sentences, which are usually not present in the evaluated benchmarks, there are multiple layers of dependencies (e.g., adjectives modifying nouns, prepositional phrases providing additional information, relative clauses, etc). In these cases, each word might have multiple ancestors in the syntactic tree (e.g., \"light blue\" depends on \"screen,\" which depends on \"laptop\"), leading to a more complex factorization; it can be indeed difficult to parse or resolve correctly. More importantly, it may become even more apparent that these relationships are not causal: for example, \"a black laptop computer made of plastic\" does not cause \"a screen that has a light blue glass front.\" -- these are separate descriptions that happen to be part of the same sentence. May be this the reason why the model fails or struggles in this case?\n\n- In complex sentences, for example, \"A black laptop computer made of plastic, with a screen that has a light blue, green, purple and white glass front, with a text pattern on it.\"; \"light blue\" depends on both \"screen\" and indirectly on \"laptop.\" -- given sentences with multiple hierarchical dependencies, how are these factors balanced? It seems that with more dependencies (e.g., through conjunctions or relative clauses), maintaining accurate conditional independence assumptions becomes harder.\n\n______\n\n- Friendly suggestion (not to pay particular attention): there is interesting related work [6, 7] that relies on graph-based representations to model relationships between entities in sentences and scenes (both language and visual inputs) -- using scene graphs as a structured representation of an image seems orthogonal to the proposed approach in this paper.\n\n- I'm also genuinely curious about the performance of COGT on Winoground -- given the ambiguity of some sentences, it is not particularly clear how finetuning using the structure yielded via STs would impact when evaluating this benchmark.\n\n[6] Singh, Harman et al. “Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality.” Conference on Empirical Methods in Natural Language Processing (2023).\n\n[7] Herzig, Roei et al. “Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs.” ArXiv abs/2305.06343 (2023): n. pag.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is easy to read, with extensive empirical results and useful qualitative examples.\n\n- Motivation: using dependency/syntactic trees to capture the grammatical structure of sentences is well motivated -- this hierarchical \n structure shows how words depend on one another, which also may help to capture the compositional meaning of sentences. In vision-language models, capturing compositional meaning impacts on how different parts of a sentence relate to one another.\n\n- This paper shows very strong results on 4 well known compositional benchmarks (ARO, SugarCrepe, VL-Checklist, ColorSwap), along with the annotated bounding boxes from the FG-OVD benchmark, and their positive and set of negative captions. \n\n- The proposed method significantly outperforms prior works that explore captioning as the learning objective for image-text pair matching (Cap / CapPa) -- using a CLIP as the feature extractor for the visual features.\n\n- The proposed approach to structuring predictions around syntactic dependencies allows the model to focus on learning relationships that are grammatically and semantically meaningful, reducing noise from irrelevant co-occurrences.\n\n- Ablation results show results when varying different components (syntactic tree parser, masking by syntactic types, and using only the last layer or both the last and penultimate layer of CLIP for extracting the visual features)", "weaknesses": "- Generalization: One downside of this work is the overreliance on CLIP visual encoder as the feature extractor -- recent work has shown stronger performance using stronger baselines (e.g., average on SugarCrepe Swap -> CLIP acc. 64.25% vs. LLaVA acc. 81.25% -- using Contrastive Region Guidance [2] LLaVA acc. 90.75% -- results reported using CLIP+CGMs acc. is 83.14%). While X-VLM focuses on fine-grained alignment via cross-attention layers (for interactions between image and text features at multiple levels), it also falls short against high-performant VLMs such as LLaVA and InstructBLIP. For example, results using COGT compared to GPT-4V results on SugarCrepe are similar  -- it is not clear if using these backbones as feature extractors would yield better results: \n\n| Model         | Add   | Replace | Swap  | Avg   |\n|---------------|-------|---------|-------|-------|\n| COGT + CLIP   | 98.26 | 87.10   | 83.14 | 89.50 |\n| COGT + X-VLM  | **98.65** | 89.17   | 84.37 | **90.73** |\n| GPT-4V       | 91.68 | **93.37**   | **86.61** | *90.55* |\n\n- Missing prior work: in [3], authors show that using BLIP, arguably a very similar to CLIP backbone, and computing the match score of generating a particular text string given an image, yield SOTA results: \n\n| Model            | ARO   |       | VL Checklist        |            |       | SugarCrepe                  |       |       |\n|------------------|-------|-------|---------------------|-----------------------------------|-------|-------|-------|-------|\n|                  | Rel.  | Attr. | Object             | Attribute                         | Rel.  | Replace | Swap  | Add   |\n| COGT + CLIP      | 87.56 | 90.26 | 78.91              | **86.07**                             | 89.37 | 87.10  | 83.14 | 98.26 |\n| COGT + X-VLM     | 87.64 | 92.30 | 80.49              | 85.87                             | 88.74 | 89.17  | 84.37 | **98.65** |\n| VisualGPTScore [3]   | **89.1**  | **95.4**  | **94.4**               | 82.1                              | **92.8**  | **95.1**   | **92.4**  | 97.4  |\n\n- Missing references: a) 3VL [1] is highly related to the proposed work: their method proposes to expand \"the text of an arbitrary image-text pair into a hierarchical tree structure using language analysis tools, 3VL allows inducing this structure into the visual representation learned by the model, enhancing its interpretability and compositional reasoning\" -- b) SynCLM [4], a syntax guided contrastive learning method, based on constituent and dependency structures of syntax trees, we propose phrase guided and tree-guided contrastive objectives for pre-training.\n\n- Syntactic Trees and Causal Graphical Models analogy: while both STs and CGMs use directed acyclic graphs (DAGs) to represent relationships between elements, STs show linguistic hierarchy, while CGMs can show hierarchies of causal influence -- STs represent grammatical structure, while CGMs represent causal relationships; these are completely different domains with no inherent connection.\n[5] identify these problems and propose to use causal grammars (as opposed to linguistic grammars, as suggested in this work). In the context of this work, STs represent grammatical relationships, but it's hard to regard them as causal ones. \nAs an example, syntactic dependencies represent grammatical relationships between words in a sentence. For example, in the sentence \"a brown bird has a small yellow head,\" the dependency tree shows that \"bird\" is the head of \"brown,\" and \"head\" is the head of \"small yellow.\" -- these dependencies are linguistic in nature, indicating how words relate to each other grammatically -- here, \"bird\" being the head of \"brown\" doesn't imply that \"bird\" causes \"brown\" to exist; it simply means that \"brown\" modifies \"bird\" grammatically.\n\n- Disentangled Factorization and Syntactic Trees: the relationships in syntactic trees do not imply that one word causes another word to appear or behave in a certain way: they describe how words are structured grammatically. Treating syntactic dependencies as causal dependencies is conceptually difficult to correlate because the nature of the relationships is fundamentally different -- the joint distribution of words in a sentence (Eq 4) reduces the complexity of modeling by focusing only on the dependencies indicated by the syntactic tree. However, the factorization is based on grammatical dependencies, not causal dependencies.\n\n- An important aspect of causal reasoning is the ability to perform interventions (using Pearl's do-calculus) or reason about counterfactuals (what would happen if something were different). In COGT, there is no notion of intervening in one word to see how it would change another word's appearance or meaning -- instead, the model predicts words based on their syntactic dependencies without considering how one word might causally influence another through an intervention.\n\n\n[1]  Yellinek, Nir, Leonid Karlinsky, and Raja Giryes. \"3VL: using Trees to teach Vision & Language models compositional concepts.\" arXiv preprint arXiv:2312.17345 (2023).\n\n[2] Wan, David, et al. \"Contrastive region guidance: Improving grounding in vision-language models without training.\" arXiv preprint arXiv:2403.02325 (2024).\n\n[3] Lin, Zhiqiu et al. “Revisiting the Role of Language Priors in Vision-Language Models.” International Conference on Machine Learning (2023).\n\n[4] Zhang, Shuai et al. “Syntax-guided Contrastive Learning for Pre-trained Language Model.” Findings (2022).\n\n[5] Tenenbaum, J. B., Griffiths, T. L., & Niyogi, S. (2007). Intuitive theories as grammars for causal inference. In A. Gopnik & L. Schulz (Eds.), Causal learning: Psychology, philosophy, and computation (pp. 301–322). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195176803.003.0020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fjAjcm1rKO", "reviewer": "Reviewer_AYGs", "summary": "The paper proposes to train a causal graphical model decoder on top of the 2-top-layer features extracted from a frozen pre-trained image encoder (clip / x-vlm). The decoder is used to \"early-fusion\" match between a textual caption parsed by a dependency parser and extracted image features. Trained on relatively small COCO train data following the NegCLIP paper protocol, the above small (38M params) decoder is able to achieve impressive image-text matching results on multiple compositional reasoning benchmarks, extending SOTA performance on those. Comparison is made to a variety of recent encoder-only methods (late fusion) showing significant gains. However, no comparison is made to modern LLM-decoder based LMMs that are known to also perform quite highly on the tested benchmarks (e.g. on SugarCREPE). That said, 38M params decoder is not apples-to-apples with a strong LLM :-)", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- small training, small decoder, great results\n- seems to be useful for CR checking, although not tested, can potentially aid in slow-fast type approaches to retrieval tasks (akin how original BLIP v1 did it) - might be a good idea to test, would make the contribution stronger I think\n- contains adequate ablation", "weaknesses": "- no comparison to decoder LMMs (all the LLM-alignment llava-style methods) - those afaik perform strongly on those old CR benchmarks, especially with CapPa style inference, but multiple choice generation (answer with single letter, etc) also works well\n- could test with CLIP-blind benchmark like eyes-wide-shut\n- no applications beyond CR - eg to improve retrieval using eg slow-fast approach, where CLIP retrieves first and the proposed method (being early fusion it is heavier) filters\n- It is not clear what 45 \"standard syntactic categories\" (line 288) are used in the LUT, I would expect a clear explanation on that in the paper and not sending to a reference. E.g. I would like to understand what the method is blind to? eg if some words get grouped to the same bucket etc...\n- the write-up of the method is a bit convoluted and hard to follow, some details are omitted sending to prior work, ideally, the paper should be improved to make it more self-contained\n- comparison to \"methods trained on datasets larger then COCO\" is not 100% fair both ways, as those datasets might also be much lower quality then COCO (eg synthetic or non-curated from the internet etc)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)