Cola: A Benchmark for Compositional Text-to-image Retrieval
Authors: Arijit Ray
Filip Radenovic
Abhimanyu Dubey
Bryan A. Plummer
Ranjay Krishna
Kate Saenko
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: compositionality
vision-language
TL;DR: A benchmark to test models' capability to compose the correct attributes to the correct objects in the presence of hard distractors, and a light weight adaption to improve this capability in pre-trained models.

Abstract: Compositional reasoning is a hallmark of human visual intelligence. Yet, despite the size of large vision-language models, they struggle to represent simple compositions by combining objects with their attributes. To measure this lack of compositional capability, we design Cola, a text-to-image retrieval benchmark to Compose Objects Localized with Attributes. To solve Cola, a model must retrieve images with the correct configuration of attributes and objects and avoid choosing a distractor image with the same objects and attributes but in the wrong configuration. Cola contains about 1.2k composed queries of 168 objects and 197 attributes on around 30K images. Our human evaluation finds that Cola is 83.33% accurate, similar to contemporary compositionality benchmarks. Using Cola as a testbed, we explore empirical modeling designs to adapt pre-trained vision-language models to reason compositionally. We explore 6 adaptation strategies on 2 seminal vision-language models, using compositionality-centric test benchmarks - Cola and CREPE. We find the optimal adaptation strategy is to train a multi-modal attention layer that jointly attends over the frozen pre-trained image and language features. Surprisingly, training multimodal layers on CLIP performs better than tuning a larger FLAVA model with already pre-trained multimodal layers. Furthermore, our adaptation strategy improves CLIP and FLAVA to comparable levels, suggesting that training multimodal layers using contrastive attribute-object data is key, as opposed to using them pre-trained. Lastly, we show that Cola is harder than a closely related contemporary benchmark, CREPE, since simpler fine-tuning strategies without multimodal layers suffice on CREPE, but not on Cola. However, we still see a significant gap between our best adaptation and human accuracy, suggesting considerable room for further research. 
Project page: https://cs-people.bu.edu/array/research/cola/

+ Cola: A Benchmark for Compositional Text-to-image Retrieval
Arijit Rayl.3
Filip Radenovic
Abhimanyu Dubey²
Bryan A. Plummer
Ranjay Krishna
Kate Saenko
(array, bplum, saenko (bu. edu, ( (#iLipradenovic, dubeya | con, ranjayêcs wash i inntinntonnn .edu
Boston University, Meta AI (FAIR). Uniiversity of Washington
Abstract
Compositional reasoning is a hallmark of human visual intelligence. Yet despite the size of large vision-language models, they struggle to represent simple com- positions by combining objects with their attributes. To measure this lack compositional capability, design Cola, rext-to-image retrieval benchmark to Compose Objects Localized with Attributes. To solve Cola, a model must retrieve images with the correct configuration of attributes and objects, and avoid choosing a distractor image with the same objects and attributes but in the wrong configuration. Cola contains about 1.2k composed queries of 168 objects and 197 attributes on around 30K images. Our human evaluation finds that Cola is 83.33% accurate, similar contemporary compositionality benchmarks. Using Cola as a testbed, we explore empirical modeling designs to adapt pre-trained vision-language models reason compositionally. We explore 6 adaptation strate- gies on 2 seminal vision-language models, using compositionality-con -centric test benchmarks Cola and CREPE. We find the optimal adaptation strategy is train multi-modal attention layer that jointly attends over the frozen pre-trained image and language features. Surprisingly, training multimodal layers on CLIP performs better than tuning larger FL.AVA model with already pre-trained multi- modal layers. Furthermore, our adaptation strategy improves CL.P and FLAVVA to comparable levels, suggesting that training multimodal layers using contrastive attribute-object data key, as opposed using them pre-trained. Lastly, we show that Cola is harder than closely-related contemporary benchmark, CREPE, since simpler fine-tuning strategies without multimodal layers suffice on CREPE, but not Cola. However, we still see significant gap between our best adaptation and human accuracy, suggesting considerable room for further research. Project page: attps: /ttes://ccspeeeee pttp:/plclppppp.eeep/ee/ppe.p.le bu. edu array/rraaa///aaaa www.a//esearcc/ce....
Introduction
Compositionality is a fundamental characteristic of human intelligence, allowing meaning of the whole [as] a function of the meanings of its parts" [9]. In language, the whole is sentence made up of words like nouns and adjectives. In vision, the whole is an image made up of visual elements like objects and attributes [27, 19]. For example, the expression "round white table" is a composition of the noun "table" and adjectives "round" and "white", visually represented in the leftmost photo of Fig. 1. Recent work has consistently identified that this type of compositionality. that between objects and their attributes something existing vision-language models struggle to represent [54, 34, 24]. Instead, they disperse attributes and ground them to distractor objects; for instance, they incorrectly match "round white table" the second left photo in
elicit "the
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Colo Benchmark
Single Object
Multi Object
table
round, white
Small wood table to the right large dark chair
Tan wood chair by large smooth
Round white table
Round table X
Tan wood chair by large brown smooth table
Small wood table to the right large dark chair X
Figure We present Cola, where a model has to Compose Objects Localized with Attributes. To solve Cola, model must match the correct image to the correct caption, not distractor image with the same objects and attributes but in the wrong configuration. We explore the design space of possible mechanisms to adapt existing models to this task; show that simple multimodal adaptation method to finetune pre-trate pre-trained vision-lane asioonlanguuae representations works best.
Fig. by grounding the attribute "round" to the round plate instead of the table. Queries involving two objects are even more challenging, see Fig. (right).
In this paper, we study the ability of large vision-language models to Compose Objects Localized with Attributes (Cola). Unlike related baselines that study compositionality using relationships [54] and scene graphs [34], we focus on attribute-object bindings, since finding objects with correct attributes crucial in many applications. For example, an embodied AI assistant told clean the "small wood table to the right of the large dark chair" should not start cleaning the "tan wood chair by the large brown wood smooth table." Additionally, object-attribute bindings should fundamentally be easier than compositions with relationships or scene graphs. However, we find that existing models still struggle with this simpler binding.
To explore these issues, we propose the Cola benchmark for composing objects localized with multiple attributes. Cola contains two kinds of compositions: single-object queries (Fig. 1 left) and multi-object (Fig. right). In each case, a model should associate the objects in the query with the correct attributes and ignore difficult distractor compositions where the query attributes are attached to distractor objects. Multi-object queries are harder since they showcase more compositions.
Unlike image-to-text setting in contemporary benchmark CREPE [34], Cola evaluates models using text-to-image, where text queries are used to retrieve the correct image from a set of im- ages. This is consistent with previous benchmarks for vision-language models [20, 41, 62]. Further, text-to-image retrieval is harder than image-to-ter tt-gettt-text retrieval because image encoders are weaker at distinguishing fine-grained differences in images for a given text than text encoders are at distinguish- ing fine-graing aieeggrined text [54]. Moreover, text-to- iooieaa is better aligned with practical applications, such user giving text instructions to machine to find certain objects.
Using Cola as a development testbed, our experiments add to the ongoing discussion that pre- rr-itrinnd vision-language models perform poorly on compositions [54]. Hence, we explore 6 finetuning strategies on 2 seminal vision-language models- CLIP [42] and FLAVA [51]- to find adaptation strategies that encourage compositionality the most. We finetune using 3 datasets (GQA [17],
CLEVR [23], and PACO [43]) and evaluate on testbeds (Cola as well as CREPE [34]). While exploring effective pre-training strategies is another valid avenue of exploration,
limit our work to
adaptation strategies since training a model from scratch expensive and can only be executed by a handful of research organizations.
We find that the best- performirn architectural choice during adaptation to be a multi-modal trans- former encoder-decoder [52, 51, 25] to further encode the visual and language representations from pre-trained model. Multimodal adaption performs significantly better than tuning the unimodal encoders (that encode just vision or language features), or tuning the prompt. Surprisingly, this adaptation improves both CLIP and FLAVA to produce comparable finetuned models, even though our CLIP model has fewer parameters and FLAVA was already pre-trained using multi-modal transformer

---

layers. This suggests that training multimodal layers using contrastive attribute-object data is key, as opposed to using them pre-trained on web data. Our adaptation also significantly outperforms standard ways to adapt/tune foundational models such as prompt-tuning [63], linear-probing [2], or tuning a comparable number of split-encoder layers. Similar to recent work identifying that structural compositionality is present in models but absent in their representations [30], our work finds that while pre-tra pre-trained representations might not exhibit compositionality, they can be adapted to do so. However, the stark difference between human accuracy and our best adaptation suggests considerable room for further research using our benchmark.
2 Related Work
Compositionality and image retrieval. Compositionality is a key aspect of human intelligence [10], especially in vision and language [7]. Vision-language compositionality has been explored for visual question answering [1], composed image retrieval (e.g., X in the style of Y) [48], and generation [55]. Compositionality is one crucial aspect of improving robustness to diverse queries, theme heavily explored in the vision-language community [45, 44, 50, I, 8J. With the recent popularity of foundation models, various works focus testing their compositional reasoning [34, 54, 24, 60). Compared to CREPE [34] and ARO [60]. a model must distinguish between difficult images in our case. Text-to- difficult-images is harder because distinguishing between difficult images (for a given caption) is harder than distinguishing between difficult captions [54]. Whereas benchmarks like Winoground [54] primarily evaluate broad and complex relational compositionality (e.g.. "man hugs woman from behind" vs "woman hugs man from behind"), we specifically focus on attribute object bindings in queries. This is motivated by practical applications such as an embodied agent trying to retrieve a custom object (like "a metal wrench with red rubber handle") in cluttered workspace with similar distractor objects [38]. Most works in the area of attribute-object image retrieval either focus on single attributes [18, 40] or multiple attributes in very niche domains with centered images and plain backgrounds of dresses [15], animals [58], shoes[59], or birds [56]. contrast, we focus on scenes with multiple objects and attributes where distractor objects also have the same attributes.
dision-language aligment. Recently, there has been flurry of image-text alignment models to learn the similarity of matched images and text in various ways. Some models use separate unimodal encoders [42, 21] for the image and text, whereas some [51, 32, 3, 11, 49] use multimodal encoders as well. Various strategies such as hard negative mining [32], concept distillation [39], and maintaining the momentum of image-text mappings [16] have been employed push performance. We focus on testing and improving the attribute-object binding capability of such models and choose the most seminal model, CLIP [42], which is widely adopted in various concurrent vision-language research/applications [53, 26, 46). Our approaches do not use any box annotations unlike recent text localization models [25, 33, 61], which we also see to underperform on text-to-image retrieval.
Adapting foundational models. Since training a new VLM from scratch is expensive, we wish to formulate simple adapter that improves the compositional attribute-object binding. Various works explore adapting foundation models [4] with prompt-tuning [63], linear-probing [2], and fine-tuning with residual connections [13]. Prompt-tuning [31] learns the embedding layer of the word inputs and keeps the model frozen. Inspired by the success of prompt-tuning [31], some works have also explored prompting in the vision [22, S] and vision-language [63, 47], and also for single attribute-objer compositions [36]. Our optimal finetuning strategy improves significantly over prompt and fine-tune toniin for attribute-object compositions in even more difficult settings. Our multi-modal strategies are similar to MAPL [35], except our lightweight adapter attends over language and vision representations, whereas MAPL only attends over language.
3 B Cola benchmark
Our goal is to adapt languaee features to improve the compositional binding of attributes to objects. Specifically, we aim to improve the classification of a query involving single or multiple objects with multiple attributes in an image. Images and language are composed of atomic concepts such as attributes and objects. The atomic concepts ("square", "plate") form certain compounds ("square plate"), and then the scene a combination of various such compounds ("square plate on white table"). Hence, we create a benchmark where we form queries using compositions of such
3

---

atoms and test a model's 's ability to distinguish between images that correctly contain the atoms in the correct composition to distractor images that contain them in the wrong composition. In total, the Cola benchmark contains about 1236 composed queries from 168 objects and 197 attributes on around 30K images from 4 datasets.
Cola contains two query types discussed below: singlc-object compounds and multi-object queries.
Retrieval using single-object queries. Single-object queries have multiple attributes grounded on one object. For examplc, the query "square white plate," " which is of the form, Q = a a2o, where a, E A is drawn from finite set of possible attributes and o E O is similarly category drawn from a finite set of objects. With this query, a model should associate the images with the correct attachment of attributes ("square," "white"" to the object ("plate"), and ignore incorrect attachments of the same attributes and objects (like square table but not square plates). Hence, the task is a text query for image retrieval among difficult distractors. We first create a list of queries with more than one attribute for an object. Next, we curate a set of images where at least one of the query words is present in the image. For example, for "square white plate," all images containing "square" objects, "white" objects, or "plates" are in the list of images to retrieve from. The goal of the retrieval problem is to score the images having the correct attachment of the attributes the query higher than others. We build the test set for single object queries using three datasets with object and attribute annotations: 1) GQA [17]: After filtering for objects with at least attribute annotated, we have 320 single-object queries composed of 1 14 objects and 1 114 attributes on 1952 images. The objects and attributes comprise common objects, making this split useful for practical applications. 2) CLEVR [23]: We have object shapes- cubes, cylinders and spheres, composed with 8 colors. 2 materials, and 2 sizes on 15K images. 3) PACO [43]: This split consists of objects similar to GQA. We have 400 queries composed of 51 objects and 61 attributes on 7921 images.
Retrieval with multi-object queries. Drawing on existing literature [34], multi-object query contains multiple objects, each with its own set of attributes. For example, "square white plate on top of brown wooden table," which is of the form, Q di 020) + 034402, where a; A is drawn from a finite set of possible attributes and 0j E O from a finite set of objects. In this setting, we want to check if the model gets confused with the wrong configuration of objects and attributes. Thus, we find distractor image-query pairs where the attributes and objects are switched. An example image for query Q = a101 + a202 would be of the form I' = 0201 + 0102. In other words, we switch the attributes of the two objects. We curate these distractors to ensure that 0r # O2 and @1 # 02- The retrieval task, framed with this formalism, is to rank the correct images for the correct captions such that it is ranked higher than the distractor images: to learn a relevance encoding (I, Q) for image I and query Q such that f[I Q)= > fII.Q) Q) & f(I', Q') f(I.Q") The test set is built using test split of the Visual Genome [28] dataset.
Filtering Cola multi-object with crowd workers. We use the object, attribute, and relationship annotations in the Visual Genome dataset [28] to create the multi-object queries. We filter the image-caption pairs with object and attribute compositions swapped as described above. We conduct a human-annotated cleaning of this filtered test set. We display the images and I' and queries Q and Q' 10 crowd workers and ask them choose which image is most relevant to which query. We only keep the image-query pairs where the majority of crowd workers can correctly assign the correct image to the query. After filtering, we are left with 210 data points (840 image-query pairs) with 1680 possible image-query matches. The human agreement (accuracy) on our validation set is 83.88% an average of 8.33 out of 10 humans agree that the first image matches to the first caption and second image to the second caption. Some qualitative examples are provided in Fig. 2a.
4 Exploring finetuning strategies with Cola
Given an image (4) and a query (Q), Cola evaluates a model f(1, Q) by measuring how well associates the correct image to the input query. Existing prained models don't perform well on this task since they fail to distinguish fine-grain me--graiedd differences in attribute-objer compositions. Hence, we explore finetuning strategies that use a dataset of image-language pairs where the language descriptions contain objects and attributes. Details of finetuning datasets are described in Sec. 5.3. We follow the standard finetuning paradigm by sampling batches of images and text from the attribute- object Cola finetuning dataset. Specifically, we match the correct images to the correct queries in each batch and minimize the NCELoss typically used in contrastive learning [42, 51]. This finetuning

---

Scuare White
Trainrd
Phoce
Image VIT
Teist Transforme
thal onde light loreee
Catarcatatte inkren
Transformer Encodor
Wood chai white
Brien heler murtee to
Transformat Decoder
(a)
(b)
Figure 2: a) Cola multi-object setting validation set: a human-cleaned difficult validation set for lesting attribute-object binding. The two images have similar objects and attributes but in different configurations. A model must match the correct images to the correct captions. b) The optimal adaptation strategy (MM-Adapter); a lightweight multimodal transformer encoder on top of frozen pre-trained encoders. The multimodal encoder crafts stronger representation by cross-attending to image patches and text tokens to attach the correct attributes to the correct objects. The stronger representation then trained to align with the frozen text representation.
step aims to improve the compositional binding of attributes and objects in pretrained dision-language features. This is in contrast training the multimodal layers on random batches of web image-text data.
Disjoint finetuning strategies Since CLIP [42] is commonly used for various tasks and is of the more lightweight vision-language foundation models, we focus most of our finetuning strategies with CLIP in mind. Although, also later use these finetuning strategies the newer and larger FLAVA [51] model. CLIP [42] consists of two encoders: one that encodes the input image and one that similarly encodes the input text. The output representations of the two modalities are used as separate embeddings for the image and text. To adapt these models for specific task/capability, researchers commonly use linear-probing or prompt-tuning. Linear-prou serearrprobbig trains linear layers on top of the frozen visual and text encoders using the finetuning dataset. Prompt-tuning learns the word embeddings of the query adapt the finetuning dataset domain without changing the weights of the model [63]. Other methods fine-tune the later layers of both the encoders. All these adaptation methods tune the parameters of the two encoders separately.
Joint multimodal strategies We hypothesize that the above common adaptation strategies don't 't ap- propriately capture the cross-modal interaction required for strong attribute-object binding. However, CLIP is significantly more lightweight than recent multimodal models. Hence, we explore lightweight multi-modal adaptation strategies adapt CLIP. We describe the best-performing multimodal adapta- tion strategy found in our experiments which is also depicted in Fig. 2b. This multi-modal adaptation borrows a transformer encoder-decoder [52, 51, 25] to attend over the image and language repre- sentations jointly. Let M = [4: Q]. denote the concatenated image patch representations extracted from CLIP's visual encoder and the token-level representations from the query. We compute self-attention over M using transformer encoder A = Att M). Finally, we use a classify token (a token randomly initialized), referred to as [CLS], that aatesds to all the self-attended features A using a transformer decoder to produce M-- This type of cross- coos-dnttnniing to self-attended features similar to FLAVA|[1] FLAVA[51)/MDTTT [25].
The standard practice in most multi-modal basedd prediction models would be to learn a linear layer to classify the output [CLS] ] token embedding [25, 51, 32, 11]. However, instead of learning a linear predictor, we compute the cosine similarity of [CLS] to the representations of the query tokens: di from the frozen text encoder, Q. We posit that aligning
frozen text encoder trained on larger scale data will act as a regularizer, helping performance on unseen compositions. Hence, the final score for given image-query pair is f(I +₂- M No out qi where N, is the number of tokens in the query. These two ablations are referred to as MM-pred and MM-adapter
query comes from [CLS], and the keys and values come from self-attended features, A.
5

---

("adapter" since we can think of the latter as adapting the image features to align better with text) in our experiments.
We also tried various flavors of computing a multi-modal adaptation inspired by FIBER [11] and ALBEF [32], which use cross attention between text and image. We would like stress that while exploring newer ways to fuse vision and language features is a valid avenue of research, we are interested in exploring the common themes current fusion methods that encourage compositionality the most to drive future research. We report the best method for simplicity and include the accuracies from other strategies in Table 3 in the supplemental.
Evaluation Setup
We evaluate models on the two types of queries described in Sec. 3 Cola and CREPE [34] datasets. All models are trained using Pytorch [37] and use the Huggingface transformers library [57]. Implementation details such as hyperparameters are provided in Sec. of the supplementary.
5.1
Metrics
Single object queries. For this type of query, we report the Cola MAP the mean average retrieval precision over difficult distractors. We further differentiate the mAP between seen and unseen queries. We split Cola into seen and unseen sets by removing some attribute-object pairs from the training set. For example, "square white plate" is unseen if this combination is absent in finetuning; however, "square white bowl" or a "square plate" may be present. In our test set, we have 320 (150 unseen, 170 seen) queries on 1950 images for GQA, 96 (32 unseen, 64 seen)' queries on 22500 images for CLEVR, and 400 (200 seen, 200 unseen) queries on 7921 images in PACO.
Multi-object queries. Recall that we have two images and two captions, and the task is to match the correct caption to the correct image. If we denote the prediction score for an image and query to be M)). we regard a prediction be correct if f(I, > f(I', & f(I', M') > M"). where images 1 and /' are paired to captions M and M' respectively. Using this criterion, we compute the Cola multi-object accuracy. The random accuracy is 25% since there are four ways to match the two captions to the two images. We also evaluate on contemporary dataset, CREPE [34], where the task is inverse. For CREPE, we compute an image-to-text (I2T) accuracy, where a model must match the correct text from two choices to the given image. Note that there is only one image for the two caption choices in CREPE [34]. The random accuracy is 50% since it is a binary task.
5.2 Explored finetuning strategies
Recall that the best-performing finetuning approach found is multimodal adaptation (MM- Adapter) for tuning pre-trained image and text features as described in Sec. 4. We compare against popular tuning methods like linear probing, tuning the prompt embeddings (prompt-tuning), and fine-tuning the whole model (FT all) or the last two layers (FT Late). These adaptations are applied separately the base model for comparison. More details in the supplementary (Sec. 3). Since our adaptation uses multimodal attention. we also compare it to seminal model that uses multimodal attention in pretraining. We chose FLAVA [51] since is one of the recent models after CLIP which is bigger, more accurate [51] and has easily available pre- pre-trained weights.
5.3 Finetuning datasets
The Cola training sets are also built the same way as described in Sec. using the training splits of GQA [17, 28). CLEVR [23] PACO [43], and Visual Genome [28]. For GQA, the training split contains 1381 objects and 601 attributes that compose 27078 queries on 74K images. For CLEVR, we have 3 shapes composed with 8 colors and 2 sizes 70K images. Finally, for PACO, we have 75 objects and 55 attributes that compose 18696 queries on 37883 images. The Cola multi-object
'we also computed the Fl score and see trends remain the same; more details in the supplementary. See. 2. page
we report "seen" only on 32 classes to avoid disbalance with unseen. "All" MAP is on all 96 classes. "Seen" trends hold same with 64 classes well; more details in appendix.

---

GQA
CLEVR
PACO
All
Unseen
Seen
All
Unseen
Seen
All
Unseen
Seen
CLIP
36.53
39.06
34.24
15.38
15.01
15.32
12.21
8.64
15.79
+Linear
40.44
42.87
38.24
47.96
29.43
46.75
14.22
6.75
21.68
+prompt-tunc
37.40
40.69
34.43
29.61
23.17
28.05
12.76
5.92
19.61
+FT all
38.81
40.85
36.95
52.32
19.00
47.95
14.58
6.49
22.66
+FT late
42.19
44.61
40.01
64.06
27.53
67.48
15.66
8.74
22.58
f. +MM-Pree (us)
45,99
48.6
43.64
75,80
51.98
80.72
15,49
8.00
22.94
g. +MMM-Adapter (us)
46.83
48,86
44.99
88.21
89.52
77.00
18.56
11.47
25.66
h. FLAVA
39.65
42.18
37.37
15,41
13.27
15.93
12.53
7.29
17.76
+Linear
37.07
39,96
34.46
19.30
17.53
18.52
11.65
7.90
15,39
j. +FT-late
39.58
42.26
37.16
77.95
72.72
66.42
12.82
5.79
19.84
k. +MM-Pred (us)
47.12
51.53
43.13
90.43
85.78
86.07
18.57
10.71
26.44
1. +MM-Adapter (us)
48.54
52.55
44.91
91.10
86.64
87.39
19.36
11.16
27.55
Table 1: mAP results the Cola single object compounds setting. Our multimodal adaptation
(MM-Adapter) performs better than common tuning methods. Further, multimodal attention to adapt the image representation (MM-Adapter) generalizes better than using simply as a prediction head (MM-Pred). MM-Adapter on CLIP is better than tuning the pre-trained multimodal attention layers of the bigger FLAVA (+FT late). MM-Adapter further improves FLAVA.
compounds training split has 551, 1,980 multi-object compounds on 71,174 images. Only the test split cleaned using human annotations. For datasets built on GQA [17] and Visual Genome [28]. leverage the annotations to explore the effects of different kinds of data queries. We use the region descriptions (denoted as CagionCap in the tables) in Visual Genome test if linguistic diversity helps over templated captions (Cola single objects and multi-object). We also compare to hard negatives from the Cola multi-object pairs. We finally have a combined setting where we use all data.
6 Results
Recall that we evaluate two settings for Cola the single-object compounds setting and the multi-object compounds setting as defined Sec. 3. We discuss the quantitative results below and some qualitative results are shown in Fig. 3.
6.1 Cola Single-object retrieval
Multimodal adaptation is more effective than other tuning methods: In Table 1, compared to prompt-tuning (row c), fine-tuning to--ttuing all CLIP (row d), or fine-tuning a few of the later layers (row e), tuning a multimodal attention layer of same/lesser parameters has higher mAP (row f and g). Linear probing (row b), although cheaper, significantly underperforms. This is not surprising since multimodal attention
the image regions and text tokens offers more flexibility to the model to learn to bind the right attributes to the right object region. Tuning the whole model is also worse than tuning the later unimodal layers (row e). This might be because fine-tuning the whole model requires larger batch sizes with significantly more data. In Fig 3, we show the comparison of tuning unimodal layers our multimodal adaptation (since tuning the unimodal layers closest in performance). Qualitative examples from each of the other adaptation methods are displayed in Figs. 5-11 in the supplementary.
Using pre-trained multimodal attention layers is not enough training them on attribute-object compositions is key: In Table we that MM-Adapter (row g) on CLIP VIT B-32 (151 M params) outperforms tuning the last two multimodal layers of FLAVA B-16 [51] (241M params) model (row i) tuning linear probe (row i). Surprisingly, tuning the last FLAVA multimodal layers (row worse than replacing them and training using MM-Pred and MM-Adapter layers (rows k and l). This suggests that training multimodal layers during adaptation (as opposed to pre-training) is key.
MM-Adapter better than using multimodal attention as a prediction head: Recall that one of the ablations of our approach is aligning the output of the multimodal encoder the frozen
7

---

Multi-Obj Acc
T21
I2T
Cola
CREPE [34]
Single-Object Cola GQA
Data Type
Random
25,00
50.00
Unseen
Seen
Human
83.88
a. RegionCap
0,4711
0,4965
0,4481
CLIP
21.42
77.43
SingleObj
0.4683
0.4886
0.4499
a. + Linear
30.47
87.35
MultiObj
0,4641
0,4795
0,4501
b. Prompt-tune
27.14
80.81
HardNeg
0.4688
0.4843
0.4548
C. + FT all
34.76
82.39
Combined
0.4788
0.4983
0,4612
d. FT late
36.19
87.14
Multi-Object
e. + MM-Pred (our)
41.42
77.84
E. MM-Adapter (our)
40.95
87.02
Cola
CREPE
g. FLAVA
24.76
65.10
RegionCap
0.3114
0.8833
h. Linear
22.38
55.10
b. SingleObj
0.2745
0.9023
i. FT
22.38
58.11
MultiObj
0.3975
0.8702
J. MM-Pred (our)
39.04
81.37
d. HardNeg
0.3483
0.8775
+ MM-Adapter (our)
40.47
74.81
Combined
0.3893
0.8798
(a)
(b)
Table 2: Results
multi-object compounds setting our Cola task and CREPE. Simpler
methods suffice on CREPE, but not on Cola, suggesting that Cola is harder. Red-orange-yellow is in decreasing accuracy order. MM-Adapter and MM-Pred CLIP perform well average both. b. Table showing the effect of the data type used in the contrastive batch training. Having multi-object captions in the data helps Cola performance while maintaining CREPE performance.
text embedding, making multimodal image-feature "adapter" (MM-Adapter). This contrasts to using the multimodal module as a prediction head with linear layer (MM-Pred) As shown in Table 1, -dd-ttpterr outperforms MM-Pred (row g vs f), especially on unseen classes and on exhaustively annotated datasets like CLEVR [23] and PACO [43]. We posit that aligning to the frozen text representation like regularizer since was pre-trained on more data.
6.2 Multi-object retrieval
Simpler methods suffice for CREPE, but not for Cola, suggesting that Cola is harder task: As shown in Table 2a, linear probing (row a) or simple fine-tuning (rows and d) suffice for CREPE [34]. However, our MM-Adapter improves further on Cola (row vs row c, d), while maintaining performance on CREPE. This also suggests that text-to-image image matching is harder than image-to-text matching, which is also reflected in Winoground [54].
Baseline CLIP and FLAVA perform below chance; If we evaluate off-the-shelf CLIP [42] and FLAVA [51] on our Cola dataset, This is consistent with the findings in Winoground [54].
in Table 2a that it performs below random (row and g).
Training late multimodal layers from scratch help, CLIP+MM-Adapes performs better overall: We improve performance by training multimodal layers from scratch on top of CLIP and FLAVA as shown in Table 2a, row e, f. k. Interestingly, as shown in Table 2a, linear probing or tuning the pre- trained multimodal layers of FLAVA hurts performance (row vs row h and i). This could be because tuning adversely perturbs the parameters trained large-scale data. Finally, CLIP+MM-Adane (row f) performs comparably well as tuning multimodal layers on FLAVA (row vs and k).
6.3 Effect of fine-tuning data
Difference between free-form captions and templated Cola queries is minimal. Hence, Cola templated queries are useful: In Table 2b (row vs d, e), see that templated queries perform

---

o'whlte
tnll
Round white table (Top 3t
CLP
CUP www.oroorrroooom (044)
-M0 Aagaaee rours
sting
Saby mraningg standing gealls
Small wood table 1o the
la wood clair as larg:
Squaro white plase (Top 31
standing brown gratte
+T Lane
CUP
X
BMA wbatterereeeeeeeeeeeeeeeeeeeee.. (ou))
MMA
Figure 3: Qualitative results of multi-object matching (left) and retrieving single object with multiple attributes (right).
redorada Park laro plaoe on metal whise
overenns san-cllenttural attriattenaijeets
Alrinal improverents on arpelallenr dbjects
CUP 6TT4
2001
Pulure whan colors arbiiganss
(Mak pllce losts white, "arra"" satpacttio)
improvenent on acciadedd butes like becogroands
Aeresal improveennn conmenlly reprosented attributts Oue Suu Cup
Mili Lovk to the righ seall
Seail lowil on witt
Brick witia well Chp
STOP
-Hoostaper par
WOLFRUM
Palure
Figure 4: Qualitative results on cases where models struggle with multiple object-attribute composi- tionality (left). Cases where we see the most improvement and the least on single-object compositional retrieval are shown on the right.
just as well as free-form region descriptions. This shows that Cola queries are still useful for attribute-object binding despite being free-form.
Having multiple objects with multiple attributes in a caption helps: When we combine single in Table 2b that performance increases especially
object captions with multiple object captions, on the multi-object Cola setting while maintaining performance on the single-object setting (row b vs c). The multi-object and single-object captions are formed on the same number of images. Combining all types of data along with hard negatives (row e) doesn't 't seem to affect performance much.
6.4 Quality of Cola Benchmark
Our Cola MAP Ps are harder and are more difficult to improve on: We also computed the per- formance based on the standard formulation of mAP, where all images are included in the list to

---

retrieve from. In contrast, Cola mAP only includes hard distractors as defined in Sec. 3. We see that trends remain the same with the standard mAP. However, our Cola mAP harder to improve on. The standard MAP (supplementary, Table 2) improves by more than 2x on GQA and 10x on CLEVR. In contrast, we only improve by 1.09x on GQA and 2x on CLEVR with our harder Cola MAP.
Ambiguous colors, spatial relationships, and size are some common themes where models underperform in Cola benchmark: We analyze the types of compositional queries that models find difficult on our benchmark. In the qualitative examples shown in Fig. 4, we see that compositions involving ambiguous colors (due to lighting) and spatial relationships are difficult for multiple-object cases. This is likely because spatial relationships are often inconsistently annotated in training data- e.g. "to the left of can sometimes be from the viewer or image perspective. More examples are in the supplemental (Fig. 4, 16, 17).
Significant improvements non-salienttoccluded objects. Improvements on larger objects minimal: On attribute-object compositional retrieval for single objects, we see the most improvements using adaptation method are non-salient occluded objects (like small sign), shown in Fig. (right). Queries that are commonly represented in training sets (like "clear blue sky" skies most commonly blue) have minimal improvements from pre-trained models.
Discussion and Limitations
This work finetunes vision- lsiion-angguag models to test design choices for compositional sttribute-ojecct retrieval. Thus, compared to the original pre-trained models [42], we may lose some other generic capabilities, such as question answering, captioning, etc. For example, our best adaptation scores 83% zero-shot on CIFAR1O [29], as compared to 87% for pre-trained CLIP. However, by just using 5% of CIFAR training data (less than epoch), we reach 92% accuracy while maintaining performance on our Cola task. We would also like to stress that our goal is not to train new foundation model but to explore design choices that allow for compositional reasoning. While we focus on attribute-object compositionality, there is still significant room for exploration of other types of compositional structures such as relationships, scene graphs, and counting [14, 12]. A great avenue for future work could be collecting more detailed annotations (about the type of objects and compositions in the queries) on larger sets of images to help pinpoint themes where models are failing. Further testing is also required to see how it fares with sensitive attributes and objects whether it is predisposed towards attaching incorrect attributes to objects because of racial/political biases in the data [6]. Additionally, it important re-evaluate our results in the context of newer vision-language models that are being proposed. Finally, we would like to point out that our finetuning strategy isn't 't specific compositions, suggesting its potential applicability for adaptation to other downstream tasks, especially since similar strategies and modules have been used for other computer vision tasks [52, 51, 25].
Conclusion
We present new task, Cola, to test the compositional attribute-object binding of vision-language models. This is important for various practical applications like an assistive agent requiring an understanding of fine-grained differences between objects in cluttered workplaces. We explore the architectural choices of adapting large vision-language models that encourage such reasoning. We show that a light-weight multimodal adaptor can improve this capability in pre-trained vision- language model as strong baseline for further research. We hope that Cola serves strong benchmark and our adaptation choices as strong baselines for improving compositional vision- language intelligence.
Acknowledgements. We wish to thank Dhruv Mahajan and Jang Hyun (Vincent) Cho for their valuable guidance in the initial phases of the project. We also wish to thank the anonymous reviewers for their thoughtful comments and suggestions. This material is based upon work supported, in part, by DARPA under agreement number HR001 12020054 awarded to Kate Saenko and Bryan Plummer BU. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the supporting agencies.
10

---

References
[1] Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh, C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset, Apr. 2017. AXXi:1704.08244 [es]. 3
[2] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes,
Nov. 2018, arXiv:1610.01644 644 [cs, stat],
[3] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Amtoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a Visual Language Model for Few-Shot Learning. Nov. 2022. arXiv:2204. 1 14198 [cs]. 3
[4] Amit Alfassy, Assaf Arbelle, Oshri Halimi, Sivan Harary, Roei Herzig, Eli Schwartz, Rameswar Panda, Michele Dolfi, Christoph Auer, Kate Saenko, PeterW J. Staar, Rogerio Feris, and Leonid Karlinsky. FETA:
Towards Specializing Foundation Models for Expert Task Applications, Dec. 2022, arXiv:2209.03644 [cs].
[5] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring Visual Prompts for Adapting Large-Scale Models, June 2022. M.KIv:2203.172744 [cs]. 3
[6] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on faimess, accountability and transparency, pages 77-91. PMLR,
2018. 10
[7]
Noam Chomsky. Aspects of the Theory of Syntax. The MIT Press, 50 edition. 1965. 3
[8] Gordon Christie, Ankit Laddha, Aishwarya Agrawal, Stanislaw Antol, Yash Goyal, Kevin Kochers- berger, and Dhruv Batra. Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes, Sept. 2016. arXiv:1604.02125 [cs]. 3
[9] MJ Cresswell. Logics and languages. 1973. 1
[10] M. J. Cresswell. Logics and Languages. Synthese, (0(2):375-387, 1973. Publisher: Springer. 3
[11] Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu,
Ce Liu, Yann LeCun, Nanyun Peng. Jianfeng Gao, and Lijuan Wang. Coarse-to-Fine Vision-Language
Pre Pre-training with Fusion in the Backbone, Nov. 2022. arXiv:2206. 076443 [cs]. 6,
[12] Mona Gandhi, Mustafa Omer Gul, Eva Prakash, Madeleine Grunde-Mcll auglaug,,,,, Ranjay Krishna, and Mancesh A Agrawala. Measuring compositional consistency for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5046-5055, 2022. 10 [13] Peng Gao, Shijic Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongteng Zhang, Hongsheng Li, and Yu Qiao. CLIP-Adaperr: Better LisionLLanguag Models with Feature Adapters, Oct. 2021. arXiv:2110.0444 les]. 3
[14] Madeleine Grunde-Melaughhiee Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for compositional ppatio-tempone reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1287-112297, 2021. 10
[15] Sheng Guo, Weilin Huang, Xiao Zhang, Prasanna Srikhanta, Yin Cui, Yuan Li, Hartwig Adam, Matthew Scott, and Serge Belongic. The iMaterialist Fashion Attribute Datasct. In 2019 JEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 31 311-1116,, Scoul, Korea (South), Oct. 2019.
IEFE. 3
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning, Mar. 2020. Xiv: 19: 191 1, 1.05722 [cs]. 3
[17] Drew A. Hudson and Christopher D. Manning. GQA: A New Dataset for Real-World Visual Reasoning
and Compositional Question Answering. May 2019. arXiv: 9902009506 [cs]. 2, 2.4. 6, 7
[18] Phillip Isola, Joseph J. Lim, and Edward H. Adelson. Discovering states and transformations in image collections. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
1383-1391, Boston, MA, USA, June 2015. IEEE. 3 [19] Jingwei Ji. Ranjay Krishna. Li Fei- and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10236 102477 2020. 1
[20] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916 16. PMLR, 2021. 2
[21] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parckh, Hieu Pham, Quoe V. Le. Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Languaage Representation Learning With Noisy Text Supervision, June 2021. urXiv:2102.05915 [cs]. 3
[22] Menglin Jia, Luming Tang. Bor-Chun Chen. Claire Cardie, Serge Belongie, Bharath Hariharan, and
Ser-Nam Lim. Visual Prompt Tuning, July 2022. arXiv:2203.121 [cs]. 3
[23] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning,
Dec. 2016. 161..06990 [cs]. 2, 4. 6,
[24] Aishwarya Kamath, Sara Price, Jonas Pfeiffer, Yann LeCun, and Nicolas Carion. TRICD: Testing Robust
Image Understanding Through Contextual Phrase Detection. 2023. 1,3
11

---

[25] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.
MDETR - Modulated Detection for End-to-End Multi-Modks Understanding, Oct. 2021. MtXiv:210.122663
[es]. 2. 10
[26]
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. DiffusionCLIP: Text-Guided Diffusion Models for
Robust Image Manipulation, Aug. 2022. arXiv:2110.0271) [es].
[27] Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting Language and Vision Using Crowdsoureed Dense Image Annotations. International Journal
of Computer Vision. 123( I 3:3273,, May 2017.
[28] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual Genome: Con- necting Language and Vision Using Crowdsourced Dense Image Annotations, Feb. 2016. arXiv:1602.07332
[cs]. 4, 6.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. I Learning multiple layers of features from tiny images. 2009. 10
[30]
Michael A Lepori, Thomas Serre, and Ellie Pavlick. Break it down: Evidence for structural compositionality
preprint [31] Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficient Prompt Tuning, Sept. 2021. arXiv:2104.0865
in neural networks. arXiv preprint arXiv:2301. 230804401.,, 2023. 3
[cs]. 3
[32] Junnan Li. Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation, Oct.
2021. arXiv:2107.077 077007651 [cs]. 5,5,
[33] Liunian Harold Li. Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang. Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded Language- Image Pre-training, June 2022. arXiv:21 12.03827 [es].
[34] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. CREPE: Can Vision-Language Foundation Models Reason Compositionally?, Jan. 2023, arXiv:2212.07796 [cs]. 1, 1, 3,
4, 6,8
[35] Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. MAPL: Seaamcter-fffciiee Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot
Prompting. Oct. 2022. wxiv:2210.07177 [es]. 3
[36] Nihal V. Nayak, Peilin Yu, and Stephen H. Bach. Learning to Compose Soft Prompts for Compositional
Zero-Shot Learning, Sept. 2022. arXiv:2204.081 03574 [cs]. 3
[37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Veto,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Parformance Deep Learning Library, Dec.
2019. 112iv:1912.01703 [cs. stat]. 6
[38] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. VirtualHome: Simulating Houschold Activities Via Programs. In 2018 IEEE/CVF Conference on Computer
Vision and Pattern Recognition. pages 8494-8502, Salt Lake City, UT, June 2018. IEEE. 3
[39] Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende. Yash Patel, Yi Wen. Vignesh Ramanathan. and Dhruv Mahajan. Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainin, Jan. 2023, arXiv:2301.00 102280 [cs]. 3
[40] Filip Radenovic, Animesh Sinha, Albert Gordo, Tamara Berg, and Dhruv Mahajan. Large-Scale Attribute- Object Compositions, May 2021. arXiv:2105. 1011111.1111 11373 [cs]. 3
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR.
2021. 2
[42] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, Feb. 2021. arXiv:2103.00020 [cs]. 2, 3,
4, 4,5, 8, 10
[43] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yiwen Song, Abhimanyu Dubey, and Dhruv Mahajan. PACO: Parts and Attributes of Common Objects, Jan. 2023. arXiv:2301.01995 [es]. 2, 4,
6.
[44] Arijit Ray, Gordon Christic, Mohit Bansal, Dhruv Batra, and Devi Parikh. Question Relevance in VQA: Identifying Non-Visual Visual And False-Premise Questions, Sept. 2016. arXiv:1606.0661 [cs]. 3
[45] Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lee, and Giedrius Burachas. Sunny and Dark Outside?! Im- proving Answer Consistency in VQA through Entailed Question Generation, Sept. 2019. arXiv:1909 (4496
[es]. 3
[46]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-Resolution Image Synthesis with Latent Diffusion Models, Apr. 2022. arXiv:21 12. 10752 [cs]. 3
[47] Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen- Yu Lec, Kate Sacnko, and Tomas Pfister. Prefix Conditioning Unifies Language and Label Supervision, June 2022. arXiv:2206.00 0.1 125 (es],
12

---

[48] Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen- Yu Lee, Kate Saenko, and Tomas Pfister. Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval, Feb. 2023.
arXiv:2302.03088 [cs].
[49]
Madeline Chantry Schiappa, Michael Cogswell, Ajay Divakaran, and Yogesh Singh Rawat. Probing conceptual understanding large visual-language models, 2023. 3
[50] Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Sycle-Consistency for Robust Visual Question Answering. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6642-6651, Long Beach, CA, USA, June 2019, IEEE. 3
[51]
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A Foundational Language And Vision Alignment Model, Mar. 2022.
arXiv:21 12.04482 [es]. 2, 3, 4, S. 8, 10
[52] Hao Tan and Mohit Bansal. LXMERT: Learning Cross (toss-Moddalty Encoder Representations from Trans- formers, Dec. 2019. AXii:1998.004999 [es]. 2,5, 10
[53] Reuben Tan, Arijit Ray, Andrea Burns, Bryan A. Plummer, Justin Salamon, Oriol Nieto, Bryan Russell, and Kate Saenko. Language-guide
audio-visual source separation via trimodal consistency, 2023. 3 [54] Tristan Thrush, Ryan Jiang. Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. Apr.
2022. arXiv:2204.03162 [cs]. 2, 3
[55] Ben Usman, Dina Bashkirova, and Kate Saenko. Disentangled Unsupervised Image Translation via Restricted Information Flow, Nov. 2021.
[cs].
[56] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The Caltech-UCSD
2irds-200-2011 Dataset, July 2011 1. Issue: 2010-001 Num Pages: 8 Number: 2010-001
Place: Pasadena,
CA Publisher: California Institute of Technology. 3
[57] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: Statc-of-the-Are Natural I Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Oct. 2020. Association for Computational Linguisties. 6
[58] Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-Shot Learning A Compre-
[es]. Aron Yu and Kristen Grauman. Fine-Grained Visual Comparisons with Local Learning. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 192 199, Columbus, OH, USA, June 2014. IEEE. 3
hensive Evaluation of the Good, the Bad and the Ugly, Sept. 2020. arXiv:1707
[59]
[60] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models hchave like bags-of-words, and what to do about it?, Oct. 2022, arXiv:2210.01936
[es]. 3
[61] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yon-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. GLIPv2: Unifying Localization and Vision- Language Understanding, Oct. 2022. arXiv:2206.0S88 [es].
[62] Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang. Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5579-5581 June 2021.
2
[63] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to Prompt for Vision- Vision-Language Models. International Journal of Computer Vision, 130(9))2337-224 Sept. 2022. arXiv:2 109.01 134 [es].
5
13


Reviews:
[{"review_id": "8n0kavBWtf", "reviewer": "Reviewer_pbZZ", "summary": "", "questions": "", "limitations": "The addressed limitations are worth exploring in the future work.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed benchmark is practical and could fill a gap in the task of compositional text-to-image retrieval. The two types of queries show a practical significance. Experiments seem to be sound in demonstrating the meaning of the proposed benchmark Cola.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "B4OKBnjita", "reviewer": "Reviewer_dcEa", "summary": "", "questions": "", "limitations": "I see no particular limitations of this work (though again I don't work in VQA)", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is well organized and provides very clear examples of what is available in the dataset and how the dataset was collected. They also provide extensive benchmark using pre-trained and fine-tuned systems against their dataset", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "eOrj0UIuxg", "reviewer": "Reviewer_QhKB", "summary": "", "questions": "", "limitations": "The authors talk about the limitations of their work and the fact that fine-tuning the pre-trained networks for this image retrieval task might interfere with the models' other capabilities such as question answering, and image captioning.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The problem that this work is tackling is very interesting and practical, especially in the domain of grounded language learning and multimodal learning.\n2. The gap in performance between the human baseline and SOTA large vision-language models indicates there is an ample opportunity for future research.\n3. The method of selecting distractor images is efficient and makes the task more challenging which is evident based on the results when compared to another benchmark.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "8eEabx7XlX", "reviewer": "Reviewer_H1og", "summary": "", "questions": "", "limitations": "There are only two model architectures, CLIP and FLAVA. These architectures are also similar or like the proposed approached (FLAVA has a cross-attention transformer). It is hard to make conclusions based off limited approaches.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed approach maintains an open-world paradigm. \nTheir approach shows noticeable improvements over the other fine-tuning approaches.\nThey are studying an interesting problem and proposing a dataset that encapsulates multiple domains of object-attribute combinations and images.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "oba70VdrtR", "reviewer": "Reviewer_wAA7", "summary": "", "questions": "", "limitations": "1. The statistics regarding the dataset are largely absent in the paper. The authors could include the distribution of object-classes, the distribution of properties for the objects etc. The purpose of this is to showcase the lack of any biases that could have caused the low results. For example, choosing objects and their properties which are particularly hard for a model. Such examples would essentially show that a model is bad at working with certain object-attribute pairs, rather than actually having a lack of compositional behaviour.\n2. The authors are requested to also include performance of a text-only model to showcase the absence of any further bias in the dataset.  ", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "The problem the authors have presented their contributions for is important for deep learning. The simplicity of the tasks, paired with the low results that the baseline produce on them, makes for a good case (which is well-accepted) that compositionality is a hard task for even recent deep learning models.\n\nThe fine-tuning strategies, being task-agnostic, presented by the authors are a good addition to the dataset the authors present.\n\nThe experimentation, while not particularly extensive, is still enough, in my opinion, to show the usefulness of the dataset the authors present.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors propose Cola, a text-to-image retrieval benchmark, to test the ability of large vision-language models when it comes to compositional groundings. The new dataset can help examine the models' ability to detect the correct image corresponding to the textual description of objects based on their attributes when there is a challenging alternative image (distractor image). All the reviewers appreciate the merits of the constructed Cola benchmark dataset, regarding that the task is interesting, challenge. Also the proposed approach shows noticeable improvements over the other fine-tuning approaches. 

The paper can be accepted. The authors are highly suggested to consider the reviewers' comments and incorporate them into their final version and make corresponding revisions.
Decision: Accept (Poster)