Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality
Authors: Harman Singh
Pengchuan Zhang
Qifan Wang
Mengjiao Wang
Wenhan Xiong
Jingfei Du
Yu Chen
Venue: EMNLP 2023 Main (2023)
Keywords: Vision-Language Compositionality
Systematic Generalization
Vision-Language Contrastive Learning
Multimodal Foundation Models
TL;DR: Significant improvements in compositional reasoning capabilities of Vision Language models via a new scene-graph guided coarse-to-fine contrastive learning technique.

Abstract: Contrastively trained vision-language models have achieved remarkable progress in vision and language representation learning. However, recent research has highlighted severe limitations of these models in their ability to perform compositional reasoning over objects, attributes, and relations. Scene graphs have emerged as an effective way to understand images compositionally. These are graph-structured semantic representations of images that contain objects, their attributes, and relations with other objects in a scene. In this work, we consider the scene graph parsed from text as a proxy for the image scene graph and propose a graph decomposition and augmentation framework along with a coarse-to-fine contrastive learning objective between images and text that aligns sentences of various complexities to the same image. We also introduce novel negative mining techniques in the scene graph space for improving attribute binding and relation understanding. Through extensive experiments, we demonstrate the effectiveness of our approach that significantly improves attribute binding, relation understanding, systematic generalization, and productivity on multiple recently proposed benchmarks (For example, improvements 
up to $\mathbf{18}$% for systematic generalization, $\mathbf{16.5}$% for relation understanding over a strong baseline), while achieving similar or better performance than CLIP on various general multimodal tasks.

Coarse-to-Fine Contrastive Learning in Image-Text-Grrph Space for Improved Vision-Language Compositionality
Harman Singh", Pengchuan Zhang', Qifan Wang',
Mengjiao Wang', Wenhan Xiong', Jingfei Du Yu Chen
MMeta AI *Anytime.Al
harmansingh. wigehhiiit@@mmii witdeegaal..c..
Ipengchuanzhang, wqfer, mengjiaow, xwhan, jingfeidu) @meta. com ychendanyt ime.aii.tttmmmiiccommm
Abstract
Snstemattiin Sccres
Contrastively trained Assionlanguuae models have achieved remarkable progress in vision and language representation learning. How- ever, recent research has highlighted severe limitations of these models in their ability to perform compositional reasoning over ob- jects, attributes, and relations. Scene graphs have emerged as an effective way to under- stand images compositionally. These are graph- structured semantic representations of images that contain objects, their attributes, and re- lations with other objects scene. this work, consider the scene graph parsed from text as proxy for the image scene graph and propose graph decomposition and augmen- tation framework along with coarse-to-fime contrastive learning objective between images and text that aligns sentences various com- plexities to the same image. We also introduce novel negative mining techniques in the scene graph space for improving attribute binding and relation understanding. Through extensive ex- periments, we demonstrate the effectiveness of approach that significantly improves at- tribute binding, relation understanding, system- atic generalization, and productivity on multi- ple recently proposed benchmarks (For exam- ple, improvements up to 18% for systematic generalization, 16.5% for relation understand- ing over strong baseline), while achieving similar or better performance than CLIP on var- ious general multimodal tasks.
60
45
(b)
80
A80 Scores
75
-23.1%
+9.1%
70
The green plant and the white bird
The white plant and the green bird
CLIP X
NegCLPP M
(c)
Figure 1: (Left) a) A typical example from the ARO benchmark for testing attribute understanding of VLMs. VLMs struggle with matching the image to the cor- rect caption (in green). (Right) Average scores of Mo- saiCLIP (our method) compared with NegCLIP and CLIP on prominent compositionality benchmarks for measuring b) Systematic Generalization c) Attribute, Relation, and Word Order understanding.
et al., 2022: Thrush et al., 2022; Ma et al., 2022). In particular, they struggle with binding correct attributes to the correct objects, understanding rela- tions between objects, generalizing systematically to unseen combinations of concepts and to larger and more complex sentences.
Some works have made progress this problem. Yuksekgonul et al. (2022) show that hard negative mining of images and during fine-tuning is a promising first step to improving compositionality. However, performance gains are highly dependent on how clean the training data is, and generaliz- ing to unseen combinations of concepts remains challenge. Doveh et al. (2023) use LLMs for hard negative mining and Sescante-Bonille et al. (2023) explore using synthetic datasets to improve compositional understanding in VLMs. Synthetic datasets lead domain gap compared natural datascts. We aim to develop a purpose ap- proach for improving compositionality all such
1 Introduction
Recent progress in contrastive learning using large- scale image-text data for joint image-text represen- tation learning has led to Vision-Language mod- els (VLMs) like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) that show remarkable zero- shot classification and retrieval capabilities. How- ever, recent works have shown that these models struggle at compositional reasoning (Yuksekgonul
Work done while at Meta.

---

contrastively trained VLMs.
fine-tuning datasets and over four compo- sitionality benchmarks (11 datasets) prove the efficacy of MosaiCLIP for improving com- positionality.
In this paper, we consider a scene graph repre- sentation of the image and text. We observe that multiple sub-graphs of the text scene graph with different semantic complexities can be matched with the same image. Performing this matching improves fine-grained and hierarchical understand- ing of text and thereby, of images. We achieve this by developing a scene graph-based text decompo- sition strategy that creates a seene graph for any given text, decomposing it into sub-graphs, and matching an image to multiple sentences derived from these sub-graphs (See Fig. 2 for an overview). Each sub-graph represents distinct part of the image, aligning well with CLIP's original image- text matching objective. Focused on improving attribute binding and relation understanding, we de- velop novel hard negative graph creation strategies which helps VL contrastive learning. We provide a novel Image-to-Multi-Tenn contrastive loss for matching individual images to multiple sentences. Our approach of matching texts of different com- plexity (from coarse-grained to fine-grained) to the image leads to fine-grained and hierarchical text understanding. Our resulting model is MosaiCLIP. Our approach leads to significant improvements across compositionality benchmarks. For example, Figure 1 b) and c) shows that MosaiCLIP improves performance by 11.5% and 9.1% CREPE and ARO dataset a strong baseline and by > 20% over CLIP. Our contributions encompass:
Related Work
Contrastive Vision-L.nnguage Pre-training: Large-scale contrastive learning for Vision and Language is utilized to create models like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021). These models showease impressive performance on variety of tasks, including image classification, text and image retrieval, image captioning (Mokady al., 2021), object detection (Zhong al., 2022; Li al., 2022c) etc.
Visio-Linguistic Compositionalliy: Various studies have introduced benchmarks for assess- ing the compositional reasoning abilities of vision-language foundation models (VLMs). For instance, Winoground (Thrush et al., 2022) is handpicked collection of 400 test cases, each comprising two images and two sentences, Sentences have the same word content and differ word-order. Diwan et al. (2022) show that the Winoground dataset tests additional challenges along with compositionality, including handling ambiguous image-text pairs and unusual examples. Yuksekgonul et al. (2022) proposed the ARO benchmark for probing VLMs ability understand Attribute, Relations, and Word-Order.
Ma al. (2022) proposed CREPE for measuring two aspects of compositionality: systematic generalization and productivity. All benchmarks suggest that contrastively trained VLMs have severe difficulty in compositional reasoning. As remedy, NegCLIP (Yuksekgonul et al., 2022) and Teaching SVLC (Doveh et al., 2023) create targeted rule-based and LLM-guided hard negative sentences, SyC -Canclnte-Bonilla et al., 2023) fine-tunes CLIP with million scale synthetic images-text pairs, for improving relational and attribute understanding. We observe that previous methods are either highly dependent on how clean the training data is, use expensive LLM's for data augmentation or use synthetic datasets that require special solutions to resolve the synthetic-to-reat domain gap. We hence develop coarse-to-fine contrastive learning framework that matches images with texts of multiple complexities, which serves as a purpose solution to improve ine-grained and hierarchical text understanding,
A novel graph-based text decomposition and augmentation framework and a coarse-to-Pie toneiie contrastive learning objective for matching im- ages to text sub-graphs of varying complexity.
-egative mining techniques using graph transformations of the text scene graphs, that are seamlessly coupled with our text decom- position strategy, and applied over any text.
A thorough analysis for understanding why MosaiCLIP improves vision-language com- positionality, disentangling the effect of im- age and text encoders and providing novel tree-score based analysis showing that Mo- SaiCLIP exhibits improved hierarchical text understanding.
Extensive experiments over three model ar- chitectures, two pre-training datasets, three

---

thereby improving compositionality.
and all graphs are first converted to text via tem- plates, converted embeddings using transformer- based (text) encoders, and matched to image em- beddings.
Scene Graphs are structured representations of visual scenes, consisting of objects, their attributes, and relationships between objects. Scene graphs are beneficial for a range of tasks including image retrieval (Wu et al., 2019; Johnson et al., 2015), image captioning (Yang et al., 2019), and image generation (Johnson et al., 2018) among others.
3.3 Scene Graph Guided Text Decomposition
Scene graphs are succinet representations of im- ages. However, an image scene graph generator used for generating a scene graph for any given input image is expensive to train since it requires supervised scene graph annotations for training (Li et al., 2017; Xu et al., 2017; Zhang et al., 2019), and also leads to issues like low coverage biased generations against the long tail nature of objects and relationship annotations. We instead use the text scene graph created using an off-the-shelff text scene graph parser' (Wu et al., 2019). This serves as a proxy for the scene graph of (part of) the im- age and is assumed to be a sub-graph of the image scene graph, also depicted by Figure 2.
3 Methodology
3.1 Overview
Here we present the key high-level ideas of our approach. We first present a graph-centric view of the standard image-text matching objective in CLIP, which serves as a motivation to develop our approach (Sec. 3.2). We create scene graphs de- rived from the text, decompose them into multiple sub-graphs (Sec. 3.3) and apply augmentations on these sub-graphs create negative sub-graphs (Sec. 3.4) which are used as hard negatives in a batch. Sec. 3.5 formally defines the Maage-to-Multi--- Text and Text-to- Image losses used for a batch of V-L inputs which is key for learning from multiple pos- itive and negative texts derived from sub-graphs. Matching images with coarse-to-fine sub-graphs results in improved fine-grained and hierarchical understanding of text. Sec. 3.6 provides two- stage curriculum learning strategy for improved fine-tuning performance.
Let the text scene graph obtained be GT (Vr, Er), where Vr represent the nodes of the graph, which are either objects or their attributes. ET are the edges of the graph that represent rela- tions between objects. See Fig. 2 for an example of a text scene graph. As shown in the figure, we decompose this scene graph into multiple positive sub- graphs Pg {gi, 92, - , ,gkt. k < M, where M is the max number of decomposed sub- graphs and is hyperparameter. Each sub-gra sub-graph a representation of a part of the image. We then convert sub-graphs to sentences so that they can be easily processed by transformer- coders commonly used to train CLIP. For this, we use a simple template-based approach. For e.g., we create templates of the form "(N1} {R) (N2)" if we need to convert a graph having two nodes (N1, i N₂) and a relation R, into sentence format. Corresponding to each sub- graph, we obtain one positive text for the image, creating a positive text set P. {tz, 12, f3.................... .t))
3.2 Image-Texte Texp-Graph Alignment
(text) en-
Our approach builds on the idea that the standard image-text contrastive learning in CLIP can be viewed as a matching between an image scene graph and its sub-graph. Formally, given an image- text pair (I, the image can be viewed by its scene graph, G, E7), The text scene graph is given by Gr EVr.Er). Then Gr C Gr. According this assumption, during contrastive learning in CLIP, we implicitly bring the repre- sentation of the image scene graph close to one of its sub-graph (the text scene graph). Now, let Sg - [glg C G} represent the set of sub-graphs of graph G. According to the assumption above, g E Sg, > g E Sg,, Hence Vg Sgr, (g. becomes a correct matching pair during contrastive learning. We match multiple sub-graphs of the text scene graph to the same image, while also including hard negative sub-graphs in the batch. Matching between graphs is an implicit concept,
3,4 Negative Sub-Graph Creation
Corresponding to sub-graphs in Pg. we create neg- ative sub-graphs No = [" "g2, "gaa..... Sub- graphs in Ng are a minimally perturbed versions of the positive sub-graphs in Pg. Similar to positive sub-graphs, we convert sub-graphs in Ng to text using the same template-bas based approach, and ob- tain N₁ ("t1, "12, "t,,.... Texts in N, serve

---

Imanee Scare Genph
Ted
Scare Goppl
Aniil and a0lig 01
Scena Goon
dlon and loodng vitha bege Cataee
(a) Text scene graph creation and dccomposition (Sec. 3.3).
Fine-Graired Negative Sub-Grephts
Scare: Craghh:
G1
G2
G3
G4
Gs
Neg Sub- Graphs
Subgrapher
P0
Text: Tigi
TT
12
TT
RT,
PTg
TT
ST,
4T
'T
TT2
balbe
bown
Imagee:
Coonse-Griired Positvve and Negative Sub-Grephs
9p1
(b) Hard negative graph ereation from positive sub-gran sub-graphs (Sec. 3.4).
(c) Coarse-to-fine contrastive learning (Sec. 3.5).
Overview of our approach. a) Depiction of the scene graph of an image (hypothetical) and a scene graph parsed from text. The text scene graph a sub-graph of the image scene graph. The text scene graph is decomposed into sub-graphs from which b) minimally perturhed hard-negative sub-graphs are created. e) The Ground truth similarity matrix used for a batch of data during contrastive learning. Solid boxes represent a match between the image and the corresponding text. Different from CLIP, each image can be matched multiple texts in our method.
Figure 2:
hard negative texts in a given batch, see Fig. 2. We focus on creating negative sub-graphs that improve the attribute binding and relation under- standing capabilities of the model, for which we use the following strategies for negative graph cre- ation: We first consider an external set of objects (N), attributes (44, and relations (R).
3.5 Coarse-to-Fine Contrastive Learning in Image- Text-Graph Space
Given an image-text batch during training B { t, ] consider separately the batch of im- ages B, { {x;} and a batch of texts By [t The sentences in the text batch are first converted to scene graphs to obtain a batch of scene graphs BG = { followed by decomposi- tion to sub-graphs to obtain the positive sub-graph batch Bg {g. ) m > n. negative sub- graphs are sampled and added to the batch to obtain B, )= We convert these sub- uub-graphs text to obtain the final text batch B, = {t²) (/]=t m+r Consider an image encoder model Se parame- terized by 8, text encoder fø parameterized by Φ. For any image æ, text á fo(a:) is the un- normalized image feature, and (t) is the unnormalized text feature. As common practice, the features are normalized to obtain u = // [|] and v = // ||] The Image-to-Multi trastive loss is given by:
1) Node Swapping and Replacement: We swap nodes in sub-graphs, these can be swaps of nodes which are attributes objects. We also replace nodes with external nodes from A, A based on their type. 2) Edge Replacement: We replace edges with randomly sampled edges from the exter- nal relations set, R. 3) Connecting Sub-graphs: Here we join two sub-graphs. For this, we use one sub-graph from Pg. and another random graph created using nodes and edges sampled from ex- ternal sets NN, A, R. This creates overall hard negative graph. Sub-graphs are joined by sim- ply joining nodes from both graphs through a ran- domly sampled edge from R. These strategies result in minimally perturbed hard negative sub- graphs for improving attribute and relation under- standing. We define multiple graph transforma- tions (fy : G + P(G)} - fret, fatrr; fobj using the above techniques and create hard negative sub- graphs. See Appendix Sec. B for more details regarding negative sub-graph creation.
con-
|5r
exp(Teu Uk) ==1 exp(Tri! vz))
CMC 12t
log
P())
8. Σ
kEP())
11 9x C C.). where P(i) = (kik E [1, [,
The Text-to-Image contrastive loss is only calcu-

---

lated for the positive texts. is given by:
attribute, relation and word-order understanding, SVO (Hendricks and Nematzadeh, 2021) measures verb (relation) understanding. VL-Checklist (Zhao et al., 2022) measures relation, attribute and ob- ject understanding. We use CREPE--rodduccivityy (Ma et al., 2022) for measuring model's ability to productively generalize to more complex and long sentences. Methods for improving compositional- ity should be tested general downstream tasks used to evaluate the quality of learned representa- tions of language and vision. For this, we utilize the popular ELEVATER benchmark (Li et al., 2022a) consisting of 20 datasets and ImageNet (Deng et al., 2009) following prior work (Doveh et al., 2023). Baselines: We compare with all recent techniques used for improving compositionality of CLIP style models including NegCLIP (Yuksekgonul et al., 2022), Teaching SVLC (Doveh et al., 2023) and Syn-CLIP (Cascante-Bonile with CLIP (Radford et al., 2021) as well as CLIP- FT (fine-tuned) on datasets we use. See Appendix Sec. F for more details.
18
p(J) U/) Di=l exp Tu. 03))
exp(Tu
(2i
log
Br
j=1
where gpli) S Gj. B, | B,ee) J. in which Bpee BTT represent the texts in B,, obtained from positive and negative sub-graphs respectively. The overall loss is LMMSSUuL (C + /2
3.6 Curriculum and Robust Fine-tuning
For fine-tuning experiments, we develop two- stage curriculum learning strategy motivated by recent work (Goyal et al., 2022; Wortsman al., 2022: Kumar et al., 2022) that show how fine- tuning can distort pre-trained features and closely mimicking the contrastive pre-training objective while fine-tuning CLIP can help mitigate this prob- lem (Goyal et al., 2022). However, our coarse-to- fine contrastive learning objective naturally devi- ates from pre-training in two ways. a) Existence of hard negative texts in the batch, and b) Hav- ing multiple positive and negative texts for an im- age. This can lead to gap in pre-training vs fine- tuning objective, and a lower than optimal perfor- mance after fine-tuning. To solve this, our two- stage curriculum learning strategy first fine-tunes the model while sampling (at max) single posi- tive and negative sub-gn -ob-graph per image, followed by fine-tuning it with multiple positive and negative sub-graphs. The hardness of data in this curriculum learning setup is defined by the amount of differ- ence the fine-tuning setup has as compared to the pre-training setup. According to this intuition, it is easier for the model to first learn to handle hard negatives in batch and then learn to handle mul- tiple positive and hard negative sentences at once. We see consistent improvements using this strategy compared to a direct one-step fine-tuning, which we term as NosuiiC in our ablations. For better performance on con-compositonal tasks, we use the robust fine-tuning approach (Wortsman et al., 2022) of weight space ensembling of the vision encoder, before and after fine-tuning. This model is called MosaiCLIP
al., 2023) along
Training and Evaluation Details:
Fine-tuning: NegCLIP (Yuksekgonul et al., 2022) was developed by fine-tuning CLIP on the COCO dataset (Lin et al., 2014), however, COCO images might overlap with benchmarks like CREPE and ARO which may lead to confounding of results. Hence we consider 2 additional similar sized fine- tuning datasets randomly sampled from CC-12M (Sharma et al., 2018; Changpinyo et al., 2021) and YFCC-15M (Thomee et al., 2016) and call them CC-FT, YFCC-FT. We also use CC3M (Sharma et al., 2018) for comparing with recent baselines. We fine-tune the commonly used OpenAl CLIP- ViT-B32 model and report results on all datasets, except for CREPE dataset which tests the system- atic generalization for which we used OpenCLIP (Ilharco et al., 2021) models pre-trained on (CC- 12M, YFCC-15M), fine-tune them on (CC-FT, YFCC-FT), and report results on [CC-12M, YFCC- 15M) splits of CREPE. See Appendix E.3 for more information evaluation datasets.
Pre-training: We pre-train MosaiCLIP, NegCLIP and CLIP on two prominent large-scale pre-training datasets, CC-12M and YFCC-15M, and use two different backbones (ResNet-50 and Swin-Tiny) following prior work (Yang et al., 2022) and report zero-shot performance on all test datasets. See Appendix H.I for hyperparameters details.
4 Experiments
Evaluation Datasets:
We test MosaiCLIP and baselines on large scale benchmarks that require compositional reasoning: CREPE--Syttematiciyy (Ma et al., 2022) measures systematic generaliza- tion, ARO (Yuksekgonul et al., 2022) measures

---

FincTun. data
COCO
CC-FT
YPCC-FT
Benchmark
ARO
VLC SVO
ARO
CREPE
VLC SVO
ARO
CREPE
VLC SVO
Meta
Method
Rel.
Air.
Ord.
Avg.
Avg.
Rel.
Attr.
Ord.
CU
AU
Avg.
Avg.
Rel.
Aur.
Ord.
CC
AU
Avg.
Avg.
Avg.
Random
50.0
50.0
20.0
50.0
50.00
50.0
50.0
20.0
14.3
20.0
50.0
50.00
50.0
50.0
20.0
14.3
20.0
50.0
50.00
38.35
CLIP
59.8 63.2
53.3
70.8
83.58
59.8
63.2
53.3
45.1
35.0
70.8 83.58
59.8
63.2
53.3
39.8
39.5
70.8
83.58
60.60
CLIP-FT
58.9
65.3
38.4
71.3
90.15
58.1
63.3
42.7
45.8
35.6
70.1
88.56
51.4
63.1
25.3
36.4
38.3
68.9
85.27
57.73
NegCLIP
72.7
75.6
71.5
88.36
63.1
52.1
39.0
70.4
67.57
MosaiCLIP
78.0
87.1 81.4
90.67
80.4
72.4
88.73
66.9
84.4
41.5
75.1
85.36
74.29
Table 1: Fine-tuning results the ARO, CREPE Systematicity, VL-Checklist (VLC) and SVO benchmark (total 10 datasets). Abbreviations Rel.:= VG-Relation, Attr:: VG-Atribution, Ard:average of ARO-Flickr and ARO-COCO Order results, CU: HN-CCompUnssen, AU: HN-Atom-Unseen. See Sec. for more details.
Finc'Tun, data +
CC3M
Scowe Ditteweee da
Cabedha
Benchmark
VL-Checklist
ARO
8
rencuLS
Kuizas
CHAALOO
Method
Obj.
Attr.
Rel.
Rel.
Attr.
Ord.
Grsns
CLIP
81.6
67.6
63.1
59.9
63.6
53.3
CLIP-FT
79.0
64.7
54.3
41.7
59.3
25.2
Syn-CLIptI
70.4
69.4
71.4
66.9
65.1
Figure 3: MosaiCLIP's average score difference with
Teaching SVLC!2
85.0
72.0
69.0
NegCLIP on 20 datasets from ELEVATER benchmark.
MosaiCLIP
86.4
73.7 71.9
83.7
78.0
79.4
significant gain in ability to systematically generalize unseen combinations of concepts. Across pre-training settings, MosaiCLIP improves over NegCLIP by up to 42.5%, 4.9% when evaluated against HN-Comp (CU), HN-Atom (AU) hard negatives respectively. Significant improvements are observed in attribute and relation understanding, giving gains of up to 8.3%, 12.0% respectively across pretraining settings. We also note that order understanding of MosaiCLIP is worse than that of NegCLIP for the CC-12M pre-training dataset, while better than NegCLIP for the YFCC-15M dataset. Notably, there is a large variance in NegCLIP's performance across pre-training datasets as seen in Table 3, and it also performs poorly when the pre-training dataset has higher noise (e.g. YFCC-15M), MosaiCLIP is fairly consistent and more robust to the change in the pre-training dataset. In Appendix C.5 we find that MosaiCLIP can provide improvements over NegCLIP while using as low as 0.3x of the total pre-tnainn or fine-tuning data.
Basscante-Bonille al., 202v)c al., 2023)
Table 2: MosaiCLIP vs contemporary works that use Isynthetic data or LL..'ss. See Appx. D. for details.
4.1 Results
In this section we provide experimental results in both pre-training and fine-tuning settings to show the efficacy of our approach. These are as follows:
Fine-tuning: Main fine-tuning results are shown in Table 1 and 2, where we fine-tune CLIP models using our method and compare to baselines. Notably, we see that the general- ization performance on unseen compounds and atoms as measured by the CREPE dataset is up to 18% higher than NegCLIP. Additionally MosaiCLIP shows upto 16.5%, 5.3%, 32.3% of improvement over NegCLIP in understanding relations, attributes and word order respectively. MosaiCLIP also shows consistent improvements in the verb understanding task as measured by the SVO dataset. Additional Comparisons: We also compare with latest contemporary works in Table 2 and Appendix Sec. D. We find significant improvements (upto 14% on ARO) over models that use LLMs or synthetic data for making CLIP more compositonal.
Results on classification and retrieval: On average, MosaiCLIP achieves +3.3%, +6.3% 6.3% better performance the ELEVATER classifi- cation benchmark compared to NegCLIP and CLIP while pre-training and maintains similar accuracy as CLIP while fine-tuning. We also try using our method along with the robust fine-tuning technique (WiSE-FT) so that performance degra-
Pre-training: Table shows pre-training results over all benchmarks. CREPE results show

---

Pre-training data +
CC-12M
YFCC-13M
Arch.
Benchmark +
ARO
CREPE
VLC
SVO
ARO
CREPE
VLC
SVO
Meta
Method +
Rel.
Attr.
Ord.
CU
AU
Avg.
Avg.
Rel.
Attr.
Ord.
CU
AU
Avg.
Avg.
Avg.
Random
50.0
50.0
20.0
14.3
20.0
50.0
50.00
50.0
50.0
20.0
14.3
20.0
50.0
50.00
36.33
CLIP
51.0
56.6
25.5
44.1
37.3
65.6
82.21
53.8
56.2
18.4
39.6
41.7
66.2
76.27
51.03
Swin-1
NegCLIP
82.4
66.8
59.7
80.3
39.6
70.0
82.04
73.6
58.9
35.5
47.1
41.5
66.0
76.10
62.82
MosaiCLIP
84.3
76.8
55.5
92.1
44.5
72.4
85.62
74.7
66.1
35.8
89.6
45.3
71.8
77.87
69.46
CLIP
52.9
59.7
22.6
42.9
36.7
66.2
82.13
57.8
55.1
18.3
38.9
38.9
64.8
75.60
50.90
RN-50
NegCLIP
80.5
66.5
60.5
82.0
41.4
69.5
82.03
68.0
58.5
37.1
67.2
41.5
66.1
75.18
64.00
MosaiCLIP
82.0
78.5
55.4
92.6
44.4
72.6
83.86
76.3
68.9
38.2
90.2
45.0
72.3
77.42
69.83
Table Pre-training results on all compositionality benchmarks (4 benchmarks, datasets) over four expt. settings for abbreviations and Sec. 4.1 for more details.
(two pre-training datasets, two backbones). See Table
0.33
0.43
0.10
MUSOFLL
MoDNCCLP
20.35
3.1
0.33
0.5
0.22
0.25
0.1
0.40
CC 12M Vncc-1MM
CC 12-32 VECC-5M
Roth
Auh
Compastly
SwinTTry Tiny
RR-50
Encturid Encoder
Aretuned Excode Excodar
(a) CREPE- Productiiyy
(b) Tree Scores
(c) A-O-Relltion
(d) ARO-Atributee
Figure 4: a) Results on CREPE-Produuctiviy dataset b) Tree-score comparison MosaiCLIP with NegCLIP: MosaiCLIP shows improved hierarchical understanding of language. e) and d) Selectively fine-tuning of image, text encoders and measure performance on different datasets. Also see similar results for SVO in Figure 10.
Model
COCO
Flickr30K
AVG.
aspect of compositionality. Although we do not explicitly train our models for generalization to longer sentences, the improved hierarchical language understanding using methods lead to emergent behavior such that MosaiCLIP generalizes better than NegCLIP and CLIP more complex sentences. We can see this effect in Fig. a) and Appendix Fig. 8 and 9. We report the average of retrieval over swap and atom splits and find MosaiCLIP significantly improves over NegCLIP by upto 15% across different text complexities (4-12).
12T
T2I
12T
T2I
CLIP
20.7
13.1
36.2
24.1
23.5
NegCLIP
20.1
12.9
38.6
23.3
23.7
MosaiCLIP
25.9
16.5
44.5
29.5
29.1
Table 4: Comparison of Recall @ I scores of MosaiCIIP with NegCLIP and CLIP. All models are tre-traind on YFCC-15M with swin- Tiny backbone
dation during fine-tuning is minimal, as shown in Appendix Table 9, See Fig. 3 for average results ELEVATER over four training settings and Table 4 for results on retrieval benchmarks where we see a +5.4 point improvement over NegCLIP, We use the popular Karpathy splits having a 5K and IK sized test set for obtaining the COCO and Flickr30k retrieval scores respectively. Hence MosaiCLIP's training strategy improves or maintains the quality of learned representations while improving compositonality. Figures 11-14 show detailed results ELEVATER.
Application to more advanced VLMs: While our focus in this work has been on CLIP style, dual encoder models due to their various benefits, we believe our methods are model agnostic and aimed at improving contrastive learning through our coarse- to to-fine learning framework and negative mining techniques. In this section we test our model advanced VLM, BLIP. We modified BLIP's original image-text contrastive learning objective and create two variants, one called B where we use NegCLIP style hard negatives and the other BLIP+MosaiCLIP which uses our methods of scene graph guided
Productivity: As defined by Ma al. (2022), a productive VL model can handle arbitrarily long and complex sentences and is an important

---

text decomposition and negative sub-graph creation. We fine-tune BLIP model taken from the official BLIP repository and use the "BLIP w/ ViT-B and CapFilt-L model (pre-trained on 129M examples)" as our base model. Results for fine-tuning experiment using COCO dataset is shown in Table 5, We use the hyperparameters used by the official codebase (for the task of fine-tuning on COCO dataset for image-text retrieval). For each setting, we report performance of four models, namely BLIP (before fine-tuned version), BLIP-FT (vanilla fine-tuned version), BLIP+NegCLIP, BLIP+MosaiCLIP. The model are evaluated on the ARO dataset to measure attribute. relation and word-order understanding, using the evaluation scripts provided by the authors of the dataset (Yuksekgonul et al., 2022). We find that
models. See Fig. 4 c.d for the results. Notably, we find that 1) Language encoder has significant scope for improvement over NegCLIP's language encoder, and MosaiCLIP able to successfully exploit this potential and deliver an enhanced compositional understanding of language, which is evident by performance increase of +3.77 over NegCLIP when only the language en- coder is fine-tuned, 2) Improvements brought by MosaiCLIP over NegCLIP in the text encoder are always higher than improvements the image encoder. This is evident from Fig. 4 c,d where the performance increase over NegCLIP when only the language encoder is fine-tuned is always higher as compared to when only the image encoder is fine-tuned; for example, 3.7% 0.0%, 6.9% 1.8% for ARO-Relation, ARO-Auuibution. 3) MosaiCLIP brings significant improvements on the image encoder side (higher than NegCLIP) without using any image negative mining, unlike NegCLIP.
shown in Fig. 4 c,d.
Model
Rel
Attr
Ord
Avg
BLIP
53.5
91.0
53.5
66.0
BLIP-FT
58.9
88.4
58.9
68.7
BLIP+NegCLIP
63.6
90.7
63.6
72.6
MosaiCLIP improves hierarchical text under- standing: For further understanding MosaiCLIP's improved compositional understanding. we provide a novel analysis by considering the recently proposed Tree-Score (Murty et al., 2022) that measures the degree to which transformer (text) encoder processes text in hierarchical manner. We hypothesize that having tree-like hierarchical computation over language can be leading factor for explaining the compositionality (or lack thereof) of CLIP-like models. Along with this, we have previously shown that the language encoder has the most prominent effect in improving compositionality in the case of MosaiCLIP These two reasons motivate the use of tree-score to compare the language encoder's hierarchical understanding capability. Fig. 4 a) shows that MosaiCLIP's language encoder has higher tree-scores than NegCLIP's language encoder, suggesting that MosaiCLIP performs more tree-like computations. This explains the im- proved language compositionality of MosaiCLIP since hierarchical tree-structured computation allows the language encoder to better understand input text compositionally. thereby improving vision-language compositionallity This is in line with the hypothesis that human's semantic understanding of sentences involves a hierarchical (tree-structured) computation which has significant evidence (Crain and Nakayama, 1987; Hale et al.,
MosaiLL
69.9
91.1
69.9
77.0
Table Comparison of BLIP (Li al., 2022b) and fine- tuned version of BLIP with BLIP models that have in- tegrated NegCLIP and MosaiCLIP methodology while training. Fine-tuning has been performed on COCO.
compared to vanilla fine-tuning, both NegCLIP and MosaiCIIIP methodologies bring improvements to relation and word order understanding, while maintaining or improving performance on attribute understanding. The MosaiCLIP methodology significantly improves relational reasoning perfor- mance and word-order understanding compared to the NegCLII methodology, up 6.3%. Attribute understanding performance remains nearly the same as the baseline BLIP performance, with the MosaiCLIP methodology bringing in slight gains over NegCLIP'ss methodology. On average Mo- saiCLIP's methodology brings more improvements to BLIP than NegCLIP or vanilla fine-tuning.
4.2 Analysis
We provide a detailed analysis of our models and baselines, across different dimensions follows:
Disentangling MosaiCLIP improvements: We quantify the relative importance of the vision and language side by freezing the language and vision encoder individually while fine-tuning all

---

the brown table and the black cat black table and
the striped vase and the white table the white striped CLIP
the wet sand and the black hird the black the bird
CLIP
NegCLIP McssiCLIP
NegCLIP
McssiCLIP
CLIP
NegCLIP MMssiCLIP
giraffe
the broccoli
holding eandwich
the trees is in front of the giraffe
the dish is on the broccoll
the sandwich holding the box
CLIP
NegCLIP
MossiCLIP
CLIP
NegCLIP
MosaiCLIP
CLIP
NegOUP
MosaiCLIP
Figure 5: Qualitative analysis on ARO dataset ARo-:trO-t Bottom: ARO-Relation). Models highlighted in blue match the image to the correct sentence (in green) while the models in white match the image to the incorrect sentence (in red). Here, models are taken from fine-tuning experiments COCO from Table 1.
2018; Pallier et al., 2011) and this leads to their compositional generalization capability.
of sub- grab-graphs are presented in Appendix Sec. C.
Benchmark +
ARO
CREPE
VLC
SVO
Meta
MosaiCLIP is Robust: Noisy texts often have meaningful sub-texts which can be exploted by MosaiCLIP, hence MosaiCLIP often achieves consistent performance increase regardless of noise in the pre-training fine-tuning dataset. For example, NegCLIP achieves significantly low performance on ARO when fine-tuned with YFCC-FT (having more noise in text) as compared CC-FT or COCO as shown in Table 1. NegCLIP takes a > 10% hit in performance across various ARO datasets when the fine-tuning dataset is changed from COCO to YFCC, whereas, Mo- saiCLIP achieves similar performance using both datasets. Appendix Sec. D.3 shows that pre-trained MosaiCLIP robust to natural distributon shifts.
Method
Rel.
Ord.
CU
AU
Avg.
Avg.
Avg.
MosaiCLIP
80.4 69.8 85.5
72.4 40.9 77.6 88.73
73.6
MosaiCLUPsaccree 79.0 80.6
40.2
77.7
72.4
MosacCLIPPsst..rr 78.8
69.4
82.6
67.5
41.2
76.4
88.08
72.0
Table 6: Effect of Curriculum learning and Robust Fine- tuning (MosaiCLIPwsser using CC-FT data.
Fine-tuning data Method
COCO
CC-FT
YFCC-FT
Rel.
Attr.
Rel.
Attr.
Rel.
Attr.
MosaiCT.IP
82.6
78.0
80.4
69.8
74.3
66.9
without fret without factr
81.7
76.6
78.8
68.7
73.5
66.2
77.7
73.2
70.5
68.2
69.0
65.9
without frets Jaser
79.0
70.4
68.8
64.9
57.4
63.6
Table 7: Effect of different sesitivve-negtivee sub-graph types sampled while training. Results are presented on the ARO benchmark.
Qualitative Analysis: We take MosaiCLIP, NegCLIP and CLIP fine-tuned on COCO and filter out examples from the ARO dataset where MosaiCLIP and NegCLIP's disagree. Some notable examples in Fig. 5 include cases where NegCLIP and CLIP often struggle to understand simple concepts like understanding the color of the cat and table (top-left Fig. 5 or understanding the "is holding" relation b/w sandwich and the box in bottom-right Fig. 5.
Conclusion
We present a method to improve the compositional reasoning capabilities of contrastively trained large vision-language models. In particular, we provide a coarse-to-fine contrastive learning framework and scene graph-based text decomposition strategy for matching subgraphs of the text scene graph having varying complexity an image during con- trastive learning. We also develop hard negative graph creation strategies focused on improving at- tribute binding and relation understanding capabili- ties. Our techniques leads to significant improve- ments in compositional reasoning capabilities. We investigate the reasons for improved composition- ality and present novel finding based on language encoder tree-scores, suggesting that our models learn improved fine-grained and hierarchical text understanding, which is likely the key reason for improved vision and language compositionality of MosaiCLIP as compared to baselines.
4.3 Ablations
Table 6 and Appendix Tables 8,9 show the effect of curriculum learning and robust fine-tume meee-tnnniig where we find that curriculum learning can bring consis- tent improvements of up to 1.2% on average and robust-finetuning (WiSE-FT) technique performs the best zero-shot tasks (i.c. minimal forget- ting while fine-tuning), while still improving over NegCLIP by about 5% on compositional reasoning tasks. Table 7 shows the effects of different kinds of sub-graphs sampled during training. More de- tails including the effect of sampling larger number

---

6 Limitations
scene-graph as input and requiring it to generate sentence. This approach might be effective but may also lead to higher computational cost while training.
Computational Cost: Although MosaiCLIP leads to significant performance increase on several benchmarks that test compositional reasoining, it requires higher per-batch computational cost while training. For this we give a detailed analysis on the computational cost in Appendix C.6 and show that simply providing more compute to prior methods in the form of larger batch sizes does not improve compositional reasoning. We also show ways tackle this computational cost, by using less data in Appendix C.5, since MosaiCLIP is data efficient and can provide improvements over baselines with as low as 0.3x of the total data. This along with our ablations in Appendix C.I gives some control to any practitioner to vary either the training dataset size the number of sub sub-graphs in our method, and obtain clean tradeoff between accuracy and compute. As future work we would like develop a coarse-to-fine grained objective requiring minimal extra computation cost per batch. Future work should also look at decreasing the extra computational cost incurred by contemporary methods like Syn-CLIP (Cascante-Bonill et al., 2023) and Teaching SVLC (Doveh et al., 2023).
A Acknowledgements
We thank anonymous reviewers for their insightful suggestions that helped in greatly improving our paper. We also thank Aditi Khandelwal, Animesh Sinha, Abhishek Kadian for their helpful comments and suggestions on this work.
References
Paola Cascante-Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Güll Varol, Aude Oliva, Vicente Ordonez, Rogerio Feris, and Leonid Karlinsky. 2023. Going beyond nouns with vision & language models using synthetic data.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12M: Pushing web- scale image-text pre-training recognize long-tail visual concepts. CVPR.
Stephen Crain and Mineharu Nakayama, 1987. Struc- ture dependence in grammar formation. Language, 63(3):522 543.
Other Vision Language Models: In our current work we primarily aim to improve the compo- sitionality of CLIP-Style, dual-tower models trained using large scale contrastive learning, since they severely lacked compostional reasoning capabilities as shown by (Yuksekgonul et al., 2022). Many other VLMs exist such as those that undergo cross modal interactions between vision and language such as BLIP (Li al., 2022b), X-VLM (Zeng et al., 2021), LXMERT (Tan and Bansal, 2019). Although our methods show promise in improving more advanced VLMs like BLIP as shown in Section 4 and Table 5, more thorough analysis will be beneficial to study the extent to which methods improve vision-language contrastive learning for these models.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 258.255.222 leee
Anuj Diwan, Layne Berry. Eunsol Choi, David Harwath, and Kyle Mahowald. 2022. Why is winoground hard? investigating failures in visuolinguistic compo- sitionality. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 2236-2250, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Sivan Doveh, Assaf Arbelle, Sivan Harary, Bli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. 2023. Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2657-2667 6667-2668.
For simplicity, we currently use manually curated templates to convert sub- graphs to sentences, however, this can lead to similar looking and synthetic sentences. Large language models like GPT-4 (OpenAI, 2023), BLOOM (Mitchell al., May 2021-May 2022) should be looked into for developing sentences from scene graphs, by directly giving the LLM a
Sentence Templates:
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer. Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang. Michael Lingelbach. Aidan Curtis, Kevin Tyler Feige- lis, Daniel Bear, Dan Gutfreund, David Daniel Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenen- baum, Josh Medermott, and Daniel LK Yamins. 2021. ThreeDWorld: A platform for interactive

---

mulli--mdaa physical simulation. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out- of-distribution. arXiv preprint arXiv:2202. 200.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. 2022. Finetune like you pretrain: Improved finetuning of zero-shot vision models.
Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, et al. 2022a. El- evater: A benchmark and toolkit for evaluating visual models. Advances in Neural Information Processing Systems, 35:9287- 9301.
John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan Brenman. 2018. Finding syntax in human encephalography with beam search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b. Blip: Bootstrapping language-image pre-training for unified risionllanguage understand- ing and generation. In International Conference on Machine Learning, pages I 12888-1228 -2900. PMLR.
Melbourne, Australia. Association for Computational Linguistics.
Lisa Anne Hendrieks and Aida Nematzadeh. 2021. Probing mmage-language transformers for verb un- derstanding. Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 3635-3644, Online. Association for Computational Linguisties.
Liunian Harold Li. Pengchuan Zhang, Haotian Zhang. Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang. ct al. 2022c. Grounded language-image pre-training. In Proceedings of the IEFF//VV Conference on Com- puter Vision and Pattern Recognition, pages 10965- 10975.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. 2021a. The many faces of robustness: A critical analysis of out- of-distribution generalization. ICCV.
Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang. and Xiaogang Wang. 2017. Scene graph generation from objects. phrases and region captions. In Proceedings of the IEÉE international conference on computer vision, pages 1261-1270.
Dan Hendrycks, Kevin Zhao. Steven Basart, Jacob Stein- hardt, and Dawn Song. 202 Natural adversarial examples. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 15262-15271.
Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang. Yue Xu, Mingyang Chen, 7e Ma. Shiyi Wang, Hao-Shu Fang, and Cewu Lu. 2019. Hake: Hu- man activity knowledge engine. arXiv preprint urXiv:190400653
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Lud- wig Schmidt. 2021. OpenCLIP.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects context. Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham. Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916, PMLR.
Zixian Ma. Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. 2022. Crepe: Can lasion-language foundation models reason com- positionally? arxiv preprint ar Xxxv:2212.007996
Justin Johnson, Agrim Gupta, and Li Fei-Fei. 2018. Im- age generation from scene graphs. In P Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1219-1228.
Margaret Mitchell, Giada Pistilli, Yacine Jernite, Ez- inwanne Ozoani, Marissa Gerchick. Nazneen Ra- jani, Sasha Luccioni, Irene Solaiman, Maraim Ma- soud, Somaieh Nikpoor, Muñoz Ferrandis Carlos, Stas Bekman, Christopher Akiki, Danish Contractor, David Lansky, Angelina McMillan-Major Tristan Thrush, Suzana Iliè, Gérard Dupont, Shayne Long- pre, Manan Dey, Stella Biderman, Douwe Kiela, Emi Baylor, Teven Le Scao, Aaron Gokaslan, Julien Lau- nay, and Niklas Muennighoff. May 2021-May 2022. Bigscience, bigscience language spen-science open- access multilingual (bloom) language model. Inter- national.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei- Fei. 2015, Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668-3678.
Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations.

---

Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:211 NE1111109774
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for visio-iinguistcc compositionality. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 5238- 5248.
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. 2022. Characterizing intrin- sic compositionality in transformers with tree projec- tions. arXiv preprint arXiv:2211.01228.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. 2019. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32.
OpenAI. 2023. Gpt-4 technical report
Christophe Pallier, Come-Domin Devauchelle, and Stanislas Dehaene. 2011 I. Cortical representation of the constituent structure of sentences. Proceedings of the National Academy of Sciences, 108(6):2522- 2527.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. 2022. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7959-7971.
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen. Quan Tran, and Abhinav Shrivastava. 2021. Learning to predict visual attributes the wild. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Puttern Recognition (CVPR), pages 13018-13028.
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang. Lei Li, Weiwei Sun, and Wei-Ying Ma. 2019. Uni- fied visual-semantic embeddings: Bridging vision and language with structured meaning representa- tions. In Proceedings of the IEFE/CVT Conference on Computer Vision and Pattern Recognition, pages 6609 -618.
Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. 2020. Grounded situa- tion recognition. In Computer VisionEECCV 2020: 16th Earopean Conference, Glasgow, UK, August 23- 28, 2020, Proceedings. Part 16, pages 314-332. Springer.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages PMLR.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei- Fei. 2017. Scene graph generation by iterative mes- sage passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410-5419.
Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. 2022. Uni- fied contrastive learning in image-text-label space. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 19163- 19173.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do imagenet classifiers generalize to imagenet? In International conference PMLR.
on machine learning, pages
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypemymeed image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2556-2565, Melhourne, Australia, Association for Computational Linguistics.
Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pages 10685-10694.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP)) pages 5100-5111 111, Hong Kong, China. Association for Com- putational Linguistics.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022. When and why vasion-language models behave like bag-of-words models, and what to do about it? arXiv preprint a1Xiv:2210.0193..
Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. 10cm100m: The new data multimedia research. Communications of the ACM, 59(2)64-73.
Yan Zeng, Xinsong Zhang, and Hang Li. 2021. Multi-grained vision language pre-training: Align- ing texts with visual concepts. arXiv preprint 1...v:2111.0826..

---

Ji Zhang, Kevin Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro, 2019. Graphical contrastive losses for scene graph parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pages 11535-11543.
Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei 7hang, Lijuan Wang, Yejin Choi, and Jian- feng Gao. 2021. Vinvi: Revisiting visual representa- tions in vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5579-5588.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. 2022. Vl-checklist: Evaluating pre-trained vision- language models with objects, attributes and rela- tions.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chun yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Yuan, Yin Li, al. 2022. Regionclip: Region-based language-iman pretrain- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16803.

---

Appendix
objects in them, and these can have multiple tributes for the objects. We develop and use scene graph transformations that take a sub-graph as in- put and return (set of) modified versions the graph mninimally-perturheed negative sub-graphs for the image). For this, we define three graph transformations follows:
A Background
Contrastive Language-Image pre-training (Rad- ford et al., 2021) (CLIP) aims to learn general- purpose representations of vision and language us- ing paired image-text data. This is achieved using contrastive learning in the image-text space. In particular consider pre-training dataset of size n, X T,
Sobj G - P(G) takes input a single object scene graph, where the object has attributes 1g- For each attribute, E A.- random attribute a' is sampled uniformly random from A. We finally obtain a set sub- sub-graphs Goby E P(G) where P(.) denotes the power set. Each E Gobj contains one object node connected with an attribute node which is sam- pled from A.
(0,,4) 4-)--1 Here X' and T are the space of images and text, respectively, and Ti, t are images and text the dataset. Also, consider access to image and text encoders, that we represent by so X' Idd and fø T Rd re- spectively, To learn distributed representations for images and text, the following contrastive losses are used:
fret G P(G) takes input sub- aub-graphs having one relation edge and gives output set of sub-graphs Gret E P(G) where each g E Gred has either object nodes shuffled, re- placed by an external object node n' sampled uniformly at random from N,, and/or relation replaced by external relation r' sampled uni- formly at random from R. Along with this, we also join the input positive sub-graph with a random sub-graph created by sampling dom nodes and edges from N, A, R.
B
exp(ru vU) exp(Tuff
L121 =
log
(1)
1
18
exp(Tu3' mz)) Σド exp(Tu? uz))
C
log
(2)
Σ B
Where B represents the batch during one iteration of training. u4 Bi the l2 normalized embed- dings of tú; 6;, where ti fe()); 6; (tt). I the temperature parameter and trainable. The overall loss is Lelip- = (Crz: + 22
Sattr G P(G) takes input sub-graphe graphs having one relation edge and gives output a set of sub-graphs Gattr E P(G) where each g E Gattr has attribute nodes shuffled, and/or replaced by an external attribute node a' sam- pled uniformly at random from A.
B Scene Graph Decomposition
Here we provide additional details for text scene graph decomposition. Denote the text scene graph obtained from the scene graph parser by Gr (Vr, ET), where Vr represent the nodes of the graph, which are either objects their attributes. Er are the edges of the graph that represent rela- tions between objects. Let G denote the set of all possible scene graphs. We first consider an external set of objects (W), attributes (AA), and relations (R) that we use for creating negative sub-graphs. In practice, we create this set from Visual Genome (VG) dataset (Krishna et al., 2016). Following sample total of 1594
forj. faur broadly aims at improving the model's attribute understanding, while Srel broadly tar- gets improved relation understanding. For each positive sub-graph, we sample all possible neg- ative subgraphs using foljs fattr and make positive-negative sub-graph pairs (gpos:: (gnog: }). These pairs can be classified into three categories C = {Conjy Crel; Cattr) according to the transforma- tion that created the negative sub-graphs. We sam- ple sub-graph pairs from these categories according to probabilities p,, i É {1,2, 3} corresponding the three categories respectively, and Σp, 1. These probabilities pendix Section H. for more details. Multiple sub- graph pairs can have common positive or negative sub-graphs, and sampling these pairs would result
Zhang et al. (2021), entities that have 30 instances of them in the VG dataset. The attribute and Relation list contains 524, and 50 unique instances, respectively. Hence [ = 1594, [A] 524, [ = 50. We first sample all possible sub-graphs having one or two
hyperparameters; see Ap-

---

in duplication, hence for each image, we make sure deduplicate sub-graphs so that all sub-grape sub-graphs, and therefore the text made from them are unique for a given image in a batch. After sampling, all sub-graphs are transformed to text using simple templates, as explained in Scction 3.3.
Unseen
70.0
Accuracy
57.5
C Ablations and Model Analysis
1001 0/2
6/1006612)
(8/4) (99) (48) 1500080021
Number of (pos, neg) subgraphs
C.1 Sampling more subgraphs
Figure 7: Effect of increasing the number of positive and negative subgraphs on CREPE Systematicity bench- mark, when fine-tuning MosaiCLIP (Here we use Open- CLIP RN-50 model pre-trained on CC-12M and fine- tune it on CC-FT).
We analyze the effect of increasing the maximum number of sub-graphs sampled for any given image in batch of data during training. See Figures 6 and 7, in which we test the performance on ARO and CREPE benchmarks (averaged over three fine- tuning datasets considered in this work), increase the max positive and negative sub-graphs per image. We find that as we increase both pos- itive and negative sub-graphs for an image, the performance steadily increases up to a point for all datasets, after which the performance can either flatten out, increase, or even decrease in some of the datasets. This is intuitive since larger num- ber of positive and negative sub-graphs per image leads gap w.r.t the pre-training stage as de- scribed in Sec. 3.6. Also, different compositional splits require different reasoning skills, and as we keep sampling positive and negative aub-graphs for an image, it is natural for certain types of positive and negative sub-graphs to be more pronounced, depending on the dataset statistics, and this can have varied effects different datasets.
Sec. B. Results are presented in Table We observe that both fret and fattr as described Appendix Sec. B, are useful for improving relation and attribute understanding (as measured on the ARO benchmark), across fine-tuning datasets.
C.3 Effect of curriculum training
As shown in Table 8, in all fine-tuning results, we can see consistent improvements when using our curriculum learning strategy, such as upto 2% on systematic generalization, and sometimes more than 6% as seen for ARO-Order results when the fine-tuning dataset is YFCC-FT.
C.4 Effeet of robust fine-tuning
Relation
Attritation
Among many other techniques developed for mit- igating forgetting in large models when they are fine-tuned, one prominent one is robust fin-tuning- WiSE-FT, (Wortsman et al., 2022). Following Wortsman et al. (2022) we perform weight-space ensembling the image encoder before and after fine-tuning using our method and call this model DoosiCLIPPwsF-FF The results on compositional- ity benchmarks can be seen in Table 8 while re- sults on 21 multimodal tasks from ELEVATER and ImageNet can be seen in Table 9. We find that MosaiCLIPPiss-FT has a slight performance de- crease on some compositonal benchmarks as com- pared to MosaiCLIP, however, it is significantly better than NegCLIP on most benchmarks. The real benefit of using MosaiCLIPwisEFFT is that it leads to least forgetting, and there is little to no performance degradation on 21 tasks as showin Table 9.
71.3
Accuuracy
03.1
14
00.3
(171 (12) 1994 1900 (60)) 10e (1
(1/1) (19): 174m (Moo evR) (570088771
Number (pos, neg) subgraphs
Figure 6: Effect of increasing the number of positive and negative subgraphs on ARO benchmark when fine- tuning MosaiCLIP. Results averaged over fine- tuning datasets considered in this work
C.2 Effect of different sub-graph types
Here we analyze the effect of sampling different kinds of sub-graphs from the original scene graph the text. In particular, we measure the effect of graph transformations that we define in Appendix

---

Fun:Tun. data
COCO
CC-FT
YFCC-FT
Benchmark
ARO
VLC
SVO
ARO
CREPE
VLC SVO
ARO
CREPE
VLC
SVO
Meta
Method 4
Rel.
Anz.
Ord.
Avg.
Avg.
Rel.
Attr
Ord.
CU
AU
Avg.
Avg.
Rel.
Attr.
Ord.
CU
AU
Avg.
Avg.
Avg.
CLIP
99.8 63.2
53.3
70.8
83.58
59.8
63.2
53.3
45.1
35.0
70.8
83.58
59.8
63.2
53.3
39.8
39.5
70.8
83.58
60.60
NegCLIP
81.7 72.7
85.7
75.6
90.20
71.5
65.4
84.5
53.1
37.5
72.4
88.36
57.8
63.1
52.1
38.8
39.0
70.4 83,90
67.57
MosaiCLIP
82.6
78.0
87.1
81.4
90.67
80.4
69.8
85.5
72,4
40.9
77.6
88.73
74.3
66.9
84.4
48.8
41.5
75.1
85.36
74.29
MosaiCLLINsCuniis
71.1
MosaiCLIP WISE.FT
82.5 76.2 86.6
80.3
89.65
78.8
69.4
82.6
67.5
41.2
76.4
88.08
69.4
67.0
79.4
48.1
43.6
74.2
83.71
72.88
Table Ablating the effect Curriculum learning and Robust fine-tuning. MLsaiCLIIP NoCurric refers to the version of our model without any curriculum learning. MosaiCLIP wiSE-rT refers to the version where the image encoder of the final model (after fine-tuning) and before fine-tuning are weight-space ensembled. CLIP and NegCI.IP scores are also shown for reference. See Appendix Sec. C.3.
Method
ZS(21) Compositional Score (Meta Avg.)
Method
Fraction of data
ARO
SVO
Rel.
Attr.
Avg.
CLIP
56.4
60.60
NegCLIP 56.8
67.57
NegCLIP
Full
73.6
58.9
76.10
MosaiCLIPNocee NoCumic
55.8
73.23
0.3x
71.6
60.6
70.82
MosaiCLIPwisde WISELT
56.8
72.88
0.5x
74.3
60.8
74.04
MosaiCLIP 55.7
74.29
MosaiCLIP
0,6x
74.7
63.8
75.76
0.8x
77.0
66.3
77.22
Table 9: Zero Shot accuracy 21 multimodal datasets from ELEVATER and ImageNet. Results are average the three fine-tuning datasets. MosaiCLIP has negli- gible drop in performance in general (compared to the gains on compositionality benchmarks), and one can boost performance by using MosaiCLIIPase. FT which has equal performance as compared to NegCLIP on muldimodal datasets. Meta Avg. Compositional Score is taken from Table 8. Second best results underlined. Conclusion: One can use MosaiCLIP for getting the best compositional reasoning capabilities with minimal performance degradation multimodal tasks, and use Pos for no degradation in performance on multimodal tasks, while still perform- ing well on compositional reasoning.
Full
74.7
77.87
Table 10: Data efficiency MosaiCLIP during pre- training. Numbers in blue are lowest numbers that are within 1% or greater than NegCLIP performance. Pre- Training dataset: YFCC-15MM
Method
Fraction of data
ARO
SVO
Rel.
Attr.
Avg.
NegCLIP
Full
71.5
65.4
88.36
0.3x
70.8
67.7
88.70
0.5x
74.5
68.6
88.80
MosaiCLIP
0.6x
75.3
69.3
88.76
0.8x
78.2
69.8
88.98
Full
79.0
69.6
88.91
Table 11: Data efficiency of MosaiCLIP during fine- tuning. Numbers in blue are lowest numbers that are within 1% or greater than NegCLIP performance. Fine- tuning dataset: CC-FT, Curriculum learning has not been used for these experiments.
C.5 Data efficiency
We find that our technique leads significant data efficiency requiring about 0.3x-0.6x fo the total fine-tuning or pre-training data to match or exceed NegCLIP performance. Results are shown in Ta- bles 10 and 11.
CLIP's compositionality, as also done by previous methods Syn-CLIP (Cascante-Bonilla et al., 2023) that generate data using external graphics engines, and Teaching-SVis SVL. (Doveh et al., 2023) which LLMs requiring massive compute even during inference.
C.6
Computational cost
Even though MosaiCLIP uses the same global batch size of image-text pairs, it requires pute compared NegCLIP or CLIP owing the fact that decomposing sub-graph leads a larger effective text-batch size and hence a larger con- trastive learning matrix. is a common practice in literature to trade-off larger compute for improving
Providing NegCLIP with more compute: One can argue that providing more compute to Neg- CLIP can lead to better performance, however, on the contrary we found that NegCLIP's perfoormm performance

---

decreases as batch size scaled (from 256 4096, much beyond MosaiCLIP's text or image batch size), as shown in Table 12.
Table show that our approach is competitve or bet- ter at attribute, relation and object understanding measured by the VL-Checklist benchmark (Zhao et al., 2022). Zero Shot performance on 21 datasets suffers minimally using our approach, and is even better than (Zhao et al., 2022). It is to be noted that both approaches Syn-CLIP (Cascante-Bonil 2023) and Doveh et al. (2023) are orthogonal to our approach and combining them with our coarse- to-fine understanding approach will likely result in much better performance overall, as compared to individual techniques. In particular, Syn-CLIP (Cascante-Bonie
Performanc--ompute Tradeoff: It is to be noted that MosaiCLIP performance continues to increase up to a threshold, as sub-graphs are increased as shown in Table and 6 hence this provides a clean tradeoff between number of sub-graphs and com- pute, and a practitioner can choose the number of -ob-graphs their compute availablility. Along with this, in Appendix Sec. C.5 we showed that we can achieve improved performance compared to NegCLIP with as low as 0.3x data closing the gap between NegC and MosaiCLIP compute even more. It is to be noted that MosaiCLIP is drop in replacement for CLIP after training and requires the same inference cost CLIP
et al..
et al., 2023) faces the issue of having long captions for images, and they aver- age out embeddings of parts of the caption before matching it to the image. This issue can be eaily resolved using our framework which handle multiple positive captions for an image. Per- forming this ablation would be future work for us, once synthetic datasets like that used by Cascante- Bonilla et al. (2023) are open-sourced and gain more popularity. Our approach can similarly also include captions generated from LLMs, as explored by Doveh et al. (2023).
easily
Batch Size (B)
ARO
SVO
Rel.
Attr.
Avg.
512
68.9
65.6
88.68
1024
67.6
65.1
88
2048
65.7
64.2
88.72
4096
62.5
63.7
88.11
Table 12: Performance of NegCLIP with increasing batch size. A batch size of B corresoonds to an effective batch size of 8 in NegCLIP after image and text negative mining. Fine-tuning dataset: CC-FT.
D.2 Standard deviations for fine-tuning results
Here we provide fine-tuning results on the CC-FT dataset with standard deviations over random seeds where OpenAl LLIP-VV-B--22 fine-tuned on CC-FT using MosaiCLIP and baseline tech- niques. See Table 14 for the results. The main paper Table 1 have average results for CC-FT while for COCO and YFCC-FT fine-tuning datasets, the results are for one seed. We do-not run multiple pre-training experiments since they significantly more costly.
D Additional Results and Experiments
D.1 Comparison with recent baselines
We compare with recently published and contem- porary works Besscante-Bonillee et al., 2023; Doveh et al., 2023). Doveh et al. (2023) show that one can create rule-based hard negative sentences and Large Language Models (LLMs) based hard negative sen- tences and use them when training CLIP style mod- els to obtain an improved model that is better at handling tasks that require compositional reason- ing. We fine-tune on CC3M (Sharma et al., 2018) for a fair comparison with Doveh et al. (2023). Re- sults are reported in Table 13. A fair comparison with Syn-CLIP possible since their synthetic dataset is not released. However in Table 13 we find that performance dif- ference large between MosaiCLIP and Syn-CLIP showing that our general coarse-to-fine grained approach is better than using targeted synthetic datasets for inducing compositional understanding in VLMs. Comparisons with Doveh et al. (2023) in
D.3 Robustness to natural distribution shifts
We find that pre-trained MosaiCLIP shows robust- ness to natural distribution shifts as measured by ImageNet natural distribution shifts benchmark. Results are presented in Table 15. We believe that MosaiCLIP sees a larger variety of texts in the form of sub-graphs which can provide it with extra supervision for tackling natural distribution shifts. Intutively, sub-graphs can lead to diversity of texts being seen by the model during training and this might lead to broader coverage of concepts and concept combinations, resulting in improved robustness. Along with this a coarse to fine hi- erarchical understanding of texts and thereby, of
Beaccatte-oonille et al. (2023) is not

---

Benchmark +
VL-Checklist
ARO
ZS(21)
Method
Obj.
Attr.
Rel.
Rel.
Attr.
Ord.
Avg.
CLIP
81.6
67.6
63.1
59.9
63.6
53.3
56.4
CLIP-FT
79.0
64.7
54.3
41.7
59.3
25.2
56.9
Syn-CLIP (Casconte Bonilla et al., 2021)
70.4
69.4
71.4
66.9
65.1
55.3
Teaching SVLC*
85.0
72.0
69.0
54.8
MosaiCLIP MosaiiLLINNCurriic NoCurric
86.4
75.0
69.6
83.2
78.6
77.3
54.9
MosaiCLIPWISEFFTT WISE-FT
86.5
73.6
72.2
82.6
77.0
79.9
55.9
MosaiCLIP
86.4
73.7
71.9
83.7
78.0
79.4
53.5
Table 13: Comparison MosaiCLIP with recently published and contemporary works Syn-CLIP (Cascante-Bonils al., 2023) and Teaching SVLC Doveh (2023). Results are reported on VL-Checklist, ARO and Average Zero Shot results on 21 datasets from ELEVATER and Imagenet. Performance numbers of these models are reported from their respective papers (blank fields (-) are not reported in respective papers). !Uses million-scale synthetic data for fine-tuning. Usse external Large Language Models (L.LMs) like BLOOM (Mitchell et al., May 202 I-May 2022) for text augmentation and hard negative text creation. See Sec. D.1 for more details.
Benchmark
ARO
SVO-Probes
Method +
Rel.
Attr.
Ord.
Obj.
Subj.
Verb.
CLIP-FT
58. 18:063
63. 33:0.8
42.7:10.18
93.1710.11
88 64
83.87:10.03
NegCLIP
71.5.10.40
65.4
84.510.11
92.90.10.09
88.1610.11
84.02.10.02
MosaiCLIP NoCurrie
79.0:10.66
69.6110.19
80.610.17
93.37:10:4
89.7410.13
83.62:10.04
MosaiCLIP
80.4:10.63
19.21
85.5
93.45.10.4
89.39
83.35.10.05
Table 14: Fine- Tuning Results on CC-FT dataset with standard deviations across random seeds. These results correspond the CC-FT fine fine-tuning results in main paper Table 1. Here the base model which is fine-tuned using different techniques is
images should intuitively help in improving per- formance on robustness benchmarks given that the model will now be able recognise details in im- ages and texts more accuractely.
ing results, since it is unclear if the performance increase using any method comes from the fine- tuning dataset (COCO) being close to the domain of test datasets, or if it's the fine-tuning method- ology that leads to an increase in performance. Hence, for rigourous experimentation of the devel- oped methods, one must use other datasets to fine- tune contrastively trained VLMs. We randomly sample similar sized (100k datapoints) from popu- lar pre-training datasets CC-12M and YFCC-15M. and call these smaller datasets CC-FT and YFCC- FT. To train NegCLIP, hard negative sentences and images are required, for which we first use the code released by (Yuksekgonul et al., 2022)² to create hard negatives sentences as well as sample three hard negative images for each image based on OpenAl CLIP ViT-B/32 features, strictly fol- lowing (Yuksekgonul et al., 2022). For compar- ing with contemporary works (Doveh et al., 2023). Mexcante-Bonilly al., 2023) (as shown in Table
E
Dataset Details
Here provide detailes about datasets used for fine-tuning, pre-training and evaluating models in this study. summary is shown Table 16
E.1 Fine-tuning datasets
Following NegCLIP (Yuksekgonul et al., 2022) we use the COCO dataset released by (Yuksekgonul et al., 2022) having 109k samples that had hard neg- ative sentences that (Yuksekgonul et al., 2022) cre- ate for training NegCL.I.. As mentioned in the main paper, COCO dataset images used for creating Visual Genome (Krishna et al., 2016), and this is further used create datasets such as CREPE (Ma et al., 2022), ARO (Yuksekgonul et al., 2022) and a part of VL-Checklist (Zhao et al., 2022). This can lead to confounding and potentially mislead-
htttps: thub. www.:tttbbb.ococooeo wttp:/wttg/////////// vision- vision-Language models -arllll-r--eeee are-bows

---

ImageNet-A
ImageNet-R
ImageNet-S
ImageNet- V2
Arch.
Data
Method
Top1
Top5
Topl
Top5
Top1
Top5
Top1
Top5
CLIP
6.4
24.5
42.6
68.8
22.2
45.5
28.2
54.1
CC-12M
NePCLIP
6.6
25.0
43.1
68.7
22.2
45.4
29.4
55.2
Swin-T
MosaiCLIP
9.1
29.4
48.6
74.3
27.2
52.6
33.6
61.6
CLIP
10,9
34.2
20.6
42.0
6.4
16.7
26.1
49.9
YFCC-15M
NegCLIP
11.4
35.6
20.0
41.7
6.0
16.0
27.2
50.7
MosaiCLIP
14.6
40.2
22.3
44.9
6.8
17.7
32.0
57.2
27.4
41.4
67.8
21.7
29.8
CC-12M
NeICLIP
27.7
41.0
66.9
21.7
30.2
RN-50
MosaiCLIP
11.1
35.6
52.1
76.9
29.5
55.4
37.0
66.5
CLIP
13.4
37.3
17.2
37.2
4.9
13.6
25.8
49.4
YFCC-15M
NegCLIP
12.9
38.0
18.0
37.3
5.1
14.7
26.0
49.0
MosaiCLIP
17.4
46.6
21.0
42.7
6.5
16.9
32.2
57.9
Table Results on ImageNet- Natural Distribution Shifts datasets. MosaiCLIP leads improved robustness to natural distribution shifts. NegCLIP performs similarly CLIP. Models are zero-shot tested ImageNet-A (Hendrycks et al., 2021b), ImageNet-R (Hendrycks et al., 2021a), ImageNet-S(ketch) (Wang et al., 2019) and ImageNet-V2 (Recht al., 2019).
Benchmark/Dataset
#Examples
#Subtasks
Subtask Examples
Datasets Used for Creation
Compositional Reasoning (Evaluation)
ARO
77K
3
Attribute, Relation,
Visual Genome, COCO, Flickr
Order understanding
CREPE-Systematicct
642K
2
Systematic generalization
generalization
Visual Genome
VL-Checklist
410K
3
Attribute, Relation
Visual Genome
Object understanding
HAKE, VAW, SWIG
SVO-Probes
48K
3
Verbs (Relations)
understanding
CREPE-Productivity
183K
9
Productivity
Visual Genome
Fine- Monn-Tning datasets
COCO
109K
CC-FT
100K
YFCC-FT
100K
CC-3M
3.11M
Pre- Sre-Trainig datasets
CC-12M
11.26M
YFCC-15M
14.20M
Citations: ARO AYKKSkkkekgonul et al., 2022), CREPE(Ma et al., 2022), "L-Checklist(Zhao al., 2022),
SVO(Hendricks and Nematzadeh, 2021), Visual (enome((rishm) et al., 2016), COCO(Lin et al., 2014), Plockr(Youugg et al., 2014), HAKE(Li et al., 2019), (Pham et al., 2021), SWiG(Pratt et al., 2020)
Table 16: Details of datasets used in this study for testing compositional reasoning, for fine-tuning and pre-training models. See Appendix Sec. for more details.
2). CC3M (Sharma al., 2018) since it's used by these baselines, and makes direct com- parison possible with them.
E.2 Pre-training datasets
We use popular and standard large scale pre- training datasets CC- CC-1 et al,, 2021)

---

and YFCC-15M (Thomee et al., 2016) for pre- training all models in this study, including CLIP, NegCLIP and MosaiCLIP.
VL-Checklist (Zhao al., 2022): This benchmark created by combining annotations from datasets like Visual Genome (Krishna et al., 2016), SWiG (Pratt et al., 2020), HAKE (Li et al., 2019), VAW (Pham et al., 2021). Each image in the resulting dataset has two captions, a positive and a negative. The positive caption is taken from the source dataset of the image, while the negative caption differs from the positive in only one word which makes it hard negative and helps in testing compositional and fine-grained understanding of VLMs across various dimensions like attributes, relations, and size and locations of objects.
E,3 Evaluation datasets
Here we list the evaluation detailes used in this study and also provide short description for each CREPE-Systematicity (Ma ct al., 2022): CREPE provides systematic generalization datasets to test models trained on popular pre-training datasets including CC-12M and YFCC-15M. While creating CREPE, Ma et al. (2022) make sure split the dataset into and unseen parts, which correspond weather the model has seen not seen the combination of concepts, when pre-trained with popular pre-training datasets. We measure and report performance on both seen and unseen splits in our work.
Baselines:
Here we list the baselines used in this study and also provide a short description for each. CLIP(Radford et al., 2021): Our first baseline is CLIP model released by OpenAl CLIP(Radford et al., 2021) and OpenCLIP (Ilharco et al., 2021). In particular we use the ViT-B/32 model for fine-tuning results Table 1 of the main paper, except for CREPE dataset, which requires using models pre-traoined on specific datasets, for which we use ResNer-50 (RN-50) models pre-trained on 12M and YFCC-15M released by OpenCLIP repository* (Ilharco et al., 2021).
ARO (Yuksekgonul et al., 2022): This benchmark consists of four datasets, including VG-Relation, VG-Attribution, COCO-Order, and Flickr-Order. The first measure attribute and relation understanding of VL models, respectively, and the last two measure the word order understanding of VL models. VG-Relation and VG-Attribution consist of tuples having an image and two texts (one positive and one negative), and the model's task is to match the image with the correct text. order datasets have four negative texts and one positive text for each image, and the task is again match the image with the correct text. SVO-Probes (Hendricks and Nematzadeh, 2021): This dataset consists of tuples having two images and one text. All texts and images have a subject, verb, and object, and the images differ in only one of subject, verb, or object. This dataset helps in understanding if VI. models can compositionally understand combinations of objects having relation between them. The original dataset contains 48K examples.
CLIP-FT: For disentangling the effects of fine-tuning data, and fine-tuning methodology, create CLIP-FT baseline where simply fine-tune the pre-trained CLIP model on the dataset at hand, by using the standard contrastive learning technique used by CLIP.
NegCLIP(Yuksekgonu et al., 2022) [ICLR 2023]: NegCLIP is trained using negative mining of texts and images. Yuksekgonul et al. (2022) create sentence level hard negatives by swapping different linguistic elements. They also additionally include hard-negative images and their corresponding texts in the batch by fetching K nearest neighbours (K=3) for each image in the feature space con- structed using a pretrained CLIP model.
CREPE-Productivity (Ma et al., 2022): Produc- tivity dataset tests the model's ability to generalize to longer and more complex sentences, with complexity ranging from 4 atoms to 12 atorns, where an atom can be an attribute, relation, or object. The CREPE- PRoPut-Prod dataset has number of test sets for each sentence complexity ranging from 4 atoms to 12 atoms.
Teaching SVLC(Doveh et al., 2023) [CVPR 2023]: This method uses LLM's like BLOOM (Mitchell et al., May 2021-May 2022) along with rules to create additional positive and negative sentences for each image while fine-tuning CLIP. Syn-CLiPCCascamte-B Bonilla et al., 2023) [Ariv
'Some image links provided by the the original repository(ht / //gitub.b..////////////.......bt com/ deepmind a probes) were broken. In total, 36k data points were retrievd and used in this study.
'https: thub. com/m foundat i ions/ open_clip

---

2023]: Syn-CLIP uses million scale syn- thetic dataset to fine-tune CLIP and improve it's performance on compositional reasoning tasks. The synthetic data is created using a 3D physics-based simulation platform built on Unity3D, called ThreeDWorid (Gan et al., 2021), 1).
results on both the seen and unseen split, for both hard negative retrieval sets (Comp and Atom) that are used when evaluating performance on CREPE by Ma et al. (2022). Detailed Fine- tine-tuning results on CREPE-Systematicity dataset on both the seen and unseen splits are provided in Table 21. These are an extension to the CREPE-Systematie provided in the main paper Table 1. Detailed Pre- training results for CREPE- Syytem-yyt dataset are provided in Table 22 which are an extension the CREPE-Systematicity results provided the main paper Table
This contemporary work is complementary to our data-centric approach and we believe our methods help fine-tuning with synthetic datasets as well. Cascante-Bonille et al. (2023) in their paper showed how dense and long captions can be obtained for synthetic images and which require splitting into sub-captons followed by averaging of features from all captions while fine-tuning CLIP. This is one avenue where we believe our method can be useful since our method inherently allows matching of images to multiple texts. This is part of future work, once such synthetic datasets are released and are easily available.
results
H Reproducibility
Here we provide necessary details to reproduce our work, that might not have been included in the main paper.
H.1 Training and hy perparameter details
Fine-tuning: For all fine-tuning experiments, we follow Yuksekgonul et al. (2022) for hyperparam- eters. In particular, all models are fine-tuned for 5 epochs, with batch size of 256, using cosine learning rate schedule with 50 steps of warmup and random-crop augmentation during training. used
G Detailed Experimental Results
the main paper Table and Table 3 we had provided concise results for some datasets, based on lack of space due to extensive experimental results. Here we provide detailed results on these datasets:
AdamW W is used for optimization. le as the initial learning rate. Training is performed using 4 NVIDIA A100 GPUs for all models. From the ARO dataset, 10% examples from attribute and relation splits are used as validation examples, and the rest used the test set for all models. On all other datasets, evaluate zero-shot performance. For MosaiCLIP, we find that sampling a maximum of 3 positive and negative sub-graphr per image during fine-tuning gives the best result on the ARO validation set and hence is used in all our exper- iments (including pre-training experiments). For MosaiCLIP, we keep sub-graph sampling proba- bilities as P2 = P3- We vary P1 in {0, 0.08, 0.15) while finettunig dataset. We choose the best model according the ARO val-set and keep the hyperparameters the same for all other fine-tuning datasets.
G.1 VL-CCheklist:: detailed results
Detailed Fine-tuning results on VL-Cheeklist dataset provided in Table 17. These tension the VL-Checklist results provided in the main paper Table 1. Detailed Pre-training results for VL-Checklist dataset are provided in Table 18 which are an extension to the VL-Checklist results provided in the main paper Table 3.
ex-
G.2 SVO-Probes: detailed results
Detailed Fine-tuning results on SVO-Probes dataset are provided in Table 19. These are an ex- tension to the SVO-Probes results provided in the main paper Table 1. Detailed Pre-training results for SVO-Pro Probes dataset are provided in Table 20 which are extension to the SVO-Probes results provided in the main paper Table 3.
the randomly chosen YFCC
Pre-training: For pre-training experiments, we fol- low the training protocol used in Yang et al. (2022); Radford al. (2021). In particular, all models are trained for 32 epochs, with batch size of 4096, ing cosine learning rate schedule with 5000 steps of warmup and random-crop augmentation during training. AdamW is used for optimization. The initial learning rate is le - 3, and weight decay is
G.3 CREPE-Systematiccgs detailed results
Here we provide detailed results on CREPE- Systematicity dataset used for measuring system- atic generalization. In the main paper we had only provided the results related to systematic general- ization (i.e., the unseen split), but here we provide

---

Benchmark +
VL-Checklist
Fine-tuning data
CC-100K
YFCC-100K
COCO
Method
Obj.
Attr.
Rel.
Obj.
Attr.
Rel.
Obj.
Attr.
Rel.
CLIP
81.6
67.6
63.1
81.6
67.6
63.1
81.6
67.6
63.1
CLIP-FT
81.9
69.3
60.9
80.7
68.1
60.2
83.7
66.7
59.3
NegCLIP
82.1
71.4
70.3
81.0
68.1
67.1
85.2
67.2
63.0
MosaiCLIP NoCurric
86.0
77.2
77.7
84.0
72.2
75.1
89.2
70.4
72.6
MosaiCLIP WISE-FT
WiSE-F
85.3
71.4
72.4
83.6
69.5
69.6
88.5
75.5
77.0
MosaiCLIP
86.0
76.8
78.4
84.1
72.1
74.8
89.0
70.1
71.3
Table 17: Fine-tuning results on the VL-Checklist benchmark, for testing compositionality in terms of attribute, relation and object understanding. OpenAl CLIP VIT-B-32 pre-trained model is used as the base model for fine- tuning. See See. G.I for more details.
Benchmark
VL-Checklist
Benchmark +
SVO-Probes
Pre-training data
CC-12M
YFCC-15M
Arch.
Data
Method
Ohj
Subj
Verb
Avg
Arch.
Method
Obj.
Altr.
Rel.
Obj.
Attr.
Rel.
CC12M
CLIP
88 43
82,58
79.33
82,21
NegCLIP
88.38
81.83
79.40
82.04
Swin-T
CLIP
73.6
63.0
Swin-T T
MosaiCLIP
91.89
87.11
82.20
85.62
NegCLIP
75.0
67.7
67.4
71.2
66.5
60.3
MosaiCLIP
80.0
72.9
64.4
79,3
71.3
64.8
STOC-15
CLIP
83.38
77.09
76.27
NegCLIP
84.07
76.87
72.28
76.10
CLJIP
75.5
62.7
60.5
73.2
62.8
58.3
MosaiCLIP
86.20
79.24
73.61
77.87
RN-50
NegCLIP
MosaiCLIP
79.2
73.2
65.3
80.1
71.6
65.1
CC-12
NegCLIP
CLIP
87.58
87.86
82.47
82.54
79.42
79.45
82.03
82.13
RN-50
MosaiCLIP
90.18
85.22
80.48
83.86
Table 18: Pre-training results VL-Checklist bench- mark, for testing compositionality in terms of attribute, relation and object understanding. Results for both back- bones Swin- Swin-Tiny and RN-50 are shown. See Sec. G.1 for more details.
YFOC-15M 13M
CLIP
82.61
76.21
72.27
75.60
NegCLIP
81.40
76.05
72.06
75.18
MosaiCLIP
84.25
79.83
73.29
77.42
Table 20: Detailed Pre-training results the SVO-
Probes dataset. See Sec. G.2 for more details.
Benchmark
SVO-Probes
Method
Obj
Subj
Verb
Avg
CLIP
88.13
83.85
78.76
83.58
set to 0.1. Training is performed using 64 NVIDIA
CLIP-FT
93.17
88.64
83.87
88.56
A100 GPUs. NegCLIP's hard negative text cre-
CC-100K
NegCLII
92.90
88.16
84.02
88.36
ation method often results in no negative text for some texts in the pre-training dataset. Removing all such image-text pairs with possible hard neg- ative text results in poor performance for NegCLIP (due fewer data to pre- on). If we include these image-text pairs, the text batch size might dif- fer for different GPUs since some image-text pairs are without hard negative texts and this causes in- stabilities. We hence keep a cache of sentences from previous batches and add it to the batch negative examples that all GPUs have the same text batch size during training. The same is done for MosaiCLIP since not all images might have the same number of unique positive and negative sub-graphs available. For NegCLIP we create hard negative sentences using code released by (Yuksek-
MosaiCLIP NoCarric
93.37
89.74
83.62
88.91
MosaiCLIP
WISE-FT
92.65
88.69
82.90
88.08
MosaiCLIP
93.45
89.39
83.35
88.73
YFCC-1 POCC-100K 100K
CLIP-FT
89.63
85.83
80.36
85.27
NegCLIP
88.43
84.05
79.21
83.90
MosaiCI. NoCuriic
89.49
85.59
79.83
84.97
MosaiCLIP
WISE.FT
87.86
84,97
78.30
83.71
MosaiCLIP
89,93
86.45
79.71
85.36
CLIP-FT
93.60
91.37
85.48
90.15
COCO
NegCLIP
93.59
91.43
85.58
90.20
MosaiCLIP NoCurric
94.14
92.22
84.23
90.20
MosaiCLIP wISR-FT
93.13
92.00
83.75
89.65
MosaiCI.IP
94.16
93.04
84.82
90.67
Table 19: Detailed Fine-tuning results on the SVO- Probes dataset. See Sec. G.2 for more details.

---

(Pre-training, Fine-tuning)
CC-12M, CC-100K
YSCC-ISM, YFCC-10OK
Retrieval Set
Comp
Atom
Comp
Atom
Method
Split +
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
CLIP
48.3
45.1
39.2
35.0
42.0
39.8
43.4
39.5
CLIP-FT
48.5
45.8
40.0
35.6
39.1
36.4
42.4
38.3
NegCI.IP
55.1
53.1
41.5
37.5
41.9
38.8
42.8
39.0
MosaiCLIP NoCurite
71.4
71.1
45.3
40.2
50.1
46.6
44.9
40.5
MosaiCLIP FISE.FT
68.4
67.5
46.1
41.2
48.9
48.1
46.2
43.6
MosaiCLIP
73.1
72.4
46.2
40.9
52.3
48.8
45.7
41.5
Table 21: Fine- tuning results on the CREPE Systematicity datasets. We take OpenCLIP models pre-trained on CC-12M, and YFCC-I5M, fine-tune them on CC-100K, and YFCC-I00K, respectively, and test them CC-12M, YFCC-15M split of CREPE dataset, respectively. See Sec. 4.1 for more details. We recalculate CLIP results since Ma et al. (2022) do not normalize CLIP embeddings before taking the dot product for text and image embeddings, resulting in an incorrect score.
Pre-training data
CC-12M
YFCC-15M
Retrieval Set
Comp
Atom
Comp
Atom
Arch.
Method &.
Split
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
Swin-T
CLIP
45.9
44.1
41.7
37.3
40.2
39.6
42.9
41.7
NI
76.4
80.3
45.1
39.6
47.3
47.1
43.2
41.5
MosaiCIIIP
85.3
92.1
49.3
44.5
80.7
89.6
48.2
45.3
CLIP
44.9
42.9
40.9
36.7
38.7
38,9
40.6
38,9
RN-50
NegCLIP
78.6
82.0
46.8
41.4
61.5
67.2
41.5
MosaiCLIP
85.3
92.6
47.8
44.4
80.1
90.2
46.6
45.0
Table 22: Pre-training results on CREPE Systematicity datasets. Models are pre- pre-trained using CC-12M and YFCC-15M datasets and tested the corresponding CC- 12M and "FCC-15M, split of the CREPE dataset. Results for both backhones Swin-Tiny and RN-50 are shown. See Sec. 4.1 for more details.
gonul et al., 2022). For MosaiCLIP training, for hard negative text createdusing NegCLIP's swapping technique, fol- lowed by positive and negative subgraphs created using our method. Sub-graph sampling probabili- ties are kept as p2 = P3, P1 = 0.15.
H.3 Computing Infrastructure and Run-Time:
each image, we always
We NVIDIA A100 GPUs for all experi- ments. Pre-training experiments took about 1.5 days per model while using 64 GPUs. Fine-tuning experiments on CC-FT, YFCC-FT and COCO took about 45 mins each and experiments on CC3M took 5 hours per model, while using 4 GPUs.
H.2 Tree-Score details:
Murty al. (2022) devised method calculate the tree-score of a transformer over a given dataset sentences D. This tree-score measures the func- tional tree-structuredness of a given transformer encoder. See Murty et al. (2022) for exact details for the algorithm to calculate the tree-scores. We use the code released by the authors for the pur- pose of calculating tree-scores for CLIP's language encoder. In practice we use 5K sentences from the COCO-validation set as the held ouot test set II) over which we calculate the tree-scores.
H.4 Model Parameters:
We use standrad CLIP models and as part of all models, is transformer language encoder having 12 layers, 8 attention heads and 512 as it's width. For vision encoders 23M trainable parameters and 2. Transformer vi- sion encoders a) Swin-Tiny with patch-size 4 and window size following (Yang al., 2022) and b) ViT-B-32 which has patch size 32, 12 layers and 12 attention heads.
1. ResNet-50 hvaing

---

11.5 Evaluation Metries:
0.50
065
HOCLL
D.40
Strictly following the respective papers and re- leased code", for ARO, VL-Checklist, SVO we use accuracy as the metric as defined by the re- specitve papers. And for CREPE-Productivvy, and CREPP--ystematicity we use Recall@ our metric of evaluation.
0.50
Recaligs
0..5
0.30
0.20
0.15
0.15
10
Complexity
Complexit
(a) Finetuning data: CC- 100k
(b) Finetuning data: COCO
H.6 Summary Statistics of results:
We provide standard deviation results using 3 ran- dom seeds Appendix Section D.2 for Fine-tuning experiments on the CC-FT dataset. For all other datascts, including the expensive pre-training runs we use single seed for our experiments.
Figure 8: Fine-tuninig Results CREPE Productivity (generalization to longer and more complex sentences). Fine-tuning datasets mentioned below each figure.
0.35
NeOLLP
for Sor
0.30
Rocalli
0.35
0.90
0.25
0.25
0.15
9 11
7
10 12
Complexity
Complexity
LSB (a) Swin-Tiny, CC-12M
(b) Swin-Tiin YFCC-15M
0.35
0.40
Boc8
0.30
0.15
B 9 10 11 Complexity
9 10 11 12 Complexiy
(c) RN-50, CC-12M
(d) RN-50, YFCC-15M
Figure 9: Pre- Pre-Training Results CREPE Productivity (generalization longer and more complex sentences). Pre- Training model and datasets are mentioned below each figure.
92
Accuracy SVO
+1.2
86
84
Text
Both
Finetuned Encoder
(a)
'ARO: https://gitttub.comm http://ubb...oo///e/..t.ee vision- bo SYO- SVO-Probes ht https: http://ttt....cceeeee/////eeeeee
Figure 10: Extension of Figure 4 c). d). Selectively fine- tuning of image, text encoders and measure performance SVO-Probes dataset.
VL-Checklist: https://githab..em
VL. Ch-chhecklss
'CREPE Code: https:: ///t thub. cww..com//te. Com/RATVVE TVNT.ab/
CREPE

---

CUP NegCUP P
60-
MOSSCCLIP
40
1000
4.7%
+3 UN
20
1.4
Orforpatts
5973
RnticeS05p
KOC2007
Evesuicip
Cauntryll
Caltensot
OHAREO
CHARI00
Kestia
FRR2013
Flawersedd
Foodies
Grees
Hatellames
RCam
Menss
Stanfordcar
Figure 11: Comparing of CLIP, NegCLIP and MosaiCLIP on 20 datasets of from the ELEVATER (Li al., 2022a)
benchmark. Models this graph are pretrained with CC-12M data have Swin-Ting Tiny as the vision backbone. See Sec. 4.1 for more details.
80-
+2.2%
NegCUP PassiCUUP
5G
5%
40
+12
112.50 255
20
2.20
40.0%
1.85
6472
10C2
EuresatCpp
Country223
Cattcchios
CHArto
coARI00
FBR2013
Airoat
Romensed
GrSRS
Hotellemer
ACam
Stantorcca
Figure 12: Comparing of CLIP, NegCLIP and MosaiCLIP on 20 datasets from the ELEVATER (Li al., 2022a)
benchmark. Models in this graph pretrained with YFCC- I.M data and have Swin-Tiny the vision backbone. See Sec. 4.1 for more details.
80
NogCLP
60
+1200
+0 04
MosalCUPP
+0.3%
10.0%
20
1..3%
13.25
25
3972
VDC2107
PupseeSp
Gauntrylt
Gatecrtot
CrAatd
CHAncod
Kisttist
870
FEA2013
Airositt
Ruwersted
Footloy
G1588
Hatdllemee
Nest
Stanfurdaar
Figure 13: Comparing of CLIP, NegCLIP and MosaiCLIP on 20 datasets of from the ELEVATER (Li al., 2022a)
benchmark. Models in this graph are pretrained with CC- 12M data and have ResNet-50 as the vision backbone. See Sec. 4.1 for more details.
60
+3.0%
128G
440%
+2.0%
20
-2.9%
6.8%
43.0%
9512
vOC2007
Eu
Hountyyl
Ca
OHAROO
CHAR180
DTD
Alrak
Pamen 182
Foodint
Grse8
Hnnatlemer
ACam
Moiss
Stanordea
Resion
Figure 14: Comparing of CLIP, NegCLIP and MosaiCLIP 20 datasets of from the ELEVATER (Li al., 2022a)
benchmark. Models in this graph are pretrained with YFCC- -5M data and have ResNet-50 as the vision backbone. See Sec. 4.1 more details.


Reviews:
[{"review_id": "zjQZ0XO3Yu", "reviewer": "Reviewer_Ut7T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for an in depth review of our paper, for believing that our idea is interesting and novel, and for considering the experimental results as impressive. We also thank the reviewer for suggesting important experiments which we believe would greatly enhance the paper for future readers.\n\nPlease see our response below:\n\n>#### **1. Lack of enough comparison with previous works. There are also other works utilizing more types of negative samples such as DeCLIP, etc, which is not compared in the experiments.**\n\n**Difference w.r.t methods like DeCLIP:** Our negative mining strategy focuses on improving compositional reasoning capabilities of Vision Language Models and hence we compare MosaiCLIP with all previous methods that improve vision-language compositional reasoning (SynCLIP, Teaching-SVLC, NegCLIP). Our focus on compositionality is different and orthogonal to methods like DeCLIP that do not use negative samples focused on compositional reasoning capabilities like relation understanding or attribute binding and also do not perform coarse-to-fine learning like in our case. \n\nDeCLIP uses extra losses and negative mining strategies, including self-supervision on images and text, multi-view supervision and nearest neighbor supervision to improve their method’s data efficiency. Most negative mining techniques studied in literature do-not promote compositional reasoning. We find that although these additional losses provide extra supervision for data-efficient learning, the problem of poor compositional reasoning still remains (as shown below).\n\nSee the following table for results of DeCLIP and a related method, SLIP on attribute, relation, order understanding and systematic generalization. Here, we consider models pretrained on YFCC-15M obtained from DeCLIP github repository and compare it with MosaiCLIP pretrained on YFCC-15M. (best results of RN-50 architecture models are in **bold**, best VIT-B-32 architecture results in **_bold+italics_**)\n\n|           |        | ARO  |      |            |              | CREPE     |             |           |             |\n| --------- | ------ | ---- | ---- | ---------- | ------------ | --------- | ----------- | --------- | ----------- |\n|           | Arch   | Rel  | Attr | COCO-Order | FLickr-Order | Comp-Seen | Comp-Unseen | Atom-Seen | Atom-Unseen |\n| Random    | \\-     | 50   | 50   | 20         | 20           | 14.3      | 14.3        | 20        | 20          |\n| DeCLIP    | ViTB32 | 55.3 | 57.7 | 19.3       | 20.6         | 40.3      | 43.7        | 45.6      | _**46.6**_        |\n| SLIP      | ViTB32 | 59.6 | 57.0 | 23.7       | 25.0         | 44.7      | 44.1        | 45.4      | 43.8        |\n| CLIP      | RN-50  | 57.8 | 55.1 | 18.5       | 18.0         | 38.7      | 38.9        | 40.6      | 38.9        |\n| DeCLIP    | RN-50  | 54.0 | 56.3 | 17.4       | 18.1         | 38.8      | 41.1        | 45.8      | **44.8**        |\n| NegCLIP   | RN-50  | 68.0 | 58.5 | 36.9       | 37.3         | 61.5      | 67.2        | 43.5      | 41.5        |\n| MosaiCLIP | RN-50  | **76.3** | **68.9** | **37.6**       | **38.9**         | **80.1**      | **90.2**        | **46.6**      | **45.0**        |\n\n\n**Observations:**\n\na] DeCLIP is slightly worse than CLIP on attribute, relation and order understanding, and better than CLIP on systematic generalization capabilities as measured by the CREPE dataset\n\nb] NegCLIP is a stronger compositional reasoner as compared to DeCLIP, SLIP in general as depicted by most of the above results.\n\nc] MosaiCLIP improves over DeCLIP and SLIP in terms of compositional reasoning.\n\nOur method is model agnostic and orthogonal to DeCLIP, hence the extra losses/self-supervision used by DeCLIP can also be used in our framework to further improve data efficiency and performance of our approach. This is left as future work but we would work on implementing this and hope to include this experiment in the final version of our paper.\n\n>#### **2. I wonder if the performance improvement is brought by the proposed method or just a larger batch size brought by more negative samples. Are NegCLIP/CLIP and the proposed method in comparison using the same text batch size? If not so, it's a necessary comparison that adds more negative text samples in the original CLIP or NegCLIP so that the total text bz is the same as the proposed method.**\n\n\nThank you for bringing this up, this is an important consideration to establish the usefulness of our method. We were aware of this and have provided experiments to study this in the paper itself.  **See Appendix A.3.6** where we provide an analysis of the computational cost of our Model vs NegCLIP. **To summarize and answer your questions**,\n\na] MosaiCLIP’s effective batch size is larger compared to CLIP (by 6 to 9 times), due to text decomposition. NegCLIP’s effective batch size is larger by 8x since it uses image and text negative mining.\n\nb] MosaiCLIP uses a slightly larger effective batch size compared to NegCLIP in most of our experiments, (by a factor of 9/8=1.125). We use 3 positive and 6 negative sub-graphs per image (see Appendix section A.3.1, Fig 6 and 7 for more details on this choice). \n\nc] We find that providing NegCLIP with even larger batch sizes does not improve its performance, and performance stays below MosaiCLIP always. We also see a decrease in performance on some datasets when batch size for NegCLIP is increased which is possible because as batch size increases, negative images and texts sampled by NegCLIP can be false negatives for other elements of the batch.\n\nd] MosaiCLIP outperforms NegCLIP even with a lower batch size as shown in the table below.\n\n\nHere, a global batch size refers to batch size without any negative mining or text decomposition. Effective batch size refers to C=n*m where n and m are the number of images and text after negative mining/decomposition.  B=256 In the following table. \n\n\n\n| Method    | Global batch size (#I \\* #T) | Effective Batch size, C=n\\*m | ARO  |       | | SVO   |\n| --------- | ----------------------------- | ----------------------------------------------------- | ---- | ----- |  ----- | ----- |\n|           |                               |                                                       | Rel. | Attr. | Ord. | Avg.  |\n| MosaiCLIP | B\\*B                          | B\\*9B = 9B\\*B                                         | **79.0** | **69.6**  | **85.5** | **88.91** |\n|           | B\\*B                          | B\\*6B = 6B\\*B                                         | 78.2 | 68.0  | 85.3| **88.92** |\n| NegCLIP   | B\\*B                          | 2B\\*4B = 8B\\*B                                        | 71.5 | 65.4  | 84.5 | 88.36 |\n|           | 2B\\*2B                        | 4B\\*8B = 32B\\*B                                       | 68.9 | 65.6  | 83.9| 88.68 |\n|           | 4B\\*4B                        | 8B\\*16B = 128B\\*B                                     | 67.6 | 65.1  | 84.3| **88.93** |\n|           | 8B\\*8B                        | 16B\\*32B = 512B\\*B                                    | 65.7 | 64.2  | 83.8| 88.72 |\n|           | 16B\\*16B                      | 32B\\*64B = 2048B\\*B                                   | 62.5 | 63.7  | 82.4| 88.11 |\n\n\n\n\nAlong with this, in the paper we provide additional analysis showing that MosaiCLIP can improve over NegCLIP with as low as 0.3x-0.6x of the total data, which can further reduce the computational cost. Computational cost can also be traded off with performance by reducing or increasing the number of positive and negative subgraphs sampled as shown in Table 6 and 7 of the paper.\n\n\n**Providing CLIP with a larger batch size:**\n\n[1,2] measure various OpenAI-CLIP and OpenCLIP models trained on up to 32k batch size and find that the compositional reasoning performance is still very poor. An intuition for this finding is that, even if the batch size scales to very large values, it is very unlikely that close hard negatives for sentences and images exist in the same batch, which are essential for the model to learn to distinguish between changes in relations or attributes of objects. Due to this even with very large batch size, the required fine-grained understanding skills are not learned.\n\n\nReferences\n\n1. Yuksekgonul et al, When and why vision-language models behave like bag-of-words models, and what to do about it? ICLR 2023\n\n2. Ma et al, CREPE: Can Vision-Language Foundation Models Reason Compositionally? CVPR 2023\n\n\n>#### **3. Evaluation on image-text retrieval is missed.**\n\nOriginally we followed previous works like Syn-CLIP and Teaching SVLC and used the ELEVATER and ImageNet benchmark as the datasets to show that improving compositional reasoning improves or maintains the representation learning capabilities of MosaiCLIP. \nWe agree with the reviewer that it’s important to test our models on image-text retrieval tasks as well.\nSee the table below for image-text retrieval scores in different settings and different datasets including COCO and Flickr30K. We use the popular Karpathy splits and use the 5K and 1K test set for COCO and Flickr30k retrieval scores respectively:\n\n\nModels with Swin-Tiny backbone, pre-trained on YFCC-15M and tested on retrieval datasets:\n\n|           |   COCO      |            |    Flickr.    |           |    AVG.    |\n| -------------- | ------------ | ------------ | ------------ | ------------ | --------- |\n| Model     | I2T R@1 | T2I R@1 | I2T R@1 | T2I R@1 |      |\n| CLIP      | 20.7    | 13.1    | 36.2    | 24.1    | 23.5 |\n| NegCLIP   | 20.1    | 12.9    | 38.6    | 23.3    | 23.7 |\n| MosaiCLIP | **25.9**    | **16.5**    | **44.5**    | **29.5**    | **29.1** |\n\n\nOpenAI -CLIP Models with ViT-B-32 backbone, fine-tuned on COCO and tested on both retrieval datasets:\n\n|           | COCO    |         | Flickr  |         | AVG  |\n| --------- | ------- | ------- | ------- | ------- | ---- |\n| Model     | I2T R@1 | T2I R@1 | I2T R@1 | T2I R@1 |      |\n| CLIP      | 50.1    | 30.1    | 78.5    | 58.9    | 54.4 |\n| NegCLIP   | **56.2**    | 40.9    | **78.6**    | 67.3    | 60.8 |\n| MosaiCLIP | **56.1**    | **44.8**    | 75.6    | **70.2**    | **61.7** |\n\n\n**Observations:**\n\na] During pre-training, MosaiCLIP significantly improves retrieval scores compared to NegCLIP and CLIP,, with an increase up to +6.2 Recall@1 points across T2I and I2T metrics.\n\nb] During fine-tuning, we see that performance of MosaiCLIP is better or equal to NegCLIP for COCO retrieval, while it’s worse than NegCLIP for text retrieval and better than NegCLIP for image retrieval. On an average, MosaiCLIP improves over NegCLIP.\n\n**Please let us know if you have any other questions or concerns and we would be happy to answer them and engage in discussions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They proposed to parse scene graphs from the text for contrastive Image-text pre-training. The text scene graphs enable multiple positive samples and hard negative mining, which facilitate contrastive training. The idea is interesting and novel to some degree.\n2. The experimental result is impressive, showing a decent gain of the proposed model over previous methods. ", "reasons_to_reject": "1. Lack of enough comparison with previous works. There are also other works utilizing more types of negative samples such as DeCLIP, etc, which is not compared in the experiments.\n2. I wonder if the performance improvement is brought by the proposed method or just a larger batch size brought by more negative samples. Are NegCLIP/CLIP and the proposed method in comparison using the same text batch size? If not so, it's a necessary comparison that adds more negative text samples in the original CLIP or NegCLIP so that the total text bz is the same as the proposed method.\n3. Evaluation on image-text retrieval is missed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposed MosaiCLIP, a framework to decompose text into scene graphs for image-text contrastive learning. It incorporates hard-negative mining via text scene graph transformations and provides a coarse-to-fine contrastive learning strategy. The efficacy of MosaiCLIP is validated through comprehensive experiments across multiple architectures, datasets, training fashions, and compositional benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nVWkUP3FTG", "reviewer": "Reviewer_JAaq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for an in depth review of our paper, for recognizing our efforts in terms of thorough and reliable experimentation, and for believing that our work is novel, inspiring and brings fresh perspectives to the field.\nPlease see our response below:\n\n>#### **1. Indeed, the method's success heavily relies on the quality of scene graph generation. If errors occur during scene graph generation, it may lead to subsequent inaccuracies in the results. Ensuring a reliable and accurate scene graph generation process is crucial for the overall effectiveness of the approach.**\n\na] In our experiments, we make use of a simple and popular, rule based text scene graph parser (Wu et al., 2019) which is frequently used in literature and is based on the stanford scene graph parser. This parser is not the best parser available, and can contain some errors, such as missing edges or objects. \nHere are few examples sentences from the COCO Validation set, for which the corresponding scene graphs have some kind of error:\n\n| Sentence (Image Caption)                                                      | Error in scene graph                                         |\n| ------------------------------------------------------- | ----------------------------------------------------------------- |\n| A person driving a car on the street with a white car in view  | Missed a relation between car and street: (car, on, street)   |\n| Two men standing behind a cow in front of a house. | Missed relation between men and house: (men, in front of, house) |\n| A bathroom with a toilet and sink and a bathtub sitting on a hardfloor | Missed object: hardfloor     \n\nWe generally find most of the errors are in missed relations and few in missing objects.\n\nHowever, even these graphs with errors are useful because they contain informative sub-graphs that the MosaiCLIP method can use and perform coarse-to-fine learning. MosaiCLIP is tolerant to some type of errors, for example missing edges and objects, because even if some objects or edges are missing in the text scene graph, we can still create useful and even accurate positive and negative sub-graphs from the remaining graph and our formulation of image-text-graph alignment would still hold (as described in Section 3.2 of the paper). It is to be noted that the text scene graph is only used for constructing more positive/negative examples during training and it's **not used for inference** and hence it does not lead to any cascading inference time errors.\n\nb] To give more support, we refer to Analysis Section 4.2 of the paper, where we discuss that MosaiCLIP is robust to noise in data. A pre-training dataset like YFCC is very noisy as compared to CC-12M, and this noise leads to a less than optimal scene graph. However, meaningful sub-graphs are still present which lead to effective learning. We can imagine a similar effect will take place if the scene graph parser itself causes more errors and we expect that MosaiCLIP will be able to handle it to some extent.\n\nWe do not search for SOTA scene graph parsers in our work, and find that our approach can already achieve good performance using a readily available open source parser. Replacing this parser with any better parser is straightforward and should result in better performance of our model.\n\n\n>#### **2. The process of extracting scene graphs may consume significant computational resources, and in situations where the scene is complex, it might not be possible to obtain complete or accurate scene graph information. This can potentially harm the model's performance.**\n\nWe use a text scene graph parser and we believe the computational resources it consumes are significantly lesser as compared to the large GPU resources required for CLIP training, which makes the overhead marginal. We make the following points:\n\na] Scene graph parsing happens on CPUs since it's largely a rule based system. We can perform multiprocessing to parallelize it on one machine, and we can use multiple machines in parallel as well, which is a common setup for practitioners training CLIP. No expensive GPU resources are used in this process.\n\nb] The Scene graph generation process can be done offline before training the model, and has to be done only one time for any number of experiments with a dataset. This will save computation required to parse scene graphs for the same data point on every epoch (32 epochs in the common pre-training setup). \n\nc] In our python implementation, YFCC-15M  can be parsed in 17.6 minutes on a single machine with 96vCPUs. This time is further reduced to under 5 minutes if 4-8 nodes are used. Compare this with the time taken for pre-training where we use 8 nodes having 8 GPUs each, i.e., 64 GPUs in total and train for about 1.5 days. Hence the cost of creating scene graphs is marginal as compared to total resources spent on training CLIP.\n\nWe discussed the accuracy and noise of scene graph generation as part of the previous answer.\n\n>#### **3. Compared to other state-of-the-art models in the same field, the performance of this method is not particularly outstanding.**\n\nWe would like to seek further clarification regarding the term \"state-of-the-art models\" as mentioned by the reviewer in the context of our work. We have endeavored to delve deeper into the reviewer’s question in our response below:\n\na] Our primary focus in this paper is on improving vision-language compositional reasoning and in this process we want to maintain or improve the general representation learning performance of VLMs which make them useful for general tasks like classification and retrieval.\nWe identified all SOTA baselines for the compositional reasoning tasks that we focus on, which include models like SynCLIP, Teaching-SVLC, NegCLIP and we compare them with MosaiCLIP in Tables 1,2,3. \n\nIn the broader field of Vision-Language representation learning, there exists other SOTA baselines like BLIP, XVLM, Flava which show good classification and retrieval performance on datasets like ImageNet and MSCOCO, however, they lack severely in terms of compositional reasoning as shown by [1,2]. Hence these methods can’t be considered SOTA for tasks related to compositional reasoning like attribute binding, relational reasoning, order understanding.\n\nIf by sota models, the reviewer is referring to general VL models like BLIP, XVLM, Flava, we would be happy to include comparisons with these models. We compare them in the table below:\n\n|           | Rel  | Attr | COCO-Order | FLickr-Order | Average |\n| --------- | ---- | ---- | ---------- | ------------ | ------- |\n| Random    | 50   | 50   | 20         | 20           | 35    |\n| BLIP      | 59   | 88   | 32.1       | 36.9         | 54.0    |\n| XVLM      | 73   | **87**   | 36.2       | 46.3         | 60.6    |\n| FLAVA     | 25   | 73   | 3.9        | 12.9         | 28.7    |\n| CLIP      | 59.8 | 63.2 | 59.2       | 47.5         | 57.4    |\n| MosaiCLIP | **82.6** | 78.0 | **86.3**       | **87.9**         | **83.7**   |\n\nAccuracies for BLIP, XVLM, Flava are taken from [1].\n\nWe observe that MosaiCLIP has better overall compositional reasoning capability as compared to baselines considered above which shows that it is possible to have the benefits of CLIP (eg. inference efficiency, strong zero-shot generalization capabilities) while also having improved compositionality. In particular, MosaiCLIP is better at relational reasoning and word order understanding, while also closing the gap between CLIP and the best model (XVLM) in terms of attribute binding.\n\nWe will include this table in the final version of the paper for the benefit of future readers.\n\nb] We would also like to mention that our methods are model agnostic and aimed at improving contrastive learning through our coarse-to-fine learning framework and negative mining techniques. We believe our methods can directly be used to improve other VLMs like BLIP, X-VLM since these models also use contrastive learning while training. We are performing an experiment on BLIP to test this hypothesis and would be happy to provide the results soon during the discussion phase. We have very limited GPU resources during this rebuttal period (compared to when we submitted our paper), hence it is taking more time than expected.\n\nPlease let us know if your question/concern has not been answered through our response, we would be happy to engage in more detailed discussion.\n\nReferences\n\n1. Yuksekgonul et al, When and why vision-language models behave like bag-of-words models, and what to do about it? ICLR 2023\n2. Zhao et. al, VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations. EMNLP 2022\n\n\n**Please let us know if you have any other questions or concerns and we would be happy to answer them and engage in discussions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is commendable that the experiments in this study were conducted in a thorough and reliable manner, providing substantial evidence for the model's performance. The validation of the motivation behind the proposed approach adds further credibility to the research findings.\n\n2. The method is indeed novel and inspiring, offering fresh perspectives in the field.", "reasons_to_reject": "1. Indeed, the method's success heavily relies on the quality of scene graph generation. If errors occur during scene graph generation, it may lead to subsequent inaccuracies in the results. Ensuring a reliable and accurate scene graph generation process is crucial for the overall effectiveness of the approach.\n\n2.The process of extracting scene graphs may consume significant computational resources, and in situations where the scene is complex, it might not be possible to obtain complete or accurate scene graph information. This can potentially harm the model's performance.\n\n3.Compared to other state-of-the-art models in the same field, the performance of this method is not particularly outstanding.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This article presents a scene graph-based image-text contrastive learning method. By incorporating scene graphs, the fine-grained control of contrastive learning is achieved, and experimental results demonstrate performance improvement compared to the baseline.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vocEqkBxaX", "reviewer": "Reviewer_oZZx", "summary": "", "questions": "What are the templates when there are multiple nodes in the sub-graphs? Could you provide some specific examples?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reviewing our paper in great detail, for recognizing our technical innovations, and believing that MosaiCLIP is well-designed, with the results being comprehensive and convincing and each component of MosaiCLIP being important. We also thank the reviewer for recognizing our efforts in writing a sound paper with properly recognized limitations.\n\nPlease see our response below:\n\n\n>#### **1. As discussed in the Limitation section, it is unclear whether the proposed method can bring improvement to more advanced VLMs like BLIP.**\n\nWe understand the reviewers concern, however as mentioned in our paper and limitations section, our primary focus is on two tower encoder models like CLIP, since they enjoy various benefits like strong zero shot generalization capabilities, inference efficiency, wide applicability etc due to which there have been numerous works in literature that try to improve them, for example [1,2,3]. Having said that, we believe our methods are model agnostic and aimed at improving contrastive learning through our coarse-to-fine learning framework and negative mining techniques. We believe our methods can directly be used to improve other VLMs like BLIP, X-VLM since these models also use contrastive learning while training. We are performing an experiment on BLIP to test this hypothesis and would be happy to provide the results soon during the discussion phase. We have very limited GPU resources during this rebuttal period (compared to when we submitted our paper), hence it is taking more time than expected.\n\nReferences\n\n1. Mu et. al, SLIP: Self-supervision meets Language-Image Pre-training.\n\n2. Li et. al, Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm.\n\n3. Goel et. al, CyCLIP: Cyclic Contrastive Language-Image Pretraining\n\n\n>#### **2. There is no evaluation on standard image-text retrieval tasks (e.g., on COCO), in addition to the evaluation on compositional reasoning benchmarks.**\n\nOriginally we followed previous works like Syn-CLIP and Teaching SVLC and used the ELEVATER and ImageNet benchmark as the datasets to show that improving compositional reasoning improves or maintains the representation learning capabilities of MosaiCLIP. \nWe agree with the reviewer that it’s important to test our models on image-text retrieval tasks as well.\nSee the table below for image-text retrieval scores in different settings, on COCO and Flickr30K datasets. We use the popular Karpathy splits and use the 5K and 1K test set for COCO and Flickr30k retrieval scores respectively:\n\n\nModels with Swin-Tiny backbone, pre-trained on YFCC-15M and tested on retrieval datasets:\n\n|           |   COCO      |            |    Flickr.    |           |    AVG.    |\n| -------------- | ------------ | ------------ | ------------ | ------------ | --------- |\n| Model     | I2T R@1 | T2I R@1 | I2T R@1 | T2I R@1 |      |\n| CLIP      | 20.7    | 13.1    | 36.2    | 24.1    | 23.5 |\n| NegCLIP   | 20.1    | 12.9    | 38.6    | 23.3    | 23.7 |\n| MosaiCLIP | **25.9**    | **16.5**    | **44.5**    | **29.5**    | **29.1** |\n\n\nOpenAI -CLIP Models with ViT-B-32 backbone, fine-tuned on COCO and tested on both retrieval datasets:\n\n|           | COCO    |         | Flickr  |         | AVG  |\n| --------- | ------- | ------- | ------- | ------- | ---- |\n| Model     | I2T R@1 | T2I R@1 | I2T R@1 | T2I R@1 |      |\n| CLIP      | 50.1    | 30.1    | 78.5    | 58.9    | 54.4 |\n| NegCLIP   | **56.2**    | 40.9    | **78.6**    | 67.3    | 60.8 |\n| MosaiCLIP | **56.1**    | **44.8**    | 75.6    | **70.2**    | **61.7** |\n\n**Observations:**\n\na] During pre-training, MosaiCLIP significantly improves retrieval scores compared to NegCLIP and CLIP,, with an increase up to +6.2 Recall@1 points across T2I and I2T metrics.\n\nb] During fine-tuning, we see that performance of MosaiCLIP is better or equal to NegCLIP for COCO retrieval, while it’s worse than NegCLIP for text retrieval and better than NegCLIP for image retrieval. On average, MosaiCLIP improves over NegCLIP.\n\nc] Improvements in compositional reasoning improves or maintains performance on retrieval benchmarks.\n\n\n>#### **3. What are the templates when there are multiple nodes in the sub-graphs? Could you provide some specific examples?**\n\nNodes can be either attributes or objects, here are some examples of templates with more than two nodes (Ni, Ai, Ri refers to objects, attributes and relations):\nFor sub-graphs having 2 object nodes, where each may or may not have attribute nodes\n- {A1} {N1} {R1} {A2} {N2}\n- {N1} {R1} {A2} {N2}\n- {A1} {N1} {R1} {N2}\n- {N1} {R1} {N2}\n\nExample: “{Blue} {pen} {on top of} {white} {table}” will be a sentence as a result of the first template above\n\nFor sub-graphs having 3 object nodes, where each may or may not have attribute nodes. We join two sentences and one.\n- {A1} {N1} {R1} {A2} {N2} and {A2} {N2} {R2} {A3} {N3}\n- {N1} {R1} {A2} {N2} and {A2} {N2} {R2} {A3} {N3}\n- {A1} {N1} {R1} {N2} and {N2} {R2} {A3} {N3}\n- {A1} {N1} {R1} {A2} {N2} and {A2} {N2} {R2} {N3}\n- … (variations where multiple objects dont have attributes)\n\nExample: Given a sub-graph having relations: (pen, on top of, table); (table, behind, bed) and objects pen, table, bed having attributes blue, white, comfortable -- we can have a sentence using the first template as:\n“{Blue} {pen} {on top of} {white} {table} and {white} {table} {behind} {comfortable} {bed}”\n\n\n\n>#### **4. Some abbreviations are inconsistent, e.g., Fig.3 (line 452) and Figures 4 (line 467).**\n\nThank you for pointing this out. We will fix these in the final version of the paper.\n\n\n**Please let us know if you have any other questions or concerns and we would be happy to answer them and engage in discussions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed coarse-to-fine contrastive learning framework, which targets the viso-linguistic compositional reasoning problem, is intuitive and each component of MosaiCLIP is well-designed.\n* The experiments are comprehensive and the results are convincing, which validate the superiority of MosaiCLIP in viso-linguistic compositional reasoning. The ablation studies suggest that every component of MosaiCLIP is conducive.\n* The analysis on the reason behind MosaiCLIP’s improvement is instructive.\n* The limitations of this work are properly recognized and discussed. \n* The paper is well-written and easy to follow.", "reasons_to_reject": "* As discussed in the Limitation section, it is unclear whether the proposed method can bring improvement to more advanced VLMs like BLIP.\n* There is no evaluation on standard image-text retrieval tasks (e.g., on COCO), in addition to the evaluation on compositional reasoning benchmarks.", "questions_to_authors": "What are the templates when there are multiple nodes in the sub-graphs? Could you provide some specific examples?", "paper_topic_and_main_contributions": "This paper aims to improve the compositional reasoning ability of vision-language models (VLMs). To address this problem, a coarse-to-fine contrastive learning framework (MosaiCLIP) is proposed, which has three major technical innovations:\n* **Scene graph guided text decomposition**: The image caption is converted into a graph structure using a text scene graph parser and the sub-graphs are extracted as positive samples in image-text contrastive learning. Specifically, the sub-graphs are converted to sentences based on a template before being fed to the VLM text encoder.\n* **Negative sub-graph creation**: Hard negative sub-graphs are created by three operations, i.e., (1) node swapping and replacement, (2) edge replacement and (3) Connecting sub-graphs.\n* **Curriculum and robust fine-tuning**: To bridge the gap in training objectives between conventional CLIP model and MosaiCLIP, a two-stage fine-tuning strategy is proposed. In the first stage, each image is only associated with a single positive sub-graph and a negative sub-graph. In the second stage, each image has multi-stage positive and negative sub-graphs.\n\nThe empirical studies demonstrate that MosaiCLIP outperforms existing CLIP-based VLMs in terms of visio-linguistic compositional reasoning. Moreover, MosaiCLIP is more robust when the image captions in the training data are noisy. The reason behind the improvement of MosaiCLIP is also explained based on the improved Tree-Score of the text encoder.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Some abbreviations are inconsistent, e.g., Fig.3 (line 452) and Figures 4 (line 467)."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tries to improve the concept compositionally for contrastive VL models. The main novelty comes from the reliable negative example constructions from text scene graph. Overall, it is a solid paper.
Decision: Accept-Main