Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining
Authors: Emanuele Bugliarello
Aida Nematzadeh
Lisa Anne Hendricks
Venue: EMNLP 2023 Main (2023)
Keywords: vision-and-language
multimodal
pretraining
zero-shot
fine-grained
TL;DR: We propose two approaches to learn from weakly-supervised visual relation data, which improve zero-shot fine-grained understanding in pretrained vision-and-language models.

Abstract: Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. In this work, we take a step further and explore how we can tap into supervision from small-scale visual relation data. In particular, we propose two pretraining approaches to contextualise visual entities in a multimodal setup. With verbalised scene graphs, we transform visual relation triplets into structured captions, and treat them as additional image descriptions. With masked relation prediction, we further encourage relating entities from image regions with visually masked contexts. When applied to strong baselines pretrained on large amounts of Web data, zero-shot evaluations on both coarse-grained and fine-grained tasks show the efficacy of our methods in learning multimodal representations from weakly-supervised relations data.

Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining
Emanuele Bugliarello xd.c
Aida Nematzadeh
Lisa Anne Hendricks
"Google DeepMind
*University of Copenhagen
Abstract
Entities
Helrer
Perton
Snowboand
Recent work in vision-and iisoonandd-laane and-language pretrain- ing has investigated supervised signals from ob- ject detection data learn better, fine-grained multimodal representations. In this work, we take step further and explore how we can tap into supervision from small-scale visual relation data. In particular, we propose two pretraining approaches to contextualise visual entities in multimodal setup. With verbalised scene graphs, we transform visual relation triplets into structured captions, and treat them as additional image descriptions. With masked relation prediction, we further encourage relat- ing entities from image regions with visually masked contexts. When applied to strong hase- lines pretrained on large amounts of Web data, zero-shot evaluations on both coarse- grained and fine-graing fine-grained tasks show the efficacy of our methods learning multimodal represen- tations from weakly-superviee relations data.
Relations
Captions
Showboarder flying through the
Snowhoarder the
ALBEF
Browborrder fling trough the
X-VM
Snowboarder flying dhough the [Mlmal
ReX-VLM
Figure I: Overview of (i) the types image annotations and (ii) the pretraining tasks and models used in this work. addition to captions and entities, we study the benefits of modelling visual relations that link entities.
Recent work (Yao et al., 2022b; Zeng et al., 2022; Zhang et al., 2022, i.a.) shows that leveraging entity localisation, such as bounding boxes, from supervised data improves performance on down- stream tasks like visual QA (Antol et al., 2015) and visual reasoning (Suhr et al., 2019). Interestingly, modelling visual locations is also crucial to learn fine-grained image-text mappings (Bugliarello et al., 2023). Motivated by this promising research thread, in this work, we further explore the bene- fits of supervised data in multimodal pretraining by leveraging structured visual information in the form of relations in scene graphs (Eilliott and Keller, 2013; Johnson et al., 2015; Krishna et al., 2017).
1 Introduction
Current and-aanguae models (VLMs) are pretrained on large amounts of image-text pairs col- lected from the Web, and shown to perform remark- ably on variety of downstream applications (e.g., Tan and Bansal, 2019; Bugliarello et al., 2021; Rad- ford et al., 2021; Li et al., 2021; Zeng et al., 2022; Gan et al., 2022). Nonetheless, recent work has highlighted their limitations in fine-grained tasks, where precise understanding of both modalities is required to correctly select a positive match against a negative one. Examples of such tasks include verb understanding (sitting vs. standing; Hendricks and Nematzadeh 2021), word order (water in boutle vs. bottle in water; Thrush et al. 2022), spatial rela- tions (above vs. below; Liu et al. 2023) and other linguistic phenomena (Parcalabescu al., 2022; Nikolaus et al., 2022; Yuksekgonul et al., 2023).
A scene graph is a data structure that describes the content of a visual scene by expressing its en- tities (e.g., helmet), their attributes (e.g., red), and their relationships (e.g., person wear red helmet; sec Figure 1). While a large body of work has fo- cused on generating scene graphs (Lu et al., 2016; Xu et al., 2017: Peyre et al., 2017; Zellers et al., 2018: 8: Tang et al., 2020; Sharifzadeh al., 2021, 2022; Chang et al., 2023, inter alia), they have also been used in other applications, such as im-
'Work completed during an internship at DeepMind. denotes equal senior contribution. Correspondence to: Emanuele Bugliarello <emamucle@@ ku.k..k...

---

age retrieval (Johnson et al.. 2015), image gen- eration (Johnson et al., 2018) and image caption- ing (Yao et al., 2018; Yang et al., 2019).
of a visual scene goes beyond individual entities, and understanding their semantic relationships is a key step towards better VLMs. Nevertheless, this is an under-explored area of research. We hence investigate two approaches to better impose the structure of visual scenes from scene graphs (see Figure 2). The first approach, verbalised scene graphs (VSG), provides different view of im- age by associating it with a description of the rela- tionships between entities in the image. Our second approach, masked relation classification (MRC), predicts the relation between two entities in the same image when their cross cross-modal representa- tions are obtained from visually masked contexts.
However, the use of scene graphs in vision-and- language (V&L) pretraining has received limited attention. In contrast to prior work (Yu et al., 2021; Lee and Kang, 2022) that relied on rela- tions inferred from captions through masked lan- guage modelling, we use small dataset of human- annotated scene graphs, which go beyond the salient ones that are referred in a caption. To make full use of this rich learning signal, we introduce (i) a novel objective and (ii) a new method for data- to-text generation, which both explicitly aim to induce structure in the image and its connection in text. Our results show that modelling limited amount of scene graph data in addition millions of Web-crawled image-text pairs further improves coarse- and fine-grained skills in VLMs.
Setup. Given an image I, it can be associated with three types of annotations in our framework. Cr Cx- - {c} denotes collection of strings that de- scribe the image I (i.e., captions). &y = {(e)) is set of entities present in the image. Entities are defined by a label 4 (i.e., string such as "cat" "duck with sunglasses") and their spatial location as bounding box coordinates- in the image: e; = (4i, 2r, yn x,! v, 9I = {(e", r, e,))) is a collection of subject- relation object triplets link- ing two entities (ef and ee) via a string (r²) of their spatial relation, such as "below" or "in front of."
Contributions. In this work, 1) we aim at im- proving fine-graine tne-grained understanding in VLMs by modelling the structure of visual scenes during pre- training. In particular, we rely on small amount of
scene graphs,' and propose two novel pretraining approaches: verbalised scene graphs (VSG) and masked relation classification (MRC). When compared to strong baselines, 2) we show their effectiveness during pretraining on both fine- and coarse- searse-grained zero-shot tasks. Our models achieve overall better fine-grained abili- ties, such as state-of-the-a in visual spatial rea- soning, whilst keeping competitive performance on coarse-grained retrieval. 3) We shed light on the individual contributions and interactions of our proposed methods. We find, for instance, that VSG enables dense caption understanding, and that, at scale, modelling relations can be more effective than modelling entity locations for fine-grainc un- derstanding. Finally, 4) we revisit the standard prac- tice of selecting the last checkpoint in V&L pre- training, showing that COCO Dev TR @1 leads to better models, especially coarse- garseegrained tasks.
2.1 VSG: Verbalised Scene Graphs
Inspired by Bugliarello and Elliott (2021); Yu et al. (2021) and work in data-to-text generation (e.g., Kukich, 1983; Gardent et al., 2017; Agarwal et al., 2021), we explore the role of scene graph annota- tions for fine-grained understanding by generating text descriptions that encode entities and relations.
Given an image I and its scene graph GI, we first sample K triplets from Gz. (xx) Second, we ensure a fixed order in the triplets by sorting them based on the spatial loca- tion of their subject entities, represented by their centre location, (gI. .gk). we verbalise them into single caption: "[CLS] ri 1 [SEP] ... 12 rg 1% [SEP]," where [CLS] and [SEP] are special text tokens used in our baselines to learn a sentence-level representation and to separate between two phrases, respectively,
Finally,
2 Learning Visual Relations
Recent work in modelling spatial layout of entities in an image (Zeng et al., 2022) has been shown to be effective for fine-grained V&L understand- ing (Bugliarello et al., 2023). Yet, the information
As shown in Figure 2 (left), once verbalised, the resulting scene graph strings are simply treated analogously to image captions Cz. In our exper- iments, our models are pretrained with the three objectives used by ALBEF (Li et al., 2021): LA = Le + LITM + LMLM; where LMIM is the masked
WWe note that 'learning where only a subset of training data is given with labels' (i.e. incomplete supervision) is one of three types of weak supervision. We refer to Zhou (2017) for a relevant review of research in weakly lcarning.

---

XM
XM
[SEP]
sofa [SEP]
Cat
Laptop
Laptop In front
Laptop [MAS]]
[SEP]
eat (SEP)
VSG
MRC
Figure 2: Overview of our proposed approaches. VSG applies the standard pretraining objectives used on image captions to verbalised scene graph annotations. MRC predicts the relation (e.g., "in front of"") from predefined vocabulary between two entities when encoded with their visual context masked (shown dark patches here).
VM: vision model; LM: language model; XM: cross-moda model; CL: contrastive learning loss, ITM: image-text matching loss; MLM: masked language modelling loss; MRC: masked relation classification loss.
language modelling loss, LCL. is the image text contrastive learning loss, and LITM is the cross- modal image-text matching loss. That is, Lvsa is equivalent to LA but applied to verbalised scene graphs; but note that VSG data could, in theory, be used with any image-text losses applied in VLMs.
Models. We focus our analysis four models: ALBEF and X- X-VLM, and their corresponding relation-enhanced models (REALBEF and R VLM, respectively). For reference, we also test two strong systems: CLIP (Radford et al., 2021 I), a popular dual-encoder; and BLIP-2 (Li et al., 2023), VLM with frozen large image and text models.
2.2
MRC: Masked Relation Classification
Our second proposed objective aims at following the progress that masked predictions have had in NLP (e.g., Devlin et al., 2019; Zhang et al., 2019) and Computer Vision (e.g., Bao et al., 2022; He et al., 2022). In particular, we were inspired by X-VLM (Zeng et al., 2022), which learns to better localise entities by solely considering an entity's image region when applying image-tex text losses.
ALBEF (Li et al., 2021) widely used VLM that achieves strong downstream performance by effectively combining key components for V&L learning, such as a contrastive objective and cross- attention, in its design. In ALBEF, an image and a caption are first independently encoded with a vision (VIT; Dosovitskiy et al. 2021; Tou- vron et al. 2021) and text (BERT; Devlin et al. 2019) Transformer (Vaswani et al., 2017), respec- tively: and then fused in dual-stream cross- modal Transformer (Bugliarello et al., 2021). The model is pretrained with three objectives (ef', Sec- tion 2.1): masked language modelling, image-tex contrastive learning, and image text matching.
As shown in Figure 2 (right), given a scene graph triplet (e", r, e²) sampled from Gz, we first sep- arately encode its subject and object entities by masking their visual context. Second, we pool the final cross- moosl representation for the two enti- ties (represented by the final features of the [CLS] token in our models). Finally, we concatenate them into single vector, which is then processed by a two-layer MLP and mapped to an output space of V labels, corresponding to the top-V most fre- quent relations in our scene graph data. The model is then trained to predict (i.e., classify) the correct oubject-object relation with cross-entropy loss.
V-V (Zeng et al., 2022) uses the same com- ponents and objectives as ALBEF, but additionally learns locate visual concepts in the image given the associated texts. It does so by predicting an entity's s bounding box (bbox) coordinates given the visually grounded representation of its label (e.g., 'black swan'). Moreover, Bugliarello et al. (2023) showed that X-VLM also learns to ground an object label by applying ALBEF's losses to visually-masked image, which they collectively refer the visually-masked ALBEF (VMA) loss. These objectives allow it to acquire strong fine-grained understanding abilities, outperforming larger models, such as Flamingo (Alayrae et al., 2022) and BLIP-2 (Li et al., 2023), on these tasks.
3 Experimental Setup
We validate the effectiveness of our approaches by enhancing two strong VLMs four, diverse fine -ne-grained and two coarse-grained benchmarks. App. A provides details reproduce our work. Our models can accessed and verified online.?
Phttps://githhe com/e bwtp///bbu/wwaakaaeeee retps//waak-lllaiin--ll..e

---

Dataset
Img
# Cap
Ann
linguistic phenomenon (e.g, saying that an image shows six zebras instead of four for counting).
Image captions
SBU (Ordonez et al., 2011)
0.9M
0.9M
SVO-Probes (Hendricks and Nematzadeh, 2021) evaluates verb understanding by asking a model to compare a caption with two images: one that matches it, and one that is semantically differ- ent in its corresponding subject, verb, or object. Stanford Paragraphs (Krause et al., 2017) 7) is dataset of paragraphs describing images in unified stories (one paragraph annotation per image). Para- graphs give a coherent natural language description for images, requiring both ane--grined image un- derstanding and long-term language reasoning." 4
COCO (Lin et al., 2014)
0.IM
0.5M
VG (Krishna et al., 201 7)
0.1M
0.8M
CCas (Sharma et al., 2018)
1.8M
LSM
CC (Changpinyo al., 2021)
11.2M
1122M
Object detection
COCOOD (Iiin al., 2014)
0.1M
0.4M
VGoo (Krishnu ul., 2017)
0.IM
1.8M
VGgD (Krishna et al., 2017)
0.IM
2.9M
Scene graphs
GQA (Hudson and Manning, 2019)
0.IM
1.4M
Table I: Statistics of our pretraining corpora.
All these tasks are framed as image- text match- ing, a common pretraining objective of VLMs. On VSR, model'ss prediction is correct if the match- ing score is greater/lower than 50% for true/false label. On the other benchmarks, a model's predic- tion is correct if the score for the positive image- text pair is higher than that of the negative pair(s). Moreover, by evaluating through foils, which con- tain a single difference compared to the truth (e.g., only a word differs between true and foil captions), VALSE and SVO-Probes allow to quantitatively measure specific fine-grained V&L abilities.
Pretraining data. We pretrain all models for 200K/500K steps the same, publicly available 4M/14M datasets originally used by the authors, and, unless otherwise specified, we use the final checkpoint for evaluation. In particular, we rely on three types of pretraining data: image captions, ob- ject detection and scene graphs. Table lists their statistics, where '# Ann' denotes the total number of entities identified by bbox-label pairs in object detection data, and the total number of relations in scene graphs. The unique number of relation strings in GQA scene graphs (expanding the origi- nal ones in VG) is equal to 310, which determines the size of the output vocabulary for our masked relation classification (MRC) method.
Results
Table 2 shows performance on fine-grained tasks that cover wide range of multimodal abilities of our baselines, relation-enhanced models, well current strong dual- and cross-encoder models. 5
Benchmarks. We report zero-shot performance on coarse-gre coarse-grained retrieval in Flickr30K (Young et al., 2014) and COCO (Lin et al., 2014), and on four English fine-grained understanding datasets.
Enhanced visual spatial reasoning capabilities. We validate the effectiveness of our proposed ap- proaches in modelling visual relations by evaluat- ing the task of visual spatial reasoning (VSR). This is benchmark that focuses spatial lations, and we expect our approaches to signif- icantly improve upon their baselines here. Our pro- posed 1VMM model substantially outper- forms its X- X-VLM baseline by +6.8/3.0pp on the Dev/Test sets, setting a new statc-of-the-an on zero-shot VSR. Moreover, REX-VLM 13M consis- tently outperforms the other models on the related subtasks of 'spatial relations' and 'actant swap' of VALSE (see Table 5 in App. B.1). We also observe
VSR (Liu et al., 2023) tests for 65 types of vi- sual spatial relationships (e.g., under, in front of) grouped into seven categories (e.g., adjacency, orientation). Each sample consists of an image- sentence pair; a model needs predict whether the sentence correctly describes the spatial relation berween two entities the image. We zero-shot evaluate models on the 'random' split, and report accuracy on both the Dev and Test sets due to their low correlation (Bugliarello et al., 2023).
VALSE (Parcalabescu et al., 2022) examines six linguistic phenomena, such plurality, actions and coreference. Given an image, model is asked to distinguish real captions from foils (Shekhar et al., 2017), where a foil is constructed from a caption by altering a word or phrase that realises a specific
"Note that the images Stanford Paragraphs are a subset of VG. Recent VLMs use VG data during pretraining, which could justify the high performance we observe in our results. 'Though these fine- grained tasks do not explicitly require scene graph understanding or generation, we hypothesise that by training with this data, our models will gain hetter fine- grained image-text text understanding.
'We note that only 1.8 and 11.2M data points were avail- able for CC and CC22 respectively, at our time of study.

---

Model
VSR Random
VALSE SVO-Probes
Stanford Paragraphs IR@1/5
Name
Role
Dev /Tsst Acc
Ace;
Acc,
TR@ 1/5
CLIPa00M
N/A
64.0
81.6
45.3/73.1
53.4/ 80.1 /3.4/95.2
11.I--120M
61.2 61.5
74.0
86.5
81.1 / /94.3
ALBEFsM
BASELINE
63.7 / 60.1
69.4
86.6
79.8 79.5
94.9
REALBEFSM
+RELATIONS
64.0 60.2
69.6
86.2
85.1 85.1/97.4
85.8 997.2
X-VLMM
+LOCALISATION
63.5 62.3
69.5
87.3
79.8/ /9.88
81.4 /95.0
+BOTH
65.0/ 61.8
70.9
87.3
87.4 97.8
87.8 99744
ALBEF
BASELINE
6
72.2
86.7
73.7 77.1
90.3
REALBEFIM
+RELATIONS
66.3
70.4
87.5
86.7 /97.5
86.5 / 97.22
X- X-VLM
+LOCALISATION
61.1 / 60.5
71.3
87.3
80.3 94,9
76.8 92.4
+BOTHI
68.4/ / 63.5
73.3
88.1
89.3 98.0
88.8 97.77
Table 2: Overall results on zero-shot fine- graiggd benchmarks. Models pretrained on 3M/13M images are evaluated after 200K/500K steps, respectively. Values underlined in green (red) denote gains (losses) of relation-enhanced models on their baselines. TCLIP cannot be directly evaluated on VSR since it requires true/false predictions for a given image-texx text input, while CLIP is only trained with a contrastive loss. Best results are in bold.
consistent gains when modelling relations on top of ALBEF (REALBEF) 13M gains +3.8/0.8pp). which shows that even just modelling relations (without modelling objects) is helpful for VSR. These re- sults show that our approaches to modelling rela- tions play an important role in tasks that require spatial reasoning. Finally, we see that modelling visual relations when pretraining on fewer images only results in slightly better VSR Dev accuracy. It is interesting to note that both our models fur- ther increase their performance gains when moving from 3M to 13M images, despite now only having 0.6% of the images annotated with scene graphs.
Flickr30K
COCO
Model
IR 1/5
TR
IR@0 1/5
TR 1/5
CLIPaazos
88.0798.7
68.7/90.6
584/81.5
/7.8
BLIP-2129M
95.5/
86.7/97.1
80.7
64.2 /1.2/85.2
ALBEF
77.9/92.7
63.3/
63.6/ 86.1
77.5
REALBEF
75.5/92.3
59.5/82.6
62.7
86.2
44.7
X-VLMMM
/8.2/94.1
61.8 84,1
63.9
86.7
47.9
77.7
REX-VLMM
78.6 93.9
61.8/ 84.8
66.9/
87.5
48.9 76.2
ALBEF
66.1 82.2/95.5
85.8
64.8 86.6
49.1 77.6
REALBEF
65.3 80.3/93.7
86.0
66.4
87.8
49.1 7
X-VLM
83.3/95.6
66.2 H6.3
64.1 86.4
49.1 74.8
REX-VLM
80.3 /
66.6 / 87.0
66.9 88.9 50.2 / 77.0
Table 3: Overall results on zero-shot coarse-grained benchmarks. Models pretrained on 3M/13M images are evaluated after 200K/500K steps, respectively. Values in green (red) denote gains (losses) of relation-enhanced models on their baselines. Best results are in bold.
Improved fine-grained understanding. In addi- tion to VSR, REX- MLM performs best across all the fine-grained tasks, which test models for much broader range of finengrained skills. It gains +1.7pp on VALSE and +0.8pp on SVO-Probes, and REX- VLM gains + +1.4pp on VALSE. These re- sults confirm that visual relations can provide use- ful signal towards fine-grained understanding, even when only available for a tiny percentage of pre- training data. On the other hand, REALBEF mod- els on par with their baselines. Recall that they model relations between entities without explic- itly learning about the entities themselves. That is, it is harder for ALBEF models to learn relations without doing localisation. Moreover, comparing X-VLM and REALBEF on VALSE and SVO- Probes, we see that modelling objects (X-VLM) on top of ALBEF is slightly better than solely modelling relations (REALBEF).
captions with two main predicates than one. We thus consider testing our models for the ability to understand long, fine-grained descriptions of im- ages on the task of zero-shot mmage-paragraph trieval. In fact, paragraphs are longer, more in- formative, and more linguistically complex than sentence-level captions. VEX-VLMM3MM achieves 89.3 TR@1 and 88.8 @R@1 I (+9.0pp and + +12.0pp compared to X-VLM13M) Such high performance is largely preserved when training on 3M images (87 4pp and 87.8pp), and it carries over to REAL- BEF models well. Overall, relation-enhanced models gain from +5.6pp to +12.8pp on this task.
Competitive coarse-grained retrieval. Finally, evaluate our relation-enhancee models zero- shot image-text retrieval tasks to verify that their gains on fine-grained tasks do not hamper perfor- mance on coarse coarse-grained tasks. Table 3 lists perfor- mance on the Flickr30K and COCO datasets. Our REX- VLM models achieve similar or better per-
Substantially better fine-grained understanding on dense captions. Thrush et al. (2022) showed that current VLMs struggle more when matching

---

ALBBIF33
X-VLMigM
10+
-956
+090
MIRC
(VRG MRC
VALSE
VSS Bey
VaR
500
Pine 18ght
FOR TKgI
COCO Tiai
Walls
V0R Boat
800
Pos 1981
FIRK lngit cocorige
Figure 3: Difference in performance when adding approaches ALBEF 13M and X- X-VLM models.
ALREF 3M
ALBHF136 13k
-ovalisctonn (C-VMM)
-Loaiaaioo d VLM
10-
"Kehnsee (MAASSL))
Bich VIM-
VALSE
VIR Dov
VSR Text
SVD
Pure TRig
F3R TRg! coco cago
VALSE
VSR
VSR Toat
Pore Tegit
FHR Tagg)
coco
Figure 4: Performance difference when adding supervision for localisation, relations and both approaches.
ing both VSG and MRC typically leads to the best performance. VALSE, we find that us- ing either approach independently decreases accu- racy, while using them together increases it. It is clear that VSG is instrumental to perform well on maraggppr retrieval for both models. How- ever, VSG hurts performance coarse sarse-grained retrieval tasks. This likely because scene graphs are treated equally to image captions here, although being distributionally different. Finally, we see similar patterns for ALBEF models, although they often gain more from VSG than from MRC.
X-VLMM13 Uasoral
60
40
20
Figure 5: Ablations of our VSG approach.
formance than their X-VLM baselines, especially COCO (+2.8pp TR@ and +1.2 IR@1) 1). That is, learning visual relations from small amount of an- notated images especially effective on in-domain data (relation-annotatee images are subset of COCO images) for the task of zero-shot image-text retrieval. On the other hand, REALBEE EF models tend to perform slightly worse than their baselines, especially on Flickr30K and when trained for 200K steps 3M images. This shows that modelling relations without objects hinders performance coarse-grained understanding in this setup. Figures 8 and (App. B.2) show that this is due sub- optimal data mixing rather than modelling limita- tions at scale. For instance, REALBEFAM matches or outperforms ALBEFMM when trained for longer.
What matters for long context understanding? As discussed above, our VSG approach plays a crucial role towards dense caption understanding. In VSG, we propose alternative view of an im- age by creating textual description that consists of sequence of subject-rclation-obye sampled from the image's scene graph. In our main approach, we verbalise scene graph annotations by (i) sampling 8 relations per image, and (ii) sorting them based on the subject entity's bounding box coordinates. We ablate both of these choices in Fig- ure 5, where we pretrained X- models for 200K steps by adding VSG variants with (i) fewer relations instead of 8), and (ii) without sorting them. We find that while sorting the relations is not critical perform well on Stanford Paragraphs, the number relations an important factor. This not only significantly boosts paage--rragrahh re-
triplets
Approach ablations. Figure shows the indi- vidual contributions of our proposed approaches towards the final models' performance. Looking at our BEX-VLM models, we see that combin-

---

trieval, but also leads to smaller yet consistent gains on COCO, which contrasts previous findings on the effectiveness of long descriptions for caption retrieval (Hendricks et al., 2021). We hypothesise this is due to the nature of our descriptions, which encode local relations between entities, rather than being long, complex natural captions.
fine-grained tasks, In fact, while our baseline's per- formance starts degrading after 250K steps, REX- VLM continues improving over time, reaching its best yet results at 500K steps." 7 We can also see that, by the end of our training, relation-enhanced models typically achieve better performance than the best results given by our baselines, confirming the validity of our results from the previous sec- tion. Likewise, the evaluation curves show that our models and baselines can achieve compara- ble coarse- coarse-grained retrieval performance, and that longer training can help relation-cnhancee models close the gap with the baselines (see Figure in App. B.2). Given fixed number of steps, we leave it for future work to investigate pretraining sched- ules that better balance coarse- and fine-grained tasks
Localisation or relations? We finally analyse whether it is more effective to model visual loca- tions of entities or their relations in visual scenes. To do so, we compare the performance gains obtained on top of ALBEF by X- L-VLM (local- isation), REALBEF (relations) and REX-VLM (both). For models trained on 13M images, Fig- ure 4 (right) shows that modelling relations alone is typically better than modelling objects. A no- table exception is noarse-grained retrieval on out- of-distribution Flickr30K, where both modelling localisation and, especially, relations decrease per- formance. Combining both objectives results in the best results. When training on 3M images, Figure 4 (left) shows similar results but with object locali- sation giving larger contributions. Taken together, we see that current VLMs can learn more from modelling relations than localisation when trained at scale. Finally, we see that adding and mod- elling a small amount of supervised data (as done by REALBEF 3M. 3-VLM and REX- VLM 3M) is typically more effective than adding 1 11M addi- tional image-text pairs crawled from the Web (i.e., ALBEF13M) for tne-grrined understanding.
to obtain single checkpoint that per- forms well on both kinds of skills.
Checkpoint selection strategies. As shown above, our relation-enhancee models achieve high- est performance on most tasks at the end of train- ing. On the other hand, this is not the case for our ALBEF and X-VLM baselines. We hence revisit the concept of checkpoint selection in pre- trained VLMs, as recent work simply trains for a fixed number of steps (e.g.. 200K or 500K). Specif- ically, we analyse how using different tasks (Dev split when possible) for checkpoint selection af- fects performance on other benchmarks. That is, for each model, we select the checkpoint that gives the highest score on a given Dev task, and evaluate it across tasks. In Figure 7, we show the difference in performance (y-axis) obtained using different checkpoint selection strategies (æ-axxs) compared to the fixed checkpoint results reported in Tables and 3, averaged across all models." Overall, we find that COCO Dev TR leads to better check- point selection for all coarse-grained benchmarks (and a small improvement overall). However, we do not see consistent pattern for fine-grained tasks, probably because they more varied in terms of skills they test compared to coarse coarse-grained retrieval tasks. For instance, using SVO-Probes re- sults better VSR but worse VALSE performance. Table App. B shows fine- and coarse-grained performance when selecting checkpoints based on COCO Dev TR@1 1. While REX- VLM still outper-
5 Analysis of Learning Dynamics
Recent work on V&L pretraining typically trains for a fixed number of steps, and then selects the last checkpoint to report performance on different tasks. However, Bugliarello et al. (2023) showed that current, strong models achieve peak perfor- mance on different fine-grained tasks at different stages of pretraining. This motivates us to study the pretraining dynamics of our models, and to reassess the current practice of choosing the last checkpoint by investigating how different checkpoint selection strategies affect the performance on our tasks.
Convergence rates. Performance for X- X-VLM models pretrained on 13M images is shown in Fig- ure 6.6 We find that our REX- VLM model requires longer training to achieve peak performance across
"W note that fine-grained understanding of REALBEFsM and REX-VLME start degrading after 350K steps (App. B). 'In Figure 7, our average results do not include RRALBEF and CEX-VIMMpp as they consistently achieved highest performance at the last, fixed checkpoint.
'Figures and 9 in App. B.2 show similar patterns for all the models pretrained on 3M and 13M images, respectively.

---

VALSE Adtr
VSR Avg.
VSIk
SNO Probes
Stunfeet Dev 18651
Stanford hragras lest TRi51
PlickroKK Dev TRi61
FiiderOKK Toat TRtgt
0O00 Dey TR
COCO leat Titt
D-2
54-
75
Figure 6: Pretraining dynamics of our X VLM models when learning from 13M images.
collected from the Web (e.g. Lu et al., 2019; Chen et al., 2020; Radford et al., 2021; Alayrac et al., 2022: et al., 2022; Li et al., 2022b, 2023), a recent line of work investigates the chal- lenge of learning fine-grained image-text map- pings. FILIP (Yao et al., 2022a), LOUPE (Li et al., 2022a), RegionCLIP (Zhong et al., 2022), Pyra- midCLIP (Gao al., 2022) and HiCLIP (Geng et al., 2023) propose different fine-grained align- ment methods for dual-encoder networks. On the other hand, GLIP (Li et al., 2022c; Zhang et al., 2022), Fiber (Dou et al., 2022), PEVL (Yao et al., 2022b), MVPTR (Li et al., 2022d), X- K-VLM (Zeng et al., 2022) and PaLI (Chen et al., 2023b) show the benefits of learming cross-modes representa- tions from additional supervised object detection data. Finally, there is increasing interest in training VLMs that perform well range of coarse- and fine une-grained vision and language tasks (Lu et al., 2020; Wang et al., 2022; Lu et al., 2023; Zou et al., 2023; Chen et al., 2023a; Beyer et al., 2023).
Tec
0.87
0.15
1.10
0.44
0.28
F30K Text TRig
0.67
-0.23
0.85
0.78
0.02
0.12
Text 1
0.65
0.26
0.75
0.20
COCO Text TR
1.36
0.59
1.17
1.57
0.36
0.66
Ang. Ase-
-0.29
-0.06
-0.26
-0.04
0.54
-0.46
SVO Ast.
Actz
0.33
0.15
0.12
0.31
-0.05
0.53
Aug.
0.63
0.77
0.89
0.31
VSR AastAgg.gg......... Ace
0.34
0.67
0.68
0.68
-0.18
0.85
Avg.
0.57
0.29
0.55
0.76
0.20
0.41
FR9K De 1881
FHOK Tav TRal
GOOD 1889 1881
1003 De 1300
VAUSE Ave Aoc-
SY0 Av.. A00
Figure 7: Average test performance differences (g-axis) with respect to fixed checkpoints for all models except for REALBEFIJMM and REX- VLM according to dif- ferent checkpoint selection tasks (x-axis), For detailed results on each model, see Figure 10 in App. B.3.
forms the other models on fine-grained tasks, find that the baselines perform on par on coarse- grained tasks. Finally, we note that, while different checkpoint selection strategies result slight vari- ations in task performance, the ranking models does not change much. This is shown in Figure 11 (App. B.3), where we compute the Spearman rank correlation coefficients between COCO Dev TR@) and the other strategies, across all models. The high rank correlation coefficients across all strategies and evaluation tasks demonstrates that REX-VLM robustly outperforms the other models.
Scene graphs and multimodality. The structural representations of scene graphs has been explored in the context of different V&L tasks, such image-text retrieval (Johnson al., 2015; Schus- et al., 2015; Schroeder and Tripathi, 2020; Ge et al., 2023), image captioning (Yao et al., 2018;
Yang et al., 2019), and visual QA (Qian et al., 2022:
Koner et al., 2021; Lee et al., 2019; Shi et al., 2019).
Only two studies have, however, investigated the role of scene graphs in V&L pretraining. ERNIE- ViL (Yu et al., 2021) first extracts scene graphs from the captions with off the-shehlf model, and then proposes MLM-based object, attribute, and lationship prediction tasks to learn cross-modal de- tailed semantic alignments. Lee and Kang (2022), in addition to extracting subject-relae cetetdrrelattoboppeeeaee triplets from captions with an off-the-shelf model,
6 Related Work
Fine-grained VLMs. While the vast majority VLMs are solely pretrained on large-scale data

---

also generate paired visual features based on the en- tities oulput by an object detection model and their co-occurrences in the VG dataset (Krishna et al., 2017). Unlike them, we rely on a small sample of human-annotated scene graphs, and propose two methods for relation prediction in V&L pretraining. Furthermore, we are the first to show the benefits of modelling scene graphs towards acquiring better fine-grained skills during multimodal pretraining.
setup-wherein the data can contain concepts be- yond those represented in English and Western so- cieties (Liu et al., 2021)-and towards safe and reli- able deployment of multimodal systems. A promis- ing direction to mitigate this limitation is to devise bootstrapping strategies to enrich a massive number of images with rich scene graph annotations.
From an experimental angle, we measure zero-shot performance of pretrained vision-and- language models (VLMs). Due to resource con- straints, we only pretrain our models once. Al- though we observe consistent gains of our ap- proaches with respect to their baselines, we note that Bugliarello et al. (2021) showed that pretrain- ing given model with different seeds can result in different performance when fine-tuned on several downstream tasks, like visual question answering or visually grounded reasoning. Further investiga- tion required to assess the variance pretrained VLMs in zero-shot (fine-grained) evaluations.
7 Conclusion
Previous work in multimodal pretraining has shown the importance of modelling objects (using locali- sation data) in improving the performance of both coarse- and fineggrained tasks. In this paper, we investigate if supervision from relational data-by modelling relations between objects in a visual scene-can improve performance on these tasks. In particular, rely on scene graph annotations, under-explored data structure for multimodal pretraining, and propose two approaches for lever- aging relations between entities in an image: 1) MRC. C. a pretraining objective that predicts the rela- tion between the objects in two image regions; and 2) VSG, a versatile data-to-text generation recipe that converts scene graphs into captions, that can then be fed to any VLM. When applied to strong VLMs, we find that our methods improve their fine- grained understanding, with REX-VLN VLM achieving state-of-the-art spatial reasoning abilities, as well as strong performance on other tasks too.
Morcover, even though the proposed approaches can be applied most recent VLMs, we only eval- uate two Aichitectures-ALBEE and X-VLM-- due to computational constraints. Although X- V VLM the current state-of-the-art for most fine- grained understanding tasks, it would be instructive to measure how our approaches transfer to mod- els that process images through learnable visual queries (Alayrac et al., 2022; Li et al., 2023, i.a.)
We also note that some of our evaluation datasets are quite small, and encourage the community create larger evaluation sets to reliably measure progress in coarse- and fine-grained V&L skills.
We hope that our work motivates further research in improving fine-grained understanding in VLMs. Given the promise of our results with a few anno- tated images, an interesting future direction is to study how to best scale up our approaches with ma- chine generated data, egg. by generating pseudo- labels from off-the-shelf scene graph generators from either images or captions, or both.
Finally, in this paper, revisit the idea of check- point selection for pretrained VLMs. While recent work simply trains for fixed number of steps, we find that using COCO validation TR @ leads overall better models in our evaluations. Yet, our findings are only based on a handful of models. We encourage the community to investigate this line further, especially since current VLMs may learn different skills at different stages of pretraining.
Limitations
Our paper investigates the benefits and limitations of learning structured information in visual scenes from scene graph annotations.
Ethics Statement
Collecting such rich annotations from humans is time consuming, and it cannot be casily scaled up to millions of images. While our work shows that models pretrained at scale can still benefit from a limited number of scene graphs, differences were less significant on anttof-diitribution images. This aspect is especially relevant in a multilingual
In this work, we include additional supervision to guide models into learning visual relations and improve performance on a variety of vision-and- language tasks. However, biases in multimodal datasets are well documented (Meister et al., 2022) and, without further mitigation, we expect our mod- els to learn them. Furthermore, our datasets include

---

images with faces, and there is no mechanism for people to remove themselves from these datasets, Multimodal models like ALBEF and X- X-VLM can be used for a variety of vision-and-languagge tasks including image and video retrieval, video description, and visual question answering. Ben- eficial upplications of such models include better human-computer interaction, or visual description and question answering for the visually impaired. However, these models can also be used for harm- ful applications such as surveillance.
Roman Ring, Francisco Ruiz, Alvaro Sanchez, Ros- alia Schneider, Eren Sezener, Stephen Spencer, Sri- vatsan Srinivasan, Wojeiech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. 2020. The Deep- Mind JAX Ecosystem.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022. BEiT: BERT pre-training of image transform- ers. International Conference on Learning Repre- sentations.
Lucas Beyer, Bo Wan, Gagan Madan, Filip Pavetic, An- dreas Steiner, Alexander Kolesnikov, André Susano Pinto, Emanuele Bugliarello. Xiao Wang, Qihang Yu, Liang- -hicch Chen, and Xiaohua Zhai. 2023. A study of autoregressive decoders for multi-tasking in computer vision. ArXiv, abs/2303.17376.
Acknowledgements
The authors would like to thank Aishwarya Agrawal, Laurent Sartran, Jovana Mitrovic, Sahand Sharifzadeh, Chris Dyer and the Google DeepMind Language Team for feedback on this project.
Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki. and Desmond Elliott. 2021. Multimodal Pretraining Unmasked: A Aeta-Analysis and a Unified Frame- work of Vision-and-Languaae BERTs. Transactions of the Association for Computational Linguistics, 9:978-994.
References
Emanuele Bugliarello and Desmond Elliott. 2021. The role of syntactic planning in compositional image captioning. Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 593-607, Online. Association for Computational Linguistics.
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. Knowledge graph based synthetic corpus generation for knowledge- snowledge-enhanced language model pre-training. In Proceedings of the 2021 Con- ference of the North American Chupter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. pages 3554-3565 Online. As- sociation for Computational Linguistics.
Emanuele Bugliarello, Laurent Sartrain, Aishwarya Agrawal, Lisa Anne Hendricks, and Aida Ne- matzadeh. 2023. Measuring progress in fine-grained Dision-and-languaae understanding. In Proceedings of the 61th Annuul Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Toronto, Canada. Association for Computational Lin- guistics.
nannBBatiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Mal- colm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Saman- gooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowski, Ricardo Bar- reira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: visual language model for few-shot learning. In Advances Neural Infor- mation Processing Systems.
Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li. Xiaojiang Chen, and Alex Hauptmann. 2023. com- prehensive survey of scene graphs: Generation and application. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(1):1-26.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web- scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 3558-3568.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question an- swering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425- 2433.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman. Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani. Daniel M. Salz, Mario Lucic, Michael Tschamnen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang. Ceslee Montgomery, Paulina Pietrzyk. Marvin Ritter. A. J. Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim M. Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Steiner, Yang Li, Daniel Keysers, Anurag Arnab,
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhu- patiraju, Jake Bruce, Peter Buchlovsky, David Bud- den, Trevor Cai, Aidan Clark, Ivo Damihelka, An- toine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan,

---

Yuanzhong Xu, Keran Rong. Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. 2023a. PaI.I- X: On scaling up multilingual vision and language model. ArXiv, abs/2305.18566
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. 2022. Pyramid- CLIP: Hierarchical feature alignment for vision- language model pretraining. In Advances Neural Information Processing Systems.
Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Pier- giovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue. Ashish V Thapliyal, James Bradbury. Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Ruiz, An- dreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. 2023b. PaI.:: A jointly-scaled multilingual language-image model. In The Eleventh International Conference on Learn- ing Representations.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Beeez-Beltrachin.. 2017. The WebNLG challenge: Generating text from RDF data. In Pro- ceedings of the 10th International Conference on Natural Language Generation, pages 124-133, San- tiago de Compostela, Spain. Association for Compu- tational Linguistics.
Xuri Ge, Fuhai Chen, Songpei Xu, Fuxiang Tao, and Joemon M. Jose. 2023. Cross-modal semantic en- hanced interaction for image-sentence retrieval. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 1022-1031.
Ven-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu, 2020. UNITER: Universal image-text representation learning. In European Conference on Computer Vision, pages 104-120. Springer.
Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. 2023. HiCLIP: Contrastive language-image pretraining with hierarchy-aware at- tention. In The Eleventh International Conference Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Dollár, and Ross Girshick. 2022. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 16000-16009.
Lisa Anne Hendrieks, John Mellor, Rosalia Schneider, Jean- Bean-Baptiste Alayrac, and Aida Nematzadeh. 2021. Decoupling the role of data, attention, and losses in multimodal transformers. Transactions of the Associ- ation for Computational Linguistics, 9:570-585.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Lisa Anne Hendricks and Aida Nematzadeh. 2021. Probing image-language transformers for verb derstanding. In Findings of the Association for Com- putational Linguisties: ACL-IJCNLP 2021, pages 3635-3644, Online. Association for Computational Linguistics.
Zi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, Jianfeng Gao, and Lijuan Wang. 2022. Coarse-to-fine vision- language pre-training with fusion in the hackbone. In Advances in Neural Information Processing Systems.
Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reason- ing and compositional question answering. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Patten Recognition (CVPR), pages 6700- 6709.
Desmond Elliott and Frank Keller. 2013. Image de- scription using visual dependency representations. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1292-1302, Seattle, Washington, USA. Association for Computational Linguistics.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. 2018 Image generation from scene graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1219-1228.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei- Fei. 2015. Image retrieval using scene graphs. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 3668- 3678.
Zhe Gan, Linjie Li, Chumyuan Li, Lijuan Wang. Zicheng Liu, Jianfeng Gao, et al. 2022. Vision-language pre- training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, (4(3-4):1633355

---

Rajat Koner, Hang Li. Marcel Hildebrandt, Deepan Das. Volker Tresp. and Stephan Günnemann. 2021. Graphhopper: Multi-hop scene graph reasoning for visual question answering. In The Semantic Web ISWC 2021: 20th International Semantic Web Con- ference, ISWC 2021, Virtual Event, October 24-28, 2021, Proceedings, page 111-27 Berlin, Heidel- berg. Springer- Verlag.
In Advances in Neural Information Processing Sys- tems, volume 34, pages ciates, Inc.
Curran Asso-
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang. et al. 2022c. Grounded language-iman
pre-training. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognition, pages 10965- 10975.
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for gener- ating descriptive image paragraphs. In Proceedings of the IEFE Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 317-325.
Zejun Li, Zhihao Fan, Huaixiao Tou, Jingjing Chen, Zhongyu Wei, and Xuanjing Huang. 2022d. Mvptr: Multi-level semantic alignment for vision-languee pre- pre-training via multi-stage learning. In Proceedings of the 30th ACM International Conference on Mul- rimedia, MM '22, page 4395-4405, New York, NY, USA. Association for Computing Machinery.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Vi- sual genome: Connecting language and vision us- ing crowdsourced dense image annotations. Int. J. Comput. Vision, 133 73.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Computer Vision ECCV 2014, pages 740 755, Cham. Springer Inter- national Publishing.
Karen Kukich. 1983. Design of a knowledge-based re- port generator. In Proceedings of the 21st Annual Meeting on Association for Computational Linguis- rics, ACI. '83, page 145-150, USA. Association for Computational Linguistics.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier. and Desmond EI- liott. 2021. Visually grounded reasoning across lan- guages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10467-10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ju-Hee Lee and Je- Won Kang. 2022. Relation enhanced vision language pre- pre-training. In 2022 IEEE Interna- tional Conference on Image Processing (ICIP), pages 2286-2294
Soohyeong Lee. Ju-W Whan Kim. Youngmin Oh, and Joo Hyuk Jeon. 2019. Visual question answering over scene graph. In 2019 First International Confer- ence on Graph Computing (GC), pages 45-50.
Fangyu Liu, Guy Edward Toh Emerson, and Nigel Col- lier. 2023. Visual spatial reasoning. Transactions of the Association for Computational Linguistics.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In European Conference on Com- puter Vision.
Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang. Qi Tian, and Sil- iang Tang. 2022a. Fine-gran uineggraieed semantically aligned vision-language pre-training. In Advances in Neural Information Processing Systems.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolin- guistic representations for iision-and-language tasks. In Advances in Neural Information Processing Sys- lems, volume 32. Curran Associates, Inc.
Jumnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In Proceedings of the 40th Interna- tional Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi. and Aniruddha Kembhavi. 2023. UNIFIED-IO: A unified model for vision, language, and multi- modal tasks. In The Eleventh International Conference on Learning Representations.
Junnan Li, Dongxu Li. Caiming Xiong, and Steven Hoi. 2022b. BLIP: Bootstrapping language-image pre- training for unified vision- language understanding and generation. In Proceedings of the 39th Interna- tional Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12888-12900. PMLR.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020. 12-in- 1: Multi-task vision and language representation learning. In Pro- ceedings of the IEEE/CYF Conference on Computer Vision and Pattern Recognition (CVPR).
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation.
Nicole Meister, Dora 7hao, Angelina Wang, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky. 2022. Gender artifacts in visual datasets. arXiv preprint 220.09220

---

Mitja Nikolaus, Emmanuelle Salin, Stephane Ayache, Abdellah Fourtassi, and Benoit Favre. 2022. Do iision-and-language transformers learn grounded predicate-noun dependencies? In Proceedings of the 2022 Conference on Empirical Methods in Nat- urul Language Processing, pages 1538-1555, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Sahand Sharifzadeh, Sina Moayed Baharlou, and Volker Tresp. 2021. Classification by attention: Scene graph classification with prior knowledge. Proceedings of the AAAI Conference on Artificial Intelligence, 35(6):5025-5033
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: A cleaned, hyperyymdd, image alt-text dataset for automatic im- age captioning. Proceedings of the 56th Aunual Meeting of the Association for Computational Lin- guisties (Volume 1: Long Papers), pages 2556-2565. Melbourne, Australia, Association for Computational Linguistics.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. 1m2text: Describing images using 1 million captioned photographs. In Advances in Neural In- formation Processing Systems, volume 24. Curran Associates, Inc.
Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Au- rélie Herbelot, Moin Nabi, Enver Sangineto, and Raf- faella Bernardi. 2017. FOIL it! find one mismatch between image and language caption. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 255-265, Vancouver, Canada. Association for Computational Linguisties.
Letitia Parcalabescu, Michele Cafagna, Lilitta Murad- jan, Anette Frank, lacer Calixto, and Albert Gatt. 2022. VALSE: A maskinndependent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 0253-8280, Dublin, Ireland. Association for Computational Linguistics.
Jiaxin Shi, Hanwang Zhang, and Juanzi Li. 2019. Ex- plainable and explicit visual reasoning over scene graphs. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Julia Peyre, Ivan Laptev, Cordelia Schmid, and Josef Sivie. 2017. Weakly-supervised learning of visual relations. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. 2022. How to train your VIT? Data, aug- mentation, and regularization in vision transformers. Transactions on Machine Learning Research.
Tianwen Qian, Jingjing Chen. Shaoxiang Chen, Bo Wu, and Yu-Gang Jiang. 2022. Scene graph refinement network for visual question answering. IEEE Trans- actions on Multimedia, pages 1-1.
Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang. Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in pho- tographs. In Proceedings of the 57th Anmual Meet- ing of the Association for Computational Linguistics. pages 1418-6428, Florence, Italy. Association for Computational Linguistics.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IICNLP) pages 3100-5111, Hong Kong, China. Association for Com- putational Linguistics.
Brigit Schroeder and Subarna Tripathi. 2020. Structured query- based image retrieval using scene graphs. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) Work- shops.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning. 2015. Gen- erating semantically precise scene graphs from tex- tual descriptions for improved image retrieval. In Proceedings of the Fourth Workshop on Vision and Language, pages 70-80. Lisbon, Portugal. Associa- tion for Computational Linguistics.
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanprect Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference Com- puler Vision and Pattern Recognition (CVPR), pages 5238-5248.
Sahand Sharifzadeh, Sina Moayed Baharlou, Martin Schmiut, Hinrich Schütze, and Volker Tresp. 2022. Improving scene graph classification by exploiting knowledge from texts. Proceedings of the AAAI Con- ference on Artificial Intelligence, 36(2)2188--1199.

---

Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolles, and Herve Je- gou, 2021. Training data-efficient image transform- ers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMIR.
Knowledge enhanced rision-language representa- tions through scene graphs. Proceedings of the AAAI Conference Artificial Intelligence, 35(4):3208 3216.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Ye- ung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. CoCa: Contrastive captioners are image-text foun- dation models. Transactions on Machine Learning Research.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. & ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and why vision-language language models behave like bags-of- words. and what to do about it? In The Eleventh Interna- rional Conference on Learning Representations.
Peng Wang. An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. OFA: Unifying ar- chitectures, tasks, and modalities through simple -aaeeenee----euennee learning framework. In Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23318-23344 PMLR.
Rowan Zellers. Mark Yatskar, Sam Thomson, and Yejin Choi. 2018. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Yan Zeng, Xinsong Zhang, and Hang Li. 2022. Multi- grained vision language pre- pre-training training: Aligning texts with visual concepts. In Proceedings of the 39th International Conference on Machine Learning, vol- ume 162 of Proceedings of Machine Learning Re- search, pages 25994-26000 PMLR.
Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei- Fei. 2017. Scene graph generation by iterative mes- sage passing. In Proceedings of the IEEE Confer- ence Computer Vision and Pattern Recognition (CVPR).
Haotian Zhang. Pengchuan Zhang, Xiaowei Hu, Yen- Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. 2022. GLIPv2: Unifying localization and vision-language understanding. In Advances in Neu- ral Information Processing Systems.
Xu Yang. Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEI/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR).
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li. Xin Jiang, and Chunjing Xu. 2022a. FILIP: Fine- grained interactive language- image pre-training. In International Conference on Learning Representa- tions.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441 1451, Florence. Italy. Association for Compu- tational Linguistics.
Ting Yao. Yingwei Pan, Yehao Li, and Tao Mei. 2018. Exploring visual relationship for image captioning. In Proceedings of the European Conference on Com- puter Vision (ECCV).
Yiwu Zhong. Jianwei Yang, Pengchuan Zhang, Chun- yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. 2022. RegionCLIP: Region- based language- image pretraining. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 16793-16803.
Yuan Yao, Qianyu Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2022b. PEVL: resition-enhanced pre-training and prompt tnning for vision-language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11 104-11117, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
thi-Hua Zhou. 2017. A brief introduction weakly pervised learning. National Science Review, 5(1):44 53.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li. Chunyuan Li, Xiyang Dai. Harkirat Behl, Jian- feng Wang. Lu Yuan, Nanyun Peng. Lijuan Wang. Yong Jae Lee, and Jianfeng Gao. 2023. General- ized decoding for pixel. image, and language. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognition (CVPR), pages 15116-15127
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual demotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE-ViL:

---

A Experimental Setup
Hs Hoperparsuctee
REALBEF
X-VLM
ReXALMM
Leaming nie
Adamo
0.965)
009.099 0:95)
In this section, provide further details the experimental setups that we used for our studies. Our ALBEF and X-VLM models are imple- mented in JAX (Babuschkin et al., 2020) and employ a ViT-B/16 image encoder pretrained on ImageNet-21k (Steiner al., 2022) that processes images with a resolution of 224x224 pixels. AL- BEF models have 212M parameters, while X- VLM models have 21 214M parameters. For MRC, we use a two-layer MLP with a ReLU nonlinear ac- tivation function, further adding 5.7M parameters during pretraining. VSG is parameter-free.
step
5000
5000
Droperr
Eniiicc (VMA/BOO)
Rdiatinn (MBC)
Relations (VSG)
Batch sizes Sampling catics
$12000
5124:128122
1024 1004 15
1:00.0
24:1:1
2:1.50000
211.::1
Table 4: Hyperparameter configurations for best- performing models. # Entities (VMA/BBOX) refers to the number of objects sampled from each image batch compute the VMA and BBOX losses in X- VLM models. '# Relations (MRCY refers to the number of subject-relation relation -bjjcct triplets sampled from each image in batch to compute the MRC loss. '# Relations (VSG)' refers to the number of triplets sampled from each image in batch to compute the VSG loss. Batch sizes and sampling ratios refer to different data types and losses, as cappionseentities
We pretrain our baselines and relation-enhancee models xx2x2 TPUv4 slice for up to 500K steps (5 days). Each model is pretrained once, using the same hyperparameters as the baselines whenever applicable. For VSG, we sample 16 re- lations per image in order to fit within the TPU memory. For MRC, we follow the same setup as for VMA/BBOX in X-VLM, by sampling 4 enti- ties per image (and their 2 corresponding relations). group datasets according their 'type' (i.e., captions, detection or graphs), and sample batches containing data from a sin- gle dataset at a time. Within group, we sample datasets uniformly at random, as this was shown to be more effective for captioning data (Hendricks et al., 2021). We also experimented with sampling VSG and MRC data with a weight of 1.5, but found 1.0 to lead to lower pretraining loss. We use maximum sequence length of 36 text tokens for all tasks except for VSG, for which we use 112 tokens to up to subject-relation--hjjer triplets per caption. For masked language mod- elling tasks, we mask 25% of the text tokens in a caption, ensuring that all tokens that belong to a word are masked. Hyperparameter configurations for best-performing models are listed in Table 4.
ples train X- X-VLMaaMXXVLLM and corre- sponding relation-enhanced models.
B Results
During pretraining,
In this section, we provide complementary results.
B.1 Results by Subtask
Tables 5 to list performance on the subtasks of our fine-grained benchmarks when pretraining our models for a fixed number of steps (see Section 4). On VALSE, we find that REX-VLM models are especially useful to improve understanding of existence, counting (when pretrained on 3M im- ages), spatial relations, actant swap and coreference. Their performance is on par in plurality, but note that the 1LBEF base- line tops all other models on the coreference and Foil-it! subtasks.
On VSR, we observe significant, consistent gains of REX-VLM models in adjacency and projective relations. REX-VLM addition- ally boosts topological relations, while REX- VLM boosts directional relations. When learning relations on top of ALBEF, we observe similar trends for REALBEF 13M but to a slightly smaller degree, indicating that it is helpful to learn object locations to better understand relationships between objects.
We typically report performance after (i) 200K steps when training on 3M images, and (ii) 500K steps when training on 13M images. Compared to the total number of data points seen throughout pretraining in the original papers, our models are typically trained on fewer examples. Li et al. (2021) trained ALBEF AM/ALBEF 14M on 154.5/456M samples, while we use 102.5/256M samples to train ALBEFss/ALLEE: I3M and corresponding relation-enhanced models. Zeng et al. (2022)
On SVO-Probes, REX-VLMISM gains + +lpp on subject and object understanding, but less on verb understanding, compared to X-VLM 13M- The gains for subject understanding are even larger for REALBEF 13 with respect to ALBEF
trained X-VLM 4M/X-V/LLII 16M on approximately 315/921.5M samples, while we use 205/51 12M sam-

---

Existence quantifiers
Plurality numher
Counting sns.f
Sp.rel.f relations
Action actant swap
Coreference
Model
Foil-it!
Avg.
halanced
adv.f
repl.f
standard
clean
Random
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.00
CLIP
66.9
56.2
62.1
62.5
57.5
64.3
75.6
68.6
52.1
49.7
88.8
64.0
BLIP-2
83.6
79.6
70.2
68.7
68.0
65.6
84.4
63.2
62.6
58.7
96.0
74.0
ALBEFAM
77.0
77.4
64.2
66.9
53.3
77.0
71.0
55.6
57.9
57.7
95.3
69.4
REALBEFMM
84.0
73.0
65.6
66.6
66.1
70.7
71,8
52.2
56.5
58.7
94.3
69.6
X-VLMaM
79.6
77.3
65.1
67.6
55.7
76.3
73.6
50.8
58.2
51.9
95.4
69.5
RDX-VLMMM
82.6
76.9
66.6
69.9
67.1
76.4
69.9
52.6
55.5
65.4
95.2
70.9
ALBEF 3M
75.4
78.0
68.1
70.7
68.0
76.4
74.8
55.7
60.6
61.5
96.1
72.2
REALBEFFSM
74.9
77.8
67.7
68.8
64.1
75.9
72.7
53.0
56.4
55.8
95.2
70.4
X-VLM
74.7
79.2
65.4
69.2
73.2
75.7
74.8
53.5
54.8
51.9
95.8
71.3
REX- ML
87.3
78.0
69.7
69.9
72.5
79.4
74.7
56.7
56.6
55.8
95.0
73.3
Table 5: Performance on the VALSE benchmark according to pairwise ranking accuracy, Best results are in bold. ssss. Counting small numbers. adv. Counting adversarial. repl. Action replacement. $ Sp.rel. Spatial relations.
Model
Adjacency Directional
Orientation
Projective
Proximity
Topological
Unallocated
Overall
Random
50.0 50.0
50.0 50.0
50.0/50.0
50.0/50.00
50.0/50.0
50.0 50.0
50.0/50.0
50.0/ 50.0
BLIP-2
59.8/54.9
50.0/43.3
52.5/57.1
59.8/
56.2/
66.4/
75.0/ 66.7
61.2/
ALBEFTM
55.6
45.5 / 42.2
67.8 1/88/66.2 56.2
64.2/62.7
56.2 / 52.0
69.8/65.0
71.9/47.J
63.7 / 60.1
REALBEFaM
55.3/
54.5 48.
63.4/
66.6 64.0
59.4/ 54.5
66.4/ 62.9
68.8 / 60.8
64.0/ 60.2
X-VLMaM
50.0/43, /3.3 56.1/54.9
57.1
63.0/ 66.6
60.9 55.3
69.5 66.0
68.8 / 56.9
63.5/ 62.3
REX- MMM
59.8/58.1
56.8 / 48.99
/5.3
/7.1/
56.2 / 55.3
67.8/ 62.4
75.0 72.5
65.0 61.8
ALBEF
54.5/56.7
45.5
61.0/57.1
57.8/ 61.1/60.5
64.1 14.1/66 64.6
61.6/
60.4 59.4
REALBEF13M
56.8/56.7
/3..3
66.1
64.8/65.2
64.1 / 52.8
66.8 64.0
87.5 / 56.9
61.3
X-VLM
45.5 56.8/58.8
66.1
56.2 61.9/612
55.3
64.1 64.5
62.5 56.9
61.1 60.5
REX- LLMM
67.4/ 60.2
50.0/47.5
67.8/533
68.4/ 67.0
60.9 / 56.1
72.5/ 67.3
75.0/ 51.0
68.4 63.5
Table 6: Dev/Test results the VSR Random dataset. Best results in bold.
Model
Subj.
Verb
Obj.
Avg.
B.2
Pretraining Dynamics
Random
50.0
50.0
50.0
50.0
Bugliarello et al. (2023) showed that current, strong models achieve peak performance on different fine- grained tasks at different stages of pretraining. This motivates us to study the pretraining dynamics of our models. Performance for models pretrained on 3M and 13M images is shown in Figures 8 and 9.
CLIP (VIT-B/32)
83.6
79.0
88.1
81.6
BLIP-2
87.6
84.6
91.7
86.5
ALBEFAM
87.3
84.6
92.2
86.6
REALBEFaM
87.8
83.5
93.0
86.2
X-VLMsM
88.8
85.3
92.3
87.3
REX-VLMMM
88.2
85.2
92.8
87.3
ALBEF
86.9
84.9
92.0
86.7
We see that models performance, especially of our coarse-grand coarse-grained baselines, tends to fluctuate con- siderably on VSR tasks. For instance, X- VLM 's accuracy VSR Dev decreases during pretraining. Looking at relation-enhance models, we find that they benefit from more training steps than the base- lines. For instance, when pretrained on 3M images. they achieve peak fine-grained results after 350K- 400K steps, while ALBEF and X- X-VLM do so within 200K steps (which is where we evalu- ate our models in Section 4). This is even more relevant when pretraining 13M images, where our baselines' performance starts dropping after 250K steps, while our models are still improving at 500K steps. Longer pretraining and designing bet- ter schedules that balance coarse- and fine- ine-grained tasks, and the different subtasks are promising di-
REALBEFIRM
88.8
85.2
93.4
87.5
X-VLM
88.1
85.5
92.3
87.3
REX- (M
89.1
86.1
93.3
88.1
Table 7: Performance on the SVO-Probes benchmark according pairwise ranking accuracy. Best results in bold.
(+1.9/1.4p) for subject/object understanding). However, these improvements are smaller when training on 3M images, likely duc to our relation- enhanced models requiring longer training to achieve top performance (see App. B.2). Overall, we note that verb understanding still the most challenging aspect of SVO- WVO-Probes and that relation- enhanced models improve less for this subtask.

---

VALSE
VSR Avg.
VSIK
SNO Probes
Stunfeet
63
67
91
54
14
Stanfond hergrabs lest
PiikkroK Dev 1Ri31
FiveriiK Yeat TRgi
0O0O Dev TR
COCO Teat Titt
D
ALDEE
54-
Figure 8: Pretraining dynamics of our models when learning from 3M images.
VALSE Avg. Acty
VSR Dev Asp. Ace
VSR Tes Ave. Ace
SVD-Puobs Avp. Anc-
Stanfied Pararrals Day TRoi
51-
30
50
53-
Stanford Paragrass Test TRig1
Piikrr0K Dev 1Ri51
Flkckr3KK Test TRg!!
0000 Dev 1R
COCO Tea Iatt
75
AM
59
16
201
4x8
200
400
200R
4006
206K
402x
Figure 9: Pretraining dynamics of models when learning from 13M images.
rections for future work to obtain a single check- point that performs well on both types of tasks. Finally, Figures 12 15 show performance for our two proposed approaches when applied inde- pendently and together during pretraining of AL- BEF and X- L-VLM models on 3M and 13M im- ages. On VSR, relation-enhancee models gen- erally reach peak performance when combining both VSG and MRC. On VALSE, their perfor- mance degrades with respect the baselines when using VSG alone. Moreover, looking at coarse- grained retrieval tasks throughout pretraining, we see that VSG degrades performance whist MRC can achieve on par or superior performance than the baselines. Interestingly, when combined, the final performance is closer to the stronger MRC objective.
Table lists the performance our models when performing checkpoint selection based on COCO Dev TR@1 1. Figure 10 lists the individ- ual gains/losses of our models on each evaluation task according to different checkpoint selection strategies, when comparing against the standard ap- proach of using the last checkpoint (200K steps for models trained on 3M images, and 500K steps for models trained on 14M images). Finally, Figure reports the Spearman rank correlation coefficients between COCO Dev TR and the other strate- gies, across all models. Here, the typical high coef- ficients indicate that the order with which models ranked for given task according to any strat- egy is mostly the same. That is, our findings from Section 4 hold regardless of the chosen checkpoint selection strategy.
B.3 Checkpoint Selection Strategies
As discussed in Section 5 and shown in Figures 8 and 9, there difference in convergence rates between relation-enhanced models and coarse- grained ones, with our models often requiring more steps to achieve peak performance. Here, we aim at complementing our discussion from Section 5.

---

VSR Random
VALSE
SVO-Probes Ace,
Stanford Paragraphs IR 1/5 TR 1/5
Flickr30K
COCO
Model
Dev / Test Acc
Acc,
IR@1/
TR@1/
IR@1X 1/5
T/@115
CLIP
N/A 61.2/61.5
64.0
81.6
45.3/73.1
53.4 80.1 83.4/95.2
88.07
/90.6
58.4/815
37.8/6244
BLIP-212M
74.0
86.5
81.1 /94.3
95.5/
/6.7/97.1
80.7
64.2/85.2
FLB
61.4/60.3
68.9
87.4
77.9/ 94.5
79.8 /4.77
78.4
61.2/ 84,6
63.9
86.5 86.5
47./74.9
REALBEFSM
65.5/ 60.2
70.8
86.7
85,7/ 97.6
85.8/97. /9.3
76.5/93.0
61.7 84.0
63.9/86. 86,7
77.3/
X-VLMaM
64.0/ 61.7
68.5
87.0
80.4 /0.4/94.9
80.2 95.00
80.4 94.8
63.7 / 86.4
66.4/87.8
76633
REX- VLM
66.6/ 62.6
71.2
87.2
87.1/ / 98.0
88.3 17.4
77.6 94,4
61.8/ 84,3
66.8/88.2
88.2
49.4/6..2
ALBEF IM
61.4/613
71.4
87.7
82.3 96.1
/9.6/
14.1/99.6
67.3/ 87.6
49.7 66.8
76.5
REALBEF13M
63.5/61.5
71.1
87.7
87.1 / 97.5
86.3 97.2
80.3 93.6
64.9 / 85.6
66.5 87.9
48.8/75.9
X-VLMISM
62,5
72.2
87.7
84.7 / 96.8
85.4/ 96.3
84.2/ 96.6
68.0
87.8
67.3 88.4
50.7 76.9
REX- MM
/
73.5
88.0
89.9 / 97.8 88.7 98.0
81.1 94.8 65.7 / 87.3 67.41 89.0
50.3/ 77.3
Table 8: Overall results fine- (left) and coarse- grarine (right) benchmarks. Models are evaluated at best COCO Dev TR@ I. Values underlined in green (red) denote gains (losses) of relation-enhanced models on their baselines. CCIP cannot directly evaluated on VSR since requires true/false predictions for given image-text input, while CLIP is only trained with contrastive loss. Best results are in bold.
ALBEF
X-VLMSH
ReALBEF
ReX-VLM3H
P30K Text ERABT
4.1
-19
016
A1
60
AL
13
53
19
13
30
22
15
L5
22
22
L2
0.0
0.
43
0.0
-4.
43
130K Test TR@1
33
22
1.4
12
12
95
05
4.1
1R601
4..
081
26
-15
1.2
15
92
9.
0%
4.3
VALSE Avg. Acey
444
-26
40.9
404
50
466
1.0
5.
-10
-1.
0.7
38
1.1
0.5
0.5
11
12
0.7
0.0
0.
611
05
04
4.5
SVO Avg. Ace,
-411
00
-33
-00
05
0.
-60
Ang.Aoe-
455
4.6
60
-36
3.0
-36
0.5
0.5
40
4.2
0.9
0.0
95
08
6.
ALBBF
X-VLM11H
ReALBEF LOW
ReX-VLMy3M
.09
F50K The TR&1
4:3
411
9.
65
4.1
4.3
05
67
14
1.4
37
-33
3.9
05
3.5
-3.0
0.8
0.5
68
0%
6.3
com Tea (Rgg
85
6.6
65
1.2
13
1.3
43
40
42
3.4
-12
011
01
01
4.3
0000 The TR@i
05.
Aug.
111
58
41
04
SV0 Aug Ase
666
02
06
86
03
24
55
30
41
60
30
36
4.6
0.5
40
-51
0.
VSR Dev Aug. Ace
1.5
1.0
.08
-17
461
4.1
VSR Test Ace
5
92
35
13OK be Ieat
TRgU
COCO be 1Ral
TRaI
VALSE Ave Abe
FIOK Day TRai TeeT
COCO De 1R001
CoCL Day TRaI
SVO AMC
E30K Des ER001
FA0K Day TRBI
COCO Dos DRgU
COCO De 1381
SVO Auc-
DOO De TR8I
COCO IRQU
CCCO D. TRQU
FBOK Dos
SVO
SVD
VALSE
VALSO
Figure 10: Performance differences (y-axis) with respect to fixed checkpoints for all models according to different
checkpoint selection tasks (x-axis).
K5
79.6
926
X5.7
77.
62.6
COCO 21
90.5
$1.0
952
100.
92.9
95.2
90.5
83.3
65.7
853
YALABANG
92.9
22.9
26.2
78.5
95.2
85.7
92.6
VSL.er Ang.
51.5
57.1
92.9
905
18.1
92.2
97.6
VS8
68.1
Heek
VAtSc.4g..a Apu
Amn
210 Aug Am
2546 Ien 803
2964 mM87
Figure II: Spearman rank correlation coefficients of different checkpoint selection tasks (z-axis) with using COCO Dev TR @ for our evaluation tasks (x-axis).

---

VALSE
Staford Parggaba Dee TR
Stafood Parggabt Dev IR@)1
Preagrad TR()
Panyrah [R
32
FlickA0KK TRiil
Ficck10K Text IR
0000 Teat TRGil
COCD Tesl IRG0
0
48-
5lckk50K Dev TRoc1
Flckk50K Dev IR301
DOCD TRA01
COCD Dev IRA01
0-0
50
46
0
46
56-
44
10
05
271K
100K
000
593K
065
1900.
200K
2006
4005
500k
06.
1900
2008
2005
403
200K
590K 00.
100K
2006
JUOK
Figure 12: Pretraining dynamics of our approaches on ALBEF models pretrained on 3M images.
VALSE Avp. Ace.
VSR Dev Avp. Ase
VSR Tea Avg. Are
SVG Probes Avg. Acc.
60
Stanfoodd
Stunfood
Stanford TR301
Stanford o
10
76
74
72
RR/01
TRRU
IR.b
.o.......
66
MRC ReALaCrue
76
Fickr30K Dev TR@8
Fickr30K Dev IR/p
COCO Dex TR@8
COCO Dev IR81
44
Figure 13: Pretraining dynamics of our approaches on ALBEF models pretrained on images.

---

VALSE
66
62
41
50
Stafoed Pargraba Dee TR@)
Saaford Parggabt Dee IR@)
Paragaae TRta
Panyrass [R
85-
FiskrAKK TRit
Fickr1kK Teat IR
COCD Teat TRGe
COCD Tisi TRG01
66
76-
30-
5OKck Dev TRocl
Fick:S0K Dev IR30]
DOCD TRA01
COCD Dev IRA01
70
62
36
05
271K
200K
0005
593K
065
1900
200K
2006
4005
500k
06
190K
2008
2006
403K
200K
60
100K
2006
JU0K
590K
Figure 14: Pretraining dynamics of our approaches on X-VLM models pretrained on 3M images.
VALSE Avp. Ace-
VSR Dev Avp Ane
VSR Tel Arg. Are
SVC-Prober Avg.Aos Acc.
67
60-
59
Stanfoodd
Stunfood
Starford TR301
Stanford
100
36
10
TR
TRiU
16
58
Fic Des TR@8
Fickr30K RR/e
COCO Dex TR@8
COCO Dev IR,81
66
38-
Figure 15: Pretraining dynamics of our approaches on X-VLM models pretrained on 13M images.


Reviews:
[{"review_id": "mE4TmgJUmo", "reviewer": "Reviewer_JtYm", "summary": "", "questions": "See the reasons to reject", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our paper! We’re glad to see that you found our experiments solid and extensive, and that our approach is important for the scene graph area. Indeed, we are the first to show the effectiveness of using scene graphs to improve fine-grained understanding in pretrained vision-and-language models (VLMs), and hope future work will investigate this further.\n\n> Why do we pre-train the model in a weakly-supervised manner yet providing some prior information on the locations (the verbalized sentences are sorted based on locations). Would the authors like to explain why we remove the exact mapping information but reserve the location prior?\n\nWe are not sure we completely understand the ablation you are proposing, are you suggesting to use objects’ bbox coordinates and their relations in the same loss? \n\nIf that’s the case, our MRC loss implicitly does that by visually masking two entities separately, and then trying to predict their relation (see Fig2 right), where the visual mask is computed from the objects’ bboxes. \n\nIn the VSG loss, we simply use the bbox coordinates to order the visual content in a scene. Yet, our ablation in Fig5 shows that sorting based on bbox coordinates is not necessary to obtain strong representations. It is also true that we do not use such fine-grained mapping between the objects and their locations in the VSG loss (which is an interesting idea for follow-up work!). Still, we also remind that our main baseline model (X-VLM) already modelled bbox locations during pretraining.\n\nWe investigate the benefits of learning visual relations between entities for fine-grained skills in VLMs. The comparisons between ALBEF and X-VLM models disentangles the information of modelling object locations. Interestingly, in L399-422, we find that modelling relations alone is more effective than object localisation when training VLMs at scale. \n\nPlease let us know in the discussion phase if we misunderstood your concern, and we’d be happy to engage further to clarify and answer your questions.\n\n> Reproducibility\n\nIn App A, we meticulously made sure to provide information for all criteria listed in the EMNLP 2023 reproducibility checklist (https://2023.emnlp.org/calls/main_conference_papers/#reproducibility-criteria). The codes for ALBEF and X-VLM are also open source. We use the same evaluation data as Bugliarello+(ACL’23), which is publicly available at https://github.com/e-bug/fine-grained-evals. The pretraining data is also publicly available and it consists of standard datasets that have been used by the community (COCO, SBU, Visual Genome, Conceptual Captions). We hope this resolves your concerns and are happy to provide additional details if needed.\n\n> Excitement\n\nWe hope that our response addresses your concerns with our study, though we would also appreciate clarification on how our work will \"significantly benefit from another round of revision\" that is reflected in the excitement score. From our understanding, the main concern is a lack of an ablation (which we think is not a “key weakness” that motivates a score of 3). We are happy to include additional missing ablations in the extra page if you can clarify what is needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments are solid and extensive (although misses some discussions, which I will explained in the reasons to reject)\n\n2. The contribution of this paper to scene graph area is important.", "reasons_to_reject": "I want to see more discussions of ablation study. More specifically, I am very curious about the strategy design. Why do we pre-train the model in a weakly-supervised manner yet providing some prior information on the locations (the verbalized sentences are sorted based on locations). Would the authors like to explain why we remove the exact mapping information but reserve the location prior? I did not see the ablation study.", "questions_to_authors": "See the reasons to reject", "paper_topic_and_main_contributions": "This paper studies the effect of visual relation in multi-modal pretraining. More specifically, the authors propose a pre-training strategy to include the visual relation annotations together with natural language. The extensive experiments show that the proposed strategy can boost the VLM trained on vision-language annotation.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IK6H3HJChE", "reviewer": "Reviewer_VUQu", "summary": "", "questions": "1. In section 2, what does l_{i} mean?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our paper! We’re happy that you found our analysis comprehensive, and our strategy to be versatile and effective, as demonstrated through rigorous testing on several benchmarks. We agree that emphasising visual relations is pivotal for a nuanced understanding, and we hope that others, like you, will gain valuable insights into the task's intricacies from our paper!\n\n> Visual scene graphs have been widely used not only in the vision but visual-language domains.\n\nWe agree! We provided a few relevant citations for usage of scene graphs in L057-065, with further emphasis on vision-and-language domains in L522-544. However, they have been understudied for multimodal pretraining. This is where our work fills the gap. We demonstrate, for the first time, that adding scene graphs for a small number of images on top of millions of image–text pairs from the Web results in improved coarse-grained and fine-grained skills.\n\nWe are only aware of two previous studies that investigated the use of scene graphs for vision-and-language pretraining. In addition to the core question that we investigate in this paper (intrinsic performance in both coarse- and fine-grained tasks), we highlight the difference in methodology in L530-544. \n\nIf you have specific references that you think are missing, we’d be happy to integrate them in our revision. \n\n> Without datasets, code of methods and checkpoints, it is hard to verify the results reported in the paper.\n\nWe use the same evaluation data as Bugliarello+(ACL’23), which is publicly available at https://github.com/e-bug/fine-grained-evals. The pretraining data is also publicly available, and it consists of the standard datasets that have been used by the community (COCO, SBU, Visual Genome, Conceptual Captions). The codes for ALBEF and X-VLM are also open source, and we think that the details in the paper (see App A) should be enough to reproduce our method. We would, of course, be happy to integrate any details that you think are missing. We will reply to any further inquiry by the community should the paper be accepted, and we will try to release our checkpoints.  \n\n> In section 2, what does l_{i} mean?\n\nl_{i} is the label (e.g., car, duck) for the entity e_{i}. Thank you for pointing this out, we’ll make it clear when we revise our paper.\n\n> Reproducibility\n\nIn addition to the clarification of datasets and code above, in App A, we meticulously made sure to provide information for all criteria listed in the EMNLP 2023 reproducibility checklist (https://2023.emnlp.org/calls/main_conference_papers/#reproducibility-criteria). We hope this weakens your concerns on being able to reproduce our results.\n\n> Excitement\n\nWe hope that our response clarifies your concerns with the position and reproducibility of our study, and we’d be happy to engage further during the discussion phase. We are delighted that you found our work important and thorough. We did not find any major concerns and suggestions that would make our paper “significantly benefit from another round of revision.” So we hope you’ll consider increasing your score according to your overall recommendation, thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Emphasizing visual relations is pivotal for a nuanced understanding in Visual Language Pretraining (VLP).\n- Leveraging verbalized scene graphs proves effective during pre-training. This strategy is versatile, allowing its integration into other VLP models, as showcased in the adaptation of the ALBEF loss function in this research.\n- The proposed methodology's efficacy is demonstrated through rigorous testing on several benchmark datasets. Coarse-grained retrieval assessments utilized the Flickr30K and COCO datasets. In contrast, fine-grained evaluations employed datasets like VSR, VALSE, SVO-Probes, and Stanford Paragraphs. The subsequent analysis in the results section offers valuable insights into the task's intricacies.", "reasons_to_reject": "- Visual scene graphs have been widely used not only in the vision but visual-language domains.\n- Without datasets, code of methods and checkpoints, it is hard to verify the results reported in the paper.", "questions_to_authors": "1. In section 2, what does l_{i} mean?", "paper_topic_and_main_contributions": "This paper introduces a weakly-supervised learning approach for enhancing visual relations in multimodal pretraining by leveraging verbalized scene graphs. To optimize the learning of visual relations, the method builds upon the loss function introduced by ALBEF, adapting it to harness the structured information from the verbalized scene graphs. Additionally, the research introduces a masked relation classification as a supplementary loss function to further improve the learning process.\n\nThe effectiveness of the proposed approach is empirically validated across various benchmark datasets. For coarse-grained retrieval, the experiments were conducted on Flickr30K and COCO datasets, while for fine-grained retrieval, datasets such as VSR, VALSE, SVO-Probes, and Stanford Paragraphs were used. The results section provides a comprehensive analysis, shedding light on the nuances of the task and highlighting the pivotal insights drawn from the experiments.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XZ23goJTbG", "reviewer": "Reviewer_tnr9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our paper! We’re happy that you found our work well-motivated, with a detailed analysis, and with extensive experiments that show the effectiveness of our novel method.\n\n> Visual representation is missing to demonstrate whether the model indeed possesses the ability for fine-grained comprehension.\n\nThis is a good point, thank you for raising it! We did not include any visual representations for two main reasons.\n\nFirst, VALSE and SVO-Probes are evaluated through foils, which contain only a single difference between the true and false pairs (e.g., only a word differs between true and false captions). That is, the only way that a model can succeed is indeed to correctly understand that difference. Hence, we felt that the quantitative results obtained through these fine-grained benchmarks were enough to highlight the fine-grained abilities gained by the models.\n\nSecond, it is not trivial to demonstrate fine-grained comprehension in VSR and Stanford Paragraphs. For instance, VSR focuses on relation understanding (e.g., above, next to), and it is not clear to us what would be the expected behaviour (Should the model pay attention to both objects? Should it focus on the contact point?).\n\nHaving said that, if you have specific suggestions, we’d be happy to include them in the extra page that we would be granted should the paper be accepted. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work is well-motivated and has extensive experiments to show the priority and effectiveness over several recent models. The proposed method is novel and effective. And the analysis of the experiments are detailed.", "reasons_to_reject": "Visual representation is missing to demonstrate whether the model indeed possesses the ability for fine-grained comprehension.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on multimodal pretraining for fine-grained multimodal understanding. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7EqGlyokCz", "reviewer": "Reviewer_y9T8", "summary": "", "questions": "- The relative decrease in the performance of REALBEF 3M compared to ALBEF 3M across most coarse-grained tasks suggests that the performance of the method in this paper might be influenced by the model architecture and scale. How do you view this phenomenon?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our paper! We’re glad that you found it well-written, with comprehensive experiments and analysis that support state-of-the-art performance on diverse fine-grained benchmarks from a relatively small amount of data.\n\n> The experiments primarily focus on tasks related to scene graphs, which results in limited coverage of other multi-modal tasks.\n\nWe note that none of the tasks that we evaluate models on require scene graph understanding or generation. We hypothesised (and later verified) that successful modelling of scene graphs could lead to improved visual relation understanding on VSR. Moreover, we show that our methods improve general fine-grained vision-and-language understanding, while keeping competitive performance on coarse-grained tasks. The fine-grained tasks that we benchmark on cover a wide range of multimodal abilities. For instance, SVO-Probes measures verb understanding, and VALSE covers six different linguistic phenomena: existence, counting, plurality, relations, actions and coreference. We provide a breakdown of performance across each subtask in App B.1.\n\n> The baseline and backbone models might be somewhat outdated, given the current proliferation of multi-modal models based on LLaMA.\n\nWe would like to point out that Bugliarello+(ACL’23) showed that X-VLM is the current state-of-the-art model for fine-grained understanding, outperforming LLM-based models like Flamingo and BLIP-2. As such, we simply reported the performance of the best *concurrent* work (BLIP-2; ICML’23) as a reference for the reader.\n\nWe also want to stress that multimodal extensions of LLaMa [1, 2] are *concurrent* work and, according to *ACL policies (last bullet point of https://www.aclweb.org/adminwiki/index.php/ACL_Policies_for_Submission,_Review_and_Citation) lack of comparison against them should not be considered a valid reason for rejection. \n\nMoreover, our work aims at improving strong multimodal models. As Reviewer VUQu stressed, our VSG data-to-text strategy is versatile, and it can seamlessly be applied to LLaMA-based models too, as well as any other large model.\n\nHaving said that, if you believe that adding a specific baseline would be a nice-to-have addition to our results, we will include it when we revise our paper should it get accepted.\n\n> The connection between weak supervision and the motivation behind this paper is not very close as the proposed method distinctly differs from some traditional understandings of weakly supervised approaches. A more accurate description might be a multi-modal pretraining method suited for low-resource scenarios.\n\nWe note that, unfortunately, the term “weakly-supervised learning” is overloaded. Our usage aligns with one type of weak supervision: incomplete supervision, defined in [3] as “learning where only a subset of training data is given with labels.” We refer to [3] for a relevant review of weakly-supervised learning.\n\nIf you and other reviewers, however, think that the title is misleading, we’ll consider updating it accordingly.\n\n> The relative decrease in the performance of REALBEF 3M compared to ALBEF 3M across most coarse-grained tasks suggests that the performance of the method in this paper might be influenced by the model architecture and scale. How do you view this phenomenon?\n\nGood observation! We find that this is due to suboptimal data mixing. In fact, as shown in Figure 9 and 12 in the Appendix, ReALBEF_3M matches or outperforms ALBEF_3M when trained for more steps. We will revise our Conclusion to highlight this point, and encourage future work to investigate this multi-task learning problem further.\n\n> Line 614 in the section of Limitations has a typo \"??\".\n\nThank you, we’ll fix it!\n\n> Excitement\n\nWe hope that our response helped clarify your concerns. (1) None of our evaluation tasks is related to scene graphs. (2) LLaMa-based models are concurrent to us, and do not present any distinctive fine-grained abilities. (3) We use “weakly-supervised learning” in a similar way as the literature. If you no longer think that our paper would “significantly benefit from another round of revision,” we’d be thrilled if you considered increasing your score accordingly.\n\nWe’ll be happy to engage further during the discussion phase, thank you!\n\n---\n\n[1] Zhang et al. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. (2023). arXiv preprint 2303.16199\n\n[2] Gao et al. LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model. (2023). arXiv preprint 2304.15010\n\n[3] Zhou. A brief introduction to weakly supervised learning. (2018). National Science Review. https://doi.org/10.1093/nsr/nwx106", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- By adding masked relation classification loss in multimodal pre-training, the proposed method achieves state-of-the-art performance on diverse fine-grained benchmarks.\n- The proposed method achieves significant improvements with a relatively small amount of data.\n- The paper is well-written. It seems that experiments and analysis are comprehensive and the figures and tables are visually appealing.", "reasons_to_reject": "- The experiments primarily focus on tasks related to scene graphs, which results in limited coverage of other multi-modal tasks.\n- The baseline and backbone models might be somewhat outdated, given the current proliferation of multi-modal models based on LLaMA.\n- The connection between weak supervision and the motivation behind this paper is not very close as the proposed method distinctly differs from some traditional understandings of weakly supervised approaches. A more accurate description might be a multi-modal pretraining method suited for *low-resource scenarios*.", "questions_to_authors": "- The relative decrease in the performance of REALBEF 3M compared to ALBEF 3M across most coarse-grained tasks suggests that the performance of the method in this paper might be influenced by the model architecture and scale. How do you view this phenomenon?", "paper_topic_and_main_contributions": "This paper introduces supervised visual relation data into the process of multimodal pre-training to enhance the fine-grained multimodal understanding ability of VLMs. Specifically, the pre-training is divided into two parts: 1) *verbalized scene graphs* applies the standard multimodal pre-training objectives used on image captions to verbalized scene graph annotations; 2) *masked relation classification* predicts the relation from a pre-defined relation vocabulary between two entities. Experimental results show the proposed method can improve the VLMs' visual spatial reasoning and fine-grained understanding capabilities.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 614 in the section of Limitations has a typo \"??\"."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on enhancing fine-grained multimodal understanding through a weakly-supervised learning approach that incorporates visual relation annotations into multimodal pretraining. The method introduces a novel strategy, including verbalized scene graphs and masked relation classification, and demonstrates its effectiveness through extensive experiments on benchmark datasets. Despite some minor concerns, reviewers unanimously praised the soundness of this work with excitement. Overall, the paper makes a valuable contribution to the field of multimodal pretraining, and can benefit from addressing the limitations and providing a more extensive discussion of its applicability to various multimodal tasks in the camera ready version.
Decision: Accept-Main