Incorporating Structured Representations into Pretrained Vision \& Language Models Using Scene Graphs
Authors: Roei Herzig
Alon Mendelson
Leonid Karlinsky
Assaf Arbelle
Rogerio Feris
Trevor Darrell
Amir Globerson
Venue: EMNLP 2023 Main (2023)
Keywords: Vision and language models; Scene graphs; Visio-linguistic compositionality
TL;DR: We propose to improve pretrained VLMs by designing a specialized architecture and a new training method that utilizes a small set of scene graph annotations from Visual Genome that are richer and reflect structured visual and textual information.

Abstract: Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. However, recent works have shown that even the best VLMs struggle to capture aspects of compositional scene understanding, such as object attributes, relations, and action states. In contrast, obtaining structured annotations, such as scene graphs (SGs), that could improve these models is time-consuming and costly, and thus cannot be used on a large scale. Here we ask whether small SG datasets can provide sufficient information for enhancing structured understanding of pretrained VLMs. We show that it is indeed possible to improve VLMs when learning from SGs by integrating components that incorporate structured information into both visual and textual representations. For the visual side, we incorporate a special ''SG Component'' in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene. Our method improves the performance of several popular VLMs on multiple VL datasets with only a mild degradation in ZS capabilities.

Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs
Roei Herzig 1.3, Alon Mendelson" Leonid Karlinsky", Assaf Arbelle Rogerio Feris', Trevor Darrell', Amir Globerson
Tel-Aviv University, 2UC Berkeley, 3 SIBM Research, "MIT-IBM Watson AI Lab
Abstract
has been explored extensively in many previous works (Xu et al., 2017; Herzig et al., 2020; Yang et al., 2022). In particular, datasets with seenc graph (SG) annotations, such as the Visual Genome (VG) dataset (Krishna et al., 2017), have been col- lected and used to improve scene understanding models. However, such datasets are expensive to collect at scale and relatively small compared those used in training VLMs. This raises the fol- lowing questions: (1) Can small datasets containing SG annotations be utilized finetune VLMs and improve compositional scene understanding? (2) How should the model and training be adapted to best use this data? Here we show that it is indeed possible to improve VLMs using image-SG pairs by integrating components that incorporate struc- ture into both visual and textual representations.
Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) per- formance a variety tasks. However, recent works have shown that even the best VLMs struggle to capture aspects of compositional scene understanding, such object attributes, relations, and action states. In contrast, ob- taining structured annotations, such as scene graphs (SGs), that could improve these models is time-consuming and costly, and thus cannot used on large scale. Here we ask whether small SG datasets can provide sufficient infor- mation for enhancing structured understanding of pretrained VLMs. We show that it is indeed possible to improve VL.Ms when learning from SGs by integrating components that incorporate structured information into both visual and tex- tual representations. For the visual side, we in- corporate a special "SG Component" the im- age transformer trained predict SG informa- tion, while for the textual side, we utilize SGs generate fine-gre mne-ggained captions that highlight different compositional aspects of the scene. Our method improves the performance of sev- eral popular VLMs on multiple VL datasets with only mild degradation in 7S capabilities.
Our first step is to convert SGs into highly de- tailed captions. A naive approach would be to finetune VLMs on these image-text pairs, however, we have found that this approach does not suffi- ciently improve performance." This is also aligned with recent work (Doveh et al., 2022; Yuksekgonul et al., 2023), showing that contrastive learning ap- proaches allow the model to concentrate mainly on object labels disregarding other important as- pects, such as relations and attributes. To alleviate this issue, we take inspiration from these works and use the SG to generate hard-negative captions that highlight structural aspects. For example, if an SG contains edge "dog--hassingcate then we can reverse that edge into "cat-chasing-doo and generate corresponding negative caption.
1 Introduction
In recent years, vision and language models (VLMs) such as CLIP (Radford et al., 2021) have demonstrated impressive results and extraordinary zero-shot capabilities when trained massive datasets containing image-text pairs (e.g., LAION 400M Schuhmann et al., 2021). However, recent empirical studies (Thrush et al., 2022: Yuksekgonul et al., 2023; Ma et al., 2022) have shown that even the strongest VLMs struggle to perform composi- tional scene understanding, including identifying object attributes and inter-object relations.
Next, we turn to introduce structure into the vi- sual representation. Inspired by prompt learning approaches (Jia et al., 2022; Herzig et al., 2022a: Zhou et al., 2022a), incorporate into the im- age transformer encoder a of "Adaptive Scene
Understanding the structure of visual scenes is fundamental problem in machine perception and
'VG contains 100K image-SG pairs, which x1000 smaller than the large-scale VLMs pretraining datasets. 2Scc our ablations in section 4.5.
"quul contribution.

---

Visual Genome image Scene Graph Pairs
Vision and Language Contrastive Learning
Scene Graph Prediction
LATON Image-Text Pairs
Pretrained VL Model
Irishre
Text Trommer Encoder
Image Transformer Encoder
"An outdoor walkway
Pasitive Negative
grass covered
Caption Generation
Adaptive Scene Graph
hill"
Graph
Image
Tokenss
Image-Text Pairs
Image-Scene Graph Pairs
Figure 1: Scene Graphs Improve Pretrained Linguae Models. Vision and Language models (VLMs) are typically trained on large-scale image-text pairs (Left). propose to improve pretrained VLMs by utilizing a small set of scene graph (SG) annotations (Right) from the Visual Genome dataset, that is richer reflects structured visual and textual information. Specifically, we design a specialized model architecture and a new finetuning scheme when learning from SGs as follows: (i) Use the graph to generate fine-grained positive and negative captions that highlight different compositional aspects the scene, (ii) Finetune the pretrained model using contrastive learning with the generated captions from the SGss, along with image-text pairs from LAION. (iii) Predict SG information (object, relations, and their coordinates) from an image by incorporating "Adaptive Scene Graph Tokens" into the image transformer encoder. During inference, these learned tokens, which capture structured information from the SG, interact with the patch tokens and CLS tokens to improve compositional seene understanding.
Graph Tokens", which interact with the patch and CLS tokens via attention. By training these tokens predict SG information, the encoder can capture better structured representations.
highlighting the effectiveness of our approach.
2 Related Work
Vision and Language Models. In recent years, popular VLMs, such as CLIP (Radford al., 2021), BLIP (Li al., 2022b), BLIP2 (Li et al., 2023), and others, have shown impressive results and ex- traordinary zero-shot capabilities. These models are trained with image-text pairs to align the two modalities in a joint embedding space. However, recent empirical studies (e.g.. VL-CheckList (Zhao et al., 2022), Winoground (Thrush et al., 2022), and ARO (Yuksekgonul al., 2023)) have shown that these models do not perform well on tasks that require compositional understanding, including re- lations between objects, and their attributes. Specif- ically, has been shown that VLMs tend to learn a "bag of objects" representation, leading them to be less structure-aware. Several recent works, such as NegCLIP (Yuksekgonul et al., 2023), Count- CLIP (Paiss al., 2023), and SVLC (Doveh al., 2022), haven shown that hard negative generation from image-text pairs using language augmenta- tions could improve fine-grained understanding. Unlike these works, we utilize image-SG pairs to improve structured representations by predicting SG information from the image encoder.
However, the above task of predicting SG from the image transformer encoder is challenging, as it deviates from the initial VLM training objective. Toward this end, we designed a technique tailored the SG tokens, which parameterized the SG to- kens independently from the patch tokens. In partic- ular, we modify the transformer layers throughout the network by decomposing the parameters into two distinct sets: SG and patch tokens. We have found that this allows better learning of the graph prediction task while still maintaining zero-shot performance. We name our proposed approach SGVL (Scene Graphs for Vision- Mod- els). See Figure for an overview.
To summarize, our main contributions are as follows: (i) We propose to exploit small set of SG annotations that is rich with visual and textual information for enhancing compositional scene understanding in pretrained VLMs; (ii) We introduce a new finetuning scheme that captures structure-related information using the visual and textual components when learning from SG labels. Specifically, for the visual side, we incorporate special "Adaptive SG tokens" in the image trans- former encoder, and train these to predict SG in- formation; (iii) Our method shows improved per- formance on CLIP, BLIP, and BLIP2 on several benchmarks: Winoground (Thrush et al., 2022), VL-CheckList (Zhao et al., 2022), ARO (Yuksck- gonul et al., 2023), and VSR (Liu et al., 2022),
Multi-task Prompt Tuning. The concept of prompt tuning for efficient finetuning of language models was introduced by Lester et al. (2021), and later explored in vision models (Jia et al., 2022: Wang et al., 2022) and VLMs (Ju et al., 2022; Zhou et al., 2022b). Several recent works (Asai et al., 2022; Sanh et al., 2022; Vu et al., 2022), have

---

Belationshige Matching
Adapted with LaRA;
Scone Graph Loss 4.g
Objects Matthing
Trantsorrer Layer with
Adaptive Scene Graph Tokens
Image
Clases Brocen
Eyp
MLPP
MLP3G SG
Text Transformer Encoder
Image Transformer Encoder
ron
weoting
frldw payshin
SG-oo-Tsst
Patcl PatelTTokonn
Adaptive SG Tokeus
Attention
thenwinn ftoaaz
Graph--Root Roohh
thnouing
G7 Classox
Negatives
Kp.VP
Qun Nam Ken
and Butes
Frisbee
Image- Scene Graph pairs
Adaptive
Figure 2: Our Scene Graphs for Vision-Languag Models (SGVL) Approach. Our key goal to capture drructure-eelate information in visual and textual components when learning from SGs. For the textual side, we generate captions and negative captions using graph (Graph- Based Negatives and SG-to-text modules), For the visual side, incorporate into image transformer an additional set of learnable "Adaptive Scene Graph Tokens" to predict SG information. The transformer parameters are partitioned into two distinct sets, one for patch tokens and one for SG tokens (shown on the right). These tokens predict objects (pink tokens & arrows) and relationships (cyan tokens & arrows) from the image, and these predictions are matched with the ground-trin souund-tuthha SGs via hipartite matching. allow open vocahulary prediction, we embed the SG categories using the text transformer encoder. Last, we use LoRA adapters for finetuning, keeping all other model parameters frozen.
explored prompt tuning in the context of multi- task learning in NLP, followed by works in vi- sion, and specifically in video domain (Avraham al., 2022; Herzig al., 2022a), and VL (Shen et al., 2022). Unlike these works, we add multiple prompts (which we refer to as Adaptive SG tokens) to VLMs in order to learn compositional scene in- formation via the SG prediction task. We designed a technique tailored to the SG tokens, which decom- poses the parameters throughout the transformer layers into two distinct sets: SG and patch tokens. This allows better learning of the graph prediction task while still maintaining zero-shot capabilities.
used to evaluate for compositional understanding in VLMs (Ma et al., 2022). This works differs from approach, as this work only proposed a benchmark, and not new training method, while our approach proposes a method for fine-tuning pretrained VLMs using scene graphs.
Scene Graphs for VL Models
We begin by describing the standard VL trans- former architecture and the scene graph annotations (Section 3.1). We then introduce our structural com- ponents for both language (Section 3.2) and vision (Section 3.3), and the training losses (Section 3.4). Our method is illustrated in Figure 2.
Learning Structured Representations. Struc- tured representations have been shown to be ben- eficial in many applications: video understand- ing (Herzig et al., 2019, 2022b; Wang and Gupta, 2018), relational reasoning (Baradel et al., 2018; Battaglia et al., 2018; Jerbi et al., 2020), VL (Chen al., 2020; Li al., 2020; Tan and Bansal, 2019), human-object interactions (Kato al., 2018: Xu al., 2019), and even image & video genera- tion (Bar et al., 2021: Herzig et al., 2020). In particular, SGs (Johnson et al., 2015; Xu et al., 2017; Herzig et al., 2018) have been extensively used to provide semantic representations in wide range of applications (Johnson et al., 2018; Raboh et al., 2020; Yu et al., 2021; Cong et al., 2023). Un- like these previous works, here we propose novel architecture design that utilizes scene graph data, demonstrating that small amount of acene-graph data can be used to improve pretrained VLMs via fine-tuning. Finally, SG data has been recently
3.1 Preliminaries
VLMs are typically trained with image-text pairs: X; = (4, T.). Each of these modalities is pro- cessed by separate encoder, and the training ob- jective is to map the embeddings using contrastive learning. Next, we briefly describe the encoders. Language Transformer Encoder Ey- The text encoder is a transformer (Vaswani et al., 2017) as described in CLIP and others, where a CLS token is appended to the beginning of the text, and the final CLS embedding is used as the text embedding. Vision Transformer Encoder Ej. typical vision transformer model (Dosovitskiy et al.. 2021) takes an image / as input, extracts N non-overlapping patches, and projects them into lower-dimension d, followed by adding spatial position embeddings,
We refer to these patches as "patch tokens".

---

resulting in new embedding zz. This forms the input tokens the vision transformer encoder:
[ECLS, 21, 22. AN]
(I)
where 2CLS is a learnable token. The input a is fed into a standard transformer, and the final represen- tation of the CLS token is the image embedding. Scene Graphs (SGs). Our motivation is to improve VLMs through structured annotations from an SG dataset. Formally, an SG is tuple G (V, E) defined as follows: (i) Nodes V set of n objects. Every object in the SG contains class label, a bounding box, and attributes. (ii) Edges of m edges. These relationships are triplets (i, j) where i and j are object nodes, and is the cate- gory of the relation between objects i and j. More details of the SG preprocessing are in Section B.1. Problem Setup. As mentioned above, we use image-SG pairs (Ig.G) from VG during finetun- ing to improve structure understanding along with standard LAION image-text pairs (I, Next, we describe the textual and visual components that capture structure in both modalities.
FARS
Figure " "Adaptive SG Tokens" Visualization. The predic- tions of object tokens (pink) and relationship tokens (cyan) are shown for images that are not the VG training data.
and propose a set of predefined graph-based rules (See Section B.2) that modify SGs and make them semantically inconsistent with the image. Next, these SGs are transformed into negative textual captions, which are used with specified loss motivate the model to focus on structural aspects.
A set
3.3 Structural Visual Component
Scene Graphs Tokens. We propose to capture structure-related information in the image encoder by predicting SGs. Toward this end, we add set of "SG tokens" that are learned prompts designed to predict objects and relationships. The SG tokens consist of two groups: (i) "object tokens" that rep- resent objects, their locations and their attributes. and (ii) "relationship tokens" that represent rela- tionships and their locations.
3.2 Structural Language Component
We begin by describing how we transform an SG into text and then explain how to manipulate the SG with our Graph-Based Negatives to further capture the structure in the model. For a visualization, see Figure 7 in the supplementary.
Formally, we define a fixed set of ñ learned ob- ject prompts and denote them by pf. pg. PPh E R 1x Similarly, we define fixed set of m. relationships prompts and denote them by ph p5. PP E Bixd, We refer to these prompts as the learned SG tokens. The SG tokens con- catenated with the standard CLS and patch tokens to obtain the following inputs to the transformer:
Scene Graph-to-Text. Given an image I and corresponding SG, G, we use G to generate a tex- tual caption for the image. We iterate over the connected components of G one by one. For each component, we iterate over the edges, and for each edge between object nodes 01 and 02 with relation r, we generate the text 01, T, 02 (e.g., "cat chasing dog"). If node has attribute, we prepend it the node's object category. Last, we generate single caption by concatenating the captions of the connected components separated by period.
2 = [ECLS, 21,.... EN, pi: ... P Pi. Pal
(2)
Next, the transformer processes the input 2, re- sulting in a new representation for each token. We use the new SG tokens representations to predict object and relationship labels and localization, by passing them through two feed-forward networks (FFNs). These predictions are supervised with ground-truth SG annotations through matching process. Figure visualizes the SG tokens learned by our model. More details are in Section 3.4. Adaptive SG tokens. We have found that the SG prediction task is challenging as deviates from the initial VLM training objective. To alleviate
Graph-Based Negatives (GN). We have found that using the scene graph data solely as image- text pairs with contrastive loss is not enough to force the model to develop structural understand- ing. As shown in recent work (Yuksekgonul et al., 2023), the commonly used contrastive learning al- lows the model to concentrate mainly on object labels disregarding other important aspects, such as relations and attributes. In order to provide more focus on such aspects, we exploit the SG structure

---

this issue, we consider a modification of the im- age transformer tailored specifically to the SG to- kens. Specifically, we decompose the parameters throughout the transformer layers into two distinct sets: SG and patch tokens. This allows better learn- ing of the graph prediction task. In more detail, recall that transformer layers have matrices for mapping tokens to queries, keys, and values. We denote these existing matrices for the patch tokens by Qp, Kp, Vp. We introduce separate set of matrices Qsc: Ksc, Vsa that is used with the SG tokens. Importantly, the attention is performed over all tokens (patch and SG). Similarly, for the MLP component in the transformer layer, we also have a different version for the patch tokens (MLPp) and the SG tokens (MLPsg).
relationship token predicts the embedding of the phrase "dog chasing cat". More details are in the next section.
3.4 Training and Losses
During training our batch contains image-texi pairs (1, T) along with image-SG pairs (Ig, G). We use the latter to generate positive captions (Tp) and negative captions (Tn). We finetune our model us- ing these inputs while optimizing the losses below. Image-Text Loss. Our image-text loss is com- prised of contrastive loss and graph negatives loss. Contrastive Loss: We apply contrastive loss on image-texi pairs, as Radford et al. (2021). Hence, the loss calculated as follows based on standard pairs and those generated from SGs:
Parameter Efficiency. To perform efficient fine- tuning, we use LoRA adapters for the VLM. Specif- ically, for each trainable matrix
Cont == Contrastive(E,(() Er(T)) (4)
Rux (e-g.- Qp or W QsG), we let Wo denote its
where T Ig and TUUUT. Graph- Based Negative Loss: For each image-SC pair we apply loss that drives the embedding of IG to be more similar to that of Tp than Tw:
W = pretrained weights, and parameterize the learned matrix as:
W = Wo + AB
(3)
(ll..IP)
where E u X r and B E r x v are r-rank matri- ces and r is a hyperparameter! 4 We note we use two distinct r: Tp and rsc for weights associated with the patch and SG tokens (as described above), respectively. During training, Wo is kept frozen, additional
LGN
Σ -log (CUaTT)) (uaiN
IG.TP.NN
where C(.,,) is the cosine similarity between the image and text embeddings. Finally, the image-text loss is weighted sum of both losses:
while A, B are learned. Overall, trainable parameters are only 7.5% of the model.
Open Vocabulary SG Prediction. The SG tokens predict the annotated SGs category information for objects, relationships, and attributes. A naive im- plementation of this idea would require a prediction head for each category. However, the VG dataset contains approximately 70K object categories and 40K relationship categories, and thus poses sig- nificant challenge. Previous work (Xu et al., 2017) introduced split containing only 100 object la- bels and 50 relation labels, which limits the dataset. Rather than restricting our data in this way, we use an open vocabulary approach. Namely, we utilize the text encoder to embed the categories from the SG components. Next, we use these embeddings in training to supervise the SG tokens. For example, if node i has category "dog" and attribute "black", we train one of the object tokens to predict the embedding of the phrase "black dog". This also applies to the prediction of relationship tokens, e.g.,
Lrr == LCont + AGN
(5)
and AGN is hyperparameter. For more informa- tion, see Section B.4 in supplementary. Scene Graph Loss. The graph G contains several annotations: the set of object categories O, their set of bounding boxes, the set of relationship cate- gories Rt, and their bounding boxes. As we do not aim to predict the object and relationship categories directly, but rather use the embeddings from the VLM, we extract the category embeddings with the text encoder Ex: Ô = (r(O) E ((++))xd and R = Er R)) E IR. (m 1)xd We note that (n + 1) and (m + 1) classes are due to "no object" and "no relationship" classes. These class embeddings together with the bounding boxes are the SG ele- ments that we aim to predict from the SG tokens.
We next describe the prediction process. The image encoder outputs a set of object tokens and set of relationship tokens. We apply two separate FFNs to predict bounding boxes and class embed- dings. To calculate probabilities over n + 1 and
'For the SG-token parameters, we set Wo to the value of the corresponding parameters of the patch tokens.

---

Winoground
VL-Checklist
ARO
VSR
ZS
Datasct
Datasets Avg Object
Flickr30K Order
COCO Order
All Dataset
Tasks
Model
Text
Image
group
Attribute
Relation
Avg
Avg
FLAVA
32.3
20.0
14.5
54.6
70.6
46.6
12.9
3.9
54.1
ViLT
34.8
14.0
9.3
73.3
85.0
62.0
22.4
18.7
LLaVA
24.8
25.0
13.0
65.5
83.1
83.0
98.1
97.5
MiniGPT44
23.3
18.0
9.5
71.3
84.2
84.1
99.4
98.9
CLIP
30.7
10.5
8.0
65.5
80.6
78.0
59.5
46.0
56.4
BIII
39.0
19.2
15.0
75.2
82.2
81.5
27.9
24.9
56.5
49.0
BLIP2
42.0
23.8
19.0
77.8
84.9
84.9
33,9
32.3
61.9
52.5
NegCIIIP
29.5
10.5
8.0
68.0
81.4
81.3
91.0
86.0
55.1
NegBLIP
42.5
24.0
18.5
78.2
83.7
81.9
85,0
84.7
57.9
48.2
NegBLIP2
41.5
26.0
20.5
79.0
87.0
88.2
91.8
88.2
62.
51.7
CLIP-SGVI. (ours) 32.0 (41.3) 14.0
14.0
(43.5) 9.8
(+1.8)
72.0
(+6.5)82.6
(+2.0)82.0
(+4.0 (+4.088.00 82.0
(422.5) 78.2
(432.2)
54.3 (-2.1)
BLIP-SGVL (ours) 42.8
(43.))77.3
(48.121155
(+6.5
81.8
(-6,6)85.2
(+3.0)
(-0.4) 70.0 (+42.1 1710 (+46.1
62.4 (+5.9)
48.0 (-1.0)
BLJP2-SGVVL (ours) (ours) 42.8 (+0.)28.5 (+4.5)23.3 (+4.3 81.2 (+3.088.44 (+3.5)88.8
(+3.9) (+3.9 (7.0 (+43.1) (+44.7
63.4 (+1.5) 51.4 (-11)
Table 1: Winoground, VL-Checklist, ARO, VSR, and Zero-Shot (ZS) Results. Gains losses are relative to the base models.
m + classes, we compute the cosine similarity of the predicted class embeddings with the GT class embeddings. Ô and R followed by softmax.
images and corresponding SGs, and LAION 400M is a large-scale image-text pair dataset that was automatically curated from the Internet.
Next, to determine which SG tokens correspond to which GT objects and relationships, the predictions of the SG tokens with the ground- truth SG. We follow the matching process and loss computation as DETR (Carion al., 2020), ex- cept that in our case, objects and relationships are matched separately. Our final SG loss LsG is the sum of the objects matching loss Lo and the rela- tionships matching loss LR. The complete match- ing process and losses details are in Section B.3. We optimize final loss the sum of the scene graph loss and the image-text loss:
match
Evaluation. We evaluate our method on four main benchmarks for compositional scene under- standing: VL-Checklist (VLC) (Zhao et al., 2022). Winoground (Thrush et al., 2022), ARO (Yuksek- gonul et al., 2023), VSR (Liu et al., 2022), and zero- shot classification. (1) VLC combines samples from the following datasets: VG (Krishna et al., 2017), SWiG (Pratt et al., 2020), VAW (Pham et al., 2021), and HAKE (Li et al., 2019). For each image. two captions are given, positive and negative. The negative caption is constructed by modifying one word in the positive caption that corresponds structural visual aspect (e-g., attribute). We report results on a combined VLC dataset excluding VG. (2) Winoground probes compositionality in VLMs. Each sample is composed of two image-text pairs that have overlapping lexical content but are dif- ferentiated by swapping an object, a relation, or both. For each sample, two text-retrieval tasks (text score), and two image-retrieval tasks (image score) are defined. The group score represents the com- bined performance. recent study (Diwan et al., 2022) has shown that solving Winoground requires not just compositionallty but also other abilities. The study suggested subset (NoTag) that solely probes compositionality. We report results on the full dataset and the NoTag split. (3) ARO proposes four tasks that test sensitivity order and composi- tionality: VG Relation, VG Attribution, COCO & Flickr30k Order. Since our approach trained on VG, we report only the COCO and Flickr30k order tasks. (4) VSR tests spatial understanding. The
Totta LIT + AsaLsa
(6)
where AsG hyperparameter.
re
4 Experiments and Results
We apply our SGVL approach to three popular VLMs: CLIP, BLIP, and 5LIP2.5 We evaluate several VL. compositional datasets following the standard protocol for each model. Additional results and ablations are in Supp. Section A.
4.1 Datasets
We describe the training and evaluation datasets below. More details are in Supp. Section D. Training. For training, we use image-SG data from Visual Genome (VG), along with a small subset (less than 1%) of the LAION dataset as "standard" image-text pairs. VG is annotated with 108, 077 'More details of BLIP2 modifications are in Section B.5.

---

Winoground
VI.-Checklist
NoTag Image
Attribute
Object Size
Relation
Model
Text
Group
Action
Color
Material
Size
Location
Action
CLIP
30.4
11.1
8,2
68.1
70.2
73.1
52.9
81.0
80.1
78.0
BLIP
44.8
23,8
19.2
79.5
83.2
84.7
59.8
83.0
81.3
81.5
B1.IP2
50.0
31.9
26.7
81.0
86.2
90.3
61.7
85.4
84.3
84.9
CLIP-SGYL(ours) [33.3 (x28)
14.0 (22.) 8.7
(+0.5)
76.6
(08.5)
78.7
(18.5) 81.3
(18.2)
59.7
(46.8)
88322
(.2.:82.0
(+1.9/8133 (43.3)
BLIP-SGVL(ouns) 45.9 (<1.) 34.3 (+10.5/25.6
(+6.4)
79.2 (40.3)
94.5
(+113/91.9 (+7.2) 73.3
(+13.5
[6.4
(33.483.9
(+2.68819 (+0.4)
BL.I225GGVL.oorr)5111 51.7 (+1.7) 37.2 (45.3) 29.0 82.4 (41.4) 91.7 (45.5) 92.2 (41.9) 70.1 (48.4) [89.0 (14.)87.6 (+3.)/88.8 (43.9)
Table 2: Wineground and VI--Chekllit Results, Results for SGVL and baselines on VL-Checkllit subsets and the NoTag split of Winoground, For VI.-Checklist we exclude the Visual Genome datasct.
Model
Adjacency Directional Orientation Projective Proximity Topological Unallocated
Average
BLIP
55.4
48.9
53.7
58.2
56.5
55.6
63.4
56.5
BLIP2
56.2
47.9
59.8
62.5
55.8
66.7
66.3
61.9
BLIP-SGVL (ours)
57.7 (+2.3)
54.1 (+5.2)
57.8
(.4.1)
63.8 (15.6)
57.8 (+1.3)
64.8
(+8.)
68.8 (+5.4)
62.4 (+5.9)
BLIP2-SGVL (ours)
59.8 (+3.6)
56.9 (+9.0)
58.5 (-13)
61.5
(-10)
59.7 (+3.9)
70.0 (+3.3)
66.8 (+0.5)
63.4 (+1.5)
Table 3: VSR Results. Results SGVL and baselines on the VSR dataset.
dataset consists of pairs of an image and a descrip- tion of the spatial relations between two objects shown in the image. The VLM should classify the pairs as either true or false. We do not evaluate models here as CLIP does not allow such classification in straightforward manner.
4.3 Baselines
In the experiments, we compare our SGVL ap- proach to three VLMs: CLIP (Radford et al., 2021), BLIP (Li et al., 2022b), and BLIP2 (Li et al., 2023) to which SGVL was applied. Addition- ally, we compare with NegCLIP (Yuksekgonul et al., 2023), which uses negative text augmenta- tions and also demonstrated strong results on these datasets. For a fair comparison, we also implement BLIP/BLIP2 versions of the NegCLIP approach, which we refer to as NegBL.P/NegBLIP.. Last, we also show comparison with the statc-of-the-art VLMs: FLAVA (Singh et al., 2021) and ViLT (Kim et al., 2021), as well newly introduced genera- tive VLMMs (GVLMs), such as LLaVA (Liu et al., 2023) and MiniGPT-4 (Zhu et al., 2023).
Zero-Shot (ZS) Classification. We evaluate 21 classification datasets following the protocol from ELEVATER (Li et al., 2022a). The evaluation in- cludes datasets such as ImageNet, CIFAR 100, and others. We report the average results over the 21 tasks in Table I and Table 4b in Ablations.
4.2 Implementation Details
We implemented SGVL using Pytorch. For code and pretrained models, visit the project page https://alommendel...
siithb..o/SSo Our code and training procedures are based on CLIP, BLIP and BLIP2. For CLIP, we use the ML- Foundation Open-CLIP repository (Ilharco et al., 2021), and for BLIP/BLIP2 we use the official implementations. We use the ViT/B-32 model ar- chitecture for CLIP, V6 for BLIP and ViT-g for BLIP2. The models are initialized with orig- inal weights released by the respective authors. The training batch contains 256/32/32 image-text pairs and image-SG pairs for CLIP/BLIP/BLIPP- SGVL respectiely... For more info, please refer to Supp. Section D. Last, we select the best check- point for each experiment using the validation set, which is composed of VG samples from VLC that are not in the test set (as mentioned above, we report VLC results excluding VG).
4.4
Results
Results are shown in Table 1, and demonstrate that CLIP/BLII//LLI22-GGGV outperforms the pre- trained base models across several datasets. These improvements come at the price of a slight degra- dation in zero-shot performance. This may be due to several factors: (1) Finetuning with negative cap- tions may harm the ZS performance, as observed in Yuksekgonul et al. (2023); (2) Finetuning for SG prediction deviates from the original VLM training objective; (3) We use only 1% of LAION image- text pairs for finetuning. Additionally, our method outperforms the Neg basclines all datasets ex- cept ARO, which measures the sensitivity text order. For this task, the text augmentations in Yuk- sekgonul et al. (2023) are more appropriate. Finally, we note that we do not evaluate CL.I--bsed models on VSR (See VSR in Section 4.1).
'https:

---

(a) Scene Graph Utilization
(b) Adaptive SG Tokens
(c) Sparse Vs.
Dense SGs
Model
Text Image Group
Model
WG 7S mAP
Model
Text Image Group
BLIP
39.0 19.2
15.0
BLIP
15.0 49.0
30% of Graph 33.2 26.5
18.0
Briph Text (GT)
40.3
20.5
16.5
BLIP
SG Tokens
20,0 47.5 16.1
70% of Graph 40.7 26.5
20.0
BLIP+GT+Grrpp Neg. (GN)
40.5
25.5
19.0
BLIP Adap. SG Tokens 21.5 48.0 17.7
w/o Relations 40.5 20.5
15.8
BIJP+GT+GN+SG Tokens 42.8
27.3
21.5
Entire Graph 42.8 27.3
21.5
Table 4: Ablations on the Wineground Dataset. We show (a) The contribution of our proposed components, utilizing the SG information, (b) The benefits of our adaptive SG tokens. Reported metries are: Winoground group score (WG), zero-shot (ZS) on ELEVATER, and mAP of SG prediction. (c) Importance of the image-SG comprehensiveness. More ablations are in Section A.
Table and Table show the performance of our method on fine- ane-ggrined Winoground, VLC, VSR splits. For most splits, our method is sig- nificantly better or comparable to the pretrained models. In Figure 4, we show where improves upon the baseline and where it still fails. For more results, please refer to Section A.3.
tribution of each component. By finetuning the model with only positive captions generated from the SGs (BLIP+GT), we obtain +1.3/41.3/41. for Winoground Text/Image/Group scores over the BLIP baseline. When the graph-based negatives (BLIP+GT+GN) are added, an improvement of +1.5/46.3/43.2 achieved compared to the base- line. Finally, when fully implementing our SGVL approach, i.e. adding the SG prediction task from the SG tokens, we achieve improved results over the baseline of +3.8/+8. 1/6.5. This highlights the contribution of the model components.
model
4.5 Ablations
We perform comprehensive ablation on the Winground dataset with our SGVL approach us- ing the BLIP model? (see Table 4). More ablations are in Section A.1. We also provide ablations on all datasets in Section A.2.
Adaptive scene graph tokens. Our approach uses different transformer parameters for SG and patch tokens, unlike the standard transformer, which uses the same parameters for all tokens. Thus, we next examine what happens when SG and patch tokens use the same parameters. In Table 4b, we report the performance of two variants on three tasks: Winoground group score (WG), Zero-shot classi- fication (ZS) on ELEVATER, and SG prediction (mAP metric). The first variant, BLIP + SG Tokens, refers to the addition of tokens dedicated to pre- dicting SGs with the same transformer parameters shared between them and other input tokens. The second variant, BLIP + Adaptive SG Tokens, refers to our technique that introduces parameters specific to the SG tokens in every transformer layer (see section 3.3). We can see that the second variant out- performs the SG token addition all tasks. This demonstrates how our modification to the image transformer encoder improves SG prediction and VL performance without compromising ZS.
Finetuning on VG without SGs. We com- pare our approach to naive finetuning with captions provided by the VG dataset. We finetuned BLIP on textual descriptions from VG, resulting in 40.0/20.5/16.0 for Winogroundf Text/Image/Grrupp scores, while our BLIP-SGVL achieves 42.8/27.3/21.5 indicating the improve- ments are not solely due to the VG data.
SG prediction without SG tokens. Our model predicts SG information from specialized SG to- kens. A naive approach might have been to predict from the CLS token. Thus, we consider variant that we refer to as BLIP MT (multi-task), which is a simple implementation of object and relation prediction. This variant does not include SG to- kens and instead predicts the graph using object and relation MLPs on top of the CLS token. Using exactly the same training recipe and data our BLIP-SGVL, this ablation achieves 38.5/22.5/17.5 for Winoground Text//mage/Group scores, while our BLIP-SGVL achieves 4 justify- ing our design choice of SG tokens.
Using sparse vs. dense SGs. In this ablation, we investigate whether the density of the SG in VG affects performance, since SGs contain denser and richer information than standard captions. Towards this end, we train SGVL with sparsified versions of the graph. Specifically, we train two variants where objects and relations from the graphs are ran- domly removed (30% and 70%) and a third variant in which all relations are removed. As can be seen
Scene graph utilization ablation. Our model consists of three parts: converting SG into cap- tions, adding hard-negatives, and adding adaptive SG tokens. In Table 4a, we examine the con-
"Wo chose BIIP due to its lower computational require- ments (c.g., BLIP2 requires A 1100).

---

Winoground
Winoground
Yoangperoes plynntasecl and and ball
Vesin parse postephunebal and
arta percos
poturt car
Ayeera permn proue
lalykgg Bove
Stwecshon B45749
yeang prose
VL- #L-Checklist
VL-Checklist
Sendeelde Lake
Povan wolk mech
Aoour
Cray platicr
Shorppome
VSR
VSR
Ta donaterragi
The dag soadde the sklin
The dag bht sagcol able
The be
Thaaey aintre: of prion
Figure 4: Predictions visualization Winoground, VL-Checklist, and VSR. The left panel shows where model succeeds and the baseline fails. The right panel shows where our model still fails. For VI.-Checklist examples, true captions are in green and false captions in red. Our model outperforms the baseline in samples that require understanding relationships between illustrate the complexity and ambiguity of the samples.
objects and binding attributes to objects. The failure
in Table 4c, our results show that our model per- forms better when the graphs are denser, richer, and describe the image more accurately, highlighting the motivation utilize SGs for VLMs.
els. We hope these findings will encourage future work to generalize our approach beyond the VL regime using other types of dense annotations, such segmentation maps and depth maps.
Training without LAION image-text pairs. In SGVL, we train simultaneously with image-text pairs from LAION and image-SG pairs from VG. To test the effectiveness of simultancous training, we train only with Image-SG pairs and obtain degradation of -2.0/-1.2// 1/2 for Text/Image/Groop scores compared to BLIP-SGY SGVL. In addition, observe degradation of 1.1 % in ZS perfor- mance compared to our BLIP-SGVL model. This indicates that simultaneous training is beneficial.
Limitations
As mentioned above, our work proposes a spe- cialized model architecture and a new finetuning scheme for learning from SGs. We demonstrate improved performance on several different models and datasets. Nevertheless, itations. First, we leverage SG annotations since they are rich and reflect structured visual and tex- tual information. However, the quality of the data is crucially important to our method, and thus poor data may result in lower performance. Finally, our improvements may be restricted by the fact that these annotations are rare and expensive to collect.
work has some lim-
5 Conclusions
Structured understanding of complex scenes is a key element of human perception, but its modeling still remains a challenge. In this work, we pro- pose a new approach for incorporating structured information into pretrained VLMs from SG data improve scene understanding. We demonstrate improved performance four benchmarks prob- ing compositional scene understanding with only mild degradation 7S performance. Our findings suggest that only a small amount of high-quality annotations may sufficient to improve such mod-
Acknowledgements
This project has received funding from the Euro- pean Research Council (ERC) under the European Unions Horizon 2020 research and innovation pro- gramme (grant ERC HOLI 819080). Prof. Dar- rell's group was supported in part by DoD, includ- ing PTG and/or LwLL programs, as well as BAIR's industrial alliance programs.

---

References
Roei Herzig, Ofrr Abramovich, Elad Ben- Avrrhaam,, As- saf Arbelle, Leonid Karlinsky, Ariel Shamir, Trevor Darrell, and Amir Globerson. 2022a. Promptono- myvit: Multi-task prompt learning improves video transformers using synthetic scene data.
Akari Asai, Mohammadreza Salehi, Matthew E. Peters, and Hannaneh Hajishirzi. 2022. Attentional mixtures of soft prompt tuning for parameter-effiien multi- task knowledge sharing. ArXiv, abs/2205. 11911.
Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. 2020. Learning canonical representations for scene graph image generation. In European Conference on Computer Vision.
Elad Ben Avraham, Roei Herzig, Kartikeya Mangalam, Amir Bar, Anna Rohrbuch, Leonid Karlinsky. Trevor Darrell, and Amir Globerson. 2022. Bringing image scene structure to video via frame-clip consistency of object tokens. In Thirty-Sixii Sixth Conference on Neural Information Processing Systems.
Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach. Trevor Dar- rell, and Amir Globerson. 2022b. Object-region video transformers. In Conference Computer Vi- sion and Pattern Recognition (CVPR).
Amir Bar, Roci Herzig. Xiaolong Wang. Anna Rohrbach, Gal Chechik, Trevor Darrell, and A. Globerson. 2021. Compositional video synthe- sis with action graphs. In ICMI..
Roci Herzig, Elad Levi, Huijuan Xu, Hang Gao, Eli Brosh, Xiaolong Wang, Amir Gloherson, and Trevor Darrell. 2019. Spatio-temporal action graph net- works. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0-0.
Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. 2018. Object level vi- sual reasoning in videos. In ECCV, pages 105-121.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Al- varo Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro. Ryan Faulkner, et al. 2018. Relational in- ductive biases, deep learning, and graph networks. arXiv preprini
Roei Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, and Amir Globerson, 2018. Mapping images to scene graphs with coomutation-invariinn structured prediction. In Advances in Neural Information Pro- cessing Systems (NIPS).
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer Vision ECCV 2020, pages 213-229.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller. Hannaneh Hajishirzi, Ali Farhadi, and Lud- wig Schmidt. 2021. Openclip.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In ECCV.
Achiya Jerbi, Roei Herzig. Jonathan Berant, Gal Chechik, and Amir Globerson. 2020. Learning object detection from captions via textual scene attributes. ArXiv, abs/2009. 14558.
Yuren Cong, Wentong Liao, Bodo Rosenhahn, and Michael Ying Yang. 2023. Learning similarity be- tween scene graphs and images with transformers. ArXiv, abs/2304.00590.
Menglin Jia, Luming Tang. Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. 2022. Visual prompt tuning. In European Conference on Computer Vision (ECCV).
Anuj Diwan, Layne Berry. Eunsol Choi, David F. Harwath, and Kyle Mahowald. 2022. Why is winoground hard? investigating failures in visuolin- guistic compositionality. In Conference on Empirical Methods in Natural Language Processing.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. 2018. Tm- age generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1219-1228.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. image is worth 16x16 16 words: Transformers for image recognition at scale. ICLR.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma. Michael Bemstein, and Li Fei- Fei. 2015. Image retrieval using scene graphs. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668-3673
Chen Ju. Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. 2022. Prompting visual-language mod- els for efficient video understanding. In European Conference on Computer Vision (ECCV).
Sivan Doveh, Assaf Arbelle, Sivan Harary, Rameswar Panda, Roei Herzig, Eli Schwartz, Donghyun Kim, Raja Giryes, Rogerio Feris, Shimon Ullman, et al. 2022. Teaching structured vision&language con- cepts to vision&language models. arXiv preprint arXiv:2211.11733.
Keizo Kato, Yin Li, and Abhinav Gupta. 2018. Com- positional learning for human object interaction. ECCV.

---

Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-land aad-language transformer without convolu- tion region supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 5583-5594.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In NeurlPS.
Nly Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Diederik P Kingma and Jimmy Ba. 2014, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona
Gandhi, Irena Gao, and Ranjay Krishna. 2022. Crepe: Can vision- laaion- foundation models reason com- positionally? ArXiv, abs/2212.07796.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32- 73.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. 2023. Teach- ing clip to count to ten. ArXiv, 1et.22222.10666.
Khoi Pham. Kushal Kafle, Zhe Lin, Zhihong Ding. Scott Cohen, Quan Tran, and Abhinav Shrivastava. 2021, Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 13018- 13028.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficint prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059 Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. 2020. Grounded situation recognition. In European Conference on Computer Vision, pages 314-332. Springer.
Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Yong Jae Lee. Houdong Hu, Zicheng Liu, and Jianfeng Gao. 2022a. Elevater: A benchmark and visual
Moshiko Raboh, Roei Herzig, Gal Chechik, Jonathan Berant, and Amir Globerson. 2020. Differentiable scene graphs. In WACV.
toolkit for evaluating amguugg--uggment models. Neural Information Processing Systems.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages PMLR
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In ICML..
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. arXiv preprint 2xdV:22011110866
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized intersection over union. In The IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR).
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-semantics aligned pre-training for lasion-language tasks. ECCV 2020.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, 7aid Alyafeai, Antoine Chaffin, Amaud Stiegler, Arun Raja, Manan Dey. M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim. Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Mingyang Chen, Ze Ma. Shiyi Wang. Hao-Shu Fang, and Cewu Lu. 2019. Hake: Hu- man activity knowledge engine. arXiv preprint arXiv:
Tsung- Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, D. Ramanan, Piotr Dollár, and C. L. Zit- nick. 2014. Microsoft coco: Common objects in context. ECCV.
Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. 2022. Visual spatial reasoning. ArXiv, abs/2205.00363.
Christoph Schuhmann, Richard Veneu, Romain Beau- mont, Robert Kaczmarzzyk, Clayton Mullis, Aarush

---

Katta, Theo Coombes, Jenia Jitsev, and Aran Komat- suzaki. 2021. L00m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114.
Jingkang Yang. Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. 2022. Panoptic scene graph generation. In European Conference Com- puter Vision.
Sheng Shen, Shijia Yang. Tianjun Zhang. Bohan Zhai, Joseph Gonzalez, Kurt Keutzer, and Trevor Dar- rell. 2022. Multitask vision-language prompt tuning. ArXiv, abs/2211.11722
Fei Yu, Jiji Tang. Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Eriie-vil Knowledge enhanced vision-language representations through scene graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4):3208-32) -2116.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2021. Flava: A foun- dational language and vision alignment model. 2022 IEFE/CVE Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 15617-15629.
Mert Yuksckgonul, Federico Bianchi, Pratyusha Kalluri,
Dan Jurafsky, and James Zou. 2023. When and why vision laioo-langgage models behave like bags-of- words. and what to do about it? In International Conference Learning Representations.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. 2022. Vl-checklist: Evaluating pre-trained vision- language models with objects, attributes and relations. arXiv preprint arXiv:2207.000 00222.....2.....
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from trans- formers. pages 5099-5110. Association Compu- tational Linguistics.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh. Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 5238- 5248.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. 2022a. Conditional prompt learning for vision-language models. 2022 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 16795-16804.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. 2022b. Conditional prompt learning for vision-language models. IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Pa- pers), pages 5039-5059, Dublin, Ireland. Association for Computational Linguistics.
Xiaolong Wang and Abhinav Gupta. 2018. Videos as space-time region graphs. In ECCV.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy. and Tomas Pfister. 2022. Learning to prompt for continual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149.
Bingjic Xu, Yongkang Wong. Junnan Li, Qi Zhao, and M. Kankanhalli. 2019, Learning to detect human- object interactions with knowledge. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 2019-2028.
Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei- Fei. 2017. Scene Graph Generation by Iterative Mes- sage Passing. In CVPR, pages 3097-31066

---

Supplementary Material for "SGVL"
21.5
20.5
Here we provide additional information about our experimental results, qualitative examples, imple- mentation details, and datasets. Specifically, Sec- tion A provides more experiment results, Section B provides more additional method details, Section C provides qualitative visualizations illustrate our approach, and Section D provides additional imple- mentation details.
18.5
18
17.5
0.25
0.5
0.75
SG Data Amount
Figure 5: Image-SG Pair Data Size. We report the perfor- mance of our model Winoground group score function of the amount of scene-graph data used during training (per- centage of the available data).
A Additional Experiment Results
data but were generated in a manner that approx- imated our graph- based negatives. Since we do not have the graphs in this setup, we apply the fol- lowing augmentations: (i) Swapping asymmetric relations We swap the nouns that are relevant the relation by using standard parser. (ii) Re- lation falsification The relation replaced with one from a closed set of relations we manually annotated in order to obtain the wrong semantic meaning, (iii) Attributes swapping We swap at- tributes from a closed set of attribute categories that we manually annotated (e.g.. color, etc.). The Text/mmage/Group scores compared to the BLIP baseline are 14.0/-0.7/-0.8 while using BLIP with our graph-based augmentations (without SG kens) obtains + 1..5//66//4444 O compared to the BLIP baseline. It can seen that the generated negatives from LAION improve only the Text score while applying our craph-based negatives improves all the metrics. This indicates that the main reason for the improvement is the structured information contained in the descriptions generated from the scene graphs
We hegin by presenting several additional ablations (Section A.I) that further demonstrate the benefits of our SGVL approach. Next, we present the BLIP model ablations for all datasets (Section A.2). Last, we present additional results (Section A.3).
A.1 Additional Ablations
In what follows, we provide additional ablations that further illustrate the benefits of SGVL. For all ablations, finetuned the BLIP model using the same recipe, in which only LoRA adapters are learned, and the training batch consists VG image-SG pairs and LAION image-text pairs. The importance of the SG data. To examine the significance of the information provided by scene graphs, we suggest learning SGs without any useful information. Thus, we run experiment in which the SGs completely random. This ablation obtains Winoground 37.8/18.7/14. 7/14.0 compared our BLIP-SGVL 42.8/27.3/21.5 for Text/Image//Group scores (the BLIP baseline ob- tains 39.0/ 99.0//9.2/15.0) This illustrates that the ap- proach we employ is not merely a regularization, but also provides important information about the SGs that can be used by pretrained VLMs.
Scene graph token representations. To analyze what the scene graph tokens learned, we can eval- uate the ability of object and relationship tokens to be utilized explicitly for the auxiliary task as simple SG predictor in images. This is accom- plished by predicting the scene graphs Visual Genome based the learned SG tokens. We com- pared the learned SG tokens with a BLIP model extended with object and relationship heads, as explained in BLIP-MT variant in the main paper (See Section 4.5). Our model achieved an mAP of 17.7, while the BLIP MT achieved an mAP of 14.4. These results indicate that the scene graph tokens learn meaningful and useful representations. Token specialization. Our SGVL approach learns different specialization for each scene graph to- ken in Figure 6. We visualize the bounding box center coordinates predicted by 10 different object
The effect of image-SG data size. In this experi- ment, train method using varying amounts of image-SG pairs of data (10% 40%, 70%, and 100% of the dataset) in order to examine the effect the data portion. Figure shows the Winoground group score performance as a function of the image- SG pairs data portion. As can be seen, the positive slope suggests that adding image-SG data consis- tently improves results.
Training with negatives from non-graph data. To demonstrate the importance of structured in- examine the
formation in textual descriptions, performance of the model when only LAION cap- tions are used. Specifically, we trained using gen- erated negatives that were not derived from SG

---

88
18-
66
34-
82-
12
0.2
244
5.b 98
0.1
0.2
0.3
0.4
0.5
0.8
0.7
08
6.
Figure 6: Tokens Specialization. We visualize the box predictions of 10 random object tokens (left) and 7 relationship tokens (right) on all images from the COCO validation set. Each box is represented as point with the normalized coordinates of its center. Colors indicate the predictions made by different tokens.
Winoground
VL-Checklist
ARO
VSR
All Dataset Image
All Datasets Avg Object
Flickr30K Order
COCO
All Dataset
Model
Text
group
Attribute
Relation
Order
Avg
BLIP
39.0
19.2
15.0
75.2
82.2
81.5
27.9
24,9
56.5
BLIP Graph Text (GT)
40.3
20.5
16.5
76.0
80.8
77.5
28.0
24.6
$7.2
BLIP GT + Graph Neg. (GN)
40.5
25.5
19.0
80.0
84.0
81.2
69.6
70.0
61.4
BLIP GT + GN SG Tokens
42.8
27.3
21.5
81.8
85.2
81.9
70.0
71.0
62.4
Table 5: Scene graph utilization ablation results for all datasets.
Winoground
VL-Checklist
ARO
VSR
Graph Pred.
ZS
All Datasets Avg Text Image Attribute Object Relation
All Dataset
Flickr30K COCO
All Dataset
mAP Score
21 Tasks
Model
Order
Order
Avg
Avg
BLIP
39.0 19.2
15.0
75.2
82.2
81.5
27.9
24.9
56.5
49.0
BLIP Tokens
39.8 26.5
20.0
81.0
84.5
81.2
67.6
67.0
61.9
16.1
47.5
BLIP + Adaptive SG Tokens
42.8 27.3 21.5
81.8
85.2
81.9
70.0
71.0
62.4
17.7
48.0
Table 6: Adaptive SG tokens results for all datasets.
tokens and 7 random relationship tokens for all im- ages in the COCO val set. We observe that these tokens are specialized in different locations in the image, whereas the relationship tokens tend to be centered since their boxes larger and spread over greater area.
with only a mild zero-shot degradation.
A.3 Additional Results
We start by presenting zero-shot classification re- sults tested with linear probing. Next, we show additional results when finetuning image-text pairs from COCO in Table 7. Moreover, we show addi- tional results all splits in Winoground in Table 8, as suggested Diwan al. (2022). Last, we also present fine-graine anc-grrined results NegCLIP, NegBLIP, and NegBLIP2 baselines on VL-Cheklist, and VSR datasets in Table 9 and Table 10. For all tables, the difference between the baselines and our SGVL approach is denoted by gains and losses (Xx. As discussed in our paper, we believe that the slight degradation in zero-shot classification is due to fine-tuning with negative captions and graph pre- diction tasks that deviate from the original VLM training objective. This phenomenon is also visible
A.2 Ablations All Datasets
The main paper only includes ablation results for Winoground in Table 4a and in Table 4b. Here, in Table and Table 6, we include additional abla- tion results. Specifically, we have performed a com- prehensive ablation using our SGVL approach with the BLIP model all datasets: Winoground, VL- Checklist, ARO, and VSR. It is evident from Ta- ble that the utilization of our approach for both vi- sual and textual components is important. Last, Ta- ble shows that the "Adaptive SG Tokens" improve SG prediction and VL performance for datasets

---

Winoground
"L-Checklist
ARO
VSR
All Dataset Image
Datasets Avg Object
Flickr30K Order
COCO Order
All Datasct
Model
Text
group
Auribute
Relation
Avg
CLIP-SGVI. (COCO)
32.8
11.8
10.0
73.5
83.3
81.0
83.0
79.4
ALIP-SGVLL (COCO)
43.8
26.0
21.3
81.9
85.5
82.0
70.7
71.2
61.7
BLIP2-SGVI (COCO)
46.0
26.5
22.8
81.2
88.8
88.8
77.4
78.2
63.0
Table 7: Winoground, VL--Checklist, ARO, VSR Results when finetuning COCO image-text pairs instead of LAION
in recent works (Doveh et al., 2022; Yuksekgonul et al., 2023). Here, also report the classifi- cation performance achieved with linear probing. For CLIP-SGVL, when using 5-shot, 10-shot lin- ear probing, the difference on classification tasks with respect to CLIP (in the same setting) changes to -0.3%, +0.1 % and with 20-shot to +0.5%. This demonstrates that partial fine-tuning does not cause degregation as previously observed in other studies.
B Additional Modeling Details
We begin by presenting additional model details regarding our graph preprocessing procedure (Sec- tion B. Next, we describe our method for creat- ing graph- based negatives (Section B.2), which is illustrated in Figure 7. We provide more details our scene graph loss (Section B.3) and some modi- fications made to our loss calculation when training BLIP (Li et al., 2022b) and BLIP2 (Li et al., 2023) (Section B.4). We conclude by describing more detail our approach when using the BLIP2 model (Section B.5).
In order to demonstrate that our method can also be used with image-text pairs other than LAION, we report in Table results when used with image- text pairs from COCO (Lin et al., 2014). It can be seen that the results are comparable our origi- nal version, which indicates the flexibility of our approach.
B.1 Graph Preprocessing
We next describe how we process the image-SG pairs create our training dataset. Our guiding principle is to create image-SG training pairs where the graphs are dense enough but not too large, in order allow structured and short descriptions. To this end, given an image / and a corresponding graph G(V, we extract the sub-graphs by tak- ing a random walk on the graph. The random walk is initialized by randomly picking a relationship from the graph (edge E E and nodes e1, 12 E V such that (v1, 22)) and ends when a node that has outgoing edges is reached, resulting in sub- graph G E1). Next, the image is cropped to the union of the bounding boxes of all objects (o E Vi) in the extracted sub-graph, resulting im- age 71- We finish the process by adding new nodes and relationships to G1 from the residual graph G, = (V/V,. . E/E;) that are visible in 4, We use G1 and In as training sample only if the derived G1 contains at most 10 objects (i.c. |E 10). This process creates SGs composed of connected components that are all DAGs with a single Hamil- tonian path, which facilitates caption generation.
Table 8 shows the performance on fine-grained splits on Winoground. It can be observed that our method generally outperforms or is comparable to pretrained models for most splits. Although we present all splits, we note that Diwan et al. (2022) suggested that only the samples from the NoTag split are actually probing compositionality, while the other splits are difficult solve for various additional reasons. Thus, the NoTag the most important split to evaluate.
Table 9 shows the performance on fine-grained splits of the VL-Checklist dataset, including the "Attribute", "Object", and "Relation" splits. As can be seen, our method improves both the pretrained baselines CLIP, BLIP, and BLIP2, as well as the NegCLIP, NeI and NegBLIP2 baselines on those fine-grained splits.
Table 10 shows the performance on fine-grained splits of the VSR dataset, which test spatial un- derstanding in VLMs. Similarly, we find that our method improves on most splits and across models when compared with pretrained and Neg baselines.
B.2 Graph-Based Negatives
Overall, our approach improves multiple archi- tectures (CLIP, BLIP, and BLIP2) on a variety of VL datasets with only mild degradation in zero- shot performance.
In order to generate negative image captions, we propose a set of predefined rules that when ap- plied to an image-scene graph pair, result in a

---

Ambiguosly Correct
Unusual Image I
Unusual
Visually Difficult
Complex Reasoning
No
Text
Compositional
Tag
Model
T
G
T
G
T
T
G
CLIP
30.4 15.2 13.0
25.0
8.9
5.4
30.0
16.0
10.0 76.7 36.7 33.3 15.8 0.0 0.0
24.4
7.7
3.8
30.4
11.1 8.2
BLIP
39.1 17.4
15.2
37.5
16.1
14.3
30.0
14.0
8.0
50.0 33.3
30.0
29.0
10.5
10.5
24,4
7.7
2.6
44.8
19.2
BLIP2
28.3
19.6
30.4
17.9
14.3
38.0
14.0
12.0
53.3
26.7
26.7
34.2
13.1
20.5
7.6
2.5
50.032.0
NegCLIP
28.2 4.3 4.3 17.9
7.2
3.6
36.0 8.0 8.0 66.7 30.0 26.7 10,5 2.6 2.6
21.8
7.7
5.1 33.2 12.2 8.7
NegBLIP
43.521.7
8.7 42.9 19.6
14.3
28.0
18.0 12.0 56.7 43.3 33.3
28.9
15.8 10.5
32
10.2
3.8
47,0 22,6
NegBLIP2
23,9 35.7 17.8 17.8 36.0 14.0 14.0 63.3 35.3 30.0 23.7 15.8 13.1
24.3
9.0
6,4
49.4 27.9
CLIP-SGVI. (ours)
37.0 13.0 8.7 28.6 10.7 7.1 32.0 10.0 8.0 70.0 40.0 30.0 18.4 5.3
24.4 16.7 12.8 33.2 14.0 8.7
BLIP-SGVL (ours)
45.6 7 21.7 44,6 23.2 4 38.0 24.0 18.0 46.7 43.3 40.0 23.7
18.3
15.8
33.3
11.2
7.7
45,9 25.6
BLIP2-SGVL (ours) 43.5 26.1 23,9 39.3 26.8 21.4 34.0 18.0 16.0 60.0 46.7 46.7 26.3 18.4 15.3 29.5 11.5 6.4 51.7 37.2 29.0
Table 8: Results on Winoground for all the splits presented in Diwan et al. (2022). We report T, I, and G for Text retrieval, Image retrieval, and Group retrieval.
Attribute Material
Object
Relation Action
Model
Action
Color
Size
State
Location
Size
CLIP
68.1
70.2
73.1
$2.9
63.3
81.0
80.1
78.0
BLIP
79.5
83.2
84.7
59.8
68.8
83.0
81.5
BLIP2
81.0
86.2
90.3
61.7
70.1
85.4
84.3
84.9
NegCLIP
66.7
74.9
78.4
54.8
63
81.9
80.9
81.3
NegBLIP
82.6
88.3
89.3
60.8
70.1
81.0
79.6
83.0
NegBLIP2
82.1
88.7
90.7
63.0
70.0
87.2
86.8
88.2
CLIP-SGVL (ours) 76.6 (+8.5) 78.7 (+8.5) 81.3 (.8.2) 59.7 (+6.8
62.0
(-13)
83.2
(+2.2)
82.0 (+1.9) 81.3 (+3.3)
BLIP-SGVL (ours)
79.2 (4.3)
94.5 (+11.3)
91.9 (47.2)
73.3
(+13.5)
70.0
(.1.2)
86.4
(+3.4)
83.9
(x2.6) 81.9 (10.4)
BLIP2-SGVL (ours) 82.4 (+1.) 91.7 (65.) 92.2 (+19) 70.1 (+8.4) 69.6 (4.5)
89.0 (+4.6) 87.6 (+3.3) 88.8 (+3.9)
Table 9: VL-Checklist Results. We report VL-Checklist (Zhao et al., 2022) on all splits of the Attribute, Object, and Relation tests, excluding the Visual Genome dataset. The difference between base models and SGVL is denoted by (+X).
Model
Adjacency Directional Orientation Projective Proximity
Topological
Unallocated
Average
BLIP
55.4
48.9
53.7
58.2
56.5
55.6
63.4
56.5
BLIP2
56.2
47.9
59.8
62.5
55.8
66.7
66.3
61.9
NeIBLIP
56.0
48.0
$5.7
59.0
55.6
58.7
63.2
57.8
NegBLIP2
56.3
48.0
57.4
61.5
55,8
67.8
70.5
61.2
BLIP-SGVL (ours)
57.7 (+2.3)
54.1
(+5.2)
57.8
(+4.1)
63.8
(65.6)
57.8 (+1.3)
64,8 (+8.8
68.8 (e5.4)
62.4 (+5.9)
BLIP2-SGVL (ours)
59.8 (43.6)
56.9
(+9.0)
58.5
(-1.3)
61.5 (-1.0)
59.7 (43.9)
70.0 (+3.3)
66.8 (+0.5)
63.4 (+1.5)
Table 10: VSR Results. We report accuracy on all splits of the VSR (Liu et al., 2022) dataset.
negative scene graph that incorrectly describes the image. Our scene graph to caption scheme trans- forms negative scene graphs into negative captions that are semantically inconsistent with the images they accompany. For each training sample we ran- domly apply one of the following negative rules focused on object attributes and relationships in the graph: (i) asymmetric relations swapping we call a relationship asymmetric, if for two ob- jects b, aRb - -bRo. We manually anno- tated a relation as asymmetric out of the 300 most common VG relations. We use these to generate negative scene graph by searching for edge e (F1, 12) E E representing asymmetric rela- tionship, and modify the graph by replacing e with an edge 6n = (V2, v1). For example, in the case
of a graph describing the phrase "dog chasing cat", such a negative will result in the phrase "cat chasing dog". (ii) relation falsification we replace rela- tions in the graph with false relations. For example, turn "cup on table" to "cup under table". For negatives focused on object attributes, first scan the dataset and split the attributes into the following categorics: color, material, size, state. Next, we use this split to perform two types of negatives: (i) altributes falsification we replace attributes for ob- jects in the graph with false attribute from the same category. For example, turning "blue ball" to "red ball". (ii) attributes swapping for two objects that are annotated with attributes from the same category. Given that such a pair has been found we switch between the attributes, result-
search the graph

---

ing for example, "silver spoon and golden knife" from "golden spoon and silver knife".
where 07 Rixn+1 and of E Rixm+1, Next, we need to match the predictions of the SG tokens with the ground-truth SG, in order to determine which SG tokens correspond to which ground-truth ob- jects and relationships. We follow the matching approach as in DETR (Carion et al., 2020), ex- cept that in our case, objects and relationships are matched separately. We describe the object match- ing below. Given permutation a over the object tokens, we define the matching-com between the permutation and the GT by:
B.3 Scene Graph Loss
This section provides detailed explanation of our Scene Graph loss, as mentioned in the main paper in Section 3.4.
As explained in the paper (See Section 3.4), we incorporate SG tokens into the model, which are used to predict the SG that corresponds to the im- age. We next explain this process. The graph G contains several annotations: the set of object cat- egories 0= {o the set of bounding boxes DO {bę )= and of relationship categories R [r:] We also augment the relationships with set of bounding boxes, that are constructed as the union of the boxes of the corresponding ob- jects, and denote these by _ = {b H-
s(o) Σ (1a2o098010)))) +
1 (oozt)) (Cgiou (b6, Br(s) + | - decalli Jan
Namely, check the compatibility between the GT and permuted objects both in terms of object category (i.e., the probability assigned by the query to the GT object o;) and in terms of bounding boxes (i.e., how well the predicted box matches the GT one). Here Lyion is from (Rezatofighi et al., 2019). The optimal matching ē is found by optimizing this score: â arg minges s(o). Finally, we use the optimal matching & from above calculate the following Objects loss:
As we use open vocabulary approach, we do not aim to predict the object and relationship cate- gories directly, but rather use the embeddings from the VL model. Thus, we extract the embeddings of these labels with the text encoder ET to get class embeddings: Ô = (O E R(n+1)xd and R = (Rt) C 1mxd We note that (n + 1) and (m+1) classes are due to "no object" and "no relationship" categories (denoted as Ø), which are represented with learned embeddings.
Thus far we described the elements of the SG that we aim to predict from the SG tokens. We next describe the prediction process, and the correspond- ing losses. The image encoder outputs a set of ń object queries and rn relationship queries. We ap- ply two forward networks, FFN,, and FFN.. on top of these queries to predict bounding boxes and label embeddings respectively. Specifically, given the final representations of 7th object token and h relationship token for image I, which we denote (T) and (T) respectively, we predict:
Lobjects log ga (ox) +
(,0 (giion (b%, Bocs)) + ||b bocolli) J(2)
The relation matching and loss L is calculated in similar manner, and the total scene graph loss is the sum of Lovj and L
LsG = Lobj + L
(13)
B.4 BLIP Image- Text Loss Details
Besides image and text unimodal encoders trained using a contrastive loss, BLIP (Li et al., 2022b) and BLIP2 (Li et al., 2023) also includes an image- grounded text encoder that uses additional image- text cross-attention layers. The encoder is equipped with a binary classification head (a linear layer) and is trained to predict whether an image-text pair is positive (matching) negative (unmatching). In the training procedure described by the authors, the encoder uses hard negative mining strategy cal- culate an additional loss for all image-text pairs in the batch. When training our BLIP/BLIP2-SGVL (10) models, we apply this loss as well. Additionally,
6, FFNN6 (F) êÚ (F((()
(7)
by = FFFN éx = FFN.
where Gy by E R are bounding box predictions and eg. de E Riidd are class embeddings predic- tions. Next, use the class embeddings matrices to predict probabilites over n and m + classes:
hh = SoftMax(e.

---

we use this encoder add another term our graph-basce negative loss (CGN). Let (I, denote the positive score given by the encoder to some image-text pair (I, then the following term is added to LON!
error analysis on Winoground and VL-Checklist evaluate the success and errors of our method and the baselines. This illustrates which examples our BLIP-SGVI model is successful on, in contrast to the BLIP model.
D Additional Implementation Details
Σ -log Pp.
(14)
Our SGVL approach can be used on top of va- riety of VL models. For our experiments, we choose the CLIP (Radford et al., 2021). BLIP (Li et al., 2022b) and BLIP2 (Li et al., 2023) mod- els as they are among the most popular and easy- to-use methods. These models are implemented based on the Open-CLIP library (Ilharco al., 2021) and the BLIP/BLIP2 code base (available at https: //g thub. com/ eftms/aalslfor/LLAVIS We implement SGVL based on these repositories. As described above, our approach is trained using both the original image-text pairs from LAION and the image-SG pairs we curate from Visual Genome. In particular, for CLIP-SGVL use 3M image-text pairs, while for BLIP/BLIP2-SGVL we use 750K due computational constrains.
with Ia, Tp, TN denoting the VG images, positive and negative captions in the batch, respectively.
B.5 BLIP2 Model Details
this section, we describe how we incorporate our "Adaptive SG Tokens" into the BLIP2 (Li et al., 2023) model. Recall that the BLIP2 model ar- chitecture is based on a Q-Former module that consists of two transformer sub-modules: (a) an image transformer that interacts with frozen im- age encoder to produce the visual features used for contrastive learning; and (b) text transformer that performs both the functions of text encoder (producing the textual features required for con- trastive learning) and a text decoder. The inputs to the image transformer sub-module are a set of learnable query embeddings. These queries inter- act with each other through self-attention layers, as well as with frozen image encoder features through cross-attention layers. When applying our SGVL approach to BLIP2, we add our SG tokens as an additional set prompts in parallel to the learnable queries. Therefore, our SG tokens interact with the learnable queries and the frozen image encoder fea- tures. We apply our adaptation technique to these tokens as we do in BLIP and CLIP (see Adaptive SG Tokens in section 3.3) and use the exact same procedure for the graph prediction task.
In our experiments, we trained our CLIP-SGVL on V100 GPUs for 32 epochs with a batch com- prised of 256 image-text pairs and 8 image-SG pairs. We use Adam optimizer (Kingma and Ba, 2014; Loshchilov and Hutter, 2017) with B1 = 0.9, B2 0.98, and 6 1e 6. We use Ir 5c with cosine scheduler, and weight decay 0.2 for regularization. For BLIP-SGVL, we trained on V100 GPUs and for BLIP2-SGVL on A100 GPUs. We train both for 8 epochs with a batch comprised of 32 image-text pairs and 8 image-SG pairs. We use Adam W optimizer (Kingma and Ba, 2014) with 31 = 0.9, B2 = 0.99, and le 8. We use Ir 5e 5 with cosine scheduler, and weight decay of 0.05 for regularization.
Qualitative Visualizations
D.1 VL-Checklist
Figure 7 shows a visualization of the generation process of captions, including positive captions as well as negative captions, based on our Graph- based Negatives module. As shown in the figure, captions generated from scene graphs are much more focused on describing fine- mme-ggaieed details. Furthermore, we show in Figure visualizations of scene graph tokens predictions for images from Vi- sual Genome, which the model was not trained on. It can be seen that although the model has not been trained on these images, the predictions are reason- able. Finally, we show in Figure 9 and Figure 10
Dataset. VL-Checklist (Zhao et al., 2022) a new study that combines the following datasets: Visual Genome (Krishna et al., 2017). SWiG (Pratt et al., 2020), VAW (Pham et al., 2021), and HAKE (Li et al., 2019). For each image, two captions are given, positive and negative. The positive cap- tion is derived from the source dataset and is co- herent with the visual structure of the image. the negative caption constructed by modifying one word the positive caption that corresponds to a structural aspect in the image. To correctly solve

---

Image & Scene Graph Pair
EoCeS
10an
Birowling
"trees green purk Man darowing white frisbee"
Asymettzccal Relations
man
"crees green park White fridbee Birowing
Graph Negatives
Throwiing
Scene Graph to Text
white
"tees white park man Barowing green frishee"
Inoex
man
Attribotes Sinding
In
throwing
Figure 7: Visualization of some of our Graph-based Negatives well the SG-to-Text module. We show the generation process of positive captions (green) and negative captions using the graph (red).
sample the model needs identify the caption Faithfully describing the image. Specifically, VL- Checklist evaluates the following structured con- cepts: (I) Object: identifying whether objects men- tioned in the text appear in the image invariantly to their spatial location and size, (2) Relation: spa- tial or action relation between two objects, and (3) Attribute: color, material, size, state, and action bounded to objects. We report results bined VL-Checklist dataset excluding VG.
retrieval tasks. A recent study (Diwan al., 2022) has shown that solving Winoground requires not just compositional understanding but also other abilitics such as commonsense reasoning. The study proposed new split to the dataset differ- entiating the samples by the source of their hard- ness. Specifically, the split of the samples into the following categories is follows: Non Composi- tional There are 30 samples in this category that do not require compositional reasoning. Visually Difficult The model must be able detect an item that is visually difficult to identify (small, blurry, in the background, etc.) in order to sort these sam- ples correctly. This category includes 38 samples. Ambiguously Correct This category includes 46 samples where at least one caption accurately de- scribes both images or doesn't t quite describe any of the images. Unusual Text & Unusual Image There are 106 samples in these categories, all of which contain unrealistic or awkward texts or images that make it difficult to solve them with a VL model. Complex Reasoning This category consists 78 samples that require common sense reasoning or knowledge of the world around us. No Tag These are vanilla Winoground examples that solely probe compositional understanding.
com-
Inference details. We use the official data and code released by the authors which is available at https: //github. thub. wwt.coooooooeee test sample consists of image and two captions. For CLIP, we compute the cosine similarity be- tween the image and the captions and report the positive caption as one with the higher similar- ity. For BLIP/BLIP2 we use the ITM head, which predicts both positive and negative score for each pair. We consider the caption with the higher posi- tive score to be the correct one.
A
D.2 Winoground
Dataset. Winoground (Thrush et al., 2022) a new challenging dataset that evaluates the ability of VL models to capture compositionality in vi- sion & language. The dataset contains 1600 tests across 400 samples. Each sample is composed of image-text pairs (To, Co), (I, C1). The pairs have overlapping lexical content but are differenti- ated by a swapping of an object, relation, or both. To correctly solve the sample the model needs to correctly solve two text retrieval and two image
Inference details. We use the official data and code released by the authors which is avail- htwpee/innnacccecaaatttaaateeeee co/ co/datasets/
able winground.. For testing, the pairs are facebook/ brtebook/winogeee given, and text score, an image score, and a group score for sample is computed in the following way: The text score is and only if image Io

---

has a higher similarity to caption Co than C1, and image I has higher similarity caption C1 than and only caption Co has a higher similarity to image To than image / and C₁ has a higher similarity to image I than image fo. The group score is 1 if and only if both text and image scores are 1. Thus, the random chances for both the image and text score, is 1/4 while for group score it is 1/6. Similarities between image-text pairs computed in section D.I.
evaluate in zero-shot manner we test our model and baselines using all samples from the train, val-
Co. Similarly the image score
idation, and test splits. The spatial relations are divided into 7 categorie Adjacency, Direc- tional, Orientation, Projctive, Proximity, Topolog- ical, Unallocated. We report results according to these categories, as well as the average over all spatial relations.
Inference details. We the official data released by the authors which is available at
https://gi com/ cambridgel visual souull-sppaial--ressoiinn We do not evaluate CLIP on this task since the task requires assigning a true or false label to an image-text pair. CLIP, however, does not allow this to be done in a straightforward manner, and Therefore only the BLIP/BLIP2 models can be used. We use the ITM head determine whether the sample true or false.
D.3 ARO
Dataset. ARO (Yuksekgonul et al., 2023) (Attri- bution, Relation, and Order) a new benchmark that tests compositionality VL models. The au- thors propose four tasks that are sensitive to order and composition, namely Visual Genome Relation, Visual Genome Attribution, COCO& Flickr30k Order. Since our approach is trained on Visual Genome, we report only the COCO and Flickr30k order task (PRC). For the order task, image-text pairs from the mentioned datasets are used. The words the text are reordered in order to create false captions for the image, according to the fol- lowing perturbations: nouns and adjectives shuffle, everything but nouns and adjectives shuffle, tri- grams shuffle and words within trigrams shuffle, Inference details. We use the official data and code released by the authors which is available at Lassuanlanggage-meee inference, each sample consists of an image and five textual descriptions. The similarity of each text to the image is measured as in section D.1, and the text with the highest similarity to the image is reported the real caption.
D.5 Finetuning Datasets
In our work, we use the LAION dataset as "stan- dard" image-text pairs, along with image-SG data pair from Visual Genome (Krishna et al., 2017) (VG). Visual Genome is annotated with 108, 077 images accompanied by their corresponding scene graphs. On average, images have 35 entities, 21 re- lationships, and 26 attributes per image. Addition- ally, there are approximately 70K object categories and 40K relationship categories.
thub. com/mertyg/ During
general, Vi- sual Genome scene graphs can be viewed as dense knowledge representations for images, similar to the format used for knowledge bases in natural language processing.
D.6 Licenses and Privacy
The license, PII, and consent details of each dataset in the respective papers. In addition, wish to emphasize that the datasets we use do not contain any harmful or offensive content, as many other papers the field also use them. Thus, we do not anticipate a specific negative impact, but, as with any Machine Learning method, we recommend to exercise caution.
D.4 VSR
Dataset. VSR (Liu et al., 2022) VSR (Visual Spa- tial Reasoning) is a new benchmark for measuring the spatial understanding of language mod- els. The VSR dataset consists of natural image-text pairs in English, each example contains an image and a natural language description of the spatial relationship between two objects shown in the im- age. The VI. model needs to classify images and captions either true false, indicating whether a caption accurately describes the spatial relation- ship. The dataset has more than 10K image-text samples, derived from 6,940 COCO images and covers 65 spatial relations. The dataset is split into a train, validation and test sets, however, since we

---

Blckk
ATSCE SOCCO
Brok Suldees
Bise SiOr
la post
TEAT
mellbrake
light
Figure 8: Scene Graph Prediction. We show the predictions of the "scene graph tokens" images from Visual Genome that trained by model.
Varegaaeaa wan playing and ball
Weang peroox playing and priis ball
Wateria battle
A bailk
vater
Lalddgg llee Dowes
Bowers ledploge
An old icorecs youra persis
Ziame 24rean
The can uxdernaat perooe
The persecte indemmadiic
The big firthe paice
Meazaraceke
Black
Nég furthe perzen
cararnahed
true smadted aito acar
worting sweaks office peroon whvicet
The proses wlo is treei ottir persen wentingg
Thore sone yelliwse and
There sus blae and mony
Thane mooee sncodwardees
Thare are moee akerr
Licevos dbeatiig
Figure 9: Error Analysis on Winoground. We demonstrate on the in green where our BLIP- SGVL model succeeds, while the baseline BLIP model fails. On the right, in red, we can observe examples in which our BLIP-SGVLL model fails. As visible, our model improves samples that require understanding relations between objects, binding attributes to objects, and counting objects.

---

Poser tents redect ride
Parous heMin
Perszz Beron mote tadquot
Peros Feronn dran doir
Permos oplis
Peross solt romtush
loing baranes Jamenng
Stacking bisd Szandiza
Siker digganee Bright green dupense
Netl
Panceboll dag Pewsefee skg
Wooden Tixt Foor
Suppy Verder Felleeder
Porsen bords Penss caud barte
Tulllae
bwach
Suor
Figure 10: Error Analysis on VL-Checklist. We demonstrate on the left in green where our BLIP-SGYL model succeeds, while the baseline BLIP model fails, On the right, in red, we can observe examples in which our BLIP-SGVL model fails. True captions are in green and false captions in red. As visible, some of the samples on which our model fails are ambiguous or visually difficult to solve.


Reviews:
[{"review_id": "jJF2yZ26kY", "reviewer": "Reviewer_639u", "summary": "", "questions": "I would appreciate it if the authors can address the questions below, in the rebuttal stage:\n\n**Q1:** Using scene graph, or more boardly structured information, is not a very novel idea now. The difference of the proposed method against some prior works need to be clarified.\n\n**Q2:** More state-of-the-art comparison, including these latest methods and more advanced VLM.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Shared Comments\n\nWe thank the reviewers for their insightful comments. We are encouraged that they found the proposed idea of leveraging small scene graph datasets **could benefit the research community immensely** `(dPsS, VrYK)` and **could successfully compensate for the lack of compositional understanding** `(VrYK)`. Furthermore, they found the proposed framework **reasonable and effective** `(639u)` and the architectural changes and losses **easy to implement** `(dPsS)`. Finally, they observed **comprehensive ablations** `(639u, dPsS)`, demonstrating **significant improvements** over the proposed tasks `(639u)`. We are pleased that the reviewers found the paper **well written** `(639u, dPsS)`, and the **diagrams capture** the novelty aspect. \n\nA brief summary of the main aspects of our rebuttal response:\n1) We performed additional experimental comparisons to other methods, including SOTA VLMs `(639u)`.\n\n2) We discussed the comparison between our work and other related works and explained in more detail our novelty compared to other works `(639u)`.\n\n3) We performed a comprehensive analysis of the degradation in performance in certain partitions of the evaluated benchmark and the zero-shot degradation `(VrYK)`.\n\n4) We performed an experiment replacing 1% of LAION with CC3M for training `(dPsS)`.\n\nWe next address all concerns and look forward to an open and constructive discussion with the reviewers.\n\n\n### Response to R639u\n\n\nWe thank the reviewer for the insightful comments. In the following, we provide a response to the questions raised in the review:\n\n**Novelty and Related work**. Thank you for suggesting these prior works. We will incorporate the discussion and results below into the final version.\n\nThe main insight of our work, as highlighted by `dPsS` and `VrYK`, is that the standard VLM architecture can be modified such that a small amount of richly annotated scene graph data can successfully compensate for the lack of compositional understanding in popular pretrained VLMs. We show that roughly 100K scene graph annotations from Visual Genome, used with a specialized model architecture and a new finetuning scheme, can improve the granular understanding of pretrained VLMs. This differs from the previous works mentioned by the reviewer.\n\n\nMore specifically, previous work [1] used scene graphs (parsed from texts) for masked multi-modal learning focused on objects, attributes, and relations. The approach used cross-modal transformer encoders without changing the encoder architecture itself. We, on the other hand, consider a novel architecture design for the vision component of dual-encoders that is explicitly targeted at predicting complete scene graphs from images alone. Our new architecture can be incorporated into a variety of vision-language models, and we show that it results in empirical improvement when used for fine-tuning existing models. In addition, to the best of our knowledge, we are the first to show that a small amount of scene-graph data can be used to improve the granularity of vision-language models via fine-tuning. Finally, [1] uses scene graphs that are parsed from textual captions alone, while our method emphasizes the use of dense and rich scene graph annotations from dedicated scene graph datasets such as Visual Genome.\n\nMoreover, we also differ significantly from the works described in [2] and [3]. [2] proposed several text augmentations (*no scene graphs are used*), specifically intended to enforce structured understanding on *only the textual level*. In contrast, our work exploits scene graphs on both the *visual and language sides*. In particular, our approach utilizes scene graphs to enforce structured understanding *using the visual side* (e.g., predicting scene graphs), which is not the case in [2]. Lastly, [3] proposed a new benchmark by constructing a dataset from scene graphs based on Visual Genome (which we used for training in our work). This differs from our approach, as this *work only proposed a benchmark, and not a new training method*, while our approach proposes a method for fine-tuning pretrained VLMs using scene graphs.\n\n\n**Comparison to state-of-the-art**. Following your comment, we present below additional comparisons to state-of-the-art VLMs. Specifically, SVLC [2], InstructBLIP, LLaVa, Mini-GPT4, FLAVA, and ViLT (the latter two were reported in Table 7 of the supplementary) on Winoground and VL-Checklist. We used results from papers where available and ran the models ourselves otherwise. We note that we did not report results for ERNIE-ViL, as it is not currently state-of-the-art for VL. It can be seen that our approach (SGVL) outperforms the other baselines except in one category.\n\n\n|              | Winoground Text | Winoground Image | Winoground Group | VL-Checklist Attribute | VL-Checklist Object | VL-Checklist Relation |\n|--------------|:---------------:|:----------------:|:----------------:|:----------------------:|:-------------------:|:---------------------:|\n|   SVLC [2]   |       28.0      |       8.75       |       5.75       |          69.3          |         82.3        |          82.1         |\n|     LLaVa    |       24.8      |       25.0       |       13.0       |          65.5          |         83.1        |          83.0         |\n|   Mini-GPT4  |       23.3      |       18.0       |        9.5       |          71.3          |         84.2        |          84.1         |\n| InstructBLIP |       38.0      |       25.0       |       19.8       |          76.7          |         84.5        |        **91.0**       |\n|     FLAVA    |       32.3      |       20.0       |       14.5       |          54.6          |         70.6        |          46.6         |\n|     ViLT     |       34.8      |       14.0       |        9.3       |          73.3          |         85.0        |          62.0         |\n|   CLIP-SGVL (Ours)  |       32.0      |       14.0       |        9.8       |          72.0          |         82.6        |          82.0         |\n|   BLIP-SGVL (Ours)  |       42.8      |       27.3       |       21.5       |        **81.8**        |         85.2        |          81.9         |\n|  BLIP2-SGVL (Ours)  |     **42.8**    |     **28.5**     |     **23.3**     |          81.2          |       **88.4**      |          88.8         |\n\n**Presentation of the visual prediction**. Thank you for this suggestion! Following your comment, we will include the visual prediction in the main paper from the supplementary materials, as the camera ready includes a 9-page limit, one more page than the current draft.\n\n**Reproducibility**. It is important to note that we have included our code base in the supplementary, and thus the results should be reproducible. We will also release the weights and models publicly upon acceptance.\n\nWe hope the above points have clarified and addressed your concerns. We would be happy to provide any further clarifications as requested.\n\n\n\n\nReferences:\n\n[1] Yu, Fei, et al. \"Ernie-vil: Knowledge enhanced vision-language representations through scene graphs.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 4. 2021.\n\n[2] Doveh, Sivan, et al. \"Teaching structured vision & language concepts to vision & language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Ma, Zixian, et al. \"CREPE: Can Vision-Language Foundation Models Reason Compositionally?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Techniqually, the proposed framework is reasonable and effective.\n\n- Compared with conventional VLM such as CLIP, BLIP, it shows a significant improvement on the proposed task.\n\n- This paper is well-written and easy-to-follow.\n\n- The ablation studies are very extensive.", "reasons_to_reject": "- The idea to leverage scene graph for vision-language model is not very enough now. In the past few years, there are already some references. Unfortunately, the authors do not discuss their diffference in the reference, which may in turn negatively impact the novelties of this work. For example:\n\n[1] Yu, Fei, et al. \"Ernie-vil: Knowledge enhanced vision-language representations through scene graphs.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 4. 2021.\n\n[2] Doveh, Sivan, et al. \"Teaching structured vision & language concepts to vision & language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Ma, Zixian, et al. \"CREPE: Can Vision-Language Foundation Models Reason Compositionally?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n- Lack state-of-the-art comparison in the experimental section. Not only the above reference, but also some state-of-the-art VLM on the scene understanding task.\n\n- Another minor issue is the presentation of this paper: I noticed the visual prediction are all put into the supplementary materials. It would be better to put several in the main text, as this is one of the key results for this paper.", "questions_to_authors": "I would appreciate it if the authors can address the questions below, in the rebuttal stage:\n\n**Q1:** Using scene graph, or more boardly structured information, is not a very novel idea now. The difference of the proposed method against some prior works need to be clarified.\n\n**Q2:** More state-of-the-art comparison, including these latest methods and more advanced VLM.", "paper_topic_and_main_contributions": "This paper proposes an interesting vision-language pre-training method, which leverages the scene graph pairs to jointly train the text and image encoder.\nTechnically, the impact of scene graph is reflected from the adaptive tokens and the prompts. \nIn the output stage, it can additional yield the scene graph output. \nThe framework is pre-trained on LAION, and benchmarked on four scene understanding datasets and one zero-shot classification dataset. \nCompared with conventional VLM, such as CLIP and BLIP, it achieves a better performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Just one presentation issue:\n\nThe visual prediction are all put into the supplementary materials. It would be better to put several in the main text, as this is one of the key results for this paper.", "Justification_for_Ethical_Concerns": "No ethical issues."}, "extra_text": {"Missing_References": "Yes, as I mentioned above, some key references on leveraging scene graph and using it to train VLM are missing. Not only the related work section but also the experimental comparison need to enrich them. For example:\n\n[1] Yu, Fei, et al. \"Ernie-vil: Knowledge enhanced vision-language representations through scene graphs.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 4. 2021.\n\n[2] Doveh, Sivan, et al. \"Teaching structured vision & language concepts to vision & language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Ma, Zixian, et al. \"CREPE: Can Vision-Language Foundation Models Reason Compositionally?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023."}}, {"review_id": "iVwqiqnn9i", "reviewer": "Reviewer_dPsS", "summary": "", "questions": "NA", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Shared Comments\n\nWe thank the reviewers for their insightful comments. We are encouraged that they found the proposed idea of leveraging small scene graph datasets **could benefit the research community immensely** `(dPsS, VrYK)` and **could successfully compensate for the lack of compositional understanding** `(VrYK)`. Furthermore, they found the proposed framework **reasonable and effective** `(639u)` and the architectural changes and losses **easy to implement** `(dPsS)`. Finally, they observed **comprehensive ablations** `(639u, dPsS)`, demonstrating **significant improvements** over the proposed tasks `(639u)`. We are pleased that the reviewers found the paper **well written** `(639u, dPsS)`, and the **diagrams capture** the novelty aspect. \n\nA brief summary of the main aspects of our rebuttal response:\n1) We performed additional experimental comparisons to other methods, including SOTA VLMs `(639u)`.\n\n2) We discussed the comparison between our work and other related works and explained in more detail our novelty compared to other works `(639u)`.\n\n3) We performed a comprehensive analysis of the degradation in performance in certain partitions of the evaluated benchmark and the zero-shot degradation `(VrYK)`.\n\n4) We performed an experiment replacing 1% of LAION with CC3M for training `(dPsS)`.\n\nWe next address all concerns and look forward to an open and constructive discussion with the reviewers.\n\n\n### Response to dPsS\n\n\nWe thank the reviewer for the insightful comments. In the following, we provide a response to the questions raised in the review:\n\n**Consideration of more datasets for fine tuning**. Thank you for this comment! We present results below of our method for BLIP/BLIP2 when using the training set of the COCO dataset, instead of the LAION dataset as reported in the paper. We note that this COCO training data is roughly of the same size as the ~1% of the LAION dataset used in our submission. It can be seen that the results are similar to those reported in the submission.\n\n|                    | Winoground Text | Winoground Image | Winoground Group | VL-Checklist Attribute | VL-Checklist Object | VL-Checklist Relation |\n|--------------------|:---------------:|:----------------:|:----------------:|:----------------------:|:-------------------:|:---------------------:|\n|        BLIP        |       39.0      |       19.2       |       15.0       |          75.2          |         82.2        |          81.5         |\n|        BLIP2       |       42.0      |       23.8       |       19.0       |          77.8          |         84.9        |          84.9         |\n|  BLIP-SGVL (LAION) |       42.8      |       27.3       |       21.5       |          81.8          |         85.2        |          81.9         |\n| BLIP2-SGVL (LAION) |       42.8      |       28.5       |       23.3       |          81.2          |         88.4        |          88.8         |\n|  BLIP-SGVL (COCO)  |       43.8      |       26.0       |       21.3       |          81.9          |         85.5        |          82.0         |\n|  BLIP2-SGVL (COCO) |       46.0      |       26.5       |       22.8       |          81.2          |         88.8        |          88.8         |\n\nRegarding other scene-graph datasets, we can experiment with the Open Images dataset, and report results in the final version (we cannot complete this within the rebuttal timeline).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well written and the diagrams clearly capture what the authors intended to show in their architectural improvements. Utilizing small scene graph datasets to better enable VLMs to create more granular multimodal representations can benefit the research community immensely. The authors provide comprehensive ablation studies and test their methodology using popular VLMs/Datasets. The loss functions and the architectural changes introduced are uncomplicated and easy to follow/implement.", "reasons_to_reject": "It would have been better if more scene graph datasets were considered other than VG and 1% of LAION.", "questions_to_authors": "NA", "paper_topic_and_main_contributions": "The paper is about enhancing structural understanding of images by Vision Language Models(VLMs) using additional signals contained in scene graphs associated with the image. Since scene graph annotation is a costly process, the authors specifically look at whether small scene graph datasets can provide sufficient information to the VLMs during training/finetuning. The authors present a novel way to incorporate scene graph information into visual and textual representations by using adaptive scene graph tokens, positive and negative captions, modifications to the transformer architecture and introduce related losses. The authors try their methodology on various popular VLMs/Datasets and present encouraging results.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "J6WeppqCO7", "reviewer": "Reviewer_VrYK", "summary": "", "questions": "Line 263: How do you represent locations for relationship tokens?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Shared Comments\n\nWe thank the reviewers for their insightful comments. We are encouraged that they found the proposed idea of leveraging small scene graph datasets **could benefit the research community immensely** `(dPsS, VrYK)` and **could successfully compensate for the lack of compositional understanding** `(VrYK)`. Furthermore, they found the proposed framework **reasonable and effective** `(639u)` and the architectural changes and losses **easy to implement** `(dPsS)`. Finally, they observed **comprehensive ablations** `(639u, dPsS)`, demonstrating **significant improvements** over the proposed tasks `(639u)`. We are pleased that the reviewers found the paper **well written** `(639u, dPsS)`, and the **diagrams capture** the novelty aspect. \n\nA brief summary of the main aspects of our rebuttal response:\n1) We performed additional experimental comparisons to other methods, including SOTA VLMs `(639u)`.\n\n2) We discussed the comparison between our work and other related works and explained in more detail our novelty compared to other works `(639u)`.\n\n3) We performed a comprehensive analysis of the degradation in performance in certain partitions of the evaluated benchmark and the zero-shot degradation `(VrYK)`.\n\n4) We performed an experiment replacing 1% of LAION with CC3M for training `(dPsS)`.\n\nWe next address all concerns and look forward to an open and constructive discussion with the reviewers.\n\n\n### Response to VrYK\n\n\nWe thank the reviewer for the insightful comments. In the following, we provide a response to the questions raised in the review:\n\nError analysis. Thank you for your suggestion. We performed error analysis on the cause of degradation in performance in the evaluated subsets below.\n\nFor Table 2, we believe that the minor degradation in state and action splits of VL Checklist may be the result of our generated negative captions being less focused on these aspects. Following your comment, we designed a model with additional negatives such that more attributes of this kind (action, state) are replaced with opposite attributes during finetuning. We note that we only included the 300 most common attributes in our submission. In order to allow more action attributes and state attributes to be replaced, we increased this number to 500. We report again performance on VL-Checklist *Attribute* splits for the base models (CLIP, BLIP, BLIP2), our models from the submission (CLIP-SGVL, BLIP-SGVL, BLIP2-SGVL), and the new mentioned models (CLIP-SGVL^, BLIP-SGVL^, BLIP2-SGVL^). It can be seen that this mitigates the degradation for these splits.:\n\n\n|             | Action       | Color        | Material     | Size         | State        |\n|-------------|--------------|--------------|--------------|--------------|--------------|\n|     CLIP    |     68.1     |     70.2     |     73.1     |     52.9     |     63.3     |\n|     BLIP    |     79.5     |     83.2     |     84.7     |     59.8     |     68.8     |\n|    BLIP2    |     81.0     |     86.2     |     90.3     |     61.7     |     70.1     |\n|  CLIP-SGVL (Ours)  |  76.6 (+8.5) | 78.7  (+8.5) | 81.3  (+8.2) | 59.7  (+6.8) | 62.0  (-1.3) |\n|  BLIP-SGVL (Ours)  | 79.2  (-0.3) | 94.5 (+11.3) | 91.9  (+7.2) | 73.3 (+13.5) | 70.0  (+1.2) |\n|  BLIP2-SGVL (Ours) |  82.4 (+1.4) | 91.7  (+5.5) | 92.2  (+1.9) | 70.1  (+8.4) | 69.6  (-0.5) |\n|  CLIP-SGVL^ (Ours) | 78.1 (+10.0) | 78.5  (+8.3) | 81.5  (+8.4) | 59.7  (+6.8) | 65.0  (+1.7) |\n|  BLIP-SGVL^ (Ours) |  80.5 (+1.0) | 94.5 (+11.3) | 91.7  (+7.0) | 73.5 (+13.7) | 71.4  (+2.6) |\n| BLIP2-SGVL^ (Ours) |  83.5 (+2.5) | 91.7  (+5.5) | 92.4  (+2.1) | 69.8  (+8.1) | 72.0  (+1.9) |\n\nWe note that the rest of the results were not affected, and therefore, we have not included them in this analysis.\n\nFor Table 3 in the submission, there is a minor degradation in the performance of BLIP2-SGVL on the *orientation* and *projective* splits of VSR. We note that these two splits include spatial relations such as: “left of”, “right of”, “in front of”. As mentioned in the VSR paper (see Figures 1 and 2 in [1]), interpreting these spatial relations requires choosing a frame of reference. For some images, a statement can be both true and false, depending on this choice. We compare the results of BLIP2 and BLIP2-SGVL per spatial relation for these two subsets (see table below). The results show that degradation is indeed specific to these relations. We, therefore, hypothesize that some VG annotations might have a different interpretation of these relations, and therefore fine-tuning results in degradation.\n\n\n|            | On top of   | beneath     | beside      | behind      | Left of     | Right of    | under       | In front of | below       | above       | over        | In the middle of |\n|------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|------------------|\n| BLIP2      | 74.5        | 61.2        | 67.6        | 64.2        | 51.3        | 51.5        | 59.8        | 63.5        | 59.1        | 54.1        | 55.4        | 60.0             |\n| BLIP2-SGVL (Ours) | 75.5 (+1.0) | 63.5 (+2.3) | 70.5 (+2.9) | 66.2 (+2.0) | 47.5 (-3.8) | 45.5 (-6.0) | 62.8 (+3.0) | 60.0 (-3.5) | 60.0 (+0.9) | 55.4 (+1.3) | 56.0 (+0.6) | 61.0 (+1.0)      |\n\nZero-shot degradation. Thank you for raising this concern. As mentioned in our paper, we believe that the slight degradation in Zero-shot classification in our SGVL models is the result of fine-tuning with negative captions and our graph prediction tasks that deviate from the original VLM training objective. This phenomenon is also visible in recent works [2, 3]. Classification performance is improved if our models are tested with linear probing. For CLIP-SGVL, when using 5-shot, 10-shot linear probing the difference on classification tasks with respect to CLIP (in the same setting) changes to -0.3%, +0.1% and with 20-shot to +0.5%. We will report the N-shot results in the final version.\n\n**How do we represent location for relationship tokens?** The location of the relationships in an image are represented by the union of bounding boxes of objects comprising them.\n\n**“I don't think unsupervised training is a natural extension of this work. The main insight brought by this work is the necessity of a small amount of densely annotated data. Future directions might include generalizing this approach beyond the VL.”** Thank you for your suggestion! We absolutely agree that this is a main insight of our work. Indeed, future directions might include generalizing this approach beyond the VL domain or applying it to other types of dense structured data (such as, segmentation masks, sketch, depth maps, normal maps etc.). We would emphasize this point further in the final version.\n\nWe hope the above points have clarified and addressed your concerns. We would be happy to provide any further clarifications as requested.\n\nReferences:\n\n[1] \"Visual Spatial Reasoning\", TACL'23. \n\n[2] \"Teaching structured vision & language concepts to vision & language models. CVPR 2023.\n\n[3] “When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?”. ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper demonstrates that the standard contrastive language-image pretraining cannot sufficiently imbue a model with compositional understanding capabilities. Therefore, the authors propose to leverage information about relations and attributes from scene graphs. It's demonstrated that a small number of scene graph annotations could successfully compensate for the lack of compositional understanding. The results would encourage the community to adapt models for better compositional scene understanding with moderate effort.", "reasons_to_reject": "The proposed method does not consistently improve performances. As mentioned in Line 485-496, zero-shot performance was degraded. Table 2&3 also show that the  proposed method harmed performances on certain partitions of the evaluation benchmark. I'm expecting more error analysis on the cause of such degradation. For example, is there a commonality of all tasks where performances were harmed by SGVL, such that during application, users could wisely choose when to adopt BLIP vs. BLIP-SGVL?", "questions_to_authors": "Line 263: How do you represent locations for relationship tokens?", "paper_topic_and_main_contributions": "The work proposes to teach vision-language Transformers structured information leveraging scene graphs (SGs). Supervision signals are obtained from SGs in two ways: 1)  positive and negative captions are generated with different compositional substituents; 2) object labels, relations and coordinates are used to supervise a set of special \"scene graph tokens\".\n\nThe learnable scene graph tokens possess a separate set of attention and feedforward-layer parameters. They are allowed to interact with regular patch tokens through self-attention, which can be later leveraged for predicting the text embedding of objects or relations.\n\nThe entire model is finetuned from CLIP/BLIP via LoRA and outperforms various baselines in hard VL benchmarks requiring compositional scene understanding.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line603 qualitative annotations --> high-quality annotations\n\nLine605-606: I don't think unsupervised training is a natural extension of this work. The main insight brought by this work is the necessity of a small amount of densely annotated data. Future directions might include generalizing this approach beyond the VL domain or using other types of dense structured data (e.g. segmentation masks, sketch)\n"}}]

Author rebuttal:

Meta-review: The paper aims to use scene graph to improve the VL models. The scene graph supervises the vision branch via two tasks: 1. direct supervise with token prediction. 2. serves to construct hard negatives in the language branch. The overall paper quality is good.
Decision: Accept-Main