Measuring Progress in Fine-grained Vision-and-Language Understanding
Year: 2023

Abstract:
While pretraining on large-scale image–text data from the Web has facilitated rapid progress on many vision-and-language (V&L) tasks, recent work has demonstrated that pretrained models lack “fine-grained” understanding, such as the ability to recognise relationships, verbs, and numbers in images. This has resulted in an increased interest in the community to either develop new benchmarks or models for such capabilities. To better understand and quantify progress in this direction, we investigate four competitive V&L models on four fine-grained benchmarks. Through our analysis, we find that X-VLM (Zeng et al., 2022) consistently outperforms other baselines, and that modelling innovations can impact performance more than scaling Web data, which even degrades performance sometimes. Through a deeper investigation of X-VLM, we highlight the importance of both novel losses and rich data sources for learning fine-grained skills. Finally, we inspect training dynamics, and discover that for some tasks, performance peaks early in training or significantly fluctuates, never converging.

heading: 1 Introduction
text: Fine-grained multimodal skills (e.g., understanding relationships and recognising verbs) require identifying and relating various entities across both image and text modalities. Vision-and-language models (VLMs) need such skills to robustly perform well on real-world vision-and-language (V&L) applications; e.g., a coarse-grained model tested on image retrieval to “find an image where something is on a sofa” might incorrectly return an image of a cat sitting below the sofa. As another example, in captioning, a model might incorrectly describe an image where “someone is selling a sweater” as “someone is buying a sweater,” if it does not have a precise understanding of the two verbs.
∗Work completed during an internship at DeepMind. ‡denotes equal senior contribution. Correspondence to: Emanuele Bugliarello <emanuele@di.ku.dk>.
However, common V&L benchmarks (e.g., Lin et al., 2014; Goyal et al., 2017; Suhr et al., 2019) do not explicitly shed light on such fine-grained understanding. Indeed, in the last few years, there has been an increase in the number of benchmarks which demonstrate that current, coarsegrained models struggle with fine-grained understanding (Hendricks and Nematzadeh, 2021; Parcalabescu et al., 2022; Salin et al., 2022; Thrush et al., 2022). Meanwhile, more models have been designed specifically to learn a better mapping between visual and textual modalities (e.g., Yao et al., 2022a,b; Zeng et al., 2022; Gao et al., 2022). While such models perform well on coarse-grained retrieval and other downstream tasks, they have not been directly evaluated on fine-grained understanding. Consequently, it is unclear if the performance gains are due to tighter, more fine-grained representations introduced by model innovations at the pretraining stage. To fill this gap, we analyse several recent models with innovations designed for a better image–text alignment and their corresponding baselines on a suite of fine-grained benchmarks. We centre our study on three key questions.
First we consider: Which models perform well on fine-grained tasks? To answer this, we evaluate models from four different model families trained with different amounts of pretraining data, as well as recent architectures that leverage frozen large language models (LLMs). We observe that modelling innovations have more impact than simply scaling image captions from the Web. Furthermore, explicitly modelling localisation can improve performance, but it is crucial how it is done, and simply using localisation data is not enough.
Our observations motivate our next question: How do data and losses impact fine-grained understanding? We focus our study on the best performing model, X-VLM (Zeng et al., 2022), which learns to map specific objects and regions (not a full image) to a label (word or phrase describing the
1559
Benchmark Task Examples Subtasks Example Subtasks
Fine-grained Tasks
SVO-Probes Verb 48K 3 subject, verb, object understanding VALSE V&L 14K 6 existence, counting, grounding spatial relations VSR Spatial 2K 7 adjacency, directional, reasoning proximity relationships Winoground Compositional 800 8 pragmatics, object swap, reasoning relation swap
Coarse-grained Tasks
region). We reformulate the X-VLM loss to better disentangle the contribution of data and losses, observing that more data does not improve performance unless paired with losses designed to learn a mapping between regions and labels. Furthermore, the diversity of class labels is important for performance on coarse-grained retrieval, and region descriptions (as opposed to single word labels) are crucial for performance on fine-grained tasks.
Finally, it is unclear if all fine-grained skills are learned at the same time during training so we consider: How does fine-grained understanding evolve during training? Surprisingly, we find that while performance steadily improves on coarse-grained retrieval tasks through training, performance fluctuates substantially on many fine-grained tasks, with some skills, like counting, becoming increasingly worse. Additionally, performance across different fine-grained tasks that should test for similar skills are not always well correlated.
Contributions. In this work, we 1) provide indepth analyses of how data and modelling decisions impact performance on fine-grained tasks, and 2) further disentangle the gains given by data and pretraining losses on our best performing model (XVLM). Our results suggest that to make progress in fine-grained understanding, modelling innovations (e.g., through object-centric losses) as well as data quality and richness are more effective than scaling up Web data alone. Finally, we 3) shed light on VLMs’ pretraining dynamics and suggest that future work should revisit pretraining strategies in order to consistently improve across several tasks.

heading: 2 Benchmarks
text: We describe the recent (English) benchmarks proposed to measure fine-grained V&L understanding
in zero-shot setups.1 See Table 1 for an overview. SVO-Probes (Hendricks and Nematzadeh, 2021) focuses on verb understanding: it tests whether a model can identify if an image matches a sentence, and includes negative images which differ on a specific part of speech (Subject, Verb, and Object). The dataset consists of 421 verbs and over 48K image–sentence pairs.2 The authors show that their baselines fail more in situations requiring verb understanding than other parts of speech.
VALSE (Parcalabescu et al., 2022) consists of six tasks that cover basic linguistic phenomena, such as plurality, actions and coreference. For each task, given a visual input, a model is asked to distinguish real captions from foils (Shekhar et al., 2017), where a foil is constructed from a caption by altering a word or phrase that realises a specific linguistic phenomenon (e.g., semantic number of nouns). The authors show that VLMs can identify objects in images, but struggle to ground their interdependence with specific linguistic indicators.
VSR (Liu et al., 2023) tests for 65 types of visual spatial relationships (e.g., under, in front of) grouped into seven categories (e.g., adjacency, orientation). Each sample consists of an image– sentence pair; a model needs to predict whether the sentence correctly describes the spatial relation between two objects in the image. We evaluate models in a zero-shot setup on the ‘random’ split.3
Winoground (Thrush et al., 2022) is an expertcurated benchmark aiming to test models’ compositional reasoning. Given two images and two captions, the goal is to match them correctly; wherein both captions contain the same set of words, but in a different order. The authors define three scores: Text (whether a model can match the correct caption for a given image), Image (vice versa), and Group (whether a model can match each pair). Several competitive VLMs have been shown to often perform close to or below random chance.
We also report zero-shot performance on coarsegrained retrieval in Flickr30K (Young et al., 2014) and COCO (Lin et al., 2014) in our analysis.
1We note that two more datasets require fine-grained skills to be solved and that they are not part of our analysis. ImageCoDe (Krojer et al., 2022) requires comparing a caption within a multi-image context, a setup not suitable for zeroshot evaluation of current single-image VLMs. Yuksekgonul et al. (2023) propose the ARO benchmark to evaluate VLMs’ attribution, relation, and order understanding. However, the data had not been released as of the ACL deadline.
2Only 30,578 pairs were available as of Nov 2022. 3Note that VSR has recently been updated, but we expect the findings from our experiments to hold on the revised splits.

heading: 3 Evaluated Models
text: Recent work has shown that two components are crucial ingredients of strong coarse-grained VLMs (e.g., Li et al., 2021; Alayrac et al., 2022; Chen et al., 2023): 1) a contrastive objective that aligns vision and language modalities, and 2) a cross-attention mechanism that fuses the two modalities. As we are interested in high performance on both fine- and coarse-grained tasks, to select models for our study, we surveyed recent work that uses these building blocks,4 but also incorporates new losses or data that can potentially improve fine-grained V&L understanding. We find that many recent models build on ALBEF (Singh et al., 2022; Yang et al., 2022; Hao et al., 2023) (which we also study as a coarse-grained baseline).
Other than strong performance on coarsegrained and downstream tasks, we also considered: 1) the possibility to study the role of new modelling innovations and data for fine-grained skills, and 2) the availability of open-source code and pretrained weights. This resulted in four models briefly described next (more details in App. A.1). Table 2 codifies the main differences in pretraining objectives and data used by these models. Recall that previous work does not evaluate these models on fine-grained benchmarks.
ALBEF (Li et al., 2021), with strong downstream performance, matches all our criteria and serves as a coarse-grained baseline. ALBEF is a dual-stream encoder (Bugliarello et al., 2021) that first encodes images and captions independently, and then fuses them with cross-modal attention.
BLIP (Li et al., 2022b) uses an autoregressive language model (LM), and employs a dataset bootstrapping technique (CapFilt) to generate synthetic captions and to remove noisy pairs from large-scale
4By studying models with well-established modules, we expect our findings to be more informative for future work.
Web data. BLIP outperforms ALBEF on most coarse-grained downstream tasks; thus, we study BLIP as another coarse-grained baseline to test if its generative LM and data contributions also lead to better fine-grained understanding.
PEVL (Yao et al., 2022b) is a fine-grained model building on ALBEF, but leverages more supervised datasets such as referring expressions, captions with visual coreferences, object detection and region descriptions data, etc. (see Table 2). Unlike ALBEF, PEVL is explicitly trained to learn fine-grained representations of entities by predicting their coordinates in a unified masked language modelling framework (similar to Pix2Seq, Chen et al., 2022): bounding box coordinates corresponding to a given entity are added in the caption as “A cat < 10 73 206 175 > is napping.”
X-VLM (Zeng et al., 2022) is our second finegrained model that enhances ALBEF by adding both new losses and additional supervised data. In contrast to PEVL, X-VLM models visual position through an additional bounding box prediction head that regresses the object’s bounding box (bbox) coordinates. The authors use both object detection labels and region descriptions to learn coarse- and fine-grained alignments (we provide an in-depth analysis of this model in Section 5).
We remark that PEVL and X-VLM were the only open-source fine-grained VLMs at the time of our evaluation, and both of them build on top of ALBEF. In addition to these core models, we also evaluate a dual-encoder network (CLIP; Radford et al. 2021) as well as recent architectures that rely on frozen, autoregressive (L)LMs: CLIPCAP (Mokady et al., 2021), FLAMINGO (Alayrac et al., 2022) and BLIP-2 (Li et al., 2023). As these models perform generally worse than our best fine-grained model, X-VLM, and differ significantly from it, we do not discuss their performance further. For more details, we refer the reader to Tables 6 to 11 in App. B.1.

heading: 3 ALBEF14M 88.6 69.4 58.3 32.5 16.2 12.7

heading: 4 BLIP14M 48.7 67.8 49.7 36.5 18.5 14.5

heading: 9 BLIP129M 51.4 68.8 46.9 35.5 15.0 11.7

heading: 4 Which Fine-grained Models Perform Well on Fine-grained Tasks?
text: We compare two strong VLMs (ALBEF and BLIP) with two models with explicit object modelling (i.e., fine-grained; X-VLM and PEVL). We evaluate on fine-grained tasks (see Table 3) to determine if recent object-centric models improve on tasks designed to measure fine-grained skills—an evaluation missing from previous work. We also include results on CLIP and BLIP-2 in Table 3 to highlight how well fine-grained models perform, even though pretrained with less data and having fewer parameters (as shown in Table 6 in App. B.1).
Experimental setup. All our fine-grained benchmarks only require models to predict a matching score for a given image–text pair, a common task that current V&L models—including all of our evaluated models—are pretrained to solve. On VSR, a model’s prediction is correct if the matching score is greater/lower than 50% for a true/false label. On the other benchmarks, a model’s prediction is correct if the score for the positive image–text pair is higher than the score of the negative pair(s).5 We evaluate the public models released by the authors on GCP.6 Code to reproduce our analysis is online.7
ALBEF vs. BLIP. We first compare our two coarse-grained baselines. A key difference between ALBEF and BLIP is that the former is trained with masked language modelling (MLM), while
5We evaluate SVO-Probes using pairwise ranking accuracy to benchmark models without a binary classification head (we note that Hendricks and Nematzadeh 2021 used accuracy).
6https://cloud.google.com/. 7https://github.com/e-bug/fine-grained-evals.
the latter uses autoregressive language modelling (LM) for text; with BLIP outperforming ALBEF on downstream tasks when pretrained on the same 14M images. Performing the same comparison on fine-grained benchmarks, we find that ALBEF14M outperforms BLIP14M on all tasks (largely on SVOProbes and VSR) except on Winoground. Likewise, Table 6 (App. B.1) shows that other visualconditional LMs, such as CLIPCAP models, also struggle with fine-grained understanding. This might be due to the fact that our evaluation relies on image–text alignments and does not test for generation, where the LM objective is often preferred. Given these results and the fact that ALBEF is more similar to our fine-grained models, we compare against ALBEF in most of our discussion.
Effectively modelling object positions improves fine-grained understanding. Overall, we find that X-VLM consistently outperforms all other evaluated approaches (see Table 3). This trend holds in both the 4M and 16M pretraining setups. When trained on the same 4M images as the ALBEF baseline, X-VLM with explicit object modelling, notably improves over all benchmarks (gaining 1.3pp on SVO-Probes, 3.3pp on VALSE, 5.7pp on VSR, and 14.8/11.2/11.5pp on Winoground). Importantly, X-VLM4M also outperforms ALBEF14M (trained on 10M more data points). This result shows the importance of explicit object modelling for a range of fine-grained tasks, including ones that are dissimilar to the supervised localisation task (e.g., verb understanding).
X-VLM16M, which adds CC12M as well as object detection data from OpenImages and Objects365 to X-VLM4M’s data, achieves even higher overall gains in most fine-grained benchmarks. On VALSE, it closes the gap with a larger model trained on supervised data from many downstream tasks (12-in-1; Lu et al. 2020), and on VSR it achieves similar accuracy to LXMERT (Tan and Bansal, 2019) fine-tuned on 50% of VSR training data (67.9pp). Moreover, on Winoground, XVLM4M significantly outperforms previous coarsegrained models, including a large-scale dualencoder (CLIP, Group score of 8.0; Radford et al., 2021) and a strong, larger cross-modal Transformer (UNITERLarge, Group score of 10.5; Chen et al., 2020), as shown in Table 6 in App. B.1.
Not all object modelling improves fine-grained understanding. Like X-VLM, PEVL also mod-
els visual locations of objects. However, it does so by expecting (masked) bbox locations as part of its input caption. Surprisingly, PEVL14M performs much worse than X-VLM16M on all tasks; in fact, it performs on par with the ALBEF14M baseline, despite being originally initialised with its checkpoint and further tuned to model visual object locations.8 We conjecture that modelling objects as input prompts is less beneficial than directly predicting object locations with a classification head (X-VLM), as the former does not directly influence the object’s representations in the text modality.
Modelling objects has more impact than increasing data. In Table 3, we observe that, not surprisingly, increasing data for a given family (e.g., ALBEF4M to ALBEF14M) results in improved performance on most benchmarks. However, interestingly, the fine-grained X-VLM4M, trained on 4M data points, outperforms all BLIP129M variants—a coarse-grained model trained on 129M data points (compare row 2 with rows 9–11). Similarly, while increasing the data from 4M to 14M results in improvements across most tasks for the coarsegrained ALBEF14M, these performance gaps are smaller than what we gain from modelling objects on top of ALBEF4M. That is, the average performance gap between ALBEF4M and X-VLM4M is bigger (+5.2pp) than that observed when increasing data from ALBEF4M to ALBEF14M (+1.0pp). This result highlights that simply scaling data, without modelling innovations, might not be enough for notable improvements on fine-grained tasks.
We also find that scaling data can hurt performance on some benchmarks. For example, on Winoground Image and Group scores, XVLM16M and BLIP-VIT/L129M perform worse than their corresponding models trained on less data, X-VLM4M and BLIP14M, respectively.9 Looking at performance by subtasks, we find that scaling Web data leads to worse performance on several of them, such as Image scores in most Winoground tasks, and VALSE’s existence, counting adversarial and coreference for BLIP-VIT/L129M (more details in App. B.1). We
8We evaluate three different models released by the authors, which differ in their pretraining and fine-tuning data. All the variants perform similarly, and as a result, we only report PEVL14M, which underwent a second-stage pretraining on multiple supervised tasks (App. B.1 lists all the models).
9While BLIP129M performs worse than BLIP14M on a few benchmarks, this might be because the data size is significantly increased without scaling the model size. Thus, we compare against BLIP-VIT/L129M, which uses a larger image encoder.
conjecture that pretraining on noisy Web data— where the language in an image–text pair does not always faithfully describe the image—might diminish the fine-grained alignments learned from smaller, cleaner datasets (Hendricks et al. 2021 report similar trends on coarse-grained tasks).
Takeaways. We observe that modelling object positions in images provides a strong signal for fine-grained understanding; but, how we model this information is crucial: simply pretraining a model with bbox positions in input does not lead to better off-the-shelf representations. We also see bigger gains on fine-grained tasks when modelling objects compared to scaling the pretraining data.

heading: 5 Data & Losses for Fine-grained Tasks
text: Recent fine-grained models build on coarse-grained ones by introducing additional training data (e.g., object detection data in X-VLM and PEVL) and new losses (e.g., bounding box regression loss in X-VLM). We study how data and losses influence fine-grained understanding, focusing on X-VLM as it outperforms other models on fine-grained benchmarks. While Zeng et al. (2022) perform ablations to show the importance of their new objective function, they do not study the impact of data and losses independently; moreover, they do not evaluate on find-grained benchmarks. We start with a description of X-VLM, emphasising details in its pretraining procedure, that we reveal to have significant impact on the final performance.

heading: 5.1 What are X-VLM Data and Losses?
text: The X-VLM architecture consists of the same modules as ALBEF: a vision, a text, and a crossmodal Transformer (Vaswani et al., 2017) encoder (see App. A.1 for details). Given an image–text pair, ALBEF performs two forward passes (as shown in Figure 1): first, the model computes a contrastive learning loss (LCL) and an image– text matching loss (LITM). In a second pass, it masks text inputs to compute a visually-grounded masked language modelling loss, LMLM. After the two forward passes, ALBEF is trained with LA = LCL + LITM + LMLM.
Data. While ALBEF is only pretrained on image–caption data, X-VLM additionally pretrains on object and region detection data. Object detection data consists of an object or attribute–object label (e.g., “dog” or “brown dog”), an image, and
Confidential + Proprietary
a bounding box; region detection data consists of a short phrase (e.g., “a cute brown dog”), an image, and a bounding box. Other multimodal Transformer models have used detection data (Hendricks et al., 2021; Li et al., 2020; Bugliarello et al., 2021; Zhang et al., 2021), but usually the bounding boxes are discarded, and objects or region descriptions are paired with the entire image. In contrast, a close examination of the X-VLM codebase10 reveals that X-VLM effectively makes use of bounding boxes.
BBOX loss. To take advantage of additional bounding box (bbox) data, X-VLM introduces an objective, Lbbox, which learns to regress to object locations from object detection and region description data (see Figure 1 for an overview).
VMA loss. The X-VLM paper presents two losses, LA and Lbbox. However, LA operates over two input types: image–text pairs from captioning data and image–text–bbox triplets from object detection data. Thus, it is hard disentangle the impact of the data and the losses on performance. We reformulate LA into two losses,11 operating over: (a) image–text pairs, LA, as in ALBEF; or (b) image–text–bbox pairs, that we denote visually masked ALBEF loss, LVMA. For LVMA, the visual and cross-modal encoders only attend to the image patches that correspond to the object bbox coordinates via an attention mask (see Figure 1). This results in an object-centric visual view for grounding the text label through the pretraining objectives. To compute this loss, in addition to the three forward passes described so far (CL and ITM, MLM, and BBOX losses), X-VLM performs two more passes: one where image patches outside a bounding box region are masked out to compute the visually masked CL and ITM loss, and another where text is additionally masked for the visually masked MLM loss. Section 5.3 shows both the
10https://github.com/zengyan-97/X-VLM. 11Our reformulation is equivalent to X-VLM, but it allows us to disentangle the impact of data and losses on performance.
data and pretraining techniques are key to the final model performance.

heading: 5.2 Experimental Setup
text: We re-implement ALBEF and X-VLM in JAX to ensure full control of modelling, data, and initialisation decisions.12 We initialise both models with a 224×224 ViT-B/16 visual encoder (Steiner et al., 2022), and BERTBASE (Devlin et al., 2019) weights in the text and cross-modal layers. Similar to Bugliarello et al. (2021), we pretrain our models on the exact same 4M and 14M datasets used by the authors (Table 2), but note that only 1.8M and 11.2M data points were available for CC3M and CC12M, respectively. For object detection data, we use the COCO and VG annotations released by the X-VLM authors. Following Zeng et al. (2022), we pretrain our models for 200K steps using the official hyperparameters (see App. A for more details).

heading: 5.3 Results
text: Table 4 shows the overall zero-shot performance of our ablations on three fine-grained benchmarks and two coarse-grained retrieval tasks. Row 0 is our ALBEF re-implementation, while row 10 corresponds to our X-VLM pretrained following the implementation of Zeng et al. (2022). Our controlled study allows us to quantify how each technique (losses, data, implementation details) in X-VLM contributes towards fine-grained understanding.
Data ablation. We first investigate the role of supervised detection data used to learn fine-grained relationships in X-VLM by pretraining the model, using its standard training objectives, and adding different data sources (rows 1–6).
Looking at rows 1–3, we find that region descriptions from VG (VGRD) are the most useful,
12To verify our implementation, we compare an ALBEF model trained in our codebase with one trained in the original codebase, obtaining an absolute difference below 1pp in Recall@1 on zero-shot Flickr30K and COCO retrieval tasks.
single-source signal for the model, resulting in improvements in both fine- and coarse-grained tasks. This variant is either close to or surpasses the final X-VLM variant (row 10) in all the tasks. We attribute this success to both its size (3.7M data points) and language format, wherein noun phrases, rather than simple labels, describe a given entity. In addition, object detection data from VG (VGOD) leads to similar fine-grained results as COCOOD, but significantly better zero-shot retrieval performance. VGOD is not only larger than COCOOD, but also includes a more diverse set of classes.13
We hypothesise that a large number of classes (as in VGOD) is important for coarse-grained retrieval tasks, and more descriptive phrases of VGRD (rather than single labels) significantly impact fine-grained tasks. To verify this, we disentangle the effect of data size and type: specifically, we re-train rows 2–3 on a subset of VG with the same number of images and annotations as in COCOOD. Figure 2 confirms our hypothesis: even when controlled for size, VGRD leads to notably better performance than COCOOD. On coarse-grained datasets, VGOD largely outperforms COCOOD.
Looking at multi-source supervised data (rows 4–6), our best performing model combines VGOD and VGRD data (row 6) and, surprisingly, adding COCOOD does not boost performance.
Loss ablation. We investigate the role of the two objectives used during supervised pretraining of XVLM (rows 7–9). We see that training an ALBEF model on object detection data as-is (row 7) results in similar performance as pretraining it on standard
13COCOOD and VGOD have 80 and 50k labels respectively.
image–caption data. That is, just adding more data is not enough; additional supervision in the form of the X-VLM pretraining objectives is crucial.
Compared to Lbbox (row 9), our reformulation makes it clear that LVMA (row 8) leads, on average, to both higher fine-grained accuracy and higher recall on retrieval tasks. One potential explanation is that the visually masked forward pass directly influences the representation learned by the contrastive loss, as well as the cross-modal representations. In contrast, the regression loss only occurs after crossmodal interaction, suggesting that better alignment is important in both contrastive and cross-modal features. Finally, X-VLM achieves its best performance when combining LVMA and Lbbox.
Takeaways. Our reformulation of X-VLM allows us to conduct a careful analysis in a controlled setup on how data and losses influence X-VLM performance. We show that more data does not improve performance unless paired with additional supervisory signal, in the form of either the visually masked ALBEF loss or bbox regression. Given our observations and the fact that, as seen in Section 4 and App. B.1, X-VLM largely outperforms
the large-scale CLIP and BLIP-2 models on finegrained tasks such as VALSE and Winoground, we believe that a promising direction in fine-grained understanding will require careful model and loss design with rich data sources like VG, not just scaling up with (potentially) noisy data.

heading: 6 Dynamics of Fine-grained Tasks
text: We now analyse the dynamics of fine-grained skills for our models to investigate (i) when and whether they are acquired, and (ii) how they relate to one another, especially when they aim at measuring similar capabilities. For example, does action understanding in VALSE correlate with verb understanding in SVO-Probes? Are there skills that vastly differ from each other that they would require different modelling contributions (e.g., counting)?
Experimental setup. We evaluate checkpoints (every 10K steps) from pretraining our ALBEF and X-VLM re-implementations with 4M and 14M data points. We focus on 14M results as we see similar patterns with 4M (see App. B.2). When evaluating correlation patterns, we report both Pearson and Spearman correlation coefficients.
Different skills, different patterns. Figure 3 (top) shows how the average model performance evolves during pretraining for the four benchmarks. Interestingly, the performance on these benchmarks converges at different rates: both ALBEF and X-VLM models easily improve on SVO-Probes. Moreover, we observe that modelling objects (à la X-VLM) leads not only to better fine-grained understanding after 200K steps (Tables 3 and 4), but also to remarkably quicker learning rates.
Figure 3 (bottom) shows performance on indicative VALSE tasks, as well as on coarse-grained
image retrieval on COCO. While some skills, such as spatial relations understanding, are learned progressively during pretraining, others, such as counting, degrade after a first, short learning phase. Finally, other skills, such as coreference resolution, oscillate significantly throughout pretraining, showing how models can not properly acquire them. This is in contrast to the coarse-grained COCO retrieval task for which the performance steadily increases over time. We conclude that it is particularly important to examine the training dynamics of fine-grained tasks, and that a single checkpoint might be inadequate for a number of skills. Results on all tasks are provided in App. B.2, including on Winoground for an ALBEF4M that we pretrained on GCP using the original codebase.
Same skills, same patterns? We next investigate whether closely-related tasks in different benchmarks have high correlation throughout pretraining. While we find that VALSE action replacement and SVO-Verb have a +55/67% Pearson/Spearman correlation, there is a -13/11% correlation between VALSE actant swap and SVO-Subject.
Looking at VALSE spatial relations, we find high correlation (+75/65%) with average VSR performance, and especially with relations such as on top of, on, inside, by, and in; mostly belonging to the ‘Topological’ category in VSR. On the other hand, we find almost no correlation with several ‘Directional’ (e.g., across from) and ‘Orientation’ (e.g., parallel to) relations, as well as with some ‘Topological’ ones (e.g., touching); and even negative correlation (-40% or less) with alongside, below, toward, part of and near.
Finally, surprisingly, VSR dev and test splits are not positively correlated for all relations. While average performance is highly correlated (+77/78%),
only a few relations have Pearson/Spearman coefficients larger than 30% (in, on, above, within, and consists of). On the other hand, near, ahead of and adjacent to are negatively correlated between dev and test sets, and most relations show very low correlations between the two sets. As a result, improvement in a given relation type on the dev set, will likely not transfer at test time.
Takeaways. When tested on fine-grained benchmarks, we observe that, compared to ALBEF, XVLM is more sample efficient as it achieves higher performance with fewer training steps. Also, while some tasks steadily improve during pretraining, for others, the performance degrades or fluctuates. Moreover, surprisingly, the performance of tasks measuring similar skills but from different benchmarks do not always positively correlate.

heading: 7 Discussion
text: While recent pretrained VLMs achieve impressive performance on various downstream benchmarks (such as visual question answering and image retrieval), recent benchmarks have highlighted that they still struggle with tasks that require finegrained understanding—where a model needs to correctly align various aspects of an image to their corresponding language entities. Yet, it is still not known to which extent recent fine-grained VLMs (e.g., Zeng et al., 2022; Yao et al., 2022b; Li et al., 2022a; Dou et al., 2022) fare on such benchmarks. We address this gap by evaluating strong and fine-grained models on four benchmarks (Hendricks and Nematzadeh, 2021; Parcalabescu et al., 2022; Liu et al., 2023; Thrush et al., 2022), and encourage future work to report zero-shot finegrained performance on our selection of benchmarks, especially if models are not open-source.
Our work contributes to a growing thread of research devoted to understand what is learned by pretrained VLMs, such as studying cross-attention patterns (Cao et al., 2020), cross-modal input ablations (Frank et al., 2021), probing linguistic and visual structure (Milewski et al., 2022; Salin et al., 2022; Nikolaus et al., 2022), robustness to words order (Akula et al., 2020; Thrush et al., 2022), and incorrectly fusing image and language modalities (Diwan et al., 2022). Here, we show that object modelling through a prediction loss (as done in X-VLM) results in notable improvements across all benchmarks, outperforming models trained on much larger amounts of Web data. Our analysis
highlights that teaching VLMs concepts of objects (e.g., by masking irrelevant parts of the image) is crucial for effectively learning fine-grained skills. Though our models rely on supervised data to learn better localisation, we hope our findings can encourage researchers to design better loss functions for image–text mapping from unsupervised, Webscale data as well.
Finally, our results also highlight the challenges of evaluating fine-grained understanding: the recent benchmarks capture a variety of subtasks (from counting to relation understanding); to perform well on these subtasks, a model requires different skills. Indeed, we observe that, during training, model performance does not always increase for all subtasks, and in particular, fluctuates a lot for counting, coreference resolution, and various spatial relations. An important future direction is designing models that perform well on a larger range of these subtasks, where improving on one subtask does not degrade performance on the rest. It is unclear why benchmarks do not always correlate; possible reasons include the data itself (images selected for analysis, annotator instructions), or that different competencies are required for different fine-grained tasks. We hope future work can explore this further, possibly by closely examining data in fine-grained benchmarks or expanding the models used in analysis beyond what we used here.
Limitations
Our work focuses on assessing recent English VLMs on tasks which require fine-grained understanding. Here, we outline limitations that we believe are important considerations for future work.
First, we only examined a limited number of models. These include (i) strong coarse-grained models, such as ALBEF, CLIP, FLAMINGO and BLIP-2, and (ii) two strong fine-grained models, PEVL and X-VLM, that build on ALBEF. While we believe our selection of models is representative of strong components in pretrained VLMs (such as dual-encoder and cross-modal interactions), we could not easily evaluate different approaches towards fine-grained understanding (e.g., Yao et al., 2022a; Li et al., 2022a) as the corresponding models and code are not open-source. We hence hope our study will motivate future work to report zeroshot performance on fine-grained benchmarks.
Second, we evaluate our models in a zero–shot setting using image–text matching. Future work
could consider how fine-grained understanding improves when fine-tuning for specific tasks. As opposed to relying on image–text matching scores, alternative methods like input ablations, visualising attention or activations could also be used to gain an understanding of potential failure modes.
Third, though we note specific areas where model performance fluctuates a lot during pretraining, we look forward to future research that improves performance for various such areas, like existence and counting.
Finally, some datasets we use are quite small. For example, Winoground only has 1,600 data points. We hope that our analysis sheds light on the kinds of skills models struggle with and encourages more and larger datasets that test for these skills.
Ethics Statement
All datasets used in this work have been previously published. Multimodal datasets frequently include social biases (Meister et al., 2022), and we expect the models trained on them to reflect the biases in these datasets. Datasets also include images of people, and there is no mechanism for people to remove themselves from these datasets.
Multimodal models have many downstream uses. Some examples of beneficial applications include: more advanced image and video retrieval, visual description systems to aid the visually impaired, and interfaces which allow users to more seamlessly interact with smart home devices. Harmful applications might include surveillance, especially when imagery of people is being used without their consent, or fine-tuning a model to retrieve harmful content, such as pornographic material.
In this work, we aim to understand how models perform on fine-grained tasks which highlights current failure modes of our models. We hope insights from our work can inspire (i) novel models which perform well on a broad set of fine-grained tasks, as well as (ii) more high quality data to stress test our models. We hope our work also helps those who might use multimodal models in downstream applications better anticipate how well these models might perform on their tasks.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers, Antoine Miech, Ravichandra Addanki, Wojciech Stokowiec, Chris Dyer and the DeepMind Language Team for feedback on this project.

heading: A Experimental Setup
text: In this section, we provide further details on the experimental setups that we used for our studies.
A.1 Evaluated Models: Details
We provide more details on the models we use to evaluate progress in fine-grained V&L understanding. See Table 5 for an overview.14
ALBEF (Li et al., 2021) is a recent VLM that has gained popularity due to its design choices, effectively combining core components in V&L learning, such as a contrastive objective and cross-attention, that result in strong downstream performance. ALBEF is a dual-stream encoder (Bugliarello et al., 2021) that first encodes images and captions independently with a vision (ViT; Dosovitskiy et al. 2021; Touvron et al. 2021) and text (BERT; Devlin et al. 2019) Transformer, respectively; and then fuses them in a cross-modal Transformer. The model is pretrained with three objectives: masked language modelling (MLM), unimodal image–text contrastive learning and cross-modal image–text matching. We refer to the original work for more details. While ALBEF does not explicitly train for fine-grained understanding, it serves as an important baseline since our three other models build on top of it.
BLIP (Li et al., 2022b) is a unified V&L understanding and generation model, that can be applied to a wide range of downstream tasks. A key component to BLIP’s success is CAPFILT: a dataset boostrapping method which the authors use to generate synthetic captions and removing noisy pairs from large-scale Web data. Moreover, unlike any other model we evaluate, BLIP uses an autoregressive language modelling (LM) objective to convert visual information into coherent captions, allowing us to evaluate the potential benefits of this objective to learn fine-grained relationships. BLIP is not explicitly trained for fine-grained understanding, however, we believe it is important to assess whether generative language modelling and its data contributions that enhance downstream performance also lead to better fine-grained skills.
PEVL (Yao et al., 2022b) explicitly connects image regions and text tokens through cross-modal position modelling. Similar to Pix2Seq (Chen et al.,
14Each model’s text and multimodal layers were originally initialised with the weights of BERTBASE (Devlin et al., 2019).
2022), PEVL expresses visual positions in text by appending the bounding box coordinates corresponding to a given (annotated) entity in the caption, surrounded by two special tokens ‘<’ and ‘>’: “A cat < 10 73 206 175 > is napping.” The bounding box coordinates are discretised and added to the text vocabulary. Starting from an ALBEF14M checkpoint, PEVL is pretrained by recovering masked text and position tokens through a generalised MLM objective. The model was trained on a diverse corpus of referring expressions, captions with visual coreferences, question answering, commonsense reasoning, object detection and region descriptions data (Table 2). Unlike ALBEF, PEVL is explicitly trained to learn fine-grained, grounded representations of entities by predicting their coordinates in a unified MLM framework. We evaluate three different models released by the authors, which differ in their pretraining and fine-tuning data: PEVL14M, underwent a second-stage pretraining on multiple supervised tasks (Table 5); PEVLGRD, which was further fine-tuned for position-output tasks such as phrase grounding (Plummer et al., 2015); and PEVLVRD, which was fine-tuned for the position-input task of visual relation detection (Krishna et al., 2017).
X-VLM (Zeng et al., 2022) also aims at learning to locate visual concepts in the image given the associated texts. Similar to the ALBEF architecture, the model consists of an image encoder, a text encoder, and a cross-modal encoder. However, unlike PEVL, X-VLM models visual position through an additional bounding box prediction head: given the visually grounded representation of an object label, the model is trained to regress the object’s bounding box (bbox) coordinates. The authors use both object detection labels and region descriptions to learn multi-grained alignments. The pretraining objective is a linear combination of this bbox loss and the losses defined in ALBEF to align texts and visual concepts (for more details, see Section 5).
In addition to the above models, which we extensively discuss, we also evaluate the following models, based on dual-encoder and frozen LLMs.
CLIP (Radford et al., 2021) is a widely used dual-encoder network. The model consists of two encoders, one for images and one for text, trained to represent both modalities in a joint space via an unsupervised contrastive objectives over more than 400M image–text pairs from the Web. Due
to its simplicity and wide adoption, we report its performance as a strong, representative baseline.
ClipCap (Mokady et al., 2021) is an autoregressive encoder–decoder network. The image encoder is a pretrained CLIP model, while the text decoder is a pretrained GPT-2 (Radford et al., 2019) language model. The authors propose to learn a lightweight Transformer-based network to map CLIP embeddings into a fixed length prefix. The mapping network and the text decoder are finetuned to learn how to generate captions, while the CLIP image encoder is frozen. At inference time, the model generates the caption word after word, starting from the CLIP-based prefix. We report performance for the two released versions—one finetuned on COCO, the other on CC3M—by ranking positive and negative samples on their likelihood.
Flamingo (Alayrac et al., 2022) is a state-ofthe-art VLM capable of tackling a wide range of vision and language tasks from a few input/output examples. To achieve this, the model relies on a pretrained CLIP-like image encoder and a strong pretrained LLM (Hoffmann et al., 2022), both kept frozen. To ingest images and videos, the model learns a small fixed number of visual tokens (Lee et al., 2019; Jaegle et al., 2021). The model is pretrained to generate text from a sequence of text tokens interleaved with images and/or videos.
BLIP-2 (Li et al., 2023) is the most recent, state-of-the-art VLM based on frozen large image
encoders and frozen LLMs (Zhang et al., 2022; Chung et al., 2022). Like CLIPCAP, BLIP-2 learns a mapping network, which in this case is a Transformer model initialised from BERTBASE. The mapping network learns visual query tokens to map the visual representations to the frozen LLM in two stages: a V&L representation stage, and a generative learning stage. The model was pretrained with the same objectives and on the same 129M image– caption data as BLIP. Following the authors’ setup for image–text retrieval and matching, we use the BLIP-2 model after the first-state pretraining.
A.2 Re-implementation Setup
We re-implement ALBEF and X-VLM in JAX (Babuschkin et al., 2020) to ensure full control of modelling, data, and initialisation decisions.15 We note ALBEF’s vision encoder is initialised with a pretrained ViT-B/16 encoder (Touvron et al., 2021) with an input resolution of 256×256 pixels, but X-VLM adopts a more efficient SwinB/32 (Liu et al., 2021) encoder with input resolution of 224×224 pixels. In our re-implementation we initialise both models with a ViT-B/16 with a 224×224 input resolution pretrained on ImageNet-
15To verify our implementation, we compare an ALBEF model trained in our codebase with one trained in the original codebase. Specifically, we pretrain both models on COCO by initialising their visual encoder with a CLIP ViT-B/16 model, and their text encoder with a BERTBASE model. The two models perform similarly on both zero-shot Flickr30K and COCO retrieval tasks with a gap below 1pp Recall@1.

heading: 3 ALBEF14M 500M 88.6 69.4 58.3 32.5 16.2 12.7 90.9 75.9 73.2 54.8

heading: 4 BLIP14M 638M 48.7 67.8 49.7 36.5 18.5 14.5 82.6 78.4 70.4 57.3

heading: 5 PEVL14M♯ 500M 86.2 68.9 57.5 33.2 15.7 12.2 74.9 60.0 45.9 33.2

heading: 6 PEVLGRD♯ 502M 88.5 69.5 57.7 36.2 15.0 12.0 71.8 77.6 42.8 37.7

heading: 9 BLIP129M 638M 51.4 68.8 46.9 35.5 15.0 11.7 90.2 79.5 71.9 58.6
text: 21k (Steiner et al., 2022), to ensure that different initialisation is not responsible for the results.
We pretrain our models on the same 4M and 14M datasets that were originally used by the authors (Table 2), but note that only 1.8M and 11.2M data points were available for CC3M and CC12M, respectively. For object detection data, we use the data points released by the X-VLM authors, and interleave captioning and detection data with a 2:1 ratio following their official implementation. Following (Zeng et al., 2022), we pretrain our models for 200K steps using a batch size of 512 and 1024 samples for ALBEF and X-VLM, respectively. We pretrain once, using the same hyperparameters
as the authors.16 Training our models takes around 1.5 days on Cloud TPUv4 (a 2x2x2 slice). We evaluate our models on both fine-grained benchmarks (SVO-Probes, VALSE and VSR) and on two zero-shot, coarse retrieval tasks (Flickr30K and COCO).

heading: B Results
text: B.1 Results by Subtask Table 6 compares overall performance of our evaluated models (Section 3) with the state-of-theart models in each of four fine-grained benchmarks (Section 2). Results for each subtask are reported in Tables 7 to 11.
In addition to the core discussion in Section 4, we note that FLAMINGO achieves the overall best performance on VALSE; and that the coarsegrained BLIP-2 model performs remarkably well on our range of fine-grained tasks, especially on VALSE, VSR and Winoground. This could be due to a number of factors, such as a larger ViT encoder, the usage of visual queries and the different formulations for the ITC and ITM objectives. We leave a
16The X-VLM authors trained for 200K steps of image captioning data, not counting batches of detection datasets. We count each batch towards the final number of steps, hence effectively training for fewer steps than Zeng et al. (2022).
deeper investigation of large VLMs to future work. Moreover, we also note that CLIPCAP well on VALSE spatial relations and action subtasks, wherein its GPT-2 backbone already performs better than most VLMs. This is further proof of the efficacy of adapting strong LMs for V&L tasks.
B.2 Full Dynamics of Fine-grained Tasks Figures 4 to 7 display pretraining dynamics for our re-implemented ALBEF4M, ALBEF14M, XVLM4M, and X-VLM14M models. For better visualisation, our curves have been smoothed by a 0.6 factor through exponential moving average.
Finally, Figure 8 shows how performance on Winoground evolves when pretraining an ALBEF4M model.17 Looking at overall performance, we see that a model’s score can vary by more than 4pp from one epoch to the next. While longer pretraining seems beneficial, some subtasks, such as Linguistic:Both and Visual:Series, fluctuate considerably; and after 20 epochs, the Image score starts decreasing on other subtasks, such as Linguistic:Object and Visual:Symbolic.
17We note that we used an image resolution of 224×224 pixels, and a batch size of 256 (instead of 512) as we pretrained on a GCP instance with 4× A100 GPUs (instead of the 8× A100 GPUs originally used by the authors).
57
58
59
60
61
62
VSR Test Avg.
ALBEF4M ALBEF14M X-VLM4M X-VLM14M 48 51
54
57
60
63
66 VSR Test Above
75 78 81 84 87 90 93 96 99
VSR Test Across
32
36
40
44
48
52
56
60
VSR Test AcrossFrom
60
64
68
72
76
80
84
88
92 VSR Test AdjacentTo
45 48 51 54 57 60 63 66 69 72
VSR Test Against
48 54 60 66 72 78 84 90 96
102 VSR Test AheadOf
48 54 60 66 72 78 84 90 96
102 VSR Test Along
63 66 69 72 75 78 81 84
VSR Test Alongside
20 30 40 50 60 70 80 90
100 VSR Test Around
36 39 42 45 48 51 54 57 60 63
VSR Test AtTheBackOf
45
48
51
54
57
60
63
66 VSR Test AtTheEdgeOf
46
48
50
52
54
56
VSR Test AtTheLeftSideOf
48
49
50
51
52
53
54
55 VSR Test AtTheRightSideOf
56 60 64 68 72 76 80 84 88
VSR Test AtTheSideOf
60
63
66
69
72
75
78
81 VSR Test AttachedTo
30 32 34 36 38 40 42 44 46
VSR Test AwayFrom
52
54
56
58
60
62 VSR Test Behind
46 48 50 52 54 56 58 60 62
VSR Test Below
60
62
64
66
68
70
VSR Test Beneath
58
60
62
64
66
68
70
72 VSR Test Beside
0
15
30
45
60
75
90
105 VSR Test Between
0 10 20 30 40 50 60 70 80 90
VSR Test Beyond
24 30 36 42 48 54 60 66 72 78
VSR Test By
66
68
70
72
74
76
78
80
VSR Test CloseTo
32 40 48 56 64 72 80 88 96
VSR Test ConnectedTo
45 50 55 60 65 70 75 80 85
VSR Test ConsistsOf
52
54
56
58
60
62
64
66
68 VSR Test Contains
0 6
12 18 24 30 36 42 48
VSR Test DetachedFrom
96
98
100
102
104
VSR Test Down
0
15
30
45
60
75
90
105 VSR Test DownFrom
60 65 70 75 80 85 90 95
100 VSR Test EnclosedBy
48
51
54
57
60
63
66
69 VSR Test Facing
39
42
45
48
51
54
57
VSR Test FacingAwayFrom
42
45
48
51
54
57
VSR Test FarAwayFrom
27
30
33
36
39
42
45 VSR Test FarFrom
36 42 48 54 60 66 72 78 84
VSR Test HasAsAPart
51
54
57
60
63
66
69
72 VSR Test In
54 56 58 60 62 64 66 68 70 72
VSR Test InFrontOf
42
45
48
51
54
57
60 VSR Test InTheMiddleOf
60
63
66
69
72
75
78
81
VSR Test Inside
34 36 38 40 42 44 46 48 50
VSR Test Into
52 54 56 58 60 62 64 66 68
VSR Test LeftOf
56
60
64
68
72
76
80
84
VSR Test Near
45
48
51
54
57
60
63
66 VSR Test NextTo
12 16 20 24 28 32 36 40 44
VSR Test Off
57
60
63
66
69
72
75
78 VSR Test On
58 60 62 64 66 68 70 72 74
VSR Test OnTopOf
40 45 50 55 60 65 70 75 80
VSR Test OppositeTo
0 8
16 24 32 40 48 56 64 72
VSR Test OutOf
45 50 55 60 65 70 75 80 85
VSR Test Outside
51
54
57
60
63
66
69 VSR Test Over
27 30 33 36 39 42 45 48 51
VSR Test ParallelTo
39
42
45
48
51
54
57
60
63 VSR Test PartOf
0 6
12 18 24 30 36 42 48
VSR Test Past
45 48 51 54 57 60 63 66 69
VSR Test PerpendicularTo
54 56 58 60 62 64 66 68 70
VSR Test RightOf
40
44
48
52
56
60
64
68 VSR Test Surrounding
60
61
62
63
64
65
66 VSR Test Touching
45 50 55 60 65 70 75 80 85
VSR Test Toward
10K 60K 110K 160K Update step
64 65 66 67 68 69 70 71 72
VSR Test Under
10K 60K 110K 160K Update step
86
88
90
92
94
96
98
100 VSR Test With
10K 60K 110K 160K Update step
8
16
24
32
40
48
56
64 VSR Test Within
10K 60K 110K 160K 0.0
0.2
0.4
0.6
0.8
1.0
10K 60K 110K 160K 0.0
0.2
0.4
0.6
0.8
1.0
10K 60K 110K 160K 0.0
0.2
0.4
0.6
0.8
1.0
Figure 7: Training dynamics on VSR Random test set subtasks. Random performance is 50%.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
3 A2. Did you discuss any potential risks of your work? 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? 7
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 2
C 3 Did you run computational experiments? 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? A
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: deeper investigation of large VLMs to future work. Moreover, we also note that CLIPCAP well on VALSE spatial relations and action subtasks, wherein its GPT-2 backbone already performs better than most VLMs. This is further proof of the efficacy of adapting strong LMs for V&L tasks. B. 2 Full Dynamics of Fine-grained Tasks Figures 4 to 7 display pretraining dynamics for our re-implemented ALBEF4M, ALBEF14M, XVLM4M, and X-VLM14M models. For better visualisation, our curves have been smoothed by a 0. 6 factor through exponential moving average. Finally, Figure 8 shows how performance on Winoground evolves when pretraining an ALBEF4M model. 17 Looking at overall performance, we see that a model’s score can vary by more than 4pp from one epoch to the next. While longer pretraining seems beneficial, some subtasks, such as Linguistic:Both and Visual:Series, fluctuate considerably; and after 20 epochs, the Image score starts decreasing on other subtasks, such as Linguistic:Object and Visual:Symbolic. 17We note that we used an image resolution of 224×224 pixels, and a batch size of 256 (instead of 512) as we pretrained on a GCP instance with 4× A100 GPUs (instead of the 8× A100 GPUs originally used by the authors). 57
58
59
60
61
62
VSR Test Avg. ALBEF4M ALBEF14M X-VLM4M X-VLM14M 48 51
54
57
60
63
66 VSR Test Above
75 78 81 84 87 90 93 96 99
VSR Test Across
32
36
40
44
48
52
56
60
VSR Test AcrossFrom
60
64
68
72
76
80
84
88
92 VSR Test AdjacentTo
45 48 51 54 57 60 63 66 69 72
VSR Test Against
48 54 60 66 72 78 84 90 96
102 VSR Test AheadOf
48 54 60 66 72 78 84 90 96
102 VSR Test Along
63 66 69 72 75 78 81 84
VSR Test Alongside
20 30 40 50 60 70 80 90
100 VSR Test Around
36 39 42 45 48 51 54 57 60 63
VSR Test AtTheBackOf
45
48
51
54
57
60
63
66 VSR Test AtTheEdgeOf
46
48
50
52
54
56
VSR Test AtTheLeftSideOf
48
49
50
51
52
53
54
55 VSR Test AtTheRightSideOf
56 60 64 68 72 76 80 84 88
VSR Test AtTheSideOf
60
63
66
69
72
75
78
81 VSR Test AttachedTo
30 32 34 36 38 40 42 44 46
VSR Test AwayFrom
52
54
56
58
60
62 VSR Test Behind
46 48 50 52 54 56 58 60 62
VSR Test Below
60
62
64
66
68
70
VSR Test Beneath
58
60
62
64
66
68
70
72 VSR Test Beside
0
15
30
45
60
75
90
105 VSR Test Between
0 10 20 30 40 50 60 70 80 90
VSR Test Beyond
24 30 36 42 48 54 60 66 72 78
VSR Test By
66
68
70
72
74
76
78
80
VSR Test CloseTo
32 40 48 56 64 72 80 88 96
VSR Test ConnectedTo
45 50 55 60 65 70 75 80 85
VSR Test ConsistsOf
52
54
56
58
60
62
64
66
68 VSR Test Contains
0 6
12 18 24 30 36 42 48
VSR Test DetachedFrom
96
98
100
102
104
VSR Test Down
0
15
30
45
60
75
90
105 VSR Test DownFrom
60 65 70 75 80 85 90 95
100 VSR Test EnclosedBy
48
51
54
57
60
63
66
69 VSR Test Facing
39
42
45
48
51
54
57
VSR Test FacingAwayFrom
42
45
48
51
54
57
VSR Test FarAwayFrom
27
30
33
36
39
42
45 VSR Test FarFrom
36 42 48 54 60 66 72 78 84
VSR Test HasAsAPart
51
54
57
60
63
66
69
72 VSR Test In
54 56 58 60 62 64 66 68 70 72
VSR Test InFrontOf
42
45
48
51
54
57
60 VSR Test InTheMiddleOf
60
63
66
69
72
75
78
81
VSR Test Inside
34 36 38 40 42 44 46 48 50
VSR Test Into
52 54 56 58 60 62 64 66 68
VSR Test LeftOf
56
60
64
68
72
76
80
84
VSR Test Near
45
48
51
54
57
60
63
66 VSR Test NextTo
12 16 20 24 28 32 36 40 44
VSR Test Off
57
60
63
66
69
72
75
78 VSR Test On
58 60 62 64 66 68 70 72 74
VSR Test OnTopOf
40 45 50 55 60 65 70 75 80
VSR Test OppositeTo
0 8
16 24 32 40 48 56 64 72
VSR Test OutOf
45 50 55 60 65 70 75 80 85
VSR Test Outside
51
54
57
60
63
66
69 VSR Test Over
27 30 33 36 39 42 45 48 51
VSR Test ParallelTo
39
42
45
48
51
54
57
60
63 VSR Test PartOf
0 6
12 18 24 30 36 42 48
VSR Test Past
45 48 51 54 57 60 63 66 69
VSR Test PerpendicularTo
54 56 58 60 62 64 66 68 70
VSR Test RightOf
40
44
48
52
56
60
64
68 VSR Test Surrounding
60
61
62
63
64
65
66 VSR Test Touching
45 50 55 60 65 70 75 80 85
VSR Test Toward
10K 60K 110K 160K Update step
64 65 66 67 68 69 70 71 72
VSR Test Under
10K 60K 110K 160K Update step
86
88
90
92
94
96
98
100 VSR Test With
10K 60K 110K 160K Update step
8
16
24
32
40
48
56
64 VSR Test Within
10K 60K 110K 160K 0. 0
0. 2
0. 4
0. 6
0. 8
1. 0
10K 60K 110K 160K 0. 0
0. 2
0. 4
0. 6
0. 8
1. 0
10K 60K 110K 160K 0. 0
0. 2
0. 4
0. 6
0. 8
1. 0
Figure 7: Training dynamics on VSR Random test set subtasks. Random performance is 50%. ACL 2023 Responsible NLP Checklist