The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models
Authors: Xinyi Chen
Raquel Fernández
Sandro Pezzelle
Venue: EMNLP 2023 Main (2023)
Keywords: Dataset
Evaluation
Zero-shot
Prompting
Visual Grounding
Language Constructions

Abstract: Despite the impressive performance achieved by pre-trained language-and-vision models in downstream tasks, it remains an open question whether this reflects a proper understanding of image-text interaction. In this work, we explore to what extent they handle basic linguistic constructions---active-passive voice, coordination, and relative clauses---that even preschool children can typically master.
We present BLA, a novel, automatically constructed benchmark to evaluate multimodal models on these Basic Language Abilities. We show that different types of Transformer-based systems, such as CLIP, ViLBERT, and BLIP2, generally struggle with BLA in a zero-shot setting, in line with previous findings. Our experiments, in particular, show that most of the tested models only marginally benefit when fine-tuned or prompted with construction-specific samples. Yet, the generative BLIP2 shows promising trends, especially in an in-context learning setting. This opens the door to using BLA not only as an evaluation benchmark but also to improve models' basic language abilities.

The BLA Benchmark: Investigating Basic Language Abilities of
Pre-Trained Multimodal Models
Xinyi Chen
Raquel Fernández
Sandro Pezzelle ILLC University of Amsterdam s.pezzelle@uva.n. nl
IvI
ILLC
University of Amsterdam chen2@uva.n nl
University of Amsterdam raquel. fernandez@@ua..ll nl
Abstract
Despite impressive performance achieved by pre-trained language-and-visse vision models in downstream tasks, it remains an open question whether this reflects proper understanding of image-text interaction. In this work, ex- plore to what extent they handle basic linguistic constructions -cttns--ettteeeeee aattie-etaaseee paasee--asvveeeeee voice, coordina- tion, and relative clauses that even preschool children can typically master. We present BLA, novel, automatically constructed benchmark to evaluate multimodal models on these Basic Language Abilities. We show that different types of Transformer- based systems, such as CLIP, ViLBERT, and BLIIP2 generally strug- gle with BLA in zero-shot setting, in line with previous findings. Our experiments, in particular, show that most of the tested mod- els only marginally benefit when fine-tuned or prompted with construction-speciine samples. Yet, the generative BLIPP shows promising trends, especially in in-context learning set- ting. This opens the door to using BLA not only as an evaluation benchmark but also to improve models' basic language abilities.
Active-Passise voice
T: the woman feeds the
T: the man fed by the woman.
F: the feeds the the woman fed by
Courdination
I: the man wears a wetsuit and car- surlbourd.
rides red bike. wears and rides red hike. F: the wears red bikini,
surlboard
T: the man who wears polo holds stufled hear.
T: the man who wears striped shirt holds
the man wears gray polo holds cow. holds a stufled bear.
It's
Paceiaa Friday
striped
Figure 1: One example for each of the linguistic con- structions included in the benchmark. Sentences each BLA dataset (both True and False ones) share the template, with 3 slots being filled by many arguments (NPs, predicates, or clauses), that we signal with differ- colors. E.g., in the retive-passive construction, the bits in red and green are the subject and object of the sentence, respectively, while the blue bit the predicate.
1 Introduction
Powered by the Transformer architecture, exten- sive pre-training, and task-specific fine-tuning, re- cent language and vision models (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019; Chen al., 2020; Li et al., 2020; Su al., 2019; Radford et al., 2021) have achieved unprecedented performance many downstream multimodal tasks. Despite the impressive results, it remains an open question whether, and to what extent, this improvement goes hand in hand with genuine understanding of im- age, text, and their interaction. In particular, the ability of models to handle linguistic skills that are essential understanding event or situation has recently been questioned. Hendricks and Ne- matzadeh (2021), ) for example, showed that these models fail in scenarios that require understand- ing verbs and verb arguments; Parcalabescu et al.
(2022) revealed a more generalized struggle of these models with phenomena that require ground- ing relations between objects; Pezzelle (2023) re- ported that their ability to ground language into vision is affected by the presence of semantically underspecified language, e.g., pronouns or loca- tives; Thrush et al. (2022) showed that no mod- els appreciate the (substantial) difference between, e.g., a lightbulb surrounding some plants and some plants surrounding a lighthulb, pointing at flaws

---

the way these models handle compositionality. Arguably, all these previous studies require mod- els do more than plain language comprehen- sion, including mathematical, spatial (Parcalabescu et al., 2022), pragmatic (Pezzelle, 2023), and com- positional reasoning abilitics (Thrush et al., 2022). While mastering these abilities is clearly desirable for any intelligent system, we notice that models may struggle with them due to their pre-training data and objectives. Indeed, these models are typi- cally trained to verify whether a fragment of text is about the content of image-via the Image-Text Matching (ITM) objective- bles the language comprehension tests adminis- tered to children to assess their lexical and gram- matical abilities. To illustrate, in these tests, chil- dren are presented with a sentence, e.g., The red monkey is being scratched by the blue monkey, and asked to either pick the corresponding image from set of alternatives (for an overview of this work, see Schmitt and Miller, 2010) or color the entities in drawing accordingly (Pinto and Zuckerman, 2019). Consistent with their goal, these tests are aimed at excluding, or at least minimizing, the need for reasoning, which has been shown to be sepa- rate from language and to recruit different brain areas (Fedorenko and Varley, 2016).
models' basic language whiitt are cur- rently generally poor. We release the BLA bench- mark and the code to reproduce our results at: https:/ htps:://iiie htt.//ggtttt...ee weeem/shieeeeeee//neee
Related Work
2.1 Basic Language Comprehension Abilities
Language comprehension abilities-in children, but also in adults, e-g., I.2 learners-are are typically assessed in a multimodal setup: the subject is ad- ministered a sentence and some visual content and asked to verify whether the two match. Common paradigms are the Picture Selection Task (PST; Gerken and Shady, 1996). where the subject is given multiple candidate images to choose from, and the more recent Coloring Book (Pinto and Zuckerman, 2019), where the subject is presented with a single black-and-whit drawing and asked to color objects in it according to the content of the sentence. Without any alternatives to choose from, in particular, the latter paradigm was introduced to minimize the recruitment of other executive func- tions connected to reasoning, such as selective at- tention and inhibition, that not relevant the assessment of genuine language comprehension.
closely resem-
In this work, we take inspiration from this line of research and investigate whether, and to what extent, language-and-vision models deal with lan- guage comprehension tasks that require virtually no reasoning abilities. We focus on three basic lan- guage constructions cnsttuuttions-ativ--piieee aative--pssive voice, coordi- nation, and relative clauses-ther even preschool children typically master (Pinto and Zuckerman, 2019; Friedmann and Costa, 2010; Frizelle et al., 2017). We refer to these constructions as Basic Language Abilities (BLA) and propose an automat- ically constructed benchmark (see Figure to as- sess pre-trained language-ande and-vision models, either a zero-shot or in fine-tuning and in-context learn- ing scenarios. We test several types of Transformer- based systems, i.e., CLIP, LXMERT, ViLBERT, BLIP2 and OpenFlamingo, and show that, while human (adult) speakers have no trouble at verify- ing these linguistic constructions, models gener- ally struggle. Yet, the generative BLIP2 model shows promising trends, especially in an in-context learning setting. This reveals that, while BLA is a challenging benchmark, it can be used not only as an evaluation tool but also to improve SotA
Using these and similar paradigms, researchers demonstrated that linguistic constructions such as a sentence's 's active-passive voice, e.g., The red mon- key scratches/is being scratched by the blue mon- key (Pinto and Zuckerman, 2019), various types of coordination, e.g., Grandma smiles and the girl sang (Friedmann and Costa, 2010), and relative clauses, e.g., He saw the girl that picked the flow- ers (Frizelle et al., 2017), are generally mastered by preschool children across various languages, although with some differences due to the stim- uli and, particularly, the experimental paradigm used. These results confirm that the comprehen- sion of these linguistic constructions involves some- what basic language abilities, as also indicated by previous evidence (Horgan, 1978; Diessel and Tomasello, 2001; McKee et al., 1998, inter alia).
In this work, we take inspiration from this line of research and aim testing models for basic lan- guage comprehension abilities that require no or little reasoning skills. Our focused, controlled ap- proach is novel compared to other work evaluating Language-and-yee vision models, that we review below.

---

2.2 Language Abilities of Pre-Trained Multimodal Models
3.1 Linguistic Constructions
BLA includes three types of linguistic construc- tions: active-passive voice, coordination, and rela- tive clauses, which we briefly describe below.
Motivated by the impressive performance of pre- trained multimodal Transformer models, a re- cent line of research investigated whether, and to what extent, this corresponds to a genuine un- derstanding of visually-grounde ing FOIL (Shekhar et al., 2017a), benchmark of minimally wrong image descriptions where all models have proven to fail, some work (Hessel et al., 2021; Parcalabescu et al., 2022) showed that Transformer- based models can almost perfectly distinguish between correct and foil sentences in a zero-shot setting. This indicated that models are good at grounding nouns in vision, likely due to their ITM pre-training objective. Con- sistently, an intrinsic evaluation of the embeddings learned by these models showed that they are bet- ter at representing highly concrete-ratee than abstract- (Pezzelle et al., 2021).
Active-Passive voice (AP) In active voice sen- tences, the agent of the action is the subject of the verb, as 'the monkey scratches the mouse', while in passive voice sentences the form of the verb in- dicates that the subject is the receiver of the action; e.g., 'the mouse is being scratched by the monkey'. Understanding the contrast between active and pas- sive voice implies being able to verify whether two sentences with different syntactic structure may have the same meaning.
language. Us-
Coordination (CO) Coordination, and in partic- ular conjunction, binds together two properties that must hold. We focus on the coordination of verb phrases joined together via the conjunction 'and', e.g.. 'the monkey eats an apple and smiles'. Mas- tering this type of coordination implies being able to verify whether both predicates (" ('eats an apple' and 'smiles") apply to the subject of the sentence.
Leveraging a FOIL-like paradigm, subsequent studies revealed that Transformer-based models struggle when dealing with verb arguments (SVO- Probes; Hendricks and Nematzadeh, 2021), nega- tion (Dobreva and Keller, 2021), numbers, spa- tial relations (VALSE; Parcalabeseu et al., 2022), and expressions requiring compositional abili- ties (WinoGround; Thrush et al., 2022; Diwan et al., 2022) or embedding semantically underspecified language (Pezzelle, 2023). Crucially, all this pre- vious work focused on phenomena and tasks that require more than basic language understanding to be properly mastered. As recently pointed out by Bugliarello al. (2023), indeed, performing well on each of these benchmarks requires models handle different skills, ranging from mathemat- ies to pragmatics and reasoning abilities.
Relative Clause (RC) Relative clauses are em- bedded clauses introduced by a relative pronoun that qualify an entity previously mentioned in the sentence. We focus relative clauses attached to the subject of the sentence and introduced by the pronoun 'who', e.g., 'the monkey who scratches the mouse tall'. Understanding sentences with relative clauses implies identifying the entity qual- ified by the relative clause (e.g., 'the monkey who scratches the mouse') and verifying whether the predicate in the main clause applies (e.g.. 'is tall").
3.2 Benchmark Format
We construct dataset of natural images and cmpplate-bssed sentences for each of the linguistic constructions: active-passive voice (AP), coordi- nation (CO), and relative clause (RC). Building on FOIL (Shekhar et al., 2017b) and FOIL-like paradigms, each datapoint in our benchmark con- sists of an image paired with 4 automatically gen- erated sentences, two correct ones, hence true, and two incorrect ones, hence false. False sentences are identical to truc ones with respect to their format and syntactic structure but contain mistake that makes them semantically incorrect for the image. Concretely, in the false AP sentences the agent and the recipient are reversed; in the false CO sentences one of the conjuncts does not apply to the subject,
Inspired by the work discussed above, we take a novel perspective and assess language-and-ving vidiin models on their genuine lexical and grammat- ical competence. We consider three linguistic constructions constretioos--cctvv--sssiie voice, coordination. and relative blauses-the have been shown to be mastered even by preschool children. In this paper, we refer to them as Basic Language Abilities.
3 The BLA Benchmark
this section, we describe our Basic Language Abilities (BLA) benchmark.

---

and in the false RC sentences the predicate of the main clause does not apply the subject. Figure shows one datapoint per linguistic construction.
datapoints
vocab
sent.
length
GRUEN
AP
613
165
6.33 +1.37
0.86 +0.002
CO
654
827
10.46
1144
0.84 10.003
RC
672
807
10.46 11.1 0.85 10.003
3.3 Dataset Construction
Table Descriptive statistics of the BLA benchmark: number of datapoints, vocabulary size, average sentence length (number of tokens), and average GRUEN score. AP: active-Passive voice. CO: Coordination. RC: Rel- ative Clause.
To generate our datapoints, we use Visual Genome (Krishna et al., 2017), a dataset of nat- ural images densely annotated with objects (bound- ing boxes around entities labeled with a WordNet synset; Miller, 1995), object attributes (such as color) and relationships between the objects in the image (predicates with their arguments).. The con- struction procedure includes the following steps:
small if their size is below a certain ratio between the area of their bounding box and that of the entire image. We use threshold of 0. % for persons and of 0.05% for other objects (such 'bikini' 'stuffed bear' in the examples in Figure 1).
I. Selection of entities and predicates Firstly, we select images in Visual Genome that include least two objects that are persons, which verify using the WordNet synsets. For AP, we select images where the two persons are the arguments of the same transitive verb (one as the agent and the other one as the recipient). We make sure that they belong to two different types (e.g., man and woman) or that they at least have two distinct attributes (e.g., rumning player and sitting player). For CO and RC, we select images where the two persons are the subject arguments of two unique predicates, i.c., two different predicates per person that apply only to that person (e.g.. 'wears wet- suit' and 'carries a surfboard' for the man in Fig- ure 1). Due to multiple annotations present in Vi- sual Genome, checking whether two predicates really different is not trivial. For example, given person may be annotated as being the subject of two predicates, 'wears a 1-shirt' and 'wears a shirt', To capture the fact that these refer to the same rela- tionship and should not be considered as different, we encode the predicates using Sentence-BEi (Reimers and Gurevych, 2019) and consider them independent predicates only if their cosine similar- ity is lower than 0.8. Morcover, for all datasets, we filter out the samples that involve reasoning language by means of handcrafted rules.3
III. Sentence construction Thirdly, we construct true and false sentences using the templates in Ta- ble 5 and Table 6 in the Appendix by randomly fill- ing them in with entities and predicates that meet the constraints above.* Since there may be a multi- tude of suitable entities and predicates per image, at this construction stage each image may end up being paired with more than set of sentences.
IV. Grammar acceptability Finally, we encode each sentence with the GRUEN pre-trained lan- guage model (Zhu and Bhat, 2020) and obtain a score that, following previous work (Parcalabescu et al., 2022), we use as a proxy for grammar ac- ceptability. We discard datapoints where any of the 4 sentences has GRUEN score equal to or lower than 0.7. If, after this filter, image is still paired with more than one set of 4 sentences, we only keep the one with the highest average GRUEN score.
3.4 Descriptive Statistics
In Table 1, we report the dataset size (number of datapoints per linguistic construction), vocabulary size, average sentence length, and average GRUEN scores. The AP dataset is the smallest of the three, the main reason being the limited number of transi- tive verbs (43) with arguments that meet con- struction constraints. The sentences in CO and RC are constructed from the same set of entities and predicates. The slightly lower number of dat- apoints and vocabulary size in RC is due to more sentences with relative clauses being discarded by the grammar acceptability filter.
II. Minimum object size Secondly, filter images where the persons identified the previous step, or the objects that are in a relationship with such persons, are too small. We consider them too
'More details the annotation in Appendix A. *We check whether the predicates in the relation- ships field are part the following list transitive verbs: https://englishooce "See Appendix B for details.
com/ traeertive-verrs/
"We turn all verbs into 3rd person singular forms of the simple present tense using Python's 's Pattern library.

---

3.5 Human Performance
image-text matching (ITM). The pre-trained check- point we use the one released by the VOLTA framework (Bugliarello et al., 2021). Here, we use the pre-trained ITM head to straightly perform the binary classification (true/false) for each <im- age, sentence> pair in our benchmark.
To assess the quality of our automatically con- structed BLA benchmark, we run a human eval- uation on 50 randomly selected datapoints for each linguistic construction dataset, using the Appen platform, $
We create 4 HITss per datapoint, each displaying the image and one of the sentences. This results in a total of 200 HITS' per dataset. The task is to judge whether the sentence is correct or incorrect given the image (therefore, random accuracy on this task is 50%). We ask 3 annotators (co-authors of the paper) to judge all HIT's and compute human accuracy by considering the majority vote. Since human accuracy exceeds 85% on all three datasets, we conclude that, for human speakers, verifying the sentences included in BLA is a trivial task. In contrast, we verified that BLA cannot be solved by (even powerful) text-only language models, which do not fare better than chance in any of its tasks,
LXMERT LXMERT (Tan and Bansal, 2019) is dual- stralm Transformer model that encodes vi- sion and language via two separate streams and combines them via cross-model layers. The model checkpoint we use is pre-trained on the same ex- act data and with the same learning objectives ViLBERT, again from the VOLTA framework. 9 Therefore, the two models are directly comparable.
4.2 Generative Models
BLIP2 BLIP2 (Li et al., 2023) is genera- tive model that uses Querying Transformer (Q- Former) combine the information from a frozen large language model (LLM) and frozen image encoder. The Q-Former contains two submodules one image Transformer and one text Transformer- that share the same self-attention layers. The Q- former is trained in two steps: first, it connects the image Transformer submodule to frozen image encoder to learn multimodal representations via image-text contrastive learning, image-grounded text generation, and image-text matching. Second, performs vision-to- tooon-to-language generation by learn- ing query embeddings that force the underlying LLM to generate text based on the visual informa- tion by the Q-former. We use TXXXL.
4 Vision & Language Transformers
We experiment with 5 pre-trained language-and- vision models: three discriminative models (CLIP, ViLBERT, and LXMERT), and two generative models (BLIP2 and OpenFlamingoo." We briefly describe the models below.
4.1 Discriminative Models
CLIP CLIP (Radford et al., 2021) has two sepa- rate Transformer-based encoders, one for language, and one for vision (here, we use the VIT-B/32 ver- sion). These encoders are jointly trained on 400M <image, caption> pairs gathered from the web. It is trained through contrastive learning for predicting high similarity scores for paired <image, caption> and low scores for unpaired samples. We compute image-text aligmment scores between an image and either its corresponding true or false sentences.
OpenFlamingo OpenFlamingo (Awadalla et al., 2023) an open-source reproduction of the Flamingo models (Alayrae et al., 2022). The model is pre-trained to generate text from a sequence of text tokens interleaved with images. It contains a frozen pre-trained CLIP-like image encoder and frozen pre-trained large language model. The two components are connected via cross-attention layers that allow the language model to attend features produced by the vision model. The mod- els are pre-trained on the LAION-2B (Schuhmann et al., 2022) and Multimodal C4 (Zhu et al., 2023) datasets. In our experiments, we use CLIP ViT- L/14 as the vision encoder and one of the 3B ver-
ViLBERT ViLBERT (Lu et al., 2019) is a dual- stream Transformer that encodes language and vi- sual inputs separately before combining them via co-attention layers. It is pre-trained on the Con- ceptual Captions (Sharma et al., 2018) dataset with two learning objectives: multimodal masked learn- ing (both word and object prediction), as well as
'ht /aappen.//ppp.ccco com/
'Further details about our experiments with GPT-2 (Rad- ford et al., 2019) are provided in Appendix H.
"We also experimented with FROMAGe (Koh et al., 2023) and MAGMA (Eichenberg et al., 2022) but decided not to in- clude them in our study due to their limited ahility to generate yes/no answers. More details are provided in Appendix F.
8 AAaiilable at https://sid. werrdadd//sareeeedrreee aQCx8cLWK7. 9 "Available at https://sid. erdd.edd.ddd blaa.dk/hhae wt//shaaeerrdireed 1p1g160IAS.

---

sions of the underlying language model.
Model / Humans
Task
Metric
5 Exp 1: Zero-Shot Evaluation
AP
CO
RC
ViLBERT
50.57
49.81
49.96
To explore whether, and to what extent, the pre- trained models described in Section 4 can deal with linguistic constructions in BLA without any task- specific fine-tuning, we evaluate them the three datasets in zero-shot setting. For each dataset, we frame the problem binary task: given an <image, sentence> pair, the models are asked to evaluate whether the sentence true or false.
LXMERT
49.31
49.77
50.00
Acc
CLIP
50.08
49.24
49.33
BLIP2
64.15
52.10
52.19
OpenFlamingo
50.73
50.15
49.52
Humans
92.00
90.00
85.00
Pt
ViLBERT
50.43
49.80
49.97
LXMERT
49.49
49.81
50.00
CLIP
50.16
49.24
49.18
ViLBERT and LXMERT can be straightfor- wardly evaluated on the binary task thanks to their pre-trained image-text matching (ITM) classifica- tion head. For CLIP, compute similarity scores between the image and each sentence, and rank the four <image, sentence> pairs according to them. We consider the top-ranked sentences as true and the 2 lower-ranked sentences as false.
BLIP2
66.73
51.88
52.15
OpenFlamingo
50.40
50.11
49.50
Humans
90.00
79.00
79.00
Pr
ViLBERT
50.84
49.82
49.96
LXMERT
48.94
49.71
50.00
CLIP
50.16
49.24
49.18
BLIP2
62.26
52.38
52.25
OpenFlamingo
54.13
50.26
49.53
We evaluate BLIP2 and OpenFlamingo by prompting. The prompt template used with BLIP2 is similar to the proposed by Li al. (2023) for Visual Question Answering: 'Question:
Humans
94.00
95.00
95.00
chance
50.00
50.00
50.00
the sentence [sentence] appropriate for this image? yes or no? Answer:', For OpenFlamingo, following Awadalla et al. (2023), we use the following prompt tem- plate: "<image>Question: Is the sentence [sentence] appropriate for this image? or no? Short Answer:'. We let the models generate a response and consider 'yes' answers as true and 'no' answers as false."
Table 2: Zero-shot model performance and human per- formance on BLA. Acc: Accuracy. Py: Precision true. Py: Precision falsee Scores are reported percentage. The highest results for each metric and task are in bold.
i.e., human accuracy on this linguistic construc- tion. Overall, this pattern of results reveals that, in a zero-shot setting, models struggle with the BLA benchmark, line with previous work inves- tigating other linguistic and reasoning phenomena (Thrush et al., 2022; Parcalabescu et al., 2022).
We use three metrics to measure model perfor- mance: (1) accuracy, measuring how well the mod- els perform on the binary task, (2) precision true, measuring how well models identify the true sen- tences, and (3) precision false, measuring how well the models identify the false sentences.
BLIP2 is the best-performing model The nighest-performin model the BLA benchmark is the generative model BLIP2. It outperforms OpenFlamingo and the best discriminative models with respect to all evaluation metrics (see Table 2). BLIP2 is the only model that consistently sur- passes chance level on all three tasks tasks- though by a small margin in both CO and RC-while the other models perform around or below chance level.
5.1 Results
All models lag far behind human performance Results by all models are reported in Table 2. As can be seen, none of the models performs anywhere close human performance on the BLA bench- mark; indeed, most of them obtain results around chance level. While BLIP2 achieves a remarkably higher accuracy on AP (64%) than on the other two datasets, this result is still very far from 92%,
5.2 Discussion
The overall poor performance obtained in the zero- shot setting indicates that pre-trained multimodal models struggle with the language comprehension abilities evaluated by the BLA benchmark. This could be due to the way in which these models are typically pre-trained, iee. maximizing cross-
10 "'Cocrreelly, as language model use instruction- finetuned model MaAPjjam--NCTTT--Innttrue "ooth models always generated 'yes''na answer.

---

modal alignment, which might not be fine-grained enough to account for the complex dynamies that intertwine language and vision. Performing well on BLA, indeed, requires understanding how en- tities interact with each other, how their attributes combine, and what attributes refer to which entity, Neither discriminative nor generative pre-trained models seem to handle these abilities.
Active-Passise
T: the gentleman kisses the woman.
T: the woman kissed by the gen- tleman.
the woman kisses the gentleman.
F; the genilemun is kissed by the
Coordination
T: the man wears Khaki pants and wears a blue shirt.
T: woman wears white and wears jeans.
At the same time, we notice that BLIP2 per- forms better than the other models, particularly on the active-passive construction. This advantage could result from the more varied data and pre- matching, image-
the mam wears khaki punis wears a white shirt.
F: the woman wears blue shirt and weurs jeuns.
Figure 2:
Two therry-picked samples where all tested models predict that the four sentences are either all true (top) or all false (bottom). revealing an inconsistent behavior. T and F in the image refer to sentences that True or False, respectively, in the BLA benchmark.
training sereetives-image--ee text contrastive learning, and amage-grounded text generation of this model, which would help the model better understand verbs and verb arguments.
5.3 Error Analysis
Exp 2: BLA-Specific Learning
We conduct an error analysis focused on the sam- ples where the models consider all four sentences as either all true or all false. Considering that, in our dataset, each image is systematically paired with two true and two false sentences (see Figure 2), these cases are interesting since they indicate that models fail to make consistent predictions- intuitively, the sentences the man holds the baby and the baby holds the man cannot be true at the same time for a given image. This, in turn, would reveal that the models are unable to correctly iden- tify the entities mentioned in the sentence.
To explore whether the models-which obtain poor performance in the zero-shot setting can learn to deal with the linguistic constructions in BLA via some degree of task-specific learning, we expose them to a relatively little amount of data from each dataset. We use these samples to fine-tune the dis- criminative models and prompt the generative mod- els to allow them to perform in-context learning.
In particular, we experiment with two types of BLA-specific learning, i.e., (1) train and test model with data from the same dataset (SD), and (2) we train a model with data from one dataset and test with data from a different dataset (DD) a cross-task setting. With the former, we evaluate whether the models can learn some key properties of the linguistic construction at hand by having exposure to the same phenomenon in other visual contexts. With the latter, we test whether the mod- els develop general language comprehension abilities that are key all linguistic constructions.
For each model except CLIP, 12 we consider all the cases where all four sentences are assigned the same predicted label, either true or false. For BLIP2, these cases constitute 54.65%, 32.75% and 38.55% of the samples in the AP, CO, and RC constructions, respectively. While these num- bers may already seem quite high, we find out they are even higher in other models. For ViL- BERT, they increase particularly for AP (86.95%), with CO (49.43%) and RC (54.0%) experiencing less dramatic increase. Similar percentages for AP, and a further increase for the other constructions, are observed for OpenFlamingo (88.5%, 68.5%, and 64.5% for AP, CO, and RC, respectively) and LXMERT (87.77%, 58.4, and 60.02%, resp.).
We downsize each dataset to include 400 sam- ples. Then, we randomly split it into training (100 samples), validation (100), and test (200) sets. While each sample in the validation and test sets has the standard format- one image and four cor- responding sentences: two truc sentences and two false ones-in the training set that we use to fine- tune or prompt our models we only keep two sen- tences per image. In particular, we randomly sam- ple one true and one false sentence to keep the supervision of the models relatively little.
These patterns reveal that models are very often inconsistent with their predictions. This suggests they have limited ability to identify the relevant entities, as well as their properties, in the image.
RRReaall that, for CLIP, we use a ranking-bused approach.

---

Charos eall
LOWER7 2eb
Bi3 ren out invaratt
Improvement (AP)
W.
Prodanad
Metric
Model
Jamees
AP
CO
RC
75
72.62
ViLBERT
11.00
0.75
1.00
Accuracy Scores (99)
P
LXMERT
4.50
1.25
-1.00
BLIP2
-2.05
0.25
4.09
ViLBERT
11.00
0.75
1.00
60-
Pj
LXMERT
4.50
-1.69
-0.31
12.0..
BLIP2
18.74
42.34
46.50
55
31.3
st.22
51 6i2
10.00
50
do
50
Table 3: Precision improvement after BLA-specific learning (SD setting) compared the zero-shot results. P. is precision_trus true, Py is loocision_falsse
Active Passive
Coordination
Relative Clause
Figure: Comparison between model accuracies in zero-shot (lighter-color bars) and BLA--peciiiic learning (darker bars). Results are obtained in the SD setting,
image? yes or no? Answer:
In DD, the setup is the same as above, except that the [true] and [false] sentences are from another BLA dataset (different linguistic construc- tion), and yet about the same image (same entities and attributes). While images in CO and RC greatly overlap, images in AP do not overlap much with those in the other two datasets. Therefore, only experiment with CO and RC this setting.
Diseriminative Models
CLIP does have a classification head, we perform fine-tuning on ViL- BERT and LXMERT. We use the training samples to fine-tune their ITM classification head. Further details are provided in Appendix D. In SD, models are fine-tuned, selected, and evaluated with data from the same dataset. In DD, they evaluated of different dataset.
6.1 Results
with data from the
BLA-specific learning generally helps As re- ported in Figure BLA-specifii learning has a generally positive impact on model performance. The generative BLIP2 is shown to improve on all three datasets, which confirms the effectiveness of the in-learning setup in boosting this model's com- prehension abilities. As for discriminative models, ViLBERT experiences the greatest accuracy im- provement compared to the zero-shot setting on AP, and a smaller (though consistent) improvement on the other two tasks; LXMERT, in contrast, does not seem to equally benefit from fine-tuning, except perhaps for a little boost on the AP construction.
Generative Models We perform in-context learn- ing with BLIP2 only. For OpenFlamingo, our pre- liminary investigations revealed that, with our data and task, using the in-context learning setup pro- posed in the original model paper led to nonsense outputs many cases. The same was observed when using the same in-context learning setup that we used for BLIP2, which we describe below.! 13
In BLIP2, we perform in-context learning using samples from the test set of a given dataset. In SD. for each <sentence, image> pair that aim evaluate, we pick one true and one false sen- tence belonging the same datapoint. Compared the sentence under evaluation [target]] these sentences have either a different voice (AP) or de- scribe attributes for different person (CO and RC). In BLIP2, we fill these sentences into a template that we use a prompt for the model. For ex- ample: 'Question: Is the sentence [true] appropriate for this image? yes no? the sentence [false] appropriate for this image? yes the
BLIP2 is the best overall model As shown in Figure 3. BLIP2 again the overall best model across the board. Indeed, it outperforms the best discriminative models by 10.1, 3.2, and 8.6 accu- racy points on AP, CO, and RC, respectively. More- over, it is the model achieving the higher relative improvement in accuracy on all BLA datasets over the zero-shot setting. It is worth mentioning that, looking at the relative improvement precision ob- tained by various models over the zero-shot setting (Table 3), BLIP2 exhibits fairly high improve- ment on all tasks, particularly CO and RC, with respect to precision_fai false. That is, spec-fcc learning particularly helps the model to better spot
Answer: yes. Question:
or no? Answer: no. Question: sentence [target] appropriate for this
Further details about these preliminary experiments and corresponding prompting setups are reported in Appendix G.

---

Improvement on Evaluation Tasks
which are shown to improve their performance over the zero-shot setting. In particular, BLA- specific learning helps the generative BLIP2, which shown to improve not only in the SD setting but also in DD, where examples of other linguistic con- structions are provided. This pattern is encourag ing and suggests that understanding these linguistic constructions may underlie some common basic language abilities dealing with the semantic proper- ties of entities, attributes, and predicates, and their interaction with the image.
Task for Learning
Model
AP
CO
RC
AP
-2.24
0.75
ViLBERT
CO
-0.38
-0.37
RC
0
-0.87
AP
-1.38
-0.38
LXMERT
CO
-0.12
0.5
RC
-0.24
-0.38
CO
11.25
Yet, their performance on the benchmark is still far from human performance, with the best over- all model (BLIP2) lagging 20 accuracy points be- hind human accuracy on AP, the highest-scoring dataset. At the same time, some linguistic structions appear to be more challenging to learn than others, with coordination experiencing much lower improvement compared AP and RC. On the other hand, AP stands out as the construction that can be best learned by the models, possibly due to the fact that it requires models to ground the entities- not necessarily their attributes.
BLIP2
RC
6.94
Table 4: Accuracy improvement after BLA-specific learning (DD setting) compared the zero-shot results
false sentences, which in turn allows it to achieve an overall higher performance on the tasks.
Active-passive voice is the most easily learned Overall, the active-passive voice construction ap- pears to be the one that models can learn to a greater extent. While BLIP2 achieves an accuracy of more than 70%, ViLBERT ranks second with a respectable 61%, which is even more notable result considering the performance around chance level in the zero-shot setting. LXMERT also out- performs the results obtained in zero-shot learning, though by a much smaller margin.
Conclusions
We introduced novel benchmark, BLA, aimed at investigating how well multimodal models under- stand basic linguistic voice, coordination, and relative clauses. We showed that the linguistic constructions BLA are challenging for current language-andd mod- els, which lag well behind human performance. Yet, the recent generative model BLIP2 exhibits better performance than discriminative models. both in the zero-shot and task-specific learning set- ting. We highlight that prompting generative mod- els with examples embedding both the same different linguistic construction promising method to improve their understanding of specific linguistic constructions. This opens the door to using BLA not only to evaluate pre-trained models but also to improve their basic language abilities.
Cross-task learning is only effective for BLIP2 As reported in Table 4, BLIP2 is the only model across the board that benefits from learning about a linguistic construction that is not the one under investigation (our DD setting). As can be seen, the improvement by BLIP2 on RC after having been exposed CO exceeds 11% accuracy compared the zero-shot learning, while the improvement in the other direction (RC to CO) is around 7%. This is not the case, instcad, for the discriminative models, for which the role played by this setup is either insignificant or even detrimental.
6.2 Discussion
Limitations
The results presented above generally show that BLA-specific learning has overall positive role in helping models understand the linguistic con- structions included in the benchmark. This sug- gests that having (even limited) experience with how these linguistic constructions work visually- grounded contexts is beneficial for these models,
The BLA benchmark currently contains three tasks while more be added for more comprehen- sive understanding of basic language ability of mul- timodal models. But our pipeline can be easily adapted to construct more tasks. In the experiment, we only investigated a limited number models, which include (i) generative models (BLIP2 and

---

Openfilamingo), (ii) discriminative models with an image-text classification head (ViLBERT and LXMERT) and (iii) discriminative models with modastmy similarity scores (CLIP), which we believe our selections are representative of current mainstream Vision-and-Lamguagee (V&L) models. Due to the in-context learning constraints for BLIP2, we only investigate its BLA-spccific cross-task learning setup on Coordination and Rel- ative Clause tasks. The two tasks are constructed with the same pipeline and contain similar deserip- tions of human attributes, so more investigation can be done to explore whether the model can improve with learning examples that are more semantically different.
of iision-and-lanuuaal berts. Transactions of the Association for Computational Linguistics, 9:978- 994.
Emanuele Bugliarello, Laurent Sartrain, Aishwarya Agrawal, Lisa Anne Hendricks, and Aida Ne- matzadeh. 2023. Measuring progress in fine- fine-grained dision-and-langue understanding. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Toronto, Canada. Association for Computational Lin- guistics.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In Computer Vision-ECCV 2020: 16th European Conference. Glasgow, UK, Au- gust 23-28, 2020, Proceedings, Part XXX. pages 104- 120.
Acknowledgements
Holger Diessel and Michael Tomasello. 2001. The de- velopment of relative clauses spontaneous child speech.
We are grateful Iacer Calixto for the valuable feedback on a preliminary version of the dataset and experiments. We want to thank Emanuele Bugliarello for his help with the VOLTA frame- work and Hongyi Li for his support with the Open- Flamingo model evaluation. We also thank the members of the Dialogue Modelling Group at the ILLC and the members of the IRLab at Ivl, Uni- versity of Amsterdam, for their insightful feedback on a draft of the paper and the experiments. Xinyi Chen is funded by the project LESSEN with project number NWA. WWA.1389.22. 1389.20.188 of the research program NWA-ORC 2020/21, which is (partly) financed by the Dutch Research Council (NWO). RF is sup- ported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No.
Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath. and Kyle Mahowald. 2022. Why is Winoground hard? Investigating failures in visuolinguistic compo- sitionality. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 2236-2250, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Radina Dobreva and Frank Keller. 2021. Investigating negation in pre-trained vision- and language models. In Proceedings of the Fourth BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, pages 350-362, Punta Cana, Dominican Repub- lic. Association for Computational Linguistics.
Constantin Eichenberg, Sidney Black, Samuel Wein- bach, Letitia Parcalabescu, and Anette Frank. 2022. augmentation of generative models through adapter-based finetuning. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 1416-24284
819455).
References
Evelina Fedorenko and Rosemary A. Varley. 2016. Lan- guage and thought are not the same thing: evidence from neuroimaging and neurological patients. An- nals of the New York Academy of Sciences, 1369.
Jean- Caan-Baptiste Alayrac. Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo; a visual language model for sho- learning, Advances in Neural Information Processing Systems, 155223166-23736
Naama Friedmann and João Costa. 2010. The child heard a coordinated sentence and wondered: On chil- dren's 's difficulty in understanding coordination and relative clauses with crossing dependencies. Lin- gua, 1 502-1515. Contrast as an information- structural notion in grammar.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hes- sel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023. Openflamingo: An open-source framework for training large autoregressive vasion-language models. arXiv preprint arx0v:2308.013900
Pauline Frizelle, Clodagh O'Neill, and Dorothy VM Bishop. 2017. Assessing understanding of relative clauses: A comparison of multiple-choice compre- hension versus sentence repetition. Journal of Child Language, 44(6): 1435-1457.
Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. 2021. Multimodal pretraining unmasked: A meta-analysis and a unified framework

---

LouAnn Gerken and Michele E Shady. 1996. The pic- ture selection task.
Sandro Pezzelle. 2023. Dealing with semantic under- specification in multimodal NLP. To appear in the Proceedings of ACI. 2023.
Lisa Anne Hendricks and Aida Nematzadeh. 2021.
Probing image-language transformers for verb un- derstanding- Findings of the Association for Com- putational Linguisties: ACL-IJCNLP 2021, pages
Sandro Pezzelle, Ece Takmaz, and Raquel Fernández. 2021. Word representation learning in multimodal pre-trained transformers: An intrinsic evaluation. Transactions of the Association for Computational Linguistics, 9:1563-1579
3635-3644.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7514-7528.
Manuela Pinto and Shalom Zuckerman. 2019. Coloring book: A new method for testing language comprehen- sion. Behavior research methods, 5 (6)):6699-263)
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Dianne Horgan. 1978. The development of the full passive. Journal of Child Language, 5(1):65-80) 80.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023. Grounding language models to images for multimodal inputs and outputs. ICML.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):9
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International jourraaaaa of computer vision, 123(1):32-
Nils Reimers and Tryna Gurevych, 2019. Sentence-bert, Sentence embeddings using siamese nett-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
73.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020. Unicoder-vi: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 11344,
Cristina Schmitt and Karen Miller. 2010. Using com- prehension methods in language acquisition research. Experimental methods in language acquisition re- search, pages 35-56.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprint arXiv:2301.122 125997
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image- text models. Advances in Neural Information Pro- cessing Systems, 35:25278-2525 252944.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint arXiv: prxXiv1908.000 1908.03577
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2556-2565.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for ladnn-ae tasks. Ad- vances in neural information processing systems, 32.
Cecile McKee, Dana McDaniel, and Jesse Snedeker. 1998. Relatives children say. Journal of Psycholin- guistic research, 27(5).
Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Au- relie Herbelot, Moin Nabi, Enver Sangineto, and Raf- faella Bernardi. 2017a. Foil it! find one mismatch between image and language caption. In ACL, pages 255 255- Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11)399-41.
Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Au- rélie Herbelot, Moin Nabi, Enver Sangineto, and Raf- faella Bernardi. 2017b. FOIL it! find one mismatch between image and language caption. In Proceed- ings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 255 265, Vancouver, Canada. Association for Computational Linguistics.
Letitia Parcalabescu, Michele Cafagna, Lilitta Murad- jan, Anette Frank, lacer Calixto, and Albert Gatt. 2022. Valse: A -ndeinddeennent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8253-8280.

---

Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. Vi-bert: Pre-training of generie visual-linguies representations. In Inter- national Conference on Learning Representations.
Voice
Template
TA
Active
the subject preticatetactive] the object
TP
Passive
the object is/are predicatepaassvee by the subject
FA
Active
the object predicate(active the subject
FP
Passive
the subject is/are predicatel by the object
Hao Tan and Mohit Bansal. 2019, Lxmert: Learning mross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 5100-5111.
Table 5: Template for caption construction in Active- Passive Voices (AP) dataset. TA. True Active. TP. True Passive. FA. False Active. FP. False Passive. Examples: TA: the man holds the woman. TP: the woman is held by the man, FA: the woman holds the man. FP: the man is held by the woman.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for visio-linguistic compositionality, In Proceedings of the IEEFSCVF Conference on Com- puter Vision and Pattern Recognition, pages 5238- 5248.
Person
Template
TP
[Glgg and 02u, /[who dilgs Gaui
TP:
Pa
the pla [Oly, and may, /[who Maps Odwy]
FP)
Pt
the pi [Alps und Gays /0lgg 14 who Aaps Daay / Ripal
PP
the P2 [Cty) and 01yy! ogyy /[who 0gg Olay! (aggg P2
Table 6: Template for caption construction Coordination (CO) dataset. TP1. True Person, TP2- True Persong. FP,. False Person, FP2. False Person,. a.Pj- Attribute; Person, EExamples: TP1: the man [wears white shirt and holds controller] /[who wears a white shirt holds controller]. TPs: the woman [wears blue shirt and sits sits on a sofa]. FP1: the man [wears a white shirt and sits on a sofa] /[ who wears a white shirt sits on a sofa]. FP2:the woman [holds controller and wears blue shirt] /[who holds controller wears a blue shirt].
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. 2023. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint urXiv:2304.069933
sofa] /[who wears a blue shirt
Wanzheng Zhu and Suma Bhat. 2020. Gruen for evaluat- ing linguistic quality of generated text. In Findings of the Association for Compulational Linguistics, ACL 2020: EMNLP 2020, pages 94-108. Association for Computational Linguistics (ACL).
language. We also added more phrases and words to the list by checking the annotations that contain prepositions. In addition, we try to avoid numer- ical descriptions in our extracted information by filtering annotations that contain numbers from one to ten. This was an effective rule based on our observation on the VG annotations.
A Visual Genome
Visual Genome (VG) is a dataset and a knowl- edge base that annotates 108K images with each image having an average of 35 objects, 26 at- tributes, and 21 pairwise relationships between ob- jects. It contains seven main components-ree descriptions, objects, attributes, relationships, re- gion gruphs, scene graphs, and question-answer together with information for image. We only used objects, attributes and relationships for our dataset construction and extracted the image size information from image. One example of the VG dataset and its corresponding annotations can be found in Figure 4.
C Sentence Generation Templates
We use sentence construction templates (Table 5 and Table 6) and the extracted information de- scribed in Section 3 to generate a set of four sen- tences (two true, two false) for given image.
D Model Finetuning Details
We fine-tune ViLBERT and LXMERT pretrained model on their entire model layers with the Image- Text Matching learning objective only. We mod- ify the pretraining code from the VOLTA frame- work (available at https: thub...tttttu.... com/e-bug/ vol ta hta/blbbmaainn/ http///aannittriiniiama/a traii_ronnaap.....// py) and adapt the hyperparameter settings for finetuning provided by the code owners. We set the learning rate to le- 5, the training batch size to 16 and the maximum training epoch to 10 after hyperparameter search-
B Avoiding Reasoning in BLA Construction
By analyzing the VG annotations that contain at least two human entities, we found the most com- mon annotations related to reasoning language is the positional description (e.g. 'on the right', 'be- hind"). We came up with a list of commonly-used positional words like 'right', 'above' and adjusted the keyword search rules to better filter reasoning

---

manuno1 wA
PeYSORAL
passanger n who spenting
the dergekcalinn FSs
diyi BY
DC SURRO Sieet
Firmily Trom Goolaadd
dyyeescale
diver redr bolvard with
What the shamrocks doing Shere?
horsB.n. D1
sariaage
symbol Sl. Patrick's
carringa 0Z
einu coli oelooin
ardtwyy mantar
liy idteeren th lis suth offecrdd
spedium
apalitanoe
Logend:
oéject
attibute
rebllonship
Imapoed symser.
derived synsel
QA palt
extracted NP
derived hyyorry of
Figure 4: One example image from Visual Genome dataset with its region descriptions, QA, objects, attributes, and relationships canonicalized by Krishna et al. (2017). One example annotations for relationships is <predicate: pulls, subject: horse, object: carriage, ...>, for attributes <carriage, green>, for object is <object_id, id, width: ... length:
ing. The model checkpoint that achieves the high- est accuracy performance on the validation set will be used for evaluation. We also experiment with only fine-tuning specific layers of the models but find out that fine-tuning the whole model achieves the best performance.
F MAGMA and FROMAGe
In the zero-shot evaluation for generative mod- els, we employ prompting to guide the models to generate constrained outputs, specifically "yes' or 'no' responses, perform the binary classifi- cation tasks BLA. We the BLIP2 prompt template and apply minor changes to it, e-g., we replace 'Question' with 'Q' and 'Answer' with following the prompting examples reported the papers of the models. Our investigation revealed that, in over 20% of the cases, MAGMA and FRO- MAGe fail to generate the desired outputs. In- deed, many answers instead provided explanations for why a sentence is correct incorrect. Since evaluating their performance would require more careful (including manual) analysis, our zero- shot experiments we focused on BLIP2 and Open- Flamingo, which exhibited higher ability to adhere to the task instructions. We share the code used to preliminary test MAGMA and FROMAGe at https://gii http:////gtuubb.co..... com/ wtt..///eiiieeeeecee//BLL.
E Human Annotation Details
We collect human annotations with Appen's inter- nal channel option. The question interface and test question setups are shown in Figure
The annotations authors of this paper, including two linguistic perts and one dataset constructor. The annotators are requested to make their judgment only based on the visual and text information provided in the questions.
collected from three
PACE:
G OpenFlamingo In-context Learning
After conducting preliminary in-context learning experiments, we observed that OpenFlamingo struggled to generate constrained 'yes' or 'no' answers. In particular, two templates. The first template, similar the BLIP2 prompt template, uses only one image in- put for both the examples and the question, as fol- lows: [<image>Question: Is the sentence [true] appropriate for this image? yes
SENTECE:
experimented with
Figure 5: Example of the Appen question interface. The golden label of this question is "No".

---

no? Short Answer: yes. Question:
Is the sentence [false] appropriate for this image? yes or no? Short :nswer:
no. Question: Is the sentence [target] appropriate for this image? yes or no? Short Answer:]. This prompt template led to nonsense outputs in most of the cases.
The second template we used closely follows the provided in the original model paper Awadalla al. (2023), which requires the to- ken <image> for image input before each ques- tion, as follows: [<image>Question:
the sentence [true] appropriate for this image? yes or no? Short Answer: yes. /llnnoofccunk/>>iinnnenn soinmuuuuniiiimeeeuussstioio the sentence [false] appropriate for this image? yes or no? Short Answer: no. < sentence [target] appropriate for this image? yes or no? Short Answer: Note that the three <image> tokens always refer to the same image. With this template, the model gener- ated more constrained answers, but only for about 40% of the cases, which is still unsatisfactory. We hypothesize this could be due, at least in part, to the properties of our questions, that longer and more complex than the examples provided in the model paper. This could harm OpenFlamingo's ability to follow instructions.
Is
Is the
Language-Only Model on BLA Tasks
As sanity check, we test whether the BLA tasks be solved by powerful text-only model, namely, GPT2 (Radford al., 2019). We calcu- late the perplexity scores for the four sentences in each datapoint and rank them such that the lower the perplexity, the higher the ranking. Similarly to our experiments with CLIP, we consider two top- ranked sentences true and the other two as false. As expected, GPT-2 performs around chance level in all tasks: 50.08% for Active-Pase Sative-Passive, 50.47% for Coordination, and 99.96% for Relative Clause.


Reviews:
[{"review_id": "NvhJdqvciR", "reviewer": "Reviewer_3JNU", "summary": "", "questions": "- The paper shows that BLIP-2 outperforms discriminative models. It would be interesting to examine the extent to which this is due to training objectives (discriminative vs. generative), or other factors such as training data and model architecture. It would be valuable to run and include other generative model baselines, as BLIP-2 is the only generative model currently presented in the paper. For example, [OpenFlamingo](https://github.com/mlfoundations/open_flamingo), [MAGMA](https://github.com/Aleph-Alpha/magma) and [Fromage](https://github.com/kohjingyu/fromage) are some vision-and-language generative models that would be valuable to include in the evaluations.\n- How do larger text-only LMs perform on this task? It would be interesting to see a text-only baseline where a model does not have access to the image. This would investigate whether it is possible to do well on BLA by simply exploiting language structure. For example, it may be interesting to see if much larger LMs (LLaMA-70B, or GPT-3) can outperform vision-and-language models.\n- In Table 3, why do you think it is the case that in-context learning with BLIP-2 on BLA examples improves P_f so much, but P_t marginally? This seems to suggest that the dataset may be imbalanced or biased towards P_f somehow?\n- In Table 1, does the sentence length refer to tokens or characters?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and comments. We address your points below:\n\n__Insufficient baselines__:  The main goal of the paper is to propose a novel dataset for evaluating the basic language abilities of current language-and-vision models and to preliminarily assess how various models representative of current ‘families’ of architectures perform on it. Yet we agree with you that testing more generative models would help to clarify whether the difference in the performance of discriminative vs. generative models we have observed generalises beyond BLIP2. To this end, we have tested Fromage (fromage_vis4 version, which uses OPT-6.7B as base LM) using the same prompts as for BLIP2. The pattern of results is very similar: like BLIP2, Fromage yields results around chance level in the zero-shot setting and improves over the discriminative models in the in-context learning setting (the in-context learning accuracy for Formage is 52.69%, 58.75% and 62.5% on AP, CO and RC, respectively). We are currently running experiments with OpenFlamingo and MAGMA (we were not able to obtain results for them yet due to time constraints). We will be happy to include the results with all these additional models to the camera-ready version of the paper. However, a systematic exploration of all available discriminative and generative models, including ablations on their pretraining objectives, parameter size, or training data, is beyond the scope of the current paper, and we leave it for future work.\n\n__Missing analysis on failure cases__: We agree that an analysis of cases where the models fail (and succeed) is interesting and can shed further light on the strengths and weaknesses of the models. Indeed, we have already carried out a preliminary qualitative analysis of a subset of each dataset, which shows interesting patterns with respect to both the type of errors that the models make (e.g., some models are shown to fail in relatively easy samples, while succeeding in more challenging samples) and the commonalities and differences between various architectures. Taking advantage of the extra page, we will include this analysis, as well as a quantitative analysis of the errors shared by various models, in the revised version of the paper. Thanks for suggesting this.\n\n__Q1__: See response to “Insufficient baselines” point above.\n\n__Q2__: Thanks for raising this point, which gives us the opportunity to further clarify why our benchmark can only be solved by a model that genuinely understands a text, a scene, and their interaction. Each sample in BLA consists of four semantically and grammatically valid sentences – we used the GRUEN language model to filter out ungrammatical/unlikely cases. Out of these linguistically valid sentences, only two apply to the target image. As such, there is no reason to expect that a language model, even if large and massively trained, should be able to perform better than chance without having access to the content of the image. We empirically tested this using GPT-2. For each sample, we obtained the model perplexity, ranked the four sentences based on it (the lower the perplexity, the higher the ranking), and computed model accuracy (see lines 409-413 in the paper about CLIP). As expected, GPT-2 was shown to perform around the chance level in all tasks: 50.08% for Active-Passive, 50.38% for Coordination, and 49.94% for Relative Clause. We will include the results of this sanity check in the updated paper.\n\n__Q3__: First, the dataset has the same number of true and false captions. False captions contain the exact same words as the correct captions while just using different word orders or switching phrases between the two sentences. Therefore, the dataset does not include any (trivial) bias toward P_f. Second, when performing in-context learning, we randomly shuffle the order for true and false examples in the context. Thus, the reported difference between P_f and P_t likely does not depend on how we use the data.\n\nUpon close inspection of the results, we conjecture that this pattern could be due to the different extent to which the model outputs a “yes” or “no” answer, e.g., 724 vs. 76 times in the RC task, respectively. In this task, the recall rate for false cases is 99.5%, while the recall for true cases is as low as 31.09%. Still, the reason why BLIP-2 outputs more positive than negative answers after in-context learning is not trivial, and surely deserves further investigation. We will add a discussion on this point in the updated version of the paper.\n\n__Q4__: In Table 1, sentence length refers to the number of tokens. We will make this clear in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The BLA dataset proposed in this paper seems like it will be a useful benchmark for evaluating the basic linguistic and language abilities of vision-and-language models. It appears to be a relatively difficult task, and existing SOTA vision-and-language models are significantly below human performance (and perform close to random chance).\n- The paper conducts analysis on several SOTA vision-and-language models, and highlights possible directions that may be useful to explore for training better vision-and-language models (e.g., model architectures that can be prompted with in-context examples).", "reasons_to_reject": "- **Insufficient baselines:** There is an interesting point highlighted in the paper that is not explored in enough detail. It suggests that generative models may be better than discriminative models on this task, perhaps due to their pretraining objective. However, the only generative model benchmarked in the paper is BLIP-2, which does not give us a good basis of comparison for this hypothesis. The improved results of BLIP-2 may also be due to the size of its language encoder: BLIP-2 has a much larger LM (11B) than the other models used in the paper. It will be useful to benchmark other generative vision-and-language models to ablate whether it is the size of the language encoder, pretraining objectives, or something else, that contributes to good performance on BLA.\n- **Missing analysis on failure cases:** The paper would be significantly stronger if there was some analysis and visualization of the failure cases of the models. What kind of examples do the models fail on? Is this because of visual or language capabilities? A more comprehensive analysis, similar to the one in Diwan et al. (2022) would significantly strengthen the paper. In the current version of the paper, there is insufficient analysis on why the models fail exactly, and what kind of examples they tend to fail on.", "questions_to_authors": "- The paper shows that BLIP-2 outperforms discriminative models. It would be interesting to examine the extent to which this is due to training objectives (discriminative vs. generative), or other factors such as training data and model architecture. It would be valuable to run and include other generative model baselines, as BLIP-2 is the only generative model currently presented in the paper. For example, [OpenFlamingo](https://github.com/mlfoundations/open_flamingo), [MAGMA](https://github.com/Aleph-Alpha/magma) and [Fromage](https://github.com/kohjingyu/fromage) are some vision-and-language generative models that would be valuable to include in the evaluations.\n- How do larger text-only LMs perform on this task? It would be interesting to see a text-only baseline where a model does not have access to the image. This would investigate whether it is possible to do well on BLA by simply exploiting language structure. For example, it may be interesting to see if much larger LMs (LLaMA-70B, or GPT-3) can outperform vision-and-language models.\n- In Table 3, why do you think it is the case that in-context learning with BLIP-2 on BLA examples improves P_f so much, but P_t marginally? This seems to suggest that the dataset may be imbalanced or biased towards P_f somehow?\n- In Table 1, does the sentence length refer to tokens or characters?", "paper_topic_and_main_contributions": "This paper proposes the Basic Language Abilities (BLA) benchmark, a new dataset for measuring the language abilities of vision-and-language models. The dataset is split into three types of linguistic constructions: active-passive voice, coordination, and relative clauses. A corpus of natural images and template-based sentences for each of these is constructed, with each image being paired with two correct sentences and two incorrect ones. Several vision-and-language models are benchmarked on it, and they show that existing models are significantly worse than human performance, highlighting possible areas for improvement in vision-and-language research.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No major missing references that I am aware of.", "Typos_Grammar_Style_and_Presentation_Improvements": "- The paper is generally well structured and easy to follow.\n- In Appendix E, it would be helpful to list the compensation provided to the human annotators."}, "extra_text": {}}, {"review_id": "QUIbM5jMHk", "reviewer": "Reviewer_pLMK", "summary": "", "questions": "How do you pronounce \"BLA\"? Bee El A, or BLAH? 😅", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and the positive comments! We pronounce the name of the dataset as ‘BLAH’, as it aims to test whether models comprehend or rather process language as merely blah-blah :)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The contributions of this paper are substantial:\n\n* The paper covers a good range of VLMs (both discriminative and generative), BLIP2 is very recent and CLIP is older but powerful and popular. I highly appreciate the testing of BLIP2, a VLM with frozen linguistic representations, since I wondered whether a VLM based on a frozen language model can capture the linguistic aspects of this benchmark better. From the results, indeed it can.\n\n* The study is well motivated and timely and I hope to see the community test even more VLMs.\n\n* The exp 2 setting (in section 6) is extremely interesting and exhaustive (whether learning one task transfers to the other).\n\n* The benchmark is automatically created, opening the potential for gathering more data if needed.", "reasons_to_reject": "I don't mean to say the study is flawless, only that I cannot find flaws.", "questions_to_authors": "How do you pronounce \"BLA\"? Bee El A, or BLAH? 😅", "paper_topic_and_main_contributions": "This paper tests VLMs (vision and language models) for their proficiency in linguistic constructions such as active-passive voice, coordination, and relative clauses in a new benchmark called BLA (Basic Language Abilities). The aim of this is to have a test for VL models that does not need reasoning for solving, since prior work has found that reasoning is hard enough for VL models. So the question arises of whether it is reasoning or even more basic things like linguistic understanding that they lack. The authors test 3 VL transformers in zero-shot, fine-tuning and in-context learning settings (exhaustive and complete experiments).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Excellent writing."}, "extra_text": {}}, {"review_id": "h7O4FaxP1J", "reviewer": "Reviewer_5kcq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and the positive comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- new benchmark to assess linguistic capabilities of VL models\n- interesting insights provided to the community", "reasons_to_reject": "I do not identify any relevant weakness that is not already discussed in the limitation section.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work presents a VL benchmark testing three basic linguistic constructions, namely active-passive voice, coordination, and relative clauses. The authors test in zero-shot and fine-tune settings 3 VL models on their benchmark. They find that although these are mastered by humans at an early age, VL models struggle with these tasks. Moreover, they find that the only generative models tested,  outperform the discriminative VL models both in zero-shot and fine-tuning settings, yet lagging behind human performance.\n\nThe paper is overall well-written and pleasant to read. The authors tackle an important and timely issue, which is assessing the capabilities of VL models to ground linguistic expressions which are easily mastered by humans.  This involves engaging in an evaluation that is not task-based, but linguistically motivated. The tasks chosen are fairly motivated and the benchmark construction seems solid. The main findings are fairly discussed and provide interesting insight into the grounding capabilities of pre-trained VL models, for instance, a very interesting finding is that all the models struggle with coordination even after exposure to the data. However the work focuses only on three models and three tasks, but this is discussed in the motivations and seems a reasonable improvement to be added in future work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers felt very positively about this paper, viewing the proposed benchmark (evaluating VLMs on particular linguistic constructions) as useful and challenging, with interesting and insightful analysis. Once concern raised by a few reviewers is the relatively limited number of pretrained models evaluated on this dataset; however the author response adequately addressed this by including a few more (generative) models. Given that the paper is already very clearly written, it should be easy for the authors to present these extra results and the promised additional qualitative analysis in the extra page in any camera-ready version to make the paper even stronger.
Decision: Accept-Main