What's "up" with vision-language models? Investigating their struggle with spatial reasoning
Authors: Amita Kamath
Jack Hessel
Kai-Wei Chang
Venue: EMNLP 2023 Main (2023)
Keywords: vision-language
spatial relations
interpretability

Abstract: Recent vision-language (VL) models are powerful, but can they reliably distinguish "right" from "left"? We curate three new corpora to quantify model comprehension of such basic spatial relations. These tests isolate spatial reasoning more precisely than existing datasets like VQAv2, e.g., our What'sUp benchmark contains sets of photographs varying only the spatial relations of objects, keeping their identity fixed (see Figure 1: models must comprehend not only the usual case of a dog under a table, but also, the same dog on top of the same table). We evaluate 18 VL models, finding that all perform poorly, e.g., BLIP finetuned on VQAv2, which nears human parity on VQAv2, achieves 56% accuracy on our benchmarks vs. humans at 99%. We conclude by studying causes of this surprising behavior, finding: 1) that popular vision-language pretraining corpora like LAION-2B contain little reliable data for learning spatial relationships; and 2) that basic modeling interventions like up-weighting preposition-containing instances or fine-tuning on our corpora are not sufficient to address the challenges our benchmarks pose. We are hopeful that these corpora will facilitate further research, and we release our data and code at https://github.com/amitakamath/whatsup_vlms.

What's "up" with vision-language models? Investigating their struggle with spatial reasoning
Amita Kamath
Jack Hessel
Kai-Wei Chang'
University of California, Los Angeles
Allen Institute for Al
(kamatha, swchang)ecs.c udla.edu, jackheallenai org
Abstract
Recent laiounlanguuae (VL) models are pow- erful, but can they reliably distinguish "right" from "left"? We curate three new corpora to quantify model comprehension of such basic spatial relations. These tests isolate spatial rea- soning more precisely than existing datasets like VQAv2, e.g., What'sUp benchmark contains sets of photographs varying only the spatial relations objects, keeping their iden- tity fixed (see Figure I: models must compre- hend not only the usual case of a dog under a table, but also, the same dog on top of the same table). We evaluate 18 VI. models, find- ing that all perform poorly, e.g., BLIP fine- tuned VQAv2, which nears human parity on VQAv2, achieves 56% accuracy on our benchmarks vs. hurmans at 99%. We conclude by studying causes of this surprising behavior, finding: 1) that popular vision-language pre- training corpora like LAION-25 contain little reliable data for learning spatial relationships; and 2) that basic modeling interventions like up-weighting preposition neepsitiion-contaiinng instances or fine-tuning on our corpora are not sufficient to address the challenges our benchmarks pose. We hopeful that these corpora will facilitate further research, and we release our data and code at https: //gi::///ii thub. com/an coi takamath/ whatsup_vie mlmm..
A dog on table
A dog right of a table
A dog left of a table
A dog right of a table
A dog right table
A dog right of table
A dog under
dog right of table
table
propose three tightly controlled bench- marks to assess model capacity for fine-grained spatial reasoning, showing that popular vision-language mod- els fall far behind human performance when asked select the correct spatial relation between two objects in an image (real examples shown).
Figure I:
1 Introduction
Pre-trained vision-language models perform well on complex tasks such as VQAv2 (Goyal et al., 2016) and Nocaps (Agrawal et al., 2019), even in the zero-shot setting (Li et al., 2023). How- ever, recent work has re-surfaced a concern that has long plagued vision-language models (Yatskar et al., 2016; Johnson et al., 2017): new multimodal models still exhibit poor behavior on simple tasks like attribute attachment, counting, etc. (Yamada et al., 2022; Thrush et al., 2022; Yuksekgonul et al., 2023; Parcalabescu et al., 2021). Despite improve- ments, models still fail to reliably capture even
basic spatial factors of images, a prerequisite for more precise and complex reasoning benchmarks.
But why? In this work, we study vision-language models' performance on basic spatial relations, such as "left of" and "right of". Existing bench- marks which aim to operationalize spatial under- standing such VQAv2 and GQA (Hudson and Manning, 2019) often conflate the evaluation of spatial reasoning with other types of reasoning. such as in the GQA question "Is there a woman to the left of the person that is wearing a wetsuit?".

---

Hence, we first curate COCO-spatial and -QA-spatial based on the COCO (Lin et al., 2014) and GQA datasets respectively, to isolate and assess more strictly only basic spatial relations. In addition, we collect a third evaluation corpus, What'sUp, with even tighter controls. The images within COCO and GQA often contain many ob- jects/relations, and exhibit biases that reflect our usual world (e.g., mug is usually on table, not under it). We manually capture controlled pho- tographs of household objects in various positions: c.g., to overcome the social bias of dogs being pho- tographed under tables, we (carefully, gently, and with many treats) placed a dog on a table and took a picture of her (see Figure 1). What'sUp consists of 205 sets of four images each, resulting in 820 images in total. Each set of images varies the un- derlying preposition that describes the relationship between two objects, e.g., one set of images con- tains a mug on, under, left of, and right of a table. Furthermore, background objects are minimized, so there is no ambiguity.
more frequent in the training data, "in the back- ground", as a case study to investigate if models may indeed "understand" spatial relationships (but are not surfacing that knowledge due to distribution mismatches); and (3) finetuning on several different relevant training sets (c.g., COCO-spatia/GQQA- spatial training sets. preposition-containie sub- sets of LAION-2B, and auto auto-generrted hard neg- atives with switched prepositions). None of these approaches dramatically improves model perfor- mance on understanding spatial relations.
In summary, our contributions are: (1) three new benchmarks evaluating spatial relations in vision-language models, alongside results of 18 VL models on them; (2) study of the training data of some of these models, with observations that could explain poor model performance on the benchmarks; and (3) study of various methods to improve model performance, with insights that could guide future research in overcoming this issue. We release code and data to encourage the same at https: thub. com/ ami takama th/ whatsup_vllm
For all three datasets, our setup is as follows: for given image, the model is given correct caption and I or 3 distractor captions, which differ only by preposition: must select the correct one. We evaluate popular vision-language models, covering various architectures (e.g., one-stack vs. two-stack), training objectives (e.g., generative vs. contrastive models), and training data. All models perform poorly across benchmarks, with many per- forming just few points above random chance and all models falling far behind human performance.
Benchmarks
Existing benchmarks include spatial reasoning questions, such as VQAv2 (Goyal et al., 2016) and GQA (Hudson and Manning, 2019). However, in- stances in these corpora often conflate several types of reasoning: in GQA, over 92% of the validation questions do so. For example, the GQA question "Are there men to the left of the person that is hold- ing the umbrella?" conflates evaluation of spatial reasoning, object relationships, and object detec- tion in contrast, our questions require only spatial reasoning about one or two objects.
Next, we investigate why these models fail to learn much about spatial relationships. All models we consider are pre-trained on large-scale image- caption corpora. We perform a corpus study of the LAION-2B dataset (Schuhmann et al., 2022), which was used to train OpenCLIP (Ilharco et al., 2021). We see that (1) common spatial prepositions occur in less than 0.2% of the training data; (2) when they do occur, they can be ambiguous or extraneous to the image, e.g.. "left" defined from the viewer's perspective vs the subject's; and (3) they can often be guessed without looking at the image, e.g., "a house above water".
Our three new evaluation corpora are presented in the same format: an image paired with sev- eral captions which differ only by a preposi- tion. What'sUp consists of tightly controlled pho- tographs we captured ourselves, whereas COCO- spatial and GQA-spatial are curated from well- recognized image datasets. One key contribution is that all instances all of our corpora require only spatial reasoning about one or two objects, e.g., in What'sUp, we cireumvent the and--and- whole problem discussed in Yamada et al. (2022) by careful construction.
We consider several modeling improvements based on these findings, including: (1) re- normalizing model probabilities account for the implicit text-only prior of captions in LAION-2B; (2) replacing the preposition "behind" with one
Figure 2 contains examples of images from each of our three benchmarks, along with the caption options each image is paired with.

---

WhatsUp (Subsct A)
Amay
mag
Amug right
right
mig table
WhattUUp (Subsat B)
A mug in frons e plate
Amag in of plate
Amug in frencel plate
A mug in frons ol plate
A mug behind a plate
Amy belind a plats
A mug benind a place
A mag bahind plase
A mug the left of plate
mug to the left of plate
A mag to the left of plate
A mug to the left of plate
mugug the right of plate
A mug the right of plano
A mug the right of plate
mug to the right of plate
COCO-spatial
GQA-sparial
person on
ddog the of
An umpin
boy markee boy to the right racect
pe son right
A dog to the rignt of bench
umpite on right
Figure 2: Examples from our three proposed benchmarks. Each image is paired with four text options What's and two text options in COCO-spatial GQA-spatial. Given single image and corresponding text options, VL model must select the correct option.
2.1 Collection and statistics
model performance, e.g. always guessing that the mug is on the table based on training priors; and (3) we are able to run specialized experiments studying model representations such as in $2.4.
What'sUp We captured 820 images of pairs of household objects in unambiguous spatial relation to each other. 408 of these (Subset A) contain an object under, left of, right of table, chair armchair. The other 412 (Subset B) contain object in front of, behind, left of right of another object on black tabletop. For a given object pair, each preposition is represented; thus each subset of Wh has equal representation each preposition.
The primary differences between the subsets are: (1) in Subset B, the two objects are closer size than in Subset A; and (2) in Subset B, there no obvious prior on the spatial relationship between the two objects, whereas in Subset A, e.g., mug would usually go on a table.
We created benchmark from the validation set of COCO (Lin et al., 2014) using detection annotation data. We select images with only one instance of each object mentioned in the text input, where the area of cach is at least 3% the area of the image. Unlike in What'sUp, these images contain objects that may embody multiple spatial relations, e.g., an object that both the top of and to the left of another object. Thus, we provide only caption options that are mutually ex-
COCO-spatial
These images were captured with a tripod, with minimal changes between images in terms of posi- tion and lighting, except for the placement of the objects. This allows the benefit of real-world im- ages, while exhibiting the controlled nature of syn- thetic images. This control has several advantages: (1) we are able to evaluate model performance on pairs or sets of images, as described $2.2; (2) we overcome textual biases that could falsely improve

---

clusive (to the left of vs to the right of, above vs below). Similarly for one-object images, we only test for mutually exclusive spatial relations (on the left vs on the right, on the top vs on the bottom). This benchmark contains 2687 images, with two caption options each.
Human estimated performance. We also esti- mate human performance on our three benchmarks, We sample 100 data points from each benchmark and, to ensure quality of the annotations, invite experts to voluntarily annotate the data. The anno- tators have all taken at least one graduate course in NLP. They are asked to determine whether the correct caption is an obvious choice, or if there is any scope for ambiguity. This estimate of human performance is 97.3% on COCO-spatial, 99% on GQA-spatial, and 100% on What'sUp.
GQA-spatial We isolated questions targeting basic spatial relations from the GQA validation dataset (Hudson and Manning, 2019), which is sourced from Visual Genome (Krishna et al., 2016). The questions we isolate are of the form "Is the object on the preposition of the image?" or "Is the object, to the preposition of objectz?", when the object(s) mentioned are all present in the image, to avoid conflation with object detection. We retain attribute-object pairs (e.g., "white car") only if the attribute does not affect the answer (e.g., there is only one car in the image), to avoid conflation with attribute detection. Similar to COCO-spatial, we select images where the area of each object in the question is at least 3% of the image. We manually filtered out noisy images, e.g., those with multiple instances of objects in the question with different spatial relations. Finally, convert these ques- tions to a templated caption format. This bench- mark contains 1451 images, with two caption op- tions each, due to the same ambiguity as in COCO- spatial of objects having multiple spatial relations.
Models. The models we study in the zero-shot setting are: CLIP (Radford et al., 2021) ViT-B/32 and ViT-L/14; a version of CLIP ViT-B/32 that has been finetuned on word order shuffling data called negCLIP (Yuksekgonul et al., 2023); a ver- sion of CLIP ViT-B/32 that has been initialized with RoBeRRaappeetrained weights (Ilharco et al., 2021); CoCa, a model trained with generative and contrastive objectives (Yu et al., 2022); XVLM (Zeng al., 2022) with 4M and 16M parameters; BLIP (Li al., 2022) with 14M and 129M param- eters; BLIP2 (Li et al., 2023) image-text matching head (ITM) and image-text contrastive learning head (ITC); and FLAVA (Singh et al., 2022). These models span various modeling choices: one- and two-stack models, generative and contrastive train- ing objectives, different training data, etc.
2.2 Evaluation
We also study several models that have been finetuned on downstream tasks: CoCa which has been finetuned on COCO captioning; two versions of XVLM- 16M that have been respectively fine- tuned on Flickr30K retrieval and COCO retrieval; and three versions of BLIP-14M that have been re- spectively finetuned on Flickr30K retrieval, COCO retrieval, and VQAv2.
Task. For all three benchmarks, the input is an image paired with several caption options that dif- fer only by the preposition they contain. The model must select the caption with the correct preposi- tion. As shown in Figure 2, for What'sUp, there are four caption options; for COCO-spatial and GQA-spatial, there are two.
Almost all of these models capable of yield- ing score representing how well given caption matches a given image. We use this score to evalu- ate whether the model "selects" the correct caption from the given options for an image. As BLIP- VQA and BLIP2-ITC have a text generation head rather than scoring head, we phrase the input set of questions, e.g. "Is the mug on the table?", "Is the mug under the table?", etc, and evaluate the model by measuring the probability of the re- sponses "yes" and "no": if the probability of "yes" highest for the gold option (or "no" is lowest for the gold option all option responses are "no"), we award a point.
Metric. The primary metric we use is the per- centage of images for which the image-text match- ing score is highest for the correct caption com- pared to the incorrect caption(s). The controlled and balanced structure of What's enables two additional metrics for that corpus: pair-wise and set-wise accuracy. Pair-wise accuracy is the ac- curacy on pairs of images that contain opposing prepositions. For example, if the model guesses correctly for "mug on table" and "mug under ta- ble", it gets one point. Set-wise accuracy is similar, but is awarded only when all four prepositions for a given object pair are guessed correctly.

---

whats-
00CO- spatial
CQA-
abilities more than contrastive objective: CoCa
Model
Avg.
Up
spatial
scores less than CLIP ViT-B/32, and BLIP2-ITC scores less than BLIP2-ITM.
CLIP VIT-B/32
31.0
47.4
46,9
41,8
CLIP VIT-L/14
26.1
49.5
47.3
41.0
NegCLIP
34.4
46.9
46.0
42.4
Supervision. XVLM is the highest-performing
RoBERT&CLIP
50.0
49.8
41.6
model of those evaluate, likely due to its more
CoCa
29.4
46.7
47.1
41.0
XVLM 4M
31.5
61.7
58.7
50.6
fine-grained supervision the bounding- level in addition to the image-level.
XVLM 16M
41.9
65.0
58.2
55.0
BLIP 14M
38.5
54.0
49.8
47.5
BLIP 129M
30.4
49.3
49.0
42.9
Finetuning. Finetuning downstream tasks ap-
BLIP2-ITM
37.6
53.0
49.8
46.8
pears to improve model performance sometimes, e.g. BLIP-VQA outperforms BLIP significantly, 44.9
BLIP2-ITC
29.0
53.7
51.0
44,6
FLAVA
30.5
52.6
51.7
but not always, e.g. CoCa-Captioning underper-
CoCa-Caption
24.1
48.6
49.5
40.8
XVLM-Flickr3OK
44.3
65.2
61.4
56.9
forms CoCa.
XVLM-COCO
42.1
71.0
68.1
60.4
BLIP-Flickr30K
33.8
54.2
48.9
45.6
Pair/Set and One-ohject/woooobpee accuracy.
BLIP-COCO
32.8
45.2
Detailed results including pair and set accuracy for
BLIP-VQA
47.8
62.0
58.4
56.0
What'sUp, and one- and two-object accuracy for COCO-spatial and GQA-spatial are presented in Appendix Table 3. All models show very poor pair and set accuracy, showing their lack of understanding of the concept of each preposition. There does not seem to be a uniform trend of model performance on one-object images two-object images.
Random/ / Text-only 25.0
50.0
50,0
41.7
Human Estimate
100.0
97.3
98.8
Table 1: Results of varied VL models on our bench- marks: models in the first section are evaluated zero- shot, and models in the second section have been fine- tuned some downstream task: COCO captioning, retrieval on Flickr30K or COCO, or VQA. All models perform poorly on basic spatial relations.
Inspection of the failure cases shows some mod- els always predicting 1-2 prepositions for all inputs, and others predicting seemingly randomly. Overall, our data allows a very precise evaluation of spatial reasoning, revealing that these models exhibit a failure to understand basic spatial relations, despite nearing human performance on VQAv2, as in the case of BLIP-VQA.
2.3 Results
The performance of the models on our benchmarks is listed Table 1. All models fall far behind human-estimated performance, with many models scoring within a few points of random chance. The number of models we evaluate allows us to draw inferences about various aspects of model design and training, as discussed below.
2.4 Visual analogies
Model architecture. XVLM and BLIP2 perform better than other models in the zero-shot setting, hinting that the increased expressiveness one- stack, cross-attention models vs the two-stack mod- els may indeed matter in this case.
Next, we study the representations of CLIP models on the What'sUp Benchmark. The models are able get some examples correct (e.g. "dog on table", "dog under table"), but they not able to get higher performance, particularly on the pair and set metrics, it hints that they are not learn- ing the generalizable concept of "under" or other spatial relations. To study whether the representa- tions encode these concepts in a generalizable man- ner, we study whether the image representations of these images exhibit the same linear analogies studied in NLP (king man woman queen) (Mikolov al., 2013). We study only CLIP vari- ants this setting, as they alone of the models we study are trained manner to encourage lin- ear recoverability. Specifically, we evaluate CLIP VIT-B/32, ViT-L/14, NegCLIP and RoBERTaCLIP.
Model size in parameters. Scaling up model size does not necessarily improve spatial reasoning capabilities. In the case of XVLM, the 16M model outperforms the 4M model; however, CLIP ViT- B/32 outperforms CLIP ViT-L/14 and BLIP 14M outperforms BLIP 129M averaged across our three benchmarks.
Training objective. Despite helping on other zero-shot tasks such as ImageNet-1K K (Deng et al., 2009; Yu et al., 2022), the generative training objec- tive does not seem to encourage spatial reasoning

---

3
Why do they struggle? Studying LAION
All models we consider in Section 2.2 utilize large- scale image-caption corpora for pretraining. Here, investigate one popular such corpus, LAION- 2B (Schuhmann et al., 2022), to better understand why spatial relations might not learned by mod- els when trained on this type of data. LAION was also used to train OpenCLIP (Ihharco et al., 2021).
Figure 3: Example of edited images with four colors.
Prepositions. We select 25 sets of 4 from What'sUp Subset A: specifically, images where objects are placed around a table. We now evaluate whether [mug on table) I [mmug under table) + I(bowl under table) is the closest to I(bowl table), compared to T(bowl left/right/under table), where I(.) is the image representation. Given 25 objects and preposition options, there are 7200 such analogies. We measure the percentage of these where our condition holds. On average, the four CLIP-based models we study achieve an analogy accuracy of only 9%. The average performance of the models when directly evaluated on the images according to our usual accuracy metric is 31%.
Prepositions occur rarely. We find that captions in the corpus contain common spatially specific prepositions like "under" or "left of" only 0.2% of the time (we additionally filter spatial prepositions that are used in non-spatial contexts, e.g., "under $25"). The individual frequency of each preposi- tion is given in Appendix Table 4.
There are several reasons why this may be the case: alt-text authors may choose not to specify prepositions they feel are obvious (e.g., house "above" the water) ambiguous (e.g., "left" from the viewer's perspective, from the subject of the image's perspective?); the preposition may not be important the writer's eyes when trying to capture holistic information about the entire im- age a short caption (e.g., "a cluttered kitchen", rather than "a fork to the left of a knife on a kitchen counter"); the writer may choose more casual lan- guage (e.g.. "next to" rather than "to the left of""), See Berg et al. (2012) for discussion of how de- scriptions manifest according to similar factors crowdsourced image captioning corpora.
Colors. As a control test for our setup, we next study whether these linear analogies appear in the representation of various colors, which CLIP has been shown to generalize to very well (e.g., correctly identifying blue cow). We isolate 25 objects from the What'sUp Bench- mark, and edit the images to attribute one of four different colors to the object: red, yel- low, green or blue, as in Figure 3. We now evaluate whether [red mug) [yeellow mug) + [yelllow bowl) is the closest to /reed bowl), com- pared to I (yyellow/reen//luee bowl), where T(-) is the image representation. Here, again, we have 7200 analogies and measure the percentage of times the condition holds. On average, the four CLiP-based models we study achieve an accuracy 61%% much higher than for prepositions. They also achieve 100% accuracy when directly evalu- ated on the color options in the same format as our basic evaluation (given one image and four cap- tion options with different colors, select the correct caption). These experiments suggest that models appear learn the concept of color attachments more effectively than spatial relations.
Prepositions can be ambiguous. Of the spatial prepositions that do occur in LAION, examination of the associated images reveals ambiguity. For example, the frame of reference could be defined from the perspective of the viewer of the photo, or of the subject of the photo we follow the same convention as CLEVR (John- son et al., 201 i.e., the perspective the viewer; however, image-text pairs in LAION are scraped from the internet, and thus follow no single conven- tion. As another example, "in front of" could mean closer to the viewer of the photo, ahead of sub- ject that is facing in certain direction in the photo. Even the same preposition with the same meaning could have very different visual appearances, e.g. "a ball under the desk" vs "a ball under the water". few examples are discussed in Figure 4.
our benchmarks,
'The linear analogy accuracy is not very high, but this is perhaps not too surprising given that even performing JPEG compression before encoding changes the image repre- sentation significantly for CL.IP (see https: //github.b. com/ al allenai/mmeen /mcce/mmc////// hitec/////sss//1

---

Image caption
4.1 Incorporating Caption Priors
The first method we consider is re-normalization of probabilities. Intuitively, some captions are more likely on average across all images. We es- timate the prior for caption by calculating its average dot product with a large set of images from a different source to avoid test set contam- ination (e.g. COCO to estimate priors of a VG caption). We then use that prior to re-normalize the caption probability for given image. Specifi- cally, compute re-normalized caption proba- bility as the difference between the un-normalized probability and the caption's 's calculated prior. This process is similar to the text-only normalization of Holtzman et al. (2021). This normalization en- codes that P(caption| image) should not depend on P(captiom).
couator obiinus in image, thus the description tralls above below doss provide much information.
Fasily pleased this striml. Only maraging approx this of darleness betause of the long days. laken between 1000pm and sumnise following day May 31 2009 Sin wics UK love the opanses curvature the trails above and below the calestial equetor. Olympus E3, 7-14mm lene, Just over 1060 expostres stucked stattaiss
COMMEI IE
The caption is transoription of the text over aid on the image, the image dows not c bus or train all.
BAREEFARTT BITEMBES DIRT BESALE
Maurr Determined Thar Was in soid neat bustrin coming right behind you the half hour determined that
made
lvommem
Tables and 6 in the Appendix contain the re- sults of models with and without considering cap- tion priors from different datasets. Overall, seems that normalizing by caption priors does not tend to improve performance on What'sUp much (al- though a slight improvement is observed in pair and set accuracies). The priors are slightly help- ful for performance on COCO-spatial and GQA- spatial, likely because those two image distribu- tions are closer to each other than either What'sUp. However, overall, this approach did not drastically improve model performance on any of the benchmarks. Thus, poor performance of vision-language models cannot be attributed en- tirely to difficult-to tovoerccmm text-only priors on correct options of the captions evaluate.
Undlea what the preposition refers to.
Loaming ects. Fabricc with sewing item and which wooden table bockgrouudd ind Directly about and copy space.
learn
Figure 4: Examples of ambiguity in spatial prepositions used in LAION captions, alongside discussions thereof.
Prepositions rarely needed satisfy the contrastive learning objective. CLIP and sim- ilar models trained contrastively rely on a large batch size to obtain negative examples that require more precise visual representations. For exam- ple, the model learns a visual representation of "Bernese Mountain Dog" rather than just "dog", as there could be several types of dogs in the 32K batch. However, this is not the case for preposi- tions. Given the combinatorial space of all possible sentences, it is unlikely that the exact same descrip- tion would apply to two images a batch with exception a specific preposition. Furthermore, some preposition-object combinations are much more common, e.g., "dog under table" vs. "dog on table". Thus, we hypothesize that the model perform well on the contrastive training objec- tive despite ignoring spatial relationships between objects the image.
4.2 Better prompts: don't fall (for) "behind"
From our study the LAION-2B dataset, we see one word that is not a basic spatial preposition, but gives information about spatial relations, and has relatively high prevalence in the data: "back- ground". This word alone appears in 0.84% of the captions, four times more than all of the other prepositions we study combined. Many of these captions describe synthetic images (e.g., "the words happy new year on a red background"). but others provide spatial information (e.g., "two people talk- ing with some flowers in the background"). The most similar preposition we evaluate is "behind", in What'sUp Subset B.
4 Data-informed attempts at improvement
this section, we operationalize our hypotheses detailed above to yield potential solutions to mod- els' struggle with learning spatial relations.
To determine whether models understand the concept of "behind" (but this knowledge may not be accessible by using that particular word), we

---

do case study of whether models trained on LAION perform better when given prompt of
Whats-
COCO- spatial
GQA- spatial
Model
Avg
Up
"background" or "behind". We take the "in front
CLIP VIT-B/32
31.0
47.4
46,9
41,8
train COCO-spatial
of" and "behind" images from What'sUp Subset B (disregarding the "left of" and "right of" images), changing the text input options to (1) "object1 be- hind objectz" and "object2 behind objecti". or (2) "objecty with objecti in the background" and "objecti with object2 in the background". This al- lows us to evaluate only performance on "behind" vs "background" without conflating other factors such as performance on other prepositions. For CLIP ViT-B/32 and CLIP ViT-L/14 (both Open- CLIP versions trained on LAION), performance on (1) is an average of 52%, just two points above random chance, whereas performance on (2) is an average of 67%.
26.7
63.9
59.5
50.0
and GQA-spatial
LATON-IM -Mppppppp
33.1
46.0
47.6
42.2
LAION-4M -Meeepp-pprpp
29.3
44.4
46.5
40.1
with neg. cap.
Random/ / Text-only Human Estimate
25.0
50.0
50.0
41.7
100.0
97.3
99.0
98.8
Table 2: Results of different types of finetuning on CLIP ViT-B/32. Even with finetuning, the results do not increase by large margin across all benchmarks.
forget standard English. This gives us 900,000 data points, which we downsample to 300,000 for compute reasons. When we finetune on this data, we see the model improves on COCO-spatial and GQA-spatial by an average of 14.6 accuracy points. But performance drops on What'sUp by 4.3 accuracy points. Plausible explanations include the image distributions being different, and that the What's Upp data contains unusual placements of objects. Also, even with significant supervised in- distribution data, performance COCO-spatian and GQA-spatial still lag significantly behind hu- man performance (by -50 accuracy points).
Discussion. This a significant jump, and shows that spatial information may indeed be present in these models, but may have to be teased out more carefully. A strong caveat to these results is that the word "background" seems to be special case: we are able to run this experiment because it appears very frequently in LAION, but we did not come across any other such words that appear frequently and provide spatial understanding. Thus, while this is an interesting thought experiment and provides hope that with more data, the issue can be mitigated, we do not believe it is the solution for models' poor performance on all spatial reasoning tasks.
Finetuning on a subset of LAION including prepositions. We next isolate a subset of LAION including the prepositions we evaluate across our benchmarks. After filtering noise, this subset con- tains 4M image-text pairs. When finetuned on this data, performance improvements are marginal. The reasons for this could be as discussed in Section 3 prepositions in LAION are ambiguous and rarely required to identify the image, even from large batch (we finetune with a batch size of 2048 across 4 NVIDIA RTX A6000 GPUs).
4.3 Finetuning
Finally, we run several experiments with finetuning. Ideally, models should be able to understand basic spatial relations without finetuning, especially as finetuning tends to lose some benefits from pretrain- ing and is tedious and expensive do for various downstream tasks. However, we experiment with some finetuning settings with CLIP ViT-B/32 to determine whether spatial reasoning can be easily learned by our models with extra training. The results are presented in Table 2.
Finetuning on LAION with hard negative captions. Taking inspiration from Yuksekgonul et al. (2023), add hard negative captions to the LAION-4M subset we curate, by programmatically switching the preposition with its opposite. This ensures that the model is forced to distinguish be- tween the two in order to meet the training objec- tive. For CLIP ViT-B/32. we observe a very high training loss, suggesting that the model cannot fit this augmented corpus. We additionally track how
Finetuming on the train equivalents of COCO- spatial and GQA-spatial. We repeat the auto- mated process to curate spatial relations data from GQA and COCO the training set (rather than the validation set, which was used create the benchmarks), dropping the filter for the objects to be at least 3% the area of the image, and dropping the human quality filter. We also combine an equal weight of COCO captions, so the model does not
AAcross several hyperparameter settings, we consistently observed loss of 5.0, compared to the loss of 0.01 for the same configuration without the very hard negatives.

---

the model allocates its probability across the batch: loss on the positive caption is similar the loss on the negative caption, which suggests that CLIP is able to narrow text options down to those two cap- tions, but cannot consistently learn which correct of the two. Experiments with ViT-B/32, ViT-B/16 and VIT-L/14 all show this pattern when finetuned on both 50% and 100% of the data, implying that, at least for the training regime we consider, scaling the data or model size does not help. It likely that an inductive bias or denser supervision is needed to enable the model to learn this, as in XVLM. The train loss curves are provided in the Appendix.
under them². Models may learn these priors and attain high scores on these benchmarks without ac- tually attending to the images (Hsieh et al., 2023) e.g., text-only GPT-1 (Radford et al., 2018) scores 27 accuracy points above random chance on spa- tial reasoning questions in VALSE. In contrast, we capture sets of photographs for What'sUp which are uncluttered, unambiguous, and contain all four preposition options for any pair of objects thus exposing any bias models may have for the "usual" relation between two objects, as well as preventing models with such a bias from leveraging it to mask their spatial understanding abilities.
Text-to-image generation has also been shown to struggle with correctly depicting spatial relations (Gokhale et al., 2022; Hu et al., 2023). Our work sheds light on why this could be the case: e.g., DALL-E 2 (Ramesh et al., 2022) uses a frozen CLIP backbone, and as we show in our work, CLIP itself struggles with spatial reasoning.
5 Related work
Spatial reasoning has long been evaluated by vision- language benchmarks: VQAv2 (Goyal et al., 2016), GQA (Hudson and Manning. 2019), NLVR2 (Suhr et al., 2018), CLEVR (Johnson et al., 2017) and Shape World (Kuhnle and Copestake, 2017) all con- tain questions requiring spatial reasoning. How- ever, many of these questions conflate several types of reasoning. Performance on these benchmarks therefore masks VL models' struggle with spatial understanding specifically.
Conclusion
In this work, we propose three new benchmarks: What'sUp COCO-spatial and GQA-spatial, to evaluate VL. models on basic spatial relations in a range of environments, with the controlled nature of What's spp allowing to evaluate pairs and sets of prepositions for given object pair. We observe that all 18 models we evaluate perform poorly on these benchmarks in zero-shot fashion. Next, we study the LAION dataset which was used to train OpenCLIP, revealing that prepositions are rare, ambiguous, and extrancous in the captions. Finally, we explore potential remedies, ultimately finding that CLIP models, at least in the regime of scale we consider, fail to even fit large-scale training set that requires precise spatial reasoning.
More recently, vision- language benchmarks eval- uating more specifie phenomena have been pro- posed, testing understanding of word order (Thrush al., 2022; Yuksekgonul al., 2023), counting (Parcalabeseu et al., 2021), object-atriibte associ- ation (Yamada et al., 2022), and compositionality (Kamath et al., 2023; Ma et al., 2022). Other work including VALSE (Parcalabescu et al., 2022), VSR (Liu et al., 2023), VL-Checklist (Zhao et al., 2023) and ReCLIP (Subramanian et al., 2022) evaluate spatial reasoning in isolation, we do in our three corpora, testing VL models' ability to match an image to the more fitting of two captions where only the spatial preposition is flipped. They show that models have room for improvement in both zero-shot and finetuned settings.
How might models solve our newly proposed evaluations going forward? Three promising future directions include: (1) Auto-generation of hard neg- atives for spatial prepositions (and beyond) during pre-training; (2) Consideration of more expressive fine-tuned models that support image-text cross- attention and mixes of contrastive and generation objectives; and (3) Thorough scaling experiments to probe for potentially promising relationships between increasing compute of vasion-language models vs. performance on our benchmarks.
However, all of the non-synthetic benchmarks above testing spatial reasoning are based on COCO (Lin et al., 2014) or Visual Genome (Krishna et al., 2016), which are sourced from Flickr. These im- ages tend to have many objects, usually in cluttered environments, which can confuse models trained with only image-level supervision (Yamada al., 2022). The images also reflect biases in our usual world, such as mugs usually being on tables and not
"s of the time of this writing, even querying Google Image Search with "a mug under/left of/right of a table" did not yield any accurate images.

---

Limitations
and Yezhou Yang. 2022. Benchmarking spatial rela- tionships in -xt-tt--mmae generation. arXiv preprint
First, the benchmarks we propose, especially Uhat'sUp, are restricted in scale compared to benchmarks like ARO (Yuksekgonul et al., 2023) and GQA (Hudson and Manning, 2019). Second, our paper focuses on investigating how and why vision-language models struggle with basic spatial relations: our methods to improve models, while grounded in observations from our investigation. do not improve model performance significantly on all of our benchmarks. Third, our work restricted to spatial reasoning. It would be interesting to per- form a wide-scale study tackling several types of reasoming.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. 2016. Making the in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398-414,
Ari Holtzman, Peter West, Vered Schwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competi- tion: Why the highest probability answer isn't 't always right. In Conference on Empirical Methods in Natu- ral Language Processing.
Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. 2023. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. In Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Acknowledgements
We thank John Hewitt, Akhila Yerukola, Liunian Harold Li, and the anonymous reviewers for helpful discussion and feedback, as well as Travis McGuire and Emily Chua for helping us take photographs of Lucy, the star of Figure 1. This work was funded by the Allen Institute for AL. AK was additionally supported by the UCLA Computer Science Depart- ment First- Year Fellowship. KC was supported in part by DARPA MCS under contract number N660011924033 ONR N00014-23-1-2780, and Sloan Fellowship. The views and conclusions con- tained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah Smith. 2023. Tifa: Accurate and interpretable text-to- image faithfulness evaluation with question answer- ing. 2023 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Drew A Hudson and Christopher D Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700-6709.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Lud- wig Schmidt. 2021. Openclip.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR.
References
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019. no- caps: novel object captioning at scale. International Conference Computer Vision, pages 8947-8956.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023, Text encoders are performance bottlenecks in con- trastive vision-language models. In EMNLP.
Alexander C. Berg, Tamara L. Berg, Hal Daumé, Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa Mensch, Margaret Mitchell, Aneesh Sood, Karl Stratos, and Kota Yamaguchi. 2012. Understanding and predict- ing importance in images. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3562-3569.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis. Li-Jia Li. David A. Shamma. Michael S. Bernstein, and Li Fei-Fei. 2016. Vi- sual genome: Connecting language and vision ing crowdsourced dense image annotations. Interna- tional Journal of Computer Vision, 123:32-73.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255.
Alexander Kuhnle and Ann Copestake. 2017. Shape- world a new test methodology for multimodal lan- guage understanding.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In ICML.
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vib- hay Vineet, Eric Horvitz, Ece Kamar, Chitta Baral,

---

Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping pegguage-image pre- training for unified vision-languagg understanding and generation. In Proceedings of the 39th Interna- lional Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12888-12900. PMLR.
Srivatsa R Kundurthy, Katherine Crowson, Lud- wig Schmidt, Robert Kaezmarezyk, and Jenia Jitsev. 2022. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty- sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami. Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. FLAVA: A foun- dational language and vision alignment model. In CVPR.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Europann conference on computer vision, pages 740-755. Springer.
Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. 2022. ReCLIP: A strong zero-shot baseline for re- ferring expression comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5198 5215, Dublin, Ireland. Association for Computational Linguistics.
Fangyu Liu, Guy Edward Toh Emerson, and Nigel Col- lier. 2023. Visual spatial reasoning. Transactions of the Association for Computational Linguisties.
Zixian Ma. Jenry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. 2022. Crepe:
Can vision-language foundation models reason com- positionally? In CVPR.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang. Huajun Bai, and Yoav Artzi. 2018. A corpus for reasoning about natural language grounded in pho- tographs. arXiv preprint arXiv: 1811 1.00491.
Tomas Mikolov, Nya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed repre- sentations of words and phrases and their composi- tionality. NearlPS.
Tristan Thrush, Ryan Jiang. Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for lisio-linguiste compositionality. In CVPR.
Letitia Parcalabescu, Michele Cafagna, Lilitta Murad- jan, Anette Frank, lacer Calixto, and Albert Gatt. 2022. VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8253-8280, Dublin, Ireland. Association for Computational Linguistics.
Yutaro Yamada, Yingtian Tang, and llker Yildirim. 2022. When are lemons purple? the concept association bias of clip. arXiv preprint arXiv:2212. 12..2423...
Letitia Parcalabescu, Albert Gatt, Anette Frank, and lacer Calixto. 2021. Seeing past words: Testing the mross-modal capabilities of pretrained V&L mod- els on counting tasks. In Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), pages 32-44, Groningen, Netherlands (On- line). Association for Computational Linguistics.
Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5534-5542.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Ye- ung. Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text founda- tion models.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and why vision-language models behave like bags-of- words. and what to do about it? In The Eleventh Interna- tional Conference on Learning Represenations.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 Improving language under- standing by generative pre-training.
Yan Zeng, Xinsong Zhang, and Hang Li. 2022. Multi- grained vision language pre- pre-training training: Aligning texts with visual concepts. In Proceedings of the 39th International Conference on Machine Learning, vol- 162 of Proceedings of Machine Learning Re- search, pages 25994-260000 PMLR
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint arX
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. 2023. Vl-checklist: Evaluating pre-trained vision- language models with objects, attributes and rela- tions.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski,

---

A Appendix
This section contains additional results. Table contains detailed results of VL models on our three proposed benchmarks. Table 4 breaks down the prevalence of various prepositions in the LAION- 2B dataset, before and after removing noisy prepo- sitions such "under $25" emphasize that direct count of word occurrence is not sufficient to understand the low prevalence of spatial relations LAION captions. Tables and contain results of the experiments targeting re-normalization of caption priors. Table 7 contains detailed results of different types of finetuning marks. Figures and 6 contain loss curves from finetuming with and without hard negative captions targeting prepositions the train loss from the lat- ter about 500x smaller than the former, and the loss on the gold caption and hard negative caption is about the same, showing that the model struggles to disambiguate between the correct caption and the hard distractor, amongst the entire batch.
our three bench-

---

Upat'sUp Suhset
What'sUp Subset
COCO-spatial
GQA-spatial
Indiv.
Indiv.
Pairs
Set 4
Indiv.
Pairs
Set of 4
One-obj
Two-obj
One-obj
Two-obj
Average
CLIP VIT-B/32
30.3
0.5
0.0
31.6
1.0
0.0
43.7
51.1
46.5
47.4
41.8
CLIP VIT-L/14
1.0
25.7
2.0
49.2
49.8
46.1
NegCLIP
32.5
5.3
0.0
36.3
2.0
0.0
47.4
46.4
45.3
46.7
42.4
RoBERTACLIP
25.2
2.4
0.0
25.0
0.0
0.0
46.3
53.6
50.8
48.8
41.6
CoCa
29.4
2.4
29.4
3.9
48.1
45.2
45.0
49.1
41.0
XVLM
40.0
23.3
23.0
2.0
58.4
62.8
54.6
XVLM 16M
50.7
31.1
1.9
33.1
10.8
0.0
65.4
64.5
63.2
53.3
55.0
BLIP 14M
38.8
23.8
0.0
38.2
5.4
0.0
54.2
53.9
49.1
50.5
47.5
BLIP 129M
4.9
30.4
3.9
44.8
50.5
47.4
42.9
BLIP2-ITM
44.9
24.3
0.0
30.4
2.0
0.0
48.3
57.7
46.0
53.6
46.8
BLIP2-ITC
35.9
3.4
0.0
22.1
0.0
0.0
55.6
51.8
52.6
49.5
44,6
FLAVA
33.7
17.5
0.0
27.2
4.4
0.0
50.3
55.0
52.2
51.2
44.9
CoCa-Caption
25.5
1.9
0.0
22.8
0.0
0.0
45.9
51.4
48.5
$0.5
40.8
XVI.M-Flick330K
45.1
16.5
0.0
43.4
17.2
1.0
63.1
67.3
64.7
58.1
56.9
XVLM--OCO
41.7
17.0
42.4
15.7
68.4
69.1
67.0
60.4
BLIP-Flickr30K
29.6
3.9
38.0
10.3
50.0
50.3
47.4
BLIP-COCO
35.7
1.9
0.0
29.9
2.0
0.0
46.4
56.4
50.3
52.6
45.2
BLIP--QA
57.8
44.2
1.9
37.7
21.1
0.0
63.6
60.5
63.8
52.9
56.0
Random chance
25.0
6.3
0.4
25.0
6.3
0.4
50.0
50.0
50.0
50.0
41.7
Table 3: Detailed results of varied llngunelanguage models on our benchmarks: models in the first section are evaluated zero-shot, and models in the second section have been finetuned on a downstream task: COCO captioning. retrieval Flickr30K COCO, VQAv2. All models perform poorly on basic spatial relations, especially under the pair and set metrics (not included in the individual averages column).
before removing noise noise
% after removing
Preposition
in front of
0010
0.0862
behind
0,0983
0.0489
above
0.0898
0.0422
on top of under
0.0183
0.0134
0.2700
0.0097
the top
0.0074
0.0050
below
0.0309
0.0040
the left 0.0059
0.0038
the right 0.0065
0.0028
at the bottom to the right of 0,0011
0.0037
0.0023
0.0005
to the left of Total
0.0009
0.0005
0.6412
0.2191
Table 4: Frequency of appearance of various prepo- sitions in LAION-2B (english). The spatial relations study represent less than 0.22% of the training data when combined, after removing noise.

---

What's Up
CoCO-spatial with priors
GQA-spatial with prions
Average
W.O. priors
with priors
priors
priors
prions
with priors
CLIP VIT-B/32
31.0
30.0
47.4
54.0
46.9
46.2
41.8
43.4
CLIP VIT-L/14
26.1
28.2
49.5
51.5
47.3
46.8
41.0
42.2
NIP
34.4
32,9
46.9
51.0
46.0
47.0
42.4
43.6
RoBERTaCLIP
25.1
25.7
50.0
50.7
49.8
51.3
41.6
42.6
CoCa
29.4
32.3
46.7
49.2
47.1
47.7
41.0
43.1
CoCa-Caption
24.1
26.5
48.6
48.9
48.6
40.8
41.3
Random chance
25.0
25.0
50.0
41.7
Table 5: Summarized results
experiments incorporating caption priors. each model we have shown the best performance from different methods of calculating caption priors. Incorporating the low caption priors improves performance in some cases, but not by a large margin overall - in many cases, even with improvement the model still performs below random chance. Detailed results are shown Table 6.
HET LIDDUR BUCCHA MUTUETE PILDDU
JMY
BUTU MICAIICHE
LOUB PUL
Figure 5: Train loss (left) and negative caption loss (right) when finetuning variants of CLIP LAION-4M-prep with hard negatives targeting prepositions, either the full dataset or half of the dataset (suffix _2M).
LAAON
co18
4%
500g
Figure 6: Train loss when finetuning variants of CLIP LAION-4M-pres without hard negatives, either the full dataset or half of the dataset. The loss is about 500x lower than in Figure 5.

---

No Priors
What's Cp Subset A
What'sUp Subset
cocO-spatial
CQA-spatial
Indiv. Avg. w.o.
Indiv. Arg. w.o.
Indix.
Set
Indiv.
Pairs
Onc-obj.
Two-ubj.
One-sbj.
Two-obj.
COCO-spatial
GQA-spatial
CLIP VIT-R/32
30.3
0.5
0.0
31.6
1.0
0.0
41.7
51.1
46.5
47.4
39.0
39,2
CLIP V/T.1/14
26.5
1.0
0.0
25.7
2.0
0.0
49.2
49.8
46.1
48.5
36.7
37.8
NegCLIP
32.5
5.3
0.0
36.3
2.0
0.0
47.4
46.4
45.1
46.7
40.2
40.6
RoBERTaCLIP
37.5
CoCa
29.4
2.4
0.0
29.4
3,9
0.0
48.1
45.2
45.0
49.1
38.2
38,0
CoCa-Caption
25.5
1.9
0.0
22.8
0.0
0.0
43.9
51.4
48.5
50.5
36.8
36.4
Random chance
25.0
6.3
0.4
25.0
6.3
0.4
50.0
30.0
50.0
50.0
37.5
37.5
COCO priors
What'sUp Subset A
What'sUp Subset
COCO-spatial
CQA-spatial Two-obj.
Indiv, Avg. w.o.
Improvement over no prior
Indiv.
Pairs
Pairs
Set of
One-obj.
Two-obj.
One-obj
COCO-spatial
CLIP VIT-B/32
29.4
8.7
30.6
2.5
0.0
47.5
45.0
-0.9
CLIP WTTL/14
31.1
5.8
25.2
5.4
0.0
46.0
47.7
37.5
0.8
NegCLIP
31.1
7.8
0.0
34.8
2.0
0.0
45.9
48.1
40.0
0..3
ReBERTSCLIP
24.3
0.0
0.0
26.0
1.5
0.0
50.3
52.3
38.2
0.8
0.0
CoCa-Captine Caption
26.7
3.9
2.0
0.0
49.2
37.5
0.7
Random chance
25.0
6.3
0.4
25.0
6.3
0.4
50.0
50.0
50.0
50.0
37,5
priors
Uhat'sUp Subset
UUp Subset
COCO-spatial
CQA-spatial One-obj
Indiv, GQA-spatial
Improvement over no prior
Indiv.
Pairs
Set of
Indiv.
Pairs
Set of
One-ohj.
Two-ohj.
Two-obj.
CLIP NIT-B/32
29.9
8.7
0.0
29.2
2.0
0.0
$1.7
56.3
41.7
2.5
CLIP L/14
30.3
5.3
0.0
25.5
5.4
0.0
30.5
52.5
39.7
1.9
NegCLIP
31.1
ReBERT-CLIP
24.8
0.0
26.7
1.5
0.0
48.1
53.2
38.2
0.7
CoCa
33.5
5.3
0.0
30.4
1.0
0.0
50.5
47.8
40.6
2.5
CoCa-Caption
26.7
2.9
0.0
26.2
1.5
0.0
50.1
47.8
31.7
1.3
Random chance
25.0
6.3
0.4
25.0
6.3
044
50.0
50.0
50.0
50,0
37.5
Table 6: Detailed results of the experiments incorporating caption priors, with different methods of calculating caption priors: no priors (top), COCO priors (middle), VG priors (bottom). Incorporating the low caption priors improves performance in some cases, but not by a large margin overall.
What'sUp Subset A
What'sUp Subset B
COCO-spatial
GQA-spatial Twu-obj.
Indix, Avg.
Indiv.
Paits
Sei
Indiv.
Pairs
Set
One-obj.
Twu-obj.
One-obj.
B/32
FT train COCO-pattal.CAA-epataa
28.2
7.3
0.0
25.2
2.0
0.0
67.2
60.7
64.4
54.6
50.0
FT on LAION-4M-pop
31.6
1.0
0.0
34.6
2.9
0.0
43.1
48.9
44.3
50.9
42.2
on prep neg. cap.
32.0
Random chance
25.0
6.3
0.4
25.0
6.3
50.0
50.0
50.0
50.0
41.7
Table 7: Detailed results of different types of finetuning on CLIP ViT-B/32.


Reviews:
[{"review_id": "yQXc44aw4g", "reviewer": "Reviewer_ciVg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank all the reviewers for their detailed and constructive reviews. Thank you to Reviewer ciVg for pointing out our “substantial contributions”, “astounding number of tested models”, “carefully introduced data capturing rare spatial relations”, and “interesting analysis”. \n\n> **Why VLMs fail at spatial reasoning / Why data-informed improvement methods didn’t succeed:**\n\n* We agree that the rarity of clearly depicted spatial prepositions in the data (see Figure 4 and Table 4), along with the training objectives not incentivizing spatial reasoning, contribute to the models’ poor performance. As to why the data-informed improvements did not succeed, we find this surprising and intriguing as well, and expect that further research is necessary to answer this question. \n* The limited improvement mirrors results shown on the Visual Spatial Reasoning benchmark (Liu et al, TACL 2023), where finetuning on in-domain spatial reasoning data only boosts performance of the top-performing model (LXMERT) to about 70%, where human performance is 95%. \n* Although finetuning with hard negatives has been shown to help with word order shuffling (Yuksekgonul et al, ICLR 2023), counting (Paiss et al, arxiv 2023) and the VL-checklist benchmark (Doveh et al, CVPR 2023), we see that for prepositions, the models struggle to fit even the training data when finetuned with hard negatives (L569-585) --- the loss is split between the correct caption and the hard negative caption for the image, hinting that even an additional loss component targeting those two in the finetuning objective (as in Paiss et al 2023 and Doveh et al, 2023) would not be helpful. \n* We believe that these results show that more research is required to improve spatial reasoning across all model architectures, potentially in the form of denser supervision.\n\n> **Previous work on spatial relations:**\n\n* Thank you --- we will expand our discussion of related work, as suggested by the other reviewers as well. Although spatial reasoning has not been extensively studied in vision-language (Liu et al, TACL 2023; Parcalabescu et al, ACL 2022), it has been studied in text-to-image generation (Gokhale et al, arxiv 2023) and is also important in vision-language navigation (Shridhar et al, CVPR 2020). Per other reviewers’ comments, we also plan to add other types of reasoning in VL beyond spatial reasoning (such as color, size, etc) to the literature review: for specific citations, please see our response to the other reviewers.\n\n\nThank you for pointing out VALSE (Parcalabescu et al, ACL 2022) --- this is relevant and interesting work; we will discuss this in the revision! We differ from this work in the controlled nature of our data, as well as the rare spatial relations we study. Consider the random sample of the spatial relations in VALSE in Table 9: GPT-4 text-only is able to resolve all of them because, unlike our work, the images depict the expected commonsense scenario (e.g. “A cow stands on a sidewalk _outside_ a building.”, “A cow stands on a sidewalk _in_ a building.”).\n\nThank you again for your thoughtful comments. Please let us know if you have any further questions or comments, we would be happy to follow up!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The contributions of this paper are substantial:\n\n* The number of tested VL models is astounding: 18!\n\n* The authors have carefully introduced new data (dog on the table) to also capture rare spatial relations.\n\n* Interesting analysis of LAION-2B which is used as a pretraining corpus for VLMs. It is surprising that the strategies which come out of these analyses, do not work too well. This is really good to know.", "reasons_to_reject": "* I have to say that even after reading this paper, I now do not feel more informed about why VLMs fail at spatial reasoning than before. It still seems that we have no idea of what goes on in these models and the combination of the training task and rarity of some prepositions in the training data are to blame, but the exploration of data-informed improvements did not succeed. Why? This makes the paper's impact onto the community less than I expected from title, abstract and introduction.\n\n* The paper does not make very clear what previous work on spatial relations does and how it differs from it.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is a great study of how well VLMs (Vision and Language Models) capture spatial relationships. It is comprehensively covering VLMs, testing 18 of them (discriminative and generative). It also provides an analysis of the LAION dataset, exploration of different pretraining strategies, but with little success.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L625: \"We **obesrve** that all\""}, "extra_text": {"Missing_References": "You cite Letitia Parcalabescu, Albert Gatt, Anette Frank, and Iacer Calixto. 2020. \"Seeing past words: Testing the cross-modal capabilities of pretrained v&l models on counting tasks\", while their follow-up work on VALSE is even more relevant because it includes spatial relations and even more phenomena: https://aclanthology.org/2022.acl-long.567/"}}, {"review_id": "S3pYvMUkxd", "reviewer": "Reviewer_4kKj", "summary": "", "questions": "Check my detailed comments", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank all the reviewers for their detailed and constructive reviews. Thank you to Reviewer 4kKj for pointing out our “interesting problem”, “clear writing” and “detailed experimental descriptions”. We will certainly put our work in a broader context, as suggested by other reviewers as well: targeting both spatial reasoning in other fields (such as image-text generation, as in Gokhale et al, arxiv 2023) and other types of reasoning in vision-language (size, color, etc, as in Zhang et al NAACL 2022). \n\n> **Three suggested citations:**\n\n* Thank you for pointing these out; we will add citations and discussion in revision. However, we believe that these works do not invalidate the novelty of our contributions. Our work studies spatial reasoning in VL models (e.g. “a dog on a table”). Zhang et al (NAACL 2022) discuss reasoning in VL models, but do not address spatial reasoning. Mirzaee et al (NAACL 2021) discuss spatial reasoning, but in text-only models, not VL models. Singh et al (Arxiv 2022) discuss one very specific spatial relation in VL models: relative elevation above the ground. \n* To our knowledge, our work is the first to (1) propose spatial reasoning data in a very controlled setting – one cannot game it with text-only priors; (2) study rare spatial relations (e.g. “a dog on a table”); (3) perform a detailed study of a large number of VL models in this setting; and (4) perform comprehensive experiments investigating why the models perform poorly. We believe that our contributions and novelty still hold in light of the relevant work (including Liu et al, TACL 2023, as discussed in L611-618, and Parcalabescu et al, ACL 2022 --- please see our response to Reviewer ciVg for discussion about this paper). \n\nThank you again for your thoughtful comments. Please let us know if you have any further questions or comments, we would be happy to follow up!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting problem\n- Clear writing", "reasons_to_reject": "- Lack of novelty\n- Shallow literature review", "questions_to_authors": "Check my detailed comments", "paper_topic_and_main_contributions": "The paper presents an examination of spatial relations in visual language models, aiming to address this crucial aspect in a scientific manner. To achieve this, the authors introduce a novel learning resource by curating three new corpora: COCO-prep from COCO, GQA-prep from GQA, and RealCLEVR from images. These corpora enable quantification of the model's ability to comprehend basic spatial relations. The evaluation results on 18 visual language models reveal a significant lack of understanding of the task, with most models performing not significantly better than random guessing.\n\nThe study of spatial understanding in transformer-based models is a topic that has been widely explored in the literature. The authors present their findings with clarity and provide detailed experimental descriptions to support their conclusions.\n\nOne major shortcoming is the degree paper's contribution to the field. While the authors have produced valuable insights into the spatial understanding of visual language models, the lack of spatial understanding in these models is a well-studied concept. It appears that previous research in this area, such as the works by Zhang et al. (NAACL2022), Mirzaee et al. (NAACL2021), and Singh et al. (ArXiv 2022), has not been adequately acknowledged or addressed in light of their own contributions. To be considered for acceptance in EMNLP, it is crucial for the paper to present a clear distinction of its unique contributions relative to the existing body of research that has been available for more than a year.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Check my detailed comments"}, "extra_text": {}}, {"review_id": "KNlX0UoLDu", "reviewer": "Reviewer_sZnW", "summary": "", "questions": "- What do the authors think can be concluded from the experiment replacing \"behind\" with \"background\"? That more data could be necessary (if not sufficient) for solving the task? Moreover, how do these results relate or compare to the findings that fine-tuning mostly doesn't help? I think both these results are very interesting, but I'm missing some real discussion on their implications.\n- Not a question, but rather a suggestion for improvement: while the Visual analogies ablation is very interesting, the motivation for having this analysis should be spelled out a bit more clearly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank all the reviewers for their detailed and constructive reviews. Thank you to Reviewer sZnW for pointing out our dataset as a “very useful resource with a controlled nature”, our “very comprehensive experiments”, and our “extremely interesting results [...] that the community should take very seriously”.\n\n> **Clarification on Human Judgments:**\n* The 100 data points were selected randomly from each benchmark. One of the authors familiar with the task but who had not previously seen the data (their native language is English) annotated whether or not they agreed with the ground truth annotation. The author tried to give a very conservative estimate (i.e. is there any possible interpretation that could reveal ambiguity in the data?). We will repeat the human study with multiple paid annotators, and report details in the final version of the paper.\n\n> **Changing the name of the dataset:**\n* We agree that the benchmark name could falsely imply that it is based on the CLEVR dataset. We will continue to brainstorm other names --- perhaps simply “what’s up”.\n\n> **Adding further discussion on top of the various results:**\n* In an extra camera ready page, we would add analyses about various model architectures, training data and objectives, and how they correlate with performance on the data. Example discussion points we had to cut for space and would add back: 1) we were surprised that CoCa did not outperform CLIP on our benchmarks --- we had thought that the generative training objective would be helpful vs. constrastive-only; 2) we were unsurprised to see XVLM outperform other models, as it has more fine-grained supervision at the bounding-box level in addition to the image-level.\n\n> **Clarification on Behind versus Background:**\n* The “behind” vs. “background” result shows that prompts can matter, i.e., in that case, the model can do better for some spatial relations, but only if the capability is accessed with a data-informed prompt. We were hopeful that fine-tuning might alleviate the “specific prompt required”  issue, but none of the four methods we used were able to surface similar improved behavior more broadly. We will add more discussion in revision.\n\n> **Suggestion regarding Visual Analogies:**\n* Thank you for your suggestion. Our motivation was to show that although the CLIP objective tends to make information linearly recoverable (as shown by strong results of a linear probe trained on Frozen CLIP embeddings on classification tasks like ImageNet), this does not occur for prepositions, otherwise they would have exhibited the same linear analogy behavior as Word2Vec. However, colors do exhibit this, serving as a control for the experiment. We will make the motivation clearer in the paper.\n\n> **Suggested references, writing clarity:**\n* Thank you for your suggested citations. Per other reviewers’ requests as well, we will put our work in a broader context, discussing both spatial reasoning in non-VL models (as recommended by Reviewer 4kKj), and other types of reasoning in VL models, as you and Reviewer 4kKj recommend.\n* Thank you also for your suggestions with respect to the writing. We will make the points you mentioned clearer, as well as improve the overall quality of the writing in the paper. The ARO benchmark we mention in Limitations is from Yuksekgonul et al 2023, which we will specify.\n\nThank you again for your thoughtful comments. Please let us know if you have any further questions or comments, we would be happy to follow up!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed dataset, though relatively small, is a very useful resource to test models' ability to deal with spatial relationships. I particularly value the controlled nature of the dataset, which prevents models from exploiting distribution/training biases. I would just recommend the authors name it differently (see my point below).\n- The set of models used for the experiments is very comprehensive and includes many different types of architectures.\n- The results are extremely interesting, and highlight one key limitation of current L&V models -- spatial understanding -- that the community should take very seriously.", "reasons_to_reject": "- It is not explained how human judgments on a sample of 100 data points were collected (if any): who are the annotators, how many per sample, how were they recruited, what was their native language, how much they were paid, what were the instructions, and what it means to compute a \"conservative accuracy\" on the task. These are important details that should be given.\n- Not a reason to reject, per se, but I don't think it is fair to name the new dataset RealCLEVR considering that it doesn't directly build on the actual CLEVR dataset (which contains images of geometrical objects with few basic features: color, texture, size, etc.). In this sense, the name is misleading, and I would recommend the authors to consider using a different one.\n- The paper lacks an insightful discussion on top of the various results: zero-shot, fine-tuning, and ablations. What do the results implicate and what do they tell us about model abilities to deal with spatial prepositions and similar expressions?\n", "questions_to_authors": "- What do the authors think can be concluded from the experiment replacing \"behind\" with \"background\"? That more data could be necessary (if not sufficient) for solving the task? Moreover, how do these results relate or compare to the findings that fine-tuning mostly doesn't help? I think both these results are very interesting, but I'm missing some real discussion on their implications.\n- Not a question, but rather a suggestion for improvement: while the Visual analogies ablation is very interesting, the motivation for having this analysis should be spelled out a bit more clearly.", "paper_topic_and_main_contributions": "The paper explores whether, and to what extent, current language-and-vision models understand spatial relationships between objects depicted in images. In particular, they focus on a bunch of relationships expressed by common English prepositions -- on, behind, on the left of, etc. -- and experiment with <image, sentence> samples from 3 datasets: COCO, GQA, and RealCLEVR, a novel dataset collected by the authors to genuinely focus on object positioning while preventing models to exploit biases due to data distribution (typical positioning, etc.). The authors evaluate several pre-trained models and show that none of them is genuinely able to understand spatial prepositions. They propose a few ways to improve models, which overall do not have a great impact on the performance. \n\nThe contributions of the paper are 1) a novel language-and-vision dataset to study spatial relationships; 2) an exhaustive investigation of current L&V models on the task; 3) an analysis of the data used to train OpenCLIP with respect to spatial prepositions; 4) some directions on how to improve model's performance on dealing with spatial relationships -- however, none of them is shown to dramatically boost model behavior.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Kuhnle, A., Xie, H., & Copestake, A. (2018). How clever is the FiLM model, and how clever can it be. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops (pp. 0-0).\n- Kuhnle, A., & Copestake, A. (2017). Shapeworld-a new test methodology for multimodal language understanding. arXiv preprint arXiv:1704.04517.\n\nAnd maybe work on similar and related reasoning abilities, such as assessing objects' size:\n- Pezzelle, S., & Fernández, R. (2019, November). Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 2865-2876).\n- Pezzelle, S., & Fernández, R. (2019, November). Big generalizations with small data: Exploring the role of training samples in learning adjectives of size. In Proceedings of the Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN) (pp. 18-23).\n\nand color:\n- Schüz, S., & Zarrieß, S. (2020, July). Knowledge supports visual language grounding: A case study on colour terms. In Proceedings of the 58th annual meeting of the association for computational linguistics (pp. 6536-6542).", "Typos_Grammar_Style_and_Presentation_Improvements": "- The abstract appears to have an overall lower writing quality compared to the paper -- I suggest the authors revise it. \n- Missing capitalizations in the title and (sub)section titles; unwanted period in the title\n- Contractions like \"doesn't\" should be avoided\n- The first sentence of the abstract is not fully understandable: since VQAv2 is not only about left and right, the rhetorical question in there seems out of place\n- random chance > chance level\n- The captions that image is paired with are: unclear sentence\n- what is the ARO benchmark?"}}]

Author rebuttal:

Meta-review: The paper investigates how Vision and Language Models (VLMs) grasp spatial relationships. It undertakes a comprehensive examination of VLMs, evaluating various models. An analysis of the LAION dataset is presented, along with the introduction of several new corpora, namely COCO-prep from COCO, GQA-prep from GQA, and RealCLEVR. Despite the meticulous experimentations and introduction of new data to understand rare spatial relations, the results indicate that most VLMs are not performing exceptionally better than random thinking in spatial reasoning. Moreover, the study supplies insights but sometimes advances our understanding of why the models fail in this domain.

**Reason To Accept**

- Contributions: The paper's contributions are notable, especially with testing a significant number of VLMs (18 in total).
- Data Introduced: The authors have integrated new data to evaluate rare spatial relations, which is commendable.
- Clarity: The findings are clearly presented, detailed experimental descriptions support their conclusions, and the writing is clear.
- Relevance: The issue of spatial understanding in models is crucial, and this paper's results shed light on existing shortcomings.

**Reason To Reject**
- Ambiguous Impact: Even after extensive exploration and analysis, the paper must significantly advance the understanding of why VLMs fail at spatial reasoning.
 - Literature Review: The paper needs to adequately differentiate its contributions from previous significant works in this area. Key works like Zhang et al. (NAACL2022), Mirzaee et al. (NAACL2021), and Singh et al. (ArXiv 2022) are insufficiently acknowledged.
- Missing Details: The paper must explain essential details about human judgments on specific data points, such as annotator information, recruitment, and calculation methods.
- Limited Insightful Discussion: The paper could benefit from a more in-depth discussion and analysis of the results, especially regarding the implications of the models' abilities concerning spatial prepositions.

**Overall**
The paper contributes valuable insights into the domain of spatial relationships in VLMs. However, the reviewers have highlighted several concerns that limit its potential impact. To be considered for a top-tier conference, it would be essential to address the gaps in a literature review, provide more in-depth discussions on the results, and clarify ambiguities around.
Decision: Accept-Main