Venue: NAACL 2024

# An Examination of the Compositionality of Large Generative Vision-Language Models

Teli Ma\({}^{\dagger,\diamond}\)  Rong Li\({}^{\dagger}\) Junwei Liang\({}^{\ddagger,\ddagger,\circ}\)

\({}^{\dagger}\) AI Thrust, The Hong University of Science and Technology (Guangzhou)

\({}^{\ddagger}\) Department of Computer Science and Engineering,

The Hong Kong University of Science and Technology

\({}^{\diamond}\) Primary author Corresponding author

{tma184, rli335}@connect.hkust-gz.edu.cn junweiliang@hkust-gz.edu.cn

###### Abstract

With the success of Large Language Models (LLMs), many Generative Vision-Language Models (GVLMs) have been constructed via multimodal instruction tuning. However, the performance of GVLMs in multimodal compositional reasoning remains under-explored. In this paper, we examine both the evaluation metrics ( VisualGPTScore, etc.) and current benchmarks for evaluating the compositionality of GVLMs. We identify the syntactical bias in current benchmarks, which is exploited by the linguistic capability of GVLMs. The bias renders VisualGPTScore an insufficient metric for assessing GVLMs. To combat this, we first introduce a **SyntaxBias Score**, leveraging LLMs to quantify such bias for mitigation. A challenging new task is subsequently added to evaluate the robustness of GVLMs against inherent inclination toward syntactical correctness. Using the bias-mitigated datasets and the new task, we propose a novel benchmark, namely **S**yntaCtically **DE**-biased benchmark (SADE). Our study provides an unbiased benchmark for the compositionality of GVLMs, facilitating future research in this direction 1.

Footnote 1: Code and dataset are available at [https://github.com/TelecMe/SADE](https://github.com/TelecMe/SADE).

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 692–705

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

A surge of research on vision-language models (VLMs) has demonstrated success in a wide range of tasks, including zero-shot visual recognition Radford et al. (2021); Gao et al. (2021); Zhou et al. (2022), visual question answering Alayrac et al. (2022); Chen et al. (2022), and image-to-text retrieval Alayrac et al. (2022); Gong et al. (2023). Previous Vision-Language Models (VLMs) have predominantly been developed using image-text contrastive (ITC) learning Radford et al. (2021); Jia et al. (2021); Li et al. (2022, 2023b) and image-text matching (ITM) Tan and Bansal (2019); Chen et al. (2020); Gan et al. (2020); Li et al. (2021); Zhang et al. (2021); Kim et al. (2021) frameworks, a category we term Encoder-based Vision-Language Models (EVLMs). With the advent of large language models (LLMs) like ChatGPT, GPT-4 OpenAI (2023) and LAMA Touvron et al. (2023a), recent studies have extended the decoder-only architecture to multimodal settings, which is named Generative VLMs (GVLMs) Liu et al. (2023); Zhu et al. (2023); Li et al. (2023); Ye et al. (2023); Gao et al. (2023); Sun et al. (2023); Dai et al. (2023). The GVLMs deviate from the EVLMs in projecting visual features into the latent lexical space of LLMs, and leveraging the auto-regressive generative capacity to solve vision-language tasks. In the training process, most work follows the recipe of freezing the main body of visual encoders and LLMs, only updating the negligible parameters of projecting layers, which is also called "bridge architecture" Rajesh et al. (2023).

Despite the emergence of research on GVLMs, the understanding of compositionality in GVLMs has remained an enigmatic black box, with no thorough investigations conducted thus far. Previous research studies Thrush et al. (2022); Zhao et al. (2022); Yuksekgonul et al. (2022); Ma et al. (2023); Ray et al. (2023) in multimodal compositionality focus on establishing retrieval-based benchmarks for evaluating EVLMs on object relations and attribute understanding, order sensitiveness of sentence elements, and atom-level understanding. The EVLMs have demonstrated abilities to discriminate positive captions from negative ones based on the image-text similarity, where the disparities between the positive and negative captions are relatively subtle, such as "an old person kisses a young person" and "a young person kisses an old person" Thrush et al. (2022).

However, we observe there exists an underlying bias towards the LLM part of GVLMs in the evaluation of the aforementioned benchmarks. During the evaluation, the log-likelihood-based

scores are widely adopted to evaluate the generative models (Fu et al., 2023; Liu et al., 2023c; Lin et al., 2023; Li et al., 2023c) to estimate the conditional probabilities of specific generations. Following Lin et al. (2023), we alias the log-likelihood score as VisualGPTScore. We examine the current benchmarks for evaluating GVLMs with VisualGPTScore and find that:

* Using VisualGPTScore to evaluate GVLMs is not sensitive to bags-of-words problems that broadly exist in the evaluation of EVLMs with similarity scores. The bags-of-words phenomenon during evaluation is due to the similarity-based metrics.
* VisualGPTScore sometimes prefers syntactical correctness rather than content-related correctness under the current benchmarks. It scores negative references with reasonable syntax but unrelated content higher than positive references. In contrast, EVLMs pay more attention to the correlation of visual content but are not sensitive to the order of tokens in references.
* A prevalent syntactical bias is present in contemporary multimodal compositional reasoning benchmarks.These benchmarks are tailored for assessing EVLMs, and the approach used to create negative references may not be effective for the evaluation of GVLMs.

Based on these observations, our contributions include:

* We quantitatively analyze the syntactical bias (namely SyntaxBias Score) that broadly exists in current benchmarks by leveraging LLMs.
* With the SyntaxBias Score, we propose a SyntActically DE-biased benchmark (SADE) based on current benchmarks for a more robust multimodal compositionality evaluation. We adopt multiple strategies to mitigate the syntactical bias in existing benchmarks. We also add a new challenging assessment in SADE to evaluate the content understanding across visual and language modalities.
* The performance of several GVLMs is reported on SADE, as well as the robustness and faithfulness to human judgments.

## 2 Background

### Generative vision-language models

In this paper, we define GVLMs as models that combine visual encoders with large language models (LLMs) trained on large text corpora. The prevailing approach in recent research connects a frozen visual encoder with an LLM by training mapping layers on images-text pairs, followed by fine-tuning using multi-modal instructional data to facilitate multi-turn conversations (Liu et al., 2023a; Gao et al., 2023; Zhu et al., 2023; Dai et al., 2023; Su et al., 2023; Gong et al., 2023b; Sun et al., 2023). This approach is anchored in the idea of treating visual tokens the same as linguistic ones. The visual tokens are mapped into a lexical embedding space and harnessed to generate textual content in an autoregressive manner. Formally, given an image \(I\) and the visual encoding \(g(I)\) from encoders like Vision Transformer (Dosovitskiy et al., 2020), the mapping process can be formulated as:

\[{\mathbf{z}}={\bf M}(g(I)),{\mathbf{z}}=\{z_{1},z_{2},...,z_{N}\}, \tag{1}\]

where \(N\) is the number of visual tokens and \({\bf M}\) is the mapping layers. Different from EVLMs that utilize image-text contrastive (ITC) or image-text matching (ITM), the training objective of multi-modal autoregressive training is to maximize the log-likelihood of the next true token. Denote the tokenized instructions as \({\mathbf{p}}\) and the output words as \(t_{i},(1\leq i\leq K)\), the GVLM training objective is defined as:

\[\max_{\theta_{M},\theta_{\sigma}}\sum_{i=1}^{K}\log P(t_{i}|{\mathbf{p}},{\mathbf{z}} ,t_{1},t_{2},...,t_{i-1};\theta_{M},\theta_{\sigma}) \tag{2}\]

where \(\theta_{M}\) refers to the learnable parameters of mapping layers \({\bf M}\) and \(\theta_{\sigma}\) refers to other tunable parameters like adapter layers in LLaMA-Adapter V2 (Gao et al., 2023), or visual abstractor and LoRA in mPLUG-Owl (Ye et al., 2023).

In comparison, the training objectives of EVLMs are based on the ITC or ITM loss between vision and language parts. Please refer to Appendix A.1 for formulations of EVLMs.

### Vision-language compositionality

Recent works on vision-language compositionality focus on introducing benchmarks to evaluate the EVLMs, mainly on CLIP (Radford et al., 2021). Winoground (Thrush et al., 2022) is one of the pioneers in building benchmarks for multimodal 

compositionality, curating 400 test items to evaluate the pragmatics, symbolic and series factors of VLMs. Afterwards, several benchmarks have been proposed to challenge the objects, relations and attributes understanding of VLMs, including VL-CheckList Zhao et al. (2022), ARO Yuksekgonul et al. (2022), CREPE Ma et al. (2023), VALSE Parcalabescu et al. (2021) and Cola Ray et al. (2023)_etc._ These benchmarks are in the form of image-text retrieval, requiring the model to differentiate positive references from negative references based on the visual contents of the images. See Fig 6 in the Appendix for the details of the image-text retrieval format. SugarCrepe Hsieh et al. (2024) is one of the most recent and similar works to ours. SugarCrepe utilizes Vera Liu et al. (2023) and TextAttack Morris et al. (2020) to detect the plausibility and grammar gaps between positive and negative references. Then, it prompts ChatGPT to generate reasonable hard negative references to reduce bias. In comparison, we partially rely on the original benchmarks, focusing on the strategy of mitigating bias by filtering and modifying them based on our defined SyntaxBias Score. All the aforementioned benchmarks are curated for evaluating EVLMs, where similarity scores between images and references serve as the criteria for selecting references. Then, the accuracy of selecting positive samples across all data samples will be reported to assess the model's compositional understanding capability.

### Evaluation metrics for multimodal retrieval

Since previous benchmarks have been carefully curated for evaluating EVLMs, image-text similarity scores naturally emerge as the metric for assessing the compositional similarity between images and references. For generative models, an intuitive way is reference-based, measuring the quality of generated captions with metrics like BLEU Papineni et al. (2002), METEOR Banerjee and Lavie (2005), ROUGE Lin (2004) and CIDEr Vedantam et al. (2015). Among the reference-based metrics, BERTScore Zhang et al. (2019) tackles superficial matching between captions and references in lexical expression, delving deeper into the semantic similarity matching. GPTScore Fu et al. (2023) proposes to leverage emergent abilities of generative models to score generated texts. Inspired by GPTScore, recent works Lin et al. (2023); Li et al. (2023); Liu et al. (2023) measure the GVLMs using the log-likelihood of directly generating reference sentences conditioned on the image. We follow the Lin et al. (2023) to abbreviate the kind of method as VisualGPTScore, which can be formulated as:

\[{\rm VisualGPTScore}(\mathbf{r}|\mathcal{I})\] \[=\sum_{t=1}^{m}w_{t}{\log}P(r_{t}|\mathbf{r}_{<t},\mathbf{p},\mathcal{ I};\theta_{GVLM}) \tag{3}\]

where \(\mathcal{I}\), \(\mathbf{r}\), \(\mathbf{p}\) represents the image, reference sentence and instructions. \(\theta_{GVLM}\) refers to parameters of GVLMs and \(w_{t}=\frac{1}{m}\). The VisualGPTScore is directly estimated conditioned on images and thus reference-free. In this work, we examine the VisualGPTScore and discuss the potential influence of using it in current benchmarks for vision-language compositionality.

## 3 Experimental setup

We introduce the configurations of experiments for the syntactical bias examination in this section.

### Model choices

We leverage two state-of-the-art GVLMs, namely LLaVA Liu et al. (2023) and MiniGPT-4 Zhu et al. (2023), to conduct experiments. LLaVA is one of the first methods to project visual features into LLaMA Touvron et al. (2023) latent space via multimodal instruction tuning. A linear projection layer and the parameters of the LLM are tuned on conversations, detailed descriptions, and complex reasoning datasets. MiniGPT-4 Zhu et al. (2023) maps visual embeddings obtained from ViT and QFormer Li et al. (2022) into Vicuna Chiang et al. (2023) via a linear projection layer. We adopt the model version of "LLaVA-7B-v0" and "Minigpt4-aligned-with-Vicuna7B" to evaluate. However, we found that when using VisualGPTScore to evaluate compositionality, both models exhibited similar patterns. Therefore, for the sake of brevity, we only present the results for LLaVA.

### Datasets

We use Winoground Thrush et al. (2022), VL Checklist Zhao et al. (2022), ARO Yuksekgonul et al. (2022) and CREPE Ma et al. (2023) in the evaluation analysis, totaling 52,189 images and 129,558 reference sentences. All benchmarks necessitate the model's selection of positive reference sentences from negative ones. For Winoground, we report text score, image score and group score as 

the paper (Thrush et al., 2022). For other datasets, Recall@1 accuracy is reported.

## 4 Evaluation Metric Examination

VisualGPTScore measures the probability of generating specific references conditioned on the given images, as defined in Eqn. 3. The generative evaluation method is based on the inherent attribute of GVLMs and used in image-text retrieval (Lin et al., 2023; Li et al., 2023c; Liu et al., 2023d). Since current benchmarks on VL compositions consists of image-text pairs, we follow Lin et al. (2023) to utilize VisualGPTscore for evaluating the VL compositionality of GVLMs. In this section, our primary focus is to examine the bias of using VisualGPTScore in current benchmarks.

### Sensitivity to bags-of-words

Previous research works have pointed out that EVLMs suffer from the bags-of-words phenomenon when doing compositional reasoning due to the pre-training recipe of matching visual and textual data in instances-level (Yuksekgonul et al., 2022b; Diwan et al., 2022). However, we observe that the bags-of-words problem is not only related to the models, but also highly correlated to the evaluation metrics, and VisualGPTScore is not sensitive to the bags-of-words phenomenon.

We explore the influence of different metrics in sensitivity to the order of tokens in sentences for GVLMs. Following CREPE (Ma et al., 2023), we randomly sample 2.5K image-text pairs from the COCO dataset (Lin et al., 2014) and adopt the following strategies to shuffle the elements of captions: _Shuffle only nouns & adjectives_, _Shuffle all but nouns & adjectives_, _Shuffle within trigrams_, _Shuffle trigrams_. Then, we calculate the VisualGPTScore, GPTScore (Fu et al., 2023) and BERTScore (Zhang et al., 2019) based on LLaVA-7B. The distribution of normalized scores are shown in Fig. 1, where x1 represents positive references and x2-x5 represents shuffled references, respectively.

It can be observed that to the same model, LLaVA-7B, VisualGPTSCore is similar to GPTScore, more sensitive to the order and structure of reference sentences compared with contrastive metric BERTScore. We also report the score distribution of the CLIP model using contrastive similarity (CLIPScore in Fig. 1), which is similar to the distribution of BERTScore results on LLaVA-7B. It implies the bags-of-words problem may be attributed to the evaluation metrics based on similarity score, but generative scores mitigate the problem to some extent.

### Sensitivity to syntax and contents

Based on the observation that VisualGPTScore mitigates the bags-of-words problem to some extent, we are curious about whether they lean more towards evaluating syntactic correctness than content relevance when assessing the compositionality of GVLMs. To examine it, we design an experiment using the test set of Flickr30K dataset (Young et al., 2014). Specifically, we sample 507 image-text pairs and construct three types of evaluation cases as shown in Fig. 2. Given an image, the task is to retrieve the positive reference from the cases below. The final scores are averaged over 507 test samples. In Case 1, each positive reference sentence is accompanied by two hard negatives with shuffled nouns, adjectives and trigrams. In Case 2, the provided negatives are fluent and syntactically correct captions sampled from COCO, which are unrelated to the visual contents. In Case 3, we keep only adjectives and nouns in the positive

Figure 1: Box plots of scaled score distributions for original (x1) and perturbed captions (x2-x5, x2:shuffle nouns & adj, x3: shuffle all but nouns & adj, x4: shuffle within trigrams, x5: shuffle trigrams). The distribution gap between the original captions and the shuffled captions is evident for the generative scores, while the contrastive score (BERTScore) is significantly less affected by the order of words. The CLIPScore sub-figure illustrates the distribution of similarity scores generated by the CLIP model, which is compared with the first three sub-figures of LLaVA-7B.

 

reference sentences by removing all the adverbs, pronouns and modifiers.

We present Recall@1 of VisualGPTScore for the GVLM (LLaVA-7B), and vision-language similarity for the EVLM (CLIP) in three evaluation cases. As shown in Fig. 3, the LLaVA model can easily discriminate the right reference sentences from the shuffled ones, reaching 98.62% with the help of VisualGPTScore. However, if the negatives are random reference sentences in Case 2, the performance degradation is up to 31.56%. In Case 3, where the sentences are syntactically incorrect, the performance drops to 27.02%. In contrast, CLIP excels at excluding negative sentences that are contextually unrelated to the image, but suffers from insensitive to syntax and sentence order.

The potential reason for the above results is the difference in the pre-training paradigm. Specifically, the generative model pre-training is to maximize the likelihood of the next token prediction in an auto-regressive manner. In contrast, the training objective of EVLM is to maximize the alignment between positive image-text pairs and minimize that between negative ones. Previous research [43] shows that CLIP takes the short-cut strategy of not encoding the order information, but only object features for retrieval/captioning tasks, which conforms to our finding. We also believe that the generative VLMs take the short-cut strategy of not fully mapping the visual and linguistic features, but leveraging the emerging capacity of LLM part to generate based on limited visual cues. _This reliance on the LLM part results in a bias towards syntactical correctness in captions under the criteria of generative score._

## 5 Benchmarks Examination

From above, we know current benchmarks are curated for evaluating EVLMs based on similarity score originally. Hence, we examine the impact of using these datasets for evaluating GVLMs with VisualGPTScore, and uncover the bias of existing datasets.

### Syntactical bias in current benchmarks

According to the observation made in Section 4, it is evident that auto-regressive vision-language models exhibit sensitivity toward the syntax and order of phrases. Hence, existing benchmarks that generate hard negatives by swapping, shuffling, or replacing specific entities promote a syntactical bias, which refers to a preference for models to

Figure 3: We report the accuracy of VisualGPTScore based on LLaVA-7B and similarity score based on CLIP in the sampled 507 image-text pairs, each pair is consisted of three cases like the example in Fig. 2.

Figure 2: **An example of three Cases of captions we construct to validate the preference of syntax and contents. Right caption: the original caption of the image, Shuffled caption: caption that the sentence elements are shuffled, Random caption: fluent and syntactically correct captions from other datasets (COCO), Content caption: caption that keeps only adjectives and nouns to keep the contents like objects and attributes. We present the normalized VisualGPTScore of every reference sentences in this example. The scores of the Right caption and Content caption may be lower compared to the Random caption (0.405, 0.322 vs. 0.432). This indicates that in this example, generative VLMs tend to prioritize syntactically correct sentences over ones that are more relevant to the content.

 

rely on the morphological structure of words. Consequently, this bias can be exploited by GVLMs to effortlessly differentiate between positive and negative samples.

To show that the bias exists in current compositional reasoning benchmarks, we conduct the ablation of utilizing both GVLMs and EVLMs to reason nonsensical images with normal reference sentences. Specifically, we construct the image-text pairs by replacing the original images with images composed of _random noises_. We observe the performance drop in both the GVLMs and EVLMs. As shown in Fig. 4, the performance degradation of CLIP (ViT-B/32) is large, approaching the Recall@1 accuracy of randomly choosing. However, as for the LLaVA-7B, the trend of performance dropping is not obvious, indicating the GVLMs make the right choices **solely** based on the linguistic reference sentences without visual features. Therefore, almost all the benchmarks lean towards evaluating the linguistic part of GVLMs, rather than the visio-linguistic understanding of GVLMs.

### SyntaxBias Score

To alleviate the syntactical bias in current benchmarks, we first quantify the bias for analysis. In an ideal scenario, in the absence of visual intervention, the quantified scores generated by GVLMs for positive and negative reference sentences should be equivalent. Therefore, we define the SyntaxBias Score to measure the syntactical discrepancy between positive and negative reference sentences. Formally, the SyntaxBias Score is calculated using the generative scores of positive and negative text produced by auto-regressive language models:

\[Score_{SyntaxBias} \tag{4}\] \[=\Delta(\sum^{m}_{i=1}w_{i}{\rm log}P(p_{i}|\mathbf{p}_{<i};\theta)\] \[-\sum^{n}_{j=1}\hat{w}_{j}{\rm log}P(n_{j}|\mathbf{n}_{<j};\theta)),\]

where \(\Delta,\mathbf{p},\mathbf{n},\theta\) represent normalization, positive tokens, negative tokens, and parameters of LLMs respectively. we leverage a strong LLM, Vicuna-13B-v1.5 (Chiang et al., 2023), to compute the SyntaxBias Score, which are normalized between \(-1\) and \(1\). We present the visualization of SyntaxBias Score distributions over different benchmarks in Fig. 5. We find that most of the mainstream benchmarks except Winoground are biased towards positive captions with distribution centers located to the right, _which makes the generative scores of GVLMs on these benchmarks overvalued._

## 6 Mitigate the Bias in Benchmarks

In this section, we propose a strategy to modify the benchmarks and mitigate the syntactical bias to provide a better evaluation of GVLMs. Specifically, we filter current datasets leveraging LLMs and add a novel challenge to evaluate visual content understanding. We name the new benchmark as SyntActical **De**-biased benchmark, abbreviated as **SADE**. In the following, we describe the filtering details of each dataset and the new challenge. Then we show human evaluation to show the effectiveness of SADE.

### Winoground

The Winoground (Thrush et al., 2022) dataset comprises 400 image-text pairs, with each pair consisting of two images and two captions. The two captions exhibit identical sets of morphemes, albeit in

Figure 4: The drop in performance of the LLaVA model when performing compositional reasoning on **nonsensical noisy images** is minimal in existing benchmarks, whereas the CLIP model exhibits a significant decrease. This indicates current benchmarks are exploited by the LLM part of GVLMs, not effective in measuring the multimodal compositionality.

 

different orders. Different from other benchmarks that construct hard negatives by simply altering the positive texts, both positive and negative texts in Winoground are fluent, meaningful, and can match related images. Thus, we include all samples in Winoground into the SADE benchmark without further mitigation, aiming to evaluate the comprehensive multimodal compositional understanding of GVLMs, especially on the _pragmatics, symbolic and series_ factors as introduced in Thrush et al. (2022).

### Relations and attributes

Real-world natural scenes are inherently intricate, encompassing a multitude of specific attributes such as colors, materials, and object relationships. Models that can tackle compositional reasoning require a nuanced understanding that goes beyond mere object-level analysis. Hence, we collect relation and attribute branches from ARO Yuksekgonul et al. (2022) and VL-CheckList Zhao et al. (2022). To mitigate the syntactical bias, we compute the SyntaxBias Score of the samples as described in Eqn. 4 and filter out ones that have a higher score than the threshold. The idea is to ensure that samples with strong syntactical bias are excluded for better vision-language compositional evaluation.

We choose the filtering thresholds of the syntaxBias Score to be close to zero (specifically, by ensuring the \(p-value\) of the SyntaxBias Score is statistically below \(1e-5\)). The filtered data includes 5,193 items from VL-CheckList and 2,328 items from Visual Genome Krishna et al. (2017) to measure relation reasoning, and 5,858 items from VL-CheckList as well as 5,193 items from Visual Genome to evaluate attribute reasoning. Specifically, for VL-CheckList, the _Relation_ branch contains two subclasses, _i.e. action_ and _spatial_, and the _Attribute_ branch includes _action_, _color_, _material_, _size_ and _state_. The number of items in each subclass is elaborated in Table 1.

### Atomic and negate

In CREPE benchmark Ma et al. (2023), the authors propose to assess the VLMs on captions that atoms are replaced or negated. The atom replacing is like _a__bus with a side, light, and window v.s. a train with a side, light, and window_, whereas the atom or sentence negating is as _Another bowl on a cloth with an orange in it. The another bowl has a re: fiction and casts a shadow v.s. Another bowl on a cloth with an orange in it. The another bowl has a reflection and casts something. There is no shadow_. There is a considerable proportion of reconstructed captions in CREPE that are fluent and coherent, thereby we also leverage the same method to filter the samples as we do for relations and attributes.

Figure 5: We visualize the distribution of _SyntaxBias Score_ in current benchmarks. The _SyntaxBias Score_ is defined as the difference between the LLM-based generative scores of positive and negative references. For ARO, VL-CheckLikist and CREPE, the distribution of the SyntaxBias Scores is situated towards the positive end (to the right of the red line), implying that these benchmarks are biased to positive captions syntactically.

 



### Replace syntactic perturbation with a content-only understanding challenge

A plethora of benchmarks perturbs the order information in the reference sentences to measure the word order sensitivity of EVLMs, which tend to treat the captions as _bags of words_ as we present in Fig. 1. The hard negative construction methods include swapping atoms, shuffling nouns, adjectives, trigrams, and all words _etc._ However, due to the intrinsic syntactical awareness of LLMs, the challenge of order perturbation is not effective in assessing the visio-linguistic compositionality of GVLMs. Hence, we abandon the order challenge and propose a content-only understanding challenge.

Specifically, we modify the positive reference sentences from COCO Lin et al. (2014) and Flickr30K Young et al. (2014), keeping only the object- and attribute-related atoms/words. Then, we randomly select fluent, coherent and meaningful reference sentences from other datasets to serve as hard negatives, which are unrelated to the visual content. Examples of this challenging task can be found in Fig. 8 in the Appendix. _The task poses a challenge and exemplifies the robustness of GVLMs against their inherent inclination towards syntactically correct reference sentences._

### Human evaluation of SADE

In order to illustrate that our proposed SADE alleviates the syntactical bias, we ask two annotators to rate the disparity between positive and negative reference sentences. The rating score ranges from -5 to 5, where the higher the score, the more reasonable text is for positive reference sentences. Conversely, the lower the score, the more reasonable the text is for negative ones. The definition of _reasonable_ comprises fluency, syntax, and the meaning of sentences. Note the reference sentences from the original dataset or SADE are agnostic to the annotators and we average the ratings of them. Table 1 clearly demonstrates that the reference sentences in our SADE benchmark substantially mitigate bias, as indicated by the score of human judgments approaching zero. The drop implies that the syntactical disparity between positive and negative reference sentences is drastically narrowed.

### Results of GVLMs on SADE

Based on the SADE benchmark, we report the performance of more concurrent GVLMs based on the VisualGPTScore metric in Table 2. It can be observed that InstructBLIP Dai et al. (2023) and Emu Sun et al. (2023) hold the top-2 positions in almost all dimensions of our benchmark. However, the abysmal performance on _Comprehensive_ and _Content_ implies the vulnerability of Emu when negative reference sentences are hard and challenging. In contrast, InstructBLIP and LLAVA-13B Liu et al. (2023a) are more robust to the _Content_ challenge and achieve high performance on hard negatives. This provides the first de-biased and comprehensive evaluation of recent GVLMs in terms of visual compositionality. **Note that we do not claim that SADE can better measure the performance of GVLMs in all aspects.** However, it can better measure their compositionality with less syntactical bias, which is supported by the reduction of SyntaxBias Score and the human evaluation in Table 1. We believe this benchmark can facilitate a unified and fair comparison for future GVLM research.

## 7 Conclusion

In this work, we evaluate the compositionality of "bridge-architecture" generative VLMs via generative multimodal score, VisualGPTScore. We examine both the VisualGPTscore and current benchmarks for evaluating the multimodal compositional

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & Comprehensive & Relation & Attribute & Atomic & Negate & Content \\ \hline LLaVA-7B Liu et al. (2023a) & 13.00 & 65.52 & 70.55 & 35.01 & 59.01 & 42.02 \\ LLAvA-13B Liu et al. (2023a) & 17.00 & 62.75 & 72.70 & 38.33 & 7.56 & 49.80 \\ MiniGPIT-7B Zhu et al. (2023) & 9.50 & 66.18 & 78.48 & 35.62 & 24.15 & 19.92 \\ mPLUCo-Owl Ye et al. (2023) & 11.00 & 65.91 & 69.04 & 34.90 & 54.61 & 35.73 \\ InstructBLip Dai et al. (2023) & **26.00** & **73.87** & 79.39 & 44.37 & 66.84 & **57.83** \\ LLaMA Adapter V2 Gao et al. (2023) & 7.75 & 58.67 & 65.07 & 31.32 & 20.26 & 10.48 \\ Emu Sun et al. (2023) & 4.00 & 68.54 & **85.84** & **51.38** & **87.20** & 2.79 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation results of GVLMs on SADE benchmark. All the models are instruction-tuned. We present the average performance of two sub-branches within the categories of _Relation_, _Attribute_ and _Content_.



understanding of GVLMs. Based on the examinations, we identify the syntactical bias that exists in current datasets for GVLMs, and define the bias with SyntaxBias Score quantitatively. We then propose a SADE benchmark that mitigates the syntactical bias and provides a better content understanding evaluation for GVLMs. We report the results of multiple GVLMs on our proposed SADE benchmark and uncover new findings of the GVLMs' capabilities.

## 8 Limitations

We discuss the potential limitations of this paper from two aspects. First, our proposed novel benchmark cannot be proved to better measure the performance of generative VLMs in all aspects, including emergent capability, vision understanding and complex reasoning. Our benchmark just evaluates the GVLMs in terms of VL compositionality more fairly by removing the syntactical bias in previous benchmarks. Second, our new benchmark is based on filtering the previous ones, and sampling from them to lower the SyntaxBias Score. Thus, the scale of the whole dataset is relatively small, limiting the generalization of the benchmark to some extent.

## 9 Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 62306257) and the Guangzhou Municipal Science and Technology Project (No. 2024A04J4390). This work was also supported by the Meituan Academy of Robotics Shenzhen. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the National Natural Science Foundation, Meituan, or the Guangzhou Government.

## References

* A. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, C. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems35, pp. 23716-23736. Cited by: SS1.
* S. Banerjee and A. Lavie (2005)Meteor: an automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72. Cited by: SS1.
* X. Chen, X. Wang, S. Changpinyo, A. Piegiovanni, B. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, et al. (2022)Pali: a jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794. Cited by: SS1.
* Y. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu (2020)Uniter: universal image-text representation learning. In European conference on computer vision, pp. 104-120. Cited by: SS1.
* W. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. (2023)Vicuna: an open-source chatbot impressing gpt-d at wth 90%* chatgt quality. See https://vicuna. lmsys. org (accessed 14 April 2023). External Links: Link Cited by: SS1.
* W. Dai, J. Li, D. Li, A. Meng Huat Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and R. Hoi (2023)InterstUDIDIP: towards general-purpose vision-language models with instruction tuning. External Links: Link Cited by: SS1.
* A. Diwan, L. Berry, E. Choi, D. Harwath, and K. Mahowald (2022)Why is winoground hard? investigating failures in visuolinguistic compositionality. arXiv preprint arXiv:2211.00768. Cited by: SS1.
* A. Dossovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, Mostafa Dehghani, Matthias Minderer, G. Heigold, S. Gelly, et al. (2020)An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: SS1.
* J. Fu, S. Ng, Z. Jiang, and P. Liu (2023)Gptscore: evaluate as you desire. arXiv preprint arXiv:2302.04166. Cited by: SS1.
* Z. Gan, Y. Chen, L. Li, C. Zhu, Y. Cheng, and J. Liu (2020)Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems33, pp. 6616-6628. Cited by: SS1.
* P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao (2021)Clip-adapter: better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544. Cited by: SS1.
* P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue, et al. (2023)LIMA-adapter v2: parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010.

 

* Gong et al. (2023) Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023a. Multimodal-gpt: A vision and language model for dialogue with humans. _arXiv preprint arXiv:2305.04790_.
* Gong et al. (2023b) Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023b. Multimodal-gpt: A vision and language model for dialogue with humans. _arXiv preprint arXiv:2305.04790_.
* Hsieh et al. (2024) Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. 2024. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. _Advances in Neural Information Processing Systems_, 36.
* Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In _International Conference on Machine Learning_.
* Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. In _International conference on machine learning_, pages 5583-5594. PMLR.
* Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. _International journal of computer vision_, 123:32-73.
* Li et al. (2023a) Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023a. Otter: A multi-modal model with in-context instruction tuning. _arXiv preprint arXiv:2305.03726_.
* Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. _arXiv preprint arXiv:2301.12597_.
* Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pretraining for unified vision-language understanding and generation. In _International Conference on Machine Learning_, pages 12888-12900. PMLR.
* Li et al. (2021) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. _Advances in neural information processing systems_, 34:9694-9705.
* Li et al. (2023c) Zejun Li, Ye Wang, Mengfei Du, Qingwen Liu, Binhao Wu, Jiwen Zhang, Chengxing Zhou, Zhihao Fan, Jie Fu, Jingjing Chen, et al. 2023c. Reform-eval: Evaluating large vision language models via unified re-formulation of task-oriented benchmarks. _arXiv preprint arXiv:2310.02569_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In _Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_, pages 740-755. Springer.
* Lin et al. (2023) Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. 2023. Visual-gptscore: Visio-linguistic reasoning with multimodal generative pre-training scores. _arXiv preprint arXiv:2306.01879_.
* Liu et al. (2023a) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. _arXiv preprint arXiv:2304.08485_.
* Liu et al. (2023b) Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. 2023b. Vera: A general-purpose plausibility estimation model for commonsense statements. _arXiv preprint arXiv:2305.03695_.
* Liu et al. (2023c) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangubo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023c. Mmbench: Is your multi-modal model an all-around player? _arXiv preprint arXiv:2307.06281_.
* Ma et al. (2023) Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. 2023. Crepe: Can vision-language foundation models reason compositionally? In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 10910-10921.
* Morris et al. (2020) John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. _arXiv preprint arXiv:2005.05909_.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_, pages 311-318.
* Parcalabescu et al. (2021) Letitia Parcalabescu, Michele Cadagna, Lilita Muradjan, Anette Frank, Iacer Calistco, and Albert Gatt. 2021. Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena. _arXiv preprint arXiv:2112.07566_.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In _International conference on machine learning_, pages 8748-8763. PMLR.

 

Kousik Rajesh, Mrigank Raman, Mohammed Asad Karim, and Pranit Chawla. 2023. Bridging the gap: Exploring the capabilities of bridge-architectures for complex visual reasoning tasks. _arXiv preprint arXiv:2307.16395_.
* Ray et al. (2023a) Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A Plummer, Ranjay Krishna, and Kate Saenko. 2023a. Cola: How to adapt vision-language models to compose objects localized with attributes? _arXiv preprint arXiv:2305.03689_.
* Ray et al. (2023b) Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A Plummer, Ranjay Krishna, and Kate Saenko. 2023b. Cola: How to adapt vision-language models to compose objects localized with attributes? _arXiv preprint arXiv:2305.03689_.
* Su et al. (2023) Yixuan Su, Tian Lan, Huyang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. Pandagpt: One model to instruction-follow them all. _arXiv preprint arXiv:2305.16355_.
* Sun et al. (2023) Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2023. Generative pretraining in multimodality. _arXiv preprint arXiv:2307.05222_.
* Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. _arXiv preprint arXiv:1908.07490_.
* Thrush et al. (2022) Tristan Thrush, Ryan Jiang, Max Bartolo,amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and language models for visio-linguistic compositionality. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 5238-5248.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruii Bhosalse, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 4566-4575.
* Ye et al. (2023) Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mplug-owl: Modularization empowers large language models with multimodality. _arXiv preprint arXiv:2304.14178_.
* Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. _Transactions of the Association for Computational Linguistics_, 2:67-78.
* Yuksekgonul et al. (2022a) Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022a. When and why vision-language models behave like bags-of-words, and what to do about it? In _The Eleventh International Conference on Learning Representations_.
* Yuksegonul et al. (2022b) Mert Yuksegonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022b. When and why vision-language models behave like bags-of-words, and what to do about it? In _The Eleventh International Conference on Learning Representations_.
* Yukawa et al. (2021) Mert Yukawa, Federic Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022b. When and why vision-language models behave like bags-of-words, and what to do about it? In _The Eleventh International Conference on Learning Representations_.
* Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Liyuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 5579-5588.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. _arXiv preprint arXiv:1904.09675_.
* Zhao et al. (2022) Tian Cheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. 2022. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. _arXiv preprint arXiv:2207.00221_.
* Zhou et al. (2022) Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krahenbuhl, and Ishan Misra. 2022. Detecting twenty-thousand classes using image-level supervision. In _European Conference on Computer Vision_, pages 350-368. Springer.
* Zhu et al. (2023) Dyeao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. _arXiv preprint arXiv:2304.10592_.

## Appendix A Appendix

### Formulations of GVLMs and EVLMs

In accordance with the discussion in the main text, we define GVLMs as models that combine visual encoders with large language models (LLMs) trained on large text corpora. The visual tokens are mapped into a lexical embedding space and harnessed to generate textual content in an autoregressive manner. Formally, given an image \(I\) and the visual encoding \(g(I)\) from encoders like Vision Transformer (Dosovitskiy et al., 2020), the mapping process can be formulated as:

\[\mathbf{z}=\mathbf{M}(g(I)),\mathbf{z}=\{z_{1},z_{2},...,z_{N}\},\] 

where \(N\) is the number of visual tokens and **M** is the mapping layers. Different from EVLMs, the training objective of multi-modal auto-regressive training is to maximize the log-likelihood of the next true token. Denote the tokenized instructions as \(\mathbf{p}\) and the output words as \(t_{i},(1\leq i\leq K)\), the GVLM training objective is defined as:

\[\max_{\theta_{M},\theta_{\sigma}}\sum_{i=1}^{K}\log P(t_{i}|\mathbf{p},\mathbf{z},t_{1 },t_{2},...,t_{i-1};\theta_{M},\theta_{\sigma}) \tag{6}\]

where \(\theta_{M}\) refers to the learnable parameters of mapping layers **M** and \(\theta_{\sigma}\) refers to other tunable parameters like adapter layers in LLaMA-Adapter V2 Gao et al. (2023), or visual abstractor and LoRA in mPLUG-Owl Ye et al. (2023).

In comparison, the training objective of EVLMs is based on the ITC or ITM loss between vision and language. Given an input image \(I\) and text \(T\), the encoded visual and linguistic features are denoted as \(f_{v}\) and \(f_{t}\). Then, two transformation matrices \(W_{v}\) and \(W_{t}\) are employed to project the visual and text features into a joint feature embedding space, which is formulated as:

\[v=\frac{W_{v}^{\top}f_{v}}{||W_{v}^{\top}f_{v}||},u=\frac{W_{t}^{\top}f_{t}} {||W_{t}^{\top}f_{t}||} \tag{7}\]

In the shared embedding space, ITC loss narrows the discrepancy of vision and language, aligning the image-text pairs in the same batch. The training objective of this process comprises two components, _i.e._\(\mathcal{L}_{v\to t}\) for text retrieval and \(\mathcal{L}_{t\to v}\) for image retrieval. The similarity of matched pairs will be maximized while unmatched ones will be minimized. The formula is:

\[\begin{split}&\mathcal{L}_{ITC}=\mathcal{L}_{v\to t}+ \mathcal{L}_{t\to v}\\ =&-\frac{1}{|\Omega_{v}^{+}|}\sum_{T_{j}\in\Omega_{v }^{+}}\log\frac{\exp(v_{i}^{\top}u_{j}/\tau)}{\sum_{T_{k}\in\Omega_{t}}\exp(v_ {i}^{\top}u_{k}/\tau)}\\ &-\frac{1}{|\Omega_{t}^{+}|}\sum_{I_{i}\in\Omega_{t}^{+}}\log \frac{\exp(u_{i}^{\top}v_{j}/\tau)}{\sum_{I_{k}\in\Omega_{v}}\exp(u_{i}^{\top} v_{k}/\tau)}\end{split} \tag{8}\]

where \(\Omega_{v}\), \(\Omega_{t}\) represent a batch of images and texts while \(\Omega_{v}^{+},\Omega_{t}^{+}\) denote positive subsets matched to image \(I_{i}\) and text \(T_{i}\). ITM loss is a binary classification loss based on the joint representation of visual and linguistic features. Compared with ITC loss, ITM loss does not maximize the distance between negative pairs.

### Granularity influence of VisualGPTScore.

To explore the influence of granularity of references in the visio-linguistic compositional reasoning, we leverage a language model to enrich the object details and relational phrases for short references in Winoground dataset, where all references are fluent and reasonable. Vicuna-13B-v1.52 is adopted as the LLM, which is instruction-following tuned based on LLaMA 2 Touvron et al. (2023b), one of the strongest open-source LLMs currently. Note that we artificially filter out nonsensical and unrelated expanded captions that are not relevant to the image and keep 282 of 400 image-text pairs finally. The expandation of references is shown in Fig. 7.

Footnote 2: [https://huggingface.co/lmsys/vicuna-13b-v1.5/tree/main](https://huggingface.co/lmsys/vicuna-13b-v1.5/tree/main)

We present the results in Table 3, and observe tha

Figure 6: An data example in current benchmarks. The image, positive and negative references are from Winoground Thrush et al. (2022).

Figure 7: An LLM is leveraged to fine-grain the references.

 

has been largely improved, indicating the fine-grained references are beneficial for text-to-image retrieval based on the definition of "Image Score" in Winoground.

### Zero-shot answer generation

Unlike EVLMs, GVLMs excel in zero-shot generation when guided by instructions, prompts, or demonstrations. We attempt to prompt and demonstrate the LLaVA and MiniGPT-4 to output the choices of positive or negative reference sentences based on corresponding images. However, we do not consider zero-shot generation of answers in our paper with two reasons. First, zero-shot answer generation cannot reflect the GVLMs' compositional understanding quantitatively, without scores or probabilities to show the confidence of judgements.

Second, when demonstrating the GVLMs to generate the option number of reference sentences directly, it is hard to acquire the direct answer due to the free-form answer format, especially considering the emergent capability is limited in relatively small-scaled GVLMs. In a limited number of instances, we observed successful model outputs where options or inference processes were accurately provided, resembling the blue line in Table 4. However, in the majority of cases, the GVLMs generated fabricated answers that were characterized by a rhetorical tone, similar to the examples shown in Table 4. Also, there are cases that the rationales are correct, but the option number is wrong, conflicting with the reasoning process of GVLMs (shown in orange line in Table 4). Hence, assessing the compositionality of GVLMs solely through direct zero-shot answer generation becomes challenging, particularly when the zero-shot capability is constrained within a relatively small-scale model like the 7B variant. Furthermore, it is not possible to quantitatively analyze the alignment of a single image-text pair using this type of evaluation method.

* Ducks are known to be efficient at swimming, and they spend a significant amount of time in the water. In this image, a duck is swimming on the water while facing the camera. The option B is most similar to the image since it describes a duck walking or swimming in the water.
* Which of the two options provides a better visual representation of the actual image?
* Which character should I output?
* Which one is most similar to the image?
* The girl is standing to the left of the giraffe, and they seem to be in a natural environment. The girl is feeding the giraffe from her hand, and the giraffe has its tongue out, reaching towards her hand. The answer is more likely to be B.

### Examples of content challenge of SADE

We present some examples of items in the Content challenge branch in our SADE benchmark in Fig. 8. Each item comprises one positive reference sentence and two negative ones. The red texts are positive reference sentences that only kept visual content-related phrases, while the black texts are negative reference sentences that were extracted randomly from other datasets. The negative reference sentences are fluent, coherent and meaningful, but irrelevant to the contents of the images.

The pure content understanding is challenging. Specifically, the intrinsic inclination of GVLMs towards syntactic correctness drives the GVLMs to prefer negative reference sentences. From the perspective of our proposed SyntaxBias Score, the bias of our Content Challenge is opposite to the current benchmarks, which is biased to the negative reference sentences in syntax. Therefore, GVLMs have to overcome the negative bias in syntax and show the robustness of visual understanding.

\begin{table}
\begin{tabular}{c c c} \hline \hline Models\&References & Text Score \\ \hline LLAvA+Original & 12.06 & 12.77 & 7.45 \\ LLAvA+Engrand & 8.51(3.59) & 37.23(45.40) & 6.38(1.07) \\ MinGPT-4M Original & 18.44 & 17.02 & 9.22 \\ MiniGPT+4Fine-grained & 6.03(12.41) & 31.91(+14.89) & 4.96(4.26) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy of LLaVA and MiniGPT-4 on original and fine-grained references of filtered Winoground dataset. The definitions of Text Score, Image Score, and Group Score is specified in Winoground [2].

 

* [150] M. C. B. Krizhevsky, A. Vedaldi, and R. Fergus. Visualizing the image with a single-scale visual network. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 2387-2396, 2014.
* [151] J. L. Friedman, S. G. Hinton, D. P. Salakhutdinov, and I. Sutskever. Learning to recognize images using convolutional neural networks. _CoRR_, abs/1412.0558v1, 2014.
* [152] H. E. Hinton, D. P. Kingma, and Z. Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6150_, 2014.
* [153] T. Warden, N. Bengio, Y. Bengio, and J. Dean. Dropout: A deep learning approach for reducing data dimensionality. _Journal of Machine Learning Research_, 20(1):1-10, 2015.

[MISSING_PAGE_POST]