Venue: NAACL 2024

# What Are We Measuring When We Evaluate Large Vision-Language

Models? An Analysis of Latent Factors and Biases

Anthony Meng Huat Tiong\({}^{1}\), Junqi Zhao\({}^{1}\) , Boyang Li\({}^{\spadesuit}\), Junnan Li, Steven C.H. Hoi\({}^{2}\), and Caiming Xiong\({}^{3}\)

\({}^{1}\)Nanyang Technological University  \({}^{2}\)Singapore Management University  \({}^{3}\)Salesforce Research

{anthonym@01, junqi.zhao, boyang.li}@ntu.edu.sg

{junnan4926, stevenhoij}@gmail.com cxiong@salesforce.com

Equal contributionsCorresponding Author.[https://github.com/jq-zh/olive-dataset](https://github.com/jq-zh/olive-dataset)

###### Abstract

Vision-language (VL) models, pretrained on colossal image-text datasets, have attained broad VL competence that is difficult to evaluate. A common belief is that a small number of VL skills underlie the variety of VL tests. In this paper, we perform a large-scale transfer learning experiment aimed at discovering latent VL skills from data. We reveal interesting characteristics that have important implications for test suite design. First, generation tasks suffer from a length bias, suggesting benchmarks should balance tasks with varying output lengths. Second, we demonstrate that factor analysis successfully identifies reasonable yet surprising VL skill factors, suggesting benchmarks could leverage similar analyses for task selection. Finally, we present a new dataset, OLIVE+, which simulates user instructions in the wild and presents challenges dissimilar to all datasets we tested. Our findings contribute to the design of balanced and broad-coverage vision-language evaluation methods.

Footnote †: Equal contributions. [topsep=0pt] \({}^{*}\)Equall contributions. [topsep=0pt] \({}^{*}\)Corresponding Author.

## 1 Introduction

Benefiting from enormous training data, large model sizes, and pretrained large language models, the current generation of vision-language models (VLMs) _(e.g.,_ Dai et al.2023; Zhu et al.2023; Liu et al.2023c; Ye et al.2023; Li et al.2023a; Awadalla et al.2023)_demonstrate competence in a wide range of tasks, including visual question-answering, optical character recognition, and spatial relation recognition. However, their broad competence pose a new challenge to the design of evaluation benchmarks, as many previous work focuses on only assessing one or a few capabilities, relying on data from a single distribution and annotation pipeline. This approach can result in test data that fails to fully represent all potential user inputs, leading to discrepancies between benchmark scores and real-world user experiences.

A prevailing evaluation strategy involves testing across a range of tasks and computing an average score Bitton et al. (2023); Xu et al. (2023); Liu et al. (2023d); Yu et al. (2023); Li et al. (2023d); Fu et al. (2023). This type of benchmarks usually involves a manual categorization of test tasks, as a benchmark that covers more categories is often considered more comprehensive and capable of measuring broad competence. For example, TouchStone Bai et al. (2023b) sorts tasks into five skills, ranging from visual recognition to visual storytelling. However, such categorizations are based on human intuition and lack support from empirical evidence.

In this paper, we promote an alternative approach that identifies vision-language (VL) capabilities underlying various tests directly from data. Inspired by the distributional hypothesis Firth (1957), we characterize test tasks using neighborhood structures inferred from transfer learning. That is, transfer learning between datasets that follow similar distributions and require similar VL capabilities will likely yield high performance. By analyzing transfer performance between a large number of source and target tasks, we can observe dataset similarity, infer shared VL capabilities, and gain insights into the VL benchmarks.

Specifically, we fine-tune four popular VLMs with different strengths, BLIP-2 Li et al. (2023c), Mini-GPT4 Zhu et al. (2023), LLaVA Liu et al. (2023c), and mPLUG-owl Ye et al. (2023), on 23 training (source) tasks and evaluate them on 29 test (target) tasks. Together with the original model performance before any finentuning (referred to as zero-shot performance), we obtain 2,784 performance measurements. After that, we examine the patterns and conduct Exploratory Factor Analysis, discovering six interpretable latent factors underlying the measurements.

The analyses reveal a few surprising findings.



First, we find that a surface-form property, the average output length, has a surprisingly strong influence on transfer performance. This suggests current evaluation results may be affected by this length bias. Second, factor analysis is capable of discovering unexpected yet reasonable factors that explain model performance. For example, we identify factors that differentiate reading text off images from multi-hop reasoning. These findings have important implications for the design of unbiased and comprehensive VL benchmarks.

Finally, to simulate real-world user instructions, we present a new vision-language dataset, Open-world Language Instruction for Visual-language Evaluation (OLIVE). OLIVE consists of 9,450 images, 30,120 unique instructions, and 47,250 responses. Empirically, we show that OLIVE has a transfer profile distinct from all other datasets we tested and hence provides a test complementary to existing tasks.

We summarize our contributions as follows:

* We promote the approach of discovering VL skills from data and demonstrate that factor analysis is a robust and effective technique for this purpose. Our large-scale experiments lead to findings that can inform the future design of VLM test suites.
* We introduce a new benchmark, OLIVE, to evaluate open-ended model responses to diverse instructions.

## 2 Analysis Techniques

The transfer performance from \(N\) source (training) tasks to \(K\) target (test) tasks on model \(m\) is stored as a matrix \(A^{(m)}\in\mathbb{R}^{N\times K}\). The performance numbers of different tasks cannot be compared directly due to differences in the scales of the evaluation metrics. Therefore, we first normalize the data. Subsequently, we apply Singular Value Decomposition and Factor Analysis. Both techniques can be understood as decompositions of the matrices \(A^{(m)}\), albeit with different mathematical formulations.

### Normalization

We obtain the raw performance number \(b_{i,j}^{(m)}\) when we train model \(m\) on the \(i^{\text{th}}\) source task and test on the \(j^{\text{th}}\) target task. The normalized performance \(a_{i,j}^{(m)}\) is obtained as

\[a_{i,j}^{(m)}=(b_{i,j}^{(m)}-b_{0,j}^{(m)})/(\max_{j^{\prime}}b_{i,j^{\prime} }^{(m)}-b_{0,j}^{(m)}), \tag{1}\]

where \(b_{0,j}^{(m)}\) denotes the performance of the pre-trained model \(m\) on target task \(j\) without finetuning on any source task, referred to as the zero-shot performance. If finetuning on source \(i\) improves over the zero-shot performance, \(a_{i,j}^{(m)}\) is a positive number. Conversely, if there is negative transfer from source \(i\), \(a_{i,j}^{(m)}\) is negative. The best source task, which in most cases is the in-domain i.i.d. training task, has \(a_{i,j}^{(m)}=1\). Hence, this normalization separates positive from negative transfers and shows how source tasks perform relative to the in-domain training data.The matrix \(A^{(m)}\) has \(a_{i,j}^{(m)}\) as its components. After separate normalizations, we concatenate the four matrices \(A^{(m)}\), corresponding to the four models we fine-tune, along the source-task dimension (the rows) and obtain the aggregate performance matrix \(A\in\mathbb{R}^{4N\times K}\).

### Singular Value Decomposition

Singular Value Decomposition (SVD) is a classic technique for learning distributed representations. Levy and Goldberg (2014) show that SVD produces word embeddings comparable to word2vec Mikolov et al. (2013). The SVD of matrix \(A\) can be written as

\[A=U\Sigma V^{\top} \tag{2}\]

We perform truncated SVD using the \(D\) largest singular values. After that, we use \(V\Sigma^{1/2}\in\mathbb{R}^{K\times D}\) as the features of the target tasks.

### Factor Analysis

It is a widely held belief that a small number of factors, known as cognitive abilities, underlie human performance on numerous mental activities Horn and McArdle (2007). To uncover these latent factors, Spearman (1904) developed the statistical technique of Factor Analysis. For modern treatments, we refer readers to Gorsuch (2014) and Barber (2012).

In this paper, we start with a premise similar to Spearman's, that a small number of VL capabilities are responsible for VLM performance on various test tasks. This belief is, in fact, implicitly shared, though often not explicitly stated, by most recent VLM evaluation papers (_e.g._, Bitton et al., 2023; Xu et al., 2023; Liu et al., 2023; Bai et al., 2023) that attempt to categorize VLM test scenarios based on intuitive justifications. In contrast, we apply Exploratory Factor Analysis (EFA) to uncover these factors from empirical data 

We assume that each source task imparts specific latent skills to a model. These skills, while not directly observable, manifest through the model's performance on related target tasks. Performances on target tasks that share the same skills tend to increase or decrease in a correlated manner. This pattern of correlation underpins our expectation that EFA will successfully unveil the underlying skills.

Mathematically, we treat the \(i^{\text{th}}\) column of \(A\), \(\mathbf{a}_{i}\in\mathbb{R}^{4N}\), as the characteristic of the target task \(i\), which we attempt to explain with \(L\) latent factors:

\[\mathbf{a}_{i}=W\mathbf{h}_{i}+\mathbf{\mu}+\mathbf{\epsilon}, \tag{3}\]

where \(W\in\mathbb{R}^{4N\times L}\) reflects how source tasks load to the \(L\) latent factors and \(\mathbf{h}_{i}\in\mathbb{R}^{L}\) reflects how target task \(i\) decomposes into the latent factors. \(\mathbf{\mu}\) is the average vector across target tasks and \(\mathbf{\epsilon}\) represents Gaussian noise. EFA differs from PCA in that it assumes the covariance of \(\mathbf{\epsilon}\) is diagonal rather than spherical. Note that the above formulation is invariant to a rotational matrix \(R\), as \(W\mathbf{h}_{i}=(WR)(R^{\top}\mathbf{h}_{i})\). We apply the Varimax rotation Kaiser (1958) to find \(R\) so that \(\mathbf{h}_{i}\) is as concentrated on as few factors as possible.

Our preliminary analysis suggests that the captioning and VQA tasks are highly correlated and predominantly load onto a single factor, likely indicative of a general VL capability. To isolate and examine other factors, we employ linear regression to control for the influence of the dominant factor. Specifically, we first perform EFA with one factor, so that \(W\) becomes a \(4N\)-by-\(1\) vector \(\mathbf{w}\). We then perform regression from \(\mathbf{w}\) to \(A\) by solving the following problem:

\[\text{minimize}~{}~{}\left\|A-\mathbf{w}\mathbf{\beta}^{\top}-\mathbf{\gamma}\mathbf{1}^{ \top}\right\|_{F}^{2}, \tag{4}\]

where \(\mathbf{\beta},\mathbf{\gamma}\in\mathbb{R}^{4N}\) are trainable parameters. Afterwards, we conduct EFA on the residuals, \(\bar{A}=A-\mathbf{w}\mathbf{\beta}^{\top}-\mathbf{\gamma}\mathbf{1}^{\top}\), which contain information about other factors indicative of more specific VL capabilities than the first factor.

We employ both parallel analysis and Velicer's Minimum Average Partial (MAP) test to determine the optimal number of factors to extract. Parallel analysis compares the eigenvalues from our sample correlation matrix against those from random data of the same size, identifying factors that explain more variance than expected by chance. Conversely, Velicer's MAP test evaluates the average squared partial correlation for each possible number of factors, pinpointing the point at which additional factors no longer meaningfully increase the explanation of variance. Both methods converge on the decision to extract six factors.

## 3 Source and Target Tasks

We gather 27 publicly available VL datasets and create variations, yielding 23 source tasks and 29 target tasks. We show the full list of tasks in Tab. 1 and describe them below. The performance metrics used are AUC for Hateful Memes, CIDEr Vedantam et al. (2015) for OpenCQA, OLIVE, and all captioning datasets, ROUGE-L Lin (2004) for MORE and New Yorker Explanation, and accuracy for the remaining tasks. To focus on end-to-end per

\begin{table}
\begin{tabular}{c c c c} \hline Intuitive Category & Task & Source & Target \\ \hline  & COCO Caption & ✓ & ✓ \\ Image & Flickr30k & ✓ & ✓ \\ Captioning & Web CapFilt & ✓ & ✗ \\  & TextCaps & ✓ & ✓ \\ \hline Generic VQA & VQAv2 & G & G, MC \\ \hline Knowledge-based VQA & OK-VQA & G & G, MC \\  & A-OKVQA & G, MC & G, MC \\  & ScienceQA & MC & MC \\ \hline OCR YQA & TextVQA & G & G, MC \\  & OCR-VQA & G & G, MC \\ \hline  & GOA & G & G, MC \\ Visual & VSR & MC & MC \\ Reasoning & IconQA & MC & MC \\  & CLEVR & ✗ & G, MC \\  & RAVEN-FAIR & ✗ & MC \\ \hline Classification & Hateful Memes & MC & MC \\ \hline  & New Yorker & & ✓ \\  & Ranking & & ✓ \\ Humor \& New Yorker & & ✓ \\ Sarcasm & Explanation & ✗ & ✓ \\  & MORE & ✗ & ✓ \\ \hline Chart & OpenCQA & G & G \\ Reading & ChartQA & ✗ & G, MC \\ \hline  & OLIVE (Ours) & ✓ & ✓ \\  & LLaVA Condensation & ✓ & ✗ \\ Open-ended & version & ✓ & ✗ \\ Generation & LLaVA Reasoning & ✓ & ✗ \\  & LLaVA Description & ✓ & ✗ \\ \hline Question & VQav2 QG & ✓ & ✗ \\ Generation & OK-VQA QG & ✓ & ✗ \\  & A-OKVQA QG & ✓ & ✗ \\ \hline \hline \end{tabular}
\end{table}
Table 1: The list of source and target tasks used in experiments. G and MC indicate the generative and multiple-choice versions of the VQA tasks respectively.

 

formance, we do not perform any separate optical character recognition.

**Image Captioning.** Image captioning is one of the most popular image-text tasks and is commonly used as a pretraining task for VLMs (Chen et al., 2022; Tiong et al., 2022). Here we select two classic datasets: COCO Caption (Lin et al., 2014) and Flickr30k (Young et al., 2014). In addition, we include TextCaps (Sidorov et al., 2020), which involves the description of textual content in images. We also include as a source task Web CapFilt, a set of synthetic image captions covering a wide variety of web images. Web CapFilt was generated by BLIP (Li et al., 2022b) for self-training. We hypothesize that its diversity could be beneficial in transfer learning.

**Visual Question-answering (VQA).** VQA is another very popular image-text task due to the versatility of the question-answering format. VQAv2 (Goyal et al., 2017) is probably the most prominent VQA benchmark, featuring more than 200,000 COCO images and 1 million questions. Other variations include knowledge-grounded VQA, OCR VQA, Chart VQA, and so forth, which we discuss below.

Performance measurement in VQA can be challenging, as there are often many correct answers to the same question. As a remedy, we create two target tasks for every VQA dataset. The first is the generative (G) version, which considers an answer to be correct only when it matches exactly one of the ground-truth answers. The second is the multiple-choice (MC) version, where the model chooses one from five options. To convert a generative VQA dataset to the MC version, we create five options for every question, including at most two correct answers to account for their linguistic variations. After that, we add incorrect choices by sampling answers from other questions and selecting those with top-k probabilities according to InvtractBLIP (Dai et al., 2023). During inference, we feed all options to the model and choose the one with the highest average word probability as the model prediction.

**Knowledge-grounded VQA.** These tasks require the model to apply world knowledge not present in the input to answer questions. ScienceQA (Lu et al., 2022) focuses on the contents of science textbooks. OK-VQA (Marino et al., 2019) is primarily about visual recognition and knowledge recall, whereas A-OKVQA (Schwenk et al., 2022) often requires an additional step of reasoning.

**OCR VQA.** TextVQA (Singh et al., 2019) and OCR-VQA (Mishra et al., 2019) are two VQA datasets that require recognition of text in images. OCR-VQA focuses on reading text from book covers, whereas TextVQA often requires locating an object before reading the text on it.

**Chart Reading.** OpenCQA (Kantharaj et al., 2022) and ChartQA (Masry et al., 2022) feature questions about the content of diagrams and charts. OpenCQA expects descriptive, long-form answers, whereas ChartQA is mainly concerned with data extraction and comparison using short answers.

**Visual Reasoning.** The term "reasoning" is used broadly in the VLM literature. It sometimes refers to shallow tasks like counting (_e.g., how many apples are in the image?_), spatial relations and grounding (_e.g., what is adjacent to the cylinder?_), but it can also include logical or algebraic operations. In this category, we include five datasets, GQA Hudson and Manning (2019), VSR Liu et al. (2023a), CLEVR Johnson et al. (2017), IconQA Lu et al. (2021), RAVEN-FAIR (Benny et al., 2021). GQA and VSR mainly contain natural images, whereas IconQA features cartoons, and RAVEN-FAIR comprises abstract diagrams. CLEVR consists of synthetically rendered images of 3D objects. Among these, RAVEN-FAIR is the most challenging, as it is derived from the Raven's test (Raven, 1938), an intelligence test designed for humans that requires complex reasoning.

**Image Classification.** Hateful Memes (Kiela et al., 2020) is a binary classification task that distinguishes hateful memes from other meme images.

**Humor and Sarcasm Understanding.** Hessel et al. (2023) show that VLMs perform poorly at understanding humor. From their paper, we adopt New Yorker Ranking, which involves selecting the best humorous caption for a cartoon from the New Yorker magazine, and New Yorker Explanation, which asks the model to explain why the cartoon and its caption invoke humor. The MORE dataset (Desai et al., 2022) involves explaining why a textual statement associated with a natural image is sarcastic.

**Question and Open-ended Generation.** We adapt three datasets, VQAv2, OK-VQA, A-OKVQA, for the task of question generation from an image and an answer. Further, we take the three subsets of LLaVA-Instruct-150K (Liu et al., 2023c), which 

respectively focus on free-form conversation, detailed description, and reasoning. We use these as source tasks but not target tasks, since these artificial data may not be representative of real use cases.

**OLIVE.** Additionally, we include a new dataset, OLIVE, a highly diverse, human-corrected multimodal dataset, which we create to simulate in-the-wild user queries to VLMs. Once a VLM is publicly released, it tends to receive substantially more diverse and idiosyncratic inputs than the available academic datasets. OLIVE is our attempt to simulate such user queries, so that we may train and evaluate the VLMs under conditions similar to their ultimate use case.

We briefly describe the data curation process here and refer readers to the Appendices A and B for more details. First, we randomly sample 9,450 images from LAION-Aesthetics (Schuhmann et al., 2022), consisting of diverse web images. We take the original LAION caption from the dataset and a few generated image captions from BLIP-2 as the complete image description. Next, we feed customized prompts to ChatGPT (OpenAI, 2023a) to generate instructions conditioned on the image description and five responses to each instruction. Thereafter, a team of data annotators is recruited to verify and refine the data in order to remove errors, shortcut biases, and harmful content. This procedure yields 30,120 unique instructions and 47,250 responses. We use 6,750 instruction-response pairs for training and another 6,750 for validation, and leave the rest for test. Fig. 1 shows three examples that require both detailed visual understanding and extensive commonsense knowledge.

## 4 Experiments

### Setup

Considering that different VLMs may exhibit different training behaviors, our analysis uses four popular VLMs: BLIP-2 (Li et al., 2023c), MiniGPT-4 (Zhu et al., 2023), LLaVA (Liu et al., 2023c), and mPLUG-Owl (Ye et al., 2023), which have mostly not been exposed to the datasets in focus. As minor exceptions, BLIP-2 and MiniGPT-4 were pretrained on COCO Caption and Web CapFilt. mPLUG-Owl has been exposed to COCO Caption, and LLaVA was pretrained on the three LLaVA datasets. We avoid models that have been finetuned on many VQA datasets, such as InstructBLIP (Dai et al., 2023), LLaVA 1.5 (Liu et al., 2023b), and Qwen-VL (Bai et al., 2023a).

For each model, we fine-tune the parameters that are trainable during their respective vision-language pretraining. On each source task, we train for 10K steps with a batch size of 192 for BLIP-2 and 128 for MiniGPT-4, mPLUG-Owl, and LLaVA. Other model details and hyperparameters are in Appendices C and D.

Figure 1: Examples of the OLIVE benchmark for different categories. From left to right: visual recognition, knowledge-based QA, and creative writing.

 



### Results

We defer the transfer performance tables to Appendix E due to space constraints. With these results, we first examine the transfer learning power of the source tasks. Within every model, we rank the source tasks by the sum of their normalized performance across all target tasks. After that, for each source task, we compute the harmonic mean of its rankings across all models. The results are in Tab. 3. A-OKVQA (MC), VQAv2, ScienceQA, A-OKVQA and OCR-VQA hold the top-5 positions.

In addition, we examine the effects of output lengths. We partition the source and target tasks into three mutually exclusive and collectively exhaustive groups according to their average output lengths: 1-3 words, 6-12 words, and more than 40 words. We show the average normalized transfer performance across these groups and the top-5 best source tasks for each group in Tab. 2.

Next, we investigate the similarity of target tasks. We perform truncated SVD on \(A\) with the first \(D=8\) singular values. After that, we compute the cosine similarity between target tasks and visualize the results in Fig. 2. With a mean similarity of \(-0.06\), OLIVE is the third least similar to other target tasks (see details in Appendix F).

Finally, we run EFA on the residual matrix, \(\bar{A}\), and present the outcomes in Fig. 3. We plot the most significant factor loading for each target task, retaining only those that exceed an absolute value threshold of 0.3. Loadings close to 1 or -1 signify strong influences of a factor on a target task. Three tasks, New Yorker Explanation and Ranking, and Hateful Memes, do not have loadings more than 0.3 on any discovered factor, suggesting they do not share with the other tasks some VL capabilities that can be discovered by EFA. The full results are available in Appendix H.

### Discussion

In this section, we highlight important findings from our experiments.

**The Output Length Bias.** Tab. 2 demonstrates that transfer performance is strongly influenced by the output lengths. In the top section, mismatch between the output lengths results in significant performance drops. In the bottom section, the best source tasks almost always have similar output lengths to the target tasks. This surprising finding shows that output length may be a shortcut feature for VLMs, suggesting that future test suites need a balance among tasks with different output lengths.

**EFA Overview.** The six EFA factors resemble hierarchical clusters from SVD features (Appendix G). For instance, both techniques identify a factor (Factor 1) or a cluster around the three captioning tasks, COCO, Flickr30k, and TextCaps. However, EFA offers insights into both the positive and negative ends of the same factor, rendering the factors more interpretable. EFA also picks up more VL skills than SVD, such as text reading versus reasoning.

It is notable that EFA is influenced by the length bias. For example, it fails to find a humor factor shared by the two New Yorker tasks. This omission occurs because the similarity between the two tasks is masked by their differences in output lengths and MC-vs-G evaluation, leading to markedly different transfer profiles in Fig. 2. Nonetheless, EFA successfully identifies several reasonable factors, and we discuss Factors 2-6 below.

**Factors 2 and 3: Generative vs. MC Evalua

\begin{table}
\begin{tabular}{c|c c c} \hline \multirow{2}{*}{Source Task} & \multicolumn{3}{c}{Target Task Output Length} \\ \cline{2-4}  & 1-3 & 6-12 & \textgreater{}40 \\ \hline
1-3 & **0.03/ 1.00** & -0.78/ 0.79 & -0.85/ 0.44 \\
6-12 & -0.43/ 0.54 & -0.43/ **0.75** & **-0.43**/ 0.48 \\ \textgreater{}40 & -0.90/ 0.43 & -0.87/ 0.28 & **-0.26**/ **0.55** \\ \hline \end{tabular} \begin{tabular}{l||c c c} \hline \multirow{2}{*}{Source Data} & \multicolumn{3}{c}{YAGv2 (1)} & Web Capfa (12) & LLVA Conversetion (48) \\ \cline{2-4}  & \multirow{2}{*}{A-OKVQA (MC)} & \multirow{2}{*}{OCO Caption (10)} & OpenCapQA (56) \\ \cline{2-4}  & & & \\ \cline{2-4}  & & & \\ \hline \end{tabular} \begin{tabular}{l||c c} \hline \multirow{2}{*}{Source Data} & \multicolumn{3}{c}{LLAVA Conversetion (48)} \\ \cline{2-4}  & \multirow{2}{*}{A-OKVQA (MC)} & \multirow{2}{*}{OCO CorbQA (10)} & OpenCapQA (56) \\ \cline{2-4}  & & & \\ \cline{2-4}  & & & \\ \hline \end{tabular} 
\begin{tabular}{l||c c} \hline \multirow{2}{*}{Source Data} & \multicolumn{3}{c}{TextCaps (12)} \\ \cline{2-4}  & \multirow{2}{*}{Fixx30K (12)} & \multirow{2}{*}{Fixx30K (12)} & FLixx30K (12) \\ \cline{2-4}  & & & \\ \hline \end{tabular}
\end{table}
Table 2: Mean normalized performance by the mean output lengths of source and target tasks. In-domain sources are excluded. In the top section, we report the mean normalized performance across all (left) and top-5 (right) source tasks.

Figure 2: Cosine similarity between target tasks computed using SVD features.



tion.** Factors 2 and 3 capture the contrast between generative and MC evaluation of VQA. Generative VQAs exhibit positive loadings on both factors while MC VQAs exhibit negative loadings. Furthermore, specialized VQAs requiring OCR skills, such as TextVQA and ChartQA, load positively on Factor 2 and negatively on Factor 3, whereas generic VQAs, such as VQAv2 and GQA, exhibit the opposite pattern.

We identify two reasons for the differences between generative and MC evaluations. First, generative evaluation requires an exact match with at least one ground-truth answer, which can result in false negatives for valid answers phrased differently. In contrast, MC evaluation compares average word probabilities and does not mandate strict word matching. Second, the exact match requirement in generative evaluation makes it more sensitive to the output lengths of source tasks.

Nevertheless, when we analyze factors internal to the generative and MC tasks (Fig. 4), we observe very similar structures. In both groups, we observe a knowledge-based QA factor, which includes OKVQA and A-OKVQA, a OCR-related factor, which includes OCR-VQA and ChartQA, and a generic or spatial relation factor, which includes GQA and VQAv2. This underscores the robustness of EFA in identifying underlying structures when provided with appropriate input data.

Figure 4: EFA results when we extract 3 factors from the 7 generative VQA tasks and the 7 MC VQA tasks separately. We merge the results for display. Cut-off for factor loadings = 0.6.

\begin{table}
\begin{tabular}{l r} \hline Source Task & Harmonic Mean Score \\ \hline A-OKVQA (MC) & 1.3 \\ VQAv2 (G) & 1.3 \\ ScienceQA (MC) & 3.8 \\ A-OKVQA (G) & 4.6 \\ OCR-VQA (G) & 6.0 \\ GQA (G) & 6.2 \\ Flickr30k (G) & 7.2 \\ OK-VQA (G) & 7.8 \\ Web Capfit (G) & 7.9 \\ IconQA (MC) & 8.4 \\ OpenCQA (G) & 9.5 \\ TextVQA (G) & 9.5 \\ VSR (MC) & 10.0 \\ Haterial Memes (MC) & 11.7 \\ COCO Caption (G) & 13.3 \\ TextCaps (G) & 15.0 \\ VQAv2 QG (G) & 15.9 \\ LLVA Conversation (G) & 17.6 \\ OLIVE (G) & 17.8 \\ A-OKVQA QG (G) & 17.9 \\ OK-VQA QG (G) & 19.7 \\ LLaVA Reasoning (G) & 21.1 \\ LLaVA Description (G) & 22.5 \\ \end{tabular}
\end{table}
Table 3: Harmonic mean ranking score for source tasks.

Figure 3: Results of EFA on the residuals \(\bar{A}\). Black arrows indicate positive loadings; red arrows indicate negative loadings. Cut-off for factor loadings = 0.3.

 



**Factors 4 & 5: Text Reading vs Reasoning.** Factors 4 and 5 distinguish between tasks that involve simple text extraction from those that require deeper and multi-hop reasoning. OCR-VQA and ChartQA, which primarily focus on reading text and numbers from images, exhibit loadings opposite to RAVEN-FAIR, ScienceQA, and IconQA, which demand strong logical reasoning skills, and opposite to A-OKVQA, MORE, and OpenCQA, which necessitate the use of external knowledge and contextual understanding. The fact that EFA can find these reasonable skills illustrates its power.

**Factor 6: Spatial Reasoning.** Factor 6 is characterized by spatial reasoning, as CLEVR and VSR are both designed for this purpose. Notably, while OLIVE shows the highest loading on Factor 6, its communality (overall variance explained) is only 0.4. The remaining variance in OLIVE is not explained by the factors identified in our analysis. This implies that although OLIVE requires spatial reasoning skills, these skills only account for a small portion of skills required by OLIVE.

**A-OKVQA and VQAv2 are effective source tasks.** These two are among the sources with the highest transfer performance (Fig. 3). They not only transfer well to VQA tasks but also to other complex tasks, such as New Yorker Ranking. We hypothesize that the large and diverse data in VQAv2 contributes to its strong transferability. Interestingly, even though A-OKVQA is 24 times smaller than VQAv2, it still transfers well. We hypothesize that the main skill that A-OKVQA requires, knowledge-enabled reasoning, is an important skill for VL competence. In contrast, OKVQA, which is designed for knowledge recall, is not as beneficial to the target tasks.

**Humor, sarcasm, and abstract reasoning remain difficult.** All models we tested struggle to understand humor and sarcasm, as captured by the New Yorker datasets and MORE. The models also perform barely above chance level on RAVEN-FAIR, an abstract reasoning task. Surprisingly, EFA is able to correctly place RAVEN-FAIR under the reasoning factor (negative Factor 4), despite the tiny variance caused by overall poor performance.

**Robustness and Inter-model Similarity.** We empirically verify the robustness of EFA using a leave-one-out analysis. We partition the aggregate performance matrix \(A\) into a training set of data from three models, \(A_{train}\in\mathbb{R}^{3N\times K}\), and a test set of data from the remaining model, \(A_{test}\in\mathbb{R}^{N\times K}\). Similar to before, we first perform a one-factor EFA on \(A_{train}\) and calculate the residuals \(\bar{A}_{train}\) and \(\bar{A}_{test}\). Then, we perform a six-factor EFA, which decomposes \(\bar{A}_{train}\) to the factor weights \(W_{train}\), the factors \(H_{train}\), and a mean vector \(\mu_{train}\),

\[\bar{A}_{train}=W_{train}H_{train}^{\top}+\mu_{train}+\epsilon_{train}. \tag{5}\]

After that, we use the identified \(H_{train}\) and \(\mu_{train}\) to reconstruct \(\bar{A}_{test}\) and calculate the L2-norm of the reconstruction error \(\epsilon_{test}\), which measures the generalization of the factor models,

\[\bar{A}_{test}=W_{test}H_{train}^{\top}+\mu_{train}+\epsilon_{test}. \tag{6}\]

We repeat the experiment five times, with different combinations of VLMs in \(A_{train}\) and \(A_{test}\). The results are presented in Tab. 4.

We observe that the factors can generalize to mPLUG-Owl and MiniGPT-4 fairly well, but not so well to BLIP-2 and LLaVA. However, if we remove BLIP-2 from the training set, as shown by the last row of Tab. 4, the factors would generalize to LLaVA. We attribute this observation to the fact that BLIP-2 is more thoroughly trained than the other three models. In particular, its average zero-shot performance across all target tasks is 47.6. LLaVA, mPLUG-Owl, and MiniGPT-4 score 29.5, 23.4, and 22.6, respectively, suggesting these three networks are much more similar to each other. Hence, we argue that the observed generalization among mPLUG-Owl, MiniGPT-4, and LLaVA indicates that EFA is robust.

**Implications of Factor Analysis.** Our findings have the following implications for the design of VL benchmarks. Firstly, to prevent shortcut learning and giving unfair advantages to any source training tasks, VL benchmarks should contain tasks of different output lengths and include both generative and MC evaluation. Secondly, instead of intuition-based categories, VL benchmarks may group tasks based on statistically discovered VL factors, and score VLMs accordingly.

\begin{table}
\begin{tabular}{c c c} \hline \multicolumn{1}{l}{VLMs in \(A_{train}\)} & \multicolumn{1}{r}{VLM in \(A_{test}\)} & \multicolumn{1}{r}{L2-norm Error} \\ \hline MiniGPT-4, mPLUog-Owl, LLaVA & BLIP-2 & 85.6 \\ BLIP-2, mPLUG-Owl, LLaVA & MiniGPT-4 & 14.2 \\ BLIP-2, LLaVA, MiMiGPT-4 & mPLUG-Owl & 16.3 \\ BiLP-2, MiniGPT-4, mPLUG-Owl & LLaVA & 63.0 \\ MiniGPT-4, mPLUG-Owl & LLaVA & 25.2 \\ \hline \end{tabular}
\end{table}
Table 4: Robustness analysis of EFA. The generalization among mPLUG-Owl, MiniGPT-4, and LLaVA indicates that EFA is robust.





### Analysis of OLIVE

**Uniqueness.** Intended to simulate real-world user instructions, OLIVE features a unique transfer profile and has very low cosine similarity with other tests (Fig. 2). Additionally, EFA only explains 0.4 of the total variance in OLIVE, indicating that the identified factors cannot fully explain model behaviors on OLIVE. These facts support our thesis that OLIVE measures an aspect of VL capabilities that few existing datasets do.

**Diversity.** We evaluate the linguistic and image and diversity of OLIVE, and compare them against those of existing datasets, such as VQAv2, GQA, and LLaVA (conversation, description and reasoning). Notably, both VQAv2 and GQA are large datasets with 658K and 956K data points respectively. LLaVA, similar to OLIVE, is generated using a GPT model, albeit without any human correction.

For linguistic diversity, we show the entropy of the multinomial word distributions in Tab. 5. OLIVE exhibits higher entropy, indicating a greater level of diversity compared to the other datasets.

For image diversity, we calculate the generalized variance for CLIP image features, which is the product of the top-k eigenvalues of the covariance matrix. This is akin to calculating the variance explained by the top-k principal components of PCA. We consider k values of 8 and 16. Tab. 6 shows that OLIVE achieves higher generalized variance than the other datasets, implying greater diversity.

## 5 Related Work

**Relationships among Tasks.** Relationships between machine learning tasks has been studied from the perspective of transfer learning and multi-task learning. A number of works focus on identifying transfer relationships from empirical data [37, 1, 1]. A typical strategy is to train a base network on multiple source tasks and test the resulting networks on target tasks. In multi-task learning, the research focuses on identifying optimal grouping of tasks that should be learned together to maximize synergy [23, 19, 18, 36, 35]. Different from the above, our work focuses on vision-language tasks and identifying latent factors and potential biases responsible for the observed performance.

**Broad-coverage Multimodal Test Suites.** As VLMs begin to excel on an ever growing set of tasks, the test suites have also grown in size. Early benchmarks contain only a few tasks. For example, Zhou et al. zhou2022 employs 4 tasks and Su et al. gui2021employs 8 tasks in 4 groups. More recent benchmarks [1, 19, 23, 24, 25, 26] uses increasingly more tasks in order to achieve broader coverage of VL capabilities, and group tasks into VL capabilities based on human intuition. For instance, Li et al. li2023learning categorizes tasks into 12 aspects focusing on spatial and temporal understanding. To the best of our knowledge, our work is the first data-driven approach to identify the VL capabilities.

## 6 Conclusions

In this work, we aim to empirically uncover latent factors and biases that contribute to the performance of various VLMs on VL tasks. Using EFA, we identify six highly interpretable factors and biases that affect performance. We also contribute a new dataset, OLIVE. We hope this research will lead to the creation of VL benchmarks with more balanced and complete coverage of VL capabilities.

## 7 Acknowledgments

Anthony Meng Huat Tiong is supported by Salesforce and Singapore Economic Development Board under the Industrial Postgraduate Programme. Junqi Zhao and Boyang Li are supported by the Nanyang Associate Professorship and the National Research Foundation Fellowship (NRF-NRFF13-2021-0006), Singapore. We thank Flitto+ for their support and expertise in OLIVE annotation.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Image Diversity & OLIVE & VQAv2 & GQA & LLaVA \\ \hline Number of images & 9 K & 123 K & 73 K & 81 K \\ \hline Generalized variance & k = 8 & **348.40** & 205.61 & 198.06 & 207.50 \\  & k = 16 & **84.28** & 66.85 & 68.41 & 70.07 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Image diversity. OLIVE has higher generalized variance than the other datasets.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Linguistic Diversity & OLIVE & VQAv2 & GQA & LLaVA \\ \hline Number of words & 4 M & 7 M & 9 M & 28 M \\ Word distribution entropy & **9.62** & 8.90 & 6.74 & 9.46 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Linguistic diversity. OLIVE has higher linguistic diversity than VQAv2, GQA and LLaVA.



Ethical Considerations

The potential risks of large language models have been discussed in literature, including Chung et al. (2022), Touvron et al. (2023), and Chiang et al. (2023). We neither propose nor release new models in this work. For easy replication, we use open-source visions-language models to analyze publicly available, academic datasets.

We consider issues of privacy, toxicity, and fair compensation in the production of OLIVE. We utilize images from LAION-aesthetics, whose privacy policy is in compliance with GDPR. For a given image, we generate an instruction and outputs using ChatGPT. To mitigate the potential issues of hallucination, toxicity, and harmful content in ChatGPT generated content, we hire an annotation company, Flitto, to thoroughly review and correct any errors presented in the data. We establish specific guidelines for annotations, prioritizing content that is free from harmful information. The identity of the annotators are anonymized to safeguard their privacy. We pay Flatito $3 USD per image to ensure fair compensation.

## 9 Limitations

We focus our study of transfer performance using only one source task instead of multi-task learning setting due to the computational constraint. Thus, we do not investigate the interaction of multiple source tasks on the target tasks. To assess the model transfer performance, our work requires a substantial amount of computation and scaling our approach to diverse models and datasets is inefficient.

## References

* Achille et al. (2019) Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. 2019. Task2vec: Task embedding for meta-learning. In _Proceedings of the IEEE/CVF international conference on computer vision_, pages 6430-6439.
* Achille et al. (2021) Alessandro Achille, Giovanni Paolini, Glen Mbeng, and Stefano Soatto. 2021. The information complexity of learning tasks, their structure and their distance. _Information and Inference: A Journal of the IMA_, 10(1):51-72.
* Awadalla et al. (2023) Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jemia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. _arXiv preprint arXiv:2308.01390_.
* Bai et al. (2023a) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023a. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. _arXiv preprint arXiv:2308.12966_.
* Bai et al. (2023b) Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. 2023b. Touchstone: Evaluating vision-language models by language models. _arXiv 2308.16890_.
* Barber (2012) David Barber. 2012. _Bayesian Reasoning and Machine Learning_. Cambridge University Press.
* Ben-David and Schuller (2003) Shai Ben-David and Reba Schuller. 2003. Exploiting task relatedness for multiple task learning. In _Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings_, pages 567-580. Springer.
* Benny et al. (2021) Yaniv Benny, Niv Pekar, and Lior Wolf. 2021. Scale-localized abstract reasoning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 12557-12565.
* Bitton et al. (2023) Yonatan Bitton, Hertik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. 2023. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. _arXiv 2308.06595_.
* Bugliarello et al. (2022) Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulic. 2022. Iglue: A benchmark for transfer learning across modalities, tasks, and languages. _arXiv 2201.11732_.
* Chen et al. (2022) Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. 2022. VisualGPT: Data-efficient adaptation of pretrained language models for image captioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 18030-18040.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Tianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_ 

* Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructible: Towards general-purpose vision-language models with instruction tuning. _arXiv preprint arXiv:2305.06500_.
* Desai et al. (2022) Poorav Desai, Tanmoy Chakraborty, and Md Shad Akhtar. 2022. Nice perfume. how long did you manipulate in it? multimodal sarcasm explanation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 36, pages 10563-10571.
* Dwivedi and Roig (2019) Kshtij Dwivedi and Gemma Roig. 2019. Representation similarity analysis for efficient task taxonomy & transfer learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 12387-12396.
* Fang et al. (2023) Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. Eva: Exploring the limits of masked visual representation learning at scale. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 19358-19369.
* Fifty et al. (2021) Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021. Efficiently identifying task groupings for multi-task learning. In _Advances in Neural Information Processing Systems_, volume 34, pages 27503-27516.
* Firth (1957) J.R. Firth. 1957. A synopsis of linguistic theory 1930-1955. _Studies in Linguistic Analysis_, page 1-32.
* Fu et al. (2023) Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. _arXiv 2306.13394_.
* Gorsuch (2014) Richard L Gorsuch. 2014. _Factor analysis: Classic edition_. Routledge.
* Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pages 6904-6913.
* Hessel et al. (2023) Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023. Do androids laugh at electric sheep? humor "understanding" benchmarks from the new yorker caption contest. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 688-714, Toronto, Canada. Association for Computational Linguistics.
* Horn and McArdle (2007) John L Horn and John J McArdle. 2007. _Understanding human intelligence since Spearman_, pages 205-247.
* Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Hudson and Manning (2019) Drew A. Hudson and Christopher D. Manning. 2019. QQA: A new dataset for real-world visual reasoning and compositional question answering. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_, pages 6700-6709. Computer Vision Foundation / IEEE.
* Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 2901-2910.
* Kaiser (1958) Henry F Kaiser. 1958. The varimax criterion for analytic rotation in factor analysis. _Psychometrika_, 23(3):187-200.
* Kantharaj et al. (2022) Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. 2022. OpenCQA: Open-ended question answering with charts. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 11817-11837, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Kiela et al. (2020) Douwe Kiela, Hamed Firooz, Arxivd Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. In _NeurIPS_.
* Kumar and Daume III (2012) Abihshek Kumar and Hal Daume III. 2012. Learning task grouping and overlap in multi-task learning. _arXiv preprint arXiv:1206.6417_.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In _Advances in Neural Information Processing Systems_, volume 27. Curran Associates, Inc.
* Li et al. (2023a) Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023a. Otter: A multi-modal model with in-context instruction tuning. _arXiv preprint arXiv:2305.03726_.
* Li et al. (2023b) Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023b. Seed-bench: Benchmarking multimodal llms with generative comprehension. _arXiv preprint arXiv:2307.16125_.
* Li et al. (2022a) Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. 2022a. Lavis: A library for language-vision intelligence.
* Li et al. (2023c) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023c. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. _arXiv preprint arXiv:2301.12597_ 

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In _International Conference on Machine Learning_.
* Li et al. (2023) Zejun Li, Ye Wang, Mengfei Du, Qingwen Liu, Binhao Wu, Jiwen Zhang, Chengxing Zhou, Zhihao Fan, Jie Fu, Jingjing Chen, Xuaning Huang, and Zhongyu Wei. 2023d. Reform-eval: Evaluating large vision language models via unified re-formulation of task-oriented benchmarks. _arXiv 2310.02569_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* ECCV 2014
- 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V_, volume 8693 of _Lecture Notes in Computer Science_, pages 740-755. Springer.
* Liu et al. (2023a) Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. 2023a. Visual spatial reasoning. _Transactions of the Association for Computational Linguistics_.
* Liu et al. (2023b) Haotiian Liu, Chunyuan Li, Yuheng Li, and Yong Bae Lee. 2023b. Improved baselines with visual instruction tuning.
* Liu et al. (2023c) Haotiian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023c. Visual instruction tuning. _arXiv preprint arXiv:2304.08485_.
* Liu et al. (2023d) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023d. Mmbench: Is your multi-modal model an all-around player? _arXiv preprint arXiv:2307.06281_.
* Lu et al. (2022) Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In _NeurIPS_.
* Lu et al. (2021) Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaadan Liang, and Song-Chun Zhu. 2021. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In _NeurIPS Track on Datasets and Benchmarks_.
* Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. OK-VQA: a visual question answering benchmark requiring external knowledge. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 3195-3204.
* Masry et al. (2022) Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 2263-2279, Dublin, Ireland. Association for Computational Linguistics.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv 1301.3781_.
* Mishra et al. (2019) Anand Mishra, Shashank Shekhar, Ajet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In _2019 international conference on document analysis and recognition (ICDAR)_, pages 947-952. IEEE.
* OpenAI (2023a) OpenAI. 2023a. Chatgtp.
* OpenAI (2023b) OpenAI. 2023b. Gpt-4.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In _Proceedings of the 38th International Conference on Machine Learning_, volume 139 of _Proceedings of Machine Learning Research_, pages 8748-8763. PMLR.
* Raven (1938) John C Raven. 1938. _Raven's Progressive Matrices: Sets A, B, C, D, E_. Australian Council for Educational Research.
* Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Code Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation imagetext models. _NeurlPS_, 35:25278-25294.
* Schwenk et al. (2022) Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In _ECCV_.
* Sidorov et al. (2020) Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020. Textcaps: a dataset for image captioningwith reading comprehension.
* Singh et al. (2019) Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In _CVPR_, pages 8317-8326.
* Song et al. (2022) Xiaozhuang Song, Shun Zheng, Wei Cao, James Yu, and Jiang Bian. 2022. Efficient and effective multitask grouping via meta learning on task combinations. _Advances in Neural Information Processing Systems_, 35:37647-37659.
* Spearman (1904) C. Spearman. 1904. General intelligence, objectively determined and measured. _American Journal of Psychology_, 15(2):201-292 

Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. 2020. Which tasks should be learned together in multi-task learning? In _International Conference on Machine Learning_, pages 9120-9132. PMLR.
* Su et al. (2021) Lin Su, Nan Duan, Edward Cui, Lei Ji, Chenfei Wu, Huaishao Luo, Yongfei Liu, Ming Zhong, Taroon Bharti, and Arun Sacheti. 2021. GEM: A general evaluation benchmark for multimodal tasks. In _Findings of the Association for Computational Linguistics_, Online. Association for Computational Linguistics.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Tiong et al. (2022) Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven C.H. Hoi. 2022. Plug-and-play vqa: Zero-shot vqa by conjoining large pre-trained models with zero training. In _Findings of the Conference on Empirical Methods in Natural Language Processing (Findings of EMNLP)_.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozieere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 4566-4575.
* Xi et al. (2023) Zhiheng Xi, Rui Zheng, Yuansen Zhang, Xuan-Jing Huang, Zhongyu Wei, Minlong Peng, Mingming Sun, Qi Zhang, and Tao Gui. 2023. Connectivity patterns are task embeddings. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 11993-12013.
* Xu et al. (2023) Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. _arXiv preprint arXiv:2306.09265_.
* Ye et al. (2023) Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mplug-owl: Modularization empowers large language models with multimodality. _arXiv preprint arXiv:2304.14178_.
* Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. _Transactions of the Association for Computational Linguistics_, 2.
* Yu et al. (2023) Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijun Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. _arXiv 2308.02490_.
* Zamir et al. (2018) Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. 2018. Taskonomy: Disentangling task transfer learning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 3712-3722.
* Zhou et al. (2022) Wangchunshu Zhou, Yan Zeng, Shizhe Diao, and Xinsong Zhang. 2022. Vlue: A multi-task benchmark for evaluating vision-language models. In _ICML_.
* Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. _arXiv preprint arXiv:2304.10592_.

 

Data Collection Protocol for OLIVE

OLIVE comprises 9,450 images, 30,120 unique instructions and 47,250 repenses. The examples can be broadly categorized into 4 groups: visual recognition, creative writing, knowledge-based, and elaborated description. Some examples are shown in Figure 1.

In our benchmark OLIVE, we use the text-only version of ChatGPT to generate instructions and outputs for each image. Specifically, we sample images from LAION-Aesthetics (Schuhmann et al., 2022) and use the BLIP-2 (Li et al., 2023c) captioning model to encode the visual information in each image into captions. These generated captions, along with the original LAION captions - which may contain entity-specific knowledge useful for generating varied instruction-output data - are then fed into ChatGPT. Additionally, for each of the aforementioned categories, we manually annotate a few seed examples, and use these as in-context examples to guide ChatGPT.

The instructions and outputs generated by ChatGPT could contain incorrect information due to model hallucination, which undermines their reliability for use as an evaluation benchmark. Recognizing this, we hire an annotation company, Flitto which recruits human annotators to thoroughly inspect and correct any erroneous data. They are task to: 1) ensure that the instructions contain minimal shortcut information, which could enable the model to produce correct outputs without having to understand the image, 2) verify the accuracy of the output and confirm that it is free from harmful content, and 3) fact-check knowledge-based information. This comprehensive review process helps to enhance the overall quality and reliability of the data.

## Appendix B ChatGPT Prompts for OLIVE

Following (Liu et al., 2023c) and (Taori et al., 2023), we construct prompts for ChatGPT (OpenAI, 2023a) to generate instructions and outputs for different categories: visual recognition, elaborated description, knowledge-based and creative writing. For elaborated description, we randomly sample from a list of instructions which inquiry about image description.

Prompt for generating creative writing instructions

You are given several image captions, each describing the same image you are observing. Using your creativity and imagination, think of a new instruction that can be induced from the image captions.

Here are the requirements:

1. Try not to repeat the verb for each instruction to maximize diversity.

2. The language used for the instruction also should be diverse. Either an imperative sentence or a question is permitted.

3. The type of instruction should be diverse.

4. The instruction must not involve counting.

5. Make the instruction challenging by not including the visual content details in the instruction so that one must use the captions to understand the instruction.

6. Replace the name of the object entity with a generic term or category, for example replace bus as this vehicle, dress as this clothing, etc.

7. The format of the instruction should follow the examples shown below. Make sure it is numbered and end with '**** 

From the generating knowledge-based instructions

You are given several image captions, each describing the same image you are observing. Using your creativity and imagination, think of a new instruction that can be induced from the image captions.

Here are the requirements:

1. Try not to repeat the verb for each instruction to maximize diversity.

2. The language used for the instruction also should be diverse. Either an imperative sentence or a question is permitted.

3. The instruction should be diverse and ask a question that requires reasoning, not just simple visual recognition.

4. Given the instruction, one should require first understanding the visual content, then based on the background knowledge or reasoning, either explain why the things are happening that way, or provide guides and help to user's request.

5. Make the instruction challenging by not including the visual content details in the instruction so that the user must use the captions to understand the instruction.

6. Replace the name of the object entity with a generic term or category, for example replace bus as this vehicle, dress as this clothing, etc.

7. The instruction must not involve counting.

8. The format of the instruction should follow the examples shown below. Make sure it is numbered and end with '###'.

Prompt for generating visual recognition instructions

You are given several image captions, each describing the same image you are observing. Using your creativity and imagination, think of a new instruction that can be induced from the image captions.

Here are the requirements:

1. Try not to repeat the verb for each instruction to maximize diversity.

2. The language used for the instruction also should be diverse. Either an imperative sentence or a question is permitted.

3. The instruction should ask about the visual content of the image, including the object types, object actions, object locations, etc. Only include instruction that has definite answers founded in the captions.

4. Include complex instruction that is relevant to the content in the image, for example, asking about background knowledge of the objects in the image, asking to discuss about events happening in the image, etc. Again, do not ask about uncertain details.

5. Make the instruction challenging by not including the visual content details in the instruction so that the one must use the captions to understand the instruction.

6. Replace the name of the object entity with a generic term or category, for example replace bus as this vehicle, dress as this clothing, etc.

7. The instruction must not involve counting.

8. The format of the instruction should follow the examples shown below. Make sure it is numbered and end with '###'.

 

List of instructions for elaborated description (part I)
* Provide a vivid description of the image.
* What is a suitable paragraph that describes this image?
* Compose a passage that depicts this image.
* What is this image about?
* What's happening in the scene?
* Can you describe the main features of this image for me?
* What are the key details in this picture?
* Can you elaborate on the elements of the picture provided?
* What do you think is going on in this photo?
* Can you provide a comprehensive description of the image?
* Describe the following image in detail.
* Provide a detailed portrayal of what's captured in this image.
* Offer an intricate description of the image you see.
* Please share a thorough run down of the image that has been presented.
* Could you elaborate on the contents of the displayed image with thoroughness?
* Clarify the contents of the displayed image with elaborate detail.

List of instructions for elaborated description (part II)

Can you offer a comprehensive portrayal of the image?
* Could you highlight and elaborate on the details of the image?
* Portray the image with a vivid comprehensive narrative.
* Analyze the image in a descriptive manner.
* Write an well-detailed depiction of the given image.
* How would you describe this photo in great detail?
* Can you give a detailed account of what you see in this image?
* Describe this image using your own words.
* Please describe what you see in the image with as much detail as possible.
* I need you to depict the image with utmost detail.
* Can you describe the image below in exhaustive detail?
* Please provide a complete description of what is shown in the picture.
* I would like you to give a detailed clarification of the contents of the displayed image.
* Could you provide a detailed and comprehensive representation of the image?
* Provide a comprehensive illustration of the image.
* Illustrate the image using a well-detailed description.
* Write a rich narrative for this image.
* Give a thorough description for the given image.
* Write a vivid account of the moment captured in this image.
* Create a narrative that is rich and vivid based on the image presented 

Fromple for generating visual recognition, knowledge-based and creative writing outputs

You are given an instruction and several image captions, each caption describing the same image you are observing. Generate an output resulting from following the instruction. Here are the requirements: 1. The output is the response to the instruction and the caption. 2. The output must utilize the information in the caption and must not contradict the caption. 3. If the output is unknown without further context, generate "unknown" as the output. 4. When using the information from the caption, directly explain the scene, do not mention that the information source is the caption. Always answer as if you are directly looking at the image. 5. Provide detailed output when answering complex instruction. For example, give detailed examples or reasoning steps to make the content more convincing and well-organized. 6. The format of the output should follow the examples shown below. Make sure it is numbered and end with '###'.

You are given an instruction and several image captions, each caption describing the same image you are observing. Here are the requirements: 1. Generate an output that describes the image in detail. 2. The output must utilize the information in the caption and must not contradict the caption. Do not include description of objects that is not presented in the caption. 3. When using the information from the caption, directly explain the scene, do not mention that the information source is the caption. Always answer as if you are directly looking at the image. 4. The format of the output should follow the examples shown below. Make sure it is numbered and end with '##'.

## Appendix C Model Details

We experiment with 4 different VLMs as follows:

* BLIP-2 utilizes ViT-G/14 Fang et al. (2023) as the image encoder and FlanT5XL Chung et al. (2022) as the LLM. We initialize BLIP-2 from the pretrained checkpoint and only fine-tune the Q-former parameters. Both the image encoder and the LLM are frozen. The total and trainable parameters are 4B and 187M respectively.
* MiniGPT-4 adopts ViT-G/14 Fang et al. (2023) as the image encoder and VicunaTB Chiang et al. (2023) as the LLM. It consists of the BLIP-2 Q-former and a linear layer connecting the image encoder and the LLM. The Q-former is initialized from BLIP-2. All parameters are frozen except the linear layer. The total and trainable parameters are 8B and 3M respectively.
* mPLUG-Owl adopts ViT-L/14 Radford et al. (2021) as the image encoder and LLaMA7B Touvron et al. (2023) as the LLM. It consists of a visual abstract module between the image encoder and the LLM. All parameters are frozen except LoRA Hu et al. (2022) parameters on the LLM. The total and 

trainable parameters are 7B and 4M respectively.
* LLaVA adopts ViT-L/14 Radford et al. (2021) as the image encoder and LLaMA\({}_{\text{7B}}\) Touvron et al. (2023) as the LLM. It consists of a linear layer in between the image encoder and the LLM. All parameters are frozen except the linear layer and LoRA Hu et al. (2022) parameters on the LLM. The total and trainable parameters are 7B and 164M respectively.

## Appendix D Additional Hyperparameters

We individually finetune models for each task using datasets in the instruction format. Only one instruction template is used per task, as preliminary experiments show using multiple templates per task degrades performance.

For all experiments using the same model architecture, we keep the hyperparameters constant. We set the training iteration to 10K steps. The batch size for BLIP-2 is 192 and 128 for the other three models. For BLIP-2, MiniGPT-4 and mPLUG-Owl, we train the model using AdamW optimizer with a weight decay of 0.05. The learning rate is linearly increased from \(1\mathrm{e}{-8}\) to \(1\mathrm{e}{-5}\) in the first 200 steps and then cosine decayed to 0. For LLaVA, we use a weight decay of 0. The learning rate is linearly rises from 0 to \(2\mathrm{e}{-5}\) across the initial 200 steps and then cosine decay to 0.

We output model performances at intervals of 1,000 iterations and select the best checkpoint using the validation set for evaluation.

All experiments are performed on a machine with 8 or 16 Nvidia A100 GPUs. On average, each experiment involves around 2 hours of training and another 2 hours of evaluation. We utilize LAVIS Li et al. (2022) library for training of BLIP-2, MiniGPT-4 and mPLUG-Owl. For LLaVA, we utilize LLaVA original author's codebase for training. All evaluations are performed on LAVIS.

## Appendix E Complete Results

In this section, we show all the experimental results from all four models. Tables 7-10 show the raw transfer learning performance, where rows denote the source tasks and columns are target tasks. Tables 11-14 show the normalized performance, where the rows (source tasks) are sorted in a descending order of average performance.

Among the source tasks, LLaVA Conversation shows strong transfer to OLIVE for BLIP-2 and LLaVA. The relatively good transferability could attribute to the fact that LLaVA Conversation and OLIVE share some similarities in data distribution since the instruction-response pairs are generated using OpenAI GPT models OpenAI (2023, 2023). However, the key difference is that OLIVE is inspected by human annotators to rectify erroneous data, while LLaVA Conversation does not undergo this correction process.

 

* [14] J. M. Cannon, R. A. B. Wilson, and D. W. Saxon, "Theoretical Population of the Universe: The Case of the New Paradigm," in _Proceedings of the XXVIII International Conference on Artificial Intelligence and Statistics_, volume 205 of _Lecture Notes in Computer Science_, pages 37-68, Springer, Berlin, Heidelberg, 1994.
* [15] L. P. Abbott, G. F. Anderson, and T. E. Barnes, "A Bayesian approach to the evolution of a new population of stars," _Annual Review of Astronomy_, vol. 101, no. 1, pp. 103-118, 1994.

[MISSING_PAGE_POST]

 

* [14] J.-P. Wang, M.-C. Zhang, and Y.-H. Chen, "A new method for the estimation of the unknown parameters in a two dimensional model," _Journal of Computational Physics: Series B_**206**(3), no. 5, pp. 798-803, 2009.
* [15] A.-M. Lin, R.-L. Li, and S.-Y. Zhu, "The estimation of the unknown parameters using the generalized least squares method," _IEEE Transactions on Signal Processing_**20**(1), no. 1, pp. 1-10, Jan 2009.
* [16] D. L. Miller, C.-S. Wu, and H.-W. Chen, "Estimating the maximum likelihood of a Gaussian mixture with application to the estimation of the unknown parameters," _arXiv preprint arXiv:1902.04432_ (2019).
* [17] P. N. Srivastava, G. E. Hinton, and I. Sutskever, "Learning the shape of a gaussian mixture with applications to image recognition," _Pattern Recognition Letters_**20**(1), no. 1, p. 105-118, 2010.
* [18] T. Krizhevsky, V. Nair, and Z. Sun, "Deep residual learning for large scale data sets," _Advances in neural information processing systems_**20**(1), no. 1, pp. 1-10, Jan 2009.
* [19] W.-T. Huang, X.-Q. Ma, and H.-W. Chen, "Classification of deep convolutional networks via stochastic optimization," _CoRR_**2014**(4), no. 1, pp. 1-10, Jan 2014.

[MISSING_PAGE_POST]

 

* [14] J. M. Cannon, R. A. B. Carter, and D. S. H. Smith, _The Origins of the New York: The Origins of the New York_, Cambridge University Press, 2003.

[MISSING_PAGE_POST]

 



* [14] M. C. D'Amico, A. J. Laird, and R. G. Wilson, _The New Journal of Physics: Volume I: Theory and Experiment_, vol. 30, p. 257-268, 1994.

[MISSING_PAGE_POST]

 

* [14] J.-P. Wang, M.-C. Zhang, and Y.-H. Chen, "A new method for the estimation of the unknown parameters in a two dimensional model," _Journal of Computational Physics: Series B_**206**(3), no. 5, pp. 798-803, 2000.
* [15] A.-M. Lin, R.-L. Li, and S.-Y. Zhu, "The estimation of the unknown parameters using the generalized least squares method," _IEEE Transactions on Signal Processing_**20**(1), no. 1, pp. 1-10, Jan 2000.
* [16] D. L. Miller, C.-S. Wu, and H.-W. Chen, "Estimating the maximum likelihood of a single Gaussian process with an application to the estimation of the unknown parameters," _arXiv preprint arXiv:0902.1018v1_ (2009).

[MISSING_PAGE_POST]

 

* [14] J. M. Cannon, R. A. B. Carter, and D. S. H. Smith, _The Origins of the New York: The Origins of the New York_, Cambridge University Press, 2003.
* [15] L. F. Anderson, P. G. Wilson, and T. W. K. White, _A Review on Astronomy_, Vol. 67, p. 98-103, 1996.

[MISSING_PAGE_POST]

 

* [14] J. M. Cannon, R. A. B. Carter, and D. S. H. Smith, _The Origins of the New York: The Origins of the New York_, Cambridge University Press, 2003.
* [15] L. F. Anderson, P. G. Wilson, and T. W. Keener, "A new approach to the identification of a single-dish galaxy in the sky," _Journal of Astronomical Society: Series A_**67**(8), no. 9, pp. 103-118, 1994.

[MISSING_PAGE_POST]

 

Mean Cosine Similarity of Target Tasks

In this section, we rank the average cosine similarity among the target tasks. We first compute pairwise cosine similarity using the SVD features of target tasks. For each target task, we take the average of all pairs that it is involved in. Finally, we rank all target tasks in a descending order.

## Appendix G Hierarchical Clustering of SVD Similarity

In this section, we perform hierarchical clustering on the SVD similarity features of target tasks using the Ward's linkage criterion which minimizes the total intra-cluster variance. In Figure 5, we show that hierarchical clustering forms meaningful clusters. For example, captioning tasks are clustered together. Generative and multiple-choice evaluated target tasks are grouped into different groups. This cluster supports the generative vs multiple-choice evaluation factor from factor analysis. However, the clusters are not as comprehensive as common factors extracted by factor analysis. For example, hierarchical clustering does not elucidate factors such as reading vs reasoning, and spatial reasoning.

## Appendix H Factor Analysis Details

Here we show all the factor loadings of the six factors from the residual matrix \(\bar{A}\). Community quantifies the proportion of variance in each target task that is accounted for by the identified factors. A low community value indicates that a task differs significantly from others in the mix.

Figure 5: Hierarchical clustering of target tasks.

\begin{table}
\begin{tabular}{l c} \hline \hline Target Task & Cosine Similarity \\ \hline OK-VQA (MC) & 0.54 \\ VQAv2 (G) & 0.54 \\ A-OKVQA (MC) & 0.54 \\ GQA (G) & 0.53 \\ OK-VQA (G) & 0.53 \\ A-OKVQA (MC) & 0.52 \\ TextVQA (MC) & 0.51 \\ YQAv2 (MC) & 0.50 \\ TexIVQA (G) & 0.50 \\ CLEVR (G) & 0.50 \\ ChartQA (G) & 0.48 \\ OCR-VQA (G) & 0.45 \\ ScienceQA (MC) & 0.45 \\ GOQA (MC) & 0.44 \\ Hateful Memes (MC) & 0.42 \\ VSR (MC) & 0.41 \\ NY Ranking (MC) & 0.37 \\ IconDA (MC) & 0.35 \\ OCR-VQA (MC) & 0.32 \\ TextCaps (G) & 0.29 \\ RAVEN-FARR (MC) & 0.27 \\ Flickr30k (G) & 0.25 \\ COCO Caption (G) & 0.22 \\ ChartQA (MC) & 0.15 \\ MORE (G) & 0.15 \\ CLEVR (MC) & 0.14 \\ OLIVE & -0.06 \\ NY Explanation (G) & -0.09 \\ OpenCQA (G) & -0.27 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Mean cosine similarity, computed from the SVD features, for each target task. The tasks are ranking by descending similarity.

 



\begin{table}
\begin{tabular}{c c c c c c c c} \hline
**Target Tasks** & **Factor 1** & **Factor 2** & **Factor 3** & **Factory 4** & **Factor 5** & **Factor 6** & **Communality** \\ \hline Flickr30k & 0.97 & -0.02 & 0.00 & 0.07 & 0.06 & -0.08 & 0.96 \\ COCO Caption & 0.93 & -0.05 & 0.00 & 0.10 & -0.02 & -0.12 & 0.90 \\ TextCaps & 0.83 & 0.12 & -0.20 & 0.07 & 0.10 & -0.10 & 0.77 \\ TextVQA (G) & -0.19 & 0.87 & 0.04 & -0.10 & -0.14 & -0.16 & 0.85 \\ VQAv2 (MC) & -0.34 & -0.74 & -0.34 & -0.01 & 0.24 & -0.02 & 0.83 \\ ChartQA (G) & -0.08 & 0.67 & -0.16 & 0.31 & -0.12 & -0.23 & 0.65 \\ OK-VQA (G) & -0.24 & 0.60 & 0.51 & -0.20 & 0.20 & 0.15 & 0.78 \\ GQA (MC) & -0.32 & -0.55 & -0.18 & -0.26 & -0.02 & 0.00 & 0.50 \\ OK-VQA (MC) & -0.43 & -0.49 & -0.30 & 0.07 & 0.22 & -0.20 & 0.62 \\ VQav2 (G) & 0.08 & 0.06 & 0.85 & 0.23 & 0.05 & -0.25 & 0.86 \\ GOA (G) & -0.22 & -0.01 & 0.75 & -0.05 & -0.21 & 0.12 & 0.66 \\ A-OKVQA (G) & -0.28 & 0.54 & 0.59 & -0.26 & 0.23 & 0.17 & 0.87 \\ TextVQA (MC) & -0.38 & -0.12 & -0.49 & 0.02 & 0.36 & -0.23 & 0.58 \\ OCR-VQA (MC) & 0.20 & -0.14 & -0.04 & 0.65 & -0.19 & -0.27 & 0.60 \\ ChartQA (MC) & -0.14 & 0.07 & -0.02 & 0.65 & 0.19 & 0.29 & 0.57 \\ RAVEN-FAIR (MC) & 0.02 & -0.01 & 0.08 & -0.40 & -0.04 & 0.17 & 0.20 \\ ScienceQA (MC) & -0.07 & 0.00 & -0.07 & -0.39 & -0.05 & -0.06 & 0.17 \\ IconQA (MC) & -0.01 & -0.09 & -0.08 & -0.34 & -0.05 & -0.10 & 0.14 \\ OCR-VQA (G) & -0.01 & 0.11 & -0.04 & -0.12 & -0.66 & 0.01 & 0.46 \\ A-OKVQA (MC) & -0.21 & -0.35 & -0.38 & -0.18 & 0.63 & -0.07 & 0.74 \\ MORE (G) & 0.22 & 0.47 & -0.22 & 0.21 & 0.54 & -0.03 & 0.65 \\ OpenCQA (G) & 0.17 & -0.07 & -0.09 & 0.11 & -0.32 & -0.24 & 0.21 \\ OLIVE (G) & -0.05 & 0.06 & 0.09 & 0.10 & -0.08 & 0.61 & 0.40 \\ CLEVR (G) & -0.17 & 0.20 & 0.16 & -0.44 & -0.34 & 0.59 & 0.74 \\ CLEVR (MC) & -0.18 & -0.13 & -0.05 & -0.07 & 0.01 & 0.55 & 0.36 \\ VSR (MC) & 0.15 & -0.26 & -0.10 & 0.10 & -0.06 & 0.50 & 0.37 \\ NY Explanation (G) & 0.13 & -0.03 & -0.04 & 0.26 & 0.21 & -0.10 & 0.14 \\ NY Ranking (MC) & -0.24 & -0.30 & 0.13 & 0.08 & -0.23 & 0.04 & 0.22 \\ Hateful Memes (MC) & 0.05 & -0.09 & -0.16 & -0.14 & -0.24 & 0.05 & 0.12 \\ \hline \end{tabular}
\end{table}
Table 16: Results of EFA on the residuals \(\bar{A}\). Cut-off for factor loadings = 0.3.

\begin{table}
\begin{tabular}{c|c c c c} \hline
**Target Tasks** & **Factor 1** & **Factor 2** & **Factor 3** & **Communality** \\ \hline OCR-VQA (GC) & 0.81 & 0.31 & 0.28 & 0.82 \\ ChartQA (MC) & 0.72 & 0.38 & 0.21 & 0.70 \\ A-OKVQA (MC) & 0.51 & 0.69 & 0.44 & 0.93 \\ TextVQA (MC) & 0.53 & 0.69 & 0.39 & 0.90 \\ OK-VQA (MC) & 0.59 & 0.64 & 0.44 & 0.95 \\ GGA (MC) & 0.23 & 0.28 & 0.93 & 1.00 \\ VQAV2 (MC) & 0.50 & 0.55 & 0.64 & 0.96 \\ \hline \end{tabular}
\end{table}
Table 17: Results of EFA on generative VQAs. Cut-off for factor loadings = 0.6.