Understanding the Role of LLMs in Multimodal Evaluation Benchmarks

## Abstract

Abstract The rapid advancement of Multimodal Large Language Models (MLLMs) has been accompanied by the development of various benchmarks to evaluate their capabilities.
However, the true nature of these evaluations and the extent to which they assess multimodal reasoning versus merely leveraging the underlying Large Language Model (LLM) backbone remain unclear.
This paper presents a comprehensive investigation into the role of LLM backbones in MLLM evaluation, focusing on two critical aspects: the degree to which current benchmarks truly assess multimodal reasoning and the influence of LLM prior knowledge on performance. Specifically, we introduce a modified evaluation protocol to disentangle the contributions of the LLM backbone from multimodal integration, and an automatic knowledge identification technique for diagnosing whether LLMs equip the necessary knowledge for corresponding multimodal questions.
Our study encompasses four diverse MLLM benchmarks and eight state-of-the-art MLLMs. Key findings reveal that some benchmarks allow high performance even without visual inputs and up to 50% of error rates can be attributed to insufficient world knowledge in the LLM backbone, indicating a heavy reliance on language capabilities.
To address knowledge deficiencies, we propose a knowledge augmentation pipeline that achieves significant performance gains, with improvements of up to 60% on certain datasets, resulting in a approximately 4x increase in performance.
Our work provides crucial insights into the role of the LLM backbone in MLLMs, and highlights the need for more nuanced benchmarking approaches. 1 1 1 Code and data are available at https://github.com/Twilight92z/Role-of-LLM .

## 1 Introduction

The rapid development of Large Language Models (LLMs) (Touvron et al., ; Bai et al., ), combined with advancements in visual encoders (Radford et al., ; Zhai et al., ) and modality bridge techniques (Liu et al., ; Dai et al., ), has catalyzed the evolution of Multi-modal Large Language Models (MLLMs) capable of comprehending diverse multi-modal inputs. Concurrently, diverse benchmarks and leaderboards have emerged to evaluate various multimodal perception and reasoning capabilities (Lu et al., ; Lerner et al., ; Yue et al., ).

While these benchmarks aim to assess multimodal capabilities, the role of the underlying LLM backbone in MLLM performance remains poorly understood. Recent studies (Tong et al., ; Yue et al., ) have highlighted that some benchmarks demonstrate an excessive dependence on the language model component, allowing MLLMs to achieve high scores even without visual inputs. This observation raises critical questions about the true nature of multimodal reasoning in these models and the extent to which performance is driven by the LLM backbone rather than multimodal integration.
Furthermore, as different MLLMs utilize LLM backbones with distinct knowledge priors learned from various pre-training corpora (Gao et al., ; Penedo et al., ), this knowledge inconsistency leads to incomparable evaluation scores when comparing MLLMs with different underlying LLMs. These issues can result in misinterpretation of evaluation scores and may misguide research and deployment of MLLMs by providing an inaccurate assessment of their true multimodal capabilities.

In this paper, we present an in-depth investigation into the role of LLM backbones in MLLM evaluation, focusing on two key aspects: (i) the extent to which current benchmarks truly assess multimodal reasoning versus relying on language capabilities alone, and (ii) the influence of LLM prior knowledge on final performance.
To address the first question, we propose an approach that goes beyond simply evaluating models without visual cues (Tong et al., ; Chen et al., ). Our method incorporates comparisons with shuffled options and transforms multiple-choice quesiont-answering (QA) formats into open-ended generation tasks, providing a more comprehensive understanding of the role of language capabilities versus true multimodal reasoning in these benchmarks.
For the second question,
we develop an automatic knowledge identification method utilizing external knowledge bases such as Wikipedia or powerful LLMs (OpenAI, ) to obtain the necessary knowledge behind each question. With these knowledge facts prepared, we examine whether the underlying LLM backbone possesses the requisite world knowledge for multimodal questions, enabling a better understanding of the obtained scores.

We select four benchmarks covering different capabilities of MLLMs: the comprehensive evaluation benchmark MMMU (Yue et al., ), ScienceQA for multimodal reasoning evaluation (Lu et al., ), and two knowledge-based VQA tasks: Viquae (Lerner et al., ) and InfoSeek (Chen et al., ).
Our experimental results with eight MLLMs reveal significant insights into the role of LLM backbones:
(i) LLMs would exploit the shortcuts in question and options, making predictions without relying on the visual inputs. For example, on the commonly adopted MMMU dataset, accuracy scores remain the same for more than 80% of samples even without visual inputs.
Further comparison of datasets and task formats suggests that knowledge-intensive VQA benchmarks requiring entity recognition from images are less affected by this issue, and LLMs could solely rely on options to achieve prediction without relying on visual inputs.
Specifically, we observe that the average performance difference between scenarios with and without visual inputs is markedly lower for multiple-choice questions in MMMU (15%) compared to open-ended questions in InfoSeek (65%).
(ii) MLLMs performance shows great dependence on the knowledge of LLM backbones. We observe that up to 50% of error rates on multimodal benchmarks could be attributed to insufficient world knowledge in the LLM backbone. Besides, MLLMs adopting knowledgeable LLMs such as LLaVA-Next-Yi-34B and InternVL2-Llama3-76B during evaluation tend to perform better, highlighting the significant impact of the LLM backbone on overall performance.
Motivated by these findings, we develop a simple knowledge augmentation pipeline to retrieve supplementary background knowledge for answering challenging VQA questions. This approach yields an average significant 36% absolute accuracy gain, with Phi-3 achieving an impressive improvement of over 60% on the Viquae dataset.
Further analysis demonstrates a trade-off between knowledge recall and the noise introduced by retrieved knowledge paragraphs.

Our study provides crucial insights into the role of LLM backbones in MLLM evaluation and highlights the need for more nuanced benchmarking approaches that can distinguish between language model capabilities and true multimodal reasoning.
These findings have important implications for the development and evaluation of future MLLMs, suggesting that both visual integration techniques and the choice of LLM backbone are critical factors in achieving robust multimodal performance.
nd the limitations in practical applications.

## 2 Method

In this section, we perform an approach to better understand the role of LLM in multi-modal evaluation benchmarks.
Figure [1](#S2.F1) provides a comprehensive overview of the method.
We begin by formally introducing the key notations essential for setting up our framework (§[2.1](#S2.SS1)).
Following this, we delve into the specifics of how we measure the significance of vision and knowledge. First we outline the methodologies for evaluating the role of vision (§[2.2](#S2.SS2)).
We then explore the methodologies for gauging the impact of factual knowledge §[2.3](#S2.SS3) and develop a knowledge-augmented framework to assist the MMLMs (§[2.4](#S2.SS4)).

Figure: Figure 1: Left:We first identify VQA questions answerable without images. Right:We subsequently decompose the process of solving visual questions into two distinct yet interrelated steps, decoupling visual perception capability from knowledge.
Refer to caption: /html/2410.12329/assets/x1.png

### 2.1 Problem Notations

VQA involves providing a model with visual input and a related question, and then requiring the model to generate an appropriate answer. Let $D$ be a given multi-modal dataset. For any data entry $d$ in $D$, we define $d$ as a triple $(I,Q,A)$, where $I$ denotes the visual input (a single image in our work), $Q$ represents the textual question, and $A$ is the corresponding answer.
We posit that MLLMs process VQA tasks through two primary stages: visual perception and knowledge reasoning. In the visual perception stage, the model extracts key information from the image input. Subsequently, we hypothesize that the model internally reformulates the original VQA question into a cognate knowledge reasoning query, denoted as $K$. This query $K$ integrates both the textual input and the extracted visual information, forming the basis for the ensuing reasoning process to derive the answer.

Regarding the representation of the multi-modal large language model (MLLM), we employ the function form $f$, where $f(I,Q),f(\varnothing,Q),$ and $f(\varnothing,K)$ denote the model’s responses to visual questions with images, without images, and to knowledge reasoning queries respectively. The visual perception step is described by $P$, with $\sim$ used instead of $=$ to reflect the non-deterministic nature of visual conception. For evaluation, we define a combination $C$ as one of these three input types. Given a dataset $D$, we calculate the Score Rate (SR) as:

$$ $\displaystyle\text{SR}_{D}^{C}=\frac{1}{|D|}\sum_{d\in D}\mathds{1}\left[f(C_{d})==A\right].$ (1) $$

For instance, $\text{SR}_{Viquae}^{(\varnothing,K)}$ represents the model’s performance on knowledge reasoning questions in the Viquae dataset. Empirically, we believe that a higher SR of a model indicates stronger performance, and vice versa.

### 2.2 Is Visual Capability Necessary?

Previous research has demonstrated that the absence of visual input often does not significantly impact model performance on certain visual evaluation datasets (Goyal et al., ; Tong et al., ; Huang et al., ). To elucidate this phenomenon, we extend prior work by systematically modifying the VQA task paradigm to assess the role of visual information under varied conditions. Our methodology involves presenting identical questions to models in image-present and image-absent contexts.
Furthermore, we introduce two critical modifications to the multiple-choice format: (1) randomization of multiple-choice option order and (2) reformulation of questions into open-ended queries. These alterations serve dual purposes: randomization of options mitigates potential biases towards specific answer types that may align with training data distributions, while open-ended reformulation allows us to evaluate whether the apparent diminished reliance on visual information is an artifact of constrained multiple-choice setups, where some options may be trivially eliminable. Our findings indicate that the presence of multiple-choice options significantly reduces both task difficulty and the necessity for visual information processing. This insight offers a nuanced perspective on the observed similarity in model performance across image-present and image-absent conditions in certain VQA datasets, underscoring the critical role of task design in accurately assessing visual reasoning capabilities.

To quantify these effects, we conduct a comparative analysis of performance differentials between image-present and image-absent scenarios at the dataset level for each model. We also introduce the Gap Rate (GR) metric, defined as:

$$ $\displaystyle\text{GR}_{D}=1-\frac{\text{SR}_{D}^{(\varnothing,Q)}}{\text{SR}_{D}^{(I,Q)}},$ (2) $$

to normalize for inherent variability in model capabilities. Theoretically, a well-constructed multi-modal dataset should elicit correct responses predominantly when visual input is provided, with performance in the absence of images approximating chance levels. Consequently, the GR serves as an indicator of a dataset’s efficacy in assessing genuine visual reasoning capabilities, with higher values suggesting a stronger coupling between visual information and task performance.

### 2.3 Do MLLMs Have Sufficient Prior Knowledge?

Based on our hypothesis that the resolution of visual tasks could be delineated into two distinct steps, upon receiving an image $I$ and a textual question $Q$, the model implicitly engages in a visual perception process $P$ to generate a corresponding knowledge reasoning problem $K$, then subsequently utilized it to make response. We formalize the entire process as follows:

$$ $\displaystyle\mathds{1}\left[f(I,Q)==a\right]=\mathds{1}\left[P(I,Q)\sim K\right]\cdot\mathds{1}\left[f(\varnothing,K)==a\right].$ (3) $$

The formula suggests that language prior knowledge and visual perception capabilities are equally crucial and indispensable. Therefore, the reason for models’ poor evaluation results may not only stem from insufficient visual capabilities but also from a lack of knowledge.

To determine whether the model’s knowledge is sufficient, we use knowledge reasoning questions corresponding to visual questions in each dataset as models’ inputs and then evaluate their performance. For datasets that do not provide corresponding knowledge reasoning questions, we directly replace image-referenced content in visual questions with given entities or invoke GPT-4(^2^22We use the GPT-4o-2024-05-13 version.) (Achiam et al., ) to convert the original visual questions (specific prompts employed are detailed in the Appendix [A.1](#A1.SS1)).

We also perform a statistical analysis about model’s correctness and errors in each visual question and its corresponding knowledge reasoning question. To quantify the analysis results, we introduce the following two indicators, Sufficiency Ratio (SuR) and Necessity Ratio (NeR), defined as follows:

$$ $\displaystyle\text{SuR}_{D}$ $\displaystyle=\frac{\sum_{d\in D}\mathds{1}\left[f(I_{d},Q_{d})==A_{d}\mid f(\varnothing,K_{d})==A_{d}\right]}{\sum_{d\in D}\mathds{1}\left[f(I_{d},Q_{d})==A_{d}\right]}$ (4) $\displaystyle\text{NeR}_{D}$ $\displaystyle=\frac{\sum_{d\in D}\mathds{1}\left[f(I_{d},Q_{d})\neq A_{d}\mid f(\varnothing,K_{d})\neq A_{d}\right]}{\sum_{d\in D}\mathds{1}\left[f(I_{d},Q_{d})\neq A_{d}\right]}.$ (5) $$

These ratios serve to elucidate the sufficiency and necessity relationship between prior knowledge and visual capability, where higher values signify a more robust relationship.

### 2.4 Can Knowledge Augmentation Improve Multimodal Capabilities?

In real-world scenarios, models often encounter the issue of insufficient knowledge due to their smaller scale or outdated information (Gao et al., ).
To mitigate the limitation caused by the absence of prior knowledge, we adopt a straightforward idea here, using the Retrieval-Augmented Generation (RAG) approach to effectively enhance the model’s knowledge and then design proper experiments for effectiveness evaluation (Weston et al., ; Cai et al., ).

We evaluate the relevance between the knowledge reasoning problem and the paragraph using cosine similarity, and then rank the paragraphs accordingly. Ultimately, the highest-ranked paragraphs are incorporated into the input to enhance the model’s knowledge base. Within the framework of RAG, for the top $n$ most relevant paragraphs $p_{1},p_{2},...,p_{n}$ from candidate knowledge document corpus $\mathcal{C}$, we articulate the calculation of Score Rate (SR) as follows:

$$ $\displaystyle\text{SR}_{D}^{\text{RAG}_{n}}=\frac{1}{|D|}\sum_{d\in D}\mathds{1}\left[f(I,(Q,p_{1},p_{2},...,p_{n}))==A\right].$ (6) $$

Specifically, we employ the state-of-the-art embedding model, NV-Embed-v2 (Lee et al., ; Moreira et al., ) as our retriever. Following the calculation of similarity, we select 1, 3, 5, and 10 as values for $n$ and evaluate the performance against the vanilla setup.

## 3 Experiments

As described in the preceding sections, we conduct extensive experiments to investigate the role of LLMs in MLLM evaluation. We first introduce the experimental settings (§[3.1](#S3.SS1).
We then discuss our findings regarding the shortcuts used by LLMs during evaluation (§[3.2](#S3.SS2)) and the knowledge deficiency (§[3.3](#S3.SS3)).
Finally, we illustrate interesting cases during our investigation (§[3.4](#S3.SS4)) and evaluate the effectiveness of our knowledge augmented method (§[2.4](#S2.SS4)).

### 3.1 Experimental Setup

Benchmarks.

The datasets we use primarily assess the models’ knowledge, generalizability, and reasoning abilities, and do not include datasets that primarily rely on visual recognition capabilities such as OCR. The specifics are as follows:

- •
Viquae (Lerner et al., ): Viquae comprises a test set of 1257 questions, is a visual version of the Named Entity Question Answering dataset, requiring the identification of named entities in images and then reasoning to answer questions based on the model’s inherent knowledge.
- •
ScienceQA (Lu et al., ): SQA consists of multimodal multiple-choice questions on various topics which is sourced from elementary and secondary school science curricula. We selected questions from the test set that have visual context for testing, totaling 2017 items.
- •
InfoSeek (Chen et al., ): InfoSeek is a dataset composed of visual information-seeking questions necessitating the model to draw upon fine-grained knowledge learned from pretraining instead of commonsense knowledge to formulate responses. We sampled 3000 questions from its validation set for testing purposes.
- •
MMMU (Yue et al., ): MMMU is composed of multimodal questions collected from university exams, quizzes, and textbooks, requiring the model to possess university-level subject knowledge and excellent reasoning abilities. We selected 648 single-image questions from a subset of its validation set for testing (further details of selected subset are available in Appendix [B.1](#A2.SS1)).

Figure: Figure 2: SR comparison of MLLMs under both image-present and image-absent conditions across four benchmarks. $f(\varnothing,Q)$ is relatively close to $f(I,Q)$, but far from Random Guess, indicating that the model’s utilization of visual information is low.
Refer to caption: /html/2410.12329/assets/x2.png

Test Models and Setup.

We conduct experiments using open-source multi-modal large models from different sources, ranging in scale from 4.2 billion to 76 billion parameters, including Qwen-VL (Qwen2-VL-7B-Instruct) (Bai et al., ), Idefics (Idefics2-8B) (Laurençon et al., ), LLaVA (LLaMA3-LLaVA-Next-8B, LLaVA-Next-Yi-34B) (Li et al., ), Phi-3 (Phi-3.5-vision-instruct) (Abdin et al., ), ChatGLM (GLM-4V-9B) (GLM et al., ), InternVL (InternVL2-Llama3-76B) (Chen et al., ) and MiniCPM (MiniCPM-V-2.6) (Yao et al., ).
We use a temperature of 0 for all models for deterministic results.
To ensure more accurate evaluation results, we employed different evaluation methods for open-ended questions and multiple-choice questions. On open-ended tasks, we evaluate the correctness of models’ responses by determining whether the candidate answers are present in the output of models through rule-matching. As to multiple-choice questions, we use DeepSeek-AI () to assess whether the model’s reasoning results are correct. The specific prompt used for determining the correctness of the model’s outputs can be found in Appendix [A.2](#A1.SS2)

Prompts.

For open-ended problems from Viquae and InfoSeek, we simply use the questions as input into the models. For multiple-choice questions from ScienceQA and MMMU, we concatenate the questions and options as the example shown in the Appendix [A.3](#A1.SS3) to form the model’s prompt without appending any additional information such as the topic in MMMU or the hint in SQA.

### 3.2 LLMs Exploit Shortcuts in Vision Tasks

**Table 1: $\text{SR}^{(I,Q)}$ and GR of MMMU in different question formats. Higher GR signifies greater utilization of visual information by models in open-ended visual tasks.**
|  | Original Option | Shuffled Option | Open-ended QA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| SR^(I,Q) | GR | SR^(I,Q) | GR | SR^(I,Q) | GR |  |
| Phi-3.5-vision-instruct | 57.6 | 11.0 | 51.9 | 6.0 | 7.1 | 10.9 |
| MiniCPM-V-2.6 | 57.4 | 8.3 | 53.5 | 13.8 | 10.3 | 34.3 |
| Qwen2-VL-7B-Instruct | 70.7 | 20.1 | 64.4 | 17.3 | 11.9 | 59.7 |
| LLaMA3-LLaVA-Next-8B | 53.1 | 6.1 | 58.2 | 19.4 | 7.7 | 36.0 |
| Idefics2-8B | 58.0 | 15.9 | 48.8 | 0.6 | 6.8 | 2.3 |
| GLM-4V-9B | 65.9 | 24.6 | 56.3 | 15.9 | 10.0 | 40.0 |
| LLaVA-Next-Yi-34B | 66.5 | 19.5 | 62.0 | 11.9 | 10.2 | 39.4 |
| InternVL2-Llama3-76B | 72.5 | 15.1 | 62.7 | 15.3 | 17.1 | 59.5 |

We conduct comparative experiments using eight models on four datasets, comparing the performance of the image-included setup with the image-excluded setup to enhance the broad applicability and reliability of the analysis. We also calculate the expected score for multiple-choice questions via randomly guessing. The outcomes are shown in Figure [2](#S3.F2). Nearly all GR values below 0.8 suggest that visual information is not always essential, echoing previous findings
(Yue et al., ; Tong et al., ).
Even on open-ended question-answering tasks like Viquae and InfoSeek, models still achieve average SRs of approximately 0.25 and 0.07 without using visual inputs. This could be due to the model having learned similar data during its training process, as the data for these datasets is sourced from Wikipedia, which is widely used in pre-training or supervised fine-tuning stages. As to multiple-choice questions like SQA and MMMU, models’ average GR on these two datasets is only 0.18 and 0.15, respectively. Such low GR values indicate a negligible role of visual input in performance.

On MMMU, we explore the correlation between question setup and the role of vision by shuffling or removing the initial options of each question, with the results visualized in Table [1](#S3.T1). Obviously, our changes to the question setup pose greater challenges to the MLLMs, as almost all models’ SRs have decreased to some extent.
Analyzing from the perspective of GR, shuffling the initial options has a relatively minor overall impact.
However, the removal of options leads to a significant increase in the maximum GR, rising from 24.6 to 59.7 percent, representing an over 100% enhancement. Combining the previous results, we believe that vision plays a more significant role in open-ended questions, as LLMs may potentially exploit shortcuts within the provided options to formulate responses.

### 3.3 MLLMs Suffers from LLMs’ Knowledge Deficiency

**Table 2: SR of knowledge reasoning questions across four datasets. Almost all models encounter the challenge of insufficient knowledge.**
|  | Viquae | InfoSeeksample | SQAIMG. | MMMUval. |
| --- | --- | --- | --- | --- |
| Phi-3.5-vision-instruct | 65.8 | 43.9 | 83.2 | 64.2 |
| MiniCPM-V-2.6 | 82.7 | 48.8 | 84.7 | 63.3 |
| Qwen2-VL-7B-Instruct | 78.6 | 45.7 | 80.8 | 72.2 |
| LLaMA3-LLaVA-Next-8B | 83.5 | 52.6 | 76.9 | 63.9 |
| Idefics2-8B | 86.4 | 55.5 | 78.8 | 59.7 |
| GLM-4V-9B | 80.0 | 53.0 | 83.1 | 63.3 |
| LLaVA-Next-Yi-34B | 91.3 | 57.7 | 79.6 | 69.3 |
| InternVL2-Llama3-76B | 94.7 | 61.6 | 88.2 | 77.2 |

Figure: Figure 3: SuR and NeR of different models across four datasets. High values indicate that possessing relevant prior knowledge is a prerequisite for solving visual tasks.
Refer to caption: /html/2410.12329/assets/x3.png

Knowledge deficiency of Large Language Models (LLMs) in Visual Question Answering (VQA) tasks are evident, even for state-of-the-art systems. As demonstrated in Table [2](#S3.T2), InternVL, despite being equipped with the powerful LLaMA3-70B, achieves an average SR not exceeding 90% across various datasets. This performance ceiling is even more pronounced in smaller models, which exhibit average SRs of approximately 70% across diverse datasets. These findings suggest that all models used in our experiments face the challenge of inadequate knowledge.

Furthermore, our analysis of SuR and NeR also substantiates the significant impact of prior knowledge on visual capability, as presented in the Figure [3](#S3.F3). Taking Phi-3 as an example, it possesses relevant knowledge for over 85% of the visual questions it correctly answered on Viquae. At the same time, over 95% of its knowledge reasoning errors on the InfoSeek dataset are accompanied by failures in their corresponding visual tasks, indicating that knowledge deficiencies are likely a significant factor impairing the model’s performance. Our findings suggest that the model heavily relies on relevant knowledge when solving visual tasks, implying the suboptimal performance of MLLMs may stem from the knowledge deficit in their backbone LLMs.

### 3.4 Case Study

We present specific cases in Figure [4](#S3.F4) that challenge our initial assumption regarding the decomposition of visual tasks into perception and reasoning steps. An example involves a question about the venue of The Beatles’ last ever live concert. In the VQA context, the model correctly identifies The Beatles in the image and subsequently deduces that Candlestick Park was the venue for their last concert. Paradoxically, when presented with the same query as a pure knowledge reasoning question without visual input, the model fails to provide the correct answer. This observed performance disparity may be attributed to the knowledge representation within the model’s architecture. We hypothesize that the relevant information is encoded in the model’s parameters in a manner that is more closely aligned with visual question-answering paradigms. Consequently, the presence of this image in the input potentially serves as a more effective retrieval cue, facilitating the model’s access to pertinent knowledge.

The model sometimes demonstrates proficiency in accurately answering knowledge reasoning queries, exemplified by its correct responses regarding Barack Obama. However, when confronted with visual questions, it exhibits a propensity for hallucination during the reasoning process, despite accurately identifying the Westminster Hall. This discrepancy suggests a misalignment between visual and textual modalities. While the model possesses the requisite knowledge, as evidenced by its performance on purely text-based queries, it struggles to effectively apply this prior knowledge to visual task resolution.

Figure: Figure 4: Top: While successfully answering the visual question, the model fail to perform well on knowledge reasoning tasks. Bottom: The model has relevant knowledge but exhibits hallucinations when addressing visual questions.
Refer to caption: /html/2410.12329/assets/x4.png

### 3.5 Retrieved Knowledge Boosts Multimodal Abilities

Since the lack of knowledge is inevitable, to compensate for this deficiency, it is natural to consider using the RAG approach to enhance the model’s knowledge. We employ the embedding model to retrieve the most relevant content from Wikipedia (June 2024 Wikipedia dump) for each knowledge reasoning question on InfoSeek and Viquae, and incorporate the information into the corresponding input.
The recall on this two datasets is presented in Table [3](#S3.T3).
The performance of all models in solving visual tasks has significantly improved after knowledge enhancement, as shown in the Figure [5](#S3.F5). Nevertheless, in contrast to the recall that increases with the number of relevant documents, the model’s SR demonstrates a trend of initially rising and then slightly decreasing., which may be attributed to the noise introduced by an excessive number of relevant paragraphs. In summary, supplementing knowledge significantly enhances the model’s performance on visual tasks, which can be applied to model evaluation to minimize differences in relevant knowledge and focus more on visual capabilities.

**Table 3: Recall of the embedding model on knowledge-intensive VQA datasets.**
|  | Recall@1 | Recall@3 | Recall@5 | Recall@10 | Recall@50 |
| --- | --- | --- | --- | --- | --- |
| Viquae | 45.2 | 65.0 | 73.3 | 82.3 | 91.7 |
| InfoSeek | 77.9 | 91.3 | 94.1 | 96.2 | 97.9 |

Figure: Figure 5: Differences in SR between scenarios without Retrieval-Augmented Generation (RAG) and those using RAG with 1, 3, 5, and 10 relevant documents. Knowledge enhancement significantly improves model performance.
Refer to caption: /html/2410.12329/assets/x5.png

## 4 Related Work

#### Multimodal Large Language Models.

Multi-modal Large Language Models (MLLMs) have made remarkable strides in recent years (OpenAI, ; Reid et al., ; Ormazabal et al., ), demonstrating an unprecedented ability to understand and generate content that seamlessly integrates visual and textual information (Fu et al., ).
Representative proprietary commercial models, such as OpenAI’s GPT-4o (Achiam et al., ), Google’s Gemini 1.5 Pro (Reid et al., ), and Anthropic’s Claude 3.5 Sonnet (Anthropic, ), have showcased impressive capabilities in various tasks. On the open-source front, models like LLaVA (Liu et al., ), Qwen-VL (Bai et al., ) and Phi-Vision (Abdin et al., ), have also demonstrated remarkable progress, particularly in their ability to comprehend multiple images or video simultaneously, expanding the scope of MLLMs from static single images to dynamic multi-frame visual content.
Our research aims to gain a deeper understanding of MLLM’s performance and limitations, with a special focus on the role of the LLM backbone.
Our experimental results show that current MLLMs rely on the LLM backbone heavily on certain benchmarks, and suffer from knowledge deficiency when facing VQA tasks demanding rich world knowledge. Based on our findings, we introduce a RAG-based method that significantly enhances model performance.

#### Multimodal Understanding Benchmarks.

The rapid advancement of MLLMs has spurred the development of diverse evaluation benchmarks. These range from specialized tasks like OCR (e.g., InfogrpahicsVQA (Mathew et al., ), ChartVQA (Masry et al., ), DocVQA (Mathew et al., )), knowledge integration (e.g., Viquae (Lerner et al., ) and Infoseek (Chen et al., )), and mathematical reasoning (e.g., ScienceQA (Lu et al., ) and MathVista (Lu et al., )), to comprehensive frameworks such as MMMU (Yue et al., ), MME (Fu et al., ), MMBench (Liu et al., ), and MMVet (Yu et al., ).
Our work contributes to this landscape by critically examining these evaluation frameworks, echoing previous findings that visual inputs may contribute less significantly in these benchmarks (Yue et al., ; Chen et al., ; Tong et al., ).
Additionally, we show that the multiple-choice format could become a shortcut that the LLMs could leverage to bypass the visual inputs and we also identify certain errors primarily stem from language knowledge limitations rather than visual perception deficiencies. These findings provide valuable insights for developing more robust evaluation benchmarks, emphasizing the need to disentangle language model capabilities from true multimodal reasoning in MLLM assessment.

## 5 Conclusion

This study provides a comprehensive analysis of the role of LLM backbones in Multimodal Large Language Model (MLLM) evaluation, shedding light on critical aspects that have been largely overlooked in previous research.
Our investigation reveals several key insights that have significant implications for the development and evaluation of MLLMs.
Our experimental findings first show that LLMs could exploit shortcuts by relying on inappropriate options in visual tasks, and that open-ended questions could offer more robust assessments.
Secondly, we identify substantial knowledge deficiencies across various datasets, where models fail to provide correct answers despite accurate visual perception. To mitigate this, we implement a Retrieval-Augmented Generation (RAG) approach, which significantly improved performance on visual tasks by enhancing the models’ factual knowledge. Further analysis reveals a phenomenon of knowledge misalignment between visual and textual modalities.

## Limitations

Since only a portion of the models used in our experiments support multi-image input, and some questions are difficult to accurately convert into corresponding knowledge inference tasks, we selected only a subset of the MMMU dataset. Additionally, due to the scarcity of multi-modal embedding models, we opted to use knowledge inference questions instead of visual questions for retrieval during the RAG process. In future work, we plan to employ multi-modal retrievers to identify the most relevant paragraphs for VQA questions and evaluate the effectiveness.

## Appendix A Prompt

### A.1 Convert VQA questions

Since some datasets do not provide knowledge reasoning questions corresponding to visual questions, we have designed a sophisticated prompt that inputs the original visual input, text question, and corresponding answer into GPT-4 to transform the question. The specific prompt is as Table [4](#A1.T4).

Table: Table 4: Prompt Template to Convert Visual Questions.

### A.2 Model Evaluation on Multiple-Choice Questions

Due to the possibility that the model may generate a lot of thinking during the answering process, and the corresponding letters for options such as ‘A’ or ‘B’ are likely to appear within the output, the rule-matching method may not be accurate enough. Therefore, we use DeepSeek to evaluate the model’s output, resulting in a more accurate assessment. The specific prompt is as Table [5](#A1.T5).

Table: Table 5: Prompt Template to Evaluate Multiple-Choice Questions.

### A.3 Input Template for Multiple-Choice Questions

We simply add corresponding prefixes to the question part and the option part. We also insert a prompt “Answer” at the end of the question to instruct the model to respond to the question. Here is an example from MMMU dataset, as illustrated in the Table [6](#A1.T6).

**Table 6: Sample Multiple-Choice Question Input.**
|  |  |
| --- | --- |
| User | Question: Identify the biome shown in **IMAGE** |
|  | Options: |
|  | (A)  taiga |
|  | (B)  tundra |
|  | (C)  rain forest |
|  | (D)  desert |
|  | Answer: |

## Appendix B Dataset Setup

### B.1 mmmu subset

In the Table [7](#A2.T7), we present the specific subsets of MMMU that we selected, along with the number of questions in each subset. Moreover, we provide the Success Rate (SR) using GPT-4 on its transformed knowledge reasoning questions.

**Table 7: MMMU Subset Sizes and GPT-4 Success Rate.**
| Type | Num. | GPT-4 |
| --- | --- | --- |
| Accounting | 30 | 56.7 |
| Agriculture | 29 | 69.0 |
| Art | 30 | 73.3 |
| Art Theory | 25 | 88.0 |
| Basic Medical Science | 28 | 85.7 |
| Biology | 27 | 51.9 |
| Chemistry | 18 | 61.1 |
| Clinical Medicine | 29 | 89.7 |
| Computer Science | 25 | 68.0 |
| Design | 30 | 80.0 |
| Diagnostics and Laboratory Medicine | 29 | 65.5 |
| Economics | 27 | 81.5 |
| Finance | 22 | 68.2 |
| Geography | 26 | 53.9 |
| History | 28 | 75.0 |
| Literature | 29 | 89.7 |
| Manage | 24 | 66.7 |
| Marketing | 29 | 82.8 |
| Math | 26 | 65.4 |
| Pharmacy | 24 | 83.3 |
| Physics | 28 | 71.4 |
| Psychology | 25 | 80.0 |
| Public Health | 30 | 83.3 |
| Sociology | 28 | 71.4 |

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 5
presentation: 1
questions: 1. In Section 3.3, Table 2, how do you conclude that the performance cap is due to insufficient knowledge? Could it not also be attributed to limited reasoning capabilities, among other factors?
2. Additionally, the writing and notation used in the paper are unclear and need improvement, e.g.
* The connection between some proposed concepts is unclear. There is a missing link between the sections “IS VISUAL CAPABILITY NECESSARY?” and “DO MLLMS HAVE SUFFICIENT PRIOR KNOWLEDGE.”
* The indicator function $\mathbb{1}\left[ P(I, Q) \sim K \right]$ lacks a clear definition of its true conditions. The notation is poorly defined, as  $P(I, Q) \sim K$  is usually interpreted as drawing  K  from the joint distribution of  I  and  Q.
soundness: 2
strengths: The idea is intuitive and easy to understand.
summary: The paper proposes an examination of issues within multi-modal evaluation from two aspects:

	1.	To what extent answers rely solely on textual information
	2.	How much external knowledge can boost the performance of multimodal questions

The paper finds that in some current benchmarks, up to 50% of questions can be solved using only textual queries. The authors introduce a retrieval-augmented generation (RAG) method to improve the language model’s performance.
weaknesses: * The issue that multimodal questions can often be solved with textual queries alone is not new; many prior works either study or are motivated by this observation (e.g., [1,2]). The paper lacks a more extensive discussion of related works that have studied this phenomenon.
* Equation 3 is confusing. From an intuitive standpoint, it doesn’t seem to hold because: 1) even without the correct knowledge, the model may still provide the correct answer, and 2) in many cases, explicit knowledge is not required (e.g., inferring relative distances between two objects in an image), so such decomposition may not apply.
* It's expected that additional information retrieved by RAG should boost the performance. It would be beneficial to discuss when RAG provides the most advantage. For instance, what types of VQA questions benefit most from RAG, and under what conditions does it perform best?
* It would be nice to provide some results on the OK-VQA dataset as well, as it also requires external knowledge which should benefit from rag. 

References:
[1] Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., … & Krishna, R. (2024). Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390.

[2] Zhang, J., Mishra, A., Patwardhan, S., & Agarwal, S. (2022). Can open-domain question answering systems answer visual knowledge questions? arXiv preprint arXiv:2202.04306.

[3] Schwenk, D., Khandelwal, A., Clark, C., Marino, K., & Mottaghi, R. (2022, October). A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision (pp. 146-162). Cham: Springer Nature Switzerland.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see the Weaknesses.
soundness: 3
strengths: + Rethinking the impact of the LLM backbone in MLLM evaluation is insightful and may advance the development of MLLM.
+ The paper is generally easy to read and follow.
+ The introduced metrics and RAG pipeline are sound.
summary: The paper explores the impact of LLM backbones on the evaluation of MLLMs, concentrating on two primary aspects: the extent to which existing benchmarks accurately measure multimodal reasoning, and the effect of LLMs' prior knowledge on performance. The paper introduces several metrics to disentangle visual perception capabilities from language prior knowledge. The study is conducted on four public MLLM benchmarks and eight open-source MLLM models. Furthermore, the authors introduce a RAG pipeline to enhance the performance.
weaknesses: - According to lines 52-55, the reviewer understands that the author aims to focus on two aspects: the influence of language knowledge on the benchmark and the MLLM models, respectively. However, the metrics introduced are all dependent on specific MLLM models, making it impossible to objectively assess the language effects on the benchmark itself. The reviewer suggests that it would be more effective to use ground-truth (human annotation) to address the authors' first question: to what extent do current benchmarks truly assess multimodal reasoning versus relying solely on language capabilities?

- While it is important to disentangle the language prior from MLLM evaluation, the task has already been considered in the construction of existing MLLM benchmarks, for example, SEED-Bench [A] (Sec. 3.3, "... For verifying question/answer pairs, we filter out questions that can be answered correctly by multiple LLMs without resorting to visual information ..."). However, such works have not been discussed in the paper.
[A] SEED-Bench: Benchmarking Multimodal Large Language Models. CVPR 2024.

- The reviewer acknowledges the empirical contributions of this work. However, it offers limited new knowledge to the community. Specifically, the impact of language prior on MLLM evaluation has already been addressed in existing benchmarks (see Weaknesses #2). Additionally, the introduced metrics rely on specific MLLM models without incorporating new ground-truth annotations (see Weaknesses #1), and the RAG pipeline presented is an established technique. The quality of the paper would be significantly enhanced if the authors could reconstruct existing popular benchmarks to disentangle them from language prior and place greater emphasis on multimodal reasoning.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The role of LLM in MLLM is a very important topic, and this paper provides an analytical tool on that topic. On the other hand, as noted in Weakness, what this paper examines is merely a more comprehensive examination of what is already known, and the findings are not surprising.

I would like to know why existing evaluations are not enough and what are the novel findings through this analysis. And, also, I would like to know what would the results be if the author used newer benchmarks for evaluation.
soundness: 3
strengths: - S1. The analysis of the contribution of the Language side in MLLM is of great community interest and is one of the key topics


- S2. Incorporating metrics such as SR, GR, SuRD, and NeRD enables more detailed analysis. It will be an important analytical tool.
summary: This paper investigates the role of LLM backbones for MLLM in the perspective of which current benchmarks truly evaluate multimodal reasoning and the effect of LLM prior knowledge.  To achieve this, this paper introduces a modified evaluation protocol to elucidate the contribution of the LLM's backbone and an automatic knowledge identification technique to diagnose whether the LLM has the necessary knowledge for the question. This paper reveals that some benchmarks allow high performance without image inputs and many error rates can be attributed to insufficient LLM capabilities. Furthermore, this paper proposes a RAG pipeline for knowledge-based VQA tasks.
weaknesses: - W1. The author's investigation into image-agnostic questions within MLLM benchmarks confirms findings from previous research [1]. Although this analysis provides a more in-depth view, the results that some benchmarks allow high performance without image inputs are not unexpected. Similarly, it is widely recognized that an MLLM’s performance depends on LLM knowledge capacity and generally improves with LLM’s size. Therefore, the finding that “many error rates can be attributed to insufficient LLM capabilities” is not surprising.


- W2. It would be beneficial to propose how to construct the image-dependent questions. For example, this paper could create a benchmark from existing datasets that requires the use of visual information to answer the questions correctly. It would be more beneficial to the community if such a benchmark could be created along with Analysis.

- W3. The evaluation dataset is not new. I would like to know how the proposed metric works on a more recent dataset such as MMStar [1].  MMStar consists of image-dependent questions. Therefore if new issues are identified using the analysis tools in this study, new findings can be given in terms of how to create image-dependent problems. 


- W4. The solution for the LLM's knowledge lacking is also just using RAGs, so there is limited novelty.


[1] Chen+, Are We on the Right Way for Evaluating Large Vision-Language Models?, NeurIPS2024

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - As I am not fully convinced of the novelty of this work, I would like to ask the authors how their discoveries regarding the necessity of visual information differ from prior works.
- Moreover, I would like to ask the authors whether one could infer the second discovery by evaluating the performance of different pre-trained language models with vision capabilities at different scales (e.g., LLAMA 7B, 30B, etc) without relying on retrieval.
- I expect that adding vision to a language model after pretraining shouldn't diminish its existing knowledge. This means we could evaluate the sufficiency of prior language knowledge for multimodal tasks as if they were language-only tasks. I ask the authors to discuss this and include tables linking knowledge sufficiency results from multimodal benchmarks to language model performance on language tasks, i.e., for models adding vision post-pretraining, report their performance on language benchmarks, and check for correlations between them and multimodal knowledge sufficiency.
soundness: 2
strengths: - The work is well-presented and motivated, and the writing is clear.
- The experiments cover four multi-modal datasets and eight multimodal language models.
- The authors introduced some changes to previous evaluation protocols (i.e., options shuffling, open-ended conversion) to minimize biases.
- The two main discoveries present valuable insights into how to design new benchmarks and what to focus on when training multi-modal language models.
summary: The paper studies the contribution of language models in multi-modal evaluation benchmarks. The authors aim to quantify the importance of language in the performance on such datasets in two ways: (i) quantify shortcut reasoning (i.e., how much these models look at the image/how many of these questions do not require image understanding), and (ii) quantify the role of prior knowledge in answering these questions (i.e., understand whether more knowledgeable backbones provides better performance). The authors select four benchmark datasets and eight models to investigate these two aspects. As for prior works, they confirm that language models tend to exploit shortcuts, i.e., do not always use visual information to answer. Moreover, they discovered that the performance on the selected benchmarks heavily depends on prior knowledge of the language model.
weaknesses: - The analysis of visual capability necessity was already studied in the literature (as also noted by the authors), leaving only the second discovery (i.e., more language knowledge implies better multimodal performance) as the main discovery of the work. Moreover, one could argue that this second point is somewhat intuitive and is more of a confirmation of it.
- The work acknowledges using only a subset of the MMMU benchmark due to limitations in the models' support for multi-image input and the complexity of converting certain questions into corresponding knowledge reasoning tasks.
- Experiments regarding different retrieval approaches are missing, e.g., using different retrieval approaches and potentially exploiting multi-modal information.

Meta-review: Paper was reviewed by four expert reviewers and received 3 x marginally below the acceptance threshold and 1 x  reject, not good enough ratings. Overall, all reviewers agree that the paper is lacking in a couple of core aspects, mainly: (1) the novelty is marginal, (2) it offers limited new insights to the community (given prior work); and (3) architectural improvements are limited to introducing RAG. Authors have provided rebuttal to address some of the concerns. Reviewers, however, found author responses only partially satisfactory with many concerns being unaddressed. As a result, the rebuttal did little to sway the reviewer opinion. 

AC has looked at the reviews, rebuttal, discussion, as well as briefly the paper itself. Overall, AC agrees with the reviewer consensus that paper is not ready for publication in ICLR in its current form. The decision therefore is to recommend Rejection at this time.

Decision: Reject