Why context matters in VQA & Reasoning: Semantic interventions for VLM input modalities

Authors: Kenza Amara, Lukas Klein, Carsten T. Lüth, Paul F Jaeger, Hendrik Strobelt, Mennatallah El-Assady
Primary area: datasets and benchmarks
Keywords: Vision Language Model, Vision Question Answering, model failure, multimodality, interpretability, semantic intervention

Abstract:
The various limitations of Generative AI, such as hallucinations and model failures, have made it crucial to understand the role of different modalities in Visual Language Model (VLM) predictions. Our work investigates how the integration of information from image and text modalities influences the performance and behavior of VLMs in visual question answering (VQA) and reasoning tasks. We measure this effect through answer accuracy, reasoning quality, model uncertainty, and attention attribution to each modality. We study the interplay between text and image modalities in different configurations where visual content is essential for solving the VQA task. Our contributions include (1) the Semantic Interventions (SI)-VQA dataset, (2) a benchmark study of various VLM architectures under different modality configurations, and (3) the Interactive Semantic Interventions (ISI) tool. The SI-VQA dataset serves as the foundation for the benchmark, while the ISI tool provides an interface to test and apply semantic interventions in image and text inputs, enabling more fine-grained analysis. Our results show that complementary information between modalities improves answer and reasoning quality, while contradictory information harms model performance and confidence. Image text annotations have minimal impact on accuracy and uncertainty, slightly increasing image relevance. Attention analysis confirms the dominant role of image inputs over text in VQA tasks. In this study, we evaluate state-of-the-art VLMs that allow us to extract attention coefficients for each modality. A key finding is PaliGemma's harmful overconfidence, which poses a higher risk of silent failures compared to the LLaVA models. This work sets the foundation for rigorous analysis of modality integration, supported by datasets specifically designed for this purpose. The code is available at https://gitlab.com/dekfsx1/si-vlm-benchmark and the tool and dataset are hosted at https://gitlab.com/dekfsx1/isi-vlm.

Decision: reject
Reviewer scores: 6, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729498710595

thread_type:
official_review

<thread_object>
creator: reviewer_9n8l

<content>
summary:
This paper presents to evaluate the robustness of VLLMs.
In particular, there are two dimensions that the advocated evaluation protocol considers: 
1) modality bias - whether VLLMs make predictions based on the linguistic relations;
2) context - whether the context helps in reasoning. 

Based on this idea, this paper collects a new dataset and then evaluates various VLLMs, including LLaVA, and Pali-Gemma.
Besides, the authors also provide some analysis from the dimension of semantic entropy and attention distribution.

strengths:
- The studied problem - the robustness of VLLMs, is practical and interesting for the research community.
- The authors adopt two different families of models for evaluation, including both LLaVA and Pali-Gemma.
- There are some more dimensions that are considered by this paper, like semantic entropy and attention distribution.

weaknesses:
- The biggest limitation of this paper lies in its limited dataset size. 
Specifically, there are only 100 instances of the collected dataset.
From this point of view, most of the conclusions from this work might be plausible and not stand.
Additionally, we cannot name a scale of such a dataset as ``comprehensive``.
- The authors are suggested to test larger model sizes, such as 13B models - LLaVA-1.5-vicuna-13B.
- It seems like there is a strong connection between this work and several well-studied problems such as modality bias (language prior) in VQA [1][2], and visual commonsense reasoning (VCR) [3].

[1] On Modality Bias Recognition and Reduction. 

[2] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning.

[3] From Recognition to Cognition: Visual Commonsense Reasoning.

questions:
See the weakness part for a detailed explanation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620508133

thread_type:
official_review

<thread_object>
creator: reviewer_gfmm

<content>
summary:
The paper investigates the limitations of Generative AI, particularly in Visual Language Models (VLMs), focusing on how the integration of image and text modalities affects performance in visual question answering (VQA) and reasoning tasks. They use only 100 samples to gain some conclusions in the paper, such as "complementary information between modalities improves answer and reasoning quality".

strengths:
1. This paper is well-written and easy to understand.
2. The experimental analysis is comprehensive.
3. The conclusions drawn are intuitively credible.

weaknesses:
1. The dataset contains only 100 samples, and the conclusions drawn lack novelty; they are basic findings that have been established in previous multimodal research. The importance of multimodal complementarity is widely recognized in the field, so the conclusions of this article lack originality.

2. Overall, this article is a fairly good technical report that provides a comprehensive experimental analysis.

questions:
Are there any other interesting conclusions or exploratory directions for uncovering the importance of multimodal complementarity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730778553400

thread_type:
official_review

<thread_object>
creator: reviewer_gzvh

<content>
summary:
This paper explores the impact of contextual information on Visual Question Answering (VQA) and reasoning within Vision-Language Models (VLMs). The study introduces the Semantic Interventions (SI)-VQA dataset and the Interactive Semantic Interventions (ISI) tool to evaluate how image and text modalities interact to affect model performance, accuracy, and uncertainty. The methodology involves benchmarking multiple VLM architectures under different configurations, integrating complementary or contradictory text with images. Experimental results indicate that integrating complementary information enhances model accuracy and reasoning quality, whereas contradictory information significantly degrades performance. Moreover, VLMs show a bias toward image inputs over textual context, with PaliGemma exhibiting notable overconfidence, leading to increased silent failures compared to LLaVA models. The study emphasizes the crucial role of modality integration and provides tools for better understanding VLM behavior in multimodal tasks.

strengths:
1)	The impact of context and different modalities on VQA has always been a noteworthy topic. This paper's discussion, incorporating VLM, is insightful for future researchers.
2)	The experimental design in this paper is very thorough, with detailed consideration given to seven different input configurations.
3)	Some of the findings in the experimental results of this paper are very interesting and offer valuable insights for the design and application of future VLMs.
4)	This paper has released the dataset and the ISI tool.

weaknesses:
1)	My primary concern is that the paper mainly describes the observed phenomena in the experimental results without providing sufficient analysis of why these results occur (though there is some experimental analysis). In particular, the paper does not explain how these results could be useful for advancing future VQA work or analyze what could be done to address some of the issues identified in the results. Additionally, some of the findings are not particularly novel, making the paper seem more like an experimental report.
2)	As the authors pointed out in the paper, the SI-VQA dataset has too few samples, with only one hundred entries. Although the authors believe these data are representative, they should at least analyze why the results from these one hundred samples are convincing. Is it because these one hundred samples are of high quality and diversity?

questions:
1)	It appears that image text annotations have little effect on some of the model's metrics; for example, the results of Q+I_A+C_+ in Figure 2a are not optimal. Could the authors analyze the reason for this phenomenon?
2)	I don't quite understand why the initial hypothesis is introduced in Section 5.1, as it doesn't seem to be strongly related to the main part of the experiments.
3)	Could the authors explain specifically how GPT-4o is used as an evaluator of reasoning ability? Since the SI-VQA dataset has only 100 samples, why didn’t the authors consider using human evaluation instead? Would that provide more accurate results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730799393035

thread_type:
official_review

<thread_object>
creator: reviewer_mkz9

<content>
summary:
This work investigates the impact of different modalities – image and text – on the performance of Visual Language Models in Visual Question Answering (VQA) tasks. 
The authors examine how the combination and interplay of these modalities affect accuracy, reasoning quality, model uncertainty, and attention attribution. 
They collect a novel dataset (SI-VQA) with controlled interventions and an interactive tool (ISI) for manipulating image and text inputs to study VLM behavior. 
This work sets the foundation for further analysis of modality integration in VQA, hightlighting the crucial role of context in guiding future model developments.

strengths:
1. This work introduces the SI-VQA dataset, which is designed to require image-based answers, ensuring that visual content is essential for solving VQA tasks. This setup allows researchers to analyze how different modalities (image, text, context) influence the model’s accuracy, reasoning, and uncertainty​.

2. Comprehensive Benchmarking of VLMs: This work establishes a robust benchmark by evaluating various state-of-the-art VLMs under diverse modality configurations. This benchmarking approach highlights the contributions and limitations of each modality, as well as the strengths and weaknesses of different VLM architectures.

3. This work introduces the ISI Tool, enabling researchers to perform semantic interventions on VLM inputs, which supports fine-grained analysis of VLM behavior.

weaknesses:
Please also refer to the questions section.

1. There are some concerns about the dataset.
2. The work lacks comparisons with other current datasets.
3. The work lacks supporting evidence for its claims.
4. The work lacks formal definitions of certain terms.

questions:
Here are the corrected versions of the reviews:

1. The proposed dataset contains only 100 samples, which is quite limited in this domain.

2. The answers are limited to "Yes" or "No." Moreover, the paper does not specify the distribution of "Yes" versus "No" answers in the dataset. This leads to the following two concerns:

   - **Model Bias**: If the dataset is heavily skewed toward one answer (e.g., mostly "Yes" answers), it could introduce bias in the models, potentially leading them to favor that answer even when the visual information suggests otherwise.

   - **Impact of Interventions**: Without knowing the baseline distribution of answers, it is challenging to isolate the true effect of the semantic interventions (complementary context, contradictory context, image annotations) on the models' performance. For example, if the dataset already has a majority of "Yes" answers, an intervention that improves performance on "Yes" questions might not necessarily reflect a genuine improvement in the model's ability to understand the visual information.

3. Even though each sample is well-annotated (i.e., an image, a corresponding question with a ground truth Yes/No answer, a text-annotated version of the image, a contradictory context, and a complementary context), there are no comparisons between the proposed dataset and state-of-the-art (SOA) datasets regarding its advantages in Image-dependent Answers and Content Domain Diversity.

4. Regarding the claim that image text annotations have minimal impact on accuracy, or even decrease accuracy, the authors list some potential reasons for this, e.g., VLMs may already extract relevant information from images. It would be helpful to provide some qualitative or quantitative results to further support these explanations.

5. The term "modality relevance" is first mentioned in the abstract. However, there is no formal definition provided for it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732214127375

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Answer to Everyone - PART 1/3

comment:
We sincerely thank all reviewers for their valuable comments. We appreciate the general consensus among the reviewers regarding the significance of our work for the community and the and the thoroughness of the experiments: "experimental design in this paper is very thorough" (Reviewer gZVH), "The experimental analysis is comprehensive and the conclusions drawn are intuitively credible" (Reviewer gFMm) or "there are some more dimensions that are considered by this paper [...]" (Reviewer 9N8L).

We acknowledge that the importance of multimodal complementarity is a well-established concept in the field. However, our work brings a unique perspective and addresses gaps in the existing literature.

Thus, next to the point-by-point responses to each reviewer, we will address the four main concerns here:

**tl;dr**\
The main criticism raised by all reviewers concerns the size of the SI-VQA dataset. Here, **(1)** we aim to clarify that SI-VQA was not intended to be a large-scale dataset for benchmarking VLMs. We will elaborate on why its limited size is appropriate and does not hinder the objectives and scope of this research. **(2)** We also want to emphasize our development of an interactive interface, the ISI tool (Appendix E), designed to engage the AI community in actively contributing to the expansion and diversification of the SI-VQA dataset. **(3)** Additionally, we emphasize the distinct focus of our work compared to previous studies addressing related topics, such as modality bias, VQA reasoning, and VLM interpretability. We highlight our contributions and novelties, demonstrating how our work advances research in explainable AI (XAI) and VLM interpretability. **(4)** Finally, we reiterate what our paper does not aim to achieve, ensuring that reviewers are clear on the objectives of our study.


1.  **Dataset Size**
Regarding the dataset size:

-   **Evaluation purpose.** We emphasize that our dataset is only used for evaluation, not for training or fine-tuning models. Its purpose is to analyze pretrained models' behavior when faced with diverse input configurations. 

-   **TinyBenchmark.** While the SI-VQA dataset we designed remains small, prior recent research [[1]](https://arxiv.org/pdf/2402.14992) demonstrates that smaller datasets can still yield meaningful results when the quality of instances is high. Each instance of the SI-VQA Dataset underwent multiple rounds of refinement, ensuring that context descriptions serve only as background information without providing enough detail to answer the question independently.

-   **Low Variance.** Despite the small size, the dataset exhibits low variance in model performance and uncertainty, as demonstrated in the Appendix H. Increasing the number of instances would not alter the conclusions drawn, as they hinge on the modality configurations rather than the dataset size. Instead, rebalancing the role of modalities, such as prioritizing text and introducing spurious visual information, would yield new insights---a direction we are actively exploring.

-   **Scalability and Community Engagement.** To ensure data quality and consistency, we intentionally limited the dataset size to 100 well-curated instances. However, we have also developed a ready-to-use interactive interface that allows the community to generate additional instances with new images and textual descriptions. This ensures that our SI-VQA dataset is extensible and can grow with greater diversity and community input. See Appendix E for details about the Interface and instructions to create new data instances and the link to the code to try it out: https://gitlab.com/dekfsx1/isi-vlm.

[1] [tinyBenchmarks: evaluating LLMs with fewer examples](https://arxiv.org/pdf/2402.14992)

*In the revised version of the paper, we integrated this argumentation in the Limitations and Future Directions paragraph in the Discussion.*
</content>
</thread_object>

</thread>
<thread>
creation date:
1732214217871

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Answer to Everyone - PART 2/3

comment:
2. **ISI Tool**

The SI-VQA dataset comprises 100 carefully curated instances, but we want to emphasize the significant contribution of the interactive interface we developed: the ISI Tool (Appendix E). This tool allows users to design their own semantic interventions and create new benchmarks, complementing the SI-VQA dataset, which serves as the backbone for our sensitivity analysis with high-quality data.

As detailed in Appendix E, the ISI Tool enables users to load new images and questions, apply various semantic interventions (e.g., occlusions, annotations, shape additions), and craft custom text descriptions to guide or mislead the model to varying degrees. While this contribution may not have been highlighted by reviewers, we believe it is a major asset for the vision-language AI community. The tool facilitates further exploration of modality bias, modality importance, and the role of multimodal complementarity.

You can try the interface yourself at https://gitlab.com/dekfsx1/isi-vlm to see how it represents a novel and valuable addition to the field. Notably, our previous tool-focused paper was accepted at six NeurIPS workshops, demonstrating the community's enthusiasm for such tools that empower collaborative contributions to enriching datasets and advancing understanding in the field.

3. **Novelty and Contribution**

**A Dataset for Model Behavior Analysis - Not A New Large-scale VLM Benchmark**

Our dataset and methodology are not designed to benchmark large vision-language models (VLMs) but to probe their behavior in specific, controlled multimodal scenarios. This unique focus on modality roles and interactions represents an important step toward understanding the limits and strengths of VLMs.

**Unique modality configurations**

Prior studies have explored modalities but face limitations like data leakage (e.g., MMMU) or oversimplified multimodal tasks (e.g., MMStar). MMStar, for instance, involves basic multi-class classification with minimal reasoning, sparse textual descriptions, and OCR-based dependencies. Many large-scale datasets also lack complete data, including ground-truth rationales for VLM answers, and feature noisy or annotated images that hinder controlled analysis. Question complexity often varies, creating inconsistency. In contrast, our curated dataset addresses these issues with natural, annotation-free images, consistently complex questions, independent text descriptions, and well-defined modality scenarios. This enables robust analysis of model performance, uncertainty, and attention mechanisms in reasoning contexts.

**Contribution to the Advancement of Vision Language Model Interpretability & Robustness**

Understanding how modalities impact VLMs by exploring complex and specific scenarios is crucial as multimodal models become increasingly opaque and diverse. Our work demonstrates how task-specific modality configurations can reveal heterogeneous effects on model performance and uncertainty. This approach not only advances our understanding of VLMs but also highlights areas where these models require improvement to achieve robustness in diverse multimodal tasks.

**Other contributions to the Advancement of the explainable AI community**

Despite the novelty aspects enumerated in the general answer to all reviewers, the conclusions of our paper are useful for advancing future VQA work:

-   **Attention is not Explanation.** Attention distribution on input tokens does not correlate to VLM performance - see ablation study about using prompt engineering to steer model's attention toward the context and Figure 6. This conclusion from the paper contributes to the ongoing debate about the role of attention and whether it is a form of explanation. Our observations support the

-   **Attention and Number of Tokens.** Our findings suggest that attention to a modality is not necessarily determined by the number of tokens associated with it. For instance, despite LLaVA-Next having approximately six times more image tokens, its attention toward those tokens remains comparable to models with fewer tokens -- like LLaVA-1.5.

**Future directions to address observations made in the paper.**

A promising avenue for further research is investigating tasks where both modalities provide semantically distinct yet equally essential information. By analyzing scenarios where successful task completion (e.g., answering a question) requires complementary inputs from both modalities, we can deepen our understanding of how models integrate and prioritize multimodal information.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732214248454

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Answer to Everyone - PART 3/3

comment:
4. **What our paper is not**

To prevent any confusion about the nature and scope of our research paper, we would like to remind reviewers and Area Chair what our paper is not about:

1\. We are not proposing a new large-scale multimodal benchmark aimed at achieving state-of-the-art results in multimodal learning.

2\. We are not attempting to resolve modality bias.

In summary, our work provides new insights into modality roles in VLMs and addresses a unique, underexplored aspect of multimodal research. We thank the reviewer for the opportunity to clarify our contributions and the broader context of our work.

*We have modified the Introduction of the revised version of the paper to clarify the scope of the paper, novelties, and contributions. In the Discussion section, we elaborated more on the arguments about the dataset's reduced size.*

For more detailed changes, see the reviewer's individual answers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732566483451

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Deadline Reminder

comment:
As the discussion phase is approaching its end soon, we wanted to kindly check if you have any remaining points for us to address. If we do not receive any further comments on our rebuttal, we assume that all points of criticism have been addressed appropriately.
We appreciate your valuable feedback and are happy to clarify anything further if needed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733175449355

thread_type:
official_comment

<thread_object>
creator: area_chair_pu2v

<content>
title:
Dec 2: the last day for reviewers to ask questions to authors

comment:
Dear Reviewers,

Thank 9N8L for confirming the authors' responses and update the review.

Dear MkZ9, gFMm, gZVH, as Dec 2 is the last day for reviewers to ask questions, can you take this chance to check authors' responses and other reviewers' comments and confirm whether you have any remaining concerns? Your constructive discussion and suggestions would be great contributions to the review process.

Best,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: This work discussed about the impact of image and text on VLMs for VQA tasks, especially the impacts on accuracy, reasoning quality, model uncertainty, and attention attribution. A high-quality dataset SI-VQA and an interactive tool (ISI) are provided to study VLM behavior.

Main Strengths: (1) The impact of context and different modalities on VQA is important research question. (2) The experimental design is very thorough. (3) The Interactive Semantic Interventions (ISI) tool is introduced to support further analysis of VLMs. 

Major Weaknesses & Concerns: (1) The proposed Semantic Interventions (SI-)VQA dataset only contains 100 instances. (2) The answers are limited to binary answers, yes and no. (3) Experimental analysis is not strong or insightful enough for future network designs, and validness of the experimental analysis is not clear, which limits the contributions of the work.

This paper received ratings as 6, 5, 5, 3. AC read reviewers' comments and authors' responses, exclude short review for final decisions, and think the current version of the paper is borderline. Considering the commonly shared concern on scale of dataset, the AC does not recommend accepting the paper and encourage the author to further improve the quality and quantity of evaluation dataset for more convincing analysis.