Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality

## Abstract

Abstract Multimodal Large Language Models (MLLMs) have emerged as a central focus in both industry and academia, but often suffer from biases introduced by visual and language priors, which can lead to multimodal hallucination. These biases arise from the visual encoder and the Large Language Model (LLM) backbone, affecting the attention mechanism responsible for aligning multimodal inputs. Existing decoding-based mitigation methods focus on statistical correlations and overlook the causal relationships between attention mechanisms and model output , limiting their effectiveness in addressing these biases. To tackle this issue, we propose a causal inference framework termed CausalMM that applies structural causal modeling to M LL M s, treating modality priors as a confounder between attention mechanisms and output. Specifically, by employing backdoor adjustment and counterfactual reasoning at both the visual and language attention levels, our method mitigates the negative effects of modality priors and enhances the alignment of MLLM’s inputs and outputs, with a maximum score improvement of 65.3% on 6 VLind-Bench indicators and 164 points on MME Benchmark compared to conventional methods.
Extensive experiments validate the effectiveness of our approach while being a plug-and-play solution. Our code is available at: https://github.com/The-Martyr/CausalMM .

## 1 Introduction

Figure: Figure 1: The comparison of conventional hallucination mitigation paradigm (e.g., VCD) and our proposed CausalMM.
Refer to caption: /html/2410.04780/assets/comparison_paradigm.png

Recent research on Multimodal Large Language Models (MLLMs) has achieved great progress in diverse applications (Yin et al., ; Jin et al., ; Yan et al., ; Zou et al., ), particularly due to their reliance on Transformer models (Vaswani, ), where performance is driven by the attention mechanism (Hassanin et al., ). In particular, such a mechanism enables the model to assign weights to input information, such as images and text, guiding the generation of outputs. However, the inherent bias in the initial parameters of the model, namely the modality priors, can negatively impact output quality via the attention mechanism (Tong et al., ; Zhao et al., ; Lee et al., ; Chen et al., ). In widely used MLLM architectures, attention that most significantly influences output can be divided into two components: visual encoder attention and Large Language Model (LLM) backbone attention (Liu et al., ).
The parametric knowledge of the visual encoder (i.e., visual priors) affects the alignment of multimodal information by affecting the visual encoder’s attention (Tong et al., ).
Similarly, the knowledge embedded in the LLM’s parameters, referred to as language priors, may compromise the model’s fidelity to multimodal inputs through attention (Lee et al., ). These biases, stemming from the visual encoder and the MLLM’s over-reliance on language priors, may lead to issues such as multimodal hallucinations, ultimately degrading model performance (Yang et al., ).
Several approaches have been proposed to enhance model output without modifying the model weights (Leng et al., ; Huang et al., ). However, as illustrated in Figure [1](#S1.F1) (a), existing decoding strategies primarily rely on statistical correlations and predetermined conclusions from posterior analysis to optimize outputs, overlooking the causal relationships among visual attention, language attention, modality priors, and model output. In this context, the attention mechanism adjusts weights solely based on parameter knowledge, which limits the model’s ability to comprehend underlying dependencies in the reasoning process, exacerbates bias, leading to problems such as multimodal hallucinations.

Modality priors are one of the confounding factors in the causal path of MLLM. We introduce a causal reasoning framework CausalMM, which can help us better capture the causal impact of effective attention on MLLM output in the presence of these confounding factors, thereby improving the performance of multimodal tasks, as shown in Figure [1](#S1.F1) (b). Specifically, we construct a structural causal model (Pearl, ) for MLLM, and use intervention and counterfactual reasoning methods under the backdoor adjustment paradigm to derive the causal effects of visual and language attention on the model output despite the confounding effect of modal priors. The CausalMM method is based on counterfactual reasoning at the visual and language attention levels, which ensures that the model output is more consistent with the multimodal input, thereby mitigating the negative impact of modal priors on performance. Experimental results show that CausalMM significantly reduces modal prior bias and improves performance on different tasks, improving 143.7 points on 6 indicators of VLind-Bench, 164 points on the MME Benchmark, and an average improvement of 5.37% on the three benchmarks of POPE.

Our key contributions can be summarized as follows: ❶ We have constructed a structural causal framework called CausalMM flexible for any MLLM, exploring the issues of visual and language priors within the framework. ❷ We apply counterfactual reasoning at the levels of visual and language attention, making the output more aligned with multimodal inputs. ❸ Through comprehensive experiments, we have demonstrated the superior performance of our method in alleviating MLLM hallucinations. In addition, our framework is plug-and-play, and can be integrated with other training-free methods for further improvement.

## 2 Related Works

Multimodal Large Language Models.
In recent years, MLLMs have seen significant advancements (Yin et al., ; Jin et al., ; Huo et al., ; Yan & Lee, ). Notable works include VITA (Fu et al., ), the first open-source MLLM capable of processing video, image, text, and audio, demonstrating robust performance across various benchmarks. Cambrian-1 (Tong et al., ) is a family of MLLMs designed with a vision-centric approach, achieving state-of-the-art performance and providing comprehensive resources for instruction-tuned MLLMs. Additionally, research on training-free reasoning stage improvements, such as VCD (Leng et al., ) and OPERA (Huang et al., ), has focused on leveraging human experience to enhance model performance without additional training (Li et al., ; Zheng et al., ). In this work, we manage to apply causal reasoning (Pearl, ) to make the MLLM automatically optimize the output.

Causal Inference in Multimodal Learning.
The field of causal inference has seen significant advancements (Pearl, ; Xu et al., ), particularly in the context of LLMs and vision systems (Zhang et al., ; Rao et al., ). Researchers have explored the integration of causal reasoning to enhance the interpretability and robustness of these models (Xu et al., ). For instance, LLMs have been shown to generate accurate causal arguments across various tasks, surpassing traditional methods (Kıcıman et al., ). A comprehensive survey has highlighted the potential of causal inference frameworks to improve reasoning capacity, fairness, and multimodality in LLMs (Liu et al., ). Additionally, recent work showcased the use of LLM-guided discovery to significantly improve causal ordering accuracy (Vashishtha et al., ). Different from previous attempts, we tend to use causal reasoning to balance the visual priors and language priors of the model output.

Modality Priors. Research on modality priors in MLLMs has seen significant advancements (Tong et al., ). Studies focused on overcoming language priors by integrating visual modules, enhancing the impact of visual content on model outputs. For instance, Zhao et al. () proposed a method to improve visual content in Visual Question Answering (VQA) tasks, which proved effective across multiple datasets. Additionally, benchmarks like VLind-Bench (Lee et al., ) have been developed to measure language priors in MLLMs, revealing a strong reliance on textual patterns. On the other hand, visual priors have been addressed by augmenting off-the-shelf LLMs to support multimodal inputs and outputs through cost-effective training strategies (Zhang et al., ).

## 3 Methodology

In this section, we construct a structural causal model of MLLM and generate different counterfactual attentions through intervention for counterfactual reasoning based on the backdoor criterion.

### 3.1 Structural Causal Model

We construct a structural causal model (SCM) to describe the relationships among various components of a MLLM (Yang et al., ; Pawlowski et al., ). In particular, our SCM captures the interactions between the visual and language modalities by modeling causal dependencies among input image ($I$), visual attention ($A_{i}$), visual token embeddings ($T_{i}$), language token embeddings ($T_{t}$), language priors ($P_{l}$), visual priors ($P_{v}$), MLLM attention ($A_{t}$), and model output ($O$).

The causal graph is formulated as follows:

- •
$I\rightarrow A_{i}$: The image input $I$ influences the visual attention layer $A_{i}$.
- •
$I\rightarrow T_{i}$: The image input $I$ directly affects the visual token embeddings $T_{i}$.
- •
$P_{v}\rightarrow A_{i}$: Visual priors $P_{v}$ contribute to the attention in the visual attention module.
- •
$P_{v}\rightarrow T_{i}$: Visual priors $P_{v}$ also influence the formation of visual token embeddings $T_{i}$.
- •
$A_{i}\rightarrow T_{i}$: Visual attention $A_{i}$ impacts the encoding of visual tokens.
- •
$T_{i}\rightarrow O$: Visual tokens $T_{i}$ contribute directly to the model’s output.
- •
$T_{t}\rightarrow A_{t}$: Language token embeddings $T_{t}$ influence the MLLM’s attention $A_{t}$.
- •
$T_{t}\rightarrow O$: Language token embeddings $T_{t}$ directly impact the final output.
- •
$P_{l}\rightarrow A_{t}$: Language priors $P_{l}$ inform the MLLM’s attention mechanism $A_{t}$.
- •
$P_{l}\rightarrow O$: Language priors $P_{l}$ directly affect the model output $O$.
- •
$A_{t}\rightarrow O$: LLM attention $A_{t}$ shapes the final output $O$.

In this causal graph, both visual priors ($P_{v}$) and language priors ($P_{l}$) serve as confounding factors, influencing the attention layers and embedding representations in both modalities. These priors are mixed into the model and can lead to biased outputs. Our goal is to quantify the causal effect of visual attention ($A_{i}$) and language attention ($A_{t}$) on the model output ($O$), while accounting for these confounding effects through intervention and counterfactual reasoning.

Figure: Figure 2: Causal diagram of counterfactual reasoning. ❶ In vision-only counterfactual reasoning, we only intervene in visual attention (i.e., the attention of the visual encoder). ❷ In language-only counterfactual reasoning, we only intervene in the multi-head self-attention of LLM. ❸ In multimodal collaborative counterfactual reasoning, we intervene in both visual and language attention at the same time and obtain the sum of their collaborative causal effects.
Refer to caption: /html/2410.04780/assets/x1.png

### 3.2 Intervention on Multimodal Attentions

We perform specific interventions on the attention layers of both the visual and language components to investigate their causal effects on the model’s output. These interventions modify the attention weights to generate counterfactual outputs, allowing us to isolate the impact of each modality.

For visual attention, we intervene by replacing the original attention map $A_{i}$ with a counterfactual state $A_{i}^{*}$, expressed as $do(A_{i}=A_{i}^{*})$. The counterfactual state $A_{i}^{*}$ can take various forms, such as random attention weights, uniform distributions, reversed scores, or shuffled attention maps. Each configuration reveals different aspects of how visual attention influences the output, independent of other factors like the image $I$ and visual processing $P_{v}$.

Similarly, we intervene in the language attention by applying $do(A_{t}=A_{t}^{*})$, where $A_{t}^{*}$ represents alternative attention states that allow us to explore the impact of the language attention module on the final output, free from the influences of $T_{t}$, $T_{i}$, and $P_{l}$.

The counterfactual attention states are specified as follows:

- 1.
Random Attention: Replace the original attention scores with random values drawn from a uniform distribution. For the visual encoder, attention scores $A_{i}(h,w)$ at spatial locations $(h,w)$ are replaced as follows:
$A^{\prime}_{i}(h,w)=\mathcal{U}(0,1)\cdot\sigma\cdot\alpha_{v},$
(1)
where $\mathcal{U}(0,1)$ is a random variable drawn from a uniform distribution, $\sigma$ represents the scaling factor for attention, and $\alpha_{v}$ denotes the normalization parameter. Similarly, for the language model, the random attention values $A_{t}(n)$ over tokens $n$ are given by:
$A^{\prime}_{t}(n)=\mathcal{U}(0,1)\cdot\beta\cdot\alpha_{l},$
(2)
where $\beta$ is the language attention scaling factor and $\alpha_{l}$ is the language normalization term.
- 2.
Uniform Attention: Assign a constant value to all attention scores. For the visual encoder, the attention at location $(h,w)$ is replaced by the average value:
$A^{\prime}_{i}(h,w)=\frac{1}{H\times W}\sum_{h,w}A_{i}(h,w)+\epsilon,$
(3)
where $H$ and $W$ represent the height and width of attention map, and $\epsilon$ is a small perturbation added to avoid exact uniformity. For the language model, the attention over $N$ tokens is distributed as:
$A^{\prime}_{t}(n)=\frac{1}{N}\sum_{n=1}^{N}A_{t}(n)+\delta,$
(4)
where $\delta$ is a small constant ensuring numerical stability.
- 3.
Reversed Attention: Invert the attention map by subtracting each attention score from the maximum value of the map. For the visual encoder:
$A^{\prime}_{i}(h,w)=\max(A_{i})-A_{i}(h,w)+\lambda,$
(5)
where $\lambda$ is an offset parameter to control the inversion. For the language model:
$A^{\prime}_{t}(n)=\max(A_{t})-A_{t}(n)+\zeta,$
(6)
where $\zeta$ is the inversion factor for language attention.
- 4.
Shuffled Attention: Randomly permute the attention scores across spatial locations for the visual encoder. The new attention map $A^{\prime}_{i}$ is created by permuting the original scores $A_{i}$:

$$ $A^{\prime}_{i}(h,w)=A_{i}(\pi(h),\pi(w)),$ (7) $$

where $\pi(h)$ and $\pi(w)$ are random permutations of the height and width indices. This intervention is specific to the visual encoder and does not apply to the language model, as token order is significant in language processing.

By conducting these interventions, we can observe the independent contributions of both visual and language attention to the model’s output, controlling for confounding factors such as the image $I$, the tokens $T_{t}$, and the model’s intermediate representations $P_{v}$ and $P_{l}$.

### 3.3 Counterfactual Reasoning

To formalize the impact of counterfactual interventions on the model output, we perform counterfactual reasoning based on the backdoor adjustment principle (Pearl, ; Li et al., ; Adib et al., ; Zhang et al., ). The backdoor criterion ensures that we properly account for confounding factors ($I$, $P_{v}$, $P_{l}$) when estimating the causal effect of attention mechanisms.
To measure the causal effect of the attention mechanisms, we use the difference in distribution between the original output ($O$) and the counterfactual output ($O_{c}$) generated under the intervention.
For the visual attention ($A_{i}$):

$$ $\displaystyle P_{effect_V}$ $\displaystyle=E_{A_{i}\sim\tilde{A}_{i}}\left[P(O|A_{i}=\textbf{A}_{i},I=\textbf{I},P_{v}=\textbf{P}_{v})-P(O|\text{do}(A_{i}=\textbf{a}_{i}),I=\textbf{I},P_{v}=\textbf{P}_{v})\right].$ $$

Here, $P_{effect_V}$ represents the causal effect of the visual attention mechanism on the model output $O$. The term $\textbf{A}_{i}$ denotes the observed visual attention, whereas $\textbf{a}_{i}$ represents the intervention applied to the visual attention.
For vision-only:

$$ $t_{next,\text{v}}=\arg\max_{i}\left(\frac{e^{\max(\ell_{i}+\gamma(\ell_{i}-\ell_{\text{cf_v},i})-\log(\epsilon)-\max_{j}\ell_{j},-\infty)}}{\sum_{j}e^{\max(\ell_{j}+\gamma(\ell_{j}-\ell_{\text{cf_v},j})-\log(\epsilon)-\max_{k}\ell_{k},-\infty)}}\right).$ $$

In this equation, $t_{next,\text{v}}$ indicates the index of the next token chosen based solely on visual attention. The variable $\ell_{i}$ stands for the original logits of the $i$-th token, and $\ell_{\text{cf_v},i}$ is the counterfactual logit derived from the visual modality.
For the LLM attention ($A_{t}$):

$$ $\displaystyle P_{effect_L}$ $\displaystyle=E_{A_{t}\sim\tilde{A}_{t}}\left[P(O|A_{t}=\textbf{A}_{t},T_{t}=\textbf{T}_{t},P_{l}=\textbf{P}_{l})-P(O|\text{do}(A_{t}=\textbf{a}_{t}),T_{t}=\textbf{T}_{t},P_{l}=\textbf{P}_{l})\right],$ $$

Where $P_{effect_L}$ denotes the causal effect of the language model attention on the output $O$. The notation $\textbf{A}_{t}$ is the observed language model attention, and $\textbf{a}_{t}$ is the intervention applied to the language model attention.
For language-only:

$$ $t_{next,\text{l}}=\arg\max_{i}\left(\frac{e^{\max(\ell_{i}+\gamma(\ell_{i}-\ell_{\text{cf_l},i})-\log(\epsilon)-\max_{j}\ell_{j},-\infty)}}{\sum_{j}e^{\max(\ell_{j}+\gamma(\ell_{j}-\ell_{\text{cf_l},j})-\log(\epsilon)-\max_{k}\ell_{k},-\infty)}}\right).$ $$

This equation describes the selection of the next token $t_{next,\text{l}}$ based purely on language attention. Here, $\ell_{i}$ is the original logits of the $i$-th token, and $\ell_{\text{cf_l},i}$ is the counterfactual logit derived from the language modality.
In a multimodal setting, the combined causal effect is given by:

$$ $\displaystyle P_{effect_M}$ $\displaystyle=E_{A_{i},A_{t}\sim\tilde{A}_{i},\tilde{A}_{t}}\left[P(O|A_{i}=\textbf{A}_{i},A_{t}=\textbf{A}_{t},I=\textbf{I},T_{t}=\textbf{T}_{t},P_{v}=\textbf{P}_{v},P_{l}=\textbf{P}_{l})\right]$ $\displaystyle-P(O|\text{do}(A_{i}=\textbf{a}_{i}),\text{do}(A_{t}=\textbf{a}_{t}),I=\textbf{I},T_{t}=\textbf{T}_{t},P_{v}=\textbf{P}_{v},P_{l}=\textbf{P}_{l}),$ $$

Where $P_{effect_M}$ represents the combined causal effect of both visual and language attention mechanisms on the output $O$.
When integrating visual and language modalities enhanced by counterfactual reasoning, the final token selection is determined by:

$$ $t_{next}=\arg\max_{i}\left(\frac{e^{\max\left(\ell_{i}+\gamma\left((\ell_{i}-\ell_{\text{cf_v},i})+(\ell_{i}-\ell_{\text{cf_l},i})\right)-\log(\epsilon)-\max_{j}\ell_{j},-\infty\right)}}{\sum_{j}e^{\max\left(\ell_{j}+\gamma\left((\ell_{j}-\ell_{\text{cf_v},j})+(\ell_{j}-\ell_{\text{cf_l},j})\right)-\log(\epsilon)-\max_{k}\ell_{k},-\infty\right)}}\right).$ $$

This equation defines the final token selection $t_{next}$ by integrating the effects of both visual and language attention mechanisms, thereby mitigating the negative influence of priors in both modalities and enabling more robust decoding strategies.

## 4 Experiments

In this section, we verify the effectiveness of the CausalMM on different benchmarks and implement ablation for different categories of counterfactual attention and number of intervention layers. The case study and gpt-aided-evaluation are in [4.4](#S4.SS4).

### 4.1 Experimental setup

#### 4.1.1 Benchmarks

VLind-Bench.
VLind-Bench (Lee et al., ) is a benchmark designed to measure language priors in MLLMs. It disentangles language priors from commonsense knowledge (CK), visual perception (VP), and commonsense biases (CB). There is significant reliance on language priors across models, and the Pipeline Score (SLP) offers insights beyond task-level evaluation.

POPE.
POPE (Polling-based Object Probing Evaluation) (Li et al., ) is a benchmark for evaluating MLLMs in accurately determining the presence or absence of specific objects in images, assessing object-level hallucination. The framework utilizes Y/N questions derived from object annotations.
Evaluation metrics include standard binary classification measures — accuracy, precision, recall, and F1 score — offering a clear quantitative assessment of MLLM performance in distinguishing real from hallucinated objects.

MME.
MME (Multimodal Large Language Model Evaluation) benchmark (Fu et al., ) quantitatively assesses MLLMs across ten perception-related and four cognition-focused subtasks. To measure object-level hallucination, it uses subsets focused on object existence and count, while attribute-level hallucinations are assessed through subsets concerning object position and color.

#### 4.1.2 Baselines

Regular setting. We use two baseline MLLMs LLaVa-1.5 (Li et al., ; Liu et al., ) and Qwen2-VL (Wang et al., ) for our baseline setting.

VCD.
Visual Contrastive Decoding (Leng et al., ) is a training-free technique that mitigates object hallucinations in MLLMs. By contrasting output distributions from original and distorted visual inputs, VCD reduces the model’s over-reliance on statistical biases and unimodal priors.

OPERA.
Over-trust Penalty and Retrospection-Allocation (Huang et al., ) is an decoding-based method that mitigates hallucinations in MLLMs. It introduces a penalty term during beam search to address over-trust issues, and incorporates a rollback strategy for token selection.

### 4.2 Main Results

Figure: Figure 3: Scores of different methods on VLind-Bench. CausalMM method significantly improves the model’s score on VLind-Bench.
Refer to caption: /html/2410.04780/assets/radar.png

Results on VLind-Bench.
As shown in the figure [3](#S4.F3), the experimental results on the VLind-Bench benchmark (Lee et al., ) are particularly interesting. On the LLaVA-1.5 model, other methods failed to achieve significant performance improvements in balancing modality priors, while the performance under the multimodal collaborative setting has made a significant leap, indicating that the visual priors and language priors of LLaVA-1.5 are balanced. The visual priors of the Qwen2-VL model has been improved, so that the language setting and the multimodal collaborative setting have achieved similar optimal performance.

This observation can be attributed to the nature of VLind-Bench, which comprises a suite of evaluation frameworks designed to elucidate the influence of various factors and to quantify the reliance on language priors. Such an evaluation paradigm imposes stringent requirements on the equilibrium of the model’s multimodal prior knowledge. Our multimodal collaborative method has notably enhanced the baseline model’s performance across all metrics, effectively achieving a balance in the model’s modal priors. Compared with other methods that follow human priors, the CausalMM method’s automatic capture of the causal effect of attention enables it to balance the bias of different modalities simultaneously. This outcome robustly substantiates the efficacy of our methodology (Liu et al., ).

Results on POPE. The experimental analysis conducted on the POPE benchmark (see Table [1](#S4.T1)), as delineated in prior studies (Li et al., ; Lin et al., ; Schwenk et al., ; Hudson & Manning, ), reveals that our proposed CausalMM demonstrates superior performance in mitigating object-level hallucinations across random, popular, and adversarial settings. CausalMM consistently outperforms existing baselines on the most evaluation metrics, indicating a robust enhancement in performance, with an average metric improvement of 5.37%.

Table: Table 1: Main results on POPE tasks. We evaluate the POPE task accuracy of various MLLMs on the MSCOCO, A-OKVQA, and GQA datasets with LLaVa-1.5 under different decoding settings. Regular refers to the scenario where direct sampling is applied. Vision, Language and Multimodal refer to vision-only, language-only, and multimodal collaboration variants of CausalMM. The bold and the underlined refer to the highest and second highest metrics under each setting, respectively. Each value is followed by the difference relative to regular setting.

Notably, both the vision-only and language-only variants of CausalMM exhibit significant improvements in effectiveness. Furthermore, the multimodal collaborative approach within our model achieves the highest accuracy, underscoring the synergistic benefits of integrating multiple modalities. Despite the observed performance decline in various baselines when subjected to popular and adversarial settings, our model maintains remarkable stability. This observation suggests that our CausalMM method is instrumental in enhancing stability. Moreover, the equilibrium of multimodal parameter priors is deemed crucial, as it can, to a certain extent, amplify the advantages conferred by the balanced priors of distinct modalities. This equilibrium is pivotal in effectively curtailing multimodal hallucinations.

Figure: Figure 4: Result comparison of different categories on MME Benchmark across different methods. In most tasks, the scores obtained by CausalMM are higher than baselines, which verifies its effectiveness.
Refer to caption: /html/2410.04780/assets/mme_light.png

Figure: Figure 5: Result comparison of perception and cognition views on MME Benchmark across different methods. In both perception and cognition dimensions, variants of CausalMM outperform the others.
Refer to caption: /html/2410.04780/assets/mme2_light.png

Results on MME. The empirical investigations conducted on the MME benchmark (Fu et al., ) offer a thorough assessment of both object-level and attribute-level hallucinations. It has been discerned that while models such as LLaVA-1.5 (Liu et al., ; ) and Qwen2-VL (Wang et al., ) exhibit commendable performance in evaluating the presence of objects, they encounter challenges when dealing with more intricate queries, notably those involving counting. As indicated in Figure [4](#S4.F4) and Figure [5](#S4.F5), our CausalMM has been instrumental in significantly enhancing the performance of these models, yielding substantial improvements.

**Table 2: Evaluation on the subset of MME perception. While most of the data are similar, the CausalMM method helps Qwen2-VL improve the performance of multiple indicators in MME Benchmark.**
| Method | OCR | celebrity | landmark | count |
| --- | --- | --- | --- | --- |
| Regular | 147.50 | 147.64 | 182.05 | 160.00 |
| Vision | 162.50 | 150.29 | 182.75 | 165.00 |
| Language | 170.00 | 168.23 | 182.50 | 160.00 |
| Multimodal | 170.00 | 168.23 | 182.75 | 165.00 |

In the domain of attribute-level evaluation, it has been observed that models are more prone to hallucinations concerning attributes like color. Our proposed CausalMM, once again, demonstrates significant improvements in this area. The CausalMM methods have demonstrated robust performance across various metrics, particularly excelling in numerical computations and counting, which also translates into an advantage in the overall score. Although the performance on tasks such as Position remains relatively consistent, the overall enhancements in the perception and cognitive categories underscore the effectiveness of these methods in reducing hallucinations.

In the context of poster and scene tasks, the language-only method has achieved the highest performance, which serves as a compelling validation of the impact of language priors on model performance. The MME fullset evaluation corroborates that our CausalMM method consistently maintains superior performance across a diverse array of tasks and models, thereby further substantiating its practical utility in enhancing the precision and reliability of MLLMs.

### 4.3 Ablation Study

Ablation on different counterfactual attention. To explore the generation of generalized counterfactual attention through interventions (Pearl, ), we evaluated four distinct types of counterfactual attention. Ablation experiments were conducted to systematically assess the impact of each type on model performance, as presented in Figure [7](#S4.F7). The results demonstrate that using random attention as the anchor for the causal effect leads to the most substantial improvement in model performance. This improvement arises because perturbed attention, when aligned with average attention, can be more clearly distinguished from the original attention. This alignment aligns with the principles of the average causal effect.

The reason for this finding is that perturbed attention, when close to the average attention level, better reflects a generalizable attention distribution pattern. Such generalizability enables a more accurate estimation of the causal effect, as it reduces the influence of outlier attention patterns that may not be representative of the overall dataset. Therefore, this approach more effectively meets the criteria for estimating the average causal effect, contributing to the observed performance improvement.

Figure: Figure 6: Ablation on different counterfactual attentions. The specific value is obtained by taking the average of all the results.
Refer to caption: /html/2410.04780/assets/image.png

Ablation on intervention cross layers.
Beyond the categorization of counterfactuals, the effectiveness of counterfactual attention depends on its application across different layers of a large language model. To investigate the influence of language priors at various depths, interventions were meticulously conducted in the early, middle, and late layers of the model. This multi-layered approach is based on the hypothesis that language priors exert varying levels of influence at different stages of language processing.

By intervening at different layers, we aimed to determine whether counterfactual attention could effectively modulate these priors. Based on the experimental results in Figure [7](#S4.F7), interventions between shallow and middle layers proved to be the most effective. We hypothesize that these layers represent the initial stages where language priors significantly impact processing. Interventions in this range can effectively establish anchor points that are influenced by language priors, thereby improving model output to a certain extent.

### 4.4 Case study

**Table 3: GPT-4o-aided-evaluation. The evaluation results of gpt4-o as an expert. The four indicators represent the overall quality, conversational, detailedness and complexity.**
| Method | All | Conv | Detail | Cplx |
| --- | --- | --- | --- | --- |
| Regular | 84.7 | 87.7 | 89.3 | 80.4 |
| Vision | 84.8 | 88.8 | 86.7 | 81.4 |
| Language | 84.7 | 88.8 | 88.0 | 80.4 |
| Multimodal | 85.0 | 88.8 | 89.3 | 80.0 |

Case Study on LLaVA-Bench. To provide a more vivid illustration of the impact of our CausalMM method, a case study was conducted on the LLaVA-Bench dataset (Liu et al., ). This study employed specific visual questions and the corresponding model responses to elucidate the enhancement in model output quality and the mitigation of adverse effects, such as hallucinations, attributable to the CausalMM method. A representative example is depicted in Figure [8](#S4.F8).
Objects like boat, which frequently co-occur with the potential ground truth object ocean, are prone to being hallucinated. However, the application of our CausalMM method notably diminishes these hallucinatory tendencies. It enables the model to discern the city situated at the base of the volcano while maintaining a coherent and informative output text. This outcome underscores the efficacy of CausalMM in refining the output and curtailing the emergence of spurious associations.

GPT-4o-aided-evaluation. Supplementing the standard benchmark assessments, we have employed the GPT-4o(^*^**[https://platform.openai.com/docs/models/gpt-4o](https://platform.openai.com/docs/models/gpt-4o)) as an evaluative referee to quantitatively measure the efficacy of our CausalMM method. The evaluation was conducted using a 10-point scoring system, with the results compiled in Table [3](#S4.T3). The results indicate that CausalMM is more adept at generating responses that align with the sophisticated evaluative standards set by GPT-4o.

## 5 Conclusion

Though promising, MLLMs are prone to biases from visual and language priors, which can degrade performance and cause multimodal hallucinations. These biases stem from the influence of the visual encoder and LLM backbone on the attention mechanism, hindering the model’s ability to align multimodal inputs effectively. To overcome this, we introduced a causal reasoning framework termed CausalMM that applies structural causal modeling to MLLMs, treating modality priors as a confounding factor. By leveraging backdoor adjustment and counterfactual reasoning at both visual and language attention levels, CausalMM demonstrates significant reductions in language priors bias and offers a plug-and-play solution compatible with other training-free approaches, providing a insightful path forward for trustyworthy multimodal intelligence.

## Appendix A Appendix

### A.1 Visualization of Counterfactual Attentions

#### A.1.1 Vision Attention

In this work, we used four commonly used counterfactual visual attentions: random, reverse, uniform, and shuffle. They represent taking random values for global attention, reversing global attention, using consistent attention values, and disrupting the original attention distribution. They can all effectively provide anchor points for obtaining causal effects, thereby helping the model improve potential modal priors. Among them, the settings of random and uniform are closest to the average value in value distribution, so they can provide the largest positive average causal effect.

Figure: Figure 10: Normal vision attention of vision encoder.
Refer to caption: /html/2410.04780/assets/normal.png

#### A.1.2 Language Attention

We visualize four similar counterfactual attentions: they represent taking random values for global attention, negating global attention, using consistent attention values, and disrupting the original attention distribution. We take three of them for visualization. Similarly, they can effectively provide anchors for obtaining causal effects, thereby helping the model improve the potential modal prior. Compared with visual attention, large language models with large parameters are not as sensitive to changes in attention as visual encoders.

Figure: Figure 15: Visualization of normal LLM attention.
Refer to caption: /html/2410.04780/assets/attn_weights_normal.png

### A.2 Case Study

We have selected some typical cases to demonstrate the effect of our method. The CausalMM method balances different modal priors to weaken the bias that may be caused by the model’s own parameter knowledge from the perspective of vision and language, so that the model’s output can be more aligned with multimodal input. This improvement is reflected in the model’s perception and cognitive ability of specific things, and the potential hallucinations of the original model have been effectively improved.

### A.3 GPT-aied-evaluation Template

For gpt-aided-evaluation, we have designed a variety of prompt templates to try to achieve a fairer evaluation. The following is a more effective template for reference.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: - Line 58 mentions the VCD method as considering only statistical correlation. I thought the VCD method also accounts for causation by using negative images to identify language priors. Why do you view the VCD method as overlooking causal relationships?
- On page 5, does 'j' refer to the order of the method for creating counterfactual attention?
soundness: 2
strengths: - The paper is well-written and easy to read.
- Performance improvements were observed in the benchmarks used for evaluation.
- The creation of counterfactual attention in various ways is novel.
summary: The paper points out problems arising from biases induced by visual and language priors in the visual encoder and the LMM backbone, and it mentions the oversight of the causal relationship between attention and the model's output. In this study, a method called CausalMM identifies modality priors as confounders, addressing them through backdoor adjustment and counterfactual reasoning.
weaknesses: - It is unclear if this method constitutes a backdoor adjustment. My understanding of backdoor adjustment involves identifying a confounder, then using it to reduce the confounder's impact, which differs from using counterfactuals as described here. Counterfactuals are typically used to measure natural direct effects or natural indirect effects, and I am curious if this method follows such an approach. Mathematical proof may be needed for line 225.
- Please explain which paths need to be blocked in Figure 2. It appears that the path from the image to visual attention and from text token embedding to LLM attention should be blocked, but aren’t those paths essential?
- This method seems closer to a contrastive decoding approach rather than backdoor adjustment. It appears to use counterfactual attention similarly to how negative samples are used.
- The intervention on attention to reduce the influence of defined priors as confounders seems to also reduce the impact of input on attention.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. **Theoretical Justification**: Can you provide a more detailed theoretical analysis or proofs to support the efficacy of your causal interventions? 

2. **Computational Complexity**: What is the computational overhead introduced by your method? How does it compare to the baseline models in terms of runtime and resource consumption?

3. **Generality of the Approach**: How well does your method generalize to other types of MLLMs, such as chameleon?

4. **Limitations and Failure Cases**: What are the limitations of your method? Are there scenarios where it does not perform well or might even degrade performance? How do you address potential negative impacts?

5. **Impact of Hyperparameters**: How sensitive is your method to the choice of hyperparameters involved in the interventions? Have you performed a sensitivity analysis?

6. **Robustness to Noise**: How does your method handle noisy or adversarial inputs? Does the causal framework improve robustness in such cases?
soundness: 3
strengths: 1. **Perspective**: The paper introduces the idea of applying causal inference techniques to address modality prior-induced hallucinations in MLLMs, which is an interesting and potentially valuable perspective.

2. **Comprehensive Experiments**: The authors conduct experiments on multiple benchmarks, including VLind-Bench, POPE, and MME, providing a range of evaluations for their method.

3. **Ablation Studies**: Inclusion of ablation studies helps in understanding the impact of different components of the proposed method.
summary: This paper addresses the problem of modality prior-induced hallucinations in Multimodal Large Language Models (MLLMs). The authors propose a causal reasoning framework, which applies structural causal modeling and counterfactual reasoning to MLLMs. By treating modality priors as confounding factors between attention mechanisms and the model's output, the approach aims to mitigate the negative effects of these priors. The method involves interventions on both visual and language attention mechanisms and is evaluated on several benchmarks.
weaknesses: 1. **Insufficient Theoretical Justification**: The paper lacks a deep theoretical analysis of why the proposed causal interventions lead to improved performance. The causal model is described, but the theoretical foundations and assumptions are not thoroughly explored or justified.

2. **Limited Novelty**: While the application of causal inference to MLLMs is presented as novel, causal reasoning has been previously applied in machine learning models, including language models. The paper does not sufficiently differentiate its contributions from existing work in causal inference applied to deep learning.

3. **Inadequate Comparison with Baselines**: For some evaluations(such as figure 3,5,6), the experimental evaluation compares the proposed method with only a partial set of baselines from the setup. Under many settings, the performance of this method underperform or only slightly outperform baseline methods as shown in table 1.

4. **Superficial Experimental Analysis**: The results, while showing improvements, lack statistical significance testing. Additionally, there is a lack of detailed analysis of where and why the method improves performance, making it difficult to assess the true impact.

5. **Applicability and Generalization**: The approach is tested on specific MLLMs, but it is unclear how well the method generalizes to other models. 

6. **Lack of Discussion on Limitations**: The paper does not adequately discuss the limitations or potential downsides of the proposed method, such as scenarios where it might not work well or possible negative impacts.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: - In Lines 58-61, the authors state that existing decoding strategies overlook the causal relationships among attention visual attention, language attention, modality priors, and model output. However, VCD [1] injects noise into the image input and acquires the output logits for contrastive decoding, which can be seen as a counterfactual intervention at the visual input. The effects of this intervention are also later shown in the attention parts and model outputs [2,3]. Could the authors make more discussion on this perspective?
- What does $\gamma$ mean in the equations in Section 3.3? More importantly, the calculation of choosing the index of the next token is not discussed. Why is it calculated in this way?
- Could you provide the results of VCD and OPERA on VLind-Bench?

[1] Leng S, Zhang H, Chen G, et al. Mitigating object hallucinations in large vision-language models through visual contrastive decoding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13872-13882.

[2] Xiao X, Wu B, Wang J, et al. Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment[J]. arXiv preprint arXiv:2405.17871, 2024.

[3] Chen Z, Xu C, Qi Y, et al. Mllm is a strong reranker: Advancing multimodal retrieval-augmented generation via knowledge-enhanced reranking and noise-injected training[J]. arXiv preprint arXiv:2407.21439, 2024.
soundness: 2
strengths: - The paper is well-written and clear motivated.
- The method is plug-and-play and does not require retraining, making it practical for existing MLLMs.
- The integration of causal inference to modify attention weights and optimize token generation in MLLMs is novel.
summary: The paper introduces a causal inference framework called CAUSALMM, which is designed to mitigate the issue of multimodal hallucinations in Multimodal Large Language Models (MLLMs) that are often caused by biases from visual and language priors. The core idea is to treat modality priors as a confounder between the attention mechanisms and the model output, and to apply structural causal modeling to address these biases. Specifically, the authors use backdoor adjustment and counterfactual reasoning at both the visual and language attention levels to alleviate the negative effects of modality priors, thereby enhancing the alignment between the MLLM's inputs and outputs.
weaknesses: - The background knowledge about causal inference is insufficient. The authors do not explain why causal inference is effective in capturing the causal impact of effective attention in MLLM output.
- Several claims lack explanations and references. The authors are advised to carefully proofread the paper and add the necessary citations. For instance:
    - Line 65: “Modality priors are one of the confounding factors in the causal path of MLLM.” Why? And what is the definition of the confounding factor?
    - Line 58-61: “existing decoding strategies……, overlooking the causal relationships among attention visual attention, language attention, modality priors, and model output.”
- In Section 3.3, the authors introduce the causal effect and the corresponding calculation of selecting the next token. But what’s next after determining the index of the next token? The authors are advised to supplement more on this point. I assume that it is going to talk about inferencing with the modified next token prediction.
- Though the proposed method is somewhat novel, the experimental results are not quite significant and robust compared with existing methods (Table 1).
- Lack of implementation details. It would be more convincing to provide more details on how to reproduce the experimental results.
- The notations should be considered more carefully. For example, in Section 3.3, it is confusing that in Line 222-223 $O$ represents the original output but in Line 226-227 it denotes the model output.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See Weakness 1.
soundness: 2
strengths: 1. This paper presents a simple yet effective method: perturbing the attention matrices in ViT or LLMs to generate counterfactual logits, and then using contrastive decoding to improve the model’s predictive probability distribution.  
    - Flexibility and Generalizability. This method is plug-and-play and offers more flexibility compared to VCD, which obtains counterfactual logits by adding noise to images. Perturbing attention allows for greater design space and importantly, can be adapted to modules of different modalities, such as ViT and LLM.
    - Simplicity and Effectiveness. Unlike VCD and its variants that often rely on Adaptive Plausibility Constraints, the proposed method does not appear to require them.
2. The experiments demonstrate the effectiveness of the method.
summary: This paper introduces a causal inference framework named CausalMM. Considering the influence of visual and textual priors on the predictions of multimodal large language models (MLLMs), the authors employ backdoor adjustment and counterfactual reasoning to mitigate these priors’ effects. Specifically, they design various methods to perturb the attention layers in ViT and LLMs, such as randomization and reversal, to obtain the perturbed model predictions (counterfactual logits). They then use contrastive decoding to enhance the model’s predictive probability distribution. The method is a plug-and-play solution and demonstrates effectiveness on mainstream models and benchmarks.
weaknesses: 1. How was the hyperparameter (e.g., $\gamma$) determined? Were hyperparameters independently tuned for different benchmarks and model/method variants?
2. The choice of base models is not comprehensive. The authors selected LLaVA-1.5 and Qwen2-VL as base models to validate their method, but introducing more models would help confirm the method’s universality. For instance, a table could be added to the appendix to show the method’s cross-model generalizability.
3. Section 3.1 and Figure 2 are hard to follow. Section 3.1 introduces very complex causal relationships, but these seem not closely tied to the method. If I understand correctly, the proposed method is largely similar to VCD: it generates counterfactual logits by perturbation and improves predictions by removing this "background noise" from the normal logits, which does not necessitate such complex preliminary explanations. I would also like to mention that the $P_{effect}$ discussed in Section 3.3 might be unnecessary.
4. The equations are also hard to follow. For example, in Section 3.3, the core of the method involves logits processing, which is unrelated to softmax. Therefore, separating logits from the softmax equation and presenting them independently would be clearer and more concise.

Meta-review: ### Summary:
This paper introduces CausalMM, a framework combining backdoor adjustment and counterfactual reasoning to mitigate hallucinations in multimodal large language models (MLLMs). The method treats modality priors as confounders between attention mechanisms and model outputs.

### Strengths:
1. Practical solution for MLLMs:
> "The method is plug-and-play and does not require retraining, making it practical for existing MLLMs" (Reviewer vNn2)

2. Reasonable empirical validation:
> "The authors conduct experiments on multiple benchmarks, including VLind-bench, POPE, and MME benchmarks" (Reviewer CnJZ)

3. Novel perspective:
> "The paper introduces the idea of applying causal inference techniques to address modality prior-induced hallucinations in MLLMs" (Reviewer CnJZ)

### Weaknesses:
1. Limited theoretical foundation:
> "The background knowledge about causal inference is insufficient. The authors do not explain why causal inference is effective in capturing the causal impact of effective attention in MLLM output" (Reviewer vNn2)

2. Marginal improvements:
> "Though the proposed method is somewhat novel, the experimental results are not quite significant and robust compared with existing methods (Table 1)" (Reviewer vNn2)

### Justification:
Despite reservations, I recommend acceptance for several reasons:

1. The paper offers a new perspective on addressing hallucinations in MLLMs through causal inference, even if the theoretical foundations could be stronger.

2. Two reviewers (VPyB and CnJZ) increased their ratings after author responses, indicating the paper's issues are not fundamental flaws but rather limitations that can be addressed.

3. The method is immediately applicable without retraining, providing practical value despite modest improvements.

However, several limitations temper our enthusiasm:
- Improvements over baselines are modest (2-3%)
- Theoretical justification remains somewhat unclear
- Implementation details could be more complete
- Comparison with simpler alternatives like VCD isn't fully convincing

While the authors addressed many reviewer concerns, questions about the method's theoretical foundations and practical impact remain. The acceptance is based more on the potential of the approach and reviewer rating improvements rather than strong conviction about the current results.

Decision: Accept (Poster)