Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
Authors: Guanyu Zhou
Yibo Yan
Xin Zou
Kun Wang
Aiwei Liu
Xuming Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Modality Priors
Multimodal Hallucinations
Counterfactual Reasoning
Deciphering Attention Causality
Primary area: foundation or frontier models, including LLMs

Abstract: Multimodal Large Language Models (MLLMs) have emerged as a central focus in both industry and academia, but often suffer from biases introduced by visual and language priors, which can lead to multimodal hallucination. These biases arise from the visual encoder and the Large Language Model (LLM) backbone, affecting the attention mechanism responsible for aligning multimodal inputs. Existing decoding-based mitigation methods focus on statistical correlations and overlook the causal relationships between attention mechanisms and model output, limiting their effectiveness in addressing these biases. To tackle this issue, we propose a causal inference framework termed CausalMM that applies structural causal modeling to MLLMs, treating modality priors as a confounder between attention mechanisms and output. Specifically, by employing backdoor adjustment and counterfactual reasoning at both the visual and language attention levels, our method mitigates the negative effects of modality priors and enhances the alignment of MLLM's inputs and outputs, with a maximum score improvement of 65.3% on 6 VLind-Bench indicators and 164 points on MME Benchmark compared to conventional methods. Extensive experiments validate the effectiveness of our approach while being a plug-and-play solution. Our code is available at: https://github.com/The-Martyr/CausalMM.

Published as a conference paper at ICLR 2025
MITIGATING MODALITY PRIOR-INDUCED HALLUCI- NATIONS IN MULTIMODAL LARGE LANGUAGE MOD- ELS VIA DECIPHERING ATTENTION CAUSALITY
Guanyu Zhou'
Yibo Yan
Xin Zou'
Kun Wang'
Aiwei Liu
Xuming Hu!
'The Hong Kong University of Science and Technology (Guangzhou)
The Hong Kong University of Science and Technology
Nanyang Technological University, 'Tsinghua University
guanyuzhou.. a @gmail com, xuminghu@hkustt edu.
ABSTRACT
Multimodal Large Language Models (MLLMs) have emerged as central focus in both industry and academia, but often suffer from biases introduced by visual and language priors, which can lead to multimodal hallucination. These biases arise from the visual encoder and the Large Language Model (LLM) backbone, affect- ing the attention mechanism responsible for aligning multimodal inputs. Existing decoding-bied based mitigation methods focus statistical correlations and overlook the causal relationships between attention mechanisms and model output, limit- ing their effectiveness in addressing these biases. To tackle this issue, pro- pose causal inference framework termed CAUSALMM that applies structural causal modeling to MLLMs, treating modality priors confounder between attention mechanisms and output. Specifically, by employing back-door adjust- ment and counterfactual reasoning at both the visual and language attention lev- els, our method mitigates the negative effects of modality priors and enhances the alignment of MILMM's inputs and outputs, with maximum score improvement of 65.3% on 6 (Lind-Bench indicators and 164 points on MME Benchmark com- pared conventional methods. Extensive experiments validate the effectiveness of our approach while being a plug-and-play solution. Our code is available at: https:: thub , com/ The Martyz/CausalMn
INTRODUCTION
Recent research Multimodal Large Language Models (MLLMs) has achieved great progress diverse applications (Yin al., 2023; Jin al., 2024; Yan et al., 2024: Zou et al., 2024b), partic- ularly due their reliance Transformer mod- els (Vaswani, 2017), where performance is driven by the attention mechanism (Hassanin et al., 2024). In particular, such mechanism enables the model assign weights input information, such as images and text, guiding the generation of outputs. However, the inherent bias the ini- tial parameters of the model, namely the modal- ity priors, can negatively impact output quality via the attention mechanism (Tong et al., 2024a; Zhao et al., 2024; Lee et al., 2024; Chen al., 2024). In widely used MLLM architectures, at- tention that most significantly influences output be divided into components: visual
la therc
JIEE The Beseege
LLM
D
Visual Contrastive Decoding
Gand Jnfor exce
Deciphering Attention Causality
(b) CausalMM (Ours)
Figure 1: The comparison of conventional halluci- nation mitigation paradigm (e.g., VCD) and our pro- posed CAUSALMM.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
coder attention and Large Language Model (LLM) backbone attention (Liu et al., 2024b). The parametric knowledge of the visual encoder (i.e., visual priors) affects the alignment of multimodal information by affecting the visual encoder's attention (Tong et al., 2024a:b). Similarly, the knowl- edge embedded in the L.M'' parameters, referred to as language priors. may compromise the model's fidelity to multimodal inputs through attention (Lee et al., 2024). These biases, stemming from the visual encoder and the MLLM's over-reliance on language priors, may lead to issues such as multimodal hallucinations, ultimately degrading model performance (Yang et al., 2023). Sev- eral approaches have been proposed to enhance model output without modifying the model weights (Leng et al., 2024; Huang al., 2024; Zou et al., 2024a). However, as illustrated in Figure (a), existing decoding strategies primarily rely on statistical correlations and predetermined conclusions from posterior analysis to optimize outputs, without systematically studying the causal relationship between visual atlention, language attention, modality priors, and model output. In this context, the attention mechanism adjusts weights solely based on parameter knowledge, which limits the model's ability to comprehend underlying dependencies in the reasoning process, exacerbates bias, leading to problems such as multimodal hallucinations.
Modality priors are of the confounding factors in the causal path of MLLM. We introduce causal reasoning framework CAUSALMM, which can help us better capture the causal impact of effective attention on MILLM output in the presence of these confounding factors, thereby improving the performance of multimodal tasks, as shown in Figure I (b). Specifically, we construct a structural causal model (Pearl, 2009) for MLLM, and use intervention and counterfactual reasoning methods under the back-door adjustment paradigm to derive the causal effects of visual and language atten- tion on the model output despite the confounding effect of modal priors. The CAUSALMM method is based on counterfactual reasoning at the visual and language attention levels, which ensures that the model output is more consistent with the multimodal input, thereby mitigating the negative im- pact of modal priors on performance. Experimental results show that CAUSALMM significantly reduces modal prior bias and improves performance on different tasks, improving 143.7 points on 6 indicators of -inn-Bench,, 164 points the MME Benchmark, and average improvement of 5.37% on the three benchmarks of POPE.
Our key contributions can be summarized as follows: 1 We have constructed a structural causal framework called CAUSALMM flexible for any MILM, exploring the issues of visual and lan- guage priors within the framework. @ We apply counterfactual reasoning at the levels of visual and language attention, making the output more aligned with multimodal inputs. @ Through compre- hensive experiments, we have demonstrated the superior performance of our method in alleviating MLLM hallucinations. In addition, our framework is plug-and-play and can be integrated with other training-free free methods for further improvement.
2 RELATED WORKS
Multimodal Large Language Models. In recent years, MLLMs have seen significant advance- ments (Yin et al., 2023; Jin et al., 2024; Huo et al., 2024; Yan & Lee, 2024). Notable works in- clude VITA (Fu et al., 2024b), the first open-source MLLM capable of processing video, image, text, and audio, demonstrating robust performance across various benchmarks. Cambrian-1 (Tong et al., 2024a) is family of MI.MMs designed with vision-centric approach, achieving state-of-the- art performance and providing comprehensive resources for instruction-tundt MILMss Addition- ally, research on training-free reasoning stage improvements, such as VCD (Leng al., 2024) and OPERA (Huang et al., 2024), has focused on leveraging human experience to enhance model per- formance without additional training (Li al., 2023b; Zheng et al., 2024). In this work, we manage to apply causal reasoning (Pearl, 2009) to make the MLLM automatically optimize the output.
Causal Inference in Multimodal Learning. The field of causal inference has seen significant advancements (Pearl, 2009; Xu et al., 2020; Cheng et al., 2023; Gong et al., 2022; Fang & Liang, 2024; Wu et al., 2022), particularly the context of L.LMs and vision systems (Zhang et al., 2023a; Rao et al., 2021; Zhou et al., 2024). Researchers have explored the integration of causal reasoning to enhance the interpretability and robustness of these models (Xu et al., 2020; Zou et al., 2023). For instance, LLMs have been shown to generate accurate causal arguments across various tasks, surpassing traditional methods (Kıcıman et al., 2023). A comprehensive survey has highlighted the potential of causal inference frameworks to improve reasoning capacity, faimesss and multimodality in LLMs (Liu et al., 2024c). Additionally, recent work showcased the use of LLM-guided discovery to significantly improve causal ordering accuracy (Vashishtha et al., 2023). Different from previous
2

---

Published as a conference paper at ICLR 2025
attempts, we tend to causal reasoning to balance the visual priors and language priors of the model output.
Modality Priors. Research modality priors in MLLMs has seen significant advancements (Tong et al., 2024a; Peng et al., 2023; Lukics & Lukács, 2022; Gema et al., 2024). Studies focused on overcoming language priors by integrating visual modules, enhancing the impact of visual content on model outputs. For instance, (Zhao et al., 2022) proposed a method to improve visual content in Visual Question Answering (VQA) tasks, which proved effective across multiple datasets. Addi- tionally, benchmarks like (Lind-Bench (Lee et al., 2024) have been developed to measure language priors in MLLMs, revealing strong reliance on textual patterns. On the other hand, visual priors have been addressed by augmenting off-the-shelf LLMs to support multimodal inputs and outputs through cost-effective training strategies (Zhang et al., 2024).
METHODOLOGY
In this section, we construct structural causal model of MLLM and generate different counterfac- tual attentions through intervention for counterfactual reasoning based on the back-door criterion.
3.1 STRUCTURAL, CAUSAL MODEL
We construct a structural causal model (SCM) to describe the relationships among various compo- nents of MLLM (Yang et al., 2021; Pawlowski et al., 2020). In particular, our SCM captures the interactions between the visual and language modalities by modeling causal dependencies among input image (/), visual attention (A,), visual token embeddings (Ti), language token embeddings (It), language priors (Pi), visual priors (P,), MLLM attention (At), and model output (O).
The causal graph is formulated follows:
I + A₁: The image input / influences the visual attention layer A,
. I + Tı: The image input directly affects the visual token embeddings T.
P. A₁: Visual priors P. contribute to the attention in the visual attention module.
P. - T;: Visual priors P. also influence the formation visual token embeddings T.
* A; > T;: Visual attention A; impacts the encoding of visual tokens.
T + O: Visual tokens T; contribute directly to the model's output.
Te > A₁: Language token embeddings T. influence the MILM's attention A,-
. T: > O: Language token embeddings T; directly impact the final output.
Pi -> A,: Language priors Pt inform the MILM's attention mechanism A,.
P. + O: Language priors Pt directly affect the model output O.
A₁ - O: LLM attention A, shapes the final output O.
In this causal graph, both visual priors (P.) and language priors (Pi) serve as confounding factors, influencing the attention layers and embedding representations in both modalities. These priors are mixed into the model and can lead to biased outputs. Our goal is to quantifyy the causal effect of visual attention (A,) and language attention (A₃) on the model output (O), while accounting for these confounding effects through intervention and counterfactual reasoning.
3.2 INTERVENTION ON MULTIMODAL ATTENTIONS
We perform specific interventions on the attention layers of both the visual and language components to investigate their causal effects on the model's output. These interventions modify the attention weights to generate counterfactual outputs, allowing us to isolate the impact each modality.
For visual attention, we intervene by replacing the original attention map A; with counterfactual state 4, expressed as do(A, = A²). The counterfactual state A can take various forms, such as random attention weights, uniform distributions, reversed scores, shuffled attention maps (Rao et al., 2021). Each configuration reveals different aspects of how visual attention influences the output, independent of other factors like the image / and visual processing P.-
Similarly, we intervene in the language attention by applying do (A, = At), where A represents alternative attention states that allow to explore the impact of the language attention module on the final output, free from the influences of T,, T;, and P.
The counterfactual attention states specified as follows:
3

---

Published as a conference paper at ICLR 2025
T
Visual Input
Output
P
Visuel Prior
Language Prior
Visuel Attention
LLM Attention
TI Visuel Taken Embedding
Text Taken Embedding
Vision
Language
Multinodal Collaboration
Causal Path
Truncated
Affected Path
Truncated Path
Figure Causal diagram counterfactual reasoning. In vision-only counterfactual reasoning, we only intervene in visual attention (i.e,, the attention the visual encoder). @ In language-only counterfactual reason- ing, we only intervene in the multi-head self-attention of LLM. @ In multimodal collaborative counterfactual reasoning, intervene in both visual and language attention at the same time and obtain the sum of their collaborative causal effects.
1. Random Attention: form distribution. For the visual encoder, attention scores A; (h, w) at spatial locations (h, w) are replaced follows:
Replace the original attention scores with random values drawn from uni-
Ai w) = u(0 1) Ove
(1)
where 2/ 1) is random variable drawn from a uniform distribution, represents the scaling factor for attention, and ag denotes the normalization parameter. Similarly, for the language model, the random attention values (r(n) over tokens are given by:
An(n) ((, 1) 8--Bo0 -B-0,,
where 3 is the language attention scaling factor and ag is the language normalization term. Uniform Attention: Assign a constant value to all attention scores. For the visual encoder, the attention location (h, w) is replaced by the average value:
A,(h, w) = HXW
Σ w.( h,w
(3)
where H and W represent the height and width of attention map. and is small perturba- tion added avoid exact uniformity. For the language model, the attention over N tokens distributed as:
A((n) Σ A.(n) +
(4)
where 8 a small constant ensuring numerical stability.
3. Reversed Attention: Invert the attention map by subtracting each attention score from the maxi- mum value of the map. For the visual encoder:
AiCh, w) max (A.) A; w) X
(5)
where À is an offset parameter to control the inversion. For the language model:
(n max (At) A((n) +..
where Ç is the inversion factor for language attention.
Shuffled Attention: Randomly permute the attention scores across spatial locations for the visual encoder. The new attention map A is created by permuting the original scores A;:
A/ w) (,( (w)).
where 7 (h) and r(w) are random permutations the height and width indices. This intervention specific to the visual encoder and does not apply to the language model, token order is significant in language processing.
observe the independent contributions of both visual and language attention to the model's output, controlling for confounding factors such as the image I, the tokens T,, and the model's intermediate representations P. and P..
By conducting these interventions,
4

---

Published as a conference paper at ICLR 2025
3.3 COUNTEREACTUAL REASONING
To formalize the impact of counterfactual interventions on the model output, perform counter- factual reasoning based on the back-door adjustment principle (Pearl, 2009; Li et al., 2023a; Adib al., 2020; Zhang et al., 2023b). The back-door criterion ensures that properly account for confounding factors (I, P... Pi) when estimating the causal effect of attention mechanisms. Under the framework of back-door adjustment, we are able to effectively obtain the causal effects of other variables under the influence of the confounding factor of modal priors. The specific proof can be found in Sec. A.I. To measure the causal effect of the attention mechanism, we use counterfactual reasoning to simulate the case of attention failure. For the visual attention (A,):
Peffect.v =E AA,r, [P(O]A, A,, I, P. P.) P(O]do(A, a.), I = I. P.)]]. Here, feat.V represents the causal effect of the visual attention mechanism on the model output O. The term A; denotes the observed visual attention, whereas a; represents the intervention applied the visual attention. For vision-only:
mmax(f,+ fassa)) log(e) max, Ej (0))
Eneatyy = arg max Σ, ( - (A.-
In this equation, Enert,y indicates the index of the next token chosen based solely visual attention. The variable & stands for the original logits of the i-th token, and fatv, is the counterfactual logit derived from the visual modality. represents the degree of confidence in the treatment effect. "J" iterates over all tokens in the denominator (to compute the softmax normalization), For the LLM attention (Ac):
Peffect_L= = AA,AA [P[O]A, = P.) - P(O a.), = - Pt).
Where Peffect_L denotes the causal effect of the language model attention on the output O. The no- tation A₃ the observed language model attention, and a, is the intervention applied to the language model attention. For language-only:
emax(t:+ -og(e)))---lgg))-))o)))))))))))) max, (0,-
Enear,11 arg max
maxt -00)
This equation describes the selection of the next token beeet based purely on language attention.
Here, & is the original logits of the i-th token, and f is the counterfactual logit derived from the language modality. In a multimodal setting, the combined causal effect is given by:
Peffect.M = A, [P(O]A; A;, A, = A, I, = T, P. = P -
P. - -(O) = do( -. P
=
Where Pef fecte_ represents the combined causal effect of both visual and language attention mecha- nisms on the output O. When integrating visual and language modalities enhanced by counterfactual reasoning, the final token selection is determined by:
mmax((, (onxx((,++((((0))) ta((tttaa,x)+ - )) lox( max, Ey. 10))
Unezt arg max
tmaxxt;++>l0
Bav. Rav
(a log(c) - max, Eu,-00)
This equation defines the final token selection Enezt by integrating the effects of both visual and lan- guage attention mechanisms, thereby mitigating the negative influence of priors in both modalities and enabling more robust decoding strategies. In all experiments use direct sampling.
4 EXPERIMENTS
In this section, we verify the effectiveness of the CAUSALMM on different benchmarks and imple- ment ablation for different categories of counterfactual attention and number of intervention layers. The case study and eppaaided-evallution are in 4.4.
4.1 EXPERIMENTAL SETUP
4.1.1 BENCHMARKS
VLind-Bench. Lind-Bench (Lee et al., 2024) is benchmark designed to measure language priors in MLLMs. It disentangles language priors from commonsense knowledge (CK), visual perception (VP), and commonsense biases (CB). There is significant reliance on language priors across models, and the Pipeline Score (SIPP) offers insights beyond task-level evaluation.
5

---

Published as a conference paper at ICLR 2025
POPE. POPE (Polling- (Polling-based Object Probing Evaluation) (Li al., 2023c) is a benchmark for eval- uating MLLMs in accurately determining the presence or absence of specific objects in images, assessing object-level hallucination. The framework utilizes Y/N questions derived from object annotations. Evaluation metrics include standard binary classification measures - accuracy, pre- cision, recall, and F1 score offering clear quantitative assessment of MILM performance in distinguishing real from hallucinated objects.
MME. MME (Multimodal Large Language Model Evaluation) benchmark (Fu al., 2024a) quan- titatively assesses MLLMs across ten perception-related and four cognition-focused subtasks. To measure object-level hallucination, it uses subsets focused on object existence and count, while attribute-level hallucinations are assessed through subsets concerning object position and color.
4.1.2 BASELINES
Regular setting. We use two baseline MLLMs LLaVa-1..5 (Li et al., 2023c; Liu et al., 2024a) and Qwen2-VL (Wang et al., 2024) for our baseline setting.
VCD. Visual Contrastive Decoding (Leng et al., 2024) is a training-free technique that mitigates object hallucinations in MLLMs. By contrasting output distributions from original and distorted visual inputs, VCD reduces the model'ss over-reliance statistical biases and unimodal priors.
OPERA. Over-trust Penalty and Retrospection-Alllese method that mitigates hallucinations in MILMM. It introduces penalty term during beam search to address over-trust issues, and incorporates a rollback strategy for token selection.
(Huang et al., 2024) is an decoding- -asse
4.2 MAIN RESULTS
Results on -Lind-Bench. As shown in the figure 3, the experimental results on the VLind-Bench benchmark (Lee et al., 2024) are particularly interesting. On the LLaVA- 1.5 model, other methods failed to achieve significant perfor- mance improvements in bal- ancing modality priors, while the performance under the mul- timodal collaborative setting has made significant leap, in- dicating that the visual priors and language priors of LLaVA- 1.5 are balanced. The visual priors of the Qwen2-VL model has been improved, so that the language setting and the multimodal collaborative setting have achieved similar optimal performance.
Sek
00.0
06.8
5vp
LP
11.0
875
89.
11.5
362
100
20,3
50.
Scb
CB
Scb
LLaVA-1.5
Qwen2-VL
Sip
Sip
Figure 3: Scores of different methods on VLind-Bence CAUSALMM method significantly improves the model's score VLind- Bench.
This observation can be attributed to the nature of -einchBench, which comprises suite of evalu- ation frameworks designed elucidate the influence of various factors and to quantify the reliance on language priors. Such evaluation paradigm imposes stringent requirements the equilibrium of the model's multimodal prior knowledge. Our multimodal collaborative method has notably en- hanced the baseline model's performance across all metrics, effectively achieving balance in the model's 's modal priors. Compared with other methods that follow human priors, the CAUSALMM method's automatic capture of the causal effect of attention enables it to balance the bias of differ- ent modalities simultaneously. This outcome robustly substantiates the efficacy of our methodology (Liu et al., 2024c).
Results on POPE. The experimental analysis conducted on the POPE benchmark (see Table 1), as delineated in prior studies (Li al., 2023c; Lin al., 2014; Schwenk et al., 2022: Hudson & Manning, 2019), reveals that our proposed CAUSALMM demonstrates superior performance in mit- igating object-level hallucinations across random, popular, and adversarial settings. CAUSALMM consistently outperforms existing baselines on the most evaluation metrics, indicating robust en- hancement performance, with an average metric improvement of 5.37 %.

---

Published as a conference paper at ICLR 2025
Table 1: Main results POPE tasks. We evaluate the POPE task accuracy of various MLLMs the MSCOCO, A-OKVQA, and GQA datasets with LLa under different decoding settings. Regular refers to the scenario where direct sampling is applied. Vision, Language and Multimodal refer to vision-only, language-only, and multimodal collaboration variants of CAUSALMM.. The bold and the underlined refer to the highest and second highest metries under each setting, respectively, Each value is followed by the difference relative to regular setting.
Dataset
Setting
Method
Accuracy
Precision
Recall
Fl Score
Random
Regular
83.53 (0.00)
92.12 (0.00)
73.33 (0.00)
81.66 (0.00)
VCD
86.40 (2.87)
94.68 (2.56)
77.13 (3.80)
85.01 (3.35)
OPERA
89.20 (5.67)
92.68 (0.56)
85.26 (11.9)
88.81 (7.15)
Vislon
86.46 (2,93)
96.27 (4.15)
75.86 (2.53)
84.86 (3.20)
Language
88.00 (4.47)
95,96 (3.84)
79.33 (6.00)
86.86 (5.20)
Multimodal
(5.40)
95.20 (3.08)
82.00 (8.67)
88.10 (6.44)
Popular
Regular
81.10 (0.00)
87.89 (0.00)
72.13 (0.00)
79.23 (0.00)
VCD
83.53 (2.43)
89.29 (1.40)
76.20 (4.07)
82.23 (3.00)
OPERA
86.83 (5.73)
88.24 (0.35)
85.26 (13.1)
86.62 (7.39)
MSCOCO
Vision
84.56 (3,46)
91.57 (3.68)
76.13 (3.00)
83.14 (3.91)
Language
87.03 (5.93)
91.80 (3.91)
88.13 (16.0)
87.17 (7.94)
Multimodal
87.13 (6.03)
86.35 (1.46)
88.20 (16.0)
87.26 (8.03)
Adversarial
Regular
78.63 (0.00)
82.96 (0.00)
72.06 (0.00)
77.13 (0.00)
VCD
81.10 (2.47)
84.47 (1.51)
76.20 (4.14)
80,12 (3.99)
OPERA
81.13 (2.50)
78.79 (4.17)
85,20 (13.1)
81.87 (4.74)
Vision
82.20 (3.57)
86.64 (3.68)
76.13 (4.07)
81.05 (3.92)
Language
81.73 (3.10)
86.28 (3.32)
75.46 (3.40)
80.51 (3.38)
Multimodal
83.70 (5.07)
87.69 (4.73)
78.40 (6.34)
82.78 (5.65)
Random
Regular
84.03 (0.00)
87.67 (0.00)
79.20 (0.00)
83.22 (0.00)
VCD
85.90 (1.87)
88.27 (0.60)
82.80 (3.60)
85.44 (2.22)
OPERA
88.23 (4.20)
86.13 (1.54)
91.13 (11.9)
84.59 (1.37)
Vision
87.66 (3.63)
90.24 (2.57)
84.46 (5.26)
87,25 (4.03)
Language
85,96 (1.93)
89.75 (2.08)
81.20 (2.00)
85,26 (2.04)
Multimodal
88.93 (4.90)
91.89 (4.22)
85.40 (6.20)
88.52 (5.30)
Popular
Regular
80.23 (0.00)
80.87 (0.00)
79.20 (0.00)
80.02 (0.00)
VCD
81.96 (1.73)
81.44 (0.57)
82.80 (3.60)
82.11 (2.09)
A- A-OKQQA
OPERA
83.40 (3.17)
78.92 (2.05)
91.13 (11.9)
84.59 (4.57)
Vision
84.03 (3.80)
83.74 (2.87)
84.46 (5.26)
84.10 (4.08)
Language
85.96 (5.73)
89.75 (8.88)
81.20 (2.00)
85.26 (5,24)
Multimodal
85.70 (5.47)
92.60 (11.7)
77.60 (1.60)
84.43 (4.41)
Adversarial
Rogular
74.26 (0.00)
72.33 (0.00)
78.60 (0.00)
75.33 (0.00)
VCD
76.10 (1.84)
72.90 (0.57)
83.06 (4.46)
77.65 (2.32)
OPERA
73.90 (0.36)
67.77 (4.56)
91.13 (12.5)
84,59 (9.26)
Vision
76.86 (2.60)
73.43 (1.10)
84,20 (5.60)
78.44 (3.11)
Language
77.43 (3.17)
74.98 (2.65)
82.33 (3.73)
78.48 (3.15)
Multimodal
77.86 (3.60)
74.41 (2.08)
84.93 (6.33)
79.32 (3.99)
Random
Regular
83.60 (0.00)
87.11 (0.00)
78.86 (0.00)
82,78 (0.00)
VCD
85.86 (2.26)
88.21 (1.10)
82.80 (3.94)
85.41 (2.63)
OPERA
88.50 (5.90)
85,45 (1.66)
92.80 (13.9)
88,90 (6.12)
Vision
87.40 (3.80)
90.53 (3.42)
83.53 (4.67)
86.89 (4.11)
Language
86.56 (2.96)
90.18 (3.07)
82.06 (3.20)
85,93 (3.15)
Multimodal
88.50 (5.90)
90.81 (3.70)
85.66 (6.80)
88.16 (5.38)
Popular
Regular
77.86 (0.00)
77.32 (0.00)
78.86 (0.00)
78.08 (0.00)
VCD
79.06 (1.20)
77.04 (0.28)
82.80 (3.94)
79.82 (1.74)
OPERA
79.80 (1.94)
73.65 (3.67)
92.80 (13.9)
82.12 (4.04)
GQA
Vision
80.80 (2.94)
79.20 20 (1.88)
83.53 (4.67)
81.31 (3.23)
Language
79.93 (2.07)
78.70 (1.38)
82.06 (3.20)
80.35 (2.27)
Multimodal
82.36 (4.50)
80.36 (2.04)
85.66 (6.80)
82.92 (4.84)
Adversarial
Regular
75.16 (0.00)
73.31 (0.00)
79.13 (0.00)
76.61 (0.00)
VCD
76.33 (1.17)
73.23 (0.08)
83.00 (3.87)
77.81 (1.20)
OPERA
75.00 (0.16)
68.43 (4.88)
92.80 (13.6)
78.77 (2.16)
Vision
76.80 (1.64)
73.43 (0.12)
84,20 (5.07)
78.44 (1.83)
Language
76.60 (1.44)
74.21 (0.90)
81.53 (2.40)
77.70 (1.09)
Multimodal
79.53 (4.37)
76.49 (3.18)
85.26 (6. (3)
80.64 (3.03)
7

---

Published as a conference paper at ICLR 2025
normal VCD
180
arlenbone
miliroodt irtemtonn
00
e5616000
geoton
phoburs
abeuk
Conmommnnne Demoniig
OG7
kigg honsaam
Figure 4: Result comparison of different categories on MME Benchmark across different methods. In most tasks, the scores obtained by CAUSALMM higher than baselines, which verifies its effectiveness.
Total Perception Scores
Method
Total Cognition Scores
Method
102
400
sormal
inton actevtions
Mangugge attontion
mulrooogi) attorinn
recoall
Wite attomions
longuege attottonn
mitinadd ateentin
Figure 5: Result comparison of perception and cognition views on MME Benchmark across different methods, In both perception and cognition dimensions, variants of CAUSALMM outperform the others.
Notably, both the vision-only and language-only variants of CAUSALMM exhibit significant im- provements in effectiveness. Furthermore, the multimodal collaborative approach within our model achieves the highest accuracy, underscoring the synergistic benefits of integrating multiple modali- ties. Despite the observed performance decline in various baselines when subjected to popular and adversarial settings, our model maintains remarkable stability. This observation suggests that our CAUSALMM method is instrumental in enhancing stability. Moreover, the equilibrium of multi- modal parameter priors deemed crucial, as it can, a certain extent, amplify the advantages conferred by the balanced priors of distinct modalities. This equilibrium is pivotal in effectively curtailing multimodal hallucinations.
Results on MME. The empirical investigations conducted on the MME benchmark (Fu et al., 2024a) offer a thorough assessment of both object-level and attribute-level hallucinations. has been discerned that while models such as LLaVA-1.5 (Liu et al., 2024b;a) and Qwen2- VL (Wang et al., 2024) exhibit commendable performance in evaluating the presence of ob- jects, they encounter challenges when dealing with more intricate queries, notably those involv- ing counting. As indicated in Figure 4 and Figure 5, our CAUSALMM has been instrumental in significantly enhancing the performance of these models, yielding substantial improvements.
In the domain of attribute-level evalua- tion, it has been observed that models more prone hallucinations concern- ing attributes like color. Our proposed CA once again, demonstrates significant improvements in this area. The CAUSALMM methods have demonstrated robust performance across various met- rics, particularly excelling in numerical computations and counting, which also translates into an advantage in the overall score. Although the performance on tasks such as Position remains relatively consistent, the overall enhancements in the perception and cognitive categories underscore the effectiveness of these methods in reducing hallucinations.
Table 2: Evaluation on the subset of MME perception.
While most of the data are similar, the CAUSALMM method helps Qwen2-VL improve the performance of multiple indi- cators MME Benehmark.
Method
OCR
celebrity
landmark
count
Regular Vision
147.50
147.6
182.05
160.00
162.50
150.29
182.75
165.00
Language
170.00
168.23
182.50
160.00
Multimodal
170.00
168.23
182.75
165.00
In the context of poster and scene tasks, the language-only method has achieved the highest perfor- mance, which serves as a compelling validation of the impact of language priors on model perfor- mance. The MME fullset evaluation corroborates that our CAUSALMM method consistently main- tains superior performance across a diverse array of tasks and models, thereby further substantiating its practical utility in enhancing the precision and reliability of MLLMs.
8

---

Published as a conference paper at ICLR 2025
Predision
600
0302
Accuracy
6.8
020
6.6
6015
Recall
610
Layers
Figure 6: Ablation on different counterfactual attentions. The specific value obtained by taking the average of all the results.
Figure 7: Ablation on intervention cross layers. We explored the relationship between the number of layers of intervention in the LLM and the causal effect.
4.3 ABLATION STUDY
Ablation on different counterfactual attention. explore the generation of generalized coun- terfactual attention through interventions (Pearl, 2009), we evaluated four distinct types of counter- factual attention. Ablation experiments were conducted to systematically assess impact of each type model performance, as presented in Figure 6. The results demonstrate that using random attention as the anchor for the causal effect leads to the most substantial improvement in model performance. This improvement arises because perturbed attention. when aligned with average at- tention, can be more clearly distinguished from the original attention. This alignment aligns with the principles of the average causal effect.
The reason for this finding is that perturbed attention, when close to the average attention level, better reflects generalizable attention distribution pattern. Such generalizability enables estimation of the causal effect, as it reduces the influence of outlier attention patterns that may not be representative of the overall dataset. Therefore, this approach for estimating the average causal effect, contributing to the observed performance improvement.
accurate
effectively meets the criteria
Ablation on intervention eross layers. Beyond the categorization of counterfactuals, the effective- ness of counterfactual attention depends on its application across different layers of a large language model. To investigate the influence of language priors at various depths, interventions were meticu- lously conducted in the early, middle, and late layers of the model. This multi-layered approach is based on the hypothesis that language priors exert varying levels of influence at different stages of language processing.
By intervening at different layers, we aimed determine whether counterfactual attention could effectively modulate these priors. Based the experimental results in Figure 7, interventions be- tween shallow and middle layers proved to be the most effective. We hypothesize that these layers represent the initial stages where language priors significantly impact processing. Interventions in this range can effectively establish anchor points that are influenced by language priors, thereby improving model output to certain extent.
4.4
CASE STUDY
Case Study on LLaVA-Bench. To provide a more vivid illustration of the impact of our CAUSALMM method, a case study was conducted on the LLaVA-
Table 3: GPT--ooaiided-evalattooo The evalua- results of gpt4-o an expert. The four indi- cators represent the overall quality, conversational, detailedness and complexity.
Bench dataset (Liu al., 2024b). This study employed specific visual questions and the corre- sponding model responses to elucidate the enhance- ment in model output quality and the mitigation of adverse effects, such as hallucinations, attributable to the CAUSALMM method. A representative ex- ample depicted in Figure 8. Objects like boat, which frequently co-occur with the potential ground truth object ocean, are prone to being halluci- nated. However, the application of our CAUSALMA method notably diminishes these hallucinatory tendencies. It enables the model to discern the city situated at the base of the volcano while maintain- ing a coherent and informative output text. This outcome underscores the efficacy of CAUSALMM in refining the output and curtailing the emergence of spurious associations.
Method
All
Conv
Detail
Cplx
Regular Vision
84.7
87.7
89.3
80.4
84.8
88.8
86.7
81.4
Language
84.7
88.8
88.0
80.4
Multimodal
85.0
88.8
89.3
80.0

---

Published as a conference paper at ICLR 2025
GPP-ao-aiided-evaluttio Supplementing the standard benchmark assessments, have employed the GPT-4o" as an evaluative referee to quantitatively measure the efficacy of our CAUSALMM method. The evaluation conducted using a 10-point scoring system, with the results compiled in Table 3. The results indicate that CAUSALMM more adept at generating responses that align with the sophisticated evaluative standards set by GPT-4o.
POSITIVE CASE
The image showcases pic- turesque scene of a mountainous island with large hill or mountain in the middle. The is- land is surrounded by a beautiful blue ocean, which adds to the serene atmosphere. The mountain appears to be volcano, giving it striking appearance. In the background, there are several small boats scattered across the ocean, likely enjoying the peaceful wa- ters. The combination of the island, moun- tain, and ocean creates breathtaking view, making it an ideal location for relaxation and exploration.
Regular:
The image features a breathtaking view large, rocky island with a lush green hillside. The island is situated in the middle of body of water, which could be ocean or a lake. The island's terrain a mix of rocky cliffs and green vegetation, creating stunning contrast against the water. In the background, a city be seen, possibly on the mainland or another island, The city's buildings are visible against the backdrop of the large island, adding the overall beauty of the scene.
CAUSALMM:
Figure 8: Positive case of CausalMM.
NEGATIVE CASE
Is there any strawberry-llaoored yogurt in the fridge?
Prompt:
Regular: Yes, there is container of strawberryfllvoore yogurt in the fridge.
CAUSALMM: Yes. there con- tainer of neewberry-flavoree yogurt in the fridge.
Figure 9: Unsolved case of CausalMM.
CONCLUSION
Though promising, MLLMS prone biases from visual and language priors, which can degrade performance and cause multimodal hallucinations. These biases stem from the influence of the visual encoder and LLM backbone on the attention mechanism, hindering the model's ability to align multimodal imputs effectively. To overcome this, we introduced causal reasoning framework termed CAUSAL MLM that applies structural causal modeling to MI.LMs, treating modality priors as confounding factor. By leveraging back-door adjustment and counterfactual reasoning at both visual and language attention levels, CAUSALMM demonstrates significant reductions in language priors bias and offers plug-and-play solution compatible with other training-free free approaches, providing a insightful path forward for trustyworthy multimodal intelligence.
"https://platfoomm. openai www.///cc///mdaeeeee.
http://mm....paaaaccccoo///o//no s/gpt
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
This work was supported by CAAI-Ant Group Research Fund; Guangdong Provincial Department of Education Project (Grant No. No.2024KNNC002);; Scientific Research Projects for the Higher- educational Institutions (Grant No. 1209)). Education Bureau of Guangzhou Municipality: Hhangzhou-HKUST(CG Joint Funding Program (Grant No. 2055033773335577 Education Bureau of Guangzhou Municipality.
REFERENCES
Riddhiman Adib, Paul Griffin, Sheikh Iqbal Ahamed, and Mohammad Adibuzzaman. A causally formulated hazard ratio estimation through backdoor adjustment structural causal model. In Machine Learning for Healthcare Conference, pp. 376-396. PMLR, 2020.
Meiqi Chen, Yixin Cao, Yan Zhang, and Chaochao Lu. Quantifying and mitigating unimodal biases in multimodal large language models: A causal perspective. arXiv preprint arXiv:2403.18. 18346, 2024.
Yuxiao Cheng, Runzhao Yang. Tingxiong Xiao, Zongren Li, Jinli Suo, Kunlun He, and Qiong- hai Dai. Cuts: Neural causal discovery from irregular time-series data. arXiv preprint arXiv:2302.07458, 2023.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03888 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purposse vision-language models with instruction tuning, 2023.
Yaxin Fang and Faming Liang. Causal-stonet: Causal inference for high-dimensional complex data. arXiv preprint arXiv:2403. 18994, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, et al. Vita: Towards open-source interactive omni multimodal 11m. arXiv preprint urXiv:2408.060 052 2024b.
Aryo Pradipta Gema, Chen Jin, Ahmed Abdulaal, Tom Diethe, Philip Teare, Beatrice Alex. Pasquale Minervini, and Amrutha Saseendran. Decore: Decoding by contrasting retrieval heads to mitigate hallucinations. arXiv preprint arXiv:2410. 18860, 2024.
Wenbo Gong, Joel Jennings, Cheng Zhang, and Nick Pawlowski. Rhino: Deep causal temporal relationship learning with history-dependent noise. arXiv preprint arXiv:22 /4706, 2022.
Mohammed Hassanin, Saced Anwar, Ibrahim Radwan, Fahad Shahbaz Khan, and Ajmal Mian. Visual attention methods deep learning: An in-depth survey. Information Fusion, 102410, 2024.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
-aarecctioonallaaatoon...o In Proceedings of the IEEE/CVF Conference
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.

---

Published as a conference paper at ICLR 2025
Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, and Xuming Hu. Mmneuron: Discovering neuron-level domain -ppecifc interpretation in multimodal large language model. arXiv preprint 1rXiv:2406.11193, 2024.
Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey. arXiv preprint arXiv:2405. 10739, 2024.
Emre KIcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. arXiv preprint 0tti::230500005: 2023.
Kang-il Lee. Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. Vlind- Measuring language priors large lingunl models. arXiv preprint arXiv:24006.mm 08702, 2024.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations large vision- languaee models through visual contrastive de- coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 13872-13882, 2024.
Wenhui Li, Xinqi Su, Dan Song, Lanjun Wang, Kun Zhang, and An-An Liu. Towards deconfounded image-text matching with causal inference. In Proceedings of the 31st ACM International Con- ference on Multimedia, pp. 6264-6273, 2023a.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation optimiza- tion. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12286-12312 2023b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 292-305, 2023c.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECC Ecton-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Xiaoyu Liu, Paiheng Xu, Junda Wu, Jiaxin Yuan, Yifan Yang, Yuhang Zhou, Fuxiao Liu, Tianrui Guan, Haoliang Wang, Tong Yu, et al. Large language models and causal inference in collabora- tion: A comprehensive survey. arXiv preprint arXiv:2403.09606 2024c.
Krisztina Sára Lukics and Ágnes Lukács. Modality, presentation, domain and training effects in statistical learning. Scientific Reports, 12(1):20878, 2022.
Nick Pawlowski, Daniel Coelho Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. Advances in neural information processing systems, 33:857- 869, 2020.
Judea Pearl. Causality. Cambridge university press, 2009.
Daowan Peng, Wei Wei, Xian-Ling Mao, Yuanyuan Fu, and Dangyang Chen. An empirical study on the language modal in visual question answering. arXiv preprint 2.X.::2305.10143. 2023,
Yongming Rao, Guangyi Chen, Jiwen Lu, and Jie Zhou. Counterfactual attention learning for fine- grained visual categorization and re-identification. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 1034-1034, 2021. 2021.
12

---

Published as a conference paper at ICLR 2025
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.
A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint
arXiv:2405.09818 2024.
Shengbang Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. arXiv preprint arXiv:2406. 16860, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1568-9578, June 2024b.
Aniket Vashishtha, Abbavaram Gowtham Reddy, Abhinav Kumar, Saketh Bachu, Vineeth N Bal- asubramanian, and Amit Sharma. Causal inference using Ilm- im-guided discovery. arXiv preprint arXiv:2310.15117 2023.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-v1: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint (rXXi:2409.12219 2024.
Yulun Wu, Robert A Barton, Zichen Wang, Vassilis N Ioannidis, Carlo De Donno, Layne C Price, Luis F Voloch, and George Karypis. Predicting cellular responses with variational causal infer- ence and refined relational information. arXiv preprint 2r.iv:2210.00112 2022.
Guandong Xu, Tri Dung Duong, Qian Li, Shaowu Liu, and Xianzhi Wang. Causality learning: A new perspective for interpretable machine learning. arXiv:2006. /6789, 2020.
Yibo Yan and Joey Lee. Georeasoner: Reasoning on geospatially grounded context for natural language understanding. arXiv preprint arXiv:2408. /366, 2024.
Yibo Yan, Haomin Wen, Siru Zhong, Wei Chen, Haodong Chen, Qingsong Wen, Roger Zimmer- mann, and Yuxuan Liang. Urbanclip: Learning text-enhanced urban region profiling with con- trastive language-image pretraining from the web. In Proceedings of the ACM on Web Conference 2024, pp. 4006-4017, 2024.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9593-9602, 2021. 2021.
Yue Yang. Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison Burchh, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable im- age classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19187-19197, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 23545.99.... 2023.
Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm- Ilms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601, 2024.
Kexuan Zhang, Qiyu Sun, Chaoqiang Zhao, and Yang Tang. Causal reasoning in typical computer vision tasks. 1xXiv:2307.1399 2023a.
Zaixi Zhang, Qi Liu, Zhicai Wang. Zepu Lu, and Qingyong Hu. Backdoor defense via deconfounded representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 12228-12238, 2023b.
13

---

Published as a conference paper at ICLR 2025
Jia Zhao, Xuesong Zhang, Xuefeng Wang. Ying Yang, and Gang Sun. Overcoming language priors in vqa via adding visual module. Neural Computing and Applications, 34(11):9011 900000915--3-,,,- 2022.
Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. Enhancing contextual under- standing in large language models through contrastive decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, 2024.
Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, and Xuming Hu. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation multimodal large lan- guage models. arXiv preprint arXiv:2408. 09429, 2024.
Guanyu Zhou, Wenxuan Liu, Wenxin Huang, Xuemei Jia, Xian Zhong, and Chia-Wen Lin. Occlu- denet: A causal journey into mixed-view actor-centric video action recognition under occlusions. arXiv preprini arX1::2411.15729, 2024.
Xin Zou, Chang Tang, Xiao Zheng. Zhenglai Li, Xiao He, Shan An, and Xinwang Liu. Dpnet: Dynamic poly-attention network for trustworthy multi-modal classification. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 3550-3559, 2023.
Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, and Xuming Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. arXiv preprint arXiv:2410.03577. 2024a.
Xingchen Zou, Yibo Yan, Xixuan Hao, Yuchong Hu, Haomin Wen, Erdong Liu, Junbo Zhang, Yong Li, Tianrui Li, Yu Zheng, al. Deep learning for cross-domain data fusion in urban computing: Taxonomy, advances, and outlook. Information Fusion, 113:102606, 2024b.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 FURTHER DEMONSTRATION
STRUCTURAL CAUSAL MODEL (SCM):
Take the three core variables mentioned in the article as an example.
VARIABLES AND THEIR ROLES:
A (attention): This represents the model's attention mechanism that aim to evaluate or manipulate.
M (modality priors): Modality priors influence both the model's attention (A) and the output (O), thus creating confounding.
O (model output): The outcome variable, which is affected both directly by A and indi- rectly through M.
CAUSAL STRUCTURE AND BACK DOOR PATHS:
The back-door path in this SCM is A < M + O, which starts with an arrow pointing into and creates a confounding junction structure.
To isolate the causal effect of A on O, the confounding influence of M must be blocked.
BACK-DOOR CRITERION:
To apply back-door adjustment, the adjustment set M must satisfy the following criteria:
MM blocks all back-door paths from O.
2. M does not include any descendants of A (i.c., variables causally influenced by A).
By intervening on A and adjusting for M, we can isolate the causal effect of A on O.
BACK-DOOR ADJUSTMENT FORMULA:
Given a sufficient adjustment set M, the causal effect P(o do(a)) identified as:
P(o do(a)) EP
P(o P P(mm
DERIVATION:
1. Starting with the interventional distribution:
P(o do(a)) Σ P(o do(a), P(m do(a))
m
2. Using the property of the intervention do(a): Under the intervention do(a), the variable is no longer influenced by M. Thus:
P(m do(a)) = P(m)
3. Replacing P(o | do(a), m) with the observational counterpart: Due to the back-door criterion, M blocks all confounding paths, allowing:
P(o do(a). m) = P(o m)
4. Combining these results:
P(o do(a)) Σ
P(o a, P(m)
m
15

---

Published as a conference paper at ICLR 2025
APPLICATION TO ATTENTION-OUTPUT FRAMEWORK:
In the context of our framework:
1. Back-door path:
The back-door path M + reflects the confounding effect of modality priors (M) the attention mechanism (A) and the model's output (O).
2. Intervention: By intervening on A, we ensure that the causal effect of attention on the output is isolated, free from the influence of modality priors.
3. Adjustment: To block the back-door path, adjust for M, computing the summation over all possible values of M to account for its confounding effect.
FULL FORMULA FOR THE FRAMEWORK:
In our framework, the causal effect of attention (A) on the model output (O) can be computed as:
P(o do(a)) = - P(o a, m) P(m)
P(o a, m): The conditional probability of the output given attention A and modality priors M.
P(m): The marginal probability of modality priors M.
By applying the back-door adjustment formula, mitigate the influence of confounding modality priors, ensuring that the attention mechanism's causal contribution to the output is properly esti- mated.
16

---

Published as a conference paper at ICLR 2025
A.2 ADDITIONAL EXPERIMENTAL RESULTS
To demonstrate the effectiveness of our approach on large multimodal language models of different architectures, we added experimental data from the Q-former-based InstructBLIP model and the iniledddingauttooggessi---eee
Chameleon model to the original experimental data from the vision pecoder-mlp-Ilae paradigm. See tab. and tab. 5 for specific data. Comparisons with more baseline methods can be found in tab. 6.
Table 4: Additional Experimental Results POPE tasks: Chameleon. We evaluate the POPE task curacy various MLLMs on the MSCOCO, A-OKVQA, and GQA datasets with Chameleon (Team, 2024) under different decoding settings. Regular refers to the scenario where direct sampling is applied. Language refer to language-only.
Dataset
Setting
Method
Accuracy
Precision
Recall
F1 Score
Random
Regular Language
61.90 69.23
57.46
91.67
70.64
63.17
92.27
74.99
Popular
Regular
65.10
59.86
91.67
72.43
Language
69.43
63.34
92.27
75.12
MSCOCO
Adversarial
Regular
60.20
56.28
91.40
69.66
Language
64.00
58.94
92.33
71.95
Random
Regular
60.37
56.26
93.20
70.16
Language
65.70
60.14
93.13
73.08
Popular
Regular Language
57.30
54.25
93.20
68.58
63.07
58.16
93.13
71.60
A-OKVQA Adversarial
Regular
53.57
51.99
93.20
66.75
Language
56.83
53.96
93.13
68.33
Random
Regular Language
60.37
56.26
93,20
70.16
68.43
62.18
94.13
74.89
Popular
Regular Language
59.37
55.76
90.67
69.05
66.73
60.81
94.13
73.89
GQA
Adversarial
Regular
52.73
51.55
90.67
65.73
Language
57.77
54.50
94.13
69.03
17

---

Published as a conference paper at ICLR 2025
Table Additional Experimental Results POPE tasks: InstructBLIP. evaluate POPE task accuracy of various MLLMs on the MSCOCO, A-OKVQA, and GQA datasets with InstructBLIP (Dai et al., 2023) under different decoding settings, Regular refers to the scenario where direct sampling is applied. Vision, Language and Multimodal refer to vision-only. language-only, and multimodal collaboration variants of CAUSALMM.
Dataset
Setting
Method
Accuracy
Precision
Recall
F1 Score
Random
Regular
80.71
81.67
79.19
80.41
VCD
84.53
88.55
79.32
83.68
Vision
87.17
92.72
80.67
86.27
Language
86.90
94.89
78.00
85.62
Multimodal
87.90
94.59
80.40
86.92
Popular
Regular
78.22
77.87
78.85
78.36
VCD
81.47
82.89
79.32
81.07
Vision
83.97
86.37
80.67
83.42
MSCOCO
I Language
83.53
87.71
78.00
82.57
Multimodal
84.90
88.35
80.40
84.19
Adversarial
Regular
75.84
74.30
79.03
76.59
VCD
79.56
79.67
79.39
79.52
Vision
81.47
81.89
80.80
81.34
Language
82.00
84.73
78.07
81.26
Multimodal
82.43
83.71
80.53
82.09
Random
Regular
80,91
77.97
86.16
81.86
VCD
84.11
82.21
87.05
84.56
Vision
87.33
85.94
89.27
87.57
Language
87.87
87.72
88.07
87.89
Multimodal
88.47
87.86
89.27
88.56
Popular
Regular
76.19
72.16
85.28
78.17
VCD
79.78
76.00
87.05
81.15
Vision
81.07
76.69
89.27
82.50
A-OKVQA
Language
82.33
79.01
88.07
83.29
Multimodal
82.13
78.45
88.60
83.22
Adversarial
Regular
70.71
65.91
85.83
75.56
VCD
74.33
69.46
86.87
77.19
Vision
74.83
69.11
89.80
78.
Language
76.27
71.07
88.60
78.87
Multimodal
75.97
70.51
89.27
78.79
Random
Regular
79.65
77.14
84.29
80.56
VCD
83,69
81.84
86.61
84.16
Vision
86.10
84.56
88.33
86.40
Language
86.67
86.86
86.63
Multimodal
87.23
86.67
88.00
87.33
Popular
Regular
73.87
69.63
84.69
76.42
VCD
78.57
74.62
86.61
80.17
Vision
77.77
72.92
88.33
79.89
GQA
Language
79.17
75.48
86.40
80.57
Multimodal
78.97
74.99
86.93
80.52
Adversarial
Regular
70.56
66.12
84.33
74.12
VCD
75.08
70.59
85.99
77.53
Vision
74.50
69.33
87.87
77.51
Language
76.30
71.81
86.60
78.51
Multimodal
75.83
71.19
86.80
78.22
18

---

Published as a conference paper at ICLR 2025
Table 6: More results POPE tasks. We evaluate the POPE task accuracy of various MLLMs the POPE benchmark with LLaVa-1.3 1.5 InstructBLIP under different decoding settings. In the table, the values taken are the averages of the three parts of the POPE benchmark (MSCOCO, A-OKVQA, GQA). Regular refers to the scenario where direct sampling is applied. Vision, Language and Multimodal refer to vision-only, language-only, and multimodal collaboration variants of CAUSALMM, DOLA stands for DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models( al., 2023).
Dataset
Setting
Method
Accuracy
Precision
Recall
F1 Score
Random
Regular
80.42
78.93
83.21
80.9
DOLA
83.00
83.06
83.13
83.00
VCD
84.11
84,20
84.33
84.13
OPERA
85.07
88.39
80.73
84.39
AGLA
87.30
88.83
85.68
87.07
Vision
86.87
87.74
86.09
86.75
Language
87.15
89.82
84.16
86.71
Multimodal
87.87
89.71
85.89
87.60
Popular
Regular
76.09
73.22
82.94
77.65
DOLA
78.99
77.12
83.13
79.85
VCD
79.94
77.84
84.33
80.80
InstructBLIP
OPERA
78.33
73.85
87.73
80.20
AGLA
81.86
80.17
85.68
82.58
Vision
80.94
78.66
86.09
81.94
Language
81.68
80.73
82.14
Multimodal
82.00
80.60
85.31
82.64
Adversarial
Regular
72.37
68.78
83.06
75.42
DOLA
74.67
71.53
83.11
76.68
VCD
76.32
73.24
84.08
78.08
OPERA
75.50
70.49
87.73
78.17
AGLA
77.29
74.09
85.67
79.16
Vision
76.93
73.44
86.16
78.99
Language
78.19
75.87
84.42
79.55
Multimodal
78.08
75.14
85.53
79.70
Random
Regular
83.72
89.30
77.13
82,55
DOLA
84.78
87.59
81.27
84.19
VCD
86.05
90.39
80.91
85,29
OPERA
88.64
88.09
89.73
87.43
AGLA
88.54
94.41
82.08
87.71
Vision
87.17
92.35
81.28
86.33
Language
86.84
91.96
80.86
85.68
Multimodal
88.79
92.63
84.35
88.26
Popular
Regular
79.73
82.03
76.73
79.11
DOLA
79.75
84.11
76.22
80.61
VCD
81,52
82.59
80.60
81,39
LLa
OPERA
83.34
80.27
89.73
84.44
AGLA
85.14
87.88
82.08
84.68
Vision
83.13
84.84
81.37
82.85
Language
84.31
86.75
83.80
84,26
Multimodal
85.06
86.44
83.82
84.87
Adversarial
Regular
76.02
76.20
76.60
76.36
DOLA
76.32
77.27
75.47
76.16
VCD
77.84
76.87
80.75
78.53
OPERA
76.68
71.66
89.71
79.46
AGLA
81.13
81.20
82.10
81.36
Vision
78.62
77.83
81.51
79.31
Language
78.59
78.49
79.77
78.90
Multimodal
80.36
79.53
82.86
80.91
19

---

Published as a conference paper at ICLR 2025
A.3 VISUALIZATION COUNTERFACTUAL ATTENTIONS
A.3.1 VISION ATTENTION
In this work, we used four commonly used counterfactual visual attentions (Rao et al., 2021): ran- dom, reverse, uniform, and shuffle. They represent taking random values for global attention, revers- ing global attention, using consistent attention values, and disrupting the original attention distribu- tion. They can effectively provide anchor points for obtaining causal effects, thereby helping the model improve potential modal priors. Among them, the settings of random and uniform are closest to the average value in value distribution, so they can provide the largest positive average causal effect.
Figure 10: Normal vision attention of vision encoder.
Figure 11: Shuffled vision attention of vision coder.
Figure 12: Random vision attention of vision coder.
Figure 13: Reversed vision attention vision en- coder.
Figure 14: Uniform vision attention vision en- coder.
20

---

Published as a conference paper at ICLR 2025
A.3.2 LANGUAGE ATTENTION
We visualize four similar counterfactual attentions: they represent taking random values for global attention, negating global attention, using consistent attention values, and disrupting the original attention distribution. We take three of them for visualization. Similarly, they can effectively provide anchors for obtaining causal effects, thereby helping the model improve the potential modal prior. Compared with visual attention, large language models with large parameters are not as sensitive to changes in attention as visual encoders.
Planboe Ropto hoamnn
Plarier Rogho Tarrn
200
240
Figure 15: Visualization normal LLM atten- tion.
Figure 16: Visualization random LLM atten- tion.
NOWiIS wiihtt HaTur
Figure 17: Visualization of reversed LLM atten- tion.
Figure 18: Visualization of uniform LLM atten- tion.
21

---

Published as a conference paper at ICLR 2025
A.4 CASE STUDY
We have selected some typical cases to demonstrate the effect of our method. The CausalMM method balances different modal priors to weaken the bias that may be caused by the model's own parameter knowledge from the perspective of vision and language, that the model's output can be more aligned with multimodal input. This improvement reflected in the model's perception and cognitive ability of specific things, and the potential hallucinations of the original model have been effectively improved.
Prompt:
How many uncut fruits are in the image?
Regular:
There are four uncut fruits in the image.
Our Method:
There are three uncut fruits in the image.
Figure 19: Case of counting task.
Prompt:
Describe this photo in detail.
The image features painting of dog wearing a Renaissance-stylee dress, giving a human-like appearance. The dog is sitting on a chair, with its head turned the side, and appears to be looking at the viewer. The painting captures the dog's unique and artistic expression, making it an interesting and eye-catching piece.
Regular:
The image features painting of dog wearing Renaissance-style dress and hat. The dog appears to be sitting down, possibly posing for the portrait. The dog's dress is long, covering its body from the neck down. The painting captures the dog's unique appearance, making an interesting and creative piece of art.
Our Method:
Figure 20: Case of description task.
22

---

Published as a conference paper at ICLR 2025
Prompt:
What material appears make up the creature?
Regular:
The creature appears be made of combination plastic and glass, as has a plastic body and glass eyes.
Our Method:
The creature appears to be made of plastic.
Figure 21: Case of perception task.
Prompt:
Describe this photo in detail.
Regular:
The image showcases a tall skyscraper towering over a cityscape. The tower is prominently visible in the middle of the scene, reaching high into the sky. The cityscape around the tower includes numerous buildings
various sizes and shapes, creating bustling urban atmosphere. There are also several cars scattered throughout the scene, indicat- ing that the area is busy with traffic. Additionally, there are three people visible in the image, likely going about their daily activities or enjoying the view of the towering skyscraper.
Our Method:
The image features tall, skinny building with large antenna top of it. The building is located in city, surrounded by other buildings, and it appears to be prominent landmark. The antenna is po- sitioned towards the top of the building, making it stand out. In the background, there are several other buildings visible, showcasing the urban landscape. The scene captures the essence of bustling cityscape with a mix of architectural styles.
Figure 22: Case of deseription task.
23

---

Published as a conference paper at ICLR 2025
Limitation of CAUSALMM
We further evaluated the effect of the CAUSALMM method based the limitations the method. The specific example is in fig. 23, We found that even after correcting some of the hallucinations caused by visual and language priors, method still did not significantly improve the acquisition of high-level semantics. We believe that the bottleneck of our method is the performance bottleneck of the vision encoder and the LLM backbone. In future work, we will explore how to maximize the positive impact of balanced modal priors when the backbone model is fixed.
study to explore
Prompt:
TW
7-77L77
What is funny about this image?
Describe panel by panel.
Regular:
The image collage of three panels, each showing different aspect of the same scene. The first panel features cell phone plugged into charger, while the second panel shows person holding a box earphones. The third panel displays close-up of the person's 's hand holding the earphones. The funny aspect of this image is that the person's 's hand is holding the earphones in the same position as the cell phone plugged into the charger in the first panel. This creates humorous visual connection between the two objects, as the person's 's hand is plugging the earphones into the charger.
Our Method:
The image is collage of three panels, each showing different aspect of cell phone being charged. The first panel features close-up of the cell phone's charging port, which is plugged into a charger. The second panel shows a person holding charger, likely preparing to plug it into the cell phone. The third panel displays person holding USB cable, which is also being used to charge the cell phone. The funny aspect of this image is that the person in the third panel is holding the USB cable upside down, which is not the correct way to plug into the cell phone. This adds a humorous element to the otherwise straightforward image of charging a cell phone.
Figure 23: Case of description task.
24

---

Published as a conference paper at ICLR 2025
A.5 GPT-AIED-EVALUATE TEMPLATE
For spt-aided-evallatiin, we have designed a variety of prompt templates to try to achieve a fairer evaluation. The following more effective template for reference.
GPT-aied-cvaluation Template
1. Image Description Evaluation: You will be provided with set of image descriptions and a list of comments about the image. Your task is to evaluate each comment for hallucinations, which are inaccuracies inconsistencies with the factual descriptions.
2. Hallucination Identification: Pay special attention comments that claim the existence of something not present in the descriptions, describe objects or attributes incorrectly, or make unrelated statements.
Judgment and Revision: For each comment, provide a judgment (hallucination, correct, or cannot judge) and, if necessary, rewrite the comment to accurately reflect the image content. Ensure that the revised comments are detailed, coherent, and free of hallucinations.
Scoring Criteria: Rate the performance the Al on scale of to for each of the following criteria:
Accuracy: How well the response aligns with the factual image content.
Detailedness: The richness of the response in necessary details, excluding hallucinated parts.
5. Output Format:
Judgment: List each comment with its judgment (hallucination, correct, or cannot judge) and reason.
Revised Sentences: Provide revised comments where necessary.
Scores: Output the scores for accuracy and detailedness, with reasons.
Example:
Region Descriptions of the Image:
[10, 20, 50, 60]: red apple on a white plate,
[70, 30, 120, 80]: A blue cup wooden table.
Comments for Evaluation:
1. The apple green.
There a spoon next the cup.
3. The atmosphere in the room is cozy.
Your Output:
Judgement:
hallucination: The description states the apple is red, not green.
2. cannot judge: The region descriptions do not mention spoon.
correct: The comment does not contradict the provided descriptions.
Revised Sentences:
1. The apple red.
Scores:
Accuracy: 7
Reason: Assistant had one hallucination, Assistant 2's response is consistent with the descriptions.
Detailedness: 68
Reason: Assistant I's response lacks necessary details due the hallucination, Assistant provides a richer description without hallucinations.
25


Reviews:
[{"review_id": "CzaNkgbrRV", "reviewer": "Reviewer_VPyB", "summary": "The paper points out problems arising from biases induced by visual and language priors in the visual encoder and the LMM backbone, and it mentions the oversight of the causal relationship between attention and the model's output. In this study, a method called CausalMM identifies modality priors as confounders, addressing them through backdoor adjustment and counterfactual reasoning.", "questions": "- Line 58 mentions the VCD method as considering only statistical correlation. I thought the VCD method also accounts for causation by using negative images to identify language priors. Why do you view the VCD method as overlooking causal relationships?\n- On page 5, does 'j' refer to the order of the method for creating counterfactual attention?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well-written and easy to read.\n- Performance improvements were observed in the benchmarks used for evaluation.\n- The creation of counterfactual attention in various ways is novel.", "weaknesses": "- It is unclear if this method constitutes a backdoor adjustment. My understanding of backdoor adjustment involves identifying a confounder, then using it to reduce the confounder's impact, which differs from using counterfactuals as described here. Counterfactuals are typically used to measure natural direct effects or natural indirect effects, and I am curious if this method follows such an approach. Mathematical proof may be needed for line 225.\n- Please explain which paths need to be blocked in Figure 2. It appears that the path from the image to visual attention and from text token embedding to LLM attention should be blocked, but aren’t those paths essential?\n- This method seems closer to a contrastive decoding approach rather than backdoor adjustment. It appears to use counterfactual attention similarly to how negative samples are used.\n- The intervention on attention to reduce the influence of defined priors as confounders seems to also reduce the impact of input on attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZNAonMtyUk", "reviewer": "Reviewer_CnJZ", "summary": "This paper addresses the problem of modality prior-induced hallucinations in Multimodal Large Language Models (MLLMs). The authors propose a causal reasoning framework, which applies structural causal modeling and counterfactual reasoning to MLLMs. By treating modality priors as confounding factors between attention mechanisms and the model's output, the approach aims to mitigate the negative effects of these priors. The method involves interventions on both visual and language attention mechanisms and is evaluated on several benchmarks.", "questions": "1. **Theoretical Justification**: Can you provide a more detailed theoretical analysis or proofs to support the efficacy of your causal interventions? \n\n2. **Computational Complexity**: What is the computational overhead introduced by your method? How does it compare to the baseline models in terms of runtime and resource consumption?\n\n3. **Generality of the Approach**: How well does your method generalize to other types of MLLMs, such as chameleon?\n\n4. **Limitations and Failure Cases**: What are the limitations of your method? Are there scenarios where it does not perform well or might even degrade performance? How do you address potential negative impacts?\n\n5. **Impact of Hyperparameters**: How sensitive is your method to the choice of hyperparameters involved in the interventions? Have you performed a sensitivity analysis?\n\n6. **Robustness to Noise**: How does your method handle noisy or adversarial inputs? Does the causal framework improve robustness in such cases?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Perspective**: The paper introduces the idea of applying causal inference techniques to address modality prior-induced hallucinations in MLLMs, which is an interesting and potentially valuable perspective.\n\n2. **Comprehensive Experiments**: The authors conduct experiments on multiple benchmarks, including VLind-Bench, POPE, and MME, providing a range of evaluations for their method.\n\n3. **Ablation Studies**: Inclusion of ablation studies helps in understanding the impact of different components of the proposed method.", "weaknesses": "1. **Insufficient Theoretical Justification**: The paper lacks a deep theoretical analysis of why the proposed causal interventions lead to improved performance. The causal model is described, but the theoretical foundations and assumptions are not thoroughly explored or justified.\n\n2. **Limited Novelty**: While the application of causal inference to MLLMs is presented as novel, causal reasoning has been previously applied in machine learning models, including language models. The paper does not sufficiently differentiate its contributions from existing work in causal inference applied to deep learning.\n\n3. **Inadequate Comparison with Baselines**: For some evaluations(such as figure 3,5,6), the experimental evaluation compares the proposed method with only a partial set of baselines from the setup. Under many settings, the performance of this method underperform or only slightly outperform baseline methods as shown in table 1.\n\n4. **Superficial Experimental Analysis**: The results, while showing improvements, lack statistical significance testing. Additionally, there is a lack of detailed analysis of where and why the method improves performance, making it difficult to assess the true impact.\n\n5. **Applicability and Generalization**: The approach is tested on specific MLLMs, but it is unclear how well the method generalizes to other models. \n\n6. **Lack of Discussion on Limitations**: The paper does not adequately discuss the limitations or potential downsides of the proposed method, such as scenarios where it might not work well or possible negative impacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "763mikOBIY", "reviewer": "Reviewer_vNn2", "summary": "The paper introduces a causal inference framework called CAUSALMM, which is designed to mitigate the issue of multimodal hallucinations in Multimodal Large Language Models (MLLMs) that are often caused by biases from visual and language priors. The core idea is to treat modality priors as a confounder between the attention mechanisms and the model output, and to apply structural causal modeling to address these biases. Specifically, the authors use backdoor adjustment and counterfactual reasoning at both the visual and language attention levels to alleviate the negative effects of modality priors, thereby enhancing the alignment between the MLLM's inputs and outputs.", "questions": "- In Lines 58-61, the authors state that existing decoding strategies overlook the causal relationships among attention visual attention, language attention, modality priors, and model output. However, VCD [1] injects noise into the image input and acquires the output logits for contrastive decoding, which can be seen as a counterfactual intervention at the visual input. The effects of this intervention are also later shown in the attention parts and model outputs [2,3]. Could the authors make more discussion on this perspective?\n- What does $\\gamma$ mean in the equations in Section 3.3? More importantly, the calculation of choosing the index of the next token is not discussed. Why is it calculated in this way?\n- Could you provide the results of VCD and OPERA on VLind-Bench?\n\n[1] Leng S, Zhang H, Chen G, et al. Mitigating object hallucinations in large vision-language models through visual contrastive decoding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13872-13882.\n\n[2] Xiao X, Wu B, Wang J, et al. Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment[J]. arXiv preprint arXiv:2405.17871, 2024.\n\n[3] Chen Z, Xu C, Qi Y, et al. Mllm is a strong reranker: Advancing multimodal retrieval-augmented generation via knowledge-enhanced reranking and noise-injected training[J]. arXiv preprint arXiv:2407.21439, 2024.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and clear motivated.\n- The method is plug-and-play and does not require retraining, making it practical for existing MLLMs.\n- The integration of causal inference to modify attention weights and optimize token generation in MLLMs is novel.", "weaknesses": "- The background knowledge about causal inference is insufficient. The authors do not explain why causal inference is effective in capturing the causal impact of effective attention in MLLM output.\n- Several claims lack explanations and references. The authors are advised to carefully proofread the paper and add the necessary citations. For instance:\n    - Line 65: “Modality priors are one of the confounding factors in the causal path of MLLM.” Why? And what is the definition of the confounding factor?\n    - Line 58-61: “existing decoding strategies……, overlooking the causal relationships among attention visual attention, language attention, modality priors, and model output.”\n- In Section 3.3, the authors introduce the causal effect and the corresponding calculation of selecting the next token. But what’s next after determining the index of the next token? The authors are advised to supplement more on this point. I assume that it is going to talk about inferencing with the modified next token prediction.\n- Though the proposed method is somewhat novel, the experimental results are not quite significant and robust compared with existing methods (Table 1).\n- Lack of implementation details. It would be more convincing to provide more details on how to reproduce the experimental results.\n- The notations should be considered more carefully. For example, in Section 3.3, it is confusing that in Line 222-223 $O$ represents the original output but in Line 226-227 it denotes the model output.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AftzyLR4rB", "reviewer": "Reviewer_gJXZ", "summary": "This paper introduces a causal inference framework named CausalMM. Considering the influence of visual and textual priors on the predictions of multimodal large language models (MLLMs), the authors employ backdoor adjustment and counterfactual reasoning to mitigate these priors’ effects. Specifically, they design various methods to perturb the attention layers in ViT and LLMs, such as randomization and reversal, to obtain the perturbed model predictions (counterfactual logits). They then use contrastive decoding to enhance the model’s predictive probability distribution. The method is a plug-and-play solution and demonstrates effectiveness on mainstream models and benchmarks.", "questions": "See Weakness 1.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper presents a simple yet effective method: perturbing the attention matrices in ViT or LLMs to generate counterfactual logits, and then using contrastive decoding to improve the model’s predictive probability distribution.  \n    - Flexibility and Generalizability. This method is plug-and-play and offers more flexibility compared to VCD, which obtains counterfactual logits by adding noise to images. Perturbing attention allows for greater design space and importantly, can be adapted to modules of different modalities, such as ViT and LLM.\n    - Simplicity and Effectiveness. Unlike VCD and its variants that often rely on Adaptive Plausibility Constraints, the proposed method does not appear to require them.\n2. The experiments demonstrate the effectiveness of the method.", "weaknesses": "1. How was the hyperparameter (e.g., $\\gamma$) determined? Were hyperparameters independently tuned for different benchmarks and model/method variants?\n2. The choice of base models is not comprehensive. The authors selected LLaVA-1.5 and Qwen2-VL as base models to validate their method, but introducing more models would help confirm the method’s universality. For instance, a table could be added to the appendix to show the method’s cross-model generalizability.\n3. Section 3.1 and Figure 2 are hard to follow. Section 3.1 introduces very complex causal relationships, but these seem not closely tied to the method. If I understand correctly, the proposed method is largely similar to VCD: it generates counterfactual logits by perturbation and improves predictions by removing this \"background noise\" from the normal logits, which does not necessitate such complex preliminary explanations. I would also like to mention that the $P_{effect}$ discussed in Section 3.3 might be unnecessary.\n4. The equations are also hard to follow. For example, in Section 3.3, the core of the method involves logits processing, which is unrelated to softmax. Therefore, separating logits from the softmax equation and presenting them independently would be clearer and more concise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)