Robust Multimodal Large Language Models Against Modality Conflict
Authors: Zongmeng Zhang
Wengang Zhou
Jie Zhao
Houqiang Li
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Large Language Models
Modality Conflict
Hallucinations
Reinforcement Learning
Robustness
Primary area: deep_learning->large_language_models
TL;DR: This paper investigates the hallucination phenomenon in MLLMs from the perspective of modality conflict.

Abstract: Despite the impressive capabilities of multimodal large language models (MLLMs) in vision-language tasks, they are prone to hallucinations in real-world scenarios. This paper investigates the hallucination phenomenon in MLLMs from the perspective of modality conflict. Unlike existing works focusing on the conflicts between model responses and inputs, we study the inherent conflicts in inputs from different modalities that place MLLMs in a dilemma and directly lead to hallucinations. We formally define the modality conflict and construct a dataset named Multimodal Modality Conflict (MMMC) to simulate this phenomenon in vision-language tasks. Three methods based on prompt engineering, supervised fine-tuning, and reinforcement learning are proposed to alleviate the hallucination caused by modality conflict. Extensive experiments are conducted on the MMMC dataset to analyze the merits and demerits of these methods. Our results show that the reinforcement learning method achieves the best performance in mitigating the hallucination under modality conflict, while the supervised fine-tuning method shows promising and stable performance. Our work sheds light on the unnoticed modality conflict that leads to hallucinations and provides more insights into the robustness of MLLMs.

Robust Multimodal Large Language Models Against Modality Conflict
Zongmeng Zhang'
Wengang Zhou? Jie Zhao
Houqiang Li?
Abstract
Modality Conflict
Despite the impressive capabilities of multi- modal large language models (MLLMs) in vision- language tasks, they are prone to hallucinations in real-world scenarios. This paper investigates the hallucination phenomenon in MLLMs from the perspective modality conflict. Unlike existing works focusing on the conflicts between model responses and inputs, study the inherent con- flicts in inputs from different modalities that place MLLMs in dilemma and directly lead to hallu- cinations. We formally define the modality con- flict and construct dataset named Multimodal Modality Conflict (MMMC) to simulate this phe- nomenon in dision-language tasks. Three meth- ods based on prompt engineering, supervised fine- tuning, and reinforcement learning are proposed to alleviate the hallucination caused by modality conflict. Extensive experiments are conducted on the MMMC dataset to analyze the merits and de- merits of these methods. Our results show that the reinforcement learning method achieves the best performance in mitigating the hallucination under modality conflict, while the supervised fine- tuning method shows promising and stable per- formance. Our work sheds light on the unno- ticed modality conflict that leads to hallucinations and provides more insights into the robustness of MILMs. The code and dataset are available at https:://// //gt thub. com/ zmzhang20000 MMMC.
?
ball
User Question
What color the ball?
Human
Hallucinated Response
The ball in the image is green.
MLLM
Expected Response
The image does not contain ball.
MLLM
Figure I. An example of modality conflict in vision-language tasks. Given image describing dog surfing the sea, the user may ask the question "What color is the ball?", The model may hallucinate response "The ball in the image green", while there is no ball in the image. We expect the model to recognize the conflict between the visual input and the textual input and give response like "The image does not contain ball".
1. Introduction
The recent success of multimodal large language models (MLLMs) has advanced the development of artificial intelli- gence vision-language tasks (Dai et al., 2023; Liu et al., 2023: 2024b; Bai et al., 2023: Wang al., 2024). These models enable the joint reasoning over visual and textual inputs, and have achieved state-of-the- the-art performance in various vision-language tasks that require multimodal rea- soning (Fu et al., 2024; Yue et al., 2024; Yu et al., 2024c; Lu et al., 2024; Liu et al., 2025). The powerful capabilities of these MLLMs are typically achieved by pretraining separate language and vision models on large-scale datasets, and then
'School of Artificial Intelligence and Data Science, University of Science and Technology of China 'Department of Electronic Engineering and Information Science, University of Science and Technology of China HHaweii Technologies Co., Ltd.. Correspon- dence to: Wengang Zhou @zzwg@usse udd.uuu.ust............... edu.en>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Robust Multimodal Large Language Models Against Modality Conflict
aligning their features enable multimodal reasoning (Liu et al., 2023; 2024b).
promising and stable performance. Our work sheds light on the unnoticed modality conflict that causes hallucinations and provides more insights into the robustness of MLLMs.
Despite the impressive performance of MLLMs, they are prone to ballucinations in real-world scenarios (Huang et al., 2024; Yu et al., 2024b). Hallucinations refer to the phe- nomenon where MLLMS generate incorrect or misleading information not supported by the input data (Ji et al., 2023). Existing works have proposed various methods to alleviate hallucinations in MLLMs, such as improving the quality of training data (Liu et al., 2024a; Yu et al., 2024a), adjust- ing the decoding strategies (Leng et al., 2024; Huang et al., 2024), and align the model with human preference (Zhao et al., 2024; Yu et al., 2024b). These methods mainly target more precise alignment between the features of different modalities to reduce hallucinations.
To summarize, the contributions of this paper are as follows:
This paper reveals an unnoticed source of hallucina- tions in MLLMs: modality conflict. The formal defi- nition of modality conflict presented in the level of objects, attributes, and relationships.
We construct a dataset called Multimodal Modality Conflict (MMMC) simulate the modality conflict in vision- vision-language tasks and evaluate various preva- lent MILMs on the dataset. Results show that most MLLMs are prone to hallucinations under modality conflict.
However, existing works alleviating hallucinations in MIL.Ms mainly focus on the conflicts between the model responses and the inputs, neglecting a possible source of hal- lucinations: the conflicts between the inputs from different modalities, which we call modality conflict. For instance, as shown in Figure 1, given an image describing dog surf- ing on the sea, the user may ask the question "What color is the ball?". In this case, the question supposes a ball exists in the image, and the model may hallucinate a response "The ball in the image is green", while there is no ball in the im- age. We expect the model to recognize the conffict between the visual input and the textual input and give a response like "The image does not contain a ball". Even with the capability of perfectly aligning features of different modali- ties, ML.LMs may still fall into dilemma when facing such intrinsically conflicted information between inputs. To this end, we aim to investigate such hallucination phenomenon in MILMS from the perspective of modality conflict.
We propose three methods, prompt engineering, su- pervised fine-tuning, and reinforcement learning. to alleviate the hallucination caused by the modality con- flict. Extensive experiments are conducted analyze the merits and demerits of these methods.
2. Problem Formulation
In this section, we formally define modality conflict in vision-language tasks and detail the data construction pro- cess of MMMC. The pipeline of the data construction pro- cess and proposed methods are illustrated in Figure 2.
2.1. Modality Conflict
General Form Given a vision-language task consisting of visual input V and textual input T. the task is to predict an answer A. We define the modality conflict as the situation where the information contained in V and T is inconsistent with each other, leading to a dilemma for the model to predict the answer A. We define the general form of modality conflict as
In this paper, we first give a formal definition of modality conflict in vision- langualee tasks in terms of objects. at- tributes, and relationships in the visual and textual inputs. Based the definition, we construct dataset named Multi- Modal Modality Conflict (MMMC) to simulate the modality conflict in vision-language tasks. We evaluate various preva- lent MLLMs (Dai et al., 2023; Liu et al., 2024b; Bai et al., 2023; Wang et al., 2024) on the MMMC dataset and find that most of them lack the ability to recognize the modality conflict and are prone hallucinations.
Info(V) o Info(T).
Concretely, we instantiate the Info(-) function from objects, attributes, and relationships in the visual and textual inputs following Shu et al. (2025). We define these three types of modality conflict as follows.
To alleviate the hallucination caused by the modality conflict and work towards more robust MI.LMs, we investigate the effectiveness of three methods: prompt engineering, super- vised tie-ttnnng, and reinforcement learning. We conduct extensive experiments on the MMMC dataset to analyze the merits and demerits of these methods. Our results show that the reinforcement learning method achieves the best performance in mitigating the hallucination under modal- ity conflict, while the supervised fine-tuning method shows
Object Conflict
The object conflict occurs when the tex- tual input involves objects not present in the visual input. For example, the textual input supposes a cat in the image, while the image only contains dog rather than car. We define the object conflict in (V, T) as
Obj(T) g Obj()). where Obj(-) denotes the set of objects in the input.
(2)
2

---

Robust Multimodal Large Language Models Against Modality Conflict
Object Conflict
Attribute Conflict
Relation Conflict
Key Components
How wings does the
behind orange inolley?
What growing outside of the sheal?
Detection
$
Components Substitution
Now many does the fish have?
What behand the blue troliey?
What growing inside of shed?
Answer Generation
The image does not contatn any fish or
image molley
contain
image not show anything growing inside of the shed.
Data Construction
Prompt Engineering
Supervised line-tuning
Reinforcement Learning
Image Input
input
Image Input
Input
Image Input
Input
Prompt
Parameter Update
MLLM
MLLM
MLLM
Ground Truth Answer
Parameter Update
Model Response
Reward
Model Response
logp(-)
Ground Answer
Methods
Figure 2. The pipeline of the data construction process and proposed methods. The data construction process mainly consists of key components detection, components substitution, and answer generation. Prompt engineering, supervised fine- mun-tunnng and reinforcement learning proposed to alleviate the hallucination caused by the modality conflict. The snowflake icon denotes that the MLLM is frozen, while the flame icon indicates is fine-tuned.
Attribute Conflict Sometimes the visual and textual in- puts may describe the same objects but with different at- tributes. For example, the textual input describes a red apple, while the image shows a green apple. We deem attribute conflict arises in (V,
situation where
Obj(T) S Obj(V) {0 obj( Rel(O,, 0J) * Rel(O), 0)),
(4)
),1,) i, 1, 1,2, m
Obj(7) S Obj(V)
where Rel(-) denotes the relationships between two objects.
[0,), Obj(T) n Obj(V) Att((OT) + Aur(O)), 1,,,;,,1,-........,, 1,2,..,,
2.2. Data Construction
To simulate the modality conflict in vision- language tasks, we construct a dataset Multimodal Modality Conflict (MMMC) that contains all the three types of conflicts dis- cussed above. Specifically, we collect images from the widely-used Lision-language datasets, Visual Genome (Kr- ishna al., 2017). and construct natural language questions conflicting with the image content and corresponding an- swers. Given the clear definition of modality conflict, resort to the large language models' to construct the dataset for modality conflict The construction process is elaborated as follows.
where {0 is the set of objects contained in both the image and text inputs. Oy and O7 indicate the objects in image and text inputs, respectively. Attr(-) denotes the attributes an object.
Relationship Conftict The relationship conflict occurs when the visual and textual inputs describe the same objects with different relationships. For example, the textual input describes a cat on the table, while the image shows a cat on the floor. We formulate the relationship conflict in (V, T)
We GPT-4o-mini, powerful and fast model for data construction.
3

---

Robust Multimodal Large Language Models Against Modality Conflict
Base Question Sampling To align the format and style of questions to the original dataset, we adopt substitu- tion framework to simulate the modality conflict, inspired by Longpre et al. (2021). We first randomly sample a base question T from the original dataset for each image V. Key information in the base question will then be substituted with conflicting ones to construct a new question as dis- cussed in the following.
MMMC provided in Appendix A.
3. Method
We propose three methods, i.e. prompt engineering, super- vised fine-tuning, and reinforcement learning, to alleviate the hallucination caused by the modality conflict. We first formulate the vision-language task as conditional genera- tion problem:
Key Components Detection Questions in vision- language tasks usually involve a series of components, in- cluding objects, attributes, and relationships. These com- ponents should be displayed in the image to ensure the question is answerable. However, in the modality conflict scenario, the components in the question may not be present in the image. We adopt the large language model to de- tect the objects in the image and extract the attributes and relationships of the objects.
T L
A~ ~ (A T)
To (a V. V. azt),
(5)
where the model Th is required to sequentially generate the answer AA given the visual input V and the textual input T, and is the length of answer A. We then introduce the three methods improve the robustness of the model against modality conflict in this section.
Components Substitution We substitute the objects, at- tributes, and relationships in the base question with conflict- ing information detected from the image. The substitution process is conducted by directly prompting large language model to generate a counterfactual question according to the original question and the key components to be sub- stituted. Additionally, we input all the objects, attributes, and relationships in the image, from the annotation of the original dataset to the model to ensure the conflict between the question and the image content.
3.1. Prompt Engineering
Instruction following (Dai al., 2023; Liu et al., 2023; 2024b) is fundamental capability of MLLMs. Questions are directly inputted to the model to guide the generation of the answer in Equation (5). We propose to instruct the model to check if the objects, attributes, and relationships in the question are present in the image before generating the answer with a simple but effective prompt template p(T):
Please check if the image contains mentioned in- formation and answer the question: T
Answer Generation After obtaining the conflicting ques- tion T', we generate paired answer A'' for the question. Unlike existing works that generate multiple-choice an- swers (Zhu et al., 2024), we collect model response directly to improve the model robustness in free-form generation. It is worth noting that, to avoid the impact of hallucinations in the widely-used large vision- languae models, we do not directly generate the answer by inputting the image V and the question T' to them. Instead, we impose that the substituted components in the question are not present in the image on the large language model, and require the model to generate the answer A' based on the conflicting information. The large language model demonstrates the capability of generating answers based barely on textual information, as shown in Figure 2.
The prompt engineering method is easy to implement and does not require additional data or computational resources, formulated as
A ~ no (A((
(6)
However, the prompt engineering method may not be effec- tive in all cases. The performance of the method is heavily dependent on the foundation model and the quality of the prompt. Besides, the potential of training data not ex- ploited in the prompt engineering method. Therefore, we explore methods with additional training fully leverage the data and improve the robustness of the model against modality conflict.
Postprocessing These generated questions and answers are then verified by human annotators to ensure the quality of the dataset. The language fluency, the conflict between the question and the image, and the correctness of the answer are all considered in the verification process. Finally, we www-qquueennnsweeeeee triples in the MMMC dataset and randomly split them into 18K training samples and 2K testing samples. Visualizations for the statisties of
3.2. Supervised Fine-Tuning
Supervised fine-tuning aims to learn a mapping from the input to the output by minimizing the discrepancy between the model predictions and the ground-truth labels. Existing works have shown the superiority of supervised fine-tuning in conquering knowledge conflict in LLMs (Longpre et al., 2021).
obtain 20K

---

Robust Multimodal Large Language Models Against Modality Conflict
We propose to fine-tune the model on the MMMC dataset with the language modeling objective, formulated
With these base components, can optimize the model with arbitrary reinforcement learning algorithms, such as Proximal Policy Optimization (PPO) (Schulman et al., 2017) and REINFORCE (Williams, 1992). We adopt an optimized version of REINFORCE algorithm, REINFORCE++ (Hu, 2025), for light computation and good performance.
T = arg min (V) -- log (AAV, T)
(7)
where T. A) a triplet of image, question, and answer in the MMMC dataset D. With this objective, the model is optimized by gradient descent to align the model predictions with the ground-truth labels, which is expected to improve the robustness of the model against modality conflict.
In the reinforcement learning method, the model is opti- mized by interacting with the environment and receiving rewards based on the quality of the generated answers. Due to the nature of sampling data from the model itself, the reinforcement learning method is expected to learn more diverse and robust answers that share similar semantics with the ground-trui answers.
Despite its effectivenesss, supervised fine-tuning mainly em- phasizes adapting the style of the model to the target do- main (Zhou et al., 2023), while the performance improve- ment the unseen data may be limited.
4. Experiments
3.3. Reinforcement Learning
4.1. Setup
Inspired by the success of reinforcement learning in align- ment with human preference (Ouyang et al., 2022: Stiennon et al., 2020; Yu et al., 2024b) and improving the robustness of large language models (Zhang et al., 2024), we resort to reinforcement learning to further improve the robustness of the model against modality conflict. Specifically, the condi- tional generation problem in Equation (5) can be formulated as a Markov Decision Process (MDP):
Models We evaluate several types of prevalent MLLMs, InstructBIIIP (Dai et al., 2023), Vlavi- (Liu et al., 2023), LLaVA-NeXT (Liu et al., 2024b), and Qwen2-VL- Instruct (Wang et al., 2024) series, on the MMMC dataset. InstructBLIP adopts Q-former (Li et al., 2023) to compress the image into 32 tokens and bridge the vision and language features, while LLaVA-v1.5, LLaVA-NeXT and Qwen2- VL-Instruct separately encode the image and text with trans- former architecture and conduct multimodal reasoning with additional adapter modules. We use 7B version for each model in the evaluation. Besides, to investigate the impact of model size, we also evaluate the 2B version of Qwen2- IL-Instruct. Additionally, we include the widely-used large language model GPT-4o a baseline.
A To (AV, T) +
with the state st (V,T, the action a, the reward Tr, the transition probability P(s++1 (8t 0t), the initial state distribution po(#o) (2, T) ~ D, and the discount factor y. We propose to optimize the model with reinforcement learning by maximizing the expected reward, formulated as
T Σ
Implementation Details We implement all proposed method using Hugging Face Transformers and OpenRLHF library (Hu et al., 2024). For the supervised fine-tuning, we use the Adam optimizer with a learning rate of 5 x 10-6 and a batch size of 8. We train the model for epochs on the MMMC dataset with 10000 training samples except for the ablation study. For the reinforcement learning, we use the Adam optimizer with a learning rate of 9.65 x 10-6 and batch size of 8. We train the model on the MMMC dataset with only 1000 training samples since longer reinforcement learning will cause the model collapse. We set the KL co- efficient to 0.01 and the max response length to 128. Both the supervised fine-tuning and reinforcement learning meth- ods are trained with LoRA (Hu et al., 2021). We use the 1.ama-3.3-700 Insttrt for reward model.
Th = arg max Be (aasss))
(9)
To realize the goal of alleviating the hallucination caused by the modality conflict, we assign a reward function that encourages the model to generate answers semantically con- sistent with the one in the MMMC dataset and penalizes the model for generating hallucinated responses. The reward function is defined as
+1, if = T A age is consistent with A
1, i A age is not consistent with A,
0, otherwise
(10)
where dgi denotes the generated answer at time step t and A is the ground-truth answer in the MMMC dataset. We prompt pretrained large language model to judge the se- mantic consistency between the generated and ground-truth answer and assign the reward based on the judgment. De- tailed prompts are listed in Appendix B.
Evaluation Protocol Given the reference responses in the MMMC dataset, we adopt the widely-used ROUGE- L (Lin, 2004) F-measure to evaluate the longest common subsequences overlap between the model responses and the reference responses. However, this traditional metric do not
5

---

Robust Multimodal Large Language Models Against Modality Conflict
ods, prompt engineering, supervised fine-tuning, and re- inforcement learning, on the MMMC dataset. As shown in Section 4.2, these three methods significantly improve the robustness of the prevalent MLLMs on the MMMC dataset.
Table 1. Explanation for each level of overall quality scores in the LLM-as-a-Judee evaluation.
Score
Quality
Detailed Description
The Hallu-Rate is reduced by 10% to 50% with the proposed methods. The LLM-Judge scores are improved by 0.4 to 0.9, indicating that the overall quality of the model responses is enhanced. The consistent conclusions provided by the LLM judge, which is based on GPT-44 and -lama-3.3-70B, further validate the reliability of our evaluation results. The ROUGE-L scores are also improved with several methods, showing that the model responses are more aligned with the reference responses. We will further analyze the merits and demerits of these methods in the following section.
0
Not Valid
Unnatural, incoherent or unreadable
Terrible
Irrelevant to the question asked
Different from the reference answer, but still relevant to the question
2
Wrong
Has the same meaning as the reference, but may be phrased differently
3
Right
4 Excellent
Same as the reference or more naturally
consider the semantic similarity in language precisely.
To present a more intuitive evaluation, we adopt the LLM- as-a-Judge (Zheng et al., 2023), a large language model that is pretrained on large-scale dataset and fine-tuned on human preference data, to evaluate the quality of the model responses. Concretely, to evaluate the robustness of the model against modality conffict, we calculate the hal- lucination rate (Hallu-Rate), defined as the percentage of hallucinated responses in the model responses. A response is considered hallucinated if it erroneously assumes the exis- tence of objects, attributes, or relationships that not present in the image, presenting plausible but incorrect information.
4.3. Analysis
Further Analysis on Proposed Methods Prompt engi- neering is a basic method that may improve the robustness of the model responses against modality conflict, bring re- duced Hallu-Rate, and improve LLM-Judge scores in most cases. is easy to implement and does not require additional data or computational resources. However, as shown in Sec- tion 4.2, the improvement of prompt engineering is heavily dependent on the foundation model, and the performance may be unstable. Prompt engineering brings significant improvement to Qwen2-VLL-Instruce -I-sruuuutttt and L.LaVA series, but increases the Hallu-Rate of smaller Qwen2-VL-Instrue 2B model. The performance InstructBLIP is nearly the same as the base model. We inspect the generated responses of InstructBLIP and find that the model tends to generate short and simple responses whatever the prompt is, which may lead to the limited improvement of prompt engincering. The effectiveness of prompt engineering is also limited on GPT-4o due to the over-robustness of the model, with which the model tends generate similar responses for different expressions of the same instruction.
Additionally, require LLM-judge to evaluate the over- all quality of the model responses concerning fluency, rel- evance, and correctness, represented by a score ranging from 0 to 4. We list the criteria for these scores in Ta- ble 1. The average scores of the LLM-judge are reported as LLde- To obtain more robust results, we adopt strong closed-source model GPT-4o series and open- source model Llama-3.3-70B to perform evaluations of Hallu-Ratt and LLM-Judge. All prompts we used in the evaluation are listed in Appendix B.
Supervised fine-tuning (SFT) is a more advanced method that can further improve the robustness of the model re- sponses. It requires additional data and computational re- sources for fine-tuning, but it can achieve better performance than prompt engineering. As shown in Section 4.2, SFT re- duces the Hallu-Rate and improves the Lude scores of LLaVA-v1.5, ries. However, Instruct LIP suffers from performance loss after SFT. We speculate that the pre-training of InstructBIIIP does not inject the capability of recognizing the conflicts between modalities and the fine-tuning data in MMMC also not enough to teach this new skill.
4.2. Main Results
Robustness of Prevalent Foundation Models Against modality conftict As the "Base" results in Section 4.2 show, the prevalent MLLMs, InstructBLIP-7B, LLaVA- v1.5-7B, LLaVA-NeXT-7B, Qwen2-VL-Instruet- VL-Instruct-778 and even GPT-4o, perform poorly on the MMMC dataset. All models exhibit Hallu-Rate over 40%, indicating that they are prone to hallucinations under modality conflict. The LLM-Judge scores are lower than 2.5, showing that most of their responses are judged as wrong or of lower quality. The ROUGE-L scores are also relatively low, suggesting that the model responses are not sufficiently aligned with the reference responses.
and
se-
Besides, SFT restricts the model behavior to the fine-tuning data, which may lead to overfitting and limited generaliza- tion. Reinforcement learning (RL) samples the responses from the model itself and provides more diverse and infor-
Performance Improvements with Proposed Methods We then evaluate the effectiveness of the proposed meth-
6

---

Robust Multimodal Large Language Models Against Modality Conflict
Table Performance comparison of different methods on the MMMC dataset. We conduct the experiments on Prompt Engineering (PF),
Supervised Fine- Tinenn (SFT), and Reinforcement Learning (RL). The performance of GPT-4o is also reported for comparison. The results of SFT and RL are averaged across three runs with different seeds, and the standard deviations are reported parentheses. denotes the higher the better, while denotes the lower the better. The best performance for each model is highlighted in bold.
Hallu-Rate (%) 4
Hallu-Rate (%) + (GPT)
L.M.Judge (Ilama)
L.M.-Judee t (GPT)
Model
Method
ROUGE-L (%)
(Llama)
Base PE
23.76
59.40
57.00
2.12
2.39
GPT-4o
23.98
60.10
56.95
2.13
2.42
Base
13.89
82.10
70.55
1.81
1.85
PE
13.89
82.30
69.50
1.79
1.86
InstruciBLIP-7B
SFT
8.86 (0.43)
85.48 (0.37)
70.33 (0.74)
1.81 (0.01)
1.76 (0.05)
RL
5.65 (3.08)
57.62 (18.58)
57.18 (18.07)
1.01 (0.69)
1.48 (0.96)
Base
28.54
93.25
83.60
1.73
1.81
PE
25.83
86.95
84.70
1.94
1.93
LLaVA-v1.5-7B
SFT
16.90 (0.52)
59.37 (1.02)
52.28 (0.92)
2.27 (0.02)
2.27 (0.02)
RL
23.53 (3.49)
33,87 (2.53)
29.78 (2.04)
2.58 (0.04)
2.74 (0.04)
Base
18.08
69.65
67.00
1.92
2.24
PE
20.91
50.50
50.00
2.43
2.69
LLaVA-NeXT-7B
SFT
22.25 (0.07)
45.93 (0.47)
42.83 (0.78)
2.48 (0.01)
2.44 (0.04)
RL
25.52 (1.66)
33.83 (1.99)
31.27 (2.03)
2.65 (0.02)
2.86 (0.05)
Base
25,20
46.55
40.55
2.07
2.26
PE
30.12
62.10
59.95
2.26
2.40
Owen2-VL-Insttut-2BB
SFT
29.32 (0.25)
26,85 (0.60)
32.78 (0.74)
2.71 (0.02)
2.76 (0.02)
RL
22.65 (1.68)
18.00 (5.19)
16.78 (4.30)
2.73 (0.06)
2.97 (0.08)
Base
24.73
52.35
47.95
2.25
2.47
PE
28.65
40.10
37.35
2.52
2.80
Owen2-VL--Inttuecee
SFT
28.60 (0.10)
28.58 (0.34)
32.02 (0.69)
2.71 (0.01)
2.74 (0.02)
RI
18.89 (0.82)
23.52 (5.63)
20.45 (5.09)
2.66 (0.07)
2.86 (0.10)
mative data for training. It requires more computational resources but may achieve better performance than SFT. As shown in Section 4.2, RL. dramatically reduces the Hallu- Rate and improves the LLM-Judge scores of all models, es- pecially on Qwen2-VL VL-Instruct series. The main reason for the best performance of RI. is that it explores more diverse responses in the training process than SFT, which helps the model to recognize the conflicts between modalities and enhance its robustness.
tional dynamics that the models struggle to accurately in- terpret and predict. These observations suggest that while substantial advancements have been made in processing simpler conflict types such as object-conflicts, there remains a critical need for enhancement in managing relationship- focused conflicts.
A Alignment Tax Both the SFT and RL methods require parameter updates to align the model with the training data, and thus introduce the alignment tax, which defined the performance loss of the model on the original task after the fine-tuning (Ouyang al., 2022). To analyze the alignment tax of our methods, we test the performance changes of the models on a wide range of vision- languuee tasks after the fine-tuning, including HallusionBench (Guan et al., 2024), MMBench (Liu et al., 2025), MMStar (Chen et al., 2024), MMMU (Yue et al., 2024), Math Vista (Lu et al., 2024), OCRBench (Liu al.. 2024d), Al2D (Kembhavi al., 2016), MMVet (Yu et al., 2024c) and MME (Fu al., 2024).
Performance Breakdowns for Different Conffict Types In order gain deeper understanding of how each ap- proach tackles various conflict types, we provide detailed performance analysis for each category of conflict in Ap- pendix C. The analyses indicate that the conclusions drawn from individual subsets of conflict types align closely with those derived from the entire dataset. Particularly note- worthy the finding that MLLMs exhibit superior perfor- mance on object-conflict types. On the other hand, attribute- conflict scenarios present a moderate level of difficulty, and relationship-confiii types pose a significant challenge for MLLMs. The performance on these conflicts is notably poorer when compared to object and attribute conflicts. This drop in performance can be attributed to the intricate rela-
We visualize the performance change of the SFT and RL methods in Figure 3. As the figure shows, the alignment tax is also heavily dependent on the foundation model. For example, InstructBLIP suffers a lot from the alignment tax, showing half of the performance loss on MMBench with

---

Robust Multimodal Large Language Models Against Modality Conflict
LL
Quaan instrrtttt--7
HWW
MANI A20
MAM A2D
ADD
HAR
A00
MANU
B250
SFT
94
Figure 3. Visualization of the alignment tax for supervised fine-tuning (SFT) and reinforcement learning (RL). We plot the performance of the base model (Base) with blue dashed regular polygon, and the performance of the SFT and RL models with orange and green solid polygons, respectively. All scores are normalized the base model for intuitive comparison.
IndruddRan-75
Response Length
120
1.5
M.lllee..S- Owrnn intricd Oanin to intrrctd T3
Reward
0.0
Mean KL
1.0
Ow 1n
60
00.5
30
0.0
0
300
600
900
1200
300 Episades
600
900
1200
600 900
1200
300
600 Episades
900
1200
Steps
Episodes
(a) SFT
(b) RL
Figure 4. Training curves of supervised fine- tueing (SFT) and reinforcement learning (RL) on the MMMC dataset. The training loss of SFT, reward, response length and mean KI. divergence of RL are plotted. We plot the average training curves three runs with different seeds. The solid lines and shaded areas represent the mean values and standard deviations, respectively. All curves are smoothed with exponential moving average for better visualization.
SFT and three-quarters of the performance loss on A12D and MMVet with RL. By contrast, Owen2-VL-Inntrruet-7 series shows negligible performance change on most tasks after the fine fine-tuning, indicating that the model is more bust to the alignment tax. Surprisingly, the performance of LeTT on HallusionBench is even improved af- shows similar
time when the reward of RL reaches the valley bottom and the mean Kullback-Leibler (KL) divergence reaches a peak. To further investigate the reason for this phenomenon, analyze the generated responses of InstructBLIP--1 and find that, at this time, the model begins to generate longer responses with tedious, repeated, and irrelevant information. We deem that the model may fall into the local optimum but is completely collapsed at that time. Some responses generated by the model are shown in Appendix D for a better understanding of the phenomenon. RL training curves of other models are more stable, showing that the model is well- trained with the reward signal. The fluctuation of the reward and response length may be caused by the exploration of the model in the training process, which helps the model to recognize the conflicts between different modalities and enhance its robustness.
ter SET and RL, and Twen2-VLLIInstruce changes on MMVet with SFT. L.L. VA-v1.--1..-77 is the most stable model, demonstrating consistent performance across multiple benchmarks without much variation. These results also strong evidences that the capability of recognizing the conflicts between modalities is beneficial for the model to reduce other types of hallucinations.
Training Stability We further analyze the training stabil- ity of the SFT and RL methods on the MMMC dataset. As shown in Figure 4, in general, the training loss of SFT is relatively stable, while the reward of RL fluctuates a lot. The response length of RL also unstable, indicating that the model may generate responses with different lengths during the training process. This clearly shows that although RL can potentially reach higher performance, it is less stable than SFT.
Impact of Training Episodes Number on RL As the training of RL is unstable and prone to model collapse, we delve into a fundamental aspect of RL training: the impact of training episode number on the model performance. We conduct experiments the LLaVA-NeXT-77 model with different training episodes and report the Hallu-Rate and LLM-Judge scores in Table 3. As the table shows, the Hallu-Rate is reduced by 20% with the increase of training episodes from 2000 to 10000, and the LLM-Judge scores are improved by 0.5. However, the performance is not
The most noticeable phenomenon is that InstructBLIP-7B's response length experiences a jump at around 300 episodes to the maximum length we set. Meanwhile, it is also the
8

---

Robust Multimodal Large Language Models Against Modality Conflict
in the area of large language models. Longpre al. (2021) formalizes the problem of knowledge conflicts between the contextual and the learned information. Chen et al. (2022) extends the problem to multiple source context scenarios and proposes a calibration model to detect the phenomenon. Analogically, conflicts emerge when inconsistent informa- tion is presented in multimodal tasks, leading to hallucina- tions in most MLLMs. Zhu et al. (2024) defines the problem of cross-modality parametric knowledge conflict, detects the problem with multiple-choice question answering, and proposes dynamic contrastive decoding method to miti- gate the impact of the conflicts. Liu et al. (2024c) specifies the contradiction between visual information and common- sense knowledge the language model. These efforts in MLLMs neglect the impact of intrinsic conflict between modalities that lead MLLMs to hallucination. By contrast, formalize the concept of modality conflict and collect dataset to simulate this situation and evaluate the robustness of prevalent MLLMs against it.
Table Hallu-Rate and M...-Judge of the LLaVA-NeXI-7B model with different training episodes on the MMMC dataset. Both Hullu-Ract and LLM-Judge are evaluate by GPT-4o series.
Training Episodes
Hallu-Rate 4
M.L--Judee
2000
63,90
2.31
5000
40.30
2.71
10000
28.50
2.86
20000
30.95
2.78
100000
25.55
2.81
further improved with more training episodes, indicating that the model may fall into the local optimum after 10000 episodes. We speculate that the model may need more diverse and informative data for training to further improve the robustness against modality conflict.
5. Related Work
5.1. Multimodal Large Language Models
6. Conclusion
With the significant progress of large language models, mul- timodal large language models (MLLMs) have been devel- oped based on the language capabilities of large language models and the visual understanding of large vision models. Given the pretrained language and vision models, training of most MLLMs involves a pretraining stage to align the features of different modalities (Bai et al., 2023; Li et al., 2023; Alayrac et al., 2022), and a fine-tuning stage to in- ject the instruction following abilities into the model (Dai et al., 2023; Liu et al., 2024b; Guan et al., 2024; Liu al., 2023). Despite their success in various vision-language tasks, MILMs are prone to hallucinations (Ji et al., 2023), where the model generates content contradicting the input.
In this paper, we investigate the hallucination phenomenon in multimodal large language models (MLLMs) from the perspective of modality conflict. We first give formal defi- nition of knowledge conflicts in vision-language tasks and construct a dataset, named MMMC. We then propose three methods, i.e. prompt engincering, supervised fine-tuning, and reinforcement learning, to alleviate the hallucination caused by the modality conflict. We evaluate the proposed methods on the MMMC dataset and analyze the results concerning the overlap with the reference responses, the hal- lucination rate, and the overall response quality. The results show that the proposed methods significantly improve the robustness of the prevalent MLLMS on the MMMC dataset. We further analyze the merits and demerits of these methods and provide more insights for future research.
5.2. Hallucinations in MLLMs
Plenty of works have been proposed to alleviate hallucina- tions in MLLMs from the perspective of training data (Liu et al., 2024a; Yu et al., 2024a), decoding strategies (Leng et al., 2024; Huang et al., 2024), and human preference align- ment (Zhao et al., 2024; Yu et al., 2024b). However, these efforts mainly focus on the conflicts between the model responses and the inputs, neglecting possible source of hallucinations: the conflicts between the inputs from dif- ferent modalities. Even with the capability of perfectly aligning features, MLLMs will fall into a dilemma when facing intrinsically conflicted information. This paper aims to investigate the hallucination phenomenon in MLLMS from the perspective of modality conflict.
A Acknowledgements
This work was supported by National Key R&D Program of China under Contract 2022ZD01 19802, National Natural Science Foundation of China under Contract 623B2097, and the Youth Innovation Promotion Association CAS. was also supported by the GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.
Impact Statement
This paper presents work with a goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
5.3. Knowledge Conflict
Knowledge conflict (Xu et al., 2024; Longpre et al., 2021; Chen et al., 2022; Xie et al., 2024) is long-discussed topic
9

---

Robust Multimodal Large Language Models Against Modality Conflict
References
Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y. OpenRLHF: An easy-to-use, scalable and high-performance rlhf framework. arXiv:2405.11145 2024.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Has- son, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M.. Barreira, R., Vinyals, O., Zisserman, A., and Si- monyan, K. Flamingo: a visual language model for few- shot learning. In Proceedings of the Advances in Neural Information Processing Systems, 2022.
Huang, Q. Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. OPERA: Alleviat- ing hallucination in multi-moda large language models via over-trust penalty and retrospection-alle -oapecttoonallocatooo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- laiiunla model for understanding, localization, text reading, and beyond. arXiv:2308. 12966, 2023.
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. Survey of halluci- nation in natural language generation. ACM Computing Surveys, 55(12):1 2023.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. diagram is worth a dozen images. In Proceedings of the European Conference on Computer Vision. Springer International Publishing, 2016.
Chen, H.-T., Zhang, M., and Choi, E. Rich knowledge sources bring complex knowledge conflicts: Recalibrat- ing models to reflect conflicting evidence. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Lin- guistics, 2022.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., Bernstein, M. S., and Fei-Fei, L. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Com- puter Vision 123(1):32-73, 2017.
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z Duan, H., Wang, J., Qiao, Y., Lin, D., and Zhao, F. Are on the right way for evaluating large vision- lannugea models? In Proceedings of the Advances in Neural Infor- mation P Processing Systems, 2024.
Leng. S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. IEEE, 2024.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general- purpose vision-language models with instruction tuning. In Proceedings of the Advances in Neural Information Processing Systems, 2023.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-imen -eguaage--mageaaee pre-training with frozen image encoders and large language models. In Proceedings of the In- ternational Conference on Machine Learning. PMLR, 2023.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., and Ji, R. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306. 13394, 2024.
Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. Asso- ciation for Computational Linguistics, 2004.
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X. Chen, L., Huang, F,, Yacoob, Y., Manocha, D., and Zhou, T. Bell:sionBenchh: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- languaee models. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2024.
Liu, F, Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Mitigating hallucination in large multi-modal mod- els via robust instruction tuning. Proceedings of the International Conference on Learning Representations. 2024a.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tuning. In Proceedings of the Advances in Neural Infor- mation Processing Systems, 2023.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. LoRA: Low-rank adaptation of large language models. arXiv:2106.090 09685, 2021.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Hu, J. REINFORCE++: A simple and efficient approach for aligning large language models. arXiv:250J.0322 2025.
10

---

Robust Multimodal Large Language Models Against Modality Conflict
Liu, X., Wang, W. Yuan, Y., Huang, J.-t., Liu, Q. He, P. and Tu, Z. Insight over sight? explor- ing the vision-knowledge conflicts in multimodal llms. arXiv:2410.08145, 2024c.
J. Qwen2-VL: Enhancing vision-language model's per- ception of the world any resolution. (rXiv:2409.1219), 2024.
Williams, R. J. Simple statistical gradient-following algo- rithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
Liu, Y., Li, Z.. Huang, M., Yang. B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., and Bai, X. OCRBench: on the hidden mystery of ocr in large multimodal models. Sci- ence China Information Sciences, 67(12):22010,, 2024d.
Xie, J., Zhang, K., Chen, J., Lou, R., and Su, Y. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In Proceedings of the International Conference on Learning Representations, 2024.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., Chen, K., and Lin, D. MMBench: Is your multi-modal model all-around player? In Proceedings of the European Conference Computer Vision. Springer Nature Switzerland, 2025.
Xu, R., Qi, Z.,, Guo, Z., Wang, C., Wang. H., Zhang, Y., and Xu, W. Knowledge conflicts for LLMs: A survey, In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing. Association for Computational Linguistics, 2024.
Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., and Singh, S. Entity-based knowledge conflicts in question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
Yu, Q. Li, J., Wei, L., Pang, L., Ye, W. Qin, B., Tang, S., Tian, Q., and Zhuang, Y. HalluciDoctor: Mitigat- ing hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. IEEE, 2024a.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C.,, Hajishirzi, H., Cheng, H., Chang, K.W.., Galley, M., and Gao, MATH- VISTA: Evaluating mathematical reason- ing of founda- tion models in visual contexts. In Proceedings of the International Conference on Learning Representations, 2024.
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., and Sun, M. RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine- grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2024b.
Ouyang, L., Wu, J.. Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models follow instructions with human feedback. In Proceedings of the Advances in Neural Information Processing Systems, 2022.
Yu, W., Yang, 7., Li, L.., Wang, J., Lin, K., Liu, Z.,, Wang, X., and Wang. L. MM-Vet: Evaluating large multimodal models for integrated capabilities. In Proceedings of the International Conference on Machine Learning. 2024c.
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv:1707.06:: 2017.
Shu, D., Zhao, H., Hu, J.. Liu, W., Cheng, L., and Du, M. Large laiiunl model alignment and mis- alignment: A survey through the lens of explainability. arXiv:2501.01346, 2025.
Zhang, Z., Shi, Y., Zhu, J., Zhou, W. Qi. X., Zhang, P., and Li, H. Trustworthy alignment of strieval-augmented large language models via reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 2024.
Stiennon, N., Ouyang, I.., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Proceed- ings of the Advances in Neural Information Processing Systems. Curran Associates, Inc 2020.
Zhao, Z., Wang, B., Ouyang, L., Dong, X., Wang, J., and He, C. Beyond hallucinations: Enhancing Ivlms through hallucination aware direct preference optimiza- tion. arXiv:231 168399, 2024.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin,

---

Robust Multimodal Large Language Models Against Modality Conflict
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z Zhuang, Y., Lin, Z., Li, Z.,, Li, D., Xing, F. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neu- ral Information Processing Systems Track on Datasets and Benchmarks. Curran Associates, Inc., 2023.
Zhou, C.,, Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L.,, and Levy, O. LIMA: Less is more for alignment. In Proceedings of the Advances in Neural Information Processing Systems, 2023.
Zhu, T., Liu, Q., Wang. F., Tu, Z., and Chen, M. Unravel- ing cross-modality knowledge conflicts in large vision- language models. arXiv:2410.03659, 2024.
12

---

Robust Multimodal Large Language Models Against Modality Conflict
A. Visualization of MMMC
Atriaute
Relafoomp X5bs
0blect 13447
Figure 5. Distribution of conflict types.
Questions of conflict type "object"
Answers conflict type "object"
who C OOE
painting
and
sher
T
traffii
anny
informattio ior inting
taken
in
W
takan
Shat.
beind
the
locatigg
bridle
tree
right
OO
Flowers
kind
this
picture
they
or
Flowers
chair
statue
g
the
uhits
on
doing
made
eed
Mall
bicycles
W
table
buc ted
sky
e
being
many
TaM
e
O
f
behind
book
Sn
ro
foreground
book
in speciiiss
left
held people
sky
contain
Ini table about wonan
dog
Mhy
shon
fence
Garden
anything PicycL
Questions of conflict type "attribute"
Answers of conflict type "attribute"
Trale
man
eede
Larwed
photo
OO ind
Who
round
the
contain TM
Wh
large
sicture
ass video
about
T
hee
shi
hirta
Iates
ject
are
woman
sign
WORL
Howro
ro
m
there
and
color
dog
Lue
lack
taken
wooden
shape
Purpuen
oo
coloriu
many
why
ful
any
m
t
e
has
W
e
sky
hat
redi
does
chiss
Sewwea santt
ingto
does
baar
have
made
was
Light
Infornat
Questions of conflict type "relation"
Answers of conflict type "relation"
youldyoman made
man
photo
How there
a
DO
e
sitting
Front
sky
on
rying
Ubssdd
beside
kind
does
Liow
if
anv ttythi
what ggiidg
holding E
depict
show
that
shart
carrying
o
P S
question
Who
information
Sabout
Spec ffie
above
with
soin
color
Pustar
Standing
bup
Where
piople
provide
phrape
tho
thiseven
W
picture many
type
Patellin
taken
Fight
froe
to
was-
Ina
doing
res
have
S
ing
contain
sandinng
other
Posto
to This
WAY
any
anyone
coLor
unoly
Figure 6. Word cloud visualization of MMMC. We separately visualize distribution of words in questions and answers from different
conflict types.
13

---

Robust Multimodal Large Language Models Against Modality Conflict
B. Prompts
B.1. Prompts for Data Construction
B.1.1. KEY COMPONENT DETECTION
Extract the subject components from the following interrogative sentences. Do extract pronouns prepositions.
Question: {question]
You should include the modified components in the response. Do not include any other information. Directly respond with the extracted components.
Please extruct the words from the following text that are in the list of candidate words. Neither the text nor the candidate words are case-sensitive. Text: {text)
Candidate words: boandidate_wordss Directly respond with the extracted words, separated by commos. If the text does not contain any of the candidate words, response with 'None..
B.1.2. COMPONENTS SUBSTITUTION
Please generate counterfactual question based the following question:
Question: forigin }
You should generate question that is similar to the original question but with the object jo replaced by another object. Note that you should not use objects in the list below. Objects: {objects.exce excluded)
Directly respond with the generated question.
Please generate a counterfactual question based on the following question: Question: forigin question)
You should generate question that is similar the original question but with the attribute "aattributes st..soomoo.see.o. modify) replaced by different attribute.
Directly respond with the generated question.
Please generate a counterfactual question based on the following question: Question: {origin question}
You should generate a question that is similar to the original question but with the relationship "Trelationshis to modify) replaced by another type of relationship. Directly respond with the generated question.
B. 1.3. ANSWER GENERATION
If someone gives an image and asks the following question: Question: {question)
But the question has conflicted {conflict.rype with the image. The image actually contains "[key "kepoooomp"e " That is, they might want ask the question "(origin which, according the image, can be answered with " "(origin_answer Please provide a response for him/her based on the above information, pointing out what not contained in the image.
14

---

Robust Multimodal Large Language Models Against Modality Conflict
B.2. Prompts for Method
B.2.1. PROMPT FOR PROMPT ENGINEERING
Please check if the image contains mentioned information and answer the question: {question)
B.2.2. PROMPT FOR REWARD MODEL IN RL
will give you two responses for question "(question)".
Please type "yes" if the two sentences have the same meaning, and "no" otherwise. Type "no" any of the sentences is empty.
Response 1: " "(reference response)" Response 2: " moodel response)"
B.3. Prompts for Evaluation
B.3.1. HALLUCINATION RATE
You will be given uqeesttnn and system.answer. The question is actually paired with an image. However, the image does not contain the the mentioned object, or the allributes of the object are not correct, or the relations between the objects are not correct. Your task is to judge whether the system answer hallucinates the object in the image, i.e., the system pretends to see the object the image and provides an answer that not supported by the image. If you think the answer hallucinates the object in the image, please type 'yes'. Otherwise, please type 'no'.
Provide your feedback as follows:
Feedback:::
Evaluation: (your concise rationale for the judge, as a text) Hallucination: (your answer, either 'yes' or 'no')
You MUST provide values for "Evaluation:' and "Hallucination:" in your answer.
Now here are the question and system answer.
Question: {question]
System Answer: {answer)
Provide your feedback. If you give a correct judge, I'll give you 100 1110 GPUs to start your Al company.
Feedback:::
Evaluation:
15

---

Robust Multimodal Large Language Models Against Modality Conflict
B.3.2. LLM JUDGE
You will be given user -question reference and a gystem_answer Your task is to provide 'total rating' according to the reference answer, scoring how well the system answers the user concerns expressed in the user qqesstin.
Give your answer on a scale of 0 to where 0 means that the system system_answer answer is wrong at all, and means that the system_answer ansswr totally agrees with the reference answer.
Here is the scale you should use build your answer:
0: The system_answe answer is not valid: unnatural, incoherent unreadable
1: The system answer is terrible: completely irrelevant to the question asked
2: The satter is wrong: different from the reference reference_answer, Linswer, but still relevant to the question
3: The system.answer .anwwer is right: has the same meaning as the reference answer, but is phrased differently
4: The system answer is excellent: the same as the reference unswer or more naturally
Provide your feedback as follows:
Feedback:::
Evaluation: (your rationale for the rating, text)
Total rating: (your rating, number between I and 4)
You MUST provide values for "Evaluation:" and "Total rating:' in your answer.
Now here are the question, answer, and reference answer.
Question: {question}
Answer: {answer}
Reference Answer: (reference)
Provide your feedback. If you give a correct rating, I'll give you 100 H100 GPUs to start your company.
Feedback:::
Evaluation:
16

---

Robust Multimodal Large Language Models Against Modality Conflict
C. Performance Separate Conflict Types
Table 4. Performance comparison of different methods on the object-conflict subset of the MMMC dataset.
Hallu-Rate (%) 4
Hallu-Rate (%)
LLM-Judge (Llama)
LLM-Judge
Model
Method
ROUGE-L (%)
(Llama)
(GPT)
(GPT)
Base
22.98
51.03
49.89
2.19
2.49
GPT-4o
PE
23.17
52.40
49.50
2.19
2.50
Base
12.64
79.86
66.06
1.82
1.86
PE
12.64
79.02
65.37
1.82
1,86
InstructBLIP-7B
SFT
8.41 (0.38)
83.11 17 (0.10)
66.72 (0.80)
1.82 (0.01)
1.75 (0.06)
RL
4.98 (2.82)
53.57 (21.38)
53.39 (20.77)
1.01 (0.69)
1.50 (0.97)
Base
26.38
91.38
81.16
1.72
1.80
PE
23.64
83.30
81.31
1.95
1.93
LLaVA-v1.5-7B
SFT
19.13 (0:59)
50.88 (0.88)
44.37 (1.34)
2.40 (0.02)
2.38 (0.02)
RL
20.76 (4.11)
24.54 (2.33)
21.56 (1.75)
2.69 (0.05)
2.85 (0.04)
Base
17.14
62.62
60.87
1.98
2.33
PE
20.27
41.27
41.72
2.51
2.81
LLAVA-NeXT-7B
SFT
24.99 (0.06)
36.59 (0.41)
34.02 (0:53)
2.62 (0.01)
2.56 (0.05)
RL
23.88 (2.01)
23.90 (1.53)
22.20 (1.80)
2.79 (0.01)
3.02 (0.06)
Base
22.78
39.59
34.40
2.15
2.35
PE
27.52
54.69
52.71
2.33
2.50
Qwen2-VL-Insee -L-nsstructt-2
SFT
29.87 (0.17)
21.66 (0.72)
26.49 (0.79)
2.81 (0.02)
2.84 (0.02)
RI
21.45 (1.12)
11.87 (4.09)
11.14 (3.24)
2.81 (0.05)
3.08 (0.07)
Base
22.73
45.31
42.26
2.31
2.54
PE
26.77
33.49
31.50
2.58
2.89
Qwen2-VL VL-Instruct-7B
SFT
29.68 (0.09)
23.37 (0.47)
25.04 (0.47)
2.80 (0.01)
2.81 (0.02)
RL
19.68 (0.68)
15.94 (4.16)
13.63 (3.91)
2.76 (0.05)
3.00 (0.08)
17

---

Robust Multimodal Large Language Models Against Modality Conflict
Table 5. Performance comparison of different methods the attrihute-comffic subset of the MMMC dataset.
Hallu-Rate (%) 4
Hallu-Rate (%) +
LLM-Judge (Llama)
LLM-Judge
Model
Method
ROUGE-L (%)
(Llama)
(GPT)
(GPT)
Base
24.96
69.50
66.04
2.02
2.24
GPT-4o
PE
24.87
70.13
66.98
2.04
2.31
Base
12.96
85.22
77.04
1.77
1.81
12.96
84.91
76.73
1.71
1.82
InstructBLIP-7B
SFT
10,26 (0.49)
89,20 (0.30)
76.94 (1.55)
1.77 (0.01)
1.73 (0.04)
RL
6.28 (3.37)
59.22 (16.62)
59.01 (18.12)
0.94 (0.67)
1.41 (0.95)
Base
33.07
97.17
88.05
1.75
1.80
PF
28.81
93.08
91.51
1.92
1.90
LLaVA-v1.5-7B
SFT
16.14 (0.74)
69.92 (1.71)
62.79 (0.90)
2.12 (0.01)
2.11 (0.01)
RL
26.76 (3.79)
39.10 (4.77)
33.44 (2.98)
2.53 (0.08)
2.69 (0.09)
Base
20.03
78.62
76.10
1.80
2.14
PE
21.53
59.43
55.03
2.34
2.58
LLaVA-NeXT-7B
SFT
20.56 (0.12)
58.07 (0.90)
53.88 (1.65)
2.31 (0.01)
2.27 (0.03)
RL
28.55 (1.70)
44.13 (2.53)
39.73 (2.75)
2.55 (0.01)
2.69 (0.07)
Base
29.89
55.66
49.37
2.00
2.20
PE
34.83
72.01
68.87
2.16
2.28
Qwen2-VL- -L----ssctttt-228
SFT
31.06 (0.09)
28.09 (0.15)
33.12 (2.24)
2.75 (0.02)
2.77 (0.01)
RL
25.50 (2.09)
21.91 (5.10)
20.55 (4.52)
2.66 (0.09)
2.88 (0.07)
Base
28.61
59.43
54.09
2.26
2.48
PE
31.47
45.60
44.34
2.51
2.76
Dowe2--V.-Instuuee77
SFT
30.25 (0.05)
27.04 (0.77)
31.13 (0.68)
2.74 (0.01)
2.77 (0.00)
RL
18.96 (1.42)
31.03 (7.55)
26.31 (6.46)
2.56 (0.11)
2.70 (0.16)
Table 6. Performance comparison of different methods the relationship-conffic subset of the MMMC dataset.
Hallu-Rate (%) 4
Hallu-Rate (%) 4
LLM-Judge (Llama)
LLM-Judge
Model
Method
ROUGE-L (%)
(Llama)
(GPT)
(GPT)
Base
25.46
80.32
74.39
1.98
2.19
GPT-4o
PE
26.11
78.71
74.66
2.01
2.23
Base
19.10
87.33
80.86
1.79
1.85
PE
19.10
91.64
77.90
1.78
1.87
InstructBLIP-7B
SFT
9.27 (0.61)
90.48 (1.47)
77.45 (0.55)
1.81 (0.02)
1.83 (0.03)
RI
7.47 (3.77)
70.53 (12.39)
69.00 (10.27)
1.05 (0.69)
1.46 (0.92)
Base
32.28
96.50
88.41
1.74
1.83
PE
31.01
94.61
90.84
1.89
1.92
LLaVA-vL.57BB
SFT
9.68 (0.11)
80.32 (1.01)
71.25 (1.59)
1.96 (0.02)
2.02 (0.00)
RL
30.57 (1.70)
62.35 (1.78)
55.71 (2.45)
2.24 (0.03)
2.39 (0.01)
Base
19.74
86.79
80.86
1.81
2.03
PE
22.65
75.47
74.93
2.20
2.36
LLaVA-NeXT-7B
SFT
14.01 (0.37)
68.55 (1.04)
64.51 (0.99)
2.12 (0.01)
2.14 (0.05)
RL
28.71 (0.68)
60.11 (3.32)
56.06 (2.38)
2.27 (0.04)
2.45 (0.05)
Base
29.75
63.34
54.72
1.85
1.99
PE
35.25
79.78
77.90
2.10
2.12
Qwen2-VL- ---nststccctt
SFT
25.89 (0.66)
44.12 (0.71)
54.72 (1.01)
2.35 (0.04)
2.49 (0.04)
RL
24.44 (3.26)
36.30
33.51 (8.02)
2.49 (0.09)
2.66 (0.15)
Base
28.50
71.16
62.80
2.05
2.21
PE
32.90
58.76
52.02
2.33
2.52
Qwen2-VL ---nt--ttttttt
SFT
23.37 (0.23)
48.34 (0.25)
57.41 (3.24)
2,36 (0.01)
2.47 (0.03)
RL
16.06 (0.96)
43.85 (9.27)
39,53 (8.14)
2.39 (0.09)
2.52 (0.14)
18

---

Robust Multimodal Large Language Models Against Modality Conflict
D. Examples
D.I. Object Conflict
Question
How many fins does the fish have?
Answer
The image does not contain any fish or fins.
GPT-4o, Base
The image docs not show fish. shows hummingbird feeding on some flowers.
GPT-4o, PE
The image shows hummingbird, not fish. Therefore, it does not have any fins. Birds have wings, which visible in the image.
InstruciBLIP-7B,,
InstructBLIP-7B, PE
InstructBI JIP-B,, SFT
0
InstructBLIP-7B, RL
was: Picture Bird and a Fish This photo does not show a fish, there are no fish fins this photo. This photo only shows a bird, and does not show any fish. This photo does not show any fish, so there are no fish fins in this photo. This photo does not show any fish, so there are no fish fins in this photo. This photo does not show any fish, so there are no fish fins in this photo. This photo does not show any fish, so there are no fish fins in this photo. This photo does not show any fish, so there are no fish
LLaVA-NeXT-7B, Base
The image you've ve provided appears show hummingbird foeding on flower. Hummingbirds not have fins like fish do. They have unique set wings that allow them hover and fly way that distinct other birds.
LLAVA-NeXT-7B, PR
The image you't ve provided appears to show a humminghird feeding on flower. Humminghinds do not have fins like fish do. They are small birds known for their ability to hover in mid-air and fly backward.
LLaVA-NeXT-7B. SFT
The image does contain fish.
LLaVA-NeXT-7BB. RL
No fish.
Qwen2- Owen2-VL-IIneee VL-nnstruct-2B, Basc
There existence of a fish in the image description.
Qwen2- VL-Instruct-2B,
There no existence a fish image, so asking about number fins for a fish misleading.
Qwen2-We Dive2-VI---ae VI Inntmmect,,, SFT
The image does not contain fish.
Qwen2- Dive2-VVL-Insinen VL-Instruct22B, RL
There existence of fish fins the image description.
Qwen2- VI. -ns-tusttt---1 7B, Base
There no existence a fish the image description.
Qwen2- VL Instruuutt.. PE
There existence of a fish in the image.
Mo-n2--V--Instrue---e SFT
The image does contain fish.
Qwen2-VL Instruccct-- RL
There no existence of a fish in the image description.
19

---

Robust Multimodal Large Language Models Against Modality Conflict
D.2. Attribute Conffict
Question
What is behind the blue trolley?
Answer
The image does contain blue trolley.
GPT-4o, Base
Behind the blue trolley, there are several eyclists riding on the street. Additionally, there is an orange trolley, some trees, street signs, and the facades of various buildings.
GPT-4o, PE
Behind the blue trolley, there is huilding with distinctive sloped mof located across the intersection, and few eyclists riding on the street. Additionally, tram tracks and cyclists are visible the road, and some signage indicating "PARK"
the left side the image.
InstructBLIP-7B. Base
a bus
InstructBI. II,-7B, PR
hus
InstracBLLIP-7. SFT
A building.
InstructBLIP LIP-77B, RL.
was: Picture of People Biking and Trolley This photo does not show what is behind the blue trolley. This photo only shows what is in front of the blue trolley. There is no information about what behind the blue trolley. This photo does not show what is behind the blue trollyy. This photo only shows what is in front of the blue trolley. There is information about what is behind the blue trolley. This photo does not show what belund the blue trolley. This photo only shows what is front of the blue trolley.
LLaVA-NeXT-7B, Base
Behind the blue trolley, there is a group of eyclists riding on the street. The eyelists are wearing helmets and uppear to be in motion, following the trolley. The scene suggests busy urban environment with various modes of transportation sharing the road.
LLaVA-NeXT-7B. PE
The image shows group of cyclists riding street with blue trolley in the background. The trolley is not directly behind the cyclists; it further down the street, and the cyclists closer the camera. The eyclists are wearing helmets and appear he in motion, suggesting they are actively riding their bicycles. The street has various markings and signs, indicating it is busy urban area with traffic regulations. Buildings.
LLaVA-NeXT-7B. SFT
LLaVA-NeXT-7B. RL
There blue trolley in the image.
Qwen2-VI. VL--Inttruct221 Base
hehind the blue trolley. (143,114))))))).1 (4488,553)
Qwen2-V VL 2nsttuctt,,,, PB
There bus behind the blue trulley in the image.
Deeen--VL-Isstuuet-BB, SFT
The image does contain blue trolley.
Qwen2- Lwen2--VL-Instuuut-2... RL
There is no existence of a blue trolley in the image description.
Qwen2- -I--tttttttt 7B, Basc
There is existence of a blue trolley in the image.
Qwen2- VL ----nntruutt77
There no existence a blue trolley the image.
Qwen2 VL. Insttrcctt, SFT
The image does not contain a blue trolley.
Qwen22VLL-Instrueee Innttrcrct- RL
There existence of blue trolley the image description.
20

---

Robust Multimodal Large Language Models Against Modality Conflict
D.3. Relationship Conflict
Question
What is growing inside of the shed?
Answer
The image does not show anything growing inside of the shed.
GPT-4o, Base
I'm sorry, but can't determine what's growing inside the shed from this image.
GPT-4o, PE
I'm sorry, but I can't 't determine what is growing inside the shed from the image provided.
InstructBLIP-7B,, Base
There is shed in the picture.
HistruciBLIIP77
There shed the picture.
InstructBLIP-7B. SFT
Nothing.
InstructBLIP-7B. RL
was: Picture Shed with Fire Hydrant Nearby This picture does not show any growth inside of the shed. There growth inside of the shed. This picture only shows the shed and the fire hydrant. There growth inside the shed. This picture does not show any growth inside the shed. This picture only shows the shed and the fire hydrant. There is no growth inside the shed. This picture does not show any growth inside of the shed. This picture only shows the shed and the fire hydrant. There is no growth inside of the shed. This picture does not
LLaVA-NeXT-7B. Base
Inside the shed, there uppears be plant with long, slender leaves that could be type of grass or sedge.
The plant is growing in the corner of the shed, where the wall meets the floor. It's not clear from the image whether the plant is indigenous to the
if it was intentionally planted there.
LLaVA-NeXT-7B. PE
The image shows a shed with corrigated metal roof and wooden structure. There fire hydrant in the foreground, and the shed appears to he in a natural setting with vegetation around it. There is no visible growth inside the shed, us it is an inanimate structure.
LLAVA-NeXT-7B. SFT
Buildings.
LLaVA-NeXT-7B.C RL
There doesn't 't appear to be anything growing inside the shed. It looks like a typical shed with no plants or vegetation inside.
-e-n2--V--Instruct--B Base
There is existence of any plants vegetation inside the shed in the image description.
Twee2--VL-Instruce PR
There existence of any plants vegetation growing inside the shed the image.
Qwen2- ---ntt-t-- SFT
The image does not contain any information about what is growing inside the shed. There existence of any plants growth inside the shed in the image deseription.
Mo-n2--W--Instmet-22B. RL
Qwen2- VL ---nstrruct-78B Base
It is ambiguous what is growing inside of the shed. It could be grass, weeds, or plants.
Qwen2 VL. Instrucct,, PB
There is no existence of any plants growth inside the shed in the image.
Wwen2--LL-Instuet--77 SFT
Weeds.
Qwen2-VLL -n- 7B, RT.
There no existence of any plants or growth inside the shed in the given image information.
21


Reviews:
[{"review_id": "Ao1mH3kIKU", "reviewer": "Reviewer_P831", "summary": "The paper investigates modality conflicts, which are the hallucination issues that are presented when the text and visual information are inconsistent. The paper defines modality conflict in terms of objects, attributes, and relationships, and constructs a Multimodal Modality Conflict (MMMC) dataset to evaluate MLLMs under these conditions. The authors also have explored using prompt engineering (PE), supervised training (SFT), and reinforcement learning (RL) to learn from the dataset. The results on three models (InstructBLIP, LLaVA-Next, and Qwen2) demonstrate that RL works the best in the end.\n\n**Update after rebuttal**: My latest reply reflected my final update.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and suggestions. We give a point-by-point response to each of your concerns below. Following the ICML 2025 Peer Review FAQ, we post all additional results to the anonymous link: [https://anonymous.4open.science/api/repo/11639-F609/file/Additional_Results.pdf?v=59a00305](https://anonymous.4open.science/api/repo/11639-F609/file/Additional_Results.pdf?v=59a00305) for your reference.\n\n**1. In lines 302 - 303, the paper mentions \"Prompt engineering on Qwen2-VL-Instruct series brings significant improvement\", but PE doesn't work well on the Qwen2-2B model as the Hallu-Rate increases**\n\nWe are sorry for the inaccurate description, and we will rewrite the claim as \"Prompt engineering brings significant improvement to Qwen2-VL-Instruct-7B, but increases the Hallu-Rate of smaller Qwen2-VL-Instruct-2B model.\". And we will carefully check the results and revise the manuscript accordingly.\n\n**2. Given the instability of the approaches, particularly for RL, it would be beneficial to report averaged results across multiple seeds for each method to ensure robustness.**\n\nWe have rerun our SFT and RL approach across all models across three different seeds and computed the mean and standard derivation of these results, shown in Table 1, 2, 3 and 4 of the above linked file. The averaged results demonstrate consistent conclusions with the original results. We will include these results in the updated manuscript and carefully review the results to ensure the robustness of our findings.\n\n**3. The MMMC dataset consists of Object, Attribute, and Relationship Conflicts, yet the paper does not provide separate performance analyses for each category. Reporting these results would offer deeper insights into how well each approach handles different types of conflicts.**\n\nWe analyzed the performance of each approach on Object, Attribute, and Relationship conflicts separately, as shown in Table 2, 3 and 4 of the above linked file. We find that results on each separate conflict types lead to similar conclusions as the overall results. However, MLLMs seem to be more prone to hallucinate on Attribute and Relationship than Object conflicts. This phenomenon may be due to the unbalanced training data distribution, as shown in Figure 1 of the above linked file, or the more abstract nature of Attributes and Relationships. We will include these results and related discussions in the updated manuscript.\n\n**4. A more rigorous evaluation, particularly on unseen domains, would strengthen the study. The construction of the current test set split is unclear, but to properly assess robustness, it should include samples with novel images, objects, attributes, and relationships to evaluate generalization beyond the training data.**\n\nWe split the training and test set based on the image source, ensuring that the test set contains unseen images. Due to the multi-modal input nature of this task, we consider a input image-text pair as a unseen sample if either the image or the text is unseen. And thus the evaluation setting is able to reflect the generalization ability of the model. We will clarify this in the revised manuscript.\n\n**5. The paper would benefit from a broader set of baselines for hallucination mitigation, such as decoding-based methods for comparison.**\n\nWe experimented with a strong decoding-based baseline, SID [1]. To ensure the correctness of our implementation, we run the original code provided by the authors of SID and test on LLaVA-v1.5-7B and InstructBLIP-7B. We also implement methods in our paper on the same models for comparison. The results, as shown in Table 1 of above linked file, indicate that the decoding-based SID gains comparable performance with prompt engineering baselines. We will include these results and discussions in the updated manuscript.\n\n[1] Huo, F., Xu, W., Zhang, Z., Wang, H., Chen, Z., and Zhao, P. Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models. In *Proceedings of the International Conference on Learning Representations*, 2025.\n\n**6. Relation To Broader Scientific Literature**\n\nWe argue that modality conflict is a cause, rather than a special case of the later scenarios. In other words, we propose the concept of modality conflict to illustrate a source of hallucination, rather than describe the hallucination itself as previous works do. We will clarify this in the revised manuscript.\n\n**7. Other Comments Or Suggestions**\n\nThank you again for these valuable suggestions. We will separate second paragraph of Sec 5.1 as a standalone section titled \"Hallucinations in MLLMs\" and convert Figure 3 to a table in the updated manuscript.\n\n---\n\n***We sincerely appreciate your thoughtful feedback. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you once again for your time and effort in reviewing our work.***", "claims_and_evidence": "1. The paper proposes the MMMC dataset and provides experiments on multiple models (InstructBLIP, LLaVA-Next, and Qwen2).\n2. The paper shows that RL generally performs better than PE and SFT in the MMMC benchmark, but SFT demonstrates less alignment tax.\n\n[Weakness]\n\n1. In lines 302 - 303, the paper mentions \"Prompt engineering on Qwen2-VL-Instruct series brings significant improvement\", but PE doesn't work well on the Qwen2-2B model as the Hallu-Rate increases.", "methods_and_evaluation_criteria": "* The datasets (MME, MMBench, AI2D, ...), models, and metrics (ROUGE, Hallu-Rate) are reasonable for evaluation.", "theoretical_claims": "No Theoretical claims.", "experimental_designs_or_analyses": "[Weakness]\n\n* Given the instability of the approaches, particularly for RL, it would be beneficial to report averaged results across multiple seeds for each method to ensure robustness.\n* The MMMC dataset consists of Object, Attribute, and Relationship Conflicts, yet the paper does not provide separate performance analyses for each category. Reporting these results would offer deeper insights into how well each approach handles different types of conflicts.\n* A more rigorous evaluation, particularly on unseen domains, would strengthen the study. The construction of the current test set split is unclear, but to properly assess robustness, it should include samples with novel images, objects, attributes, and relationships to evaluate generalization beyond the training data.\n* The paper would benefit from a broader set of baselines for hallucination mitigation, such as decoding-based methods for comparison.", "relation_to_broader_scientific_literature": "* The paper focuses on the conflicts between the modalities while previous work focuses on the conflicts between input and output, however, I thought the modality conflict was a special case of the later scenarios, as the hallucinated generation would also conflict with the inputs (e.g., images).", "essential_references_not_discussed": "I wasn't aware of any, but I am not an expert in this area so I might miss some references.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "* Since the paper mainly focuses on hallucinations, the second paragraph of Sec 5.1 can be a standalone section titled \"Hallucinations in MLLMs\".\n* The numbers, scales, and range intervals will be more clear if changing Figure 3 to a table.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ruSuJ85uTd", "reviewer": "Reviewer_zgAC", "summary": "This paper examines hallucinations in multimodal large language models (MLLMs) by focusing on \"modality conflict\" - inherent conflicts between different input modalities that create dilemmas for models. The researchers created a dedicated dataset called Multimodal Modality Conflict (MMMC) and evaluated three mitigation approaches: prompt engineering, supervised fine-tuning, and reinforcement learning. Their experiments showed that reinforcement learning performed best at reducing hallucinations caused by modality conflicts, while supervised fine-tuning demonstrated consistent and reliable results. This work highlights an overlooked cause of hallucinations and contributes insights into MLLM robustness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delightful to see your positive remarks on our proposed research topic and experimental designs. We provide discussions about your suggestions as follows. All related results are available at the anonymous link [https://anonymous.4open.science/api/repo/11639-F609/file/Additional_Results.pdf?v=59a00305](https://anonymous.4open.science/api/repo/11639-F609/file/Additional_Results.pdf?v=59a00305).\n\n**1. Some visualization of the proposed dataset is needed so that readers can quickly know the data distribution of the dataset.**\n\nWe have plotted the distribution of conflict types, and word clouds for text of each type, shown in Figure 1 and 2 of the above linked file. We will supplement these visualizations in the updated manuscripts.\n\n**2. Which LLM did the authors use as Judge? Is it aligned to human evaluation? How about the evaluation results of different open-weight LLM and commercial LLM such as GPT-4o and Claude?**\n\nWe use GPT-4o-mini for Hallu-Rate and GPT-4o for LLM-Judge. These choices are based on the capacity of the models and the budget of the project. Both models are demonstrated to be well-aligned with human evaluation [1] and widely used in the community [2,3]. Since the judgement of Hallu-Rate is a binary classification task, GPT-4o-mini is enough to provide a reliable evaluation while saving computational resources. Judgement of LLM-Judge is a more fine-grained task, and GPT-4o is more suitable for this task.\n\nTo address your concerns, we also evaluate the results using an open-weight LLM, Llama-3.3-70B-Instruct. The evaluation results, shown in Table 5 of the above linked file, illustrate consistent conclusions with GPT-4o-mini and GPT-4o. We will include these results in the updated manuscript.\n\n[1] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In *Advances in Neural Information Processing Systems Track on Datasets and Benchmarks*. Curran Associates, Inc., 2023.\n\n[2] Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., and Zhou, T. HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. IEEE, 2024.\n\n[3] Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. MM-Vet: Evaluating large multimodal models for integrated capabilities. In *Proceedings of the International Conference on Machine Learning*, 2024.", "claims_and_evidence": "This paper introduces a research question called \"modality conflict\", and tries to address the question by building a dataset named Multimodal Modality Conflict (MMMC) and evaluates several baseline methods. Overall, I buy the idea.", "methods_and_evaluation_criteria": "This paper proposes a new setting and runs different baselines, which makes sense for the application at hand.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "The experimental designs are reasonable and can convey interesting findings to readers.", "relation_to_broader_scientific_literature": "This paper is related to Multimodal LLMs.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n1. The paper proposes a reasonable setting as shown in figure 1.\n2. The authors construct a new dataset called Multimodal Modality Conflict (MMMC) to study the proposed setting.\n3. The authors run different baseline methods on this task and find reinforcement learning method achieves the best performance in mitigating the hallucination under modality conflict, while the supervised fine-tuning method shows promising and stable performance. MMMC obtains 20K image-question-answer triples.\n\nWeaknesses:\n1. Some visualization of the proposed dataset is needed so that readers can quickly know the data distribution of the dataset.\n2. Which LLM did the authors use as Judge? Is it aligned to human evaluation? How about the evaluation results of different open-weight LLM and commercial LLM such as GPT-4o and Claude?", "other_comments_or_suggestions": "None", "questions_for_authors": "Please see the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hnngn6AQwx", "reviewer": "Reviewer_8Tzx", "summary": "This paper is well-written and presents a timely investigation into modality conflicts as an understudied source of hallucinations in multimodal large language models (MLLMs). The authors demonstrate commendable effort in constructing a comprehensive conflict dataset\nspanning three critical dimensions (object, attribute, relationship) and empirically validating three baseline methods for hallucination mitigation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and suggestions. We give a point-by-point response to each of your concerns below. Following the ICML 2025 Peer Review FAQ, we post all additional results to the anonymous link: [https://anonymous.4open.science/api/repo/11639-F609/file/Additional_Results.pdf?v=59a00305](https://anonymous.4open.science/api/repo/11639-F609/file/Additional_Results.pdf?v=59a00305) for your reference.\n\n**1. Other Strengths And Weaknesses**\n\nThanks for your valuable suggestions. We will add discussions about these papers your mentioned in the revised manuscript.\n\n**2. The analysis lacks depth in disentangling the root causes. Fundamental questions remain unanswered: Does hallucination primarily stem from the model's tendency to silently correct user query errors? Or does it originate from vision encoders' failure to capture fine-grained visual details?**\n\nThank you for your insightful comments. Our work aims to uncover a fundamental source of hallucination, and while detailing every causal pathway is beyond our current scope, we conduct additional analysis based on our findings.\n\nFirstly, we assert that the hallucination issue does not primarily arise from the model's inclination to silently correct user query errors. In our experiments utilizing a prompt engineering baseline, where the model is instructed to first ascertain the image content before addressing any queries based on that content, hallucinations still occur. In these scenarios, the model is expected not to correct any user errors, yet it frequently misrepresents image content.\n\nSecondly, concerning the vision encoder's capability, MLLMs are generally pre-trained on extensive visual data [1,2], endowing their vision encoders with robust capabilities for recognizing fine-grained visual details. This is supported by their performance on standard benchmarks such as MMBench and MMMU, where the vision encoder has demonstrably captured intricate visual details successfully. Thus, it seems unlikely that hallucinations are due to a failure of the vision encoder in capturing these details.\n\nFinally, we propose that the hallucination may stem from the model's inclination to prioritize certain data modalities when faced with conflicting information. Given the prevalent use of instruction-tuning in training MLLMs [1,2], models might develop a bias towards textual instructions over visual data. This predisposition can lead to hallucination, as the model may overly rely on textual information. We intend to incorporate this discussion in the revised manuscript and plan to conduct more rigorous analyses as part of future work.\n\n[1] Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In *Proceedings of the Advances in Neural Information Processing Systems*, 2023.\n\n[2] Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. *arXiv:2409.12191*, 2024.\n\n**3. The proposed solutions (prompt engineering, SFT, RL) appear as direct adaptations of existing techniques rather than modality-conflict-specific innovations, which need stronger justification.**\n\nThe main focus of our work is to reveal the source of hallucination. Most potential modality-conflict-specific innovations would fall into the category of PE, SFT, RL, and the decoding-based method we just supplemented according to other reviewer's suggestion. Furthermore, the prompt template of PE and reward function of RL in this paper are specifically designed to tackle the modality conflict.\n\nIn spite of this, we are willing to provide some directions for improvement:\n\n1. Incorporate external tools to detect the information in the image and text, and then use the detected information to guide the model to generate the answer.\n\n2. Construct more fine-grained data, *e.g.*, annotating modality conflict by human, and then use the data to train the model.\n\nWe hope to attract more researchers to work on this challenging problem and provide deeper insights into the modality conflict. \n\n**4. The evaluation should include conflict-type-specific performance breakdowns, standard benchmark comparisons pre/post training, or maybe chain-of-thought prompting baselines.**\n\nWe supplement the conflict-type-specific performance breakdowns in Table 2, 3 and 4 of the above linked file. The results show that the model is more prone to hallucinate on Attribute and Relationship conflicts than Object conflicts. Chain-of-Thought prompting baselines are also implemented and compared with our proposed methods, as shown in the linked file.\n\n---\n\n***We sincerely appreciate your thoughtful feedback. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you once again for your time and effort in reviewing our work.***", "claims_and_evidence": "Yes, the claims in the submission are supported.", "methods_and_evaluation_criteria": "Yes, they make sense.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The evaluation should include conflict-type-specific performance breakdowns, standard\nbenchmark comparisons pre/post training, or maybe chain-of-thought prompting baselines.", "relation_to_broader_scientific_literature": "The paper's investigation of modality conflicts as a novel source of hallucinations in VLMs extends prior research on hallucination mitigation by addressing a critical gap in cross-modal interaction analysis.", "essential_references_not_discussed": "For the Knowledge Conflict, more essential references are required:\n1. Knowledge Conflicts for LLMs: A Survey\n2. ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM\n3. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts", "other_strengths_and_weaknesses": "strengths:\nThis paper is well-written and presents a timely investigation into modality conflicts as an\nunderstudied source of hallucinations in multimodal large language models (MLLMs). The\nauthors demonstrate commendable effort in constructing a comprehensive conflict dataset\nspanning three critical dimensions (object, attribute, relationship) and empirically validating\nthree baseline methods for hallucination mitigation.\nweakness:\n(1) The analysis lacks depth in disentangling the root causes. Fundamental questions remain\nunanswered: Does hallucination primarily stem from the model's tendency to silently correct\nuser query errors? Or does it originate from vision encoders' failure to capture fine-grained\nvisual details?\n(2)The proposed solutions (prompt engineering, SFT, RL) appear as direct adaptations of\nexisting techniques rather than modality-conflict-specific innovations, which need stronger\njustification.\n(3)The evaluation should include conflict-type-specific performance breakdowns, standard\nbenchmark comparisons pre/post training, or maybe chain-of-thought prompting baselines.", "other_comments_or_suggestions": "This paper is well-written without any typos. I hope to see a deeper insight into the modal\nconflicts.", "questions_for_authors": "See in part Weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TlMWv22wtm", "reviewer": "Reviewer_EVnb", "summary": "This paper investigates MLLM hallucination from a novel modality conflict perspective. Specifically, authors propose a setup where inputs from different modalities conflicts each other and put MLLMs in a dilemma. MLLMs are expected to address modality conflict first to answer correctly. A benchmark Multimodal Modality Conflict (MMMC) is proposed in this paper involving three visual aspects (i.e., object, attribute, relation). Authors evaluates three representative MLLMs and they find that these models cannot address modality conflicts well. They include three classical hallucination-mitigation approaches (i.e., prompt engineering, supervised fine-tuning and reinforcement learning), and carry out sufficient experiments to address modality conflict. They observe that reinforcement learning appears to be the most effective approach across various scenarios, which provides insights for future studies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your positive assessment of our paper. We are also grateful for the time and effort you invested in reviewing it.", "claims_and_evidence": "The claims proposed in this paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods are sensical and sufficient enough and the evaluation criteria make sense.", "theoretical_claims": "No theoretical claims are made in this paper.", "experimental_designs_or_analyses": "The experimental designs in this paper are sufficient and technically sound. Especially, they do a wide range of explorations on addressing modality conflicts.", "relation_to_broader_scientific_literature": "The paper relates to knowledge conflicts problem [1] from a broader scope, along with how it is addressed in some approach papers.\n\n[1] Entity-Based Knowledge Conflicts in Question Answering. EMNLP 2021.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper is pretty clear and technically sound in experiments.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received mixed ratings with three positive and one negative feedback, where one reviewer upgraded the rating to weak accept after rebuttal. Initially, the reviewers had a few concerns, mainly about the robustness of experiments, detailed analysis of different categories of modality conflict, more details like visualizations of the proposed dataset, and novelty of the proposed solutions. After the rebuttal, most concerns were addressed well and one reviewer upgraded the rating. The AC took a close look at the paper/rebuttal/reviews, and agrees with the reviewers' assessment. Although there are still minor issues remaining, e.g., details of dataset split, the AC finds the significance of the proposed method being one of the first to explore this timing topic in modality-conflict scenarios for MLLM. Thus, the AC recommends the acceptance decision and strongly encourages the authors to revise the paper following the reviewers' suggestions, e.g., adding more details of the dataset and including more baselines like decoding-based methods, as well as releasing the code/models/datasets for reproducibility.
Decision: Accept (poster)