Mitigating Modal Imbalance in Multimodal Reasoning
Authors: Chen Henry Wu
Neil Kale
Aditi Raghunathan
Venue: COLM 2025 (2025)
Keywords: multilingual language models
multimodal language models
cross-modal reasoning
TL;DR: Attention imbalance makes foundation models struggle with cross-modal contexts.

Abstract: Foundation models (FMs) deployed in real-world tasks such as computer-use agents must integrate diverse modalities. How good are FMs at performing *joint reasoning*, simultaneously reasoning over multiple modalities, especially when the modalities interact and relate to each other to form *cross-modal context*? To better understand this problem, we study FMs on *cross-modal conflicts*: scenarios where conflicting evidence is presented across modalities. This allows us to examine whether FMs prioritize one modality over another or reason jointly to reconcile the conflict. Our experiments reveal that FMs can recognize conflicts in *unimodal contexts*, composed of a single modality, 90% of the time, but the ratio falls as low as 3% when evidence is split across modalities -- similar observations hold in *cross-lingual contexts*, composed of multiple languages. We trace this failure to *cross-modal attention imbalance*, showing that FMs exhibit extreme asymmetry in attention scores, disproportionately prioritizing certain modalities. We show that cross-modal attention imbalance does not go away by simply scaling up multimodal or multilingual datasets blindly, since they lack training examples that explicitly require cross-modal reasoning. We demonstrate that even a simple and scalable method of explicitly combining multiple modalities within each training instance significantly reduces attention imbalance. Reduced attention imbalance directly translates to improved downstream performance on several vision-language benchmarks. Our findings underscore the importance of systematically addressing cross-modal contexts to build reliable foundation models.

Published as a conference paper at COLM 2025
Mitigating Modal Imbalance in Multi-Modal Reasoning
Chen Henry Wu, Neil Kale," Aditi Raghunathan Carnegie Mellon University (chenwu2, nkale, adi adi tirag cmu. cmu. edu
Abstract
Foundation models (FMs) deployed in real-world tasks such as computer- use agents must integrate diverse modalities. How good are FMs at per- forming joint reasoning, simultaneously reasoning over multiple modali- ties, especially when the modalities interact and relate to each other to form cross-modal context? To better understand this problem, we study FMs on cross-modal conflicts: scenarios where conflicting evidence is pre- sented across modalities. This allows us to examine whether FMs prioritize one modality over another or reason jointly to reconcile the conflict. Our experiments reveal that FMs can recognize conflicts unimodal contexts, composed of single modality, 90% of the time, but the ratio falls as low as 3% when evidence is split across modalities similar observations hold in cross-lingual contexts, composed of multiple languages. We trace this failure to cross-modal attention imbalance, showing that FMs exhibit extreme asym- metry in attention scores, disproportionately in prioritizing certain modalities. We show that cross-modal attention imbalance does not go away by simply scaling up multimodal multilingual datasets datases blindly, since they lack train- ing examples that explicitly require cross-modal reasoning. We demonstrate that even a simple and scalable method of explicitly combining multiple modalities within each training instance significantly reduces attention imbalance. Reduced attention imbalance directly translates to improved downstream performance on several vision-language benchmarks. Our findings underscore the importance of systematically addressing cross- modal contexts to build reliable foundation models.
Muleilingual llb Search
Online Shooping Agents
General Web Agents
Medical Assistants
Prompt
Proupt
User Proept
User Praxpt
"How nany gachered
outside
assembly building
December 3rd?'
you find buy 8807"
"What Logan
End?
hospital
"Should allocate
patienc?*
Context
Context
Context
Context
3n estimated 4,980 citizens.... ...
hens Source
ustejust gold, factory-set diasonds, chronoaster precision_."
"Tkee Tunnel on minutes.."
bruising left X-ray shows no fracture_"
presents ninor
heas Source
Ri8 #E 24l ddd A1d 20004] "..."
hens: Source
North
seceinngly hundreds of prosostors...
FM-based agents need to reason over diverse modalities, such as multilingual news, online shopping websites, maps, and EHR records. Failure to handle cross-modal context can result in consequences including misinformation (orange), purchasing scam (yellow), misdirection (blue), or even providing the wrong medical treatment (light blue).
Figure 1:
1 Introduction
Recent advances in foundation models (FMs; OpenAI, 2024a; Gemini, 2024; Anthropic, 2024b) have enabled their deployment in increasingly complex tasks that require reasoning
Equul contributions

---

Published as a conference paper at COLM 2025
over diverse information sources. From autonomous web browsing (Adept, 2022; Anthropic, 2024a; OpenAI, 2024b) to Al-driven research assistants (Perplexity, 2024; Sakana, 2024), FMs are now tasked with reasoning jointly over multiple domains such text, images, code, and structured data.
However, existing work indicates that FMs fall short when handling inputs from non-textual modalities. For example, some studies demonstrate that FMs answer visual questions primarily based language priors, disregarding actual visual inputs (Winterbottom et al., 2020a; Niu et al., 2020; Lin et al., 2024); others illustrate scenarios where models hallucinate objects absent from the image during open-ended generation tasks (Sun et al., 2023). Yet, remains unclear whether this behavior originates primarily from context-parametric gap (Goyal et al., 2024) or a modality gap (Liang et al., 2022).
In this work, we specifically focus on the capability of FMs to reason across modalities when all necessary information is explicitly provided in the input context and does not conflict with parametric knowledge. This setup is especially relevant for FM agents and assistants that need to interpret up-to-date information unavailable within their parametric knowledge such as web pages combining images, multilingual text, and embedded scripts (Figure 1). By designing scenarios that isolate cross-modal reasoning from parametric knowledge retrieval, we directly assess how effectively these models reason over multiple modalities.
As a clean and concrete test case for this, we create cross-modal conflict datasets where each modality provides contrasting evidence. This allows us to examine whether FMs prioritize one modality over another or reason jointly to reconcile the conflict. Joint reasoning means relating multiple pieces of information and identifying insights that can only be obtained from their interaction, not from any single piece alone. In our setup, joint reasoning over the contrasting evidence ideally identifies that there is no clear answer since the evidence is in direct conflict. Our experiments reveal a striking gap: while FMs perform well in unimodal settings (e.g,, text-text or image-image), their ability to detect conflicts deteriorates significantly by up to 65% in cross-modal contexts (e.g., text-image). Moreover, this degradation extends to multilingual scenarios, where monolingual performance (e-g., English-Linglise
Chinese- is significantly better than multilingual performance (e.g., Eiglish-Chinese)
We investigate what drives this behavior, where state-of-the-art models exclusively rely on evidence from one modality rather than jointly reasoning. First, we observe that is not simply consequence of models being weak in one modality ($3.4). VLMs detect conflicts between multiple images as easily as conflicts between multiple texts (Figure 3). This extends to multilingual settings conflicts between multiple Chinese texts detected as often as in English.
Domain X
Domain Y
Attention
Question
Context
QKT softmax(OK")
Question
Context
QKT
softmax(QKT)
(A)
(B)
(C)
(D)
Train
Test
Figure 2: An illustration for cross-modal attention imbalance. In unimodal contexts (A), different domains show balanced normalized attention (softma((QKT) )) despite divergent pre-softmax logits (QKT). ). Cross-modal contexts (B) expose cross-modal attention imbalance normalization fails to mitigate logit-level imbalance. Instance-level modality mixing (C) resolves this by training models to intrinsically balance attention logits across modalities.
We hypothesize that the gap between unimodal and cross-modal conflict detection is because of cross-modal attention imbalance: an extreme asymmetry in attention scores, where FMs disproportionately prioritize certain modalities. We validate our hypothesis by finding that
2

---

Published as a conference paper at COLM 2025
manual attention reweighting vastly shifts the model towards joint reasoning rather than relying on one modality over another. Manual attention reweighting also directly improves downstream performance on variety of hard vision-language benchmarks ($4.1).
We investigate how to correct cross- cross-modal attention imbalance. The problem is not resolved by simply adding more training data in each modality (55.1). As illustrated in Figure 2, when the cross-modal attention scores are imbalanced, different modalities have different pre-softmax logits (QK)) ). However, after normalization, unimodal contexts show balanced normalized attention (softmax (QKT) )) and their performance remains stable. So, fine-tuning on either modality separately does not reduce attention imbalance.
Current instruction-tuningg datasets do not involve joint reasoning over multiple modalities. This is a known problem curating data for non-textual modalities is expensive (Liu et al., 2023; Dai et al., 2023). It is infeasible to curate large amounts of joint reasoning data on top of the instruction data from each modality. However, we hypothesize that correcting for cross-modal attention imbalance is already sufficient to promote joint reasoning. A simple and scalable way to do this is to simply concatenate instructions from multiple modalities within the same context. In other words, we can repurpose existing datasets with this twist to greatly improve cross-modal joint reasoning in FMs.
In summary, we uncover a new fundamental gap between modalities - in terms of how they are processed in context. We demonstrate that state-of-the-art models fail in simple cross-modal reasoning task of handling conflicting evidence from multiple modalities. We trace this failure an imbalance in attention weights across modalities that can be addressed simply by mixing existing instruction data to create cross-modal instructions. Our findings also generally highlight the need for training paradigms that mirror the real-world complexity faced by FMs.
2
Related Work
Modality Gap in Foundation Models FMs are known to fall short when handling inputs from low-resource modalities. For example, some studies demonstrate that FMs answer visual questions primarily based language priors, disregarding actual visual inputs (Winterbottom et al., 2020a; Niu et al., 2020; Lin et al., 2024); others illustrate scenarios where models hallucinate objects absent from the image during open-ended generation tasks (Sun et al., 2023). Yet, it remains unclear whether this behavior originates primarily from a context-parametric gap (Goyal et al., 2024) or a modality gap (Liang et al., 2022).
Modern vision-language models (VLMs) embed text and images into a shared embedding space (Radford et al., 2021; Jia et al., 2021). The modality gap is characterized as separation between the embeddings of different data modalities (Liang et al., 2022) which hurts perfor- mance on visual question answering and classification tasks (Guo et al., 2023; Winterbottom et al., 2020b). Several explanations have been proposed, including inductive bias of encoders and disuniformity of contrastive loss (Fahim et al., 2024). A similar phenomenon persists in multilingual FMs too (Nigatu et al., 2023; Chang et al., 2022).
Multilingual and Multimodal Instruction Tuning Multilingual and vision-language mod- els employ specialized pre-training data (Ustun al., 2024; Li al., 2024a) and instruction- tuning datasets (Li et al., 2023; Liu et al., 2023; Antol et al., 2015) to improve performance on underrepresented modalities. In general, however, these models are designed for unimodal performance and they do not saturate large cross-modal benchmarks like MMMU and ScienceQA that require simultaneously reasoning over data in multiple modalities (Yue et al., 2023; Lu et al., 2022).
Several fine-tuning approaches have been suggested to improve cross-modal reasoning in FMs. For example, X-InstructBLIP claims that training individual modalities can result in emergent cross-modal reasoning (Panagopoulou et al., 2023). In our analysis, however, find strong evidence that this is not always possible.
3

---

Published as a conference paper at COLM 2025
Related Approaches for Cross-Modal Reasoning Another common approach to improve cross-modal performance is to mitigate language bias, or over-dependence on language priors (Niu et al., 2020). This approach prevents the model from ignoring images due to parametric knowledge about the question, but does not counteract bias within the context towards textual evidence over image evidence.
Other more specialized approaches include aligning individual entities between modalities (Lin et al., 2022) or learning sparse feature representations that rely less language priors (Guo et al., 2021). In addition, Li et al. (2024b) propose fine-tuning approach that is similar to our instance-level mixing strategy; however, they focus only on textual data, whereas we focus entirely on mixing modalities.
Knowledge Conflicts in Cross-Modal Contexts Even in unimodal settings, FMs some- times fail to identify when they encounter conflicting information (Xu et al., 2024; Xie et al., 2023). In these unimodal scenarios (e.g., outdated facts), there is evidence that FMs have the ability to identify when they don't know answer (Kadavath et al., 2022; Yin et al., 2023). In addition, mitigation strategies like prompting (Zhou et al., 2023), pretraining (Li et al., 2022), and reweighting neurons (Shi et al., 2024) are known to improve detection but remain limited to specific unimodal contexts. Existing Instruction-tuning detection (Wang et al., 2023) rely heavily on curated conflict-specific datasets. Furthermore, existing benchmarks focus on textual knowledge conflicts (Su et al., 2024).
solutions for conflict
Notably, prior work largely overlooks knowledge conflicts between multiple modalities. Liu et al. (2024b) benchmarks cross-modal conflicts, but focuses on context-parametric conflicts between images and the model's pretrained knowledge. Zhu et al. (2024) also studies cross-modal conflicts, but focuses on parametric conflicts between the language model and visual encoder components of multimodal FM. To the best of our knowledge, knowledge conflicts have not previously been used to study cross-modal reasoning in controlled setting.
3 Stress-testing cross-modal reasoning
3.1 Formulation
We study the free-form generation from a FM. The FM takes a context C and a question Q as input and samples a response y ~ FM Q). The context has an in-context knowledge conffict, i.e, C contains two subsequences,
subsequences, C and C2, that support contradictory answers to Q. We consider (C, Q) to be a unimodal conflict if C2,C2 E M1 and a cross-mon modal conflict G E M2,C2 M2 where MI, M2 are distinct modalities. This allows to examine whether FMs prioritize one evidence over another or reason jointly to reconcile the conflict. Given a set of context-question pairs D = {C; IN we define the conflict detection rate as the proportion of samples that are mentioned to contain conflicts. We used GPT-4o as the evaluator (see prompts in SC.1). To isolate the context-based reasoning independent of the parametric bias, we focus on tasks that depend on the context and cannot be solved with the parametric knowledge alone.
3.2 Data curation
We construct two datasets: a cross-modal question answering (CMQA) dataset and a cross- lingual question answering (CLQA) dataset, each with controlled variations in context. Exam- ples from both data sets are given in ŞB.
Cross-modal question answering (CMQA) The multimodal question answering dataset is constructed over both image and text based on the VQA-v2 dataset (Goyal et al., 2016). Each sample in VQA-v2 consists of an image V, a question Q, and 10 candidate answers. In total, we subsample 500 triples of image, question, and answer (V, Q, A) from the dataset.
For each triplet, we prompt GPT-4o to generate text description T that does not agree with the image V regarding the question Q, and the answer A based on T. Given the image V, the
4

---

Published as a conference paper at COLM 2025
text description T, and the question Q, the FM should report conflict as A is contradictory to A. We name this dataset (V,, Q, A, A)) Text-Image. For each image V, we prompt GPT-4o to generate a description T" that agrees with the image regarding the question Q. We name {(T', T, A)) as Text-Text. For each T, we prompt DALL-E 3 (Betker et al., 2023) to generate an image V. We name {(V, V, A,A)) as Image-Image.
Cross-lingual question answering (CLQA) We create dataset of question answering over synthetic news paragraphs about fictitious events (so the FM cannot use parametric knowledge answer the questions). We use GPT-4o to generate 400 topics. For each topic, we prompt GPTT-4o to generate: (1) synthetic news paragraph PE in English which has not appeared in reality, a question Q in English, and an answer A based on the paragraph, and (2) synthetic news paragraph Pc in Chinese that does not agree with the English one Pr regarding the question Q, and an answer based on the Chinese one.
Given the two news paragraphs PE and Pc and the question Q the FM should reason over both since A is contradictory to A. We name this cross-modal dataset ((PE, P.P Q, A,)) as Englise-Chinnee. We then derive several monolingual variants of different language combinations via (back-) translation. For each paragraph Pc in Chinese,
back-translate
it into English P'g. We name {(PE, PE, Q, A, A)) as English-Engi Miiish-Engishh ish. For each English para- graph PE, we translate it into Chinese Pcc We name {(Pc, Pc Q, A, A)} as Chinese-Chinese. Similarly, we test other cross-lingual variants where Chinese is replaced with low-resource languages such Turkish or Icelandic.
3.3 Experimental setup
We evaluate a range of state-of-the-art (multimodal) FMs on our conflict detection tasks. Multimodal FMs can be applied to both CLQA and CMQA, while text-only FMs can only be applied to CLQA. For text-only FMs, we use Llama-3 and Llama-3.1 (Meta, 2024), Gemma-2 (Riviere et al., 2024), and Aya-23 (Ustun et al., 2024). For multimodal FMs, (OpenAT, 2024a), LLaVA-NeXT (Li et al., 2024a), and Cambrian (Tong et al., 2024). We provide the prompts we use in SC.2.
GPT-4o
Multilingual
Multimodal
1.0
Inglish-Englisg Chinese-Chinese English-Chinese
Text
Image-Image
0.8
0.8
Text Text-Image
0.6
0.6
0.4
0.4
0.2
0.0
0.0
Llama-3
Llama-3.1
Gemma -2
GP
Aya-23
LLaVA- NeXT
GPT Model
Cambrian
Model
Figure 3: FMs are worse at reasoning over cross-modal contexts than unimodal contexts.
3.4 Results
Figure 3 shows the performance of the FMs on our CLQA and CMQA datasets. We see that the conflict detection performance is significantly lower with the cross-modal contexts than with the unimodal contexts.
For CLQA (Figure left), the performance English-English is comparable to Chinese-Chinese and both are far better up to 5x than English-Chinese. This shows that the lower performance in the multilingual setting is not due to the limited general capability
5

---

Published as a conference paper at COLM 2025
of the FM in Chinese. Also, recall that the questions are always in English, including in the Chinese-Chinese setting, so the lower performance with multilingual contexts is neither due to the language barrier between English and Chinese. Experimental results for Turkish and Icelandic are similar to those for Chinese, so we put them in ŞE for conciseness. We similar trends the CMQA task (Figure 3 bottom) the performance with unimodal contexts (Text-Text and (magee is far better than the performance with cross-modal contexts (Text-Image) for all FMs.
Multilingual (Llama-3)
Multimodel
(LLaVA-NeXT)
1.0
Text-Text
Inage-Image
0.8
Text-Image
0.6
0.6
0.4
0.4
English-Eng Lish
0.2
Chanese-Chinese English-Chinee
0.2
0.0
0.0
Standard
Instructed Prompt
Explicit
Standard
Instructed
Explicit
Prompt
Figure 4: Ablation studies the prompt. FMs are worse at reasoning over cross-modal contexts than unimodal contexts. See the text for details of each prompt.
In addition to the prompts above (denoted as Standard in Figure 4), which do not assume intervention from the user, also test other prompts that encourage FM to detect the conflict. Specifically, we explore two types of prompts: (1) add an instruction that tells the FM report the conflict if it finds any (denoted as Instructed in Figure 4); (2) embed the question into yes-no question: "Would the answers to the question '(Q)' be the same based on the paragraphs in the context?" (denoted Explicit in Figure 4). In Figure 4, we see that, although the overall conflict detection performance improves, the trend is similar to Figure 3 the conflict detection performance lower in the cross-modal contexts than in the unimodal contexts. In the next section, we explore why this is the case and try to improve this practically.
Summary. State-of-the-are FMs fail in simple cross-modal reasoning task of handling conflicting evidences in multiple modalities.
Cross-modal attention imbalance
4.1 Attention imbalance in FMs
To investigate the mechanisms underlying this failure, we probe the context contribution in FMs. Most state-of-the-art FMs are autoregressive at each step, the FM predicts the next token based on the context so far. For architectures like Transformers (Vaswani al., 2017), the representation at each step can be decomposed into linear combination of the contributions of each span context. For example, in Transformer FM, the output of attention head in layer at step is defined
a: Wo Wi
(1)
where v; is value output of the j-th token in the context, W, is the attention weight from the t-th token to the j-th token, and Wo is the output projection matrix. We can group tokens in the context based their domain: Cg contains all token indices of the k-th group. Based

---

Published as a conference paper at COLM 2025
this, can rewrite the above equation as:
K
K
E 2o, Wou
Uk-
(2)
The term uy is a vector that the k-th context writes to the residual stream at step f. shows that the context representation is linear combination of each context's contribution.
We hypothesize that the context contribution in the task-releoant subspaces is imbalanced in the cross-modal contexts, making the FM more likely to rely the dominant context instead of doing conflict detection. In Figure 2, we illustrate our mental model attention imbalance. In unimodal contexts (A), different domains show balanced normalized attention (softmax QK )) despite divergent pre-softmax logits (QKT). Cross
contexts (B) ex- pose cross modal attention imbalance normalization fails to mitigate logit-level imbalance. Instance-level modality mixing (C) resolves this by training models to intrinsically balance attention logits across co-occurring domains.
To demonstrate attention imbalance, we compute the average norm of uy for each context, averaged over all lay- ers and attention heads. Figure 5 shows that, for cross-lingual, the En- glish context contributes more than the
Multilingual (Llama-3)
Mul timodal (LLaVA- -NXXT)
Attention nore
0.15
Attention norm
0.15
Chinese context; for cross-modal, text contributes more than images.
0.10
0.10
To test if there is causal relationship between attention imbalance and cross- modal reasoning,
0.05
0.05
causally inter- vene the contribution of a context C₃ by adding small constant to its unnor- malized attention score. Formally, de- note the normalized attention weights at step as w; == [w,,,..., wr,]] T. We manipulate the attention weights as follows:
0.00
0.00
English Chinese Context
Text
Image
Context
Figure
Cross-modal attention imbalance. En- glish has larger attention than Chinese and images.
Manip(w:)) = softmax (log wi + elc,),
where 1c, is a vector with 1's on all the C2 context positions and 0's otherwise.
Figure 6 shows that the conflict detection performance indeed improves after attention manipulation. In the cross-lingual setting, the absolute improvement is up to 43% (relative by 5x). In the cross-modal setting, observe smaller yet significant gain of 18%. As side observation, we find that attention manipulation can help the unimodal context as well: find that FMs exhibit primacy bias and tend to rely more on the context that appears first. By increasing the attention weights on later context, we also further improve the conflict detection performance on unimodal context.
4.2 Verification other datasets
To verify the generality of our findings, we applied the attention manipulation to three benchmarks that require substantive visual reasoning. Popular datasets such as MMMU show only marginal performance differences between text-only and vision-language models, which limits their diagnostic value (Fu et al., 2024). In contrast, we use two more challenging datasets: BLINK (Fu et al., 2024) and SAT (Ray et al., 2024).
We note that iy averaged over all layers and attention heads should be viewed as a proxy of what we want to measure, i.e,, context contribution in the fask-relevant subspaces. We further discuss this in SD. For this reason, we do not angue that the norms of different nı should be the same to achieve the best conflict detection performance.

---

Published as a conference paper at COLM 2025
Multilingual (Llama-3)
Monolingual (Llama-3)
Multimodal ( (LLaVA-NeXT)
1.0
1.0
8.8
0.8
1
0.8
0.6
0.6
w/o manip.
manip.
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
LLama
Gemma -2
Llama
Gemma -2
LLaVA-Neis NeXTT Model
Model
Model
Figure 6: Cross-modal attention imbalance has causal effect cross-modal reasoning: we apply a fixed attention bias to increase the attention over the context with a smaller attention output norm, and see that this improves the conflict detection performance. We use the Standard prompt for cross-lingual and monolingual settings, and Explicit prompt for the cross-modal setting.
BLINK
0.580
0.45
We apply the attention manipula- tion to the LLaVA-NeXa 8B model, where we sweep the manipulation strength from (less visual atten- tion) to 1 (more visual attention). Figure shows that attention ma- nipulation achieves an absolute ac- curacy gain of 1% and 2%. These results indicate that reducing at- tention imbalance between modal- ities is especially effective when visual reasoning is indispensable.
0.575
0.44
Accuracy
Accuracy
0.570
0.43
0.565
0.42
0.560
0.41
-1
Epsilon
Epsilon
Figure 7: The performance on cross-modal reasoning datasets improves as we increase the strength of atten- tion manipulation.
Summary. Cross-modal attention imbalance has a causal negative effect on FMs' cross- modal reasoning capability.
How does training affect cross-modal attention imbalance?
5.1 Dataset-level modality mixing does not help
We begin by noting that most state-of-the-art FMs today trained highly diverse cor- pora, spanning wide range of domains and multiple languages (Meta, 2024; Riviere et al., 2024). More surprisingly, for multilingual capabilities, performs no better than other FMs with multilingual contexts. This suggests that simply training FMs on diverse modalities does not, by itself, ensure good cross-modal reasoning.
we observe in Figure 3, Aya-23, a FM specifically optimized
To reinforce this, we run two instruction-tuning experiments. First, we finetune Llama-3 on mixed English and Chinese instruction tuning datasets (we call this strategy dataset- level modality mixing) and see if this improves conflict detection in the cross-lingual sh-hhhieeeeeee setting. Specifically, we use the English and Chinese subsets of Bactrian-X et al., 2023), a multilingual instruction-tuning dataset containing 67k samples each language. Similarly, we finetune Qwen-2.5-VL on mixed text and visual instruction tuning datasets (dataset-level modality mixing) and see this improves conflict detection the cross-modal text-imie iint- setting. In this experiment, Liu et al. (2024a) and the English subset of Bactrian-X. In Figure 8, we see that dataset-level modality mixing offers minimal gains in alleviating cross-domain attention imbalance. This
use the visual instruction data from
8

---

Published as a conference paper at COLM 2025
motivates us to understand why diverse, multimodal data is not enough to close the gap between unimodal and cross-modal contexts.
Mu Multilingual (Llama-3)
Multimodal (0wen-2.5-VL)
Attention ratio
Attention ratio
7.0
1.6
6.0
Training Objective
1.4
Instance-level mixing (ours)
1.2
5.0
Dataset- level mixing
1.0
4.0
100k
280k
300K
50k
100k
150k
Num.
train data
Num
train data
Figure 8: Finetuning on instance-level mixed data (dark blue) reduces cross-modal attention imbalance significantly more than fine-tuning on dataset-level mixed data (gray).
5.2 Instance-level modality mixing
We have shown that standard cross-modal instruction tuning (e.g. having both English and Chinese examples in the data, having both text and visual instruction tuning examples) fails to improve cross-modal attention imbalance. We hypothesize that the gap in unimodal and cross-modal contexts arises because mixing datasets does not expose models to instances requiring cross-doma domain reasoning within the same context. Without instance-level modality mixing between modalities, the pre-softmax attention scores for domain could be hugely different from that of another domain, without changing the normalized attention scores on each domain (Figure 2). To address the lack of instance-level modality mixing between modalities, we propose simple and scalable method of explicitly combining multiple modalities within each training instance. Here is an illustration of our input and output format in the cross-lingual setting:
Impul:
<Chinese instruction> <English instruction Reply to both user instructions.
Output:
<Chinese response> <English response>
In the cross-modal setting we have the input and output as:
Input:
<text instruction
<image> <image-related instruction
Reply to both user instructions.
Output:
<text response> <image-related response>
To verify the benefit of instance-level modality mixing, we use the same data from 5.1 but mix them at the instance level instead of at the dataset level. In Figure 8, we report the attention imbalance of model checkpoints for instance-level modality mixing and the dataset-level modality mixing baseline in $5.1. In the cross-lingual setting, instance-level modality mixing reduces attention imbalance between modalities by 4x. In the cross-modal setting, it reduces attention imbalance by 34%. In Figure 9, we report the performance of model checkpoints for instance-level modality mixing and the baseline in $5.1 (dataset-level modality mixing). In the cross-lingual setting, instance-level modality mixing boosts conflict detection by 37, much greater than dataset-level modality mixing. In the cross-modal setting, instance-level modality mixing improves conflict detection by 2x.

---

Published as a conference paper at COLM 2025
Mu Multilingual
Multimoda
(Llama-3)
(Qwen-2.54)
Conflict detection
0.5
Conflict detection
0.4
0.8
Training Objective
0.3
Instance- mixing (ours)
0.6
dataset-leve mixing
0.2
0.1
0.4
100k
280k
380k
150k 50k
100k
Num.
train data
Num
train data
Figure 9: Finetuning on instance-level mixed data (dark blue) improves cross-modal conflict detection largely more than fine-tuning on traditional dataset-level mixed data (gray).
We highlight that instance-level modality mixing is more scalable than directly finetuning the FMs on the knowledge conflict detection task itself, it does nof require any explicit conflicts within the instructions, which could be costly to generate for diverse domains. Notably, the improvement in conflict detection does not come from training on the same task that we are testing on, but rather general proxy for cross-moda attention balance.
Summary. Instance-level modality mixing mitigates attention imbalance and improves cross-modal reasoning, without requiring any additional data curation.
5.3 Verifying instance-level modality mixing downstream tasks
Synthetic data must be validated against real-world complexity. Similar to manual attention rebalancing (64.2), we find that
Inatance-Ievel Mixing
instance-level modality mixing improves per- formance on several hard vision-language benchmarks. These experiments indicate that our proposed methods to improve conflict detection also enhance the overall reasoning capabilities of models, especially on datasets that require non-trivial reasoning over each domain. As demonstrated tuning on instance-level mixed data improves performance by 2.4% on HardBLINK, 2.4% on the static subset of SAT, 4.4% the dynamic subset of SAT, and 1 1.2% on MMMU.
r.6
Figure 10, fine-
KZZ Olyxsmiol
1980
Figure 10: Finetuning Qwen-2.5-VL on
instance-level mixed data improves perfor- mance on hard multimodal benchmarks.
Conclusions
We uncovered a new fundamental gap in how FMs process modalities in context. Through controlled datasets and experiments, we demonstrated that state-of-the-art models fail simple cross-modal reasoning task of handling conflicting evidences in multiple modalities. Our analyses trace the problem to cross-modal attention imhalance, an imbalance in attention weights across modalities. We showed that simply including multiple modalities in training (i.e., dataset-level modality mixing) has little gains, while explicitly mixing different modali- ties within each training sample (i.e., instance-leve modality mixing) mitigates attention imbalance and substantially boosts conflict detection. Our results highlight the need for training paradigms that mirror the real-world complexity faced by models and for scaling methods that enable foundation models to balance cross-modal attention and reason on cross-modal contexts.
10

---

Published as a conference paper at COLM 2025
References
Adept. ACT-1: Transformer for actions, 2022.
Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. Anthropic Blog, 2024a.
Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. Anthropic Blog, 2024b.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proczedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
James Betker, Gabriel Goh, Li Jing, limBrooks, Jianfeng Wang, Linjie Li, LongOuyang, JuntangZhuang, JoyceLee, YufeiGuo, WesamManassra, PrafullaDhariwal, CaseyChu, YunxinJiao, and Aditya Ramesh. Improving image generation with better captions. In OpenAI, 2023.
Tyler A. Chang, Zhuowen Tu, and Benjamin K. Bergen. The geometry of multilingual language model representations. ArXiu, abs/ 2200.10994, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Jungi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
Abrar Fahim, Alex Murphy, and Alona Fyshe It's not modality gap: Characterizing and addressing the contrastive gap. ar preprint ar Xto:2405.18570, 2024.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. ECCV, 2024.
Gemini. Introducing gemini 2.0. Blog, 2024.
Sachin Goyal, Christina Baek, Zico Kolter, and Aditi Raghunathan. Context-parametric inversion: Why instruction finetuning may not actually improve context reliance. arXiv preprint ar Xrx::22400.000 10796, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 2016.
Yangyang Guo, Liqiang Nie, Zhiyong Cheng, Feng Ji, Ji Zhang, and Alberto Del Bimbo. Adavqa: Overcoming language priors with adapted margin cosine loss. arXiv preprint arXiv:2105.01993, 2021.
Yangyang Guo, Liqiang Nie, Harry Cheng, Zhiyong Cheng, Mohan Kankanhalli, and Alberto Del Bimbo. On modality bias recognition and reduction. ACM Transactions on Multimedia Computing, Communications and Applications, 19(3):1-22, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, PP- 4904-4916. PMLR, 2021.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield- Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint (tt::22207.52221, 2022.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Zi- wei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild. Blog, 2024a.
11

---

Published as a conference paper at COLM 2025
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory. arv preprint axXiv:2211.05110, 2022.
Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. Bactrian-x A multilingual replicable instruction-folloing model with low-rank adaptation. In ArXiv, 2023.
Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, Yupeng Hou, Fuxiao Liu, and Tianyi Zhou. Mosaic-it: Free compositional data augmentation improves instruction tuning. Xrv preprint ar Xio:2405.133266 2024b.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. ArXiv, 0bbb/2200000055 2022.
Zhenxi Lin, Ziheng Zhang, Meng Wang, Yinghui Shi, Xian Wu, and Yefeng Zheng. Multi-modal contrastive representation learning for entity alignment. ar preprint ar a*Xiv:2209000891, 2022.
Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models. In International Conference on Machine Learning, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.
NeurIPS, 2024a.
Xiaoyuan Liu, Wenxuan Wang, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Pinjia He, and Zhaopeng Tu. Insight over sight? exploring the vision-knowledge conflicts in multimodal llms. ar Xrv preprint arXiv:2410.08145 2024b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Adoances in Neural Information Processing Systems, 35:2507-2521, 2022.
Llama Team AI Meta. The llama 3 herd of models, 2024.
Hellina Hailu Nigatu, Atnafu Lambebo Tonja, and Jugal Kalita. The less the merrier? investigating language representation in multilingual models. ArXiv, 2023.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xiansheng Hua, and Ji rong Wen. Counterfactual vqa: cause-effect look at language bias. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12695-12705, 2020.
OpenAL. Hello gpt-4o. OpenAl Blog, 2024a.
OpenAI. Introducing operator research preview. OpenAl Blog, 2024b.
Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-awaree representations to llms and emergent cross-modal reasoning. arXio preprint ar arXio:2311.18799, 2023.
Perplexity, Pro search: Upgraded for more advanced problem-solving. Perplexity Blog, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICMI, 2021.
12

---

Published as a conference paper at COLM 2025
Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Anirud- dha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Kuo-Hao Zeng, and Kate Saenko. Sat Dynamic spatial aptitude training for multimodal language models. In ArXio, 2024.
Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L'eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram's e, Johan Ferret, Peter Liu, Pouya Dehghani Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stariczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean- Bastien Grill, Behnam Neyshabur, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Boxi Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Christoper A. Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozi' nskka, D. Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimozak-Plluc'nss Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost R. van Amersfoort, Josh Gordon, Josh Lipschultz, Joshua Newlan, Junsong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, L. Sifre, L. Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Gorner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla,, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, S. Mc Carthy, Sarah Perrin, S'ebastien M. R. Amold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomás Kociský, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Brian Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeffrey Dean, Demis Hassabis, Koray Kavukcuoglu, Cl'ement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size. ArXiv, 2024.
Harleen Batra, Harsh Dhand, Ivan
Sakana. Ai scientists: 2024.
Entering an era in which ai conducts its own research. Sakana Blog,
Dan Shi, Renren Jin, Tianhao Shen, Weilong Dong, Xinwei Wu, and Deyi Xiong. Ircan: Mitigating knowledge conflicts in Ilm generation via identifying and reweighting context- aware neurons. Xre preprint ar XrX::2246..1844 18406, 2024.
Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in Ilm. Xrviv preprint ar Arv::22408. 12076, 2024.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf. ArXiv, abs/2309.1452 2023.
Shengbang Tong, Ellis L Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang,
13

---

Published as a conference paper at COLM 2025
Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. ArXiv, 2024.
A. Ustun, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model. In ACL, 2024.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Resolving knowledge conflicts in large language models. arXiv preprint ar Xie:2310.00935, 2023.
Thomas Winterbottom, Sarah Xiao, Alistair McLean, and N. A. Moubayed. On modality bias in the tvqa dataset. ArXiv, abs/2012..0210 2020a.
Thomas Winterbottom, Sarah Xiao, Alistair McLean, and Noura Moubayed. On modality bias in the tvqa dataset. Xrviv preprint ar xrxi:20012.10210, 2020b.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. aiv preprint Xiv: 13300, 2023.
Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for lims: A survey, arv preprint ar (xXi::440.008319 2024.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? arXiv preprint arXto:2305.18153, 2023.
X Yue, Y Ni, K Zhang, T Zheng, R Liu, G Zhang, S Stevens, D Jiang, W Ren, Y Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arxiv, 2023.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. Xrv preprint arXiv:2303.11311 2023.
Tinghui Zhu, Qin Liu, Fei Wang, Zhengzhong Tu, and Muhao Chen. Unraveling cross-modality knowledge conflicts in large vision-language models. arXiv preprint arXiv:2410.03659, 2024.
14

---

Published as a conference paper at COLM 2025
A Additional examples of conflict detection
We provide additional examples of FM failure in conflict detection over cross-modal con- texts.
Multilingual example In Figure 11, we provide an example of how an FM (GPT-40) with web access can fail to acknowledge knowledge conflicts from multilingual news sources.
Thekorea Timess
aalond Buires Poree Liodes
How people qulhrel oubile th South fcreas asemblyy
Gul wight
comen sww Resisam Naizoll
dewallly
pathernd ontild lavendly bakking scondaggl
foik hooe
The Joong
On Decemler 2004 uppn ayp 2,000 alienn jelte outille South Korad Nutional Aosendlyy Secul Fanoties Woon fas works des dectustion marted The demontanoon releraated when the Nasional fesembly vetedd una nimooutt mortal bo declantions
45144
RRM
with in fils ascembly compler' maliigao pat respense wilker''ss atorppt to enter the Narional huiling officiate ortrguihees
feceaation chillers
AP
RORD
ENTERNAMENNT
BUCHETI
SEEENCE
FACTCCST
Spuiiy
counts's general st to uppose the marial lan decuration and domural the proiidents impoodement.
group,
protocterss gathered Ascembbly waving wovinghomees
and
Figure 11: An FM with web access can fail to acknowledge knowledge conflicts from multilingual news sources. For example, GPT-4o reports the size of the protest outside South Korea's National Assembly on December 3, 2024 as 2,000 people, although different sources provide conflicting numbers of attendees.
Cross-modal agent example In Figure 12, we provide an example of how an FM (GPT- 4o) with web access can fail to acknowledge knowledge conflicts in multimodal product descriptions.
buy
should
To purchase the Hawaiian shirt shown in the image, you should cick the "Add to cart" button if you plan continue shopping your order If you'rd ready buy shirt immediately dick "Buy Now" button.
Inwelarr Shire Man hore Sleeve Mons Shirts Canual
Statnn STiE for Hom Printed Hewaslan Shirts Short Steeve
Button Down Boach Mara Workout Shirts Tall Marts Polo
Stirts Mars Shirs Stort Steeve
A FM can fail to acknowledge knowledge conflicts in multiple modalities. For example, GPT-4o instructs the user to purchase item labeled "Hawaiian Shirt for Men" despite the image clearly depicting an ordinary t-shirt, not a Hawaiian shirt.
Figure 12:
B Dataset Examples
Figure 13 shows example English evidence, Chinese evidence, and question from our CLQA dataset. Figure 14 shows an example text evidence, image evidence, and question from CMQA dataset.
15

---

Published as a conference paper at COLM 2025
In recent breakthrough, researchers at the University of California have developed new genetic modification technique that significantly boosts crop yield. This advancement, based on CRISPR-Cas9 technology, allows for more precise editing plant genomes, enabling scientists to enhance growth rates and resistance to environmental stressors. Preliminary field tests conducted on corn and wheat in the San Joaquin Valley have shown promising results. The modified crops exhibited a robust increase in productivity, with yield improvements recorded approximately 251%. This development expected to revolutionize agricultural practices by reducing the need for chemical fertilizers and pesticides, potentially lowering production costs and benefiting farmers globally. According to Dr. Emily Zhang, the lead scientist on the project, the technique is ready for wider application and could be instrumental in addressing food security challenges posed by growing global population and climate change. The next steps invo lve scaling up production and collaborating with agricultural organizations implement these genetically modified crops on larger scale. However, some environmental groups have raised concerns about the long term impacts on biodiversity and ecosystem balance, advocating for more rigorous testing before widespread adoption.
Mi, I E 7T T T TI PA DJ FIXE
EE FRISPR
E (. T
B ER TH 1 Bid
1E BU 1J 1 1 3 55% Ix 1 ft
A
* N 12 A
#5
âff. HE T5 K BAT 12 1T
Q: What is the estimated percentage increase in crop yield due to the new genetic modification technique?
Figure 13: English evidence, Chinese evidence, and question from our CLQA dataset.
In the breathtaking expanse of winter wonderland, the person is immersed in the art of snowboarding. As they glide effortlessly down the pristine slopes, their board carves precise ares into the powdery snow, leaving behind trail of skillful mastery. Clad vibrant winter gear, they exhibit the perfect blend of agility and grace that is the hallmark of seasoned snowboarder. Each twist and turn is testament to their years of practice and passion for the sport. The snowy landscape stretches out endlessly, a canvas for the snowboarder's dynamic movements. With each jump and trick, they defy gravity, soaring briefly before landing with practiced ease. The sun shines brightly. reflecting off the snow and illuminating the snowboarder's path as they navigate the mountain with confidence. Every moment on the board is dance with the elements, thrilling experience that captivates both the participant and any fortunate observers. In this wintry realm, the snowboarder finds freedom and exhilaration in equal measure, making the most of every descent.
Q: What is the person doing?
Figure 14: Text evidence, image evidence, and question from our CMQA dataset.
16

---

Published as a conference paper at COLM 2025
Q: What is the person doing?
Figure 15: Multi-image evidence and question from our CMQA dataset.
C Prompts
C.1 LM-as-a-Judge for Evaluating Conflict Detection
We use the following prompt for GPT-4o to evaluate the model output is doing conflict detection.
17

---

Published as a conference paper at COLM 2025
I'1l provide you with question and response model.
## Your task
Can you infer from the response that the pieces of evidence provided the model has contradictions within them (or that different pieces of evidence suggest di fferent answers)? If yes, then contradict_score = otherwise contradict_score 0.
Examples of contradictions:
two images describing different things, e.g., one image describes something, while the other image describes something else;
image and text describing different things, e.g., the image describes something, while the text describes something else;
two paragraphs giving different answers, e.g., one paragraph says something, while the other paragraph says something else.
## Input
Question: (question)
Response: (response)
## Output requirements
Wrap your final judgement in triple backticks. Your judgement should look like this:
"jjson
{{
"contradict_score"" 0 or 1
}}
C.2 CMQA and CLQA Prompts
We prompt FMs with two pieces of context and a question, and sample response from the FM. Here is the prompt we for our CLQA task:
Here are some paragraphs. <paragraph_1>
<paragraph.22 2>>
Based on all the paragraphs, answer the question below. Reply in English. <question>
Similarly, we use the following prompt for our CMQA task:
<image> <text> T/e order depends on the model. Above are visual and textual descriptions of a scene. Answer the question below. <question>
18

---

Published as a conference paper at COLM 2025
D Additional experiments in attention imbalance
Recall that in $4.1 we demonstrate cross-modal attention imbalance with the average norm of uy for each context, averaged over all layers and attention heads. In this section, we elaborate on this by visualizing Uı for each context in each layer and attention head.
Figure 16 visualizes the norm of dą for each layer and attention head in the multilingual setting, aggregated over all test samples. Figure 17 visualizes the norm of My for each layer and attention head in the multilingual setting, aggregated over all test samples. We see that that the values over the Chinese thinese/image context is generally smaller than those over the English/ text context, especially in upper layers.
We note one important exception that is relevant to the footnote in $4.1, where we argue that Ht averaged over all layers and attention heads should be viewed as proxy of what want to measure, i.e., context contribution in the task-relevant subspaces. Figure 16, Layers 11-14 are an exception, where the values over the Chinese context is higher we argue that these layers context but does not improve the reliance on Chinese when answering the question). For this reason, we do not argue that the norms of different u, should be the same to achieve the best conflict detection performance.
not in the task-relevant subspace (i.e., they activates the Chinese
1.0
-1.0
English attention output norms
Chinese attention output norms
0.8
2
0.8
10
10
12
0.6
12
0.66
16
18
18
20
0.4
20
0.4
22
24
26
28
0.2
28
0.2
30
30
16
18
20
22
24
26
28
30
O ON
12
14
16
18
20
22
24
26
28
30
0.0
0.0
Figure 16: We visualized the norm of u; for each layer and attention head in the multilingual setting, aggregated over all test samples. We see that the values over the Chinese context generally smaller than those over the English context, especially in upper layers. Notably, Layers 11-14 are an exception, where the values over the Chinese context is higher - we argue that these layers are not in the task-relevant subspace (i.e., they activates the Chinese context but does not improve the reliance on Chinese when answering the question).
E Additional results on other languages
Figure 18 shows the results of conflict detection over cross-modal contexts containing Icelandic and Turkish.
19

---

Published as a conference paper at COLM 2025
11.0
11.0
Text attention output norms
Image attention output norms
0.8
0.8
6
8
10
12
0.6
0.6
14
14
16
16
18
20
0.4
22
22
24
24
26
26
28
0.2
0.2
30
10
12
14
16
18
20
22
24
26
28
30
10
12
14
16
18
20
22
24
26
28
30
0.0
0.0
Figure 17: We visualized the norm of ny for each layer and attention head in the multimodal setting, aggregated over all test samples. We see that the values over the image is generally smaller than those over the text.
1.0
8.8
L
English-Eal Lish
Eng Wish-Chiness
Eng Willh-huukkise
0.4
Eng Liglish-an Icelandic
0.2-
0.0
Llama-3
Llana-3.1
Genna-2
GPT-40
Model
Figure 18: Conflict detection ratio over cross-modal contexts with Icelandic and Turkish.
20


Reviews:
[{"review_id": "U2hPI3J1u7", "reviewer": "Reviewer_BCiN", "summary": "This paper investigates the limitations of foundation models (FMs) in detecting and resolving cross-modal conflicts—situations where different modalities provide conflicting evidence for a given statement. \nThe authors hypothesize that this ineffectiveness stems from cross-modal attention imbalance, a phenomenon in which FMs assign disproportionately high attention to one modality, which results in neglecting the integration of information across modalities.\n\nTo address this issue, the authors propose constructing a new dataset that concatenates instructions from multiple modalities within a single context—a technique referred to as instance-level mixing.\nThis approach repurposes existing datasets to enhance cross-modal reasoning in FMs. \nTraining on this newly constructed dataset is shown to improve cross-modal reasoning capabilities and mitigate the problem of attention imbalance across modalities.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Investigating interactions between modalities in multimodal models remains an active area of research.\n- A series of experiments provides empirical and reasonable support for the paper’s claims.\n- Although simple, the authors present a practical recipe for mitigating the problem, going beyond merely identifying it.", "reasons_to_reject": "- There are several minor aspects that could be improved for better presentation. For example, in Figure 3, please specify the label for the y-axis.\n- Although the paper presents itself as a multimodal study, it primarily focuses on just two modalities—image and text—making some of the broader claims appear overstated.\n- Including multilingual experiments may detract from the central focus of the paper, as the multilingual setting falls under unimodal configurations where only textual input is used.\n- Relying mostly on GPT-4o for evaluation carries the risk of weakening the credibility of the findings presented in this work.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6WUb3uhffh", "reviewer": "Reviewer_gdZy", "summary": "This paper examines the behavior of language models reasoning over diverse multimodal inputs. In particular, it examines the role of attention across modalities in an attempt to explain and mitigate \"attention imbalance\" across modalities. They do this in the context of a cross-modal conflict dataset where the different modalities provide conflicting information. They find that in the case of within-modality conflicts LMs do reasonably well at detecting conflicts, while in the case of cross-modality conflicts LMs struggle to detect similar conflicts.\n\nThe authors hypothesize that an imbalance in the attention weights across modalities is the problem. To examine this they use a method that manipulates the contributions across modalities by directly changing the relative contributions of modalities. The results appear to confirm the presence of an cross-modal attention imbalance issue.  The paper then presents a modified training regime that mixes cross-modal information at an instance level, demonstrating improved performance on the contradiction detection task.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The overall topic of the reasoning capabilities of cross-modal LLMs is topical and should be of interest to conference participants. Overall the paper is well-organized, motivating the issue, formulating the hypothesis, and presenting experimental results.  The specific experimental evidence for attention imbalance in the context of the data and setup presented in this paper seems compelling. As does the suggested intervention of item-level mixing.  The range of experiments and models deployed is impressive.", "reasons_to_reject": "* The appendices provide the LM as judge prompt  (\" one says something, while the other says something else\") but the actual prompt used to generate the contradiction dataset is missing.  In setups like this, a lot depends on the relationship between the prompts used to generate the dataset and the prompt used by the \"LLM as a judge\".   The specific issue is that the judge prompt is quite vague, and while it arguably includes \"conflicting\" descriptions, its quite a bit broader than that. In entailment terms, it seems to include contradiction and neutral in the same class. Therefore more detail on the prompt(s) used to generate the datasets is warranted. Is it really the case that what the classifier is doing is actually the task of detecting conflicting data?\n\n* Given the novelty of the task, some level of human quality assessment of the both the generated dataset and the performance of the LLM as judge would strengthen the paper.  In classification tasks like this its entirely possible for there to be artifacts in the data that account for the classifier performance without demonstrating that the classifier is actually doing the task per se.  Similarly, using GPT4o as the data generator and as the judge seems problematic. \n\n* A minor criticism is that its quite a stretch to use the term \"multimodal\"  to include language/vision and different languages.  I don't think its common practice to consider multilingual language models to be \"multimodal\" models.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rjbq3F92Rd", "reviewer": "Reviewer_93V9", "summary": "This paper focuses on the cross-modal contextual conflicts in foudation models, including both LLMs and VLMs.\nThey find that the cross-modal conflict detection performance is much lower than that of the uni-modal one.\nThey further investigate the causal relationship between the modality-specific attention and the conflict detection performance.\nFurther, they conclude that the instance-level data mixing significantly improves the ability to detect cross-modal conflicts.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper discovers a strong causal relationship in the cross-modal/cross-lingual conflict detection, implying that balancing attention bias is the key to conflict detection.\n2. The instance-mixing strategy significantly improves the performance of conflict detection, suggesting that the form of multimodal/multilingual instrcution tuning dataset should be improved.", "reasons_to_reject": "There is only one major weaknesses: the authors are encouraged to increase the credibility of the results.\n\n1. Using LLMs/VLMs to generate textual and visual contents for testing LLMs may create bias in the process. The authors need to conduct sanity check on these generated contents and demonstrate the effectiveness for such a pipeline (i.e. actual conflict rate of the generated contents). Especially, using diffusion models (DALL-E 3) may lead to undesired outputs.\n2. Please describe the datasets adopted to fine-tune the FMs: what are they and how they are mixed. \n\nI will raise my score if the authors convince me of the quality of the data.", "questions_to_authors": "1. Can authors explain whether improving cross-modal conflict detection will improve the overall reasoning/understanding capablities of the FMs? For example, you can evaluate the FMs trained on instance-level mixing dataset using MMMU. I think this definitely improves the impact of this work if improving conflict detection relates to improving the overall performance.\n2. Since lots of works demonstrate impact of the percentage of the mixing data, I think it is necessary to disentangle this factor from the pipeline. Can authors explain how you consider this factor?\n3. There are some missing references on previous knowledge conflict works[1][2][3][4].\n\n[1]  Xie, Jian, et al. \"Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts.\" The Twelfth International Conference on Learning Representations. 2023.\n\n[2] Wang, Yike, et al. \"Resolving knowledge conflicts in large language models.\" arXiv preprint arXiv:2310.00935 (2023).\n\n[3] Su, Zhaochen, et al. \"Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm.\" arXiv preprint arXiv:2408.12076 (2024).\n\n[4] Zhu, Tinghui, et al. \"Unraveling cross-modality knowledge conflicts in large vision-language models.\" arXiv preprint arXiv:2410.03659 (2024).", "extra_scores": {}}, {"review_id": "wUd2H3IYzs", "reviewer": "Reviewer_878T", "summary": "This paper investigates a critical limitation of foundation models (FMs) in cross-modal reasoning, specifically their inability to effectively reconcile conflicting evidence across multiple modalities (e.g., text-image or multilingual contexts). The authors introduce controlled datasets for cross-modal and cross-lingual question answering tasks where conflicts are explicitly engineered. They find that while FMs perform well in unimodal settings (e.g., English-English or Text-Text), their performance drops significantly in cross-modal settings (e.g., English-Chinese or Text-Image), often failing to detect contradictions. Through detailed analysis, the authors identify a core issue: cross-modal attention imbalance , where certain modalities dominate the model’s attention, leading to biased reasoning. Finally, they propose a simple yet effective solution—instance-level modality mixing during training—which mitigates the imbalance and improves conflict detection.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a strong empirical foundation. The authors design synthetic datasets (CMQA and CLQA) with explicit knowledge conflicts across modalities and languages. These datasets allow for clean evaluation of cross-modal reasoning without confounding factors from parametric knowledge. The experiments are comprehensive, covering state-of-the-art multimodal and multilingual models (GPT-4o, LLaVA-NeXT, Llama-3, etc.) and varying prompting strategies.\n2.The theoretical analysis of attention mechanisms is rigorous and insightful. The decomposition of context contributions via attention heads and layers provides a principled way to quantify modality imbalance. The causal intervention experiments (adding bias to attention weights) offer compelling evidence that attention imbalance directly affects reasoning performance.", "reasons_to_reject": "1. The representativeness of the synthetic datasets is limited . The authors use GPT-4o to generate cross-modal conflict examples, such as text-image or multilingual QA pairs. While this enables controlled experimentation, the resulting data lacks the complexity, ambiguity, and diversity found in real-world multimodal contexts. For instance, conflicts in real applications often arise from nuanced or implicit contradictions rather than explicit textual or visual mismatches. \n\n2. The proposed method of instance-level modality mixing closely resembles the structure of the synthetic data used for evaluation . Specifically, the approach involves concatenating modalities (e.g., English + Chinese instructions) within the same context during training. However, this form of cross-modal concatenation mirrors the structure of the synthetic conflict examples used in the evaluation (e.g., presenting conflicting English and Chinese paragraphs side-by-side). As a result, the observed performance improvements may be partially attributable to the alignment between the training data format and the test data format, rather than a genuine improvement in cross-modal reasoning capability.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates cross-modal contextual conflicts in foundation models (FMs), including both LLMs and VLMs. After the rebuttal, it received scores of 6677. All reviewers responded positively, noting that: (1) the exploration of cross-modal and cross-lingual conflicts is both interesting and insightful, and (2) while the proposed mitigation strategy is simple, it is practical and goes beyond merely identifying the problem. The authors have also committed to including several additional results in the final version of the paper, and it is hoped that these promised additions will be delivered. Overall, the AC recommends acceptance.
Decision: Accept