When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs
  Preference Dynamics in MLLMs
Authors: name: Zhuoran Zhang; name: Tengyue Wang; name: Xilin Gong; name: Yang Shi; name: Haotian Wang; name: Di Wang; name: Lijie Hu
Published: 2025-11-04 04:11:31

Abstract: Multimodal large language models (MLLMs) must resolve conflicts when
different modalities provide contradictory information, a process we term
modality following. Prior work measured this behavior only with coarse
dataset-level statistics, overlooking the influence of model's confidence in
unimodal reasoning. In this paper, we introduce a new framework that decomposes
modality following into two fundamental factors: relative reasoning uncertainty
(the case-specific confidence gap between unimodal predictions) and inherent
modality preference( a model's stable bias when uncertainties are balanced). To
validate this framework, we construct a controllable dataset that
systematically varies the reasoning difficulty of visual and textual inputs.
Using entropy as a fine-grained uncertainty metric, we uncover a universal law:
the probability of following a modality decreases monotonically as its relative
uncertainty increases. At the relative difficulty level where the model tends
to follow both modalities with comparable probability what we call the balance
point, a practical indicator of the model's inherent preference. Unlike
traditional macro-level ratios, this measure offers a more principled and less
confounded way to characterize modality bias, disentangling it from unimodal
capabilities and dataset artifacts. Further, by probing layer-wise predictions,
we reveal the internal mechanism of oscillation: in ambiguous regions near the
balance point, models vacillate between modalities across layers, explaining
externally observed indecision. Together, these findings establish relative
uncertainty and inherent preference as the two governing principles of modality
following, offering both a quantitative framework and mechanistic insight into
how MLLMs resolve conflicting information.