Hidden in Plain Sight: Probing Implicit Reasoning in Multimodal Language
  Models
Authors: name: Qianqi Yan; name: Hongquan Li; name: Shan Jiang; name: Yang Zhao; name: Xinze Guan; name: Ching-Chen Kuo; name: Xin Eric Wang
Published: 2025-05-30 21:47:28

Abstract: Multimodal large language models (MLLMs) are increasingly deployed in
open-ended, real-world environments where inputs are messy, underspecified, and
not always trustworthy. Unlike curated benchmarks, these settings frequently
involve instructions that refer to missing objects or contradictory facts, rely
on ambiguous references, or request infeasible actions. In such cases, success
hinges not on task execution alone, but on a model's ability to detect when
something is silently wrong. This paper presents a systematic analysis of how
current MLLMs handle such implicit reasoning scenarios: cases where the flaw is
not explicitly stated but must be inferred from context. Using a curated
diagnostic suite spanning four categories of real-world failure modes, we
evaluate six MLLMs, including o3 and GPT-4o, and find that models frequently
fail to surface hidden issues, even when they possess the necessary perceptual
and reasoning skills. Explicit prompting reveals that the underlying
capabilities exist but are often suppressed in favor of user compliance. We
further show that simple inference-time interventions, such as cautious persona
prompting and, in particular, requiring a clarifying question, can dramatically
recover performance. Our findings highlight a persistent gap between reasoning
competence and behavioral compliance in current MLLMs and suggest practical
strategies for making these models more trustworthy in underconstrained
environments.