Can Large Multimodal Models Actively Recognize Faulty Inputs? A
  Systematic Evaluation Framework of Their Input Scrutiny Ability
Authors: name: Haiqi Yang; name: Jinzhe Li; name: Gengxu Li; name: Yi Chang; name: Yuan Wu
Published: 2025-08-06 02:13:46

Abstract: Large Multimodal Models (LMMs) have witnessed remarkable growth, showcasing
formidable capabilities in handling intricate multimodal tasks with exceptional
performance. Recent research has underscored the inclination of large language
models to passively accept defective inputs, often resulting in futile
reasoning on invalid prompts. However, the same critical question of whether
LMMs can actively detect and scrutinize erroneous inputs still remains
unexplored. To address this gap, we introduce the Input Scrutiny Ability
Evaluation Framework (ISEval), which encompasses seven categories of flawed
premises and three evaluation metrics. Our extensive evaluation of ten advanced
LMMs has identified key findings. Most models struggle to actively detect
flawed textual premises without guidance, which reflects a strong reliance on
explicit prompts for premise error identification. Error type affects
performance: models excel at identifying logical fallacies but struggle with
surface-level linguistic errors and certain conditional flaws. Modality trust
varies-Gemini 2.5 pro and Claude Sonnet 4 balance visual and textual info,
while aya-vision-8b over-rely on text in conflicts. These insights underscore
the urgent need to enhance LMMs' proactive verification of input validity and
shed novel insights into mitigating the problem. The code is available at
https://github.com/MLGroupJLU/LMM_ISEval.