PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error
  Detection
Authors: name: Yusu Qian; name: Cheng Wan; name: Chao Jia; name: Yinfei Yang; name: Qingyu Zhao; name: Zhe Gan
Published: 2025-10-27 17:57:52

Abstract: We introduce PRISM-Bench, a benchmark of puzzle-based visual
challenges designed to evaluate not only whether models can solve problems, but
how their reasoning unfolds. Unlike prior evaluations that measure only
final-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visual
puzzle and a step-by-step chain-of-thought (CoT) containing exactly one error,
models must identify the first incorrect step. This setting enables
fine-grained assessment of logical consistency, error detection, and visual
reasoning. The puzzles in PRISM-Bench require multi-step symbolic, geometric,
and analogical reasoning, resisting shortcuts based on superficial pattern
matching. Evaluations across state-of-the-art MLLMs reveal a persistent gap
between fluent generation and faithful reasoning: models that produce plausible
CoTs often fail to locate simple logical faults. By disentangling answer
generation from reasoning verification, PRISM-Bench offers a sharper lens on
multimodal reasoning competence and underscores the need for diagnostic
evaluation protocols in the development of trustworthy MLLMs.