PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error
  Detection
Authors: name: Yusu Qian, name: Cheng Wan, name: Chao Jia, name: Yinfei Yang, name: Qingyu Zhao, name: Zhe Gan
Organization: Apple

We introduce PRISM-Bench, a benchmark of puzzle-based visual
challenges designed to evaluate not only whether models can solve problems, but
how their reasoning unfolds. Unlike prior evaluations that measure only
final-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visual
puzzle and a step-by-step chain-of-thought (CoT) containing exactly one error,
models must identify the first incorrect step. This setting enables
fine-grained assessment of logical consistency, error detection, and visual
reasoning. The puzzles in PRISM-Bench require multi-step symbolic, geometric,
and analogical reasoning, resisting shortcuts based on superficial pattern
matching. Evaluations across state-of-the-art MLLMs reveal a persistent gap
between fluent generation and faithful reasoning: models that produce plausible
CoTs often fail to locate simple logical faults. By disentangling answer
generation from reasoning verification, PRISM-Bench offers a sharper lens on
multimodal reasoning competence and underscores the need for diagnostic
evaluation protocols in the development of trustworthy MLLMs.