Comment on The Illusion of Thinking: Understanding the Strengths and
  Limitations of Reasoning Models via the Lens of Problem Complexity
Authors: name: C. Opus; name: A. Lawsen
Published: 2025-06-10 21:16:53

Abstract: Shojaee et al. (2025) report that Large Reasoning Models (LRMs) exhibit
"accuracy collapse" on planning puzzles beyond certain complexity thresholds.
We demonstrate that their findings primarily reflect experimental design
limitations rather than fundamental reasoning failures. Our analysis reveals
three critical issues: (1) Tower of Hanoi experiments systematically exceed
model output token limits at reported failure points, with models explicitly
acknowledging these constraints in their outputs; (2) The authors' automated
evaluation framework fails to distinguish between reasoning failures and
practical constraints, leading to misclassification of model capabilities; (3)
Most concerningly, their River Crossing benchmarks include mathematically
impossible instances for N > 5 due to insufficient boat capacity, yet models
are scored as failures for not solving these unsolvable problems. When we
control for these experimental artifacts, by requesting generating functions
instead of exhaustive move lists, preliminary experiments across multiple
models indicate high accuracy on Tower of Hanoi instances previously reported
as complete failures. These findings highlight the importance of careful
experimental design when evaluating AI reasoning capabilities.