Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during
  Multi-Hop Analysis
Authors: name: Anushka Yadav; name: Isha Nalawade; name: Srujana Pillarichety; name: Yashwanth Babu; name: Reshmi Ghosh; name: Samyadeep Basu; name: Wenlong Zhao; name: Ali Nasaeh; name: Sriram Balasubramanian; name: Soundararajan Srinivasan
Published: 2025-08-06 17:58:36

Abstract: The emergence of reasoning models and their integration into practical AI
chat bots has led to breakthroughs in solving advanced math, deep search, and
extractive question answering problems that requires a complex and multi-step
thought process. Yet, a complete understanding of why these models hallucinate
more than general purpose language models is missing. In this investigative
study, we systematicallyexplore reasoning failures of contemporary language
models on multi-hop question answering tasks. We introduce a novel, nuanced
error categorization framework that examines failures across three critical
dimensions: the diversity and uniqueness of source documents involved ("hops"),
completeness in capturing relevant information ("coverage"), and cognitive
inefficiency ("overthinking"). Through rigorous hu-man annotation, supported by
complementary automated metrics, our exploration uncovers intricate error
patterns often hidden by accuracy-centric evaluations. This investigative
approach provides deeper insights into the cognitive limitations of current
models and offers actionable guidance toward enhancing reasoning fidelity,
transparency, and robustness in future language modeling efforts.