Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on
  Elementary School-Level Reasoning Problems?
Authors: name: Kai Yan, name: Yufei Xu, name: Zhengyin Du, name: Xuesong Yao, name: Zheyu Wang, name: Xiaowen Guo, name: Jiecao Chen

The rapid escalation from elementary school-level to frontier problems of the
difficulty for LLM benchmarks in recent years have weaved a miracle for
researchers that we are only inches away from surpassing human intelligence.
However, is the LLMs' remarkable reasoning ability indeed comes from true
intelligence by human standards, or are they simply reciting solutions
witnessed during training at an Internet level? To study this problem, we
propose RoR-Bench, a novel, multi-modal benchmark for detecting LLM's
recitation behavior when asked simple reasoning problems but with conditions
subtly shifted, and conduct empirical analysis on our benchmark. Surprisingly,
we found existing cutting-edge LLMs unanimously exhibits extremely severe
recitation behavior; by changing one phrase in the condition, top models such
as OpenAI-o1 and DeepSeek-R1 can suffer 60% performance loss on elementary
school-level arithmetic and reasoning problems. Such findings are a wake-up
call to the LLM community that compels us to re-evaluate the true intelligence
level of cutting-edge LLMs.