Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning
  Capabilities Through Evaluation Design
Authors: name: Lin Sun; name: Weihong Lin; name: Jinzhu Wu; name: Yongfu Zhu; name: Xiaoqi Jian; name: Guangxiang Zhao; name: Change Jia; name: Linglin Zhang; name: Sai-er Hu; name: Yuhan Wu; name: Xiangzheng Zhang
Published: 2025-06-05 08:09:11

Abstract: Reasoning models represented by the Deepseek-R1-Distill series have been
widely adopted by the open-source community due to their strong performance in
mathematics, science, programming, and other domains. However, our study
reveals that their benchmark evaluation results are subject to significant
fluctuations caused by various factors. Subtle differences in evaluation
conditions can lead to substantial variations in results. Similar phenomena are
observed in other open-source inference models fine-tuned based on the
Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their
claimed performance improvements difficult to reproduce reliably. Therefore, we
advocate for the establishment of a more rigorous paradigm for model
performance evaluation and present our empirical assessments of the
Deepseek-R1-Distill series models.