Are Your LLMs Capable of Stable Reasoning?
Authors: name: Junnan Liu; name: Hongwei Liu; name: Linchen Xiao; name: Ziyi Wang; name: Kuikun Liu; name: Songyang Gao; name: Wenwei Zhang; name: Songyang Zhang; name: Kai Chen
Published: 2024-12-17 18:12:47

Abstract: The rapid advancement of Large Language Models (LLMs) has demonstrated
remarkable progress in complex reasoning tasks. However, a significant
discrepancy persists between benchmark performances and real-world
applications. We identify this gap as primarily stemming from current
evaluation protocols and metrics, which inadequately capture the full spectrum
of LLM capabilities, particularly in complex reasoning tasks where both
accuracy and consistency are crucial. This work makes two key contributions.
First, we introduce G-Pass@k, a novel evaluation metric that provides a
continuous assessment of model performance across multiple sampling attempts,
quantifying both the model's peak performance potential and its stability.
Second, we present LiveMathBench, a dynamic benchmark comprising challenging,
contemporary mathematical problems designed to minimize data leakage risks
during evaluation. Through extensive experiments using G-Pass@k on
state-of-the-art LLMs with LiveMathBench, we provide comprehensive insights
into both their maximum capabilities and operational consistency. Our findings
reveal substantial room for improvement in LLMs' "realistic" reasoning
capabilities, highlighting the need for more robust evaluation methods. The
benchmark and detailed results are available at:
https://github.com/open-compass/GPassK.