The Relationship Between Reasoning and Performance in Large Language
  Models -- o3 (mini) Thinks Harder, Not Longer
Authors: name: Marthe Ballon; name: Andres Algaba; name: Vincent Ginis
Published: 2025-02-21 17:59:13

Abstract: Large language models have demonstrated remarkable progress in mathematical
reasoning, leveraging chain-of-thought and test-time compute scaling. However,
many open questions remain regarding the interplay between reasoning token
usage and accuracy gains. In particular, when comparing models across
generations, it is unclear whether improved performance results from longer
reasoning chains or more efficient reasoning. We systematically analyze
chain-of-thought length across o1-mini and o3-mini variants on the Omni-MATH
benchmark, finding that o3-mini (m) achieves superior accuracy without
requiring longer reasoning chains than o1-mini. Moreover, we show that accuracy
generally declines as reasoning chains grow across all models and compute
settings, even when controlling for difficulty of the questions. This accuracy
drop is significantly smaller in more proficient models, suggesting that new
generations of reasoning models use test-time compute more effectively.
Finally, we highlight that while o3-mini (h) achieves a marginal accuracy gain
over o3-mini (m), it does so by allocating substantially more reasoning tokens
across all problems, even the ones that o3-mini (m) can already solve. These
findings provide new insights into the relationship between model capability
and reasoning length, with implications for efficiency, scaling, and evaluation
methodologies.