ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates
Authors: name: Ling Yang; name: Zhaochen Yu; name: Bin Cui; name: Mengdi Wang
Published: 2025-02-10 18:51:47

Abstract: We present that hierarchical LLM reasoning via scaling thought templates can
effectively optimize the reasoning search space and outperform the mathematical
reasoning capabilities of powerful LLMs like OpenAI o1-preview and DeepSeek V3.
We train our ReasonFlux-32B model with only 8 GPUs and introduces three
innovations: (i) a structured and generic thought template library, containing
around 500 high-level thought templates capable of generalizing to similar or
relevant reasoning problems; (ii) performing hierarchical reinforcement
learning on a sequence of thought templates instead of long CoTs, optimizing a
base LLM to plan out an optimal template trajectory for gradually handling
complex problems; (iii) a brand new inference scaling system that enables
hierarchical LLM reasoning by adaptively scaling thought templates at inference
time. With a template trajectory containing sequential thought templates, our
ReasonFlux-32B significantly advances math reasoning capabilities to
state-of-the-art levels. Notably, on the MATH benchmark, it achieves an
accuracy of 91.2% and surpasses o1-preview by 6.7%. On the USA Math Olympiad
(AIME) benchmark, ReasonFlux-32B solves an average of 56.7% of problems,
surpassing o1-preview and DeepSeek-V3 by 27% and 45%, respectively. Code:
https://github.com/Gen-Verse/ReasonFlux