ICML 2025

Satori: Reinforcement Learning with Chain-of-Action-Thought
Enhances LLM Reasoning via Autoregressive Search
Maohao Shen * 1 Guangtao Zeng * 2 Zhenting Qi * 3
Zhang-Wei Hong 1 Zhenfang Chen 4 Wei Lu 2 Gregory Wornell 1
Subhro Das 4 David Cox 4 Chuang Gan 4 5
Abstract
Large language models (LLMs) have demon-
strated remarkable reasoning capabilities across
diverse domains. Recent studies have shown that
increasing test-time computation enhances LLMs’
reasoning capabilities. This typically involves ex-
tensive sampling at inference time guided by an
external LLM verifier, resulting in a two-player
system. Despite external guidance, the effective-
ness of this system demonstrates the potential of
a single LLM to tackle complex tasks. Thus, we
pose a new research problem: Can we internal-
ize the searching capabilities to fundamentally
enhance the reasoning abilities of a single LLM?
This work explores an orthogonal direction fo-
cusing on post-training LLMs for autoregressive
searching (i.e., an extended reasoning process
with self-reflection and self-exploration of new
strategies). To achieve this, we propose the Chain-
of-Action-Thought (COAT) reasoning and a two-
stage training paradigm: 1) a small-scale for-
mat tuning stage to internalize the COAT reason-
ing format and 2) a large-scale self-improvement
stage leveraging reinforcement learning. Our ap-
proach results in Satori, a 7B LLM trained on
open-source models and data. Extensive empiri-
cal evaluations demonstrate that Satori achieves
state-of-the-art performance on mathematical rea-
soning benchmarks while exhibits strong gener-
alization to out-of-domain tasks. Code, data, and
models are fully open-sourced1.
* Core contributors of Satori team, contributed equally to this
work. Order determined by a dice roll.
1MIT 2Singapore University of Technology and Design 3Harvard
4MIT-IBM Watson AI Lab, IBM Research 5UMass Amherst.
Correspondence to: Maohao Shen <maohao@mit.edu>.
Proceedings of the 42 nd Internationa...