Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
Authors: Maohao Shen
Guangtao Zeng
Zhenting Qi
Zhang-Wei Hong
Zhenfang Chen
Wei Lu
Gregory W. Wornell
Subhro Das
David Daniel Cox
Chuang Gan
Venue: ICML 2025 poster (2025)
Keywords: reasoning
self-improvement
large language model
post-training
reinforcement learning
Primary area: deep_learning->large_language_models

Abstract: Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains. Recent studies have shown that increasing test-time computation enhances LLMs' reasoning capabilities. This typically involves extensive sampling at inference time guided by an external LLM verifier, resulting in a two-player system. Despite external guidance, the effectiveness of this system demonstrates the potential of a single LLM to tackle complex tasks. Thus, we pose a new research problem: *Can we internalize the searching capabilities to fundamentally enhance the reasoning abilities of a single LLM?* This work explores an orthogonal direction focusing on post-training LLMs for autoregressive searching (*i.e.,* an extended reasoning process with self-reflection and self-exploration of new strategies). To achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a two-stage training paradigm: 1) a small-scale format tuning stage to internalize the COAT reasoning format and 2) a large-scale self-improvement stage leveraging reinforcement learning. Our approach results in Satori, a 7B LLM trained on open-source models and data. Extensive empirical evaluations demonstrate that Satori achieves state-of-the-art performance on mathematical reasoning benchmarks while exhibits strong generalization to out-of-domain tasks. Code, data, and models are fully open-sourced.

Satori: Reinforcement Learning with Enhances LLM Reasoning via Autoregressive Search
Maohao Shen 11
Guangtao Zeng" ? Zhenting Qi
Zhang-Wei Hong' Zhenfang Chen Wei Lu Gregory Wornell
Subhro Das 4 David Cox Chuang Gan 45
Abstract
1. Introduction
Large language models (LLMs) have demon- strated remarkable reasoning capabilities across diverse domains. Recent studies have shown that increasing test-time computation enhances LLMs' reasoning capabilities. This typically involves ex- tensive sampling at inference time guided by an external L.M verifier, resulting in a two-player system. Despite external guidance, the effective- ness of this system demonstrates the potential of single LLM to tackle complex tasks. Thus, pose new research problem: Can we internal- ize the searching capabilities to fundamentally enhance the reasoning abilities of a single LLM? This work explores an orthogonal direction fo- cusing on post-training LLMs for autoregressive searching (i.e., an extended reasoning process with self-reflection and self-exploration of new strategies). To achieve this, we propose the Chain- of-Action- Thouttt (COAT) reasoning and a two- stage training paradigm: 1) a small-scale for- mat tuning stage to internalize the COAT reason- ing format and 2) a large-scale self-improvement stage leveraging reinforcement learning. Our ap- proach results in Satori, 7B LLM trained on open-source models and data. Extensive empiri- cal evaluations demonstrate that Satori achieves state-of ttate-of-the-art performance on mathematical rea- soning benchmarks while exhibits strong gener- alization to out-of-domain tasks. Code, data, and models are fully open-sourced!
Large language models (LLMs) have demonstrated re- markable performance across a wide range of reasoning tasks, including mathematical problems (Cobbe et al., 2021;
Hendrycks et al., 2021a), programming (Chen et al., 2021;
Zhuo et al., 2024) and logical reasoning (Han et al., 2024;
Liu al., 2020). One of the key techniques enabling these strong reasoning capabilities is Chain-of-Thought (CoT) prompting (Wei et al., 2022), which allows LLMs to ad- dress complex tasks by generating a series of intermediate reasoning steps. As a result, many early efforts focus on fine- tuning LLMs using scale, high-quality CoT reasoning chains, either through human annotation (Hendrycks et al., 2021a; Yue et al., 2024) or by distilling synthetic data from more advanced models (Yu al., 2024; Toshniwal al., 2024a; Ding et al., 2024). However, human annotation is extremely labor intensive, and distillation often limits the model's reasoning capabilities to certain level.
Apart from scaling up training resources, more recent work has focused on test-time scaling, i.e., allocating additional inference-time compute search for more accurate solu- tions. This often involves extensive sampling, either by generating multiple complete solutions (Wang et al., 2023) or by sampling multiple intermediate reasoning steps (Yao et al., 2024; Wan et al., 2024). These methods typically require external feedback to guide the search process, usu- ally through training an auxiliary reward model to rate final solutions or intermediate steps (Sun al., 2024; Wang et al., 2024a). However, such two-player frameworks incur more model-deployment costs and do not internalize the search capabilities into single LLM.
Core contributors of Satori team, contributed equally to this work. Order determined by dice roll.
Orthogonal to the above work, our study investigates a new direction that enables LLMs with autoregressive search capabilities, i.e., an extended reasoning process with self- reflection and self-exploration of new strategies. Specifi- cally, introduce the Chain-of-AAciinn-Thon mechanism, which enables LLMs take various meta- actions during problem solving. Unlike conventional post- training consisting of large-scale supervised fine-tuning (SFT) and reinforcement learning from human feedback
MITT' "Singapore University of Technology and Design Harvard "MIT-IBM Watson Al Lab, IBM Research *UMass Amherst. Correspondence Maohao Shen <maohao@mn mit.edu:>
(COAT)
Proceedings of the nnnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s). htttps:/ssatoe reasoni ing..nnig..... www.gg........... github..h.... io/

---

Satori: RL with Seeii--of-ActiinThhuggg Enhances LLM Reasoning via Autoregressive Search
(RLHF), we propose a novel two-stage training paradigm: (1) small-scale format tuning (FT) stage to internalize the COAT reasoning format and (2) a large-scale self- improvement stage that utilizes reinforcement learning with "Restart and Explore" (RAH) techniques. Our approach leads to the development of Satori, a 7B LLM trained on open-source base models and mathematic data that achieve superior performance on both in-domain and out-of-domain tasks. To summarize, our contributions are threefold,
Post-training LLMs for Reasoning. Recent advance- ments have focused on extensive post-training enhance reasoning. A line of work focus on constructing high-quality instruction-tuning datasets (Hendrycks et al., 2021a; Yue et al., 2024; Yu et al., 2024; Toshniwal et al., 2024a; Ding et al., 2024), but suffers from expensive annotatoin costs. More recent research has focused on self-imprrvemmnt ap- proaches, where models are trained on data generated by themselves (Zelikman et al., 2022; 2024; Singh et al., 2024; Zhang et al., 2024a). Additionally, reinforcement learning methods, particularly those based on Proximal Policy Op- timization (PPO) (Schulman et al., 2017a; Ouyang et al., 2022), have been demonstrated to be more effective, which typically leverage reward models to guide the learning pro- cess (Sun et al., 2024; Wang et al., 2024a; Yuan al., 2024).
1. Efficiency: Satori is a single LLM capable of autore- gressive search without external guidance (Section 6 and Section A). Moreover, this is achieved with mini- mal supervision and large-scale seff-improvement.
2. Effectiveness: Satori demonstrates superior perfor- mance on in-domain mathematical reasoning tasks and outperforms the instruct model built on the same base model (Section 5.1).
Enabling LLMS with Searching Abilities. Prompting- based approaches (Yao et al., 2024; Shinn et al., 2024; Hao et al., 2023; Qi et al., 2024a) guide LLMs to search for solutions via error correction and exploring alterna- tive paths. However, such approaches cannot fundamen- tally enhance the LLM's reasoning abilities. Moreover, recent work has pointed out the difficulties of LLMS in self-correction (Zhang et al., 2024b; Kamoi et al., 2024). Recent research has pivoted toward training LLMs for self- exploration. Some focused on enabling trajectory-level search iterativeyy identify errors in previous complete re- sponses and produce improved responses (Saunders et al., 2022a; Kumar et al., 2024; Qu et al., 2024; Havrilla et al., 2024). Another line of research has explored step-level search, which enables LLMs to identify and correct mis- takes in a more fine-grained manner. Some achieve this us- ing another model to provide step-level feedback (Xi et al., 2024; Setlur et al., 2024; Zhang et al., 2024c: Guan et al., 2025; Zhang et al., 2024d), but such two-player frameworks suffer from high costs for model deployment. SoS (Gandhi et al., 2024) is another closely related work that attempts to train single LLM to perform a tree search as flattened string. However, the effectiveness of SoS has primarily been shown on simple symbolic tasks, and its ability to generalize to more complex problems remains to be explored.
3. Generalizabillty: Unlike recent research on math rea- soning, Satori exhibits strong transferability out-of- domain tasks and demonstrates universal capabilities for self-reflection and self-exploration (Section 5.2).
2. Related Work
We summarize the literature that is closely aligned with the scope of this paper (refer to Section B for more discussions).
Concurrent Work. Building on the impact of OpenAI's ol (OpenAI, 2024), significant efforts have been made within the research community to enhance open-source LLMs with advanced reasoning capabilities. The most com- mon approach relies on distilling knowledge from stronger teacher models (Huang et al., 2024a; Zhao et al., 2024; Min et al., 2024). In contrast, Satori addresses this problem from a reinforcement learning (RL) perspective and requires min- imal supervision (only 10K samples in the format tuning stage). The most related concurrent work is DeepSeek's ss recently released RI (Guo et al., 2025), which adopts similar bigh-level strategy of small- small-scale cold-start SFT fol- lowed by large-scale RL training. Although both works coincide in this high-level idea, our work differs from R1 in key methodologies, including the data synthesis framework and RL algorithmss Additionally, DeepSeck-RI focuses on training large-scale LLMs (671B), whereas our work pro- vides insights into the development of smaller-scale LLMs (7B) for research purpose. Finally, as an industry-developed model, the technical details of DeepSeck-RR -I (Guo et al., 2025) are not fully disclosed, making reproduction difficult, whereas our work is a fully transparent study that aims to open-source training data and training recipes.
3. Preliminaries
We address mathematical problem-solving by training a language model Ta to generate a solution ÿ that matches the ground truth y", given a problem prompt æ. All sequences x. y, and y consist of tokens from a predefined dictionary. Since our approach uses reinforcement learning (RL) to train the model for solving math problems, we outline the key RL concepts below.
Reinforcement Learning (RL). RL (Kaelbling et al., 1996) involves an agent making sequential decisions to max-
2

---

Satori: RL with Chain-of-Action-Thone Enhances LLM Reasoning via Autoregressive Search
Stage 1: Format Tuning
Stage 2: Selfimmrovvement
Quastice: 2-3-0
Demonetraioon Trajectory
Finet
213-5 5
Generator
Thus, the stóvar is 14.
Nest, 5+8 -14
Trajeturkos Gosenticn
Walt, I incorredly compueed 5+8 as 14. Instead, should re-compune
Satori-FT
You eecrnesily componed 5-3 141 you reocompane
Satori
Critie
Let correct my compusation: 5-88 13. Thus, the meswer should be 13,
13.
Thai, the stáver stavershhuld be
Gemerator
Figure 1: A High-level Overview of Satori Training Framework: Format Tuning (FT) + Self-improvement. First, Satori learns COAT reasoning format through imitation learning on ssall-scale demonstration trajectories. Next, Satori further leverages COAT reasoning format to self-improve via large-scale reinforcement learning.
imize the expected cumulative rewards through interactions with an environment. Here, the language model &Te acts as the agent's policy. Starting from an initial state 2p. at each step (, the agent observes the current state 2t, receives a reward r,, selects an action based on Ty. transitions to the next state 1, and continues until reaching terminal state. A trajectory is the sequence of states and actions during this interaction. RL optimizes the policy to maximize the expected rewards Σi= Ft. where L is the trajectory length.
Explore Alternative Solution (< explorel)>> Sig- nals the model to identify critical flaws in its reasoning and explore a new solution.
Each reasoning step yr is sequence of tokens, with the starting token potentially being one of the designated meta- action tokens. We refer this formulation as Chain-of- Action-Thought reasoning (COAT). In particular, typical Chain-of-Thoughn Thought reasoning (CoT) (Wei et al., 2022) can be viewed as a special case of COAT, where each reasoning step in CoT is restricted to continuation, without explicitly incorporating other types of meta-actions.
4. Method
We start this section by introducing the formulation of rea- soning and how reasoning can be formulated as sequen- tial decision-making problem. Goal: We want to train LLMs to solve problems by reasoning through multiple steps rather than directly predicting the final answer. Given a problem statement æ, the model generates a sequence of reasoning steps {y), ... )r). .r.).
Learning to Reason via RL. We formulate reasoning as sequential decision-making problem, where reasoning is a process of constructing and refining an answer step by step. Specifically, the model To starts with an input context æ (initial state z0), generates a reasoning step yr (action), updates the context by appending yı (next state R1+1 = 21 O yı. where @ denotes string concatenation), and repeats this process until it produces final answer yz. The reasoning terminates when the model signals completion (e.g., omitting EOS token). The simplest reward function can be 1(yr, = y" }. evaluates whether the final answer yt matches the ground truth y". With this formulation, we could train the model to reason using RI., aiming to generate reasoning steps that maximize the expected reward. How- ever, applying RL to reasoning presents two key challenges:
where yr. provides the final answer. However, not all intermediate steps are help- ful ful-repeating errors does not improve accuracy. Effective reasoning requires verifying correctness, identifying mis- takes, and considering alternative solutions. For instance, given x = "1+1 =?", the model might initially output y1 = 3, then recognize the mistake with 3/2 (c.g.. "Wait, let me verify...") "), before correcting it to y/3 = 2.
Chain-of-A -aaii-off-Actoo----e Aatt-A-toooooTughh reasoning (COAT). The key challenge is enabling the model to determine when to re- flect, continue, or explore alternatives without external in- tervention. To enable this, we introduce special meta-action tokens that guide the model's reasoning process beyond standard text generation. These tokens serve as hint for the model to determine when to reassess its reasoning before proceeding.
The model doesn't understand the purpose of special tokens and fails to recognize that encountering special meta-action tokens may require reflection or proposing alternatives.
Unawareness of meta-action tokens:
Long horizon and sparse rewards: Reasoning re- quires long-term decision-making with rewards only the end, which hinders learning effectiveness (Belle- mare et al., 2016). The model must take many correct reasoning steps before receiving rewards, and failures force it to restart from the initial state (i.e., the problem statement). This makes learning difficult because train- ing data associated with rewards scarce, yet rewards are essential for driving RL progress.
Continue Reasoning (< cont i nue >>; Encourages the model to build upon its current reasoning trajectory by generating the next intermediate step.
Prompts the model to pause and verify the correctness of prior reasoning steps.
Reflect (< | reflect |>):
3

---

Satori: RL with Mhain-of-Action-Though Enhances LLM Reasoning via Autoregressive Search
Overview of Proposed Method. To address the model's initial unawareness of meta-action tokens, we introduce warm-up "format-tuning" stage: we fine-tune pre-trained LLM on a small dataset featuring a few demonstrated rea- soning trajectories (Section 4. 4.... This step familiarizes the model with using and reacting to meta-action tokens. Sec- ond, to tackle the challenges of long horizons and sparse rewards, we propose a "restart and explore" (RAE) strat- egy, inspired by Go-explore (Ecoffet et al., 2019). Here, the model restarts from intermediate steps, including those points where previous reasoning attempts failed, allowing it to focus on correcting errors rather than starting from scratch. We also add exploration bonuses to encourage deeper reflection, further increasing opportunities for the model arrive at correct answers (Section 4.2).
4.2. Seff-improvemmnt via Reinforcement Learning
After format tuning, the LLM policy अy adopts the COAT reasoning style but struggles to generalize, particularly in using meta-actions for self-reflection. This limitation arises from the scarcity of demonstrations during format tuning. While collecting more demonstrations could help, it is costly and tame-consuming Instead, we explore whether the model can self-improve its reasoning via RL.
We start with the format-tuned LLM and train it using PPO (Schulman et al., 2017b) algorithm, widely used RL method. In addition to training problems x from the dataset D, we also train the model To to begin reasoning from partial trajectories generated by the format-tuned L..M. Since reasoning errors typically arise from minor mistakes rather than fundamental flaws, re-exploring from the start inefficient. Instead, we allow the model to restart from in- termediate steps correct errors and finally achieve correct answers. Inspired by Go-Explore (Ecoffet et al., 2019), we introduce the Restart and Explore (RAE) strategy.
4.1. Format Tuning Through Imitation Learning
Training a base LLM Th to perform COAT reasoning presents a significant challenge: LLMs are typically not pre-trained on COAT reasoning data that incorporates trials and errors, necessitating post-training stage to inject this capability. To address this, we introduce format tuning (FT), a method designed to train LLMs to emulate expert COAT trajectories through imitation learning. Imitation learning techniques (Hussein et al., 2017) are widely used in the robotics domain, where agents are trained using demon- stration trajectories provided by human experts (Ross and Bagnell, 2010; Ross et al., 2011; Ho and Ermon, 2016). However, generating high-quality demonstration trajecto- ries for LLMS is prohibitively expensive for complex tasks. To efficiently construct a demonstration trajectory dataset Dayn = ((x()) (9) ) we propose a multi-agent data synthesis framework that leverages three LLMs:
Algorithm 1 Restart and Explore (RAE)
input Dataset 2> = ((x1), y (8) M LLM policy we after format tuning: maximum back-track steps T
D Initialize D Dissum + 0; Initialize Dieston +
for i = 1,2...., do Given input problem x00), sample my and collect multiple initial trajectories.
Randomly select one correct trajectory g+ and one incorrect trajectory h"
> Randomly backtrack last S actions from ğ and ÿ b Obtain intermediate states at time-step T. - t,, =7 [ar(), -i ........ ... =ī_ [zo(1), UJ ..... Vz-1- Add "reflect" special token trigger self-reflection action.
Generator: Given an input problem, a generator my gen- erates multiple reasoning paths for a given input problem using classical CoT techniques.
= = [m((), (,()), ØT U .......
... Di <I reflect |>]; =i-
(a 91 H2 .......... Me-, <rrf |>).
& Update Disuunt + Diestn U 2 zi D Prestan + D Drestan U -
end
Critic: A critic ne evaluates the correctness of the reason- ing paths generated by the generator, providing feedhack to refine the reasoning and address suboptimal steps.
D Presant = (0 U Disasaan U U.D Deestan
output Augmented initial states dataset Drestatt
Additionally, a reward model म, assigns scores to the refined reasoning paths and selects the most effective path as the final demonstration trajectory.
Reward Model:
Initial States. RAE trains the model to reason not only from the problem statement but also from intermediate steps sampled from past trajectories, both correct and incorrect. This enables deeper exploration without redundant recompu- tation. As detailed in Algorithm 1, given an input problem x E D, the format-tuned LLM first generates multiple rea- soning trajectories. We then randomly backtrack T 2 0 steps and append a reflect token < rerflett |> to prompt the model to refine its reasoning. To encourage diverse exploration, correct and incorrect trajectories are stored sep- arately in restart buffers (Ditsar and Drestan). RL training then optimizes reasoning across these buffers along with the
These three models collaborate to construct high-quality demonstration trajectories (details on the trajectory synthe- sis are provided in Appendix C). For this work, we adopt the simplest imitation learning approach, behavior cloning, which utilizes supervised fine-tuning to train the LLM policy on the expert COAT demonstration trajectories Deyn. No- tably. we observe that even a small number (10K) of COAT demonstration trajectories is sufficient for To to effectively follow the COAT reasoning format.
4

---

Satori: RL with Chain-of-Actinn-hogge Enhances LLM Reasoning via Autoregressive Search
original problem dataset, sampling initial states from the merged dataset resstnn-
Sator--wan-78 (PT Stage)
Intuitively, RAE modifies the initial state distribution by starting new rollouts not only from dataset-samplee prompts but also from random partial trajectories, which encourages a more diverse initial state distribution. This technique is aligned with the insight offered in (Kakade and Langford, 2002), which argues that more uniform state coverage leads to tighter bounds of learned policy. We leave more rigor- ous theoretical justification for future work.
(RL Stage)
BO0K
Owen-2:5 M-ttthhhhhhhha Mat--att-----eeee 78-Instruc
2.5M
Qwen-2:5 Matth Moatth----nntrucc
Stage)
10r
(log-scale)
Reward Design. RAE gives the model multiple oppor- tunities to refine its reasoning, but effective reflection is key to making use of these chances. In addition to using correctness as rewards, we introduce the following bonuses rewards hints guide the model reach correct answers:
Figure 2: Number of Training Samples of Satori-Qwen- and Qwen-2.5-Mathh-B--nnttt nificantly less supervision (small-scale FT) and relies more on self-improvement (large-scale RL).
Satori-Qwen-77 requires sig-
For an initial state E Drestart and a sampled trajectory ÿ. the overall reward function r(2, y is defined as:
Rule-based Reward: Rule-based reward simply evalu- ates the correctness of the final answer.
r(2, 0) rrule (OL. a (rw(2,0)) + Tbonus (2,
Trule (uL, )) = -g==v- 1 {-1.0}
Iterative Self-improvement. RL enables a policy to self- improve from self-generated trajectories, but can also lead to a vicious cycle, where the policy converges local sub-optimum and cannot further improve. Inspired by (Agarwal et al., 2022; Schmitt et al., 2018), propose an iterative self-im self-improvement strategy to mitigate this issue. Specifically, after each round of RL training, we distill the knowledge of the current well-optimized policy into the base model through supervised fine- mee-tuning (SFT). Starting from the newly fine-tuned model, we then perform another round of RL training. Intuitively, from an optimization perspective, each round of distillation can be viewed as parameter reset mechanism that helps the policy escape local optima in the loss landscape, allowing it to continue self- liff-iipoviing (more details are included in Section D.3). In the next section, we provide empirical evidence to validate this approach.
To reinforce self-reflection, we in- troduce a reflection bonus Tbomus If the model starts from an incorrect reasoning trajectory stored in the negative restart buffer (Prestant and successfully solves the prob- lem, it obtains reward bonus, encouraging it to correct past mistakes. Conversely, if it starts from correct tra- jectory in the positive restart buffer (Ditsun) but fails to solve the problem, it incurs penalty, discouraging un- necessary revisions when it was already on the right track. Formally, the reflection bonus is defined as:
Reflection Bonuses:
if z E D Drestan and UL y*,
Tbonus (2,
-8
if E D restum and ÿr. # y*, otherwise,
0
where 3 is a bonus scale hyperparameter.
Since correct answers are rare at initial training stage, reward signals are often too sparse for effective RL training. Even with reflection, the model may fail to generate any correct reasoning trajectories, resulting in sparse reward problem. To mitigate this, we train an Outcome Reward Model (ORM) using a Bradley- Terry (BT) preference framework. The ORM rates rea- soning trajectories, assigning higher values correct (preferred) ones. For each problem æ E D, we generate multiple trajectories using Te and construct a preference dataset by pairing correct and incorrect outputs. A BT model is trained to maximize the score gap between these pairs. The ORM's 's output, a (2, 7)) E [0, 1). serves as a fine- ane-grained reward signal, helping the model further refine its reasoning. See Appendix D.3 for details.
Preference Bonuses:
5. Experiment
Implementation Details. We employ 2.ven2255-Math-7B as the base model due to its strong mathematical capabilities. Our training data is sourced from the publicly available math instruction datasets, Open and NuminaMath- COT. For the multi-agent data synthesis framework, the generator is required generate high-quality, step-by-step reasoning trajectories. Therefore, we use Qwen-2.55 2.5-Mahhh-- Instruct the generator. Meanwhile, the critic must have ro- bust instruction -olluciinn-oooowinng capabilities, so we choose Llama- 3. 3.1-70B-Innse Instruct as the critic. To ensure data quality, we filter out problems with invalid questions or incorrect la- bels, resulting in approximately 550k samples. Additional implementation details can be found in Appendix D.
5

---

Satori: RL with Chain-of-Action-TThnn Enhances LLM Reasoning via Autoregressive Search
Table Results on Mathematic Benchmarks. Satori- Qwen-7B achieves SOTA performance across five benchmarks, outperforms Owen-2.5-Matte Ma-M--aa-a---tttiee Instrccttt which uses the same base model Qwen-2.5 2owen-2.5-Math775 After round-2 training, Satori Satori-Qwen- Qwen--w 7B (Round 2) demonstrates even stronger performance on hard tasks.
Scale
Model
GSM8K
MATHS00
OlympiadBench
AMC2023
AIME2024
Avg.
GPT-4o
60.3
43.3
/
9.3
ol-preview
85.5
82.5
44.6
Large
1amm-3.1-70B-Insttee
68.0
29.4
42.5
13.3
49.5
OpenMath2-LLamma3I
94.1
71.8
30.1
45.0
13.3
50.9
QwQ-32B-Preview
95.5
90.6
61.2
77.5
50.0
75.0
Llama-3, (lama-3.1-bb-nnsi 1-8b-Instttuct
84.4
51.9
15.1
22.5
3.3
35.4
OpenMath2-LLamma3. 1-8B
90.5
67.8
28.9
37.5
6.7
46.3
MantnnaMtth-BB-CCC
78.9
54.6
15.9
20.0
10.0
35.9
Small
Qwen-2.5-7B. 2.-srr--sttttt
91.6
75.5
35.5
52.5
6.7
$2.4
Qwen-2.5-Man 1att-225--Math-7- 7B-Instuuct
95.2
83.6
41.6
62.5
16.7
59.9
Satori-Qwen-7BB
93.2
85.6
46.6
67.5
20.0
62.6
Satori-Qwen-7B (Round 2)
93.9
83.6
48.5
72.5
23.3
64.4
Table 2: Results on Out-of-domain Benchmarks. Trained only on math datasets, Satori-Qwen-7B exhibits strong transferability across diverse out-of-domain benchmarks and outperforms 2.55-Mthh-7B-nnnn by large margin. Moreover, despite not being trained other domains, Satori-Qwen-7B achieves performance comparable exceeding other small-scale general instruct models.
Scale
Model
FOLIO
BGQA
CRUXEval
Strategy
TableBench
STEM
Avg.
Llama-3, 3.1.-.0B-I-Isteeeee
65.0
58.3
59.6
88.8
34.2
61.7
61.3
Large
OpenMath2-LLama3 1-0BB
68.5
68.7
35.1
95.6
46.8
15.1
55.0
QwQ-32B-Prevvie
84.2
71.1
65.2
88.2
51.5
71.3
71.9
Llama-3.1 1amma-3.1-bb-mma -nstrruttttt
63.5
50.3
38.5
92.2
32.4
43.4
53.4
OpenMath2-Llamabe 1-8B
57.1
49.0
11.1
84.4
34.2
10.9
41.1
NuminaMath-7B-Con
53.2
44.6
28.0
77.8
29.1
11.3
40.7
Small
Owen-2.5-7B-Innsree
72.4
53.0
58.1
91.3
43.2
57.1
62.5
Qwen-2.5-Mathh 7B-Instuct
68.9
51.3
28.0
85.3
36.2
45.2
52.5
Satori-Qwen-7B
71.4
61.8
42.5
86.3
43.4
56.7
60.4
Natori-Qwen-77 (Round 2)
72.9
58.5
41.1
90.4
44.6
57.4
60.8
Benchmark and Evaluation. We conduct the main eval- uation of the models using math benchmarks to assess their problem-solving solving abilities, including GSMSK, MATH500 (a subset of the MATH test set (Lightman et al., 2023)), AMC2023, AIME2024, and OlympiadBench. Except for GSM8K, all other datasets feature competition-level prob- lems. The evaluation is performed using greedy decoding without tool integration. The main metric reported is the zero-shot @ass@1 accuracy, which measures the percentage of problems correctly solved on the first attempt. We also conduct additional evaluations on wide range of bench- marks beyond the math domain to evaluate general reason- ing capabilities. This includes logical reasoning (FOLIO (Han et al., 2024), BoardgameQA (BGQA) (Kazemi al., 2024)), code reasoning (vauXEval (Gu al., 2024)), com- monsense reasoning (StrategyQA (STGQA) (Geva et al., 2021)), tabular reasoning (TableBench (Wu et al., 2024a)), and domain-specific reasoning (STEM subsets of MMLU- Pro (Wang et al., 2024b)), including physics, chemistry, computer science, engincering, biology, and economics. For more evaluation details, please refer to Appendix D.4.
Baseline Models. We compare developed model, Satori-Qwen-77h with several industry-developed LLMs. The main comparison is between our model and Qwen-2.5- Math-7B, IB-Inutt (Yang et al., 2024a), a math-speciialized model built on the same base model (Qwen-2.5-M) 2.5een225-Math-177 as ours. Additionally, we report the performance larger mod- els, including ol-preview and QwQ-32B-Prewee -reerreve which exhibit strong reasoning capabilities and serve perfor- mance upper bounds.
5.1. Main Results on Math Domain
We present math benchmark results in Table 1, where Satori-Qwen-7BB outperforms all small-scale baseline mod- els. Notably, using Qwen-2.5-Math-77 as the base model, Aatori-Qwen-78 achieves superior performance compared to Qwen-2.5- Matt--B--Insttte cantly less supervision (i.e., less SFT data) and relying more on self-improvement (i.e., more RL data) (see Figure 2).
despite requiring signifi-
6

---

Satori: RL with Chain-of-Action-Tn Thon--of-Action-Thouhht Enhances LLM Reasoning via Autoregressive Search
5.2. Out-of-Domain Transferability
actions. Next, we maintain all other training settings, in- cluding the same amount of SFT and RI. data and consis- tent hyper-parameterss This results in typical CoT LLM (Qwen-7B-CoT) without self-reflection or self-exploration capabilities. As shown in Table 3, the performance of Qwen- 7B-CoT is suboptimal compared to Satori-Qwen- and fails to surpass 2.5e-2.5- Ma-athh---nntructt suggesting the advantages of COAT reasoning over CoT reasoning.
Although Satori-Qwen-7B is trained only on math domain datasets, we observe that it can extrapolate its reasoning capabilities to other domains. In Table 2, we evaluate Satori- Qwen-7B on a diverse set of out-of-domain benchmarks that require reasoning capabilities but are not directly related to math. Similar to the observation on the math domain, Satori demonstrates superior performance on several benchmarks, outperforming Qwen-2.5-Me M-th-Ma--ttttrreeee Instttrut.. In particular, on the challenging reasoning benchmark BoardgameQA, Satori-Qwen-77 surpasses all baseline models of the same scale. These results and demo examples in Appendix A sug- gest that Satori has acquired general reasoning capabilities rather than simply math problem solving skills. In Section 6, we present further analysis to show that this transferability emerges as a result of large-scale reinforcement learning.
Table Satori's Self-correction Capability. T-F: negative self-correction; F positive self-correction.
In-Domain
Out-of-Domain
Model
MATH500
OlympiadBench T-F
MMLUPPoSTEM T-F
T-F
F-T
F-+
F-T
Satori-Qwen-78-F7
79.4%
20.6%
65.6% 42.1%
34.4% 57.9%
59.2% 46.5%
40.8%
Satori-Qwen-7B
39.0%
61.0%
53.5%
5.3. Results on Iterative Seff-improvement
Satori Exhibits Self-correction Capability. We observe that Satori frequently engages in self-reflection during the reasoning process (see demos in Section A), which occurs in two scenarios: (1) it triggers self-reflection at intermediate reasoning steps, and (2) after completing problem, it initi- ates second attempt through self-reflection. We focus on quantitatively evaluating Satori's self-correction capability in the second scenario. Specifically, we extract responses where the final answer before self-reflection differs from the answer after self-reflection. We then quantify the percent- age of responses in which Satori's self-correction is positive (i.e., the solution is corrected from incorrect to correct) or negative (i.e., the solution changes from correct to incorrect). The evaluation results on in-domain datasets (MATH500 and Olympiad) and out-of-domain datasets (MMLUPro) are presented in Table 4. First, compared to Satori- Satori-Qwen-FT Qwen-FT which lacks the RL training stage. Satori-Qwen demon- strates significantly stronger self-correction capability. Second, we observe that this self-correction capability ex- tends to out-of-domain tasks (MMLUProSTEMM) These results suggest that RI. plays a crucial role in enhancing the model's true reasoning capabilities.
Finally, we present the results of the second-round training of Satori. As shown in Table and Table 2. compared to Satori-Qwen-77B. Satori-Qwen-7B (Round 2) demonstrates continuous performance gains across most in-domain and out-of-domain benchmarks. This suggests the significant potential of iterative self-imppovement to push the limit of LLM's reasoning performance.
6. Analysis
In this section, we provide comprehensive analysis of Satori. First, we demonstrate that Satori effectively lever- ages self-reflection to seek better solutions and enhance its overall reasoning performance. Next, we observe that Satori exhibits test-scaling behavior through RL training, where it progressively acquires more tokens to improve its reasoning capabilities. Finally, we conduct ablation studies on vari- ous components of Satori's training framework. Additional results are provided in Appendix E.
Table 3: COAT Training v.s. CoT Training. Qwen-2.5-Mathh trained with COAT reasoning format "Satori-Qwen-77 outper- forms the same base model but trained with classical CoT reason- ing format (Qwen-7B-CoT)
RL Enables Satori with Test-time Scaling Behavior. Next, we aim to explain how reinforcement learning (RL) incentivizes Satori's 's autoregressive search capability. First, as shown in Figure 3, we observe that Satori consistently im- proves policy accuracy and increases the average length of generated tokens with more RL training-time tiim compute. This suggests that Satori learns to allocate more time to reasoning, thereby solving problems more accurately. One interesting observation is that the response length first decreases from O to 200 steps and then increases. Upon a closer investigation of the model response, we observe that in the early stage, our model has not yet learned self-reflection capubilities. During this stage, RL optimization may prioritize the model
Model
GSMSK MATH500 Olym. AMC2023 AIME2024
Qwen A.thMahh77 BB Instruct
95.2
83.6
41.6
62.5
16.7
Qwen-7B-Coo (SFT+RT)
93.1
84.4
42.7
60.0
10.0
Satori-Qwen-7B
85.6
46.6
67.5
20.0
COAT Reasoning v.s. CoT Reasoning. We begin by conducting an ablation study to demonstrate the benefits of COAT reasoning compared to the classical CoT reasoning. Specifically, starting from the synthesis of demonstration trajectories in the format tuning stage, we ablate the "re- flect" and "explore" actions, retaining only the "continue"
7

---

Satori: RL with Chain-of-Action-Thon Enhances LLM Reasoning via Autoregressive Search
Table 5: Large-scale FT V.S. Large-scale Satori- twen (10K FT data + 300K RL data) outperforms same base model Qwen-2.5- Math-7B trained with 300K FT data (w/ RL) across all math and out-of-domain benchmarks.
1300
Response Length tokens)
Pollcy Acc. (96)
75
1200
70
1100
(In-domain)
GSMBK MATH500
Olym.
AMC2023
AIME2024
1ronn255-Math-7B--iae
95.2
83.6
41.6
62.5
16.7
65
Salori-Qwen-7B--
40.9
1000
Satori-Qwen-7B
93.2
85.6
46.6
67.5
20.0
60
(Out-of-domain)
BGQA
CRUX
STGQA TableBench
STEM
900
0wen-2.5-Mathh-27 1.---thh-B---nsttuee
51.3
28.0
85.3
36.3
45.2
200
400
600
800
Salori-Qwen-7B-1 (300K)
50.5
29.5
74.0
35.0
47.8
RL Training Step
Satori-Qwen-7B
56.7
Figure Policy Training Acc. & Response length RL Train- time Compute. Through RL training. Satori learns to improve reasoning performance through longer thinking.
Average Acc Across Benchmard
50.0
Liama-85 B8--88-FF
47.5
Liama-Bn-Qeejj
Resp. Len. (# tokens)
Granito
+394
4:362
42.5
1000
+330
40.0
+305
+269
37.5
500
35.0
32.5
Accuracy (%)
100
107
+2.9
30.0
Liame-8B-FT Liema-8B-Disell Granite-a-FT Gravit--B-oivell
Models
+4,2
Format Tuning v.s. Distillation. Distilling from a Stronger model "Satori-Qwen-7BB 7B) to weaker base models (Llama- 8B and Granite-8B) are more effective than directly applying for- mat tuning weaker base models.
Figure
80
+5.8
Satori-Qwen Satori-Qwen-FT
+688
60
Levell
Level2
Level3
Level4
Level5
Figure Above: Test-time Response Length v.s. Problem Diffi- culty Level; Below: Test-time Accuracy v.s. Problem Difficulty Level. Compared to FT model (Satori-Qwens Qwen-FT), Satori- Qwen uses more test-time compute to tackle more challenging problems.
(300K). Additionally, the large-scale FT model performs significantly worse on out-of-domain tasks, demonstrates RL's advantage in generalization.
to find a shot-cut solution without redundant reflection, lead- ing to temporary reduction in response length. However, in later stage, the model becomes increasingly good at using reflection to self-correct and find a better solution, leading to longer response length.
Distillation Enables Weak-to-Strong Generalization. Finally, we investigate whether distilling a stronger rea- soning model can enhance the reasoning performance of weaker base models. Specifically, we use Natori-Qwen-Ti to generate 240K synthetic data train weaker base models, Llama-3.1-8 and Granite-3.1-88 For comparison, we also synthesize 240K FT data (following Section 4.1) to train the same models. We evaluate the average test accuracy of these models across all math benchmark datasets, with the results presented in Figure 5. The results show that the distilled models outperform the format-tuned models.
Additionally, in Figure we evaluate Satori's test accuracy and response length MATH datasets across different dif- ficulty levels. Interestingly, through RL training, Satori nat- urally allocates more test-time compute to tackle more chal- lenging problems, which leads to consistent performance improvements compared to the format-tuned (FT) model.
This suggests a new, efficient approach improve the rea- soning capabilities weaker base models: (1) train a strong reasoning model through small-scale FT and large-scale RL (our Ratori-Qwen. and (2) distill the strong reasoning capabilities of the model into weaker base models. Since RI. only requires answer labels as supervision, this approach introduces minimal costs for data synthesis, i.e., the costs induced by multi-agent data synthesis framework or even more expensive human annotation.
Large-scale FT v.s. Large-scale RL. We investigate whether scaling up format tuning (FT) can achieve perfor- mance gains comparable to RL training. We conduct ablation study using Qwen-2.5-Math-75 trained with equivalent amount of FT data (300K). As shown in Table 5, on the math domain benchmarks, the model trained with lcrge-scale FT (300K) fails to match the performance of the model trained with mmall-scale FT (10K) and large-scale RI.
8

---

Satori: RL with Chain-of-Action-Thouge Enhances LLM Reasoning via Autoregressive Search
7. Concluding Remarks
S. Han, H. Schoelkopf, Y. Zhao, Z. Qi, M. Riddell al., "FOLIO: Natural language reasoning with first-order logic," in Proceedings of the 2024 Conference Em- pirical Methods in Natural Language Processing, Y. Al- Onaizan, M. Bansal, and Y.-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistics, November 2024, pp. 22017-22 031. 1, 6, 24, 36
The training framework of Satori exhibits significant poten- tial for enhancing LLM reasoning capabilities. The small- scale format tuning stage serves as a warm-up phase, al- lowing the LLM policy to internalize a specific reasoning format, while large-scale reinforcement learning (RL) plays a crucial role in incentivizing intrinsic reasoning abilities. We believe that this framework can inspire the research community to explore more methods for achieving autore- gressive search, such as developing reasoning formats with broader range of meta-actions, designing more advanced RL algorithms, and extending this approach to general domain.
J. Liu, Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang, "LogiQA: A challenge dataset for machine reading com- prehension with logical reasoning," in Proceedings of the Twenty-Nimth International Joinl Conference on Artificial Intelligence, IJCAI 2020, C. Bessiere, Ed. ijcai.org, 2020, pp. 3622-3628. 1, 1,24
Acknowledgment
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou el al., "Chain-of-thouuhh prompting elicits reasoning in large language models," Advances in neural information processing systems, vol. 35, pp. 824 24 837, 2022. 3,
This research/project was supported, in part, by the follow- ing funding resources: MIT-IBM Watson AI Lab under Agreement No. W1771646, the National Research Founda- tion, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-200 000--2000---0077) and Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier 2 Programme (MOE AcRF Tier 2 Award No.
X. Yue, X. Qu, G. Zhang. Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen, "MAmmoTH: Building math generalist models through hybrid instruction tuning," in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. 2,2,24
MOE-12EP201 22-0011).
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Zhang. J. T. Kwok, Z. Li, A. Weller, and W. Liu, "MetaMath: Bootstrap your own mathematical questions for large language models," in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. 1,
References
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano el al., "Training verifiers to solve math word problems," arXiv preprint arXiv:2 10. 14168, 2021. 1,24,35 36
S. Toshniwal, W. Du, I. Moshkov, B. Kisacanin, A. Ayrapetyan, and Gitman, "Opennathinstruct22 Ac- celerating AI for math with massive open-source instruc- tion data," arXiv preprint arXiv:2410.01566) 2024. 1, 2, 24, 27
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang. D. Song, and J. Steinhardt, "Measuring math- ematical problem solving with the MATH dataset," in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 2, 24, 36
Y. Ding, X. Shi, X. Liang, J. Li, Q. Zhu, and M. Zhang, "Un- lcashing reasoning capability of Ilms via scalable question synthesis from scratch," arXiv preprint arXiv:2410. 18693, 2024. 1, 24
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03377. 2021. 1,24
X. Wang, Wei, D. Schuurmans, Q. V. Le, H. Chi, S. Narang, A. Chowdhery, and D. Zhou, "Self- consistency improves chain of thought reasoning in lan- guage models," in The Eleventh International Conference on Learning Representations, 2023. 1
T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, N. B. Yusuf, H. Zhan, J. He, I. Paul et al., "Big- codebench: Benchmarking code generation with diverse function calls and complex instructions," arXiv preprint arXiv:2406. 15877, 2024. 1, 24
S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, ''Tree of thoughts: Deliberate problem solving with large language models," Advances in Neural Information Processing Systems, vol. 36, 2024. 1, 2, 24

---

Satori: RL with Chain-of-Action-Thouee Enhances LLM Reasoning via Autoregressive Search
Z. Wan, X. Feng, M. Wen, S. M. McAleer, Y. Wen, W. Zhang, and J. Wang, "Alphazero-like tree-search can guide large language model decoding and training." in Forty-firs International Conference on Machine Learn- ing, 2024.
themselves to think before speaking," " in First Conference on Language Modeling, 2024. 2, 24
A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. Liu, J. Harrison et al., "Beyond human data: Scaling self-training for problem-solving with lan- guage models," Trans. Mach. Learn. Res., vol. 2024, 2024. 2, 24
Z. Sun, L. Yu. Y. Shen, W. Liu, Y. Yang, S. Welleck, and C. Gan, "Easy-to-hare hord generalization: Scalable alignment beyond human supervision." in The eight Annual Conference on Neural Information Processing Systems, 2024. 1, 24
X. Zhang, C. Du, T. Pang, Q. Liu, W. Gao, and M. Lin, "Chain of preference optimization: Improving chain-of- thought reasoning in LLMs," " in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 2,
Wang, I.. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui, :hath-Shepherd: Verify and reinforce LLMs sep-by-step without human annotations," in Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Au- gust 2024, pp. 9426 9439. 1, 2, 24
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.06347 2017. 2, 24,
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L.. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, "Training language models to follow instruc- tions with human feedback," " in Advances in Neural In- formation Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., 2022. 2,24
OPenAI, "Learning to reason with IIms," 2024, accessed:
12-18. [Online]. Available: https://openai.com index/learning-toreee wo--reesoo-wtth-llee
2
Z. Huang, H. Zou, X. Li, Y. Liu, Y. Zheng. E. Chem, S. Xia, Y. Qin, W. Yuan, and P. Liu, "O1 replication journey-part 2: Surpassing ol oprevvew through simple distillation, big progress or bitter lesson?" arXiv preprint arXiv:2411.16448 2024. 2, 24
Y. Zhao, H. Yin, B. Zeng, H. Wang. T. Shi, C. Lyu, L. Wang, W. Luo, and K. Zhang, "Marco-ol: Towards open rea- soning models for open-ended solutions," arXiv preprint arXiv:2411. 14405, 2024. 2,24
L. Yuan, W. Li, H. Chen, G. Cui, N. Ding, K. Zhang, B. Zhou, Z. Liu, and H. Peng, "Free process rewards without process labels," arXiv preprint arXiv:2412.01991, 2024. 24
Y. Min, Z. Chen, J. Jiang, J. Chen, J. Deng, Y. Hu, Y. Tang, Wang, X. Cheng, H. Song et al., "Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems," arXiv preprint arXiv:2412.09411 2024. 2, 24
N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, "Reflexion: Language agents with verbal rein- forcement learning," Advances in Neural Information Processing Systems, vol. 36, 2024. 2.24
S. Hao, Y. Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu, "Reasoning with language model is planning with world model," in Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2023, Singapore, December 6- 10, 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Association for Computational Linguistics, 2023, pp. 8154-8173 2, 24
D. Guo. D. Yang. H. Zhang, J. Song, R. Zhang, R. Xu, Q Zhu, S. Ma, P. Wang. X. Bi et al., "DeepSeek-R1: In- centivizing reasoning capability in Ilms via reinforcement learning," arXiv preprint arXiv:2501. 12948, 2025. 2
E. Zelikman, Y. Wu, J. MU, and N. D. Goodman, "STaR:
Bootstrapping reasoning with reasoning," in Advances in
Neural Information Processing Systems 35: Annual Con- ference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., 2022. 2, 24
Z. Qi, M. Ma, Xu, L. L. Zhang. F. Yang, and M. Yang, "Mutual reasoning makes smaller 1lms stronger problem- solvers," arXiv preprint 2rXiv:2408.06/99 2024. 2, 24, 27
Y. Zhang, M. Khalifa, L. Logeswaran, J. Kim, M. Lee, H. Lee, and L. Wang, "Small language models need strong verifiers to self-correct reasoning," " in Findings of
E. Zelikman, G. R. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. Goodman, "Quiet-STaR: Language models can teach
10

---

Satori: RL with Chain-of-Action-Thuugh Enhances LLM Reasoning via Autoregressive Search
the Association for Computational Linguistics, ACL, Bangkok, Thailand and virtual meeting, August I 11-16, 2024, L. Ku, A. Martins, and V. Srikumar, Eds. Asso- ciation for Computational Linguistics, 2024, pp. 637-
D. Zhang, S. Zhoubian, Z. Hu, Y. Yue, Y. Dong, and J. Tang. "ReST-MCTS* LLM self-training via process reward guided tree search," in The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024. 2, 24
15 653.22 2, 24
R. Kamoi, Y. Zhang, N. Zhang. J. Han, and R. Zhang, "When can Iims actually correct their own mistakes? critical survey of self-correction of Ilms," Transactions of the Association for Compulational Linguistics, vol. 12, pp. 1417-1440, 2024. 2, 24
K. Gandhi, D. H. J. Lee, G. Grand, M. Liu, W. Cheng, A. Sharma, and N. Goodman, "Stream of search (SoS); Learning to search in language," in First Conference on Language Modeling, 2024. 2, 24
L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Re- inforcement learning: A survey," Journal of artificial intelligence research, vol. 4, pp. 237-285, 1996. 2
W. Saunders, C. Yeh, J. Wu, S. Bills, L. Ouyang, Ward, and J. Leike, "Self-critiquing models for assisting human evaluators," arXiv preprint wtXiv:2206.05802 2022. 2, 24
M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, "Unifying count-based ex- ploration and intrinsic motivation," Advances in neural information processing systems, vol. 29, 20,661.. 3
A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. lqbal, C. Bishop, R. Roelofs el al., "Training language models to self-correct via re- inforcement learning," arXiv preprint arXiv:2409.12917 2024. 2, 24
Rcoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, "Go-explore: new approach for hard- exploration problems," arXiv preprint arXiv:1901.10995, 2019. 4
Y. Qu. T. Zhang, N. Garg, and A. Kumar, "Recursive in- trospection: Teaching language model agents how to self-improve," in The eighth Annual Conference Neural Information Processing Systems, 2024. 2, 24
Hussein, M. M. Gaber, E. Elyan, and C. Jayne, "Im- itation learning: A survey of learning methods." ACM Computing Surveys (CSUR), vol. 50, no. 2, pp. 1-35, 2017. 4
A. Havrilla, S. C. Raparthy, C. Nalmpantis, J. Dwivedi- Yu, M. Zhuravinskyi, E. Hambro, and R. Raileanu, "GLoRe: When, where, and how to improve LLM reasoning via global and local refinements," " in Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. 2, 24
S. Ross and D. Bagnell, "Efficient reductions for imitation learning." in Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010, pp. 661- 668. 4
Z. Xi, D. Yang. J. Huang, J. Tang, G. Li, Y. Ding, W. He, B. Hong, S. Do, W. Zhan et al., "Enhancing I1m reason- ing via critique models with test-time and training-time supervision," arXiv preprint arXiv:2411.16577 2024. 2, 24
S. Ross, G. Gordon, and D. Bagnell, "A reduction of im- itation learning and structured prediction to no-regret online learning," in Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011, pp. 627-635. 4
A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar, "Re- warding progress: Scaling automated process verifiers for Ilm reasoning." arXiv preprint axXiv22400.08146. 2024. 2, 24
J. Ho and S. Ermon, "Generative adversarial imitation leam- ing," Advances in neural information processing systems, vol. 29, 2016. 4
J. Schulman, F. Wolski, Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint
L. Zhang, A. Hosseini, H. Bansal, M. Kazemi. A. Kumar, and R. Agarwal, "Generative verifiers: Reward model- ing as next-token prediction," " in The 4th Workshop on Mathematical Reasoning and Al at NeurIPS'24 2024. 2, 24
2017. 4
S. Kakade and J. Langford, "Approximately optimal ap- proximate reinforcement learning." in Proceedings of the nineteenth international conference on machine learning. 2002, pp. 267-274.
X. Guan, L. L. Zhang, Y. Liu, N. Shang, Y. Sun, Y. Zhu, F. Yang, and M. Yang, "rstar-math: Small Ims can master math reasoning with seol-evolved deep thinking." arXiv preprint arXiv:2501.045 04519, 2025. 2, 24
R. Agarwal, M. Schwarzer, P. Castro, A. C. Courville, and M. Bellemare, "Reincarnating reinforcement learning:

---

Satori: RL with Chain-of-Action-Tougge Enhances LLM Reasoning via Autoregressive Search
Reusing prior computation accelerate progress," Ad- vances in neural information processing systems, vol. 35, pp. 28 98955-28 971,, 2022. 5
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., "Program synthesis with large language models," arXiv preprint arXiv:2108.0777
2021. 24
S. Schmitt, Hudson, A. Zidek, S. Osindero, C. Doersch, W. M. Czarnecki, J. Z. Leibo, H. Kuttler, A. Zisserman, K. Simonyan et al., "Kickstarting deep reinforcement learning," " arXiv preprint arXiv:1803.0388 2018.
J. Dai, J. Lu, Y. Feng. D. Huang, G. Zeng, R. Ruan, M. Cheng, H. Tan, and Z. Guo, "Mhpp: Exploring the capabilities and limitations of language models beyond basic code generation," arXiv preprint arXiv:2405.1 14430, 2024. 24
H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, Sutskever, and
K. Cobbe, "Let's verify step by step," arXiv preprint arXiv:2305.20055 2023. 6, 36
C. E. Jimenez, Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, "SWE-bench: Can language mod- els resolve real-world github issues?" in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. 24
M. Kazemi, Q. Yuan, D. Bhatia, N. Kim, X. Xu, V. Imbra- saite, and D. Ramachandran, "Boardgameqa: A dataset for natural language reasoning with contradictory infor- mation," Advances in Neural Information Processing Sys- tems, vol. 36, 2024. 6, 36
C. Liu, J. Shen, H. Xin, Z. Liu, Yuan, H. Wang. W. Ju, C. Zheng, Y. Yin, L. Li et al., "Fimo: A challenge formal dataset for automated theorem proving," arXiv preprint arXiv:2309.04299 2023. 24
A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Syn- naeve, and S. I. Wang, "Cruxeval: A benchmark for code reasoning, understanding and execution," arXiv preprint arXiv:2401.03063 2024. 6, 24,36 36
O. Tafjord, B. Dalvi, and P. Clark, "ProofWriter: Generat- ing implications, proofs, and abductive statements over natural language." " in Findings the Association for Com- putational Linguisties: ACI/JCNNIP 2021, Online Event, August 1-6, 2021, ser. Findings of ACL, C. Zong, F. Xia, W. Li, and R. Navigli, Eds., vol. ACL/UCCNP 2021. As- sociation for Computational Linguistics, 2021, pp. 3621- 3634. 24
M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Be- rant, "Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies." Transac- tions of the Association for Computational Linguistics, vol. 9. pp. 346-361, 2021. 6, 24, 36
X. Wu, J. Yang, L. Chai, G. Zhang. J. Liu, X. Du, D. Liang, D. Shu, X. Cheng, T. Sun al., "Tablebench: A com- prehensive and complex benchmark for table question answering," arXiv preprint 4rXiv:2408.09174, 2024. 6, 24, 36
A. Talmor, J. Herzig, N. Lourie, and J. Berant, "Common- senseqa: A question answering challenge targeting com- monsense knowledge," arXiv preprint arXiv: 18111111.000 1.00937, 2018. 24
Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj. X. Hc, Z. Jiang et al., "Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark," arXiv preprint arXiv:2406.01574
P. Veličković, A. P. Badia, D. Budden, R. Pascanu, A. Ban- ino, M. Dashevskiy, R. Hadsell, and C. Blundell, "The CLRS algorithmic reasoning benchmark," in Interna- tional Conference Machine Learning. PMLR, 2022, pp. 084 22 102. 24
2024. 6, 36
A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin el al., "Qwen2.5-math techni- cal report: Toward mathematical expert model via self- improvement," arXiv preprint arXiv:2409.12122 2024.
L. Markeeva, S. McLeish, B. Ibarz, W. Bounsi, O. Ko- zlova, A. Vitvitskyi, C. Blundell, T. Goldstein, A. Schwarzschild, and P. Veličković, "The CLRS-Text al- gorithmic reasoning language benchmark," arXiv preprint arXiv:2406.04222 2024. 24
E. Glazer, E. Erdil, T. Besiroglu, D. Chicharro, E. Chen, A. Gunning, C. F. Olsson, J.-S. Denain, A. Ho, E. d. O. Santos et al., "Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai," arXiv preprint arXiv:2411.04872 2024. 24
Z. Qi, H. Luo, X. Huang, Z. Zhao, Y. Jiang, X. Fan, H. Lakkaraju. and Glass, "Quantifying generalization complexity for large language models," arXiv preprint arXiv:2410001766 2024. 24
I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar, "Gsm -omm-symboii:: Understanding the limitations of mathematical reasoning in large language models," arXiv preprint arXiv:2410.05229 2024. 24
L. Fan, W. Hua, L. Li, H. Ling, and Y. Zhang, "NPHard- Eval: Dynamic benchmark on reasoning ability of large language models via complexity classes," " in Proceedings
12

---

Satori: RL with seaii-of--Atiio-Thougee Enhances LLM Reasoning via Autoregressive Search
of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, L. Ku, A. Martins, and V. Srikumar, Eds. Association for Com- putational I Linguistics, 2024, pp. 4092-4114. 24
M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gu- nasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann et al., "Phi-4 technical report," arXiv preprint arXiv:2412.08900 2024. 24
Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Paper- not, and R. Anderson, "The curse of recursion: Training on generated data makes models forget," arXiv preprint arXiv: 17493, 2023. 24
Lu, L. Qiu, K.- Chang, N. Wu, S.-C. Zhu, hit, P. Clark, and A. Kalyan, "Dynamic prompt learning via policy gradient for semi-structured mathematical rea- soning," arXiv preprint UrXiv:229.1166 2022. 24
T. Rajpuro-
X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subrama- niam, A. R. Loomba, S. Zhang, Y. Sun, and W. Wang, "SciBench: Evaluating college-leve level scientific problem- solving abilities of large language models," in Forty-first International Conference on Machine Learning, ICML OpenRe-
T. Wu, X. Li, and P. Liu, "Progress regress? self- improvement reversal in post-training." arXiv preprint arXiv:2407.05013, 2024. 24
M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, Gianinazzi, Gajda, T. Lehmann, H. Niewiadomski, P. Nyezyk et al., "Graph of thoughts: Solving elaborate problems with large language models," in Proceedings of the AAAI Conference on Artificial In- telligence, vol. 38, no. 16, 2024, pp. 17 686-2-17 690. 24
2024, Vienna, Austria, July 21-27, 2024. view.net, 2024. 24
D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirami, J. Michael, and S. R. Bowman, "GPQA: A graduate-level google-proof q&a benchmark," in First Conference on Language Modeling, 2024. 24
A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffee U. Alon, N. Dziri, S. Prabhumoye, Y. Yang et al., "Self-refine: Iterative refinement with self- feedback," Advances in Neural Information Processing Systems, vol. 36, 2024. 24
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song. and J. Steinhardt, "Measuring massive multitask language understanding," 9th International Conference on Learning Reprexentations, ICIR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 24, 36 A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown al., "Beyond the imitation game: Quantifying and extrapolating the capabilities of language models," Trans. Mach. Learn. Res., vol. 2023, 2023. 24
L. Yang, Z. Yu, T. Zhang, S. Cao, M. Xu, W. Zhang. J. E. Gonzalez, and B. CUI, "Buffer of Thoughts: Thought- augmented reasoning with large language models," in The Thirty-eightt Annual Conference on Neural Information Prucessing Systems, 2024. 24
P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga et al., "Holistic evaluation of language mod- els," Trans. Mach. Learn. Res., vol. 2023, 2023. 24
J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, "Large language models cannot self-correct reasoning yet," in The Twelfth International Conference on Learning Reprexentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. 2024. 24
L. Phan, A. Gatti, Z. Han, N. Li, Hu, H. Zhang, S. Shi, M. Choi, A. Agrawal, A. Chopra, A. Khoja, R. Kim,
OpenReview.net.
J. Hausenloy al., "Humanity's 's last exam," 2025. [Online]. Available: https://axxiv.or//bb///////
24 W. Saunders, C. Yeh, J. Wu, S. Bills, I. Ouyang, J. Ward, and J. Leike, "Self-critiquing models for assisting human evaluators," arXiv preprint arXiv:2206.05880 2022. 24
S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi, "Generating sequences by learning to self-correct," in The Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. 24
S. Toshniwal, I. Moshkov, S. Narenthiran, D. Gitman, F. Jia, and I. Gitman, "Open A 1.8 million math instruction tuning dataset," in The Thihty-eight Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. 24
D. Paul, M. Ismayilzada, M. Peyrard. B. Borges, A. Bosse- lut, R. West, and B. Faltings, "REFINER: reasoning feed- back on intermediate representations," in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 Volume 1: Long Papers, St. Julian's 's, Malta, March 17-22, 2024, Y. Graham and M. Purver, Eds. Association for Computational Linguistics, 2024, pp. 1100-1126, 24
H. Luo, Q Sun, C. Xu, P. Zhao, J. Lou, Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang, "Wizardmath: Em- powering mathematical reasoning for large language models via reinforced evol-instruct," arXiv preprint arXiv:2308. 09583, 2023. 24
13

---

Satori: RL with Chain-of-Action-Thougg Enhances LLM Reasoning via Autoregressive Search
D. Zhang, J. Wu, Lei, T. Che, Li, T. Xie, X. Huang. S. Zhang, M. Pavone, Yi al., "LLama-berry: Pairwise optimization for ol-like olympiad-level mathematical rea- soning," arXiv preprint D.Xiv:2410.028885 2024.7 24
LI, E. Beeching, L. Tunstall, B. Lipkin, So- letskyi, S. C. Huang, K. Rasul, L. Yu, A. Jiang, Z. Shen, Z. Qin, B. Dong, L. Zhou, Y. Fleureau, G. Lample, and S. Polu, "NuminaMath." [htips: //huggingface.c///l--a--/a Aaaooa/a/oA CoT]shups::
matp:///ctmmmmmmamiimmooooo proggggssssssssssssssss progress-prize/lobl prize/blob/ main/reportmnuiinaddassss datpeett.ddddaaat.dd..d.. 2024. 27
Qwen, "QwQ: Reflect deeply on the boundaries of the unknown," November 2024. [Online]. Available: 27
D. Kocetkov, R. Li, L. B. allal, J. LI. C. Mou, Y. Jernite, M. Mitchell, C. M. Ferrandis, S. Hughes, T. Wolf, D. Bah- danau, L. V. Werra, and H. de Vries, "The Stack: 3 TB of permissively licensed source code," Transactions on Machine Learning Research, 2023.
Y. Zheng, R. Zhang. J. Zhang, Y. Ye, and Z. Luo, "Lla- maFactory: Unified efficient fine-tuning of 100+ language models," " in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Y. Cao, Y. Feng, and D. Xiong, Eds. Bangkok, Thailand: Association for Computational Linguisties, August 2024, pp. 400-410 35
Hu, X. Wu, Z. Zhu, Xianyu, W. Wang, D. Zhang, and Cao, "OpenRLHF: An easy-to-use, scalable and hirf-peef rlhf framework," " arXiv preprint arXiv:2405. / 2024. 35
Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," Advances neural information processing systems, vol. 30, 2017. 35
C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, Hu, X. Han, Y. Huang, Y. Zhang et al., "Olympiadbench: A challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems," arXiv preprint ar X4v.2244..11008 2024. 36
14

---

Satori: RL with Chain-of-Action-Thouue Enhances LLM Reasoning via Autoregressive Search
Appendix
Satori's Demo Examples
15
Additional Related Work
24
B.1 Post-training LLMs for Reasoning
24
B.2 Enabling LLMs with Searching Abilities
24
C Details about Data Synthesis Framework
25
Experimental Setup
27
D.I Data Processing
27
D.1.1 Prompt Templates
28
................................
D.1.2 Synthetic Data Examples
31
D.2 Format Tuning
35
D.3
Reinforcement Learning
35
D.4
Evaluation Details.
36
Additional Results
37
E.1
Ablation on Reflection Bonus
37
E.2
Ablation on Restart and Explore (RAE)
37
E.3
Ablation Preference Bonus
37
E,4
Offline Restart Buffer v.s. Online Restart Buffer
37
A. Satori's Demo Examples
2 -33g 100
(
Mathemaricaal Reasoning AMC3023)
Sipplyy opariion:
Burtor
rpolkee the poode the weats, and chiia nothe to pottioe soack tuaper woods oriple Wist dape The anal sewee eas be veuloos dee in. 2. wlene and ediiineyy
Problems
siceafa (9)
21
7--50-0 00000-0
Sabtmat opasias (3) fioe opaniai (oreeiee (4)
Solee fir a soood oypotion.
19 e (y) (i -
Satori's Bosoote:
Fas,,
the ccontisea tou cade. The cade with a sots conyer the thinogger.
2 P-yt- -0--0--- witt-
3-
dic center
"heeodin,,
Salen
Sabeciiyte S
Wast, Tet''s clandyy this The eqgantion yo 4- byy correctly Neet, on Aetercche the eqpation the worrd chiie Thes dirie witss Tharkke,
ti). The
ciede
Matusttino jy bes eepocionn (T)
poinls interetion cido, ye spstec
Tae pazaoo intenaciion upo (U, rad +* S))
dape poriay though
(r-47 11-0
25 4/ 205-1
Expezai equationn
The alape le Thrrrien, and faal stantse
Figure 6: Math Domain Example. Satori verifies the correctness of the intermediate steps and proceeds to the next reasoning step.
15

---

Satori: RL with Chain-of-Action-Tee Attio--o-cctt Enhances LLM Reasoning via Autoregressive Search
soch solving system of eqpatiins with time spent the coffee
Mathematical
Ressonnng (from ATME20224
sctation Fint, calentae the walking tinss givon sprade without soudddeinng the time speat
Latts exttch
By
coffee shop,
flsd
Priliems
UD te 235
the thes spunt wilking the ase speed.
Evsr uonius save tikinaatoiingg
shep
likezatezs
kaap tllueanth Wize sall axdecise sistin apeat adea Sahe the usbbii equatice toitey ils contioia fenuada eak
spead of kiazstes
leur We book:
Whes wolls bileretee bere call and wireter, bollding relettes qued ofher slop Seppour
takon len, anfallus the amaio speal is tte wtiee slop.
1-1-1-251
co Satori's Respore:
When sle walls of a likentters loc, Oar late spent walling
Fist se deternie thee spedds wilking slen
tiss speat the coffe dlop in aoeee. Thaneaan the thee spect volkiiz
the instaling
40 (030
The diffrenc in wilking tixms
boans
stapiity
State the datiazo the vale Hocester,, watte the exnation:
Ste- 5) tv
fi- ..== -F
Maze caud porltoo, on take Las pablti reat
5
+
quei af a- Silorette love. The sial tiee for the with on thllilng the bears. Thantion,
11:20
Now, ueal to End theckzz Ayoupadd velkingg visarde vilk gpard x kllaxter boat, "he apeal waldng
Salve the spadd etle
aparke wik toe cudbat Search
herrs
-267
Sisee the distare the oats kioester, seeer srlee the exgatioe
Sabeltine
Store roed be protiien
(x (1121(24
() -9
-20 1yT
BUTT
We rone hawe eenter of ton eynations
2
de) -92 i 21 di)
The Slas queat wakling
tiess epeat wakliggt
per
then
expoid bith exuatinee
de- #i
Tiaze velkin - Louzs
2. 2.4 4. B a Buteeee the fist oqueiia fises the aood equation achernoto if
-B-e Sylt
Convert lutza mitaton
2i [4x 244 L88
3 Looze 180 stlustee
18
Baxc scanAcc one gand in the coffee shap, Uhe lotal kaaz original exputions,
63
-1 6-15-
Life
1o4 dBe
172
Sinatione
The total case far the valk Intteding the thee quert lo the cofte akop
144--4tx
At IT 144
us s
Aller sazplioing, lnd that the loaa tine ie hoore and atuntoo. coliee aduntos wilk caboo Las, incheding thet tatsato
122
6o
Tharsten the total tiice
Tansfoo,
190 24 234
2442241 8)
sulectocapezz
the cofice skipp
(16+ ()
provinus
approceds
the probiem overty complicated havalves sanoeseasy
Figure 7: Math Domain Example. Satori identifies the mistakes in the previous solution and proposes an alternative correct solution.
16

---

Satori: RL with Chain-of-Action-TThnn Enhances LLM Reasoning via Autoregressive Search
Reasouing (from
Proldem
and
(ptena ABCDD
Isay]
roal
conttclo Ri--Clighh Aemoad deoucco, Tea)--th, high 1-5), arrea))
pate pale patr 66, petr a Ab:
Sabel 2abeary "BDY", SabedrV seay, My Sabedcy sea"",
[/bag]
Satori's Responee:
spanitiaterd ABCD Alis B11, 1), D1s,0),, ou Stockec Thaeex (sbe knewu Gesa seas fenash plogpea):
siato thet pabpo
with witiie
Wotu sotu bego suin
We apple the Steelive
A g 112-3 3-
porducts
San Kalaa 1 3-5 4:222m4-22-2-------:
Stlethtete there iaase lato the Stoeler "I Liansee fireals
Ang 6- 1--188
Watt, Let's dirits this. The application the Shorlace Theceem ie cornect, and the calculatione our the susta of the products of the courdiastes are accurates leading to the cocrect area of bricagle
Fri, fest the wos of inaegk, ABD: B1L
Ellngl werdow
Arta ritee in so milo in i mis maid
For elengle
" 10 42t 401-500 -01---h fi- 1- -1-4--1--
Neet, trd the trluad ICCI canaa: 811.3 C13.2) 516.01
Atriano 100- +00 (1-2)) - --11-3
Froily, on odd the mnee of the tos thisggh get the ara apaditateral AIC0T
Amsanno Ameznu Aemanu
Riteoal
Figure 8: Math Domain Example. Satori verifies the correctness of previous solution and initiates a different solution.
V-W 5-- rooed Blod
Mathesmatcaa Reasouing (from
Thenken
Problem
Vij [4looe] Hpood
inji ded viler Iproj
The mastnere valpo of (sed sleds eccrs slex (it, alex and the same dinetee). lerest possik nix [pni,
The feania for the progction
1-1
proly
Thrrkks, the laquu dhde valae foros v
B
rogunaade projection grot, te
Watt, The expineatios
recomppte magiittado poojection directly using [y][sejoon
skup
[seuiu x|-
[w]]
Showe (w] no loss wow 6f Gi. Theeefunn the roper fhe sugerate do the projectoo teorse:
[M]]
[are], vl- (V-W)
Gloo, I+]
ise posttct in can exprosed
rogystatee and codio angho
Isno Houff
Ietooin
The suodenera vine of di le valua Iprot,
Giess i and fwj- lave
Tharkke, lagus paidll value Janis va
Figure 9: Math Domain Example. Satori verifies the correctness of previous solution and further explores simpler solution.
17

---

Satori: RL with Chain-of-Action-Tee -cthoo-oohtttt Enhances LLM Reasoning via Autoregressive Search
Thes tinal nemiiee of posiilio perket coloe that am fiviess 17
Muthematical Reasoning (from OlyupisaBeack)
Re pomppl
2x
Probium:
Let y 1. Garpate the sunlor of pookie perfect cakes that ano disk.
Walt, The mistake the epprosch caleslating the expo- form prrfect calor, which is unatly comples und mnt eoch
Saturt's Resporant
valan of 3
Lot's a satution. lustred of
expo-
For
directly conuddr the expuramts for cads peius factor that are muk lpko of and then
Nest, wo to devertiee site the priner facco bntios 1T, The Sotoria
Latt with the yrirs funistatioe 171 and soxe on the nog equeents
We sne sttrated the annder pastis pariec cales that wr da-
p-
Tar lacrdizakkn 13d indaado prisas 3,5, 11, and in,
suaiher eage dasas oquanee tase seline
eethiiee For eppoose 111 The poefil expoceets
thet
fune braiis wottippe of shops correctly
Wast, clarity
noed fidd pelno Sectriitntin determiio the mumber porfir culer fivio..
For p 3: The eqpaane in. 17 a b. The pootles expoerate fur
Fir
(VT The pooile expeetals for
nee to copdica the potie bxteriatto of 17 and low the expecette of
perfes wr
perter
Frsi Ie's did the peine fecto natass of (o) The fectoratiion of to come tanabers bos thin ayall tutorbatton oncher these polto
Nov. reed Ceterrtion you maee thew exporiento car Seins exporct
prrfuct
pootill
Fir yease annderr expensal p in 111 grons by:
be zattibik eta We clack eac expoeeat:
Fie 30 b (eooges millng of
perfect rale Ther 1 dalio
perfoc cale a There 1 taoee
poodit equeser
ospesest
sid poodik oppannt
We will edleritto this for peice wiise lis than ogaal
For p
The uo 17: 1 last as littgge andl le ant maaiippn of 3)
snalupée of 30 odlor prize: naabbere can
yi 110 1 u tact an salner
pedfcct They
The nea pollio pefice ochos
distooe
For
Bie p 2 dedia 30 3,
P
Figure 10: Math Domain Example. 1) Satori verifies the correctness of intermediate steps in early stage. 2) Satori realizes that the pervious solution is actually erroneous and then proposes an alternative correct solution.
18

---

Satori: RL with hetti-ooo--Aciio-TThgggge Enhances LLM Reasoning via Autoregressive Search
Logical Reasoning (Drom BoardgamecAA)
Probiems
silt the billoiig posilee Your samme shorls "Irar", "Maket" oar "Unoertab""
and feende dhet The plonochmmpt
fottlel calre
indio, end oi leadar eouuies The wel lank the cadda laei da has line the lee last doer tok lest pawcr plant sne lie some fadle of toky
pelion if
footled that
the is 503 y 361 inclee les. Rubei Secunfing the out Ih's ovels in stiuutio the so courtde mue telees an task Su the pellion livee you eust seuntioy lake the caok thet ehe las fro the ben pollksn thes
pellar. polloo will acer leng, od tazk the inguel codduiaa Riket Yar ligr will nazagr perezale the
periaac the poliaa and 2 the and polias lik (Lap laari lover fnnada
ol
oe Bindd is
Talet Rood gadtonal owe Belol
Base game
doe the pellis leing bank dragss"
Satori's Hespous::
The kgoe has faeed that
and fricals that une Inc.
lige
The mottball with catter 22 tockes
The wall Midee
tuur froe
dow fedid a pouse plur aou grven fiile the laady. ydiion
bodd2 The will bengg ol tode for lae yddion ila (tke snil lea fcollell caal lae as 1S.5 a did x 35.1 iadao lass
Iidet If you that wondliing hiw the cads that de bas from the hes lut loss out frstd poon jlant rwn the givw rokle of the keddy shist on you contilaty concladel Yori can rcecitdo that it dis not bring
tuxic or ydiion.
Brtak the pelkin, you lass teo of esiinacs il the lae
contane the pelkon and 3) lie inal leaap takk for the pelion,
and "pelkaa sikk Lowa lezzg tasic the the daagoe
20u2 voachaees
The
wade pellios it ttho ligho) for Seew than fivs fiiedd.
Whor. The
colution onalyitng the ghin informatiion and rilee without direccty wlldreniing the spoctiio of whetlerr the pelllion hrings on tonk for the deagon.
stsp
dragon.
Liase's Ittenda sad Occupatiion:
The 1pm plooaiiamaaal
gring and frieods
Accoding stadd bat: aussun to peocaade pillon ira tite frow thas for brinads the liye
concitm
Seatle holl and Hlee.
indao.
The desenanea of tie les am 92.3 a 450 26.1 tubhoo
glves $4e7.
jemn ia foctonay in s9om19 colire iockes
Thee witzer the bosc
Slase the wikeze of cas footioll is los thas the mésaze of the loos the stale Bostbell the the boe.
pollius
"Jmmmsn the and wis teing trkk fhe tas pelkon.
Froe Bules If
that shencinng lake the mada Uhe sie has froes the law bat des tor buidl plant ane the grene fitall the Lanicy, what cas you certeialy conclade! You can andduk that it foes not leing
bur pilion.
"The kil cands that from the hes dow belld
tear the taidk the teabbe.
TTariio, hr and does not lenng yn lack be the pelket
will
Glenn the rake and the intemation, the oven lightl coctudos that the wad willl lovngg in of trak fis the pelltcs, degiito the untrradiioon
Bool the esabiia theraa and the information, the and will biing of teak for pellaz.
Thevrdes the teal sewee in [Thee]
analystig and focus direct qusailue. We nsed to determinn the polikan bring taak fov the drugon based the givan rukor the indurmastion about the lager and
hettaging
Lat's simpliy
the wad.
gloee
bye hac Shoeek oasy gaig sod) the ligs ressage to coseivee the yellkon (Bure))
(Bakk)
The and laln the cand that akz law bes the his bet dos aut bedid a pows plase oue tae gaves Soak ot the lawks, sccandigg to dould, tai wal doa ast being ss oll takk dee tae pellia
speeife
Gren the mainallii so ued monobatt leind for The bey reoggee that halli groedes des condttinn that aegites the sll back le lenggin helawier if boch oraifiiee Siee the snal meds Oee takes prosoionne owe Rake.
condikee loileg
sebling a power draas.
Therbeo, pallon dow
The feel unsver Fahe]]
Figure 11: Out-of-domain Example. 1) Satori identifies the potential mistakes in intermediate steps and initiates another solution. 2) Satori realizes that the pervious solution is still erroneous and then proposes an alternative correct solution.
Rosseming (from StrategyQA)
Protekmt
are givs the Sileitz fada: Thchepphn Bover retherd
quiaa and vory that let quaitppaag
pestalée
Al us jotai, Candaaaa Rane's a tonaaai yna coilig S43 parls
daddi "Toa"
he wer poor wnall Thrloogee Baer ham load"
comernmeennt co Satari Remponen:
The whk viattr Chrlateiar Borve word
wan poer. ti edtil and kape Chemnohhe Rose's spisal cori injory les los geedripogie wlecs seaes was prosyeal han the uod does This saade is dificat Se lin, to cseve and peiken mayy dull adtinios
stunes tile,
the
of tratease.
The
The oun lis mostoeet une Sitouet youly wakh a disittust anccest saaayy
imprec conditiio Wi Walkost the quaditen colutionn Rooue dut that queetion welse would liee poor, and coneiive whither the out reeatment would lave been determiviig Sutter
sadiral tontanral, leghle calinly hat le worbl hav less vilk to lor casndd Lif.
stility live,
siven cisumatanoos
Lot's weitch
survical.
wallste the opoetio wonn coukutyy Thes vale Cheietpper floo would komo be
The byy facous ccuidir are
The set of las mestived lajeey.
iepect yourry
Cheieorake Hone's apiial
and sopazed condualard candical equizaenn and copptngg toatrarai The out of Suneaoo peacty anberartil and Hally logoed the ausy popll, opedalll thee
vioo soe poor.
Rass
would
twon aftent the leve rendica The cos lis woild love apuikoor Snsecid handor
Gieas the sowniss ha cocdiiee and the ligh. oot tme sonk. il masosalle to contiale that be wolld not ham hem did love overed Me be vore poxe.
Therkoo, the Teal craser
Figure 12: Out-of-domain Example. Satori identifies the potential mistakes in intermediate steps and initiates another correct solution.
19

---

Satori: RL with Chain-of-Action-Tn -ctio--o-hcttn Enhances LLM Reasoning via Autoregressive Search
Domain-speciffe Reasoning (from MMLUPro Chemistry)
Prolicms
uader otverriee, taker onhendde stated, therraiigooee we 588.15
placed introdibett Cálrabon doae
getir most maciion bink proorare stu vui troggaation 39C
[6) wilk does)
(h) -6.33
[0] 1/6c (hutavv wak does by sestess)
-1/83
[1 1328
systee))
Resporann
The madise letena and Lytandlari real
2011 Myl'l
sysson as
givex
Mi
sher t' the stroeghests conoern and AV chrngg velerne of the systion tydrram prodhend tacke Fari, ts deveratee 24,3 grack Thmstoo,
equatica dany
tangandi prodhcts losbogen gua
coles My
56.11 glned
1bg
Show it made of reag reagrwimm winn miee I vade teahigen gas the srncher of mode of hochige gendawwd i cloa 6:17
The valanu lydicem que producd
(20633
cakolated
#RT/P
star the camber aadoo yna, & the sknd oesitea (1eea00 Lax I tesperatiec Kelvas F
sitecelenes
unnocwsaryy for cartly
aupp cocrectoy stauiity this ficuulaa wick, culcuaatin the moler umgueeium hydreges ar produced, but
them sheosoo wm beoree rerriks wvere booke and the valaoo durne is tegle urder the glas caiition The syete doss not expord corppose sgaint enternal prsean.
Was., The misatlo artla the volume change is incornectly anommdd to be anstuiids baning to the lucornet concunton that ou work dias.
lydirogen
Tet's mo menellate the grolloe cipuly-ctey
hydropes gas prodhaeed
The lalosed epuaiies
2013 Malli, Hi
spartes, Me
The of Aig tlmall. Tandon, the tradee of takko of Mi
Ibg calles My st. glted
Sn the remiie recke of It's protuced also 0.877 wel.
lydirggu sue produced:
Usig ta link uft.? alece sxi, Rt dxezts (aatx/tudk)) KJ, 208.15 is R. and P- stat:
V.022 trecte
152
Calcubate
by asstems
The
-WW Shaw the mortis contn haodor and
condder
Uravet La urnng the comerioo Butter Latte
Tlamkee
The ocnect
Figure 13: Out-of-domain Example. Satori verifies the correctness of intermediate steps in early stage. 2) Satori realizes that the pervious solution is actually erroneous and then proposes an alternative correct solution.
20

---

Satori: RL with Chain-of-Action-Tee -cthooooggtttt Enhances LLM Reasoning via Autoregressive Search
Domain-specile Reasoning (from. MMLUPro- Economica)
Probien:
What Blrsed Dhooer wodld bakee Kisnodaa theory suriiiy, lappy the guort ribe poor cocatriee
Rel cernbe ropellone
(8) Pore cocaizice sill Lave Das anc sportl nose ladl eikek of Seriap
(h) The lind frome les to inpartt cartnyy
corntry
Bida scll loss aroeth cato
(02 Pwee onende
kis goorth ratee
ligh groch
[T: Pooe cossice will lees loe gooeth nice
reparilis
lodls
The Haroe Diener mookt conenese aoodel thei másea the gaoe miee of an
snd capiallnigni mias The aoold rightt ilat the goeth make of dxteranaal by dl milu growch corlage and the capital-output rotin, whithh is correct starting analysing
entrgr capiial mit acheraed le
mils.
Wair. dailiyy Harrod modil Indred
poor connrriee
eapiral
Wast, The rxpissiation of the Harnod Doune modil cornect but the stop ant dineciiy addiress the implications for comntrie the contest Keyondan theoey aastiags
wertch to new actution. start by explataing bow the Harrod Domar model when camblond with the Koyonsta theory of savlogs suggots that hightr sactapg ratox in post countics kad
The Burne Docorr uould eraasts that lee grelth sale as eXEAARy is oetenizad be the no to awrnggs ts eqpaal Acneding the Keyaninn hony saxiags oxsris will legher switgs cain will lave eanr fands
poot orarrke sher the oitial tock shin beee vnlation to thez popelation Maker oxtaae rate oa anad to liaae capital stock ovr thas. THis tamoran is cxpital dtokk on thea agppott lightr book of sarpet and,
Was. The euplanatiios dues aue dineclly aidenss the options pewviled in the quession, w lids specilic the goonch nakes of your coustrio rio.
switch lo new vduution. Ambra each uption disermin wlich oas
implicationn Harrod-Danar modal and the Bayssiiu thoury of strringe for pour couttioo.
waluste the
[&) tiik and pour oxembe expelooee cazes gootss vater reger bew eadegg caiee
dille
pose
(B0 Floe vocaleine sill lave Bas tu gooels mise pa pooee seriap.
divecty sodaggs
(C) The livd of indieskegy detemiss grork ndee not untags
expleiily state gowd's
santnge the expitallopput
(D) The and of leaase
strtap.
aeseding Koramin Laney
euxeets that bagler Innose can lead Migher auritage
(E)
The not tean coseding
Blank uoddd whach saows dioci seakiy be wes seniage and goonts..
TTiee in not on de tra li's oecebbe way booe tightr groets vatee taey hove sigh cattagg cotie and lvo cxpital cuplaai-nttaat ratils (G) oumnm
high gooil
Thee uecomed teac coutiias tsey han loar snocila cato & they lave low
aaal lial. eaptuboongga mikas.
(13) fliss amde
will sheure Docar
bigh groets
wouider capital vals High
alono outh for high grocts
(T Thor
hasn be gott
eapliskonpps
11) Phot cousino ou sniace shut sporc pine mamudoe ummitaaa kntis
Gloen the analyiie the recd seme cnteeet lead the Homol Borae modid and the Keyamon thevey savings is that coinerke can williwe high gooets vater if they laom high savings nove and lor capilaaaae
Tlmmees tead scave
Figure 14: Out-of-domain Example. Satori engages multiple self-reflection processes during intermediate reasoning steps.
21

---

Satori: RL with Chain-of-Action-Tee -ctooo-Togttt Enhances LLM Reasoning via Autoregressive Search
Code Ressoning CRU (Fol)
You un dlon the Sikddig firdio
spai Leat: (sep, axxоptac)
aptikbon
laaed m)
recors sep jein (ow.eel
gins the follnitg apst toke th tape,
agui vactiiko fasciant::
lepst
orrpet
Satori' Roposs:
fareiio
sescher aveephtr. full the muiling ports, thon crssctrnato roconnt half adea cop
freetios Fillos
parts ocne epprine any up is croms
by sptiiting
with funetion's Inborded bohovier.
Lat' look can the faaticce zap sig was the cone laper: vatuoel, 2r,
nno lai notaaiing
ering and for engir (besono songgis het Chero movermmnoo of by
spisteed
Deteralin the angth
aptunced
Spate
teo Shiw kegtu the tart balt to the taxt steantte Jhodoc 1), and awcrd bulf the cenaliing okounare (ludes
des rww.aplateed splittesls // 23 Derttatei
1y, (1]
see (oc.rpiceool () - E.", cectubed 1 wyt_eeliee
riing repocone
splatted (1), "actater"
(4]
csseat Jooua apastted loctdd "actabes" 'pt aho the
actabee p g*
Spite the dhiing balf
Lat's norile comeet sproock
cortide
approoch toopht
Spili the atring vecaled lato une and Trwe.
Connaterate laor' and Thei with seporator 'p' tomphew.
Therken the Teal wrese pemplor
Wast, The misntoo the moureess handlng of spilt and the inlueeuuces concatenation, basligg to nu incurrect final string,
arring parts using esparator
apple the finetions legin dlnwehy
Syatt sugth:
vatabed counds
the fraetice las dimeat:: ("artubet"))
Iadf oppratte
Reseese the fert belf ire ball eupro so cotetts expis. [acabee)
Thembee
Figure 15: Out-of-domain Example. 1) Satori verifies correctness intermediate steps early stage. 2) Satori realizes that the pervious solution is actually erroneous and then proposes an alternative correct solution.
22

---

Satori: RL with Chain-of-Action-TT -ttho-oohcttt Enhances LLM Reasoning via Autoregressive Search
Tabulur Remsniigg (from TabloBench)
Problemi
glon Billwing
Poak
Coudrr
(m)
(0))
Norn Knrddeni
Beadds Decionat Repolili dill the
-31
1195
Seath Seron
3157
313)
1997
Freegwiing
Seroo
Malt
Dressnati Beysbii ofe
1480
Wakkon
Denaitaas Hejehii the Coago
3866
1570
1335
Meloirms
Beaads Ugaads
Delk Tilk el pala and liee decals
Bood es the talks secve the fiEoklag qpetios qpericn you arowe actranted fwo the palne rade cur that the seeee rasrty the care the onneejrrdngg mnbie in balls exnar proataess nctzial polks Desaocack Bipelike Coeao lhai denties hust
Sataris Peeposson
Denesrari Reyabllo
oloutton
tabli,
Mixra Starky catcee)
Kalebo (3725 suntmo
Widans axtem))
yroechmnos thas pocks
Meent Stackoo 2801
Mieza Kananie: 5812 sadars
Seal Aodez 1862 sadurs
We ver the porrleraies of theie pailo:
1884 2892
5857
1300
There uee peaks divde the istal sun of poooreeecss Ged asmagn 20055
Therken the wvreggy pereeneeen of ressttais peaks is the Democrate Revalll the Orngs that heve vn elevation of frrst 3000 meters in TTll solution. cusalder
musters
slevatione lssat 3000 melecs, kich Moust Stanlay, Mount Karidmbll Mone Moli, Witnevi. Heuilly moseron polk the Deneratte Regedite Coego with en elsarioo of lest melece
witch
Stceet Kabankk (1227 paters)
Whlted 13838 metros
reusssenno Moust S Sinzkos soor
prabe
Street Kokbaki 2213
Widact:: 1379 netees
Calolate de stus styp by sige
3831 -018 7561, 7963 126t 6855 6455 1570 10435 2e0t
draal
Eadl
Figure 16: Out-of-domain Example. Satori identifies the mistakes in previous solution and proposes an alternative correct solution.
23

---

Satori: RL with Chain-of-Action-Thuuue Enhances LLM Reasoning via Autoregressive Search
B. Additional Related Work
B.1. Post-training LLMs for Reasoning
Statc-of-the-art LLMs have achieved human-level and, in some cases, "superhuman" performance across diverse reasoning
benchmarks. These benchmarks span various domains, including mathematics (Cobbe et al., 2021; Hendrycks al., 2021a; la; Glazer et al., 2024; Mirzadeh al., 2024), programming (Chen et al., 2021; Austin et al., 2021; Zhuo et al., 2024; Dai et al., 2024; Jimenez et al., 2024; Gu et al., 2024), logical reasoning (Han et al., 2024; Liu et al., 2020; 2023; Tafjord et al., 2021), commonsense reasoning (Geva et al., 2021; Talmor et al., 2018), algorithmic reasoning (Veličković et al., 2022; Markeeva et al., 2024; Qi et al., 2024b; Fan et al., 2024), semi-structured data (Wu et al., 2024a; Lu et al., 2022), scientific knowledge (Wang et al., 2024c; Rein et al., 2024), and world knowledge (Hendrycks et al., 2021b; Wang et al., 2024b; Srivastava et al., 2023; Liang et al., 2023; Phan et al., 2025).
Recent advancements have concentrated on extensive post-training to enhance LLMs' reasoning abilities. One research direction in this area involves constructing -uunnn datasets annotated with high-quality CoT-like reasoning chains. These datasets are created either through extensive human annotation (Hendrycks al., 202 la; Saunders et al., 2022b; Yue et al., 2024) or by distilling data from more advanced models (Yu et al., 2024; Toshniwal et al., 2024b;a; Ding et al., 2024; Luo et al., 2023; Huang et al., 2024a; Zhao et al., 2024; Abdin et al., 2024; Min et al., 2024). However, human annotation is resource-intensive, and model-generated data inherently caps the student model's potential at the level of the teacher model.
More recent research has focused on self-improvement approaches, where models are trained on data generated by themselves (Zelikman et al., 2022: 2024; Singh et al., 2024: Zhang et al., 2024a). While self-training mitigates the reliance on external resources, it has raised concerns about potential "model collapse", phenomenon where the iterative use of model-generated data degrades performance (Shumailov et al., 2023; Wu et al., 2024b). Additionally, reinforcement learning methods, particularly those based on Proximal Policy Optimization (PPO) (Schulman et al., 201 7a; Ouyang et al., 2022), have been explored to enhance reasoning capabilities. These approaches typically utilize reward models, such as Process-Reward Models (PRMs) or Outcome-Reward Models (ORMs), to guide the learning process (Sun et al., 2024; Wang et al., 2024a; Yuan et al., 2024), resulting in significant performance improvements compared to supervised fine-tuning.
B.2. Enabling LLMs with Searching Abilities
Chain-of-Thoughh (CoT) prompting (Wei et al., 2022) demonstrated its potential to improve reasoning but lacked mechanisms to correct previous errors once committed. To address this, subsequent work proposed more sophisticated methods (Yao et al., 2024; Shinn et al., 2024; Besta et al., 2024; Madaan et al., 2024; Yang et al., 2024b) that prompt L.MMs to search for solutions via forward exploration, backtracking from errors, and finding alternate paths. Heuristic search methods (Hao al., 2023; Qi et al., 2024a) have also been adopted to enable more effective exploration of high-quality solutions. However, prompting-based approaches improve task-specific performance without fundamentally enhancing the LLM's intrinsic reasoning capabilities. Moreover, recent work has pointed out the inherent difficulties of current LLMs in conducting effective self-correction (Huang et al., 2024b; Zhang et al., 2024b; Kamoi et al., 2024).
Recent research has pivoted toward training LLMs explicitly for exploration and backtracking. A large body of work has focused on enabling trajectory-level search abilities, which train LLMs to iteratively identify errors in their previous complete responses and produce improved responses, relying on either human-annotated revisions (Saunders et al., 2022a) or model-generated data (Kumar et al., 2024; Qu et al., 2024; Havrilla et al., 2024) as training data. Another line of research has investigated step-level search techniques, which induce more fine-grained and real-time correction that enables LLMs to identify and correct mistakes once they occur. Some achieve this by leveraging another model to provide step-level feedback to an actor model in the reasoning process (Xi et al., 2024; Welleck et al., 2023; Paul et al., 2024; Zhang et al., 2024e; Setlur et al., 2024; Zhang et al., 2024c; Guan et al., 2025; Zhang et al., 2024d), but such two-player frameworks suffer from high costs for model deployment. The most related our work is SoS (Gandhi et al., 2024), which attempted train a single LLM perform tree search as a flattened string. However, the effectiveness of SoS has primarily been demonstrated on simple symbolic tasks, and the ability generalize to more complex problems, such as math word problems, remains to be explored.
24

---

Satori: RL with Chain-of-Action-Thoupe Enhances LLM Reasoning via Autoregressive Search
Sample Initial Trajectories
Feedback From Critic
Correct Mistakes
Question
Critic
Ask for Feedback
Generator
Verify Correctness
Critic
Provide
Iteratively Correct Mistakes
Xm
Feedback
Generator Self- Reffnmeefnnment
Correct step
Incorrect step
Wrong Trajactories
Complete
Identify mistake
by Generator
Explore new idea
Demo. Trajectories
Prefix steps
Figure 17: Demonstration Trajectories Synthesis. First, multiple initial reasoning trajectories sampled from the generator and sent critic to ask for feedback. The critic model identifies the mistake for trajectories with incorrect final answers and proposes alternative solution. For trajectories
answers, the critic model provides verification of correctness. Based on the feedback, the generator self-refines its previous trajectories, and the incorrect trajectories are sent to the critic again for additional feedback with maximum m iterations. At cach step, those trajectories with successful refinements are preserved and finally, a reward model rates and collects high-quality demonstration trajectorics to form the synthetic datasct Dyy.
correct
C. Details about Data Synthesis Framework
Sample Initial Trajectories. The details of data synthesis framework are illustrated in Figure 17. Given an input problem x D, we begin by sampling the generator Ty to generate K initial reasoning trajectories. For each trajectory 1 ....... .r.] Ty (-(x), we evaluate whether the final answer ÿ,. matches the ground-trute -ruth answer y* Based on the evaluation, the generated trajectories are divided into two subsets according their correctness, which are then processed differently in subsequent steps.
Critic and Refinement. For those incorrect trajectories, the critic ma provides feedback to help the generator address its flaws. Specifically, the critic, given the ground-truth solution, identifies the first erroneous step ÿt and generates summary Ii+1 of the mistake as reflection, along with exploration direction (hint), i.e. [ Next, we ask the generator Tg to self-refine its current trajectory based on the feedback provided by the critic, performing a conditional generation the remaining reasoning steps, [ū++:..... UL ~ x ax, .
xe( (1+2) ~ Tr: ........ y²)
....., Tt - (2)
For correct trajectories, critic n. focuses on verifying the correctness of the generator's reasoning. random intermediate reasoning step ÿt selected, and the critic provides summary explaining why the preceding steps are progressing toward the correct solution, i.e., U1+1 Te(: xx, V1..... . D1: "),, where U1+1. Similarly, the generator continues from the current solution, generating the subsequent steps as 22 ÿr.) (- "i. ...... Dr; 11+)
Finally, we check whether the final answer ŌL aligns with y" The above procedure is repeated iteratively, up maximum of m iterations, until the generator produces the correct final answer. All feedback and refinements are then contaminated to synthesize the final demonstration trajectories. Additionally, the trajectories are cost-processed by inserting meta-action tokens at the beginning of each reasoning step to indicate meta-action type.
25

---

Satori: RL with Chain-of-Action-Thuughh Enhances LLM Reasoning via Autoregressive Search
Trajectory Filtering. The above procedure may yield multiple demonstration trajectories for each problem æ. We then select the top-k (k < K) trajectories based the reward scores (ū a:) assigned by the reward model "y- This approach allows us construct diverse synthetic dataset syn containing high-quality demonstration trajectories, including (1) short-cut COAT paths that boil down CoT reasoning paths and (2) more complex COAT paths involving multiple rounds of self-reflection and exploration of alternative solutions.
26

---

Satori: RL with Chain-of-Action-Touuge Enhances LLM Reasoning via Autoregressive Search
D. Experimental Setup
D.I. Data Processing
Data Source. We construct our training dataset by combining two open-source synthetic datasets: OpenMathIn struct2 (Toshniwal al., 2024a) and NuminaMath-Cof (LI al., 2024). After a careful review of the synthetic data, we identify and remove invalid questions improve data reliability. To further enhance the quality of the dataset, we adopt the mutual consistency filtering method inspired by rStar (Qi et al., 2024a), which removes examples with inconsistent answers provided by different models. Specifically, we utilize QwQ (Qwen, 2024) to relabel the questions and compared the newly generated answers with the original answers from the source datasets. Only examples with consistent answers were retained. Additionally, we apply de-duplication tools from (Kocetkov et al., 2023) to eliminate redundant examples. Through these filtering processes, we finalized high-quality dataset with approximately 550K samples total.
Multi-agent COAT Data Synthesis. For the multi-agent demonstration trajectory synthesis framework, we utilize three the generator, Llama-3. Instrnst as the critic, and Skywork-Rewe Reward -Lamm---- 3.1-8B-v0.2 the outcome reward model. For the generator, we the temperature 0.3 and the maximum generation token limit to 2048. First, the generator samples K 100 initial solutions for each problem, dividing the generated solutions into three subsets: correct, incorrect, and invalid (those that fail to produce a final answer). Invalid solutions are discarded, and we randomly select four correct solutions and four incorrect solutions. Next, we set sampling temperature of critic to 0.2 and a maximum token limit of 256, and let critic provides feedback on these selected solutions, allowing the generator to perform conditional generation. This iterative process is repeated for a maximum of m = iterations, potentially resulting in up to 8 16 demonstration trajectories.
models: 2.5-Math--BB----
During the generation process, various situations require different prompt templates:
1. The generator produces an initial solution.
2. The initial solution is correct, and the critic verifies its correctness.
3. The initial solution is incorrect, and the critic identifies mistakes.
4. The generator generates continuations after the critic verifies the correctness of its correct initial solution.
5. The generator generates continuations after the critic identifies mistakes in its incorrect initial solution.
6. The generator fails to solve the problem after refinement, and the critic provides an additional feedback to identify errors in the generator's second attempt.
The prompt templates for these situations are detailed in Appendix D.1.1.
Among the synthetic trajectories, we categorize them into the following types:
1. Type-I: Synthetic trajectories without critic feedback, i.c., no reflection actions.
2. Type-II-I: Synthetic trajectories that include an intermediate reflection action verify the correctness of previous reasoning steps.
3. Type-II-II:
Synthetic trajectories that include an intermediate reflection action to verify the correctness previous reasoning steps, and 2) a second reflection action to correct mistakes in the previous solution, followed by explore action to propose alternative solution.
4. Type-III-I: Synthetic trajectories that include reflection action to correct mistakes the previous solution and an explore action to propose alternative solution.
5. Type-IIIIII: Synthetic trajectories that include two rounds self-reftection and self-explore.
Examples of these five types of synthetic trajectories are provided in Appendix 1.1.2. Finally, the outcome reward model applied to select the t (k=1) sample of each type from the 16 demonstration trajectories based on the reward score, such a type exists.
27

---

Satori: RL with Chain-of-Action-Thone Enhances LLM Reasoning via Autoregressive Search
D.1.1. PROMPT TEMPLATES
Prompt Template 1.1 Generator generates initial solution
<lin_start buser
Salive the following mata probl efficiently and clearly. Flease reason stop step, put your zinal within \bowed]). Froblem: i in_endi>>
start hassistant
Prompt Template 1.2.1 Generator generates continuations for correct partial solutions
<lizustart bayston
Your task in te continue writing a colution to problen. Seven a probien, a correct partial solution, along with a sarity check previous sceps, you sheuld continue the current solution to solve the problen. Think step by step using English and adhere rollowing
Step: [Brief auplanation and cstculations]
(Briet explanation and enleviations]
**Ploase do
Eng-ish.
Your solction should aonclade with "herzfoor, the fimal anaver ias \ (/boxxe((answer whare [answer] la the final number expression that selves ita problen. all steps in the solutions should be bricf and alxays start with "Stapi" and end with two explicitly reter prerious and should constituto coherent.
progression towards final ansver. inaser.)]in_and]]
<lizustart puper
*r Proniem (4 8are5.
as Partial solution with saalty chack
Steps Firs., express 16 274
as povera
MI 2'2
Stop: Subszitute these expressions
given equation:
VI (2ract 61 "m: (2"2) "2 2
Use \Erao(2" (4n1) [2n]]
property M(2")) (3m//M simplity
Stop: Use the property of exporents (a"" a (d-n))) to combine the exponents:
2"1
Stop: Since the bases are the sans, se can set the exponents equal to each other: \I 4n
corractly slmelitied and 4/ <lingatat bassistant as continued solution
and equated Ve4n
which in essencial cind the relationship betveen ((nk)
Steps fand a Mh, on newd Lo wolve this equation 1 VI and Fizat, divide equat by 2 Lo simplidy: AE 2m - (2) VI
Steps Add 2m Vfrac1HH2 Al
both aides isolate 2a. M
Steps Add i n AI Lo both wides Cind a.n Ma (Fraa(1)12) \frec(1112)
Step: Simplity the right side: 1
Theretore, the value of 1 m is: 31. \boxeel1) in in <linsstart buser end endj]s
cocinetrussionss)
Partial colution with eanity check
<lim start PSsssistant
toned solutioon
Step:
28

---

Satori: RL with Chain-of-Action-Thuuge Enhances LLM Reasoning via Autoregressive Search
Prompt Template 1.2.2 - Generator generates continuations for incorrect partial solutions
<linustart Payston
toor se continue writing colution problen. soven pronlem, incorrect portial solution whare 1ast stop wrong, along with e revisior reedback and hint for next stepo, you should cont the solution using the nint, Do not direct ly copy contents under "M Incorrect partial solation with revicion #codback and ainz for naxt stopa". Think stop by stop using rollowing tomat,
English
Step: [Brict explanation and cricuistions]
[Brief explanation and crleciations]
**Plcase do
Eng-ish...
Your solction should conclade wich "Thezefors, the final answar is: ( whare [answor] in the tinal number ar expression that solves the problem. All stops in the solutions chould be brief and always start with "Stop:" and and with two ateps, and acaps should conatiacte coberent
line breaka. Bach proqression towards finsl answer, ]nsser.<|dn_endd (in_an))> <liz_start buser
should explicitly refer
as Problen
hanging head, haif days daily grays until zeaches there full potential. Besearch sheas that soa-tourth of the gray baiza are deposited on the flest day, and the second day is when half the vemaining hair has disappaczed, leaving Devanti with 15 gray haizs on hez ful. head. calculate the total number hairs Devanti
Devanti shoulders. To propare
tied next photo shoot, Devanci decides
from hair.
Incorrect partial solution wish revision feetback and hint for next stops
Let
total naber gray haixa Devanti had initially.
Step: do the firat day, one Sourch of the gray haics are depoaited on the head. Thecefore, the number of gray haica cemining aftor the first day is:
x |[rac[]]|4]x (4)x
Sait. I racted one Sourth of insteed of adding it, bus gray haies degosited (added) the fiest day. Aloernationly, should one fourth <liaustart bassistant
find the cocal number of gray hairs after the fleat day. (iaannd)
Gondd ipnution
Let
first dayts calculation addine ona-fourth qray
VI
(teit(Tnitsaa aal gray hairs)
(Gray halis deposlttdd on the First day: [44x
3]
Stext (Total gray hairs after the first dayl [fra (Frae(51(4)x
3]
Step: Then, celculate number gray hallss renaining
day, half diseppear,
31
Stext (Gray halrx after the second day' (2rac(1) Meines (4)) (Fra
Stept Next, set up the equat fon, based on the information fon that 15 gray haira rensin after the second day-
(8:x
Steps Finally, leltss
V Cind the initial mumber of gray
M
15 Wines (5) 24
Theretore, the rinal ansver is: <lin_start buser
codisstruzsonn3>
Tncorrect partial solut fon with revixion feedback and hint for next
codtrajectoyyosciii end <lia_stort Pacsistant Continued sclution
Stop:
29

---

Satori: RL with Chain-of-Action-Thon Thhuugh Enhances LLM Reasoning via Autoregressive Search
Prompt Template 2.1 - Critic verifies correctness
General Botdell inee
You are student, tollowing guidelines:
caretully reviex your own correct portial soluticn moth peoblem, ednere the colution explain 3, You are provided with the question, the ground truth solution, and your steg-by-step partial colution.
Veriey corsectness serterce]*
"Verity: [briet explanation ccrrect
Your respense
parases "ground solction".
5. Your response should be exactly in the folingngg Fornat:
Verity: [brist explanation
correct
sertence]
t: Test Example
t+t Questior <e<questica>>>
#+t Ground truth solution <e<gt_selu:isp>>>
128 Your partial solution <<studont_colttioo>
Prompt Template 2.2.1 Critic corrects mistakes
General Guidelines
You are sbudens Your task is to carefully revies your own solusion to meth problen, and adhece to the following gu guidelinee inwss
Step [brief explanatioon 2, After this, suggest i'renaaivve thas should have taken correct the currect Incorress "Alternacivelyy (your suggested step with sentems)"
Directly point mistake with one sentemce)
First potentially
step you find explain your
3. You are provided with the ioestion, the ground truth solution, and your steg-by-step sclution.
alternative step you propose should include
like *ground trush solution".
5. Yous response should be weautly ly in the folluwing fornat:
Step <id>: (briet explaaatioo of the mietake in this with sentence
Allwenativelys [your suggested new step, with sentsnce]
#4 Text Reample
424 Question.
cocquestionss
t+ Ground truth solution <cegt
trt Your incorrect solution cocatudent solutionns
t+ Your reviev
30

---

Satori: RL with Chain-of-Action-Thoupe Enhances LLM Reasoning via Autoregressive Search
Prompt Template 2.2.2 Critic corrects mistakes for a second round
math cless. You coll inoracing with pertner to salve math problems. ems. First, your partaer provieed portial solution math problem sdeaciried mistake. Then, you continae your partner's partial solution and tinish solving the problen. New, your teas is to carefully zeviow your conzinucd solution, adbering to the tollewing quidelinas.
You student
"Your partace's partiol sclution" includes your partner's a partial solution to the problen. It migh: be long and contain several substeps. "Your contiaued solution" includes your completion ot the prebien based or your partner's partinl solution.
check ccrxectness "Your partner's portizi solution", incorrect step in "Your continued solution", and explain the reason why you made the mistake: "In step <is>t (briet explanation ot the mistake in your own solution with one sentersel*.
Your task
directly point cut tires rotentially
suggest auternative step you nave cocrest your (your suogested stop with ane seatence]"
step: "Altornativelyy
provided with question,
truth solution,
partner's partial solction,
ccatiaued solution.
The alternative stop you propocs SHOULD NOI explicitly include phrases 1ike "ground truth coletion", tas shown in the ground truth solution", etc.
only tocus your ow
NoT reter to any mistake your portner's partial
7. Your respcace should be exactly in the following fermat:
<id>: (briet explanation the mistake
solution sentence]
Aiternatively:
Iyour suggested now stop with and scatcnes]
Test Question
cesqustioo02>
t+t Ground truth solution <e9g__polu.io000
1:t Your partace's partial selution <cspattaer_solttionn
## Your oan inrded solution scsatudentsellttionn
Your iex
D.1.2. SYNTHETIC DATA EXAMPLES
Synthetic Data Example Type I: No reflection
μlnput]
Five couples plan take their boats trip down river. Bach boat can eazry 4 people. Whaz is the smallest nunber of beats souonnodate
couplex?
[Quiput]
First, we determine the total number people. Since there are five couples, ee, and each couple corstets of people,, the total number of people is:
Atines
continel/>
Bext, need to Figure out many bosts are required carry people. beat cen earry people, To the mininun number at boste, Ne divide the total numter people by the capacity ot each zoat:
sontinuus Since cannot have halt bost, xe need to round up to the next whole number, Theretore, xe need bosts to accomnodate all 10 people.
vericy, let's
people
among
Roat people
Beat 2: people
Boat people
A1l 10 people are accounted tor, and no bost exceess its capscity.
Thus, smallast number hoats that accomnedato all rive couples
1boxedi33
31

---

Satori: RL with Chain-of-Action-Thuugl Enhances LLM Reasoning via Autoregressive Search
Synthetic Data Example Type II-I: Verify correctness
a. library aas z total of 9508 books, each of which is categerized zs a biography, nevel, er seit-help book, The Hibrary's a patrons berrov 9305 books in one Neek, and the ratio ar norrowed biographies porzoved novels to borrowed selt-help books is 92:3:55. library
number of biocsaphies, noveis, sair-help bosks, hav many self-help books borsoved that week?
[Ouiput]
Pirat, ae detesnine mumber biographies, novela, and hhhlpp books Library. Since library equal nutbes of each type of book and the total nurber of books in 50, wa can divida 50 by 2:
Moxe (Nanber of biographies)
(Nurhes of novelal Mext (Namber of aelf_help books] (Ccwe(SC)
Howevws, sirow the number of books most be an integer and the problen states that the library has en eiuel number of each type of book, need Lo serevaluate the problen statenen... The problum aiates that the Library has an equal mumber of biographies,, novels, help books, biographies barrowed novels barrowed self books is (12:3153). This means the botal number books barzoved must be nulciple of (213315-00)) Sinow 30 books vare bocrowed, it is I indeed maltiple of 10.
of borrowed
<]ewclect]]>
let me stop here to verify, The de colllaaiion fon of the nomber of each type of book in the library is correct, but the Focus should be let the ratio of bocrowed books, which consiat ent with the given Lotal of 30 bocroved booka, < lonnt inse VI sepresent the namber biographien, novela, and self_help books barzoved respectively. According the giver ratio V1Rr3:5M we can writer XI b 2x 2x-1
MI
<I<lonnt inuel>
The Lotal musber books burcowed is 30:
AI b.bb + + e 30 V/<leantinue
Subat itute the expresxions for VC M. M Mr and VC into the equat Fanz
Ms) inuel?
Combine like terne:
AI 10x 30 {]<]epn:
Selve for V1 (frec/301110
cont
Pind the number of self_help books bor rowed: AI = 5x 5. Stimes 3 15 )J
Therefore, the nanber of sel feelpp books borroved that week \I 12oxedi151 3J
Synthetic Data Example Type II-II: Verify correctness then correct mistake
linput)
all pesitive integers digits, 325 sunbes of Uhis aut because "thsww Lwo Five" is valid Eaglish phaase. What is the lasges. memher Uhis sec that also a perfect square?
Consider
loft right, valid English word. For
[Qutpu]]
To Fiod the Largest menber of Lhe set of positive insegess whowe digita Torn valid English word and is also perfect to consides constrainte problen, nimber must pertect square ice digits most form valid Kng! fish word.
equace,
<Ireeleect
double check my sel seloulations step corrent ident the two main constraints problen: the number nust. bew feot aquaze and ita digits must fuom valid Engl ish word. Ljonniine Me start by checking the largest perfect squares in descending order uneil we find one whose digits form valid English word. ionontinuell
Lett' check VI.
9801 99"2 \)
The digita aew 9, 8, 0, 1. Wlin night sero one" is valid Bng! ish phrase.
9801 such number.
conditions
perfect eguare digits form valid English words, the largest
Therefore, the largest menoer of the set that ie also perfect aquare in: 3201) soriitt
But weit, The digits 9, 8, 0, do not forn a valio English vord because "zero" is not a valid word this context. clexpilores>
nethed, Me chould choek the largest perteet which is 961, and verity it its dipite torn walid Seglish word. continuel>
I will explore
let's check /r 961 V:
AL 961 - 31"2 \] The digits
9, 1. 'Bine six one" is valid English
Since V. 961 V) neets both conditione lit is pertect equare and its digits forn a valid English word), it is the largest auch number.
Theretore, the lazgest monser set that is persect square \boxedi 9611 N)
32

---

Satori: RL with Chain-of-Action-Thoue Enhances LLM Reasoning via Autoregressive Search
Synthetic Data Example Type III-I: Correct mistake
Given dn "x Amid -x" geggeeaant 01 and 9B Vx faid Aoc_/Verac (1) (2) Megeien 0 taen SAlccap am
exacde +iindty Aright)s
811ede (0, 1
81, 31eighht
D: 21
[Quipui]
inequell ityy
x Agey 0 31<jeontines inue 12
Factor out the common cerm:
x(-x 31 \geg i)clonnti
Deternins
stiiial puints setting Vimplisa continni
Aquad (eexcfor) 'quad
The critical points divide the real number Hiee into three intervals: VI (-Vinftyy 01 Mr X 10, 31 Mr and VI (s Minfty V. Be Leat point From each inteeval in the irsquality xt 1.31 Ageg 0 V).
For x Vi (-Vinity, M. choose 14 Vs AL t 114 1 31 - 12 2. Iquad (Stett Ageg 01 M
Vin
ML 11111
(1) Yquad Ogeg 0)
For VC Vin (3, Vinfty) V. choose
V)
lquad Ogeg 0) (mmtmmi
Since the inequelity is strict ('geg))), we include the aritios) pointe M x and VI V. Therefore, the solution lon to the inequelity is:
[Vfrr ingquelity Aleg M1leennt inuel>
Becall that tae base ot the logaritan in /1 waica is lees than 1. For logarithn wish base less taen 1, the ineguel (yylog Affrac11:21 Veeq 05I Tes:
Ageg
inequality incorrectly insludes critical 14 bocause V posstive natural numbers), find better solution, "he snlution to the ineguelity should be VI ViefeM" 2, Stright V since must natural aumber and VI 0 x Viegslant MM.c/continull Correct solution
overlocked comething. solution
'Nath tthe
poeitive natural nunees.<lexploee
VI considering Min "Nstabe(NI'' No
XL A 311, 2, JV zontinoll Set 11.3 31.3. renairs the same:
Vintty) zontinnne
Find the intereeation V Neep a Mr
11,, keap (1, Vattyl Va,
Therafore, the final ansvcz 15:
Al (zokediD)
33

---

3KM and Satori: RL with Chain-of-Action-TThnn Enhances LLM Reasoning via Autoregressive Search
Synthetic Data Example Type III-II: Correct mistake multiple times
In everienats ABCS, Saf, 9a5, Rc5 are the sidos appozite to the ançies 535, 9B6, 9C9 respectivaly. Given 95-45, 6iten 36, Plcos o- Veraet (5g 9, cind the area of Sitriangle ABCS,
[Ouiput]
Find Basin using Pythagoreen idemiity MAsin' \eos"2
Voow' '2. Alfft (5 (1s5 (rac((2) \frac(4115) Nain'2 C
Since VCW is an angle in a triangle, Vaain CV) is poxitive:
(5)
Next, Fird Caiin AI using the given Vsten 3V, Recell that Vrtan A Varaoilain Aldlons ATV. Tee Mixin Using Pythagorean identily CCainn2 Neos 133
MI
13k1 k72 Vimpliex 9272 k22 Vinplies 10k12 Vinglies k72 (1 Vinplina Vrrae (10g (113
Thus, Caii A 3t 3 Aedot (trac Atraci (10111:001111111 and Vloos A (trac(log:t/100 (10AA. usw haw of Sines find M
(Nain VErandelNNaia Nimpliea (Craclel 3Ngg (100)) [M vieplies Miracta) (\tract Regrt (10:11113 Vodot (2rac15312) 10 Airgl fee 10 Vodot ( 10: 11egrt \]
<)
Mait minate. celeulationn Vasin CV) is incorrect becruse Vaiin CV should be \/XFrac(2) (813) insteed ( (1022) is This time, Use the correct value of Vasin V (5gge subsequent exleulati <lcortirue ial for Moain CV)s
AI
Yein (Negett11) \
Use the correct value of Voain Cl) in the Tax of Sines Find Viel:
\fracdel//ain VfracietNNein Vimplies SVaee (10)7(1011 (A (Nage (58g) tecac(al//ersee (1erac (19: ledot (orre/ 21sgrz(5) Mippiies a 25 Vodot. (1)
deracrekegrtionn lstgrt/oc 1101 Atrectó Yodot 31egrt/2-H1111 H10) Sleget12) c|]ioonni
the tommla for asea txiangle (Area) (2))) A
\I (Aros) ( Veest (2) Vedae Vodet ( - (eraz:1):2) Vodot 12Nogez Vesot Vireci2) (5gget151) 1traeilH22 lodot (erae (1r00(122oor: AJ
1traei24kegrti22 (51)
Vegrat (211 (egtt 15get1531
(
Cheratore, the area of Mittriangic
Aioooed) 21seet (10) D....lleeeeee..e.. peleectct)
let's resssess my previous conclucton, The mistake is in the colculation ot the acea, where the value ot Visin BY should have been used iastead of Vivain CV) <ix_lerx rind better solution, (tra Yodoz (21
correct value Vosin
in the formul:: (Arex)
ledot Neraet logre 12:1 corract the calcatacion of the area using the correct value of (isin e (2)) Micooos Aseal (2) Nodot [2) Aodot Aedot (2)) 65).
Therafore, the area Nittriangl ABCV) (/
34

---

Satori: RL with Chain-of-Action -iitooo Enhances LLM Reasoning via Autoregressive Search
D.2. Format Tuning
Training Details. We perform supervised fine-tuning (SFT) on the Qwen-2.5-Math-7B base model using our syn- thetic COAT dataset with 10K unique questions. The SFT prompt template is illustrated below. We utilize co- sine learning rate scheduler with initial learning rate of 2e-5. The batch size is set to 128, the maximum se- quence length is 4096, and the model is trained for a maximum of two epochs. We add the following special tokens < <Jconting inue >,< Ief dreffect// | >,< explorel into the vocabulary. All experiments are implemented using the LLaMA-Factory framework (Zheng et al., 2024).
Prompt Template 3: SFT and RI.
< [lim_stares >user
Solve the following math problem efficiently and clearly. Please reason step by step, and put your final answerwithin \boxed <<<your instruction>>>
Problem:
< < sim_start|ll | Passistant
D.3. Reinforcement Learning
ORM Training. To construct the preference data for our ORM models, we utilize our format-tuned model, Satori-Qwen- 7B-FT, generate trajectories. Starting with our filtered training dataset of 550K unique questions, we follow these steps: (1) allow the FT model sample eight solutions for each question; (2) evaluate the correctness of these solutions and label them accordingly; and (3) select only those questions that contain correct and incorrect solutions. For these selected questions, we construct preference data by pairing correct solutions with their corresponding incorrect ones, resulting in preference dataset of approximately 300K unique questions.
For each problem æ, we allow Ty to randomly generate multiple reasoning trajectories, constructing a dataset D, with positive and negative pairs of trajectories. We select trajectories with the correct final answer as positive trajectories ÿ and trajectories with incorrect final answer negative trajectories ÿ Assuming the Bradley-Terry (BT) preference model, we optimize the reward model Tw (zr, 5) through negative log-likelihood,
Crm (+)
(e + g-> [log (ar (/, - (x, () +))
where r denotes a target reward margin. In practice, we observe that setting T >> O improves the performance of the reward model.
RL Data. Our RL training dataset consists of 300K unique questions from the preference dataset. This ensures that the questions are neither too easy (where the FT model always produces correct solutions) nor too difficult (where the FT model never succeeds). This encourages policy learn through trial and error during RL training. To further guide the model to learn self-reflection capabilities, we apply the proposed RAE technique, augmenting input problems with restart buffers, i.., intermediate reasoning steps collected from the FT model. These intermediate steps are extracted from the preference dataset, and for each question, we randomly select one correct and one incorrect trajectory, applying the back-track technique for up to = 2 steps.
Training Details. For both ORM and RL training, we implement our experiments using the OpenRLHF framework (Hu et al., 2024). For ORM training, we employ a cosine learning rate scheduler with an initial learning rate of 2e-6. The batch size is set to 128, the maximum sequence length to 4096, and the model is trained for two epochs. As the objective function, we use PairWiseLooss (Christiano al., 2017) with a margin of T = 2. For evaluation, we select the optimal ORM model checkpoint based on RM@8 performance, measured using the SFT model on held-out validation dataset. Specifically, we allow the FT model to sample eight trajectories and let ORM select the best trajectory according the highest reward score. The @ accuracy is then computed based on the selected trajectories.
For RL training, we use the PPO algorithm (Schulman et al., 2017a). The critic model is initialized from our ORM model, while the actor model is initialized from our FT model. We optimize the models using a cosine learning rate scheduler, setting the learning rate to 2c-7 for the actor model and 5e-6 for the critic model. During PPO training, we sample one trajectory per prompt. The training batch size set to 128, while the rollout batch size is 1024. Both the number of epochs
35

---

Satori: RL with Chain-of-Action-Thuuue Enhances LLM Reasoning via Autoregressive Search
and episodes set 1. The maximum sequence length for prompts and generations is fixed at 2048. Additional parameter settings include a KI. coefficient of 0.0, a sampling temperature of 0.6, and bonus scale of Tbamus 0.5.
Second-round Self-mmprovement. We begin with a set of 240K unique questions, also used in the distillation experiments shown in Table The policy of the first round of RL training serves as a teacher model to generate synthetic reasoning trajectories. Among these 240K questions and corresponding trajectories, we filter the data based on question difficulty, selecting the most challenging 180K samples for distillation. This process results in new fine-tuned (FT) model checkpoint, obtained from supervised fine-tuning (SFT) on these 180K trajectories. Since the new FT model has been trained on numerous high-quality trajectories, including reflection actions distilled from the teacher model, we do not apply restart and exploration (RAE) techniques in the second round of RL training to further encourage reflection. Additionally, we increase the sampling temperature from 0.6 to 1.2, generating eight samples per prompt to encourage more aggressive exploration to push the performance limit.
D.4. Evaluation Details.
For each model, we use the same zero-shot CoT prompt template to obtain results on all test datasets. For Satori and all its variants, we use Prompt Template 3 (Appendix D.2). We set the temperature to (greedy decoding) for every model, and collect pass@1 @1 accuracies. Details of each test dataset are as follows.
MATH500 (Lightman et al., 2023) is subset of MATH (Hendrycks et al., 202 a) of uniformly sampled 500 test problems. The distribution of difficulty levels and subjects in MATH500 was shown to be representative of the entire MATH test set.
GSM8K (Cobbe et al., 2021) is a math dataset that consists of 8.5K high-quality, linguistically diverse grade-school math word problems designed for multi-step reasoning (2 to 8 steps). Solutions involve elementary arithmetic operations and require no concepts beyond early algebra. Its test set contains 1319 unique problems.
Olyn (He et al., 2024) a bilingual, multimodal scientific benchmark with 8,476 Olympiad-leved math and physics problems, including those from the Chinese college entrance exam. We use the open-ended, text-only math competition subset, containing 674 problems in total.
AMC2023 and AIME2024 contain 40 text-only problems from American Mathematics Competitions 2023 and 30 text-only problems from American Invitational Mathematics Examination 2024, respectively.
FOLIO (Han et al., 2024) is a numan-annotater dataset designed to evaluate complex logical reasoning in natural language, featuring 1,430 unique conclusions paired with 487 sets of premises, all validated with first-order logic (FOL) annotations. Its test set contains 203 unique problems.
Boardgame (BGQA) (Kazemi et al., 2024) is a logical reasoning dataset designed to evaluate language models' ability to reason with contradictory information using defeasible reasoning, where conflicts are resolved based on source preferences (e.g., credibiliy or recency). Its test set contains 15K unique problems.
CRUXEval (Gu et al., 2024) is a benchmark for evaluating code reasoning, understanding, and execution, featuring 800 Python functions (3-13 lines) with input-output pairs for input and output prediction tasks. Given a function snippet and an input example, LLMS are tasked to generate the corresponding outputs. Its test set contains 800 unique problems.
Strategy QA (Geva et al., 2021) is a question-anssering benchmark designed for multi-hop reasoning where the necessary reasoning steps are implicit and must be inferred using a strategy. Each of the 2,780 examples includes a strategy question, its step-by-step decomposition, and supporting Wikipedia evidence.
TableBench (Wu et al., 2024a) is a tabular reasoning benchmark designed to evaluate LLMs on real-world tabular data challenges, covering 18 fields across four major TableQA categories: Fact checking, numerical reasoning, data analysis, and code generation for visualization. We test all models on fact checking and numerical reasoning subsets for simplicity of answer validation, resulting in 491 unique problems.
MMLUPrOSTEM is subset of MMLU-Pro (Wang et al., 2024b). MML.U-Pro is an enhanced benchmark designed to extend MMLU (Hendrycks et al., 202 by incorporating more reasoning focuseed questions, expanding answer choices from four to ten, and removing trivial or noisy items. We select six STEM subsets: physics, chemistry, computer science, engineering, biology, and economics (we remove the math subset as it belongs to in-domain tasks). Finally, we obtain 5371 1 unique problems in total.
36

---

Satori: RL with Chain-of-Action-Thouue Enhances LLM Reasoning via Autoregressive Search
E. Additional Results
E.I. Ablation on Reflection Bonus
Table 6: Ablation Study on Reflection Bonus.
Bonus Scale GSM8K MATH500 Olym. AMC2023 AIME2024
0.0
93.6
84.4
48.9
62.5
16.7
0.5 (default)
93.2
85.6
46.6
67.5
20.0
During RL training, introduce reflection bonus facilitate the policy learn self-reflection capabilities. The default value of the reflection bonus is set to Tretlect 0.5. To analyze its impact performance, also evaluate the model with the reflection bonus set to rrefteat 0. The results are presented in Table 6. We observe that the performance slightly degrades on challenging benchmark AMC2023 and AIME2024 when set Trefteet = 0 compared to Trefteet 0.5.
E.2. Ablation on Restart and Explore (RAE)
Table 7: Ablation Study on Restart and Explore (RAE).
Setting
GSM8K MATH500 Olym. AMC2023 AIME2024
No RAE
93.4
81.6
45.2
57.5
20.0
with RAE
93.2
85.6
46.6
67.5
20.0
Complementary to the reflection bonus, the RAE technique effectively encourages the model to self-reflect and improves the diversity initial states. We conduct an ablation study by removing the RAE technique in RL training. As shown in Table 7, the model without RAE exhibits performance degradation, confirming that RAE plays a critical role in our RL training.
F.3. A blation on Preference Bonus
Table 8: Ablation Study on Preference Bonus.
Reward
GSMSK MATH500 Olym. AMC2023 AIME2024
Only Rule-based
93.6
82.8
46.3
62.5
16.7
ORM Rule-based
93.2
85.6
67.5
20.0
To mitigate the sparse reward problem, incorporate preference bonus provided by Outcome Reward Model (ORM). To assess the impact of the preference bonus, perform ablation study by removing the ORM and relying solely on the rule-based reward and reflection bonus as the RL reward signal. The results in Table 8 demonstrate performance degradation, suggesting the effectiveness of using a hybrid reward signal that combines sparse rule-based rewards with dense ORM-based rewards.
E.4. Offline Restart Buffer v.s. Online Restart Buffer
Complementary to the reflection bonus, the restart buffer is designed to enhance the policy's self-reflection capabilities by augmenting the initial states with a diverse set of intermediate states. This includes trajectories processed from both correct and incorrect reasoning paths, which are then categorized into positive (Ditsaa)) and negative (Drestan) restart buffers, as described in Section 4.2.
In addition to constructing the restart buffer offline, also explore online restart buffer approach. Specifically, after each PPO episode, we use the updated policy to construct the restart buffer and collect rollouts from this buffer to optimize the policy, iteratively repeating this process. However, this approach is suboptimal. During PPO training, we observe that the majority of sampled trajectories are correct, leading to a significant imbalance between correct and incorrect intermediate states the online restart buffer. As a result, the model fail to adequately learn from incorrect paths, which are essential for incentivize self-reflection actions.
37

---

Satori: RL with Chain-of-Action-Thuughh Enhances LLM Reasoning via Autoregressive Search
To overcome this limitation, we opt for an offline restart buffer approach to mitigate the bias introduced by online collection.
Offline sampling ensures balanced inclusion of intermediate states from both correct and incorrect trajectories.
38


Reviews:
[{"review_id": "ucJV8EbfV9", "reviewer": "Reviewer_ZU1e", "summary": "In this paper, the authors proposes Satori as a framework for LLM reasoning. It is a two-stage framework, including Format Tuning and Self-improvement, to enhance LLM reasoning capabilities. The core contribution of this paper is the Chain-of-Action-Thought (COAT) framework, which structures LLM reasoning with meta-action tokens (
 and 
) to enable self-correction and exploration. The Self-Improvement stage utilizes RL, the Restart and Explore (RAE) algorithm, to efficiently train the model to use the COAT format effectively. Experiments on math and out-of-domain benchmarks demonstrate that Satori outperforms existing models, exhibiting strong generalization and test-time scaling behavior.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Ablations and analysis are limited: Some design choices (meta-actions, RAE parameters, reward function) could be more thoroughly justified and explored.**\n\nWe thank the reviewer for the suggestion. **We have conducted additional ablation studies to further analyze our design choices at** https://docs.google.com/document/d/e/2PACX-1vQ5lTnuQ5x6bx5Qh87cVdEq5iyrRIMLi5DYVHQUbrqB3f2Gye6mn0bHLwMcVosqddg_wp6P2JdEMfJM/pub.\n- Meta-actions: Please refer to rebuttal response 6.\n- RAE parameters: We clarify that RAE does not introduce any new tunable hyperparameters. However, to evaluate its impact, we perform an ablation study by training a model without applying RAE. The results show significant performance degradation, confirming that RAE is crucial for RL optimization.\n- Reward function: We have already ablated the reflection bonus in Appendix E. In addition, we include another ablation study that removes the preference bonus provided by the outcome reward model. The performance degrades in this setting, indicating that the preference bonus plays an important role in mitigating the sparse reward issues.\n\n**2. Analyze common failure modes and how COAT helps address them.**\n\nWe thank the reviewer for the suggestion. **We have conducted a failure mode analysis (see provided link above)**. Specifically, we find that COAT effectively identifies and mitigates five common failure modes in reasoning tasks: (1) overly complicated solution that misleads the problem-solving process; (2) numeric simplification and calculations error; (3) replacing variables with incorrect numerical values; (4) lack of comprehensive consideration; (5) wrong interpretation of the problem.\n\n**3. Benchmark scope is limited.**\n\nSince our training data consists of open-sourced math problems, it is reasonable to evaluate the model primarily on math-related benchmarks. However, we would like to emphasize that **our evaluation goes beyond the math domain with six OOD datasets logical reasoning, code reasoning, commonsense reasoning, tabular reasoning, and domain-specific reasoning**. This type of OOD evaluation is rarely explored in prior works. We would appreciate it if the reviewer could clarify in what sense the benchmark scope is considered limited, and what types of additional evaluation would be most helpful.\n\n**4. Evaluation only focuses on final answer accuracy (pass@1).**\n\nWe would like to note that zero-shot pass@1 is a widely adopted evaluation metric in LLM reasoning literature, provided that the ground-truth final answer is usually available. We agree with the reviewer that analyzing the intermediate reasoning process could provide valuable insights. However, this remains challenging due to the lack of reliable automatic verifiers capable of evaluating intermediate steps at scale. Toward a deeper understanding of our model’s behavior, **we have included a diverse set of demo examples in Appendix A that showcase different reasoning patterns of our model.**\n\n\n**5. Novelty is more in the combination than in fundamentally new concepts.**\n\nWhile the novelty is the eye of the beholder, we respectfully argue that the contributions of this work go beyond a simple combination of existing ideas. The reviewer describes our approach as a “novel combination of the COAT framework, RAE algorithm, and two-stage training for reasoning.” To the best of our knowledge, **none of these components have been introduced in prior LLM reasoning research**:\n* COAT differs substantially from classical CoT prompting in both design and motivations.\n* RAE offers a new perspective (by changing initial state distribution) to tackle the sparse reward problem in RL, a challenge has not been resolved by prior works.\n* Our two-stage training pipeline (small-scale Format Tuning + Online RL) also offers new insights for LLM post-training: SFT warm ups, RL improves.\n\n**6. Question-1: Have you tried to ablate 
 and 
 meta-actions separately? What if 
 is not used at all?**\n\nWe believe that 
 and 
 are fundamental meta-actions for reasoning tasks, and both are essential for effective problem-solving. Specifically, 
 prompts the model to evaluate its current reasoning, enabling it to identify potential errors or suboptimal steps, 
 allows the model to propose alternative solutions, which is a natural follow-up when the model realizes (via 
) that its current approach may be flawed.\n\n*** \n**If these clarifications satisfactorily address the reviewer's concerns, we kindly ask if the reviewer would consider updating the score to reflect what we believe is a paper with noteworthy contributions to the community.**", "claims_and_evidence": "Claims of superior performance and generalization are *partially* supported by empirical evidence.\n\n* Strengths: Strong performance on math and out-of-domain benchmarks compared to baselines. Ablation studies show the contribution of reflection bonus and RL. Qualitative examples illustrate COAT reasoning.\n* Weaknesses: Statistical significance is not explicitly discussed. Ablations are limited. Qualitative analysis is anecdotal. Benchmark scope is somewhat limited, and some benchmarks may be saturating. Lack of human-level performance comparison. Overall, while performance gains are shown, the strength and generality of the claims could be better supported with more rigorous and systematic evidence.", "methods_and_evaluation_criteria": "Proposed methods (COAT + RAE) and evaluation criteria (math and reasoning benchmarks) are generally sensible for the problem.\n* Strengths: COAT provides a structured approach to reasoning. RAE addresses RL challenges in reasoning. Benchmarks are relevant and challenging. Out-of-domain evaluation is a strong point.\n* Weaknesses: Evaluation primarily focuses on final answer accuracy (pass@1). Metrics that evaluate reasoning process quality would be valuable. Benchmark scope is somewhat limited.", "theoretical_claims": "This paper doesn't contain theoretical claims. No theoretical claims or proofs were checked.", "experimental_designs_or_analyses": "Experimental designs and analyses are generally sound, but could be strengthened.\n\n* Strengths: Ablation studies provide some insights. Multi-agent data synthesis framework is a creative approach.\n* Weaknesses: Ablations are limited in scope. Lack of statistical significance testing. Qualitative analysis is anecdotal. No error analysis of failure modes.", "relation_to_broader_scientific_literature": "* Relation is clear: The paper clearly relates to the broader literature on LLM reasoning, CoT prompting, test-time search, self-improvement, and RL for LLMs.\n* Specific relations: Builds upon CoT by adding meta-actions. Extends RL for reasoning by proposing RAE. Presents COAT as an alternative to data-intensive CoT fine-tuning and computationally expensive test-time search methods. Relates to SoS as concurrent work on training single LLMs for search, but argues for broader applicability of COAT.", "essential_references_not_discussed": "No essential references appear to be missing based on my current understanding of the literature.", "other_strengths_and_weaknesses": "Strengths:\n* Originality: Novel combination of COAT framework, RAE algorithm, and two-stage training for reasoning.\n* Significance: Addresses an important problem (enhancing LLM reasoning) with a practical and efficient approach. Demonstrates promising empirical results and generalization.\n* Clarity: Paper is generally well-written and easy to understand, especially with figures and examples.\n\nWeaknesses:\n* Incremental Novelty: Novelty is more in the combination than in fundamentally new concepts.\n* Limited Rigor in Evaluation: Evidence could be strengthened with more systematic evaluations, statistical significance testing, and more detailed ablations.\n* Justification of Design Choices: Some design choices (meta-actions, RAE parameters, reward function) could be more thoroughly justified and explored.", "other_comments_or_suggestions": "* Consider adding error analysis to understand common failure modes and how COAT helps address them.\n* Minor comment: In Figure 1, \"Trajectories Generation\" could be rephrased to \"Initial Trajectories Generation\" for clarity.", "questions_for_authors": "1. Have you tried to ablate 
 and 
 meta-actions separately? What if 
 is not used at all?\n2. Have you observed how COAT helps to mitigate the common failure modes? Which failure modes are best mitigated by COAT?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lC8KFYxRMy", "reviewer": "Reviewer_wWt8", "summary": "This work studies the problem of post-training LLMs for self-reflection and self-exploration capabilities. A training scheme termed COAT is proposed, which consists of two stages: (1) a small-scale SFT to initiate COAT reasoning format; (2) a large-scale RL finetuning stage to further enhance the self-reflection/exploration capabilities. For the SFT phase, a generator-critic-reward multi-agent data synthesis framework is used to construct high-quality demonstration trajectories. For the RL phase, a restart and exploration strategy is adopted, letting LLMs to reflect starting intermediate steps, and reward is carefully designed to have both rule-based correctness reward, reflection bonuses, and additional preference bonus. Extensive results demonstrate the effectiveness of COAT in reasoning tasks, in particular, the improvement over base models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Compare with more naive approaches (e.g., the ones adopted in DeepSeek-R1 with only RL finetuning).**\n\nWe appreciate the reviewers' interest in the comparison between our method and the RL-only approach (r1-zero). As noted in our response to reviewer WTyn, r1 is a concurrent work, and such comparisons are not expected according to ICML reviewer guidelines. Nonetheless, we are happy to provide this comparison and share the observations and insights we have gained, in the spirit of contributing to the broader research community. \n\nThe results and discussions can be found at https://docs.google.com/document/d/e/2PACX-1vRk1wLF9fsEVPFP2ijxNUd82LPnWZ2w-zTxVNGkkrof9yy36BNrM47JiCr6r8bga9c2Sr3q0a-S0oFF/pub\n\n* **Apple to apple performance comparison of r1-zero and Satori:** \n\nIn order To fairly compare our method with r1-zero, we keep all experimental settings consistent, including the base model, training data, and RL training framework. The only differences lie in the algorithms used, our method employs PPO with RAE, while r1-zero uses GRPO, and whether an additional SFT stage is incorporated (ours includes Format Tuning and r1-zero starts RL training from the base model). **We evaluated r1-Zero on the same in-domain and out-of-domain benchmarks, and the results show no advantage over our method**. This suggests that **the perceived strong performance of the released DeepSeek-r1 model is likely due to significant engineering investment**, such as larger-scale in-house data, a much larger base model, and more powerful infrastructure. Moreover, **we observed several challenges and unexpected behavior of training r1-zero**, which we detail below.\n\n* **Training of r1-zero could be unstable:** \n\nWe found that training r1-zero can be quite unstable. For instance, **rewards may drop sharply in later training stages, and the model sometimes begins generating repetitive random sequences**. This instability is especially common when the KL penalty is set to zero (i.e., no regularization). We hypothesize that without an SFT stage to anchor the model's behavior, the base model may produce harmful outputs early on, which can lead to a vicious cycle during optimization.\n\n* **R1-zero's behavior is hard to control:** \n\nEven when training converges, R1-Zero exhibits undesirable behaviors at inference time. Specifically, we observed two representative issues: (1) **Repetitive responses**, where the model generates repeated tokens or sentences mid-generation. (2) **Language mixing**, where responses may unexpectedly contain characters from other languages (e.g., Chinese), a phenomenon also noted by the r1 authors. These findings suggest that pure RL fine-tuning lacks model behavior constraints. While more carefully designed reward functions might help, we argue that a simpler and more effective solution is to incorporate a SFT stage, as we did with Format Tuning, to stabilize model’s initial behavior.\n\n* **R1-zero shows some \"reflection pattern\", especially using Python code verification:** \n\nWe observe that r1-zero indeed demonstrates some reflection behavior after RL training, but we found **it tends to use Python code to verify answers, even when this is unnecessary**. For example, in a multiple-choice question from the MMLU-Pro dataset (which may not require logical verification), the model attempts to verify the answer using Python, which is a redundant step in that context. This behavior suggests that without appropriate constraints, **cold-start RL can introduce unexpected patterns, beyond the desired problem-solving skills**. Moreover, a recent study has also questioned the validity of the \"Aha moment” claimed by r1-zero and suggests “Aha moment” could be a mirage [1].\n\n* **Our final takeaway: SFT warm ups, RL improves.**\n\nOur view is that **neither pure SFT nor pure RL (as in r1-Zero) is sufficient for effective LLM post-training**. Pure SFT can overly constrain the model and limit its ability to generalize beyond demonstrations. Pure RL can lead to unexpected and unreasonable behaviors, as observed with r1-zero. Instead, **we advocate for an appropriate combination of both**. This is the key insight we propose: a small-scale warm-up SFT helps the base model to get familiar with certain reasoning patterns, and a large-scale RL helps the model to utilize such reasoning patterns to incentivize its actual reasoning capability. This idea is also consistent with practices in the classical RL literature. For example, [2] demonstrates that a small amount of demonstration data can accelerate RL training by improving policy initialization. Moreover, the released DeepSeek-r1 model (as opposed to r1-Zero) also incorporates a SFT stage before RL (though the details of such SFT stage are not disclosed), which aligns with our insight.\n\n**Reference**\n\n- [1] There May Not be Aha Moment in R1-Zero-like Training — A Pilot Study, Feb, 2025.\n- [2] Deep Q-learning from Demonstrations, AAAI, 2018.", "claims_and_evidence": "Yes, the claims in this work are made cautiously with supporting evidences.", "methods_and_evaluation_criteria": "Yes, the proposed method is intuitively reasonable, a SFT phase to get initialized and a RL phase to further boost the performance. The evaluation is also comprehensive with in-domain and out-of-domain tasks with many other metrics presented.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "I have checked the results provided in the main paper and the results are convincing.", "relation_to_broader_scientific_literature": "The study of reasoning capability of LLMs are important to promote their usage in broader scientific domain, and I believe this work contribute to this field.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The paper is written very clearly, and it has been a pleasure to read it. The idea is nicely presented and the effectiveness is demonstrated with comprehensive experiments.", "other_comments_or_suggestions": "NA.", "questions_for_authors": "One question that I would love to hear the author's opinion on: while this work considers a \"more developed\" framework to enhance models' self-improvement capability (e.g., use a SFT phase first to get COAT format and a large-scale RL phase), is it possible to compare with more naive approaches (e.g., the ones adopted in DeepSeek-R1 with only RL finetuning)? In particular, I would love to see both the performance comparison and the self-improvement pattern comparison.\n\nI might have missed similar results in the paper. If that is the case, please provide a pointer.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N2O3VLjdKr", "reviewer": "Reviewer_WTyn", "summary": "The paper presents a method called Satori, which could enhance the reasoning abilities of LLMs. It does this through Chain-of-Action-Thought (COAT), a system that adds special “meta-action” tokens (like ``
`` and ``
``) to regular chain-of-thought prompts. These tokens let the model pause to check its work or try a different approach on its own. The authors train the model in two main stages: first, they fine-tune it on a small set of example trajectories to teach it how to use the new tokens. Second, they apply RL at a much larger scale, which allows the model to generate new solutions, find and fix its mistakes, and gradually improve how well it reasons.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. R1 uses a cold-start RL to develop reflection skills on its own (the “Aha moment”), but the paper relies on a specialized dataset to teach reflection-like abilities.** \n\nWe would like to respectfully clarify several important points:\n\n- **R1 is a concurrent work and comparison should not be required:** r1 was officially released around the same time as the ICML 2025 abstract deadline (end of January). **According to the reviewer guidelines, we are not required to compare with or even mention r1.** However, we chose to acknowledge r1 as a concurrent work because we believe it offers valuable context for the research community amidst the recent interest it has generated.\n- **Clarifying potential misunderstanding:** We respectfully suggest that the reviewer may have some confusion regarding r1. (1) Deepseek introduces two models: r1-zero, which is trained using pure RL from a base model, and r1, the actual released model. R1 includes a SFT stage before RL, where the SFT data is explicitly designed with a reflection pattern (see Section 2.3.1 in [1]). Thus, **r1 employs a similar strategy to our format-tuning stage, although its implementation details are not disclosed**. (2) **Rather than limiting capabilities, this SFT stage is helpful to achieve more effective RL**. Indeed, the r1 authors acknowledge the limitations of r1-zero without SFT (see Section 2.2.4 in [1]), including poor readability and language mixing issues. Furthermore, recent work [2] supports the effectiveness of incorporating SFT before RL. Several studies have also questioned the validity of the “Aha moment” claimed by r1-Zero and suggest “Aha moment” could be a mirage (see [3]), which challenges the reviewer's claim that \"r1 can develop reflection skills on its own\". \n- **Comparison results provided:** Please refer to [**our response to reviewer wWt8**](https://openreview.net/forum?id=j4FXxMiDjL&noteId=lC8KFYxRMy).\n\n**2. The performance still lags behind o1 and open-sourced r1 model.**\n- **Unfair comparison:** We respectfully argue that it is not reasonable to compare the performance of a research prototype with that of large-scale industry systems. R1 contains over 600B parameters, whereas our model is based on a 7B LLM. Furthermore, the scale of training data and infra used by o1 and r1 is not publicly disclosed but is far beyond what is accessible in a academic setting.\n- **R1 is not truly open-sourced:** While o1 is certainly closed-source, we would also like to clarify that r1 is not fully open-sourced. Only the model weights have been released. The training data, codebase, and detailed pipeline remain proprietary. Over the past two months, several efforts have attempted to reproduce r1’s performance, but few—if any—have succeeded in matching it, which further suggests that important implementation details are missing.\n- **Research Objective:** o1 and r1 are undoubtedly successful, but their performance is largely attributed to extensive resources. In contrast, the goal of academic research is not to achieve SOTA benchmark results, but rather to explore novel methodologies orthogonal to scaling up resources. For instance, our proposed RAE technique effectively mitigates the sparse reward issue in RL, which remains a major challenge for complex reasoning tasks and is under-explored in prior works. Moreover, RAE could potentially be integrated into r1’s training framework as well. We respectfully note that if benchmark performance were the sole criterion for evaluation, then almost no academic work (including LLM reasoning papers submitted to ICML 2025) could surpass current industry-developed models such as r1 or o1.\n\n**3. How does your approach mitigate the risk of critic model hacking?**\n- **Request for clarification**: In RL, the term “hacking” typically refers to reward hacking, where the model exploits flaws in a reward model to receive undeservedly high scores. By contrast, the critic model in PPO is used for estimating advantages via GAE, and does not directly influence the reward signal. We kindly ask the reviewer to clarify whether the concern is regarding to reward model or PPO’s critic model.\n- **Our approach**: Satori utilizes a hybrid reward mechanism with both a reward model and rule-based reward. (see Section 4.2). While it is possible that the reward model may become less reliable during later stages of training, our design ensures that the rule-based reward remains dominant, mitigating the risk of reward hacking.\n\n**Reference**\n- [1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, Jan, 2025.\n- [2] Demystifying Long Chain-of-Thought Reasoning in LLMs, Feb 2025.\n- [3] There May Not be Aha Moment in R1-Zero-like Training — A Pilot Study, Feb, 2025.\n\n***\n**If these clarifications satisfactorily address the reviewer's concerns, we kindly ask if the reviewer would consider updating the score to reflect what we believe is a paper with noteworthy contributions to the community.**", "claims_and_evidence": "Yes. It is clear and convincing", "methods_and_evaluation_criteria": "Yes. It makes sense.", "theoretical_claims": "They make no theoretical claims", "experimental_designs_or_analyses": "They evaluate performance on datasets including GSM8K, MATH, and several out-of-domain tasks. Their experiments validate their claim that Satori can both solve challenging math problems and generalize to new domains.", "relation_to_broader_scientific_literature": "I understand the authors mentioned r1 model in their paper. However, I still want to mention the following fact:\n\nBecause o1 and o3 models are not open-sourced, it is unclear how OpenAI trained them. However, DeepSeek’s r1 shows performance comparable to o1 and reportedly uses a cold-start reinforcement learning approach, which seems to let the language model develop reflection skills on its own (the “Aha moment”). In contrast, the paper in question relies on a specialized dataset to teach reflection-like abilities. Compared with r1’s algorithm, this approach has several constraints: (1) it requires format tuning prior to self-improvement, limiting its flexibility to learn other skills, and (2) this format tuning could negatively affect the model’s existing capabilities. Moreover, their final results still lag behind o1, suggesting that while their method works, it may be somewhat behind the latest approaches.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Their method is undeniably powerful, and they provide strong evidence for its effectiveness. However, it remains unclear whether this approach can surpass the latest open-sourced r1 model.", "other_comments_or_suggestions": "No other comments", "questions_for_authors": "In many RL settings, methods that rely heavily on pre-trained or trainable critic models can be easily hacked by LMs. How does your approach mitigate these risks? Could you explain any strategies you employ to ensure the policy cannot simply exploit the critic’s learned features?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e50pCG6L6r", "reviewer": "Reviewer_X1zu", "summary": "This paper propose a post-training framework, incluing two stages: (1)  the chain-of-Action-thought fine-tuning, i.e., format fine-tuning; (2) self-improvement RL, i.e., iterative distillation and RL. For first stage, using Qwen-2.5-Math-Instruct and Llama-3.1-70B-Instruct constructs a multi-agent data synthesis framework. For second stage, incorpating rule-based  reward, reflection Bonuses and preference bonuses into final reward. Authors evaluate their resultsing model in GSM8K, MATH500, AMC2023, AIME2024, and OlympiadBench. They achieve comparable performance with other LLMs.\n\n## Update\n\nThanks for detail reponse.  I summary my concern as follow:\n1. Overclaim of performance. As shown in table 1, the propose method did not outperform than baselines. Since this paper have no theoretical analysis, authors should highlight the up/down and numbers comparing to baselines.\n2. Need more statics on augmented data. The new format of RL data is the core contribution. For more better understanding, authors should provide ablation on data.\n3. This paper is more like a technical report than an academic paper, including data format, integrated reward and so on. Authors should provide more insights for academic community.\nFinally, i will keep my socre.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Over-claim about \"without external guidance\".**\n\nWe would like to clarify that **“external guidance” specifically refers to guidance provided by another LLM verifier at inference time (see Abstract, lines 18-19)**. Many existing LLM reasoning methods rely on extensive sampling and guidance from a verifier model (e.g., PRM-guided tree search). Regarding the cost of the multi-agent system, we believe this is analogous to the inevitable cost of curating most LLM training datasets. We will clarify these points in the revision.\n\n**2. Over-claim about \"superior performance\".**\n\nWe would like to highlight that the **performance gains on challenging MATH benchmarks are substantial**. For example, we achieved +5% on OlympiadBench, +10% on AMC2023, and +3.3% on AIME2024 compared to the best available 7B models. Moreover, **on our out-of-domain generalization experiments**, our method consistently outperforms baseline models **with an average improvement of 8.3% (see Table 2)**. While we acknowledge that the paper review may carry some degree of subjectivity, we would also like to note that **all three other reviewers appreciate the strong empirical performance**. However, we would like to tone down the performance claims in the revision.\n\n**3. No theoretical results.**\n\nWe acknowledge the reviewer’s concern regarding the lack of theoretical analysis, as our proposed training framework currently does not come with a rigorous theoretical justification. However, we would like to note that conducting a comprehensive theoretical analysis in the context of LLM, especially for complex RL training pipelines is highly non-trivial and beyond the scope of this work. We also agree that some components, particularly the proposed Restart and Explore (RAE) technique, merit further investigation. We view this as an important direction for future work and include some preliminary theoretical insights about RAE as a starting point for such analysis.\n\n\nRAE modifies the initial state distribution by starting new rollouts not only from dataset-sampled prompts but also from random partial trajectories, i.e., prompts concatenated with partial responses generated by intermediate policies. This design can be justified by Theorem 6.2 of the Conservative Policy Iteration (CPI) paper [1], It states that if $\\pi$ is an approximately optimal policy learned from one initial distribution $\\mu$, and $\\pi^*$ is the true optimal policy for a different initial distribution $\\tilde{\\mu}$, then\n\n$$\n\\eta_{\\tilde{\\mu}}(\\pi^*) - \\eta_{\\tilde{\\mu}}(\\pi) \n\\ \\le \\ \n\\frac{\\epsilon}{(1-\\gamma)^2}\n\\left\\|\\frac{d_{\\pi^*, \\tilde{\\mu}}}{\\mu}\\right\\|_\\infty,\n$$\n\nwhere $d_{\\pi^*, \\tilde{\\mu}}$ is the stationary state distribution of $\\pi^*$ under $\\tilde{\\mu}$, $\\gamma$ is the discount factor, and $\\epsilon$ is small approximation error. The concentrability coefficient $\\bigl\\|\\frac{d_{\\pi^*, \\tilde{\\mu}}}{\\mu}\\bigr\\|_\\infty$ measures how badly $\\mu$ misses states that $\\pi^*$ visits. If $\\mu$ puts little or no mass on those important states, the policy learned from $\\mu$ can perform poorly under $\\tilde{\\mu}$.\n\n\nIn the context of LLMs, the \"initial state\" corresponds to the prompt prefix. Training with a narrow or fixed set of prompts results in a limited $\\mu$, which may place little to no mass on the key states an optimal policy should visit, thereby inflating the concentrability coefficient. RAE mitigates this issue by broadening $\\mu$ through randomized partial rollouts, effectively augmenting the initial state distribution with more diverse and relevant prefixes encountered during training. While we do not have access to the true stationary distribution $d_{\\pi^*, \\tilde{\\mu}}$, this augmentation strategy reduces distribution mismatch and narrows the theoretical performance gap between $\\pi$ and $\\pi^*$. This aligns with the conclusion in [1], which argues that more uniform state coverage leads to tighter bounds.\n\nWe plan to explore a more formal and comprehensive analysis of RAE in future work.\n\n\n**4. Question-1: If you perform SFT using only your synthesized data in stage 1, would the performance differ from the current results?**\n\n**We kindly refer the reviewer to the “Large-scale FT vs. Large-scale RL” ablation study in Section 6**, where we have demonstrated that performing SFT using only the large-scale synthesized data in Stage 1 is sub-optimal compared to large-scale RL training.\n\n**5. Question-2: Is there an R1-style comparison?** \n\n Please refer to [**our response to reviewer wWt8**](https://openreview.net/forum?id=j4FXxMiDjL&noteId=lC8KFYxRMy) for details.\n \n**Reference**\n\n- [1] Approximately optimal approximate reinforcement learning. ICML, 2002.\n\n***\n\n**If these clarifications satisfactorily address the reviewer's concerns, we kindly ask if the reviewer would consider updating the score to reflect what we believe is a paper with noteworthy contributions to the community.**", "claims_and_evidence": "There are some over-claims:\n1. \"without external guidance\" -> this paper adopt a a multi-agent system in data construction, including Qwen-2.5-Math-Instruct and Llama-3.1-70B-Instruct. If author want to illustrate the efficiency, I suggest they provide more evidence about cost, FLOPS or other metric of efficiency.\n2. \"superior performance\" -> GSM8K: 93.9 (theirs) vs  95.2/91.6 (Qwen /w math). The same situation applies to MATH500. And, they also achieve 4 point improvement on average of OlympiadBench.", "methods_and_evaluation_criteria": "Yes, this paper use reasonable metric for evaluation.", "theoretical_claims": "This paper does not provide any theoretical results or insights.", "experimental_designs_or_analyses": "Yes, this paper's experimental designs are reasonable.", "relation_to_broader_scientific_literature": "The key difference between this paper and current literature is the fine-tuning data format and reward design in RL. They incorporate all suitable formats for self-improving fine-tuning. And for reward designs, they propose reflection bonuses and preference bonuses.", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "The main concerns of this paper are :\n1. there are not theoretical results.\n2. the complex and diverse training system design ultimately only achieves comparable performance, and the results presented in the paper do not support the claims made in the introduction.", "other_comments_or_suggestions": "If the paper includes more analysis on reinforcement learning theory, it would provide deeper insights.", "questions_for_authors": "- If you perform SFT using only your synthesized data in stage 1, would the performance differ from the current results?\n- And is there an R1-style comparison?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this work, the authors propose Satori, an LLM post training method which leverages Chain-of-Action-Thought (COAT) scheme to teach LLMs do reasoning. The overall pipeline has two key steps: 1) a small-scale SFT to teach LLM the COAT format; and 2) a larger-scale RL fine-tuning to teach the LLM self-reflection and exploration behaviors, via COAT reasoning format. The authors show that smaller models (e.g., Qwen 7b) trained with Satori can perform on par or exceed much larger LLM on a set of in distribution and out of domain benchmarks. 

Reviewers agree that the multi-agent data synthesis framework is a neat idea; adding structure into reasoning paths (COAT), together with the Restart and Explore (RAE) algorithm, indeed seems to bring self-reflection and exploration capabilities to the model; the experiment results are strong, the out-of-domain results are especially promising. 

Apart from a few clarification questions and asking for some additional ablation studies (which the authors did a good job addressing), a common concern raised by the reviewers is regarding the comparison with the DeepSeek R1 model (including R1-zero). This is mainly because Satori also aims to teach LLM models to do reasoning (and probably also involves some format training).

I am well aware that the R1 work was released one week before the ICML deadline (and one day before the abstract deadline), so I do not think it is necessary to compare with a concurrent work. I noticed both the authors and the reviewers that R1 is a concurrent work and thus I will not take such negative reviews into consideration. 

During the rebuttal period, the authors nonetheless managed to conduct some analyses comparing their approach with R1/R1-zero, which is impressive and much appreciated. The additional analyses actually provided some interesting insights on the necessity of properly combine SFT and RL fine-tuning. Also given the fact that the authors promise to fully open-source Satori (code, data, and model), I feel this work can benefit the ICML community by providing a very interesting but different perspective on training reasoning models. Despite the majority of Satori's experiments only used a 7b base model, its strong performance gives me confidence that a similar approach could also work on a larger model. 

After the rebuttal, one of the reviewers increased their score from 2 to 3, saying the framework is indeed a promising direction for future LLM training. 

I recommend to accept this paper. Although the average score is not outstanding, if we ignored negative comments regarding R1 comparison, the score could be higher.
Decision: Accept (poster)