Thinker: Learning to Think Fast and Slow
Authors: Stephen Chung
Wenyu Du
Jie Fu
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Reasoning
Large Language Models
Dual Process Theory
Primary area: reinforcement_learning
TL;DR: We propose a four-stage training task for LLMs—Fast Thinking, Verification, Slow Thinking, and Summarization—separating the training of intuition, verification, reasoning, and integration.

Abstract: Recent studies show that the reasoning capabilities of Large Language Models (LLMs) can be improved by applying Reinforcement Learning (RL) to question-answering (QA) tasks in areas such as math and coding. With a long context length, LLMs may learn to perform search, as indicated by the self-correction behavior observed in DeepSeek R1. However, this search behavior is often imprecise and lacks confidence, resulting in long, redundant responses and highlighting deficiencies in intuition and verification. Inspired by the Dual Process Theory in psychology, we introduce a simple modification to the QA task that includes four stages: Fast Thinking, where the LLM must answer within a strict token budget; Verification, where the model evaluates its initial response; Slow Thinking, where it refines the initial response with more deliberation; and Summarization, where it distills the refinement from the previous stage into precise steps. Our proposed task improves average accuracy from 25.6% to 27.3% for Qwen2.5-1.5B, and from 45.9% to 51.0% for DeepSeek-R1-Qwen-1.5B. Notably, for Qwen2.5-1.5B, the Fast Thinking mode alone achieves 25.2% accuracy using fewer than 1000 tokens, demonstrating substantial inference efficiency gains. These findings suggest that intuition and deliberative reasoning are distinct, complementary systems benefiting from targeted training. Additionally, we have open-sourced both the trained models and the source code.

Thinker: Learning to Think Fast and Slow
Stephen Chung"
Wenyu Du The University Hong Kong
Jie Fu Shanghai Al Lab
University of Cambridge
Abstract
Recent studies show that the reasoning capabilities of Large Language Models (LLMs) can be improved by applying Reinforcement Learning (RL) to question- answering (QA) tasks in areas such as math and coding. With a long context length, LLMs may learn to perform search, as indicated by the self-correction behavior observed DeepSeek RI. However, this search behavior and lacks confidence, resulting in long, redundant responses and highlighting deficiencies intuition and verification. Inspired by the Dual Process Theory psychology, we introduce simple modification to the QA task that includes four stages: Fast Thinking, where the LLM must answer within a strict token budget; Verification, where the model evaluates its initial response; Slow Thinking, where it refines the initial response with more deliberation; and Summarization, where it distills the refinement from the previous stage into precise steps. Our proposed task improves average accuracy from 25.6% to 27.3% for Qwen2.5-1.5B. and from 45.9% 51.0% for DeepSeek-RI. R1-ee---R------------ 1.5B. Notably, for Qwen2.5-1.5B, the Fast Thinking mode alone achieves 25.2% accuracy using fewer than 1000 tokens, demonstrating substantial inference efficiency gains. These findings suggest that intuition and deliberative reasoning are distinct, complementary systems benefiting from targeted training. Additionally, have open-sourced both the trained models and the source code.
often imprecise
Introduction
Multiple studies have shown that the reasoning capabilities of Large Language Models (LL.Ms) can be enhanced by applying Reinforcement Learning (RI.) guestion-answering (QA) tasks 0 3.
demonstrating impressive mathematical and coding performance across benchmarks. With long context lengths, an interesting emergent behavior is self-correction within the chain-of-thought (CoT), where the LLM learns to perform search, such as verifying its steps, backtracking, and trying alternative paths.
However, it has been observed that this emergent search tends to be inefficient the CoT is often long and redundant []. For example, Deepseek RI's reasoning typically involves excessive backtracking and verification [I] A likely cause is inefficient temporal credit assignment: for instance, in the GRPO algorithm used to train Deepseek RI, the entire generation sequence receives the same scalar advantage. That is, if the final answer is correct, the probability of the whole sequence increased-regardless of which parts were actually useful. As a result, futile search paths and uncertain verifications are also rewarded, long the correct solution is eventually produced. Consequently, intuition-the ability to identify promising search paths rapidly- verification- the ability to evaluate search path confidently therefore be underdeveloped.
not explicitly trained and may
A typical RL solution to this issue is to use more precise temporal credit assignment, such as incorporating a critic to compute a more accurate advantage for each token, as PPO ( However,
Equual contribution. Correspondence to: mhc480cam.cm. uk.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

indicating that the critic may not be accurate enough for token-level credit assignment. Another approach is to use lower discount rate or a shorter context length to encourage more efficient search; however, this may hinder the emergence of search behavior, as studies show that a long context length necessary for strong performance [8, [.
studies show that PPO performs similarly to GRPO 7 2
To address this dilemma, we draw inspiration from how hu- man decision-making is modeled under Dual Process Theory According to this theory, humans possess two distinct but complementary cognitive systems: System 1, which op- erates quickly and intuitively based on heuristics but prone to biases, and System 2, which is slower, more deliberate, and capable of reflective reasoning. Within this framework, a typical decision-making process unfolds as follows:
Propose Initial Rapidly
System Fast Thinking
System Stow Thinking
1. System I rapidly generates a candidate option based on intuition.
Verify and Refine Solution Carefully
2. System 2 evaluates this option through mental sim- ulation.
Figure Conceptual model of the in- teraction between Fast Thinking and Slow Thinking modes in the Thinker task, based on Dual Process Theory.
3. If the option passes verification, is implemented: otherwise, System 2 attempts to refine it.
4. If refinement fails, the process returns to System 1 for another option.
Inspired by this cetision-making process, we propose the Thinker task an alternative to the standard QA task. In a typical QA task, the model receives a question and generates final answer in a single pass. A binary reward given based solely on the correctness of the final answer. In contrast, the Thinker task decomposes the response into a four-step process:
1. Fast Thinking: The agent generates an initial answer using a small token budget.
2. Verification:
The agent evaluates the correctness of the initial answer using a small token budget. If verified, it is accepted as the final answer.
3. Slow Thinking: If the initial answer fails verification, the agent can produce another final answer, using a large token budget.
4. Summarization: The agent summarizes the reasoning from the slow thinking step into a concise summary that leads to the same final answer.
We design distinct reward signals for each step, aiming to enhance different capabilities of the agent: intuition from Fast Thinking, evaluation from Verification, refinement from Slow Thinking, and integration from Summarization. Crucially, the reward signal for each task is restricted that task alone. This separation allows for more precise temporal credit assignment by isolating learning signals for each task. For example, in the Fast Thinking task, the agent receives a binary reward based on the correctness of the initial answer, encouraging it to identify promising search paths under strict token budgets- sudgets-thereby strengthening intuition. Meanwhile, the Slow Thinking task preserves the opportunity for the agent to learn more general search strategy to refine previously incorrect answers.
The design facilitates virtuous loop between intuition and reasoning. Fast Thinking helps Slow Thinking by providing a promising initial search path, while Slow Thinking helps Fast Thinking by refining flawed intuition. This bidirectional refinement mirrors how expert human decision-making evolves through repeated interactions between intuition in System and reasoning in System 2 (Н].
Experimental results validate our approach: relative to the QA task, the Thinker task yields consistent gains across diverse math benchmarks, with average relative performance gains of 6.7% for Qwen2.5- 1.5B models and 11.1% for DeepSeek-RP Ce-Seee--R-isttiiii L-Distill--weee Saall--ewe---5BB 1.5B models. Furthermore, our analysis reveals a notable reduction in reflection patterns, suggesting more direct reasoning. In summary, the proposed Thinker task offers the following key strengths:
Specialized Training: Dedicated sub-tasks and rewards are designed to explicitly train distinct agent capabilities, providing richer and more targeted learning signals.

---

General Applicability: The Thinker task can replace standard QA tasks without imposing constraints on the choice of RL algorithm or model architecture.
Inference Efficiency:: The Fast Thinking mode, requiring minimal token generation, can be deployed standalone for simpler tasks, offering flexible trade-off between performance and computational cost during inference.
Strong Empirical Performance: Our experiments demonstrate that agents trained with the Thinker task consistently outperform those trained standard QA tasks across various benchmarks.
2 Background
In single-turn QA task. question is sampled from a dataset, and the LLM generates response the question. Concretely, let the dataset be denoted as ((()) ) where z()) denotes the i-th question, Vig is its corresponding ground-trutt truth answer, and is the size the dataset. Let
Te(- 2) denote the model's 's policy, parameterized by 0. A response a ~ To(- xx) is sampled for question æ. The objective is maximize:
J(0) = -n.y R(a, y²)],
(1)
where a Te(. r), and is the reward function, such as a binary function that returns the extracted answer from matches the ground-truth answer y", and 0 otherwise.
In more general multi-turn task, allow the dialogue to continue after the first response. Concretely, we denote 2₁ and a as the prompt and model response at turn t. The initial prompt is randomly sampled from the dataset D. To generate the subsequent prompt Zt, we define the transition function I = [[ Q{-1) (with Taib -icpuiitnicllssive,,, which determines the next prompt based on previous prompts and responses, or whether to terminate the episode. Thus, the objective in the multi-turn task becomes:
T
Re Ra 2r)
((0) = B
Σ
(2)
where a₂ To(- Zozt a0.--1), 1). that is, the response is conditioned all previous prompts and responses, and 7 is the terminal step.
3 Method
In the proposed Thinker task, we decompose the QA task into four steps. The whole task occurs within a single dialogue, meaning that the agent receives all prompts and responses from previous steps in addition to the current prompt. An illustration of the Thinker task is shown in Figure
3.1 Task Description
Step 1 Fast Thinking, In the first step, the agent is prompted to answer the given question concisely within strict token budget. The response in this step is restricted to relatively short maximum generation length (e.g.. 1000 tokens), meaning that the response will be truncated if it exceeds this length. The reward Riss in this step is defined extracted answer. Specifically, let yrast denote the extracted answer; then Rias = {yras = y The agent always proceeds the next step after the response.
binary function based on the correctness of the
Motivation.
The motivation this step is to explicitly train the agent's intuition. As the agent must generate response under strict token budget, it cannot search extensively. It is usually restricted to a few search paths, which are directly reinforced if one leads to the correct answer.
Verification. In the second step, the agent is prompted to verify whether the fast answer Wast is correct, and must output either Yes or No. The response in this step restricted a relatively short maximum generation length (e.g., 2000 tokens). The reward Rverity in this step is defined as weighted binary function based on the correctness of the verification. Specifically, yverity denote the extracted answer; then:
Step 2

---

Step 1: Fast Thinking
User:
Please answer the question concisely, This question: <Question> Step 1... Step 2 The answer
Response:
Yust
Riast
Vtast
y
Training Always proceed Inference Always proceed
Step 2: Verification
Either Yes
User:
Is your above answer correct?
Response:
Let me verify. Step 1... Step 2
The verification answer is (verry
(1
Pimatamx) 1 = Yes)
if Miau 1/1 otherwise,
Rownfy
Training I incorrect Inference If Yuuryy No
Pixet-ace
1 (Vveetyy No}
Step 3: Slow Thinking
User:
Your initial answer verified to be incorrect. Please try again. Let correct my mistake. Step Step ... The answer is Yaou
Response:
Ralow
{ yslow y
Training
y is correct Inference
Step 4: Summarization
User:
Your final answer is correct. Please summarize the steps concisely. This is the question: <Question>
Never proceed
Response:
Step 1.... Step The answer is wetperrisyyummer Yuummayy
Raummery Voummay
Vslow + clog P
(asummary Tfatt
Short generation length (o.g. 1000 tokens)
Long generation length
(o.g. 6000 tokens)
The four-step Thinker task. Each stage involves a user prompt, model response, and specific rewards and transition conditions designed to train distinct agent capabilities (intuition, evaluation, refinement, and integration). Reward function details are in the main text.
Figure 2:
{(1
Prast ):
(Iiverity = Yes)
ylast = y",
Rverify
Pfast-ace
1 [[Iverify No}
otherwise,
where Plastacc denotes the trailing accuracy of the fast thinking step (averaged over the batch). This weighting is used to balance the two classes, so to discourage the agent from always outputting Yes or No when the accuracy of the fast thinking step is too high or too low.
The transition function to the next step depends on whether the agent is training or inference mode. In inference mode, if the agent answers Yes, then the fast answer is chosen the final answer, and the episode terminates; otherwise, the agent proceeds to the next step. In training mode, if the fast answer correct, then it chosen the final answer; otherwise, the agent proceeds to the next step. The distinction between training and inference mode aims to ensure that, during training, the Slow Thinking step primarily focuses on instances where the fast answer was incorrect. This prevents the agent from needing to re-vent re-verify an already correct fast answer in the Slow Thinking stage. However, during inference, do not have access the ground -roun answer, so we must rely on the agent's verification.
Motivation.
The motivation of the second step is explicitly train the agent's evaluation capability. The agent receives a clear binary reward based on whether its verification result is correct. Verifying an answer is often easier than generating one. If the fast answer already verified to correct, there is no need to proceed further, thus saving computational cost.
Step 3 Slow Thinking, In the third step, the agent is prompted that the fast answer has been verified to incorrect and is asked to try an alternative answer. The response in this step is restricted to relatively long maximum generation length (e.g., 6000 tokens). The reward Ratow in this step is defined as a binary function based on the correctness of the extracted answer. Specifically, let ydow denote the extracted answer; then Rsoow (Vstow = y
In both inference and training modes, Ysllww always chosen as the final answer if the Slow Thinking step is exccuted. In inference mode, the episode ends here. In training mode, the agent proceeds to

---

the next step if the slow answer Uslow is correct; otherwise, the episode ends. This distinction exists because the purpose of the next step supmmmrraiiiiiinnn is to distill the long and correct response in this step to improve intuition, which is not applicable in inference mode.
Motivation. The motivation of this step is encourage the agent learn refine incorrect fast answers for difficult questions. It should learn to use the reasoning from the verification step and revise errors to arrive at the correct answer. If such refinement not possible, it should learn to try an alternative approach, leveraging the generous token budget for generation.
Step 4- Summarization. In the fourth step, the agent is prompted that the previous slow answer correct and is asked to concisely summarize the steps leading to it. Crucially, the prompt for the Summarization step includes the original question again, mirroring its presentation in the Fast Thinking prompt. The response in this step is restricted to a relatively short maximum generation length (e.g., 1000 tokens). The reward Raumnary in this step is designed based on two criteria:
1. Correctness: The extracted answer from the summary, Vsummary: should be the same as the previous slow answer, meaning it should not produce a summary that leads to an incorrect answer.
2. Consisteney:
The response should be consistent with what the model is expected to produce in the Fast Thinking mode-that is, its probability conditioned on the Fast Thinking prompt should not be unduly low. For example, directly outputting the final answer without intermediate steps is considered inconsistent, the likelihood of producing the correct answer directly under Fast Thinking mode is typically very low.
Combined, the reward function this step is defined as:
Rammary (Vanmmry V/loww + c clog P P(Usammary Tifstt)
where Fiast is the prompt Fast Thinking step (i.c., the initial prompt), log P(Osummary (iaaa is the log probability of the summary response under the Fast Thinking prompt, and e is a scalar hyperparameter. In experiments, we found that the agent sometimes still degenerates to give very short answer despite the log probability term. To mitigate this, we gate the reward to 0 if the length of the generated response is less than a low threshold (e.g., 300 tokens).
Motivation. The motivation of the final step is to reinforce concise reasoning patterns by rewarding correct and consistent summaries. A key design element is that the original question is re-presented in the Summarization prompt, mirroring its appearance in the Fast Thinking step. The agent is trained to produce a concise reasoning trace that leads the correct answer for this input. This encourages the model to form a strong association between the original question and correct, concise solution path. We hypothesize that this targeted reinforcement distills the successful but lengthy reasoning from Slow Thinking into a compact form suited to the Fast Thinking mode-thereby improving the agent's intuition. In addition intuition, this step also trains the agent's integration ability, must extract and condense key reasoning steps from the longer trace generated in the previous step.
3.2 Training with the Thinker Task
Training LLMS with the Thinker task requires partieular considerations regarding reward propagation. Since the reward at each step is specific to that step alone, it should not be propagated backward to earlier steps. This implies that the discount factor between steps should be set O, while that within each step should be high (e.g., 1) to enable effective credit assignment over tokens. The Thinker task defines only the RL environment and imposes no restrictions on the choice of algorithm or model architecture, allowing compatibility with any standard RL method (e.g., PPO) and LLM.
4 Related Work
Environment Augmentation. Our Thinker task form of environment augmentation for QA, inspired by Dual Process Theory and related to concepts like the Thinker MDP [22 [3]. While Thinker MDP provides agents with a simulated world model for interaction before action, our task structures QA into stages where self-generated intermediate outcomes guide subsequent reasoning. This contrasts with multi-attempt tasks [1] that allow iterative revision but require ground-truth access during inference, a constraint our method avoids.
5

---

RL and Reasoning in LLMs. A large number of studies have demonstrated the effectiveness of applying RL to enhance the reasoning capabilities of LLMs 2 3 Our work builds on these efforts by decomposing the QA task into the four-step Thinker task. It has been observed that L.LMs trained with RI. can produce inefficient CoT 4 leading to overthinking [55] Our work relates to strategies for controlling generation length (often termed token budgets). Examples include dynamic token budgets that scale with problem complexity [5), or user-defined budgets [T6 [7]. For instance, Muennighoff et al. [88 utilize token budget controls during the generation of CoT data for supervised fine-tuning (SFT). Concurrent works such as ThinkPrune [9]] Concise Reasoning 200 SR-Flow 211, and AdaptThink (22) modify the reward RL algorithm to encourage more efficient reasoning. While these approaches primarily focus on token budget control within a single response generation, method introduces a structured, multi-step RL task explicitly designed to train distinct agent abilities independent of the underlying RL algorithm. In addition, our work is related to methods for encouraging celf-correction in L.MMs 23 For instance, methods like Self-Refine (4 and Reflexion (25 primarily use prompt engineering and few-shot examples to enable agents to incorporate internal external feedback for refining subsequent responses. However, unlike work, these methods typically do not involve RL fine-tuning of the agent for these self-correction behaviors; instead, the correction capability is elicited at inference time through prompting.
5
Experiments
This section details the experiments conducted to assess whether the Thinker task can more effectively enhance L.M reasoning capabilities compared to standard QA task. We focus the mathematical reasoning domain here.
5.1 Experimental Setup
To evaluate the Thinker task, we fine-tune two publicly available models: Qwen2.5-1.5B (Q1.5B) (26) and DeepSeek-RR -il----iill---ee--------
(R1.5B) M. While sharing a base architecture, R1.5B has undergone additional distillation using reasoning data from DeepSeek-RI, endowiing it with stronger initial reasoning and search behaviors compared the base Q1.5B. Training both models allows us to investigate the Thinker task's ' impact on model with foundational capabilities (Q1.5B) and its ability further enhance a model already specialized for reasoning (R1.5B). For the Q1.5B runs, we use three independent seeds and report the averaged results.
We fine-tune these models using RL both the Thinker task and standard QA task (serving as our baseline). For all experiments, we employ PPO (6). Key hyperparameters include a discount rate ? 1, GAE lambda À 1. and sampling temperature of 1. No KL penalty against reference policy was applied. The Fast Thinking and Summarization stages use a 1000-token budget, Verification uses a 2000-token budget, and Slow Thinking uses 6000-token budget. Most other hyperparameters and training details mirror those in Open- Reann-Ressner-Zerg with details provided in Appendix[A]
For the training data, we utilized the 129K math question-answering dataset provided by Open- Reasoner-Zero. Each training run for both the Thinker task and the baseline required ap- proximately 7 days on two compute nodes, each equipped with 8 A100 GPUs. Our im- plementation, adapted from the Open-Reasoner--Zrro codebase 33, is publicly available at https: //githb./ thub. www. hemm/stephen-ceee shteppennchhnng--nhhhttneeeeee-e whtp:/tttnneeeet... which also includes the trained mod- els.
5.2 Training Dynamics and Evaluation Results
The training performance shown in Figure 3 measured by accuracy on the training data. For the Thinker task, we plot fast accuracy (from the Fast Thinking stage) and final accuracy (from the Slow Thinking stage). Note that the final accuracy not directly comparable Thinker agent effectively has two attempts during training (though only one during testing). We observe that the Thinker agent's fast and final accuracy improve steadily, while the baseline's 's accuracy plateaus rather quickly. This may suggest that the Thinker agent is learning the underlying intuition of the task, which requires sustained training, as opposed to merely acquiring generic search algorithm. We also observe that the fast accuracy of the Q1.5B model surpasses the baseline's. 's. Notably, while
the baseline's, ss as the

---

0.8
0.4
Accuracy
0.3
Accuracy
0.5
0.4
0.3
Baseline
0.2
Baseline
Thinker (Final)
Thinker (Final)
Thinker (Fast)
0.1
Thinker (Fast)
0.0
0.0
200
400
600
800
1k
1.2k
200
400
600
800
1k
1.2k
Training Steps
Training Steps
(a) Qwen2.5-1.5B
(b) DeepSeek-RI M--iiiiill-Queeeee 1.5B
Figure 3: Accuracy on training set. For the Qwen2.5-1.55 model (a), the shaded region represents the standard deviation across three independent seeds.
2500
6000
Response Length [tokens)
2250
Response Length (tokens)
5000
Thinker (Final)
Baseline
2000
(Fast)
4000
Baseline Thinker (Final)
Thinker (Fa st)
3000
1250
1000
750
1000
500
200
400
600
800
1.2k
200
400
600
800
1k
1.2k
Training Steps
Training Steps
(a) Qwen2.5 5.5B
(b) DeepSeek- RI--sisill Qwen- 1.5B
Figure 4: Average response length on training set. For the Qwen2.5-1.5B model (a), the shaded region represents the standard deviation across three independent seeds.
both metrics reflect a single attempt, the Fast Thinking mode achieves superior performance with significantly smaller token budget (Figure underscoring its efficiency.
Figure 4 illustrates the average response length during training, plotting both the Fast Thinking response length and cumulative length across all four stages of the Thinker task. The two models exhibit distinct behaviors. For the Q1.5B model, which lacks strong inherent self-correction capabilities, the Thinker task's 's cumulative length is greater than the bascline's s, as its structured stages introduce verification and refinement steps. Conversely, for the R1.5B model, which already possesses self-correction abilities, the Thinker agent's total response length eventually becomes shorter than the baseline's. This suggests that rather than simply adding overhead, the Thinker task refines the R1.5B model's 's existing search process, pruning inefficiencies to achieve more loken-efficien reasoning.
To evaluate the models, we consider the following common benchmarks: MATH500, AIME2024, AIME2025, GPQA Diamond, Olympiadbench, AMC23, and Minerva Math. We evaluate agent Pass 1 accuracy on these benchmarks every 50 training steps. The average performance across the benchmarks during training shown in Figure 5 The trend similar to the training performance, with the agent trained the Thinker task surpassing the baseline. Detailed breakdowns for each benchmark are provided in Appendix Notably, for the R1.5B model, a moderate gap exists between its final accuracy and fast accuracy. This disparity is likely attributable to RI.5B's strong inherent reasoning capabilities, enabling its Slow Thinking mode to effectively utilize the larger token budget for refining initial answers.
The performance of the final models is detailed in Table These models correspond to check- points, saved at 50-step intervals, that achieved the highest accuracy a validation dataset. We evaluated these final models on the aforementioned benchmarks and the additional CollegeMath benchmark; due to its large size, CollegeMath was reserved for this final evaluation and was not

---

0.55
0.26
0.50
Average Accuracy
Average Accuracy
0.24
0.22
0.40
020
Baseline
0.30
Baseline
0.18
Thinker (Hinal)
0.25
Thinker (Final)
Thinker (Fest)
Thinker (Fast)
0.16
0.20
200
400
600
800
1.2k
0
200
400
600
800
1k
1.2k
Training Steps
Training Steps
(a) Qwen2.5-1.5B
(b) Baannneek-R--istttiie Swaeiill---ee-.... 1.5B
Figure 5: Evaluation performance across seven common benchmarks. For the Qwen2.5-1.50 model (a), the shaded region represents the standard deviation across three independent seeds.
Table I: Performance comparison across various mathematical reasoning benchmarks. Average (Avg.) scores presented. All scores Pass@1 accuracy (%) averaged over 16 samples. Top score in each benchmark column is bolded. Standard errors and more statistical analysis are provided in Appendix B Results for the Q1.5B Thinker and Baseline are averaged across three independent
seeds.
MATH
AIME
AIME GPQA Diamond
Olympiad bench
AMC
Minerva Math
College Math
Method
Avg.
500
2024
2025
23
Qwen2.5-1.5B
(Q1.5B)
Pretrained
9.05
0.00
0.00
4.55
3,09
4.06
2.30
7.40
3.81
Baseline
59.82
4.10
2.43
20.52
26.05
35.36
19.25
37.42
25.62
Thinker 64.45
6.25
2.22
19.21
28.21
39.06
20.38
38.82
27.33
Thinker-Fast Fast
59.82
4.58
1.25
21.28
24.52
34.53
17.85
37.58
25.18
ORZ 58.00
3.50
1.00
16.80
SimpleRL 59.00
4.20
21.00
35.00
20.20
DeepSeek RI-Distill Prbistill-Qwen--eee -wen-1.5B (R1.5B)
Pretrained 76.21
17.50
17.92
13.76
37.46
55.94
24.82
38.85
35.31
Baseline 86.24
35.42
23.75
25.69
49.22
72.81
32.08
42.02
45.90
Thinker 88.51
38.96
26.67
37.41
55.49
83.59
34.77
42.46
50.98
Thinker-Fast
81.35
18.33
14.58
28.85
45.68
66.41
31.39
41.74
41.05
used to assess intermediate checkpoints. For a comprehensive comparison, the table includes the Fast and Final Accuracy our Thinker agent, the accuracy the fine-tuned Baseline, and results for the original Pretrained model. Since evaluation is performed in inference mode, the Thinker agent does not require external verification, making Final Accuracy directly comparable the single-attempt baselines. Furthermore, we include reported figures from Open-Reasoner-Zere (ORZ) and SimpleRL. 2 both of which also utilize PPO to train agents on standard QA tasks.
From Table T we observe that the Thinker agent consistently performs better than the other methods across almost all benchmarks. For the QL.SB model, the Thinker-Fass agent's performance is already close to that of the baseline, while the full Thinker agent achieves a 6.7% relative performance gain average compared to this baseline. For the R1.5B model, the Thinker-Fast agent performs slightly worse than the baseline but still significantly outperforms the pretrained model. This result for Thinker-Far Fast is notable, suggesting substantial efficiency gain, as it only requires 1000 token budget compared to the 8000 token budget of the pretrained model. The full Thinker-task model with R1.5B surpasses the baseline by an average relative increase 11.1%. These results collectively suggest the benefits of decomposing the single-tur QA task into the proposed four-step Thinker task. To confirm these benefits extend larger models, additional experiments on 7B models were conducted, yielding similar results as detailed in Appendix D

---

Table 2: Comparison with coneurrent works fine-tuning R1.5B models token efficiency bench- marks. Results from concurrent works are extracted from the respective papers.
MATH500
AIME24
AMC23
Method
Acc. (%)
Length
Acc. (%)
Length
Acc. (%)
Length
ThinkPrune 19
83.2
1938
27.1
5631
73.2
3039
Concise Reasoning 20
81.0
1965
30.0
6752
69.4
2936
SR-FLOW [21
85.3
36.7
77.8
AdaptThink [22]
82.0
1782
31.0
6679
Baseline
86.2
2780
35.4
5778
72.8
3938
Thinker
88.5
2501
39.0
5597
83.6
3517
Thinker-Fast
80.9
600
18.1
853
66.9
751
To evaluate token efficiency, analyze the Thinker agent three benchmarks used in concurrent work on this topic (Table The results show that our agent achieves superior performance across all three benchmarks with competitive token count. The Thinker-Fast mode is also effective on easier datasets like MATH500 and AMC23, using substantially fewer tokens. This highlights a key advantage for deployment: the flexibility to select the appropriate mode based on the task's s complexity.
5.3 Analysis and Case Study
Reflection and Response Length. Beyond overall performance, we investigated whether the Thinker task encourages more direct reasoning with less overt self-reflection self-doubt. We compiled vocabulary list of terms commonly associated with self-reflection in Deepseek R1's outputs (e.g.. "wait," "however," "alternatively") and measured the average occurrence of these reflection patterns during training, with results shown Figure 6a] We observed that the Thinker agent tends to use fewer such reflection patterns in its reasoning trace, suggesting it may be learning to reason more directly.
A detailed breakdown of response length during training is presented in Figure 6b The trends for the two agents diverge: after stabilizing, the baseline's ' response length steadily increases, likely due to increase in learned self-reflection. Conversely, the Thinker agent's 's total response length gradually decreases. This improved efficiency is not due to reduction in the length of individual stages, which remain consistent. Rather, it stems directly from the model's rising Fast Thinking accuracy. As the agent's 's intuition improves, it solves more problems correctly the Fast Thinking stage, thereby allowing it to bypass the token iikennintensive Slow Thinking stage and significantly reduce its average token length.
Baseline
Reflection Pattern Score
Thinker
M Mn
Average Token Length
2000
Baseline
Thinker - Total
Morr
2000
Thinker
Thinker Verify
Thinker Slow
1000
%
Training Steps
Training Steps
(a) Reflection Pattern
(b) Response Length
Figure 6: Average reflection pattern count and response length for DeepSeek-RI RI-Distill Saelll--wen--11 1.5B
(R1.5B) during training.

---

0.30
6000
Reflection Pattern Occurance
Thinker
Thinker
0:25
Thinker Summarzation
Average Token Length
5000
Thinker Summarizttio
0.20
4.000
0.15
3000
0.10
2000
0.05
Lece
0.00
200
400
600
800
1k
1.2k
1.4k
1.6k
1.8k
%
200
400
600
800
1k
1.2k
1.4k
1.6k
1.8k
Training Steps
Training Steps
(a) Reflection Pattern
(h) Response Length
Figure 7: Average reflection pattern count and response length for Qwen2.5-1.5B (Q1.5B during training, with and without Summarization stage.
Role of Summarization F Finally, we investigated the importance of the Summarization step through an ablation analysis. We trained Q1.5B model using a modified Thinker task where the Summarization step was removed. The average reflection pattern count and response length for the Thinker task, both with and without the Summarization step, are presented Figure Without the Summarization step, the model exhibited more frequent increase in reflection patterns, accompanied by highly fluctuating response lengths. This indicates that the Summarization step may contribute to stabilizing the learning process, potentially by discouraging degeneration into excessive reflection, though we cannot rule out that the observed divergence due to stochastic variation.
Removing the Summarization step negatively impacted Fast Thinking: average Fast Accuracy dropped from 26.75% to 24.84%. This suggests summarization, by potentially distilling concise reasoning, enhances intuition for Fast Thinking. Interestingly, final accuracy remained comparable (27.85% with vs. 27.84% without summarization), indicating that the Slow Thinking step could often compensate for the reduced fast accuracy. Detailed ablation results can be found in Appendix B
study examine how the agent's reasoning adapts across the Thinker task stages. Observations from representative outputs (see Appendix C for full examples) highlight the importance of the Fast Thinking mode. When initial Fast Thinking output is incorrect, the agent first engages in a Verification stage to assess its own answer with clear reference to the steps given in the Fast Thinking mode. If errors identified, the subsequent Slow Thinking stage involves a detailed refinement. During Slow Thinking, the agent explicitly scrutinizes the flawed reasoning from the Fast Thinking attempt and insights from its Verification, endeavoring generate new, correct solution. Furthermore, after successful correction, the agent learns to distill the long reasoning into concise explanations during the last Summarization stage.
Case Study. We conducted
6 Future Work and Conclusion
In this paper, we introduced the Thinker task, a novel approach that decomposes the standard single- turn question-answering process into structured four-stage task. This decomposition aims to provide more explicit reward signals for training distinct agent capabilities: intuition via Fast Thinking, evaluation via Verification, refinement Slow Thinking, and integration via Summarization.
Beyond the application to question-answeriine answering, this work underscores the potential of environment augmentation in RL. In RL, while significant attention is devoted to algorithm development, the environment itself is typically treated as a given problem specification. However, as demonstrated by this paper and prior research such the Thinker MDP, there considerable untapped potential in designing environments that offer richer inputs, more structured interactions, or more nuanced reward signals. Future research could explore alternative methods for enriching RL environments, perhaps by developing more dynamic tasks that adapt to an agent's 's learning state or that explicitly target the development wider array of cognitive capabilities. Such advancements environment design could unlock new levels of performance and emergent abilities in RL agents.
2This ablation study used 6000-token Verification step. The length was later reduced to 2000 tokens for the main experiments, this did not significantly affect performance.
10

---

Acknowledgment
Jie Fu is supported by Shanghai Artificial Intelligence Laboratory.
References
[1] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[2] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma. and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models the wild. arXiv preprint arXiv:2503. 2025.
[3] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero:: An open source approach to scaling up reinforcement learning the base model. arXiv preprint arXiv:2503.242200 2025.
[4] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint arXiv:2412.21188 2024.
[5] Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aware Ilm reasoning. arXiv preprint arXiv:2412.18544 2024.
[6] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.066 2017.
[7] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03266 2025.
[8] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03377. 2025.
[9] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol- preview with 1.5b model by scaling rl. Notion Blog.
2025.
[10] Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
[11] Gary A Klein. Sources of power: How people make decisions. MIT press, 2017.
[12] Stephen Chung, Ivan Anokhin, and David Krueger. Thinker: Learning to plan and act. Advances in Neural Information Processing Systems, 36:22896-22933, 2023.
[13] Kevin A Wang, Jerry Xia, Stephen Chung, and Amy Greenwald. Dynamic thinker: Optimizing decision-time planning with costly compute. In The Seventeenth Workshop on Adaptive and Learning Agents.
[14] Stephen Chung, Wenyu Du, and Jie Fu. Learning from failures in multi-attempt reinforcement learning. arXiv preprint arXiv:2503. 04808, 2025.
[15] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint 1644 2025.
[16] Pranjal Aggarwal and Sean Welleck. LI: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697 2025.
[17] Yuhui Xu, Hanze Dong, Lei Wang, Doyen Sahoo, Junnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning, arXiv preprint wrXiv:2505.05315, 2025.
11

---

[18] Niklas Muennighoff, Zitong Yang, Weijia Shi. Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint Sttiv:2501.193990 2025.
[19] Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long thain-of-thouue arXiv:2504.01290 2025.
Ilms via reinforcement learning. arXiv preprint
[20] Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning, arXiv preprint arXiv:2504.05185 2025.
[21] Yubo Dong and Hehe Fan. Enhancing large language models through structured reasoning. arXiv preprint (rXiv:2506.20241, 2025.
[22] Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. Adaptthink: Reasoning models can learn when to think, arXiv preprint arXiv:2505. 134/7, 2025.
[23] Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When Ilms actually correct their own mistakes? a critical survey of celf-correction of llms. Transactions of the Association for Computational Linguistics, 12:1417-1440, 2024.
[24] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with feedbckee..c. Advances in Neural Information Processing Systems, 36:46534-46594, 2023.
[25] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-866 8652, 2023.
[26] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[27] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505-3506, 2020.
[28] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[29] Philipp Moritz, Robert Nishihara. Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging {AI] applications. In 13th USENIX symposium on operating systems design and implementattinn (OSDI 18), pages 561-577, 2018.
[30] Evan Miller. Adding error bars evals: A statistical approach to language model evaluations. arXiv preprint 20Xi::441.0064444 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, clearly list contributions, and at the end of the introduc- tion, we further elaborate on our contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Section 6 discuss the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
This paper is an empirical study.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Section 5.1 and Appendix A
we provide detailed evaluation results and
training hyperparameters.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: We have submitted our code and data to OpenReview, and we will open-source them GitHub in the final version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
htaiai//ggiddee/deeSubbbita cocloloo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
wttgggidddd/oeSSuuunnnnn ssionlolicy) for more details. hwwpb//iiigggddde/oooooeo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 5.T and Appendix A we provide all evaluation and training hyper- parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Standard errors of the main evaluation results are reported in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details of the computational resources required are discussed Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read this code.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in Appendix E
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Our study is empirical exploration. We open-source mathematical problem sets, and the models we release are intended solely for research purposes not direct industrial application.
-ot for
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please refer to Appendix A
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All datasets, codes and models will be fully released under the license of CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
18

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We LLMs solely for grammatical checks in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non -tanddaddrrd components.
Please refer to our LLM policy (https: for what should or should not be described.
19

---

A Experimental Details
This section describes the details of the experiment in Section Our implemen- lation, adapted from the Open-Reasoner-Zero codebase [3] is publicly available at https:/ //gi githt....... www...//eeeennccuuieee.e www.hnnnngnttiiinnnerrrrrr thinner-----ree task, which also includes the trained mod- els.
A.1 Hyperparameters
We the same hyperparameters those provided in eeee 3 except that reduce the number of samples per prompt from 64 to 32 to save computational resources. One training step proceeds follows: we first randomly sample 128 prompts (rollout batch size) from the training dataset and generate 32 samples per prompt, totaling 128 x 32 = 4,096 samples. We then divide the generated samples into (12) training batch for the actor (critic), where each training batch used for one optimizer update.
We tune the coefficient in Raummary by searching over {1e-4, 1e-3, 1e-2). Other Thinker-tast task- specific hyperparameters are selected using heuristics. We use a lower sampling temperature during summarization, observe that higher temperatures tend produce less concise and consistent summaries.
For the baseline model, we use similar hyperparameters, except with generation length of 8,000 tokens. We found that 8,000 tokens yield optimal baseline performance on R1.5B.
Table 3: Hyperparameters used in experiments.
Parameter
Value
PPO
Rollout Batch Size
128
Number of Samples Per Prompt
32
Number of Epochs
Actor Learning Rate
le-6
Number of Actor Update Steps
1
Critic Learning Rate
5e-6
Number of Critic Update Steps
12
Discount Rate y
GAE Lambda X
1
Clip Ratio
0.2
KI. Loss
None
Sampling Temperature
Sampling Temperature in Summarization
0.6
Generation Length
Fast Thinking
1,000
Verification
2,000
Slow Thinking Summarization
6,000
1,000
Reward-specific
Coefficient e Raummary
1e-3
Minimum Length for Summarization
300
A.2 Prompt Templates
The prompt templates used in four stages of the Thinker task are illustrated Box A. Note that not all prompts are necessarily used. For example, in training mode, if the agent's fast answer is correct, the Slow Thinking and Summarization prompt will skipped. Please refer the main text for the termination conditions.
20

---

Box A.I: Prompt Templates for Thinker Task
Fast Thinking
Answer the below question with concise steps and output the final answer within \boxed(). Limit your response below 1000 words. This is the problem: {question)
User:
Assistant:
<Agent Response>
Verification
User: Is your answer above correct? Please verify each step and the answer care- fully. Output \boxed(Yes) if your answer is correct, \boxed(No)} if your answer is incorrect.
Assistant:
<Agent Response>
3. Slow Thinking
Your initial answer is incorrect. Now, think about the possible errors and consider alternative solutions. The reasoning process should be enclosed within <think>... "think>..<tthinkk This is the problem: {question} Let's think step by step and output the final answer within \boxed(). Assistant: <think> <Agent Response>
User:
4. Summarization
User: Your final answer is correct. Now summarize the steps leading to your final answer concisely and precisely, excluding internal reasoning. Limit your response between 300 and 1000 words.
This is the problem: {question}
Assistant: <Agent Response>
A.3
Computational Resources
Each training run for both the Thinker task and the baseline required approximately 7 days on two compute nodes, each equipped with A100 GPUs. We use the Deepspeed [27], vLLM (28), and Ray [29] library for distributed training.
21

---

B
Result Details
This section describes additional experimental results that were omitted from the main text due to length constraints.
B.1 Evaluation Results
Figure X and Figure 9 show the breakdown of Figure [5 from the main text, corresponding the evaluation results fine-tuning Q1.ST and R R1,5B on the QA task or the Thinker task during training.
The detailed evaluation results of the ablated run in which the Summarization step is removed can be found in Table labeled as SkipSum.
Table 5 and Table O present the standard error corresponding to the results reported in Table The standard errors here are computed using Equation (1) from 30
Accuracy
Accuracy
0.100
0.06
0.4
0.04
0.050
0.02
0.2
0.000
0.00
200
400
600
800
1.2k
0
200
6.0
600
BO0
1k
12k
200
400
600
8.00
1k
1.2k
Training Steps
Training Steps
Trai ning Steps
(a) MATH 500
(b) AIME
AIME
0.30
25
0.25
0.4
Accurscy
20
Accuracy
Accuracy
2.5
0.3
0.15
20
0.10
400
6.00
1k
1.2k
200
400
6.00
800
1k
1.kk
200
4.00
600
800
1x
1.2k
Training Steps
Training Steps
Training Steps
(d) GPQA Diamond
(c) Olympiad Bench
(f) AMC 23
0.20
Accuracy
0.15
0.10
Baseline
0.05
Thinker (Anal Acc)
Thinker (Past Acct
650 Training
(g) Minerva Math
Figure Detailed evaluation results of Q1.5B fine-tuned using QA task or Thinker task on individual mathematical reasoning benchmarks. The shaded region represents the standard deviation across three independent seeds
22

---

0.4
Accurecy
Accuracy
Accurecy
0.2
0.4
0.0
0.0
200
600
1.24
600
14
1.2k
Triining Steps
Training Steps
Trainingg Steps
(a) MATH 500
(b) AIME 24
AIME 25
Accuracy
Accuracy
6.0
20
0.4
0.2
200
400
600
900
1K
1.28
200
400
600
800
1k
1.2k
200
4.00
600
800
1x
1.2k
Training Steps
Training Stegs
Training Steps
(d) GPQA Diamond
(c) Olympiad Bench
(f) AMC
Accuracy
Thinker (Fost (Fost Acci
200
400
600
800
1k
12k
Training Stops
(g) Minerva Math
Figure 9: Detailed evaluation results of R1.5B fine-tuned using QA task or Thinker task individual mathematical reasoning benchmarks.
Table 4: Mathematical reasoning performance of the Thinker agent trained without the Summarization stage. Average (Avg.) scores are presented. All scores are Pass@ accuracy (%) averaged over 16 samples. All runs used a 6000-token Verification step.
MATH AIME
AIME
GPQA
Olympiad bench
AMC
Minerva College Math
Method
Avg.
500
2024
2025
Diamond
23
Math
Qwen2.5-1.5B (Q1.5B)
Thinker
64.25
6.25
2.50
23.74
28.11
40.62
19.03
38.33
27.85
Fast
61.60
6.25
2.50
26.39
24.78
35.94
18.66
37.85
26.75
SkipSum
64.30
9.17
4.17
18.62
29.11
37.50
20.82
39.42
27.89
SkipSum-Fass
60.30
5.00
1.25
20.27
24.17
30.00
19.85
38.24
24.88
23

---

Table 5: Standard analysis for the Q1.5BB model. All scores are in %. The values presented as score (standard error).
Thinker
Thinker-Fas
Baseline
Benchmark
Qs
Seed
Seed
Seed
Seed
Seed 2
Seed 3
Seed
Seed
Seed
MATH 500
500
63.25
64.72
65.38
59.42
59.67
60.38
57.98
60.72
60.75
(1.83)
(1.79)
(1.80)
(1.83)
(1.85)
(1.84)
(1.79)
(1.80)
(1.79)
AIME 2024
30
5.62
5.83
7.29
5.00
4.17
4.58
3.33
4.38
4.58
(2.51)
(3.20)
(3.34)
(2.83)
(2.69)
(3.16)
(1.58)
(2.61)
(2.49)
AIME 2025
30
3.54
1.88
1.25
1.46
1.04
1.25
3.33
1.46
2.50
(1.84)
(1.13)
(0.70)
(1.26)
(0.85)
(0.63)
(1.79)
(1.07)
(1.36)
GPQA Diamond 198
19.00
17.99
20.64
23.11
19.85
20.86
21.46
19.35
20.74
(1.69)
(1.67)
(1.78)
(1.77)
(1.84)
(1.81)
(1.80)
(1.61)
(1.74)
Olympiad Bench
675
28.02
27.94
28.67
24.82
24.13
24.60
24.54
26.46
27.16
(1.44)
(1.45)
(1.46)
(1.35)
(1.37)
(1.35)
(1.32)
(1.39)
(1.38)
AMC23
40
38.91
41.41
36.88
36.09
34.84
32.66
34.38
36.41
35.31
(6.11)
(6.38)
(6.18)
(6.43)
(5.74)
(5.78)
(5.62)
(5.67)
(5,46)
Minerva Math
272
19.90
21.83
19.42
16.98
20.06
16.50
17.78
19.88
20.08
(1,92)
(2.02)
(1.90)
(1.75)
(1.90)
(1.74)
(1.85)
(1.88)
(1.94)
College Math
2818
38.73
39.15
38.58
37.49
37.84
37.39
35.85
38.26
38.15
(0.85)
(0.86)
(0.85)
(0.85)
(0.84)
(0.84)
(0.81)
(0.84)
(0.85)
Table 6: Standard error analysis of R1.5B models on mathematical benchmarks. All scores are in %. The values are presented score (standard error).
Benchmark
# Questions
Thinker
Thinker-Fast
Baseline
MATH 500
500
88.51 (1.22)
81.35 (1.50)
86.24 (1.25)
AIME 2024
30
38.96 (7.21)
18.33 (5.23)
35.42 (7.09)
AIME 2025
26.67 (7.52)
14.58 (5.37)
23.75 (6.74)
GPQA Diamond
198
37.41 (2.24)
28.85 (2.27)
25.69 (2.00)
Olympiad Bench
675
55.49 (1.67)
45.68 (1.66)
49.22 (1.66)
AMC23
40
83.59 (4.70)
66.41 (5.94)
72.81 (5.18)
Minerva Math
272
34.77 (2.57)
31.39 (2.43)
32.08 (2.47)
College Math
2781
42.46 (0.90)
41.74 (0.89)
42.02 (0.89)
24

---

B.2 Ablation Study on Fast Thinking Mode
To understand the importance of the Fast Thinking mode in the overall Thinker task, we experiment by using a less-trained agent to generate the Fast Thinking response, while still using the fully trained agent generate responses for the remaining stages This allows us measure the impact of Fast Thinking quality on overall performance.
four earlier R1.5B Thinker-agent checkpoints (Step 0, which is the pretrained model; Step 200; Step 400; and Step 600) to generate the Fast Thinking response, and use the fully trained RI.SB Thinker-agent for the remaining stages. We evaluate final accuracy across the eight benchmarks, as in the main evaluation. The results are shown in Figure I0
Specifically,
We observe general positive correlation between the Fast Thinking accuracy of checkpoint and the final accuracy, suggesting that the Fast Thinking response has a substantial influence on subsequent stages. For instance, when use the pretrained model (Step 0) to generate the Fast Thinking response. final accuracy drops significantly from 49.8% to 36.3%. However, we also observe that this sensitivity diminishes Fast Thinking performance improves. For example, using the Step 200 model, which has a moderate Fast Thinking accuracy of 28.9%, leads to final performance of 49.18% minor drop from 49.8%.
We conjecture that this due to the robustness of the Slow Thinking mode: since it is trained specifically to handle incorrect Fast Thinking answers, it can often recover from slightly flawed initial intuition. However, if the Fast Thinking intuition is very poor (as the pretrained model), subsequent stages may fail to recover due to the lack of meaningful starting point. A qualitative analysis of how the Fast Thinking stage interacts with subsequent stages can found the case study in Appendix C which shows that the trained agent is able to correct flawed heuristics from the Fast Thinking mode during the Verification and Slow Thinking mode.
60
Average Accuracy (%)
50
30
20
10
Final Accuracy
Fast Accuracy of Checkpoint
Step
Step 200
Step 400 Step 600
Full Model
Fast Thinking Checkpoint Step
Figure 10: Final accuracy on the evaluation benchmarks of the Thinker agent (R1.5B) when its
Fast Thinking stage is generated by model checkpoints at previous training steps. The original Fast Thinking accuracy of these respective checkpoints also shown. All scores Pass accuracy (%) averaged over 16 samples. Error bars represent standard error, which are typically minor in this data.
"The section here used 6000-token Verification step. The length was later reduced to 2000 tokens for the main experiments, this did not significantly affect performance.
25

---

C Case Study
In this section, we present sampled responses from the fine-tuned R1.5B agent on the Thinker task, aiming to understand the behavior learned by the agent. Only responses with an incorrect fast answer are selected, so that the interaction between Fast Thinking and Slow Thinking can be observed.
C.1 Case Study I: Identifying Flaws in Fast Thinking (Box
Box
This example demonstrates the Thinker task's 's ability to guide the agent from an uncertain, flawed initial attempt correct, verified solution by structuring its reasoning process.
Fast Thinking. The agent adopts a quick, System- --like heuristic despite expressed uncertainty ("the side length of the larger hexagon is + 2 a + 4? Not sure.") and proceeds with this flawed assumption (S
+ 4). This leads to an incorrect perimeter, 6v3 12, which is physically implausible as it results in a negative value.
Verification. The agent directly confronts this flawed assumption. It explicitly questions the initial logic ("The side length of this larger hexagon might be + 4? Or is it + 2? Wait, reconsider. <... the relationship isn't straightforward..) This critical re-evaluation leads to the correct geometric insight, yielding the relationship = + v3 Based on this corrected understanding. the agent identifies the error in its initial reasoning and conclusion, stating, "The previously given answer was 6/33- - 12. But that would not match. <...> Thus our initial approach is wrong."
Slow Thinking. The agent then leverages the insight from Verification. It explicitly focuses on the "difference in the apothems" to re-derive S = s + v3. successful reasoning trace from Verification. The agent then systematically solves for the side length and calculates the correct perimeter, 18 4v3. Notably, it independently performs numerical check, showcasing a deeper level of deliberation and confidence in its refined answer.
4 This demonstrates clear adoption of the
Summarization. The provided summary effectively distills the core mathematical steps for solving the problem into a clear. concise, and logically consistent sequence. It accurately establishes the relationship between the side lengths of the inner and outer hexagons, correctly formulates the equation for the path's 's area, and finds the pool's perimeter efficiently. Interestingly, it also employs thinking block that reflects certain self-correction patterns observed in earlier steps.
This case highlights the agent's 's capacity for targeted error identification and conceptual correction. The progression shows a clear refinement of reasoning, moving from the System-l-like heuristic in Fast Thinking more rigorous, System-2-like approach in Verification and Slow Thinking. The explicit references between stages-Verificatton Verification critiquing Fast Thinking's 's "initial approach" and Slow Thinking building directly on Verification's 's apothem briggtttunderseen the efficacy of the structured task in fostering coherent, self-correcting thought processes.
C.2
Case Study II: Propagation of Error from Verification to Slow Thinking (Box
This example provides a counterpoint to the previous successful error-correction cases. It demon- strates a scenario in which the agent arrives at an incorrect final answer due to error propagation and insufficient depth in later-stage reasoning.
Fast Thinking. The agent, faced with a complex product of fractional parts, makes a guess. After calculating the first few terms and noting the initial product starts with 2 1, it states: "Since initial term is 2 and product involves fractions potentially leading to 1/2." This leads to the incorrect answer D (5),, driven more by the available options and superficial pattern than rigorous derivation.
Verification. This stage correctly identifies Fast Thinking's answer as likely wrong. However, introduces its own flawed heuristic: "Since each term as_. is less than 1, and multiple such terms multiplied, so perhaps the total product tends zero." While correctly prompts re-evaluation by concluding the original answer was incorrect, its reasoning toward an alternative answer (0) is based on wrong heuristies without considering the specific structure of this sequence (which actually involves a telescoping product that simplifies to d1 leading to a final answer of 1).
ag-1
Slow Thinking. Instead of uncovering the correct algebraic simplification, Slow Thinking explicitly adopts and reinforces the flawed heuristic from Verification. After refining some calculations, it
26

---

reasons: "Alternatively. since the product seems to be approaching zero due to the product of many terms each less than 1. But given the options, perhaps it's 0." This echoes Verification' logic and leads to the incorrect final answer B (0).
This demonstrates how flawed reasoning persist within the Thinker task. While Verification correctly identified Fast Thinking's initial guess as wrong, it unfortunatlly introduced its own plausible but incorrect heuristic. Subsequently, Slow Thinking, anchored on this flawed premise inherited from Verification, performing more calculations around it rather than challenging its validity. This highlights the possibility of an anchoring bias on an incorrect heuristic, and ultimately, superficial deliberation in the later stages that fails to correct the core flawed assumption.
Box C.I: Example Response from Thinker Agent: Hexagon Path Problem
Problem: A path 2 wide and with area of 36 m² has been built around a swimming pool in the shape of regular hexagon. What is the perimeter of the pool?
User: <Fast Thinking Prompt: Answer concisely...>
Assistant:
<Initial Reasoning Trace showing uncertainty about side length>
3. <...> the side length of the larger hexagon is + 2 2
4? Not sure.
<Calculations based on the assumption S = + 4>
13. So 8a = 8/3 16 = a = v3
14. Perimeter of the hexagon 6a 6(v3 2) 6v3 3- 12.
6v3 12
User: <Verification Prompt: Is your answer above correct?>
Assistant:
<Reasoning Trace Verification, questioning the = assumption (a relabeled as s)> The side length of this larger hexagon might be + 4? Or is it 2?
Wait, reconsider. <...> the relationship isn't straightforward.
<Derivation of the correct relationship S
v3 based on apothems>
<Calculations using the corrected S, leading Perimeter = 18 4
The previously given answer was 6v33 12.
But that would not match. So our approach seems incorrect.
<...> Thus our initial approach is wrong. Therefore, the original answer is incorrect.
No
User: <Slow Thinking Prompt: Your initial answer incorrect. Now, think about errors...>
Assistant: <think>
<revisiting the core geometric error from Fast Thinking>
When you add path around the hexagon, the increase in side length can be found by considering the distance from the center. <..>
The path is 2m wide, which I think refers to the distance from the pool to the path, that would mean the difference in the apothems is 2m. So:
S/3
_/3 2
<Re deriviain of S v3' consistent with Verification's 's insight>
= +
<Systematic algebraic solution using the corrected S, leading s 2y3 <alcculation of perimeter = 18 - 4v3, followed by numerical verification>
Hence, the perimeter is 6s = 18 4v3 meters. <...> Yes, that seems consistent.
Final Answer. The perimeter of the pool is 18 4v3 meters.
</think>
<arrative solution in Slow Thinking, presenting the final corrected steps>
27

---

Box C.2: Example Summary from Thinker Agent: Hexagon Path Problem
Problem: A path 2 m wide and with an area of 36 m² has been built around a swimming pool in the shape of regular hexagon. What is the perimeter of the pool?
User: <Summarization Prompt: Summarize concisely...>
Assistant:
<think>
Let the side length of the original hexagonal pool. The path surrounds the pool and forms larger hexagon. The width of the path is 2m, so the side length of the larger hexagon
s 2 (2/ = s + 4/ 3.. Wait, actually, the relationship is more complex.
+
The area of the path is the difference between the area the larger hexagon and the smaller one. The area of regular hexagon (3v3/2) (side length)?.
3. Let be the side length of the pool, then the side length of the larger hexagon is 2+2. = - 33. Wait, no, the correct relationship is that the side length increases by (2 = 4/V3.
4. The area of the path is 36 = (3v3/2) [(s 4/v3)) =
5. Simplifying the equation, we find that = 3- (2V3)/3 The perimeter is 6-s 18 -433.
Thus, the perimeter of the pool is 18 4v3 meters. </think>
Let be the side length of the original hexagonal pool. The path width is 2m. The side length of the larger hexagon (pool + path) is S = s + v3 The area of the path is 36 m², which is the difference between the areas of the larger and smaller hexagons. The area of regular hexagon
4
3v3 2 x (side length)? Therefore:
3v3
4
+
-
8
= 36
2
V3
Simplifying the term inside the brackets:
)
4
88
16
88
16
s
2
52
+
v
v
3
3
Substituting this back into the area equation:
3v3
8s
16
36
v3
Multiplying through by 3v3.
3v3 8s
3v3
16
36
3
128 + 8 = 36
Solving for s:
12s = 36- 8/3
36 8/3
2/3
$
= 3
12
3
The perimeter of the pool is 6s:
2/3
6
18 4/3
Thus, the perimeter of the pool is
18 4v3 meters.
28

---

Box C.3: Example Response from Thinker Agent: Sequence Product Problem
Problem:
Define the sequence
{an
01 = 1, 02 2, On++2 = On + 1, E N4, then
find
dos
Options: A. 2, B. 0, C.
dos
d2
User:
<Fast Thinking Prompt: Answer concisely...>
Assistant:
<Initial computation of sequence terms and first few product terms: [a2/a1] {a3/02)
1/2>
7. Notice that the product of fractional parts potentially telescopes simplifies. However,
it's complex.
8. Given the options A to D, and considering the pattern, likely answer B.0 C.I, but need to think carefully. Since initial term is and product involves fractions potentially leading 1/2.
D
User: <Verification Prompt: Is your answer above correct?>
Assistant:
<Re-evaluation of sequence and product structure, calculating [a2/a1] 2 and [09s/02] R 17>
Wait, count the number of fractional parts: <..> 96 terms.
<..> Since each term {ag/as (a=/ax=1) is less than 1, and multiple such terms multiplied, so perhaps the total product tends to zero.
Thus, the product is very small, so the overall expression would be close to zero. <..> Therefore, the original answer incorrect, the correct answer should be No as the initial answer was incorrect.
<...> now I think the final answer might be B.O.
User: <Slow Thinking Prompt: Your initial answer is incorrect. Now, think about errors...>
Assistant: <think>
<Recalculates sequence terms, [02/ai] 2. Identifies a, Pn-1 and that Fog is even, refining [2] to Fee/2>
Given that each ratio 02/ak-1 approaches the golden ratio Φ Rz 1.618. Thus, each {ax/ax-1 approaches ø - sb = 0.618.
Thus, each term {a//ot (c//ok-1} is about & 0.618.
Thus, product of 96 terms [error in count, should be 97]: 4.
Therefore, product=2 (w))06 floor(Fgs/2)
<..>> Alternatively, since the product seems to be approaching zero due to the product of many terms each less than But given the options, perhaps it's Therefore, the answer is B.O.
B
</think>
<Narative solution, reinforcing the (incorreet) conclusion that the product is 0>
D
Additional Experiments on Larger Models
To assess the scalability of the Thinker task, we fine-tuned DeepSeck-RI-Diee Disttill-Qwen-71 (R7B) [ larger version of the R1.5B model that has also undergone distillation using reasoning data from DeepSeek-RR The same set of hyperparameters and training data from the main experiments were utilized, with the exception of an increased token budget for the verification stage-from 6,000 tokens- to accommodate the more verbose reasoning of larger models. The baseline for this experiment corresponds fine-tuning the R7B model on the standard QA task.
from 2,000 to
The average evaluation accuracy across the seven benchmarks (excluding CollegeMath) during training is shown in Figure TT. The detailed performance of the best checkpoints from both runs can found in Table 7, Similar to the results observed smaller models, we found that agents trained under the Thinker task consistently perform better than those trained on the baseline QA task

---

across all benchmarks. The overall performance improved from 54.41% to 59.09%, representing relative improvement 8.6%. This suggests that larger models also benefit from the Thinker task. Additionally, we note that Thinker-Fas Fast performance improves from 41 41.05% in R1.5B to 47.19% in R7B, demonstrating that the Fast. Thinking mode scales well with model size.
0.65
0.60
Average Accuracy
0.55
0.50
0.45
0.40
0.35
Baseline
Thinker (Final)
0.30
Thinker (Fast)
0.25
200
400
600
800
1k
Training Steps
Figure 11: Evaluation performance of R7B averaged across seven common benchmarks.
Table 7: Performance comparison across various mathematical reasoning benchmarks. Average (Avg.) scores are presented. All scores are Pass@1 accuracy (%) averaged over 16 samples. Top score in each benchmark column bolded. Standard errors are provided Table S
MATH
AIME
AIME GPQA
Olympiad bench
AMC
Minerva College Math
Method
Avg.
500
2024
2025
Diamond
23
Math
(R7B)
Pretrained 84.05
37.50
28.54
17.58
37.92
36.41
34.49
40.72
39.65
Baseline
91.03
47.50
34.58
34.63
56.76
87.81
40.23
42.71
54.41
Thinker 93.04
56.25
41.46
41.51
62.12
91.09
44.39
42.84
59.09
Thinker-Fast
86.47
26.46
21.88
34.12
51.77
71.56
43.08
42.14
47.19
Table Standard error analysis of R7B models on mathematical benchmarks. All scores are in %.
The values are presented score (standard error).
Benchmark
# Questions
Thinker
Thinker-Fast
Baseline
MATH 500
500
93.04 (0.97)
86.47 (1.30)
91.03 (1.06)
AIME 2024
30
56.25 (7.60)
26.46 (6.50)
47.50 (7.20)
AIME 2025
41.46 (7.86)
21.88 (6.83)
34.58 (7.14)
GPQA Diamond
198
41.51 (2.31)
34.12 (2.54)
34.63 (2.43)
Olympiad Bench
675
62.12 (1.64)
51.77 (1.69)
56.76 (1.63)
AMC23
40
91.09 (3.67)
71.56 (6.00)
87.81 (3.96)
Minerva Math
272
44.39 (2.78)
43.08 (2.73)
40.23 (2.71)
College Math
2818
42.84 (0.89)
42.14 (0.89)
42.71 (0.88)
E
Societal Impacts
This research contributes to enhancing the reasoning capabilities of LLMs, which could positively impact areas like scientific advancement and education. By promoting more structured reasoning through the Thinker task, we aim for Al systems that are not only more performant but also potentially more understandable. However, as LLMs become more powerful, remains essential to address potential risks, including misuse and unintended societal consequences, through continued research into AI safety, ethics, and governance.
30


Reviews:
[{"review_id": "o3tNKCAijy", "reviewer": "Reviewer_hUT1", "summary": "The paper presents \"Thinker\", a method of decomposing reasoning tasks into four subtasks, and associated RL training rewards for each subtask. Motivated by Dual Process Theory for human cognition, the subtasks are \"fast thinking\", \"verification\", \"slow thinking\", and summarization.\n\nFast thinking consists of attempting to answer the question with a small token budget (1k tokens). Verification consists of using a large token budget to assess the answer produced by fast thinking. If verification rejects the fast thinking result, slow thinking uses a large token budget to produce a corrected answer. Finally, summarization produces a new short reasoning trace, trained with a reward for producing reasoning which both achieves the same answer as slow thinking, and is assigned high probability under the fast thinking prompt.\n\nModels are trained via PPO to solve math questions. It studies two models (Qwen2.5 1.5B, and DeepSeek-R1-Distill-Qwen-1.5B) and two task settings (“baseline”, i.e. standard QA reward, vs. Thinker subtasks). Thinker is shown to result in larger performance gains than the direct QA task: on Qwen, Thinker improves performance by 24%, while Baseline improves performance by 21%; and on R1, Thinker improves performance by 14%, while Baseline improves performance by 10%.", "questions": "How are the standard errors in appendix B computed? E.g. for the MATH 500 Thinker: if rewards are binary correctness, for the Thinker results, (64% accuracy), I get sqrt(.64\\*(1-.64)/500) ~= 2.1%. If we considered each of the 500\\*16 samples as IID, we’d get sqrt(.64\\*(1-.64)/(500\\*16)) ~= 0.54%; this would still be higher than reported in the paper, though note that this will underestimate standard errors due correlation between samples on the same question (e.g. see “cluster standard errors”: https://www.anthropic.com/research/statistical-approach-to-model-evals)\n\nI searched the code for “error” and “std”, but it wasn’t clear to me where these statistics are computed vs. using std for internal model normalization.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1) The paper is clearly presented, and the method is described well.\n2) Evaluations are performed on 7 datasets, consisting of 6 math benchmarks plus GPQA Diamond.\n3) The paper presents some interesting qualitative analysis; I found it interesting that “reflection patterns” seem to increase over training for both Baseline and Thinker, despite much more flat total response length.\n\n**Weaknesses**:\n\nI’m concerned about how much we can infer from the presented results. IIUC the paper's conclusions stem from 5 total PPO RL runs:\n* R1 1.5B, Thinker vs Baseline\n* Qwen 1.5B, Thinker vs. Baseline\n* Qwen 1.5B, Thinker without Summarization\n\nThe paper does provide CIs with respect to dataset examples (though for readability these should ideally be included in the main table, rather than separately in the appendix). However, with only single RL runs for each setting, we have no way of estimating variation due to RL training dynamics. E.g. figure 6 shows divergence in “reflection pattern occurrence”; is this actually caused by differences in the loss, or simply that Thinker without Summarization happened by chance to produce some high-quality reflection patterns, and so settled into a different but equally effective writing style?\n\nThe average performance improvements over the baseline are relatively minor (3% and 4%); to adequately support the paper's claim and justify the substantial increase in method complexity, the paper should compare multiple RL runs for each setting (each using different seeds for dataset shuffling and LLM sampling), and show statistical significance between baseline and thinker with respect to variation in RL runs.", "was_revised": true, "final_justification": "Overall rating updated 2->4, quality 2->3, significance 2->3. The authors' additional experiments and analysis provide good evidence that their results are not merely due to chance:\n\n1) They performed one of their RL runs with 3 independent seeds, demonstrating that the variation between seeds is small compared to the variation between runs. This addressed my most critical concern, that the headline results could be simply due to randomness. (While it would be preferable to have multiple seeds for each parameter setting, in order to rule out heteroskedasticiy, I don't have a strong reason to expect heteroskedasticity a priori.)\n2) They conducted more thorough statistical analysis, showing on which datasets improvement is significant (treating dataset examples as the random variable).\n\nAs such, I believe this paper provides value with its approach for improving model reasoning.", "rebuttal": "We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below.\n\n**Response to Weakness 1: Variance of Performance and Number of Experiments**\n\n*New Multi-Seed Experiments*\n\nWe agree with the reviewer that assessing the variance from RL training dynamics is important for validating the benefits of the proposed method. To address this concern, we have conducted new experiments for our main result (Q1.5B Thinker vs. Baseline), running each method with three different random seeds. The aggregated results are as follows:\n\n| Method            | MATH500 | AIME24 | AIME2025 | GPQA Diamond | Olympiad Bench | AMC23 | Minerva Math | College Math | Average |\n| ----------------- | ------- | ------ | -------- | ------------ | -------------- | ----- | ------------ | ------------ | ------- |\n| Baseline (Seed 1) | 57.98   | 3.33   | 3.33     | 21.46        | 24.54          | 34.38 | 17.78        | 36.21        | 24.88   |\n| Baseline (Seed 2) | 59.26   | 4.58   | 1.46     | 19.57        | 25.05          | 35.62 | 18.43        | 38.40        | 25.30   |\n| Baseline (Seed 3) | 58.44   | 2.92   | 1.04     | 19.73        | 26.52          | 35.47 | 18.84        | 38.39        | 25.17   |\n| ORZ [1]           | 58.00   | 3.50   | 1.00     | 16.80        | -              | -     | -            | -            | -       |\n| SimpleRL [2]      | 59.00   | 4.20   | -        | -            | 21.00          | 35.00 | 20.20        | -            | -       |\n| RL on DSR [3]     | 57.20   | 5.00   | 0.80     | -            | 21.20          | 30.30 | 17.60        | -            | -       |\n| Thinker (Seed 1) | 64.25 | 6.25 | 2.50 |  23.74 | 28.11 | 40.62 | 19.03 | 38.33 | 27.85 |\n| Thinker (Seed 2) | 64.53 | 6.88 | 2.29 | 19.54 | 28.03 | 39.06 | 19.44 | 38.75 | 27.31 |\n| Thinker (Seed 3) | 63.76 | 5.83 | 0.83 | 21.31 | 27.80 | 40.31 | 18.75 | 37.94 | 27.07 |\n\nWe note that our baseline results are consistent with those reported by similar methods like ORZ [1], SimpleRL [2], and RL on DSR [3], which all performed RL training on the same Q1.5B model with a standard QA task, similar to our baseline. From the above table, we observe Thinker consistently outperforms the baselines. An independent samples t-test showed that Thinker (M = 27.41, SD = 0.40) performed statistically significantly better than the baseline (M = 25.12, SD = 0.22), t(4) = 8.76, p < .001. Additional comparisons between Thinker and four concurrent works with more advanced methods are provided in our response to Reviewer fcXk. \n\nTo address the reviewer's question about the \"reflection pattern occurrence\" (Figure 6), we analyzed this metric across our new multi-seed runs. The average scores, both over all training steps (steps 0-1200) and during the final training steps (steps 1100-1200), are presented below:\n\n| Horizon              | Thinker (Seed 1) | Thinker (Seed 2) | Thinker (Seed 3) | Thinker without Summarization |\n| -------------------- | ---------------- | ---------------- | ---------------- | ----------------------------- |\n| All Training Steps   | 0.01380          | 0.01686          | 0.01774          | 0.03031                       |\n| Final Training Steps | 0.00897          | 0.02643          | 0.01116          | 0.03124                       |\n\nAcross all three seeds, we observe that Thinker models consistently exhibit lower reflection pattern scores compared to the variant without the summarization subtask. This supports our hypothesis that the summarization step encourages the agent to develop a more concise chain-of-thought, thereby reducing reflective or repetitive language. However, as noted by the reviewer, it remains possible that the `Thinker without Summarization` may have converged to a different local optimum with inherently lower reflection patterns. Addressing this possibility would require additional seeds for `Thinker without Summarization`, which are currently beyond our computational budget. Nonetheless, the results presented in the table indicate that Thinker is less likely to settle into policies characterized by high reflection patterns, which we believe sufficiently demonstrates the benefits of incorporating the summarization step.\n\n*Additional Experiments*\n\nAt the time of submission, the R7B runs were ongoing due to computational constraints, which is why their preliminary results were placed in the appendix. We have since completed these experiments and present the full results below:\n\n| Method       | MATH500   | AIME24    | AIME2025  | GPQA Diamond | Olympiad Bench | AMC23     | Minerva Math | College Math | Average   |\n| ------------ | --------- | --------- | --------- | ------------ | -------------- | --------- | ------------ | ------------ | --------- |\n| Pretrained   | 84.05     | 37.50     | 28.54     | 17.58        | 37.92          | 36.41     | 34.49        | 40.72        | 39.65     |\n| Baseline     | 91.03     | 47.50     | 34.58     | 34.63        | 56.76          | 87.81     | 40.23        | 42.71        | 54.41     |\n| Thinker      | **93.04** | **56.25** | **41.46** | **41.51**    | **62.12**      | **91.09** | **44.39**    | **43.29**    | **59.14** |\n| Thinker-Fast | 86.48     | 26.46     | 21.88     | 34.12        | 51.77          | 71.56     | 43.08        | 42.58        | 47.24     |\n\nAs shown, Thinker demonstrates consistent performance improvements over the baseline on the R7B model. This result is also consistent with our findings on smaller models and suggests that our approach scales effectively.\n\nFurthermore, to address potential concerns about method efficiency, we have also conducted two additional experiments (detailed in our response to Reviewer fcXk) showing that the token budget for the verification step can be significantly reduced while maintaining performance, thereby improving Thinker's efficiency.\n\nIn total, these new experiments bring the number of PPO runs presented to 13: \n\n- Q1.5B: Thinker vs. Baseline (3 seeds per run)\n- Q1.5B: Thinker without summarization\n- R1.5B: Thinker vs. Baseline\n- R1.5B: Thinker with short verification\n- R7B: Thinker vs. Baseline\n- R7B: Thinker with short verification\n\nEach PPO run is computationally intensive, requiring approximately 960 A100-hours (8 GPUs for 5 days). The experiments presented here, including the new runs, represent over 10,000 GPU-hours. Given these substantial costs and our limited computational budget, we have focused our resources on the most important experiments. We hope this comprehensive evaluation is sufficient to address the concerns about statistical robustness.\n\nWe will incorporate all new results and analyses into the revised manuscript.\n\n**Response to Question 1: Calculation of Standard Errors**\n\nThe reviewer is correct that the method for calculating standard errors was not explicitly defined. We apologize for this oversight.\n\nThe standard error reported in Appendix B is not calculated over the questions in the dataset, but rather over 16 independent samples for a given benchmark. The procedure is as follows:\n\n1. For a single model, we generate 16 samples per question from the benchmark with a temperature of 1.0.\n2. This process yields 16 aggregate accuracy scores for the benchmark.\n3. The standard error is then calculated over these 16 accuracy scores.\n\nThe purpose of this metric is to quantify the sampling variance of a single trained model (particularly relevant for benchmarks with a limited number of questions, such as AIME), which is distinct from the training variance observed across different RL runs. Our original intent was for this to be conveyed in the caption of Table 1, by placing the reference to the standard error after the sentence: “All scores are Pass@1 accuracy (%) averaged over 16 samples.” However, we now recognize that this implicit connection was too subtle. In the revised paper, we will include a clear and explicit definition of the standard error.\n\nWe thank the reviewer again for their valuable feedback. We hope our new experiments sufficiently address the concerns about training variance. We would be grateful if the reviewer would consider this new evidence in their final assessment. We welcome any further questions.\n\n[1] Hu, J., Zhang, Y., Han, Q., Jiang, D., Zhang, X., & Shum, H. Y. (2025). Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. *arXiv preprint arXiv:2503.24290*.\n\n[2] Zeng, W., Huang, Y., Liu, Q., Liu, W., He, K., Ma, Z., & He, J. (2025). Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. *arXiv preprint arXiv:2503.18892*.\n\n[3] Wang, Y., Yang, Q., Zeng, Z., Ren, L., Liu, L., Peng, B., ... & Shen, Y. (2025). Reinforcement learning for reasoning in large language models with one training example. *arXiv preprint arXiv:2504.20571*."}, {"review_id": "10i4tTVyyT", "reviewer": "Reviewer_fcXk", "summary": "This paper proposes the \"Thinker\" task, a four-stage RL framework for LLMs (Fast Thinking, Verification, Slow Thinking, Summarization) inspired by Dual Process Theory. By decomposing QA into structured stages with distinct rewards, the method aims to enhance intuition, evaluation, refinement, and integration capabilities. Experiments on Qwen2.5-1.5B and DeepSeek-R1-Distill-Qwen-1.5B show significant accuracy gains across math benchmarks, with Fast Thinking achieving efficiency via strict token budgets. The approach reduces redundant self-reflection but yields longer responses due to verification iterations.", "questions": "Why does Thinker achieve higher accuracy despite generating longer responses than standard QA? For instance, DeepSeek-R1-Distill-Qwen-1.5B’s responses increase in token length, yet MATH500 accuracy rises by 8.5%. Is this due to Verification’s iterative self-checks or Slow Thinking’s deep refinement?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strength\n\n1. Inspired by human cognitive systems (System 1/2), Thinker explicitly trains intuition (Fast Thinking) and deliberation (Slow Thinking), forming a virtuous loop where fast responses guide slow refinement and vice versa. This structured design aligns with psychological theories, providing a principled RL framework. \n\n2. Each stage’s reward (e.g., binary for fast thinking, weighted for verification) targets specific capabilities, enabling precise credit assignment. \n\n# Weaknesses\n\n1. In Section 5.3, the authors observe that although the Thinker task exhibits fewer reflection patterns, the verification step nonetheless produces longer responses. This phenomenon stems primarily from repeated self-verification cycles incentivized by the answer extraction strategy, which focuses solely on the final boxed output. This indicates a redistribution of inefficiency rather than a net reduction, potentially compromising practical inference gains in certain scenarios.", "was_revised": true, "final_justification": "The author addressed my concerns, especially the performance improvement on short data in the long context setting. I have adjusted my score accordingly.", "rebuttal": "We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below.\n\n**Response to Weakness 1: Inefficiency During Inference**\n\nWe acknowledge that one limitation of our method is that longer verification step is unavoidable during inference in full deployment mode (note: in fast deployment mode, the verification step is not needed though). This can indeed lead to longer response lengths for more complex tasks. To mitigate this issue, we conducted an additional experiment on R1.5B and R7B models by shortening the token budget in the verification step from 6k tokens to 2k tokens, and found that performance can be maintained while significantly reducing the number of tokens used. We present results for this \"short verification\" model in both its full (`Thinker (short verification)`) and fast (`Thinker-Fast (short verification)`) deployment modes:\n\n*Average Accuracy for Finetuned R1.5B Models*\n\n| Method | MATH500 | AIME24 | AIME2025 | GPQA Diamond | Olympiad Bench | AMC23 | Minerva Math | College Math | Average |\n| --------------------------------- | --------- | --------- | --------- | ------------ | -------------- | --------- | ------------ | ------------ | --------- |\n| Baseline | 86.24 | 35.42 | 23.75 | 25.69 | 49.22 | 72.81 | 32.08 | 42.02 | 45.90 |\n| Thinker | 87.02 | 35.62 | **27.71** | 36.08 | 54.21 | 81.72 | 33.23 | **42.77** | 49.80 |\n| Thinker-Fast | 77.21 | 11.46 | 11.46 | 30.08 | 40.39 | 59.22 | 29.23 | 41.37 | 37.55 |\n| Thinker (Short Verification) | **88.51** | **38.96** | 26.67 | **37.41** | **55.49** | **83.59** | **34.77** | 42.44 | **50.98** |\n| Thinker-Fast (Short Verification) | 80.90 | 18.12 | 12.71 | 28.69 | 44.38 | 66.88 | 31.20 | 41.57 | 40.56 |\n\n*Response Length (in Token) for Finetuned R1.5B Models*\n\n| Method | MATH500 | AIME24 | AIME2025 | GPQA Diamond | Olympiad Bench | AMC23 | Minerva Math | College Math | Average |\n| --------------------------------- | ------- | ------ | -------- | ------------ | -------------- | ------ | ------------ | ------------ | ------- |\n| Baseline | 2780.1 | 5777.7 | 5694.1 | 3815.9 | 4369.3 | 3938.2 | 3505.2 | 2769.0 | 4081.2 |\n| Thinker | 4061.0 | 8147.8 | 7819.5 | 6902.6 | 6128.0 | 5708.9 | 5883.7 | 3861.0 | 6064.1 |\n| Thinker-Fast | 648.5 | 960.2 | 919.9 | 732.8 | 843.7 | 833.1 | 720.7 | 616.8 | 784.5 |\n| Thinker (Short Verification) | 2501.0 | 5597.4 | 5355.7 | 5008.4 | 4060.8 | 3517.1 | 3976.7 | 2445.9 | 4057.9 |\n| Thinker-Fast (Short Verification) | 600.3 | 853.1 | 813.2 | 726.2 | 745.6 | 751.0 | 709.8 | 559.9 | 719.9 |\n\n*Average Accuracy for Finetuned R7B Models*\n\n| Method | MATH500 | AIME24 | AIME2025 | GPQA Diamond | Olympiad Bench | AMC23 | Minerva Math | College Math | Average |\n| --------------------------------- | --------- | --------- | --------- | ------------ | -------------- | --------- | ------------ | ------------ | --------- |\n| Baseline | 91.03 | 47.50 | 34.58 | 34.63 | 56.76 | 87.81 | 40.23 | 42.71 | 54.41 |\n| Thinker | **93.04** | 56.25 | **41.46** | **41.51** | **62.12** | **91.09** | **44.39** | **43.29** | **59.14** |\n| Thinker-Fast | 86.48 | 26.46 | 21.88 | 34.12 | 51.77 | 71.56 | 43.08 | 42.58 | 47.24 |\n| Thinker (Short Verification) | 92.16 | **56.46** | 36.67 | 41.07 | 61.80 | 88.91 | 44.26 | 42.91 | 58.03 |\n| Thinker-Fast (Short Verification) | 86.49 | 27.08 | 22.71 | 33.05 | 51.30 | 72.66 | 42.10 | 42.24 | 47.20 |\n\n*Response Length (in Token) for Finetuned R7B Models*\n\n| Method | MATH500 | AIME24 | AIME2025 | GPQA Diamond | Olympiad Bench | AMC23 | Minerva Math | College Math | Average |\n| --------------------------------- | ------- | ------ | -------- | ------------ | -------------- | ------ | ------------ | ------------ | ------- |\n| Baseline | 2313.3 | 5517.6 | 5569.4 | 3567.7 | 3896.0 | 3453.9 | 2950.3 | 2224.5 | 3686.6 |\n| Thinker | 2366.3 | 6736.2 | 6640.3 | 5416.0 | 4195.3 | 3535.9 | 3756.3 | 2171.0 | 4352.2 |\n| Thinker-Fast | 502.0 | 766.8 | 734.1 | 681.5 | 640.4 | 626.2 | 599.0 | 516.9 | 633.4 |\n| Thinker (Short Verification) | 1504.7 | 4495.2 | 4601.0 | 3903.5 | 2762.4 | 2321.8 | 2274.3 | 1398.0 | 2907.6 |\n| Thinker-Fast (Short Verification) | 500.2 | 800.3 | 746.5 | 662.8 | 645.9 | 638.7 | 592.4 | 504.3 | 636.4 |\n\nWe have also identified several concurrent works—ThinkPrune [1], ConciseRL [2], Structured Reasoning [3], and AdaptThink [4]—that also apply RL to improve mathematical reasoning on R1.5B with more advanced methods:\n\n*Comparison with Concurrent Works Finetuning R1.5B Models*\n\n| Method | MATH500 Accuracy | MATH500 Length | AIME24 Accuracy | AIME24 Length | AMC23 Accuracy | AMC23 Length |\n| --------------------------------- | ---------------- | -------------- | --------------- | ------------- | -------------- | ------------ |\n| ThinkPrune [1] | 83.2 | 1938 | 27.1 | 5631 | 73.2 | 3039 |\n| ConciseRL [2] | 81.0 | 1965 | 30.0 | 6752 | 69.4 | 2936 |\n| SR-FLOW [3] | 85.3 | n.a. | 36.7 | n.a. | 77.8 | n.a. |\n| AdaptThink [4] | 82.0 | 1782 | 31.0 | 6679 | n.a. | n.a. |\n| Baseline | 86.2 | 2780 | 35.4 | 5778 | 72.8 | 3938 |\n| Thinker | 87.0 | 4061 | 35.6 | 8148 | 81.7 | 5709 |\n| Thinker-Fast | 77.2 | 649 | 11.5 | 960 | 59.2 | 833 |\n| Thinker (short verification) | **88.5** | 2501 | **39.0** | 5597 | **83.6** | 3517 |\n| Thinker-Fast (short verification) | 80.9 | 600 | 18.1 | 853 | 66.9 | 751 |\n\nSeveral observations can be made from the results above:\n\n1. Shortening the verification tokens from 6k to 2k has little impact on final accuracy—showing a slight increase for the R1.5B model and a slight decrease for the R7B model. However, the total token length can be significantly reduced in full deployment mode.\n2. Thinker (short verification) R1.5B models outperform other concurrent works in terms of accuracy while remaining competitive in terms of token length.\n3. Thinker-Fast (short verification) R1.5B models use significantly fewer tokens than all concurrent works, while maintaining slightly worse performance on simpler benchmarks (MATH500 and AMC23), though with significantly worse performance on complex tasks (AIME24).\n\nAn additional advantage of Thinker—beyond the metrics shown above—is that users can choose the deployment mode based on task demands and computational cost constraints (e.g., fast mode for simpler tasks and full mode for complex ones) without training two separate models. In summary, Thinker offers a flexible trade-off while maintaining competitive or even superior performance compared to similar methods. \n\nThe above new results on short verification and the additional baselines will be included in the revised version of the main paper.\n\n**Response to Question 1: Thinker achieving higher accuracy despite generating longer responses than standard QA**\n\nIn general, longer responses are usually associated with higher accuracy, as the model has more tokens to perform reasoning. The DeepSeek R1 paper [5] shows that both response length and accuracy increase during training. The increase in token length during training for both the baseline and Thinker models, as shown in Fig. 5b, is consistent with this trend. Manual inspection reveals that the models generally learn to perform more self-correction during training—e.g., \"Wait, I may be wrong here, let me recheck\"—for both the baseline and Thinker models. This behavior aligns with the final reward being based on the correctness of the final answer, giving the agent an incentive to perform more verification and try additional approaches when there is room to use more tokens.\n\nWe thank the reviewer again for their valuable feedback. We believe these new baseline comparisons, along with our new results, substantially strengthen the paper. We would be grateful if the reviewer would consider this new evidence and our responses in their final assessment. We welcome any further questions.\n\n[1] Hou, B., Zhang, Y., Ji, J., Liu, Y., Qian, K., Andreas, J., & Chang, S. (2025). Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. *arXiv preprint arXiv:2504.01296*.\n\n[2] Dumitru, R. G., Peteleaza, D., Yadav, V., & Pan, L. (2025). ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models. *arXiv preprint arXiv:2505.17250*.\n\n[3] Dong, Y., & Fan, H. (2025). Enhancing Large Language Models through Structured Reasoning. *arXiv preprint arXiv:2506.20241*.\n\n[4] Zhang, J., Lin, N., Hou, L., Feng, L., & Li, J. (2025). Adaptthink: Reasoning models can learn when to think. *arXiv preprint arXiv:2505.13417*.\n\n[5] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. *arXiv preprint arXiv:2501.12948*."}, {"review_id": "c5HlKkRqdS", "reviewer": "Reviewer_1C42", "summary": "Inspired by the Dual Process Theory, the paper proposes an approach to train a model to infer using two modes of thinking (fast and slow) in the question-answering setting. Specifically, the model is trained to take a 4-step variant of CoT paradigm, involving a fast thinking as the first step which predicts the answer within a short output token budget. The second verification step check the answer and proceeds to the slow thinking step if the fast thinking answer is verified to be incorrect. The slow thinking step involves a longer output token budget. The final summarization step provides the final answer. Experiments were conducted on an array of math reasoning QA benchmarks to show the superior performance of proposed approach compared to baseline methods. Further analyses were conducted to understand the effect of training on response length and model’s reflection patterns.", "questions": "The proposed approach seems to have two tries in answering a question (one from fast and slow if fast thinking is verified as incorrect). However, most baseline methods would only have one try in answering the questions. It seems it is fairer to compare with pass@2 metrics of baseline methods where there are two tries to answer a particular question.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nProposed approach is shown to improve math reasoning performance while reducing compute with the fast thinking mode.\nThe paper is well written and easy to follow\n\nWeaknesses:\nExperiments are limited to math reasoning tasks, applicability to other domains and tasks is yet to be supported by experiments", "was_revised": true, "final_justification": "The authors have answered my questions and increased my confidence of the evaluation.", "rebuttal": "We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below.\n\n**Response to Weakness 1: Limitation to Math Reasoning Tasks**\n\nThe primary focus of this paper is on improving the math reasoning capabilities of LLMs via RL, which is a highly active and focused research area. As evidence of this focus within the community, a recent survey [1] lists multiple concurrent papers dedicated specifically to math reasoning. We are hopeful that our work can contribute to this research frontier.\n\nNonetheless, we agree that demonstrating broader applicability is valuable. We have performed an evaluation of our models on a non-mathematical benchmark, GPQA Diamond, which consists of 448 challenging multiple-choice questions written by domain experts in biology, physics, and chemistry [2]. Despite being trained exclusively on math questions, our Thinker models also perform well on GPQA Diamond compared to both the baseline and the original pre-trained models (Table 1 in the main paper). This promising generalization performance leaves room for further exploration in future work, which we will acknowledge in the paper.\n\n**Response to Question 1: Two Tries Per Question**\n\nOur Thinker agent does not get two attempts at providing a final answer during inference. The process, as depicted in Figure 2 of the main paper, is as follows:\n\n1. The model generates a fast answer in the fast thinking step.\n2. The model verifies if the fast answer is correct in the verification step. This is a step internal to the model's own reasoning process and does not rely on any external verifier.\n3. If the model verifies the fast answer as correct in the verification step, then the fast answer is defined as the final answer. Otherwise, the model goes into the slow thinking step, and the final answer will be the slow answer given in the slow thinking step.\n\nCrucially, this process results in only one final answer being produced for each question. Therefore, the comparison with baselines is fair, as both our method and the baselines submit only a single, final answer for each question (i.e., the evaluation is Pass@1 for all methods). We will explicitly update the experimental setup section in our paper to make this single-attempt evaluation process clearer.\n\nWe thank the reviewer again for their valuable feedback. We hope this clarification addresses the main concern regarding evaluation fairness. We believe that with this resolved, the strengths of the paper as noted by the reviewer—its strong empirical performance and novel approach—solidify its contribution. We would be grateful if the reviewer would consider this clarification in their final assessment and are happy to answer any further questions.\n\n[1] Li, Z. Z., Zhang, D., Zhang, M. L., Zhang, J., Liu, Z., Yao, Y., ... & Liu, C. L. (2025). From system 1 to system 2: A survey of reasoning large language models. *arXiv preprint arXiv:2502.17419.*\n\n[2] Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., ... & Bowman, S. R. (2024, July). Gpqa: A graduate-level google-proof q&a benchmark. *In First Conference on Language Modeling.*"}, {"review_id": "207yZIK9SE", "reviewer": "Reviewer_5wyw", "summary": "This paper focuses on improving the reasoning capabilities of large language models (LLMs). One representative model is DeepSeek R1, which performs well but relies heavily on backtracking and verification, often resulting in long and redundant responses. To enhance efficiency, the paper proposes a dual-process framework inspired by cognitive psychology. The method involves four steps: the agent first generates an initial answer rapidly using a small token budget, then evaluates it with a larger token budget. If the answer is deemed incorrect, the agent produces a revised response using more resources. During training, a summarization step is introduced to strengthen the model’s fast-thinking ability. Empirically, the method is evaluated on several mathematical reasoning benchmarks, showing improved accuracy over baseline approaches.", "questions": "- On GPQA, why does *Thinker-Fast* outperform the full *Thinker* model? What insights can be drawn from this result?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- Improving the reasoning abilities of LLMs through reinforcement learning is a highly active and relevant research direction, especially following the success of DeepSeek R1. This paper builds upon that line of work by decoupling reasoning into two complementary systems: a *fast thinking* component for intuitive response generation and a *slow thinking* component for verification and deliberation. The motivation is grounded in cognitive psychology, and the empirical results demonstrate the effectiveness of the proposed dual-process framework.\n\n- The paper includes thorough ablation studies and detailed analyses to validate the contributions of each component in the model design.\n\n**Weaknesses:**\n\n- It would be helpful to compare the proposed method with simpler prompting-based approaches such as *Self-Refine* or *Reflexion*, which also aim to improve reasoning through iterative refinement.\n\n- The experiments are limited to Proximal Policy Optimization (PPO). It would strengthen the paper to evaluate how the approach performs with more advanced or stabilized RL algorithms such as Gradient-Regularized Policy Optimization (GRPO).\n\n- In Figure 4(a), there appears to be some overfitting in the baseline methods. Additionally, the performance fluctuation of the proposed method after 1k training steps raises concerns about stability. It is unclear whether the observed improvement is statistically significant or due to noise.\n\n- While I am not deeply familiar with all baselines in this area, I believe there are additional recent works that apply RL to enhance LLM reasoning. A broader baseline comparison would help clarify the significance of the proposed approach.", "was_revised": true, "final_justification": "The author has addressed my questions in the rebuttal. So I think a weak accept is appropriate for this paper.", "rebuttal": "We thank the reviewer for their constructive feedback and detailed comments. We address the reviewer's concerns below.\n\n**Response to Weakness 1&4: Additional Baselines**\n\nWe agree that comparing against additional baselines is valuable. We have identified several concurrent works—ThinkPrune [1], ConciseRL [2], Structured Reasoning [3], and AdaptThink [4]—that also apply RL to improve mathematical reasoning on R1.5B. \n\n*Comparison with Concurrent Works Finetuning R1.5B Models*\n\n| Method                            | MATH500 Accuracy | MATH500 Length | AIME24 Accuracy | AIME24 Length | AMC23 Accuracy | AMC23 Length |\n| --------------------------------- | ---------------- | -------------- | --------------- | ------------- | -------------- | ------------ |\n| ThinkPrune [1]                    | 83.2             | 1938           | 27.1            | 5631          | 73.2           | 3039         |\n| ConciseRL [2]                     | 81.0             | 1965           | 30.0            | 6752          | 69.4           | 2936         |\n| SR-FLOW [3]                       | 85.3             | n.a.           | 36.7            | n.a.          | 77.8           | n.a.         |\n| AdaptThink [4]                    | 82.0             | 1782           | 31.0            | 6679          | n.a.           | n.a.         |\n| Baseline                          | 86.2             | 2780           | 35.4            | 5778          | 72.8           | 3938         |\n| Thinker                           | 87.0             | 4061           | 35.6            | 8148          | 81.7           | 5709         |\n| Thinker-Fast                      | 77.2             | 649            | 11.5            | 960           | 59.2           | 833          |\n| Thinker (short verification)      | **88.5**         | 2501           | **39.0**        | 5597          | **83.6**       | 3517         |\n| Thinker-Fast (short verification) | 80.9             | 600            | 18.1            | 853           | 66.9           | 751          |\n\nThe table above compares our method against these new baselines. To explore efficiency, we also trained a variant model using a shorter verification step (2k tokens instead of 6k). We present results for this \"short verification\" model in both its full (`Thinker (short verification)`) and fast (`Thinker-Fast (short verification)`) deployment modes.\n\nSeveral observations can be made from the results above:\n\n1. Shortening the verification tokens from 6k to 2k has little impact on final accuracy—showing a slight increase for the R1.5B model. However, the total token length can be significantly reduced in full deployment mode.\n2. Thinker (short verification) R1.5B models outperform other concurrent works in terms of accuracy while remaining competitive in terms of token length.\n3. Thinker-Fast (short verification) R1.5B models use significantly fewer tokens than all concurrent works, while maintaining slightly worse performance on simpler benchmarks (MATH500 and AMC23), though with significantly worse performance on complex tasks (AIME24).\n\nAn additional advantage of Thinker—beyond the metrics shown above—is that users can choose the deployment mode based on task demands and computational cost constraints (e.g., fast mode for simpler tasks and full mode for complex ones) without training two separate models. In summary, Thinker offers a flexible trade-off while maintaining competitive or even superior performance compared to similar methods. \n\nThe above new results on short verification and the additional baselines will be included in the revised version of the main paper.\n\nRegarding simpler prompting-based methods like Self-Refine or Reflexion, we believe a direct comparison is challenging. First, while the effectiveness of these methods has been demonstrated on large, instruction-tuned models like GPT-4, they would likely perform poorly on our smaller base models, which are not instruction-tuned. Second, a direct comparison of the final outputs would be misleading due to the vast difference in computational cost: these prompting strategies are inference-time techniques requiring no finetuning, while our methods and baselines involve substantial RL training. One might suggest resolving these issues by applying RL to these prompting methods; however, this requires designing a good reward signal, which is a significant research topic in its own right. For these reasons, we consider this comparison outside the scope of our current work.\n\n**Response to Weakness 2: GRPO vs. PPO**\n\nWe chose PPO as our primary RL algorithm because prior work [5, 6] showed that GRPO can be unstable for finetuning LLMs, sometimes leading to performance collapse. We also conducted preliminary experiments finetuning our models on the standard QA task using GRPO and confirmed the instability analyzed in [5, 6]. Given these findings, we focused our efforts on PPO, which provided more stable and reliable results.\n\n**Response to Weakness 3: Overfitting in the Baseline**\n\nWe select the final model for evaluation based on its peak performance on a held-out validation set (i.e., early stopping). For example, the results for the Qwen 1.5B baseline reported in Table 1 are from the checkpoint at training step 900, before the performance degradation occurred.\n\nRegarding statistical significance, we have conducted new multi-seed experiments to assess the variance from RL training. Please refer to our response to Reviewer hUT1 for a detailed presentation of these new results.\n\n**Response to Question 1: Thinker-Fast performing better on GPQA**\n\nWe believe this interesting phenomenon is due to GPQA's nature as a knowledge-intensive science benchmark. Unlike our other benchmarks which focus on mathematical reasoning, GPQA is knowledge-intensive, covering biology, physics, and chemistry. Our primary hypothesis is that the fast thinking step helps to mitigate hallucinations on such tasks. This hypothesis is guided by prior work [7], which suggests that long reasoning chains trained via RL can lead to a higher degree of hallucination. While the long thinking process used by the full Thinker model and our baselines is beneficial for math, it may become a vulnerability on a knowledge-based benchmark like GPQA. By explicitly constraining the generation to a short format, Thinker-Fast limits the model's opportunity for flawed reasoning and forces it to rely more directly on its pre-trained knowledge.\n\nThis also explains why Thinker-Fast may outperform the full Thinker mode on this specific task. The deployment of the subsequent verification and slow thinking steps in the full mode, while allowing more room for reasoning, appears to partially negate the anti-hallucination benefit of the fast thinking step. The advantage is not entirely lost, however, as the full Thinker mode still outperforms the baseline on GPQA. We are actively exploring the implications of this finding for controlling hallucination in RL-finetuned LLMs.\n\nWe thank the reviewer again for their valuable feedback. We believe these new baseline comparisons, along with our clarifications on the training process and algorithm choices, substantially strengthen the paper. We would be grateful if the reviewer would consider this new evidence and our responses in their final assessment. We welcome any further questions.\n\n[1] Hou, B., Zhang, Y., Ji, J., Liu, Y., Qian, K., Andreas, J., & Chang, S. (2025). Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. *arXiv preprint arXiv:2504.01296*.\n\n[2] Dumitru, R. G., Peteleaza, D., Yadav, V., & Pan, L. (2025). ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models. *arXiv preprint arXiv:2505.17250*.\n\n[3] Dong, Y., & Fan, H. (2025). Enhancing Large Language Models through Structured Reasoning. *arXiv preprint arXiv:2506.20241*.\n\n[4] Zhang, J., Lin, N., Hou, L., Feng, L., & Li, J. (2025). Adaptthink: Reasoning models can learn when to think. *arXiv preprint arXiv:2505.13417*.\n\n[5] Hu, J., Zhang, Y., Han, Q., Jiang, D., Zhang, X., & Shum, H. Y. (2025). Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. *arXiv preprint arXiv:2503.24290*.\n\n[6] Rajani, N., Gema, A. P., Goldfarb-Tarrant, S., & Titov, I. (2025). Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them. *arXiv preprint arXiv:2507.10616*.\n\n[7] Li, J., & Ng, H. T. (2025). The Hallucination Dilemma: Factuality-Aware Reinforcement Learning for Large Reasoning Models. *arXiv preprint arXiv:2505.24630.*"}]

Author rebuttal:

Meta-review: **Summary:** This paper proposes Thinker, a reinforcement learning framework for enhancing mathematical reasoning in large language models by explicitly decoupling reasoning into two components: fast thinking (intuitive generation) and slow thinking (deliberation and verification). Inspired by cognitive psychology (System 1 vs. System 2), the approach uses distinct reward structures to guide each stage—binary rewards for fast thinking and weighted verification for slow thinking—thereby enabling more precise credit assignment. Experiments are conducted across six math reasoning benchmarks plus GPQA Diamond, with qualitative analyses of reflection patterns and ablation studies. The results indicate modest but consistent improvements (≈3–4%) over strong baselines, alongside evidence of differentiated reasoning behaviors.

**Strengths:**

- Principled design: The dual-process (fast vs. slow) structure is well motivated by cognitive psychology and provides a systematic RL framework for reasoning.

- Technical novelty: Clear reward decomposition and role assignment across different stages of reasoning.

- Thorough analysis: Includes qualitative analyses (reflection patterns, response length dynamics) and ablation studies to illustrate component contributions.

**Weaknesses:**

- Statistical robustness: As raised by Reviewer hUT1, the main results rely on only single RL runs per setting. Without multiple seeds, it is unclear whether improvements are statistically significant or an artifact of stochastic training dynamics.

- Limited gains: The performance improvements (3–4%) are relatively modest compared to the added complexity of the framework.

- Restricted evaluation: Experiments are limited to math reasoning tasks, leaving generalizability to other domains unexplored.

- Efficiency concerns: While fast thinking is designed to reduce compute, slow thinking often produces longer responses, raising doubts about real inference efficiency.

- Training stability: Some figures (e.g., fluctuations after 1k steps) suggest instability, but this is not deeply analyzed.

My recommendation is acceptence after reading the reviews and the rebuttal.
Decision: Accept (poster)