Title: Thinker: Learning to Think Fast and Slow
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Reasoning

Large Language Models

Dual Process Theory
Abstract: Recent studies show that the reasoning capabilities of Large Language Models (LLMs) can be improved by applying Reinforcement Learning (RL) to question-answering (QA) tasks in areas such as math and coding. With a long context length, LLMs may learn to perform search, as indicated by the self-correction behavior observed in DeepSeek R1. However, this search behavior is often imprecise and lacks confidence, resulting in long, redundant responses and highlighting deficiencies in intuition and verification. Inspired by the Dual Process Theory in psychology, we introduce a simple modification to the QA task that includes four stages: Fast Thinking, where the LLM must answer within a strict token budget; Verification, where the model evaluates its initial response; Slow Thinking, where it refines the initial response with more deliberation; and Summarization, where it distills the refinement from the previous stage into precise steps. Our proposed task improves average accuracy from 25.6% to 27.3% for Qwen2.5-1.5B, and from 45.9% to 51.0% for DeepSeek-R1-Qwen-1.5B. Notably, for Qwen2.5-1.5B, the Fast Thinking mode alone achieves 25.2% accuracy using fewer than 1000 tokens, demonstrating substantial inference efficiency gains. These findings suggest that intuition and deliberative reasoning are distinct, complementary systems benefiting from targeted training. Additionally, we have open-sourced both the trained models and the source code.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on improving the reasoning capabilities of large language models (LLMs). One representative model is DeepSeek R1, which performs well but relies heavily on backtracking and verification, often resulting in long and redundant responses. To enhance efficiency, the paper proposes a dual-process framework inspired by cognitive psychology. The method involves four steps: the agent first generates an initial answer rapidly using a small token budget, then evaluates it with a larger token budget. If the answer is deemed incorrect, the agent produces a revised response using more resources. During training, a summarization step is introduced to strengthen the model’s fast-thinking ability. Empirically, the method is evaluated on several mathematical reasoning benchmarks, showing improved accuracy over baseline approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- Improving the reasoning abilities of LLMs through reinforcement learning is a highly active and relevant research direction, especially following the success of DeepSeek R1. This paper builds upon that line of work by decoupling reasoning into two complementary systems: a *fast thinking* component for intuitive response generation and a *slow thinking* component for verification and deliberation. The motivation is grounded in cognitive psychology, and the empirical results demonstrate the effectiveness of the proposed dual-process framework.\n\n- The paper includes thorough ablation studies and detailed analyses to validate the contributions of each component in the model design.\n\n**Weaknesses:**\n\n- It would be helpful to compare the proposed method with simpler prompting-based approaches such as *Self-Refine* or *Reflexion*, which also aim to improve reasoning through iterative refinement.\n\n- The experiments are limited to Proximal Policy Optimization (PPO). It would strengthen the paper to evaluate how the approach performs with more advanced or stabilized RL algorithms such as Gradient-Regularized Policy Optimization (GRPO).\n\n- In Figure 4(a), there appears to be some overfitting in the baseline methods. Additionally, the performance fluctuation of the proposed method after 1k training steps raises concerns about stability. It is unclear whether the observed improvement is statistically significant or due to noise.\n\n- While I am not deeply familiar with all baselines in this area, I believe there are additional recent works that apply RL to enhance LLM reasoning. A broader baseline comparison would help clarify the significance of the proposed approach.\n\nQUESTIONS:\n- On GPQA, why does *Thinker-Fast* outperform the full *Thinker* model? What insights can be drawn from this result?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
Inspired by the Dual Process Theory, the paper proposes an approach to train a model to infer using two modes of thinking (fast and slow) in the question-answering setting. Specifically, the model is trained to take a 4-step variant of CoT paradigm, involving a fast thinking as the first step which predicts the answer within a short output token budget. The second verification step check the answer and proceeds to the slow thinking step if the fast thinking answer is verified to be incorrect. The slow thinking step involves a longer output token budget. The final summarization step provides the final answer. Experiments were conducted on an array of math reasoning QA benchmarks to show the superior performance of proposed approach compared to baseline methods. Further analyses were conducted to understand the effect of training on response length and model’s reflection patterns.

STRENGTHS AND WEAKNESSES:
Strengths:
Proposed approach is shown to improve math reasoning performance while reducing compute with the fast thinking mode.
The paper is well written and easy to follow

Weaknesses:
Experiments are limited to math reasoning tasks, applicability to other domains and tasks is yet to be supported by experiments

QUESTIONS:
The proposed approach seems to have two tries in answering a question (one from fast and slow if fast thinking is verified as incorrect). However, most baseline methods would only have one try in answering the questions. It seems it is fairer to compare with pass@2 metrics of baseline methods where there are two tries to answer a particular question.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the \"Thinker\" task, a four-stage RL framework for LLMs (Fast Thinking, Verification, Slow Thinking, Summarization) inspired by Dual Process Theory. By decomposing QA into structured stages with distinct rewards, the method aims to enhance intuition, evaluation, refinement, and integration capabilities. Experiments on Qwen2.5-1.5B and DeepSeek-R1-Distill-Qwen-1.5B show significant accuracy gains across math benchmarks, with Fast Thinking achieving efficiency via strict token budgets. The approach reduces redundant self-reflection but yields longer responses due to verification iterations.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\n1. Inspired by human cognitive systems (System 1/2), Thinker explicitly trains intuition (Fast Thinking) and deliberation (Slow Thinking), forming a virtuous loop where fast responses guide slow refinement and vice versa. This structured design aligns with psychological theories, providing a principled RL framework. \n\n2. Each stage’s reward (e.g., binary for fast thinking, weighted for verification) targets specific capabilities, enabling precise credit assignment. \n\n# Weaknesses\n\n1. In Section 5.3, the authors observe that although the Thinker task exhibits fewer reflection patterns, the verification step nonetheless produces longer responses. This phenomenon stems primarily from repeated self-verification cycles incentivized by the answer extraction strategy, which focuses solely on the final boxed output. This indicates a redistribution of inefficiency rather than a net reduction, potentially compromising practical inference gains in certain scenarios.\n\nQUESTIONS:\nWhy does Thinker achieve higher accuracy despite generating longer responses than standard QA? For instance, DeepSeek-R1-Distill-Qwen-1.5B’s responses increase in token length, yet MATH500 accuracy rises by 8.5%. Is this due to Verification’s iterative self-checks or Slow Thinking’s deep refinement?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents \"Thinker\", a method of decomposing reasoning tasks into four subtasks, and associated RL training rewards for each subtask. Motivated by Dual Process Theory for human cognition, the subtasks are \"fast thinking\", \"verification\", \"slow thinking\", and summarization.\n\nFast thinking consists of attempting to answer the question with a small token budget (1k tokens). Verification consists of using a large token budget to assess the answer produced by fast thinking. If verification rejects the fast thinking result, slow thinking uses a large token budget to produce a corrected answer. Finally, summarization produces a new short reasoning trace, trained with a reward for producing reasoning which both achieves the same answer as slow thinking, and is assigned high probability under the fast thinking prompt.\n\nModels are trained via PPO to solve math questions. It studies two models (Qwen2.5 1.5B, and DeepSeek-R1-Distill-Qwen-1.5B) and two task settings (“baseline”, i.e. standard QA reward, vs. Thinker subtasks). Thinker is shown to result in larger performance gains than the direct QA task: on Qwen, Thinker improves performance by 24%, while Baseline improves performance by 21%; and on R1, Thinker improves performance by 14%, while Baseline improves performance by 10%.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1) The paper is clearly presented, and the method is described well.\n2) Evaluations are performed on 7 datasets, consisting of 6 math benchmarks plus GPQA Diamond.\n3) The paper presents some interesting qualitative analysis; I found it interesting that “reflection patterns” seem to increase over training for both Baseline and Thinker, despite much more flat total response length.\n\n**Weaknesses**:\n\nI’m concerned about how much we can infer from the presented results. IIUC the paper's conclusions stem from 5 total PPO RL runs:\n* R1 1.5B, Thinker vs Baseline\n* Qwen 1.5B, Thinker vs. Baseline\n* Qwen 1.5B, Thinker without Summarization\n\nThe paper does provide CIs with respect to dataset examples (though for readability these should ideally be included in the main table, rather than separately in the appendix). However, with only single RL runs for each setting, we have no way of estimating variation due to RL training dynamics. E.g. figure 6 shows divergence in “reflection pattern occurrence”; is this actually caused by differences in the loss, or simply that Thinker without Summarization happened by chance to produce some high-quality reflection patterns, and so settled into a different but equally effective writing style?\n\nThe average performance improvements over the baseline are relatively minor (3% and 4%); to adequately support the paper's claim and justify the substantial increase in method complexity, the paper should compare multiple RL runs for each setting (each using different seeds for dataset shuffling and LLM sampling), and show statistical significance between baseline and thinker with respect to variation in RL runs.\n\nQUESTIONS:\nHow are the standard errors in appendix B computed? E.g. for the MATH 500 Thinker: if rewards are binary correctness, for the Thinker results, (64% accuracy), I get sqrt(.64\\*(1-.64)/500) ~= 2.1%. If we considered each of the 500\\*16 samples as IID, we’d get sqrt(.64\\*(1-.64)/(500\\*16)) ~= 0.54%; this would still be higher than reported in the paper, though note that this will underestimate standard errors due correlation between samples on the same question (e.g. see “cluster standard errors”: https://www.anthropic.com/research/statistical-approach-to-model-evals)\n\nI searched the code for “error” and “std”, but it wasn’t clear to me where these statistics are computed vs. using std for internal model normalization.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}