Title: Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Slow Thinking

System Switch

Large Reasoning Model
Abstract: Large reasoning models (LRMs) have demonstrated strong performance on complex reasoning tasks, but often suffer from overthinking, generating redundant content regardless of task difficulty. Inspired by the dual process theory in cognitive science, we propose Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables LRMs to achieve efficient reasoning through adaptive cognitive allocation and dynamic system switch.
ACPO incorporates two key components: (1) introducing system-aware reasoning tokens to explicitly represent the thinking modes thereby making the model's cognitive process transparent, and (2) integrating online difficulty estimation and token length budget to guide adaptive system switch and reasoning during reinforcement learning. 
To this end, we propose a two-stage training strategy. The first stage begins with supervised fine-tuning to cold start the model, enabling it to generate reasoning paths with explicit thinking modes. In the second stage, we apply ACPO to further enhance adaptive system switch for difficulty-aware reasoning.
Experimental results demonstrate that ACPO effectively reduces redundant reasoning while adaptively adjusting cognitive allocation based on task complexity, achieving efficient hybrid reasoning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables large language models to dynamically switch between fast and slow reasoning modes using system-aware tokens. ACPO introduces an online difficulty-aware token length budget and combines it with a multi-part reward function to optimize reasoning efficiency. The two-stage training pipeline includes supervised fine-tuning followed by reinforcement learning with GRPO. Experiments on math benchmarks show improved token efficiency with competitive accuracy. The method aligns reasoning behavior with problem complexity, aiming to mitigate overthinking in LRMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The dual-process reasoning formulation (fast thinking vs. slow thinking) provides an interpretable and cognitively grounded approach to reasoning in LLMs.\n- Results demonstrate a substantial reduction in token usage without significant accuracy loss, with improvements in ACU.\n\nWeakness:\n- There is no ablation study to prove the effectiveness of the proposed reward function. Using a weighted sum of multiple reward functions may easily lead to reward hacking. \n- The optimization objective is somewhat underspecified. While the SFT stage enforces distinct fast and slow thinking patterns, the RL phase does not directly reinforce the intended structural relationship (e.g., slow thinking should be longer than fast thinking), potentially weakening the cognitive alignment.\n- The proposed pipeline is only evaluated on two math datasets without any other task, e.g., code, reasoning gym agent, etc.\n- The performance of the proposed method is worse than L1-max in MATH 500 and AIME 2024 datasets on AUC. Could the author explain why that happens?\n\nQUESTIONS:\n- The performance of the proposed method is worse than L1-max in MATH 500 and AIME 2024 datasets on AUC. Could the author explain why that happens?\n- How does the proposed reward function explicitly encourage longer slow thinking segments for more complex tasks during reinforcement learning, beyond what is learned during supervised fine-tuning? Could the authors show the distribution of the fast and slow thing token numbers under different question difficulties?\n- Can the authors provide ablation studies to demonstrate the individual contributions of each reward component and address potential reward hacking from the weighted combination? E.g., reward loss figures through training.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper find that Large Reasoning Models (LRMs) tend to \"overthink,\" producing redundant reasoning content regardless of task difficulty. This behavior leads to inefficiency on simple problems and strategic inadequacy on complex ones, where a fixed approach may fail. The root cause, according to the paper, is the models' lack of a mechanism to dynamically adjust their reasoning policy in response to task complexity. To tackle this challenge, the authors introduce the Adaptive Cognition Policy Optimization (ACPO) framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The online Token Length Budget (TLB) reward is a well-designed mechanism. Its ability to adaptively adjust the reasoning length based on sample success rates is a notable contribution.\n\n2. The proposed method appears to have significant practical value, especially given the current landscape of large-scale deployment of Large Reasoning Models (LRMs).\n\nWeakness\n1. The generalizability of the proposed method is not well-established. The training dataset is based on LIMO, a high-quality dataset for complex mathematical reasoning, and the experiments are exclusively confined to this domain. The paper does not evaluate the approach on other types of reasoning tasks, such as commonsense reasoning. Consequently, it remains uncertain whether the method is effective in scenarios beyond mathematical problem-solving.\n\nQUESTIONS:\nIn my view, a major significance of this research lies in its potential for saving computational costs during inference. Could the authors provide more analysis to quantify this advantage? For instance, it would be highly compelling to show that your method achieves comparable performance to previous approaches while using only a fraction (e.g., x%) of the reasoning tokens. This is particularly relevant as redundancy in the reasoning process is a core problem for current LRMs, which your method aims to solve.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to train the LLM to think (or to reason) in two speeds, specifically fast thinking and slow thinking. Here, the fast thinking is for system 1 for easy reasoning problems (or easy reasoning stage, such as verification after answering), while slow thinking is for complex reasoning. To achieve this, the author proposes a model to think by using special tokens, namely \\<fast_thinking\\> and \\<slow_thinking\\>. To train this model, the author proposes SFT as a cold start by collecting slow and fast thinking processes, then uses online RL with a new reward that forces the model to think efficiently.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n(1) I think the problem is well-motivated, and the method is quite novel. I do like the special tokens, \\<fast_think\\> and \\<slow_think\\>. Also, having a cold start with SFT makes sense. \n\n(2) Overall, the writing is clear\n\n(3) The experiments are also well conducted. \n\nI do not find major weaknesses in this paper, but I do have some concerns that would be great if the authors could resolve them through the rebuttal.\n\n**Weakness**\n\n(1) I think the paper has only focused on the math domain, which is one limitation. I think it would be great if the authors could extend to coding (e.g., MBPP, Humaneval) or general domains (e.g., GPQA diamond). \n\n(2) Additionally, I believe it is worthwhile to analyze the idea on more challenging benchmarks. It is great that the authors have already considered AIME 2024, but also AIME 2025, and AMC 2023 would be worth it. The motivation behind this question is that slow thinking may appear more in challenging scenarios compared to easy ones (e.g., GSM 8K). \n\n(3) I quite like the idea of SFT warmup (i.e., Cold start). However, I could not find the effect of this cold start in the paper. Can the authors provide some analysis behind this?\n\n(4) This is not a weakness rather a question. Can the author provide the intuition behind the reward design? For instance, using negative reward for a wrong answer is not common. Is there any intuition behind this? \n\n(5) I think adding GRPO as a default baseline for all datasets will be required (i.e., in Table 2 the GRPO is missing). Also, when adding more datasets during the rebuttal, I kindly ask the authors to add GRPO as a baseline.\n\nOverall, I think the paper is well written and has the strength to be accepted. I kindly ask the authors to address the minor concerns that I have (I am willing to increase the score after the rebuttal).\n\nQUESTIONS:\nSee the weakness above\n\nLIMITATIONS:\nSee the weakness above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method to teach large reasoning models (LRMs) how to incorporate 'fast' and 'slow' thinking adaptively while solving mathematical reasoning problems. A small dataset is curated using rejection sampling and LLM-as-judge to annotate correct, shorter (longer) responses as fast (slow) thinking examples. Subsequently, a language model is supervised fine-tuned for cold start to generate chain-of-thought with fast and slow thinking token markers. The ensuing RL training is primarily based on GRPO but includes two additional rewards: online token length budget that calculates the token budget adaptively from the GRPO sampled candidate answers and their success rate, and an additional reward to incentivize slow thinking for harder problems and fast thinking for easier problems (sample success rate is used a proxy for task difficulty). Three testbeds are used to evaluate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n- Highly relevant research problem: Most LRMs suffer heavily from overthinking, leading to erroneous reasoning as well as high computational cost.\n\n- Interesting approach to solution: I really like the idea of interleaving slow and fast thinking. Personal opinion: Differentiating between fast or slow thinking in absolute terms of the problem (i.e., this problem requires system 1 thinking or that problem requires system 2 thinking) is a debatable topic in cognitive science (see [1] or [2], for example). The interleaving can be thought of as an alternative to such hard binaries. I would really appreciate if the authors put some thought into this direction to further strengthen the argument of the paper.\n\n- Cheap reasoning trace: Proposed method renders the model to use tokens in an efficient manner. This (along with similar papers in this direction) can provide cheaper effective alternatives.\n\n# Weaknesses\n\n- Occasional lack of clarity: Certain major details are missing (or, I could not find them effectively), for example, how they make the model to interleave two different modes within a single chain-of-thought when the data is per-instance, what happens (in the data curation phase) is the longer vs shorter reasoning traces explore two different paths of reasoning altogether (that leads to the same answer, but uses different ways or premises), why certain experiments are not performed with GSM8K, etc.\n\n- Narrow focus: The proposed method is designed and experimented with mathematical reasoning only, and no pointers are provided for if and how it can be adapted to other reasoning tasks.\n\n- Insufficient analysis of errors: While the limitation section is quite superficial in my opinion, there is no explicit analysis of erroneous cases.\n\n[1] Keren, G., & Schul, Y. (2009). Two is not always better than one: A critical evaluation of dual-system theories.\n\n[2] Melnikoff, D. E., & Bargh, J. A. (2018). The Mythical Number Two.\n\nQUESTIONS:\n1. How do you interleave two modes in the response? Figure 1 tells that the mode annotation is per-instance and not per-token.\n\n2. Did you perform manual checking of the GPT generated annotation? Given the quality of the LIMO dataset, I doubt GPT-4 will be able to correctly identify all cases of reasoning nuances. What happens if the short and long responses are explicitly two different state-transition sequences?\n\n3. How do you differentiate between short and long (fast or slow) reasoning? Can you provide the length-wise distribution of the responses and where you put the decision boundary?\n\n3. The LIMO paper itself used the same testbeds and fewer training data. The accuracy numbers are much higher than all the numbers in this paper. I would like to have your comment on this.\n\n4. The dual-mode thinking is a debated hypothesis in cognitive science (see Strengths and Weaknesses). I would suggest the authors take into account and correspondingly position their contribution and limitations.\n\n5. How does the token budget affect the performance? Can you analyse cases where the baseline model (that used a longer reasoning trace) was able to solve the problem, but ACPO failed?\n\n6. Analysis of ACPO on distribution shift (e.g., GSM-plus [1]) compared to a standard (overthinking) model will significantly improve the quality of the paper.\n\n[1] Li, Q., et al. (2024). A comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-05"}