Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning
Authors: Xiaoxue Cheng
Junyi Li
Zhenduo Zhang
Xinyu Tang
Xin Zhao
XinYu KONG
Zhiqiang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Slow Thinking
System Switch
Large Reasoning Model
Primary area: deep_learning

Abstract: Large reasoning models (LRMs) have demonstrated strong performance on complex reasoning tasks, but often suffer from overthinking, generating redundant content regardless of task difficulty. Inspired by the dual process theory in cognitive science, we propose Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables LRMs to achieve efficient reasoning through adaptive cognitive allocation and dynamic system switch.
ACPO incorporates two key components: (1) introducing system-aware reasoning tokens to explicitly represent the thinking modes thereby making the model's cognitive process transparent, and (2) integrating online difficulty estimation and token length budget to guide adaptive system switch and reasoning during reinforcement learning. 
To this end, we propose a two-stage training strategy. The first stage begins with supervised fine-tuning to cold start the model, enabling it to generate reasoning paths with explicit thinking modes. In the second stage, we apply ACPO to further enhance adaptive system switch for difficulty-aware reasoning.
Experimental results demonstrate that ACPO effectively reduces redundant reasoning while adaptively adjusting cognitive allocation based on task complexity, achieving efficient hybrid reasoning.

Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning
Xiaoxue Cheng 1.23+ Junyi Li Zhenduo Zhang", Xinyu Tang', Wayne Xin Zhao 12,3f, Xinyu Kong", Zhiqiang Zhang Gaoling School of Artificial Intelligence, Renmin University of China
2 Beijing Key Laboratory of Research on Large Models and Intelligent Governance
3 Engineering Research Center of Generatnnnnnn Intelligent Search and Recommendation, MOE
Department of Data Science, City University of Hong Kong Ant Group chengxiaoxue@ruuc edu. en, junyili@cityy edu. edu.hk, hk, batmanfly@gmail.. com
Abstract
Large reasoning models (LRMs) have demonstrated strong performance on com- plex reasoning tasks, but often suffer from overthinking, generating redundant content regardless of task difficulty. Inspired by the dual process theory in cog- nitive science, we propose Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables LRMs to achieve efficient reason- ing through adaptive cognitive allocation and dynamic system switch. ACPO incorporates two key components: (I) introducing system-aware reasoning tokens to explicitly represent the thinking modes thereby making the model's cognitive process transparent, and (2) integrating online difficulty estimation and token length budget to guide adaptive system switch and reasoning during reinforcement learning. To this end, we propose two-stage training strategy. The first stage begins with supervised fine-tuning to cold start the model, enabling it to generate reasoning paths with explicit thinking modes. In the second stage, we apply ACPO to further enhance adaptive system switch for difficulty-aware reasoning. Exper- imental results demonstrate that ACPO effectively reduces redundant reasoning while adaptively adjusting cognitive allocation based on task complexity, achieving efficient hybrid reasoning.
Introduction
Recent advances in large reasoning models (LRMs) have demonstrated remarkable success on complex tasks such as mathematical reasoning 3 largely attributed reinforcement learning that encourages the generation of detailed, step-by-step reasoning processes. LRMs improve answer accuracy through self-reflection and self- self-verification during long reasoning paths. As the reasoning length increases, the performance of model tends to improve accordingly 22 6 77. Although the long chain-of of-thought (CoT) g reasoning LRMs is effective for solving complex problems 2 9 often leads to overthinking [10 [1), producing redundant reasoning paths. Most existing LRMs rely on fixed reasoning strategies, lacking the ability to dynamically switch between different thinking modes based on task complexity. This rigidity results in inefficient inference, particularly for simple problems that could be resolved more effectively with concise and direct reasoning.
Several recent efforts have explored long CoT compression for efficient reasoning 13 13 One
line of work fine-tunes LRMs using supervision from shorter of-thugh exemplars [4 15) encouraging the model to arrive at correct answers with fewer intermediate steps. Another line
"qqual Contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

introduces length penalties into reinforcement learning reward functions 16 17 18
explicitly discouraging unnecessarily long reasoning trajectories during training. Since many prior approaches overlook task difficulty and treat all samples uniformly with the sole objective of shortening reasoning paths, some recent methods attempt to address this by estimating length budgets offline and training the model on sampled trajectories accordingly [9]. However, such offline strategies depend on precomputed budgets and fixed preference data, restricting their adaptability and scalability.
Inspired by the dual process theory in cognitive science 20). which states that humans have two systems for thinking - fast, intuitive thinking (System 1) and slow, deliberate thinking (System 2). we pose the following research question: Can LRMs learn to dynamically switch between fast and slow thinking modes based on task complexity, enabling more efficient and adaptive reasoning?
In this paper, we propose Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables LRMs to perform efficient and adaptive reasoning through dynamic system switch between fast and slow thinking modes. To achieve this, we first introduce system-aware reasoning tokens (e.g. sfast_think>, in fast or slow thinking modes during the reasoning process. Based on these tokens, we construct dataset from which the reasoning trajectory is interleaved with fast thinking and slow thinking steps, followed by the final answer. Then, we propose a two-stage training strategy to enable dynamic system switch reasoning. In the first stage, we perform supervised fine-tuning with the constructed dataset as a cold start, establishing the model's foundational ability to generate explicit thinking process. In the second stage, we apply reinforcement learning with ACPO for further enhancement. Specifically, we introduce an online token length budget (TLB) mechanism that estimates task difficulty based on the model's sampling success rate, providing a real-time signal to adjust the reasoning budget according to task complexity. These estimates are further incorporated into reward function that guides cognitive allocation through two components: TLB reward that encourages difficulty-awee length control, and a system pattern reward that incentivizes appropriate system switch between fast and slow thinking modes.
(slow_think>) explicitly indicate the model engagement
We evaluate ACPO on a range of complex reasoning benchmarks. Unlike traditional reinforcement learning methods that focus solely on accuracy or fixed-length compression, ACPO dynamically adjusts both reasoning length and cognitive effort based on task difficulty. For challenging problems, it effectively reduces redundant reasoning steps, while for simpler tasks, it avoids overcompression and maintains high accuracy. These results highlight the advantages of difficulty- reasoning. demonstrating the effectiveness of dynamic cognitive control through adaptive reward optimization. The main contributions of this work are follows:
We introduce system-aware reasoning tokens to explicitly annotate fast and slow thinking steps in LRMs, enabling transparent reasoning paths and providing foundation for dynamic control of the model's 's cognitive strategies.
We propose ACPO, reinforcement learning framework that integrates online difficulty estimation and token length budget to dynamically calibrate the reward function and steer difficult--wware cognitive allocation.
We demonstrate through extensive experiments that ACPO effectively compresses redundant reasoning content while maintaining accuracy, achieving robust balance between efficiency and performance by avoiding overcompression and underexploration.
2 Preliminary
2.1 Group Relative Policy Optimization (GRPO)
Group Relative Policy Optimization (GRPO) 21 is reinforcement learning framework that clim- inates the need for a value function by estimating advantages in a group-relative manner. Given a specific question-answer pair (q, a), the behavior policy Tew, samples a group of G individual responses (y, )- For each token yi, in response yi. its normalized advantage is computed based on the group-level rewards (R, G as follows:
R₁ mean([R))
A
(1)
std(
2

---

Similar to Proximal Policy Optimization (PPO) 22), GRPO adopts clipped surrogate objective with an additional KL regularization term stabilize optimization:
JoRPo (0)
E (vnn) mo
G
min (rra(0)Aa,, clip (r.,(0),11 1, =, 3Dxi. (TTa ( ].
(2)
1
G
where (0))(0)
no (yc.e (0.35.ce) ae (9..11.))))))
is the importance sampling ratio for the t-th token yi.e-
2.2 Token Length Budget
The token length budget (TLB) 19 is a method introduced for estimating an appropriate token-level budget for reasoning trajectories. Instead of prescribing fixed or hard constraint, TLB adaptively adjusts the expected output length based on the sampling accuracy of candidate responses, capturing the inherent difficulty of the task. Specifically, for given question q. N candidate responses are sampled from a language model. The TLB, denoted as Lundgrt. is computed as:
Lbudget - L, + (1 p) Lmax:
where p = represents the sampling success rate, with c denoting the number of correct responses, L, denotes the average token length among the correct responses, Lmax is the maximum token length among all responses. This formulation allows the budget to adapt to task complexity, providing flexible length guidance based sampling accuracy without external supervision. In the DAST method [1)] TL.B is computed an offiin manner by sampling multiple responses for each question before training. These responses then converted to preference pairs for further fine-tuning according to their estimated TLB scores.
3 Method
In this section, we propose ACPO, reinforcement learning framework that enables dynamic and adaptive system switch of LRMs to improve reasoning efficiency and adaptability to task complexity. This framework incorporates system-aware reasoning tokens explicitly represent fast and slow thinking steps of LRMs, and integrates an online length budget estimation in RL training to dynamically adjust reasoning length based on task difficulty, guiding the model balance accuracy and efficiency through adaptive cogmitive effort allocation. We first describe the use of system wystem-aware reasoning tokens for system explicitization, followed by the construction of explicit reasoning paths for supervised fine-tuning. Next, we present the reinforcement learning process of ACPO, providing a detailed overview of the online token length budget estimation and reward design. The overall framework of the proposed ACPO is illustrated in Figure] 1
3.1 Explicit Dual Process Reasoning
The dual process theory (0) models human thinking combination of fast, intuitive processes (System 1) and slower, deliberate reasoning (System 2). Inspired by this theory, we introduce system-aware reasoning tokens explicit indicators of different thinking modes within the model's reasoning process. Specifically, we define four special tokens, teis__thikk,,, </fast_ think>, <slow_think> think> and
explicitly wrap fast and slow reasoning steps, respectively. The introduction of reasoning tokens serves two purposes. First, it makes the model's internal thinking modes explicitly observable, revealing the process of system switch. Second, provides fine -neeggaided mechanism for controlling and monitoring the model's cognitive dynamics. With this explicit thinking process, we can train the model for system switch, optimizing its cognitive allocation based on task difficulty.
3.1.1 Data Construction
To enable the model to learn explicit thinking patterns, we construct training dataset for supervised fine-tuning, Each sample consists of question paired with its corresponding answer consisting of interleaved fast thinking and slow thinking segments. To ensure the quality of the training data, we utilize the LIMO dataset (23). carefully curated, high-quality dataset of complex mathematical

---

System Explicitization
Respanse XX
SFT
Longest Response
Dual Process Ressoning
Respunse
ooo
Query
<olow Panks
<alw minks
Candidate Sampling
Respunse 3
Comparison and Annotation
fast Whink Safart thinks
Shorlest Response
car cangwers.....o wlangwerr
Response N
ACPO
YL
Accuracy
A
System
Defficult: Suceess Rate
Y2
R2
Group Norm
Online
Ys
ILB:
Ra
Avg Length
Pollcy
Max Length
System
Reward
System 2
Figure 1: The overall framework of ACPO. The upper section illustrates the system explicitization process and cold start training via SFT. The lower section presents the ACPO training phase.
reasoning tasks with high challenge and diverse knowledge coverage. These tasks demand precise, multi-step reasoning to arrive at correct answers, making thinking processes. Specifically, our data construction involves two main steps: Candidate Response Sampling and Response Comparison and Ammotation
them well-suited for constructing explicit
Candidate Response Sampling: Building the approach in TOPS 24 prompt the DeepSeek- (2) model to generate multiple candidate responses with varying lengths for each question. We adopt the prompting strategies in TOPS to encourage diversity of the reasoning lengths. After generating these candidates, we filter out incorrect responses and select the longest and shortest correct answers for further processing.
RI
Response Comparison and Annotation: We employ GPT-4 (25 as an evaluator to perform fine- grained comparison and annotation between the selected response pairs. For those reasoning steps that are present in both short and long responses, we consider them as essential and detailed components that should be marked slow thinking, as they represent critical, non-omittable reasoning processes. For other steps that appear in the long response but are omitted or summarized in the short response, we view them trivial steps that should be labeled fast thinking. Based this principle, the short response by enclosing essential reasoning steps with <slow kslow_thinke twinttthtssllle /slk//lww__. [low_thinkk think> tags and trivial reasoning steps with <fast_ feat__thin><<</fff think>> tags, An illustrative example provided in Figure
reframe
3.1.2 Supervised Fine-Tuning (Cold Start)
After completing the data construction process, obtain dataset containing 745 annotated samples with explicit reasoning tokens. This dataset used for supervised fine-tuning. allowing the model to learn to generate outputs with clearly interleaved fast and slow thinking modes. This cold start phase equips the model with the foundational ability to explicitly switch between different thinking modes. We train the model with standard cross-entropy loss to ensure its thinking process aligns with the intended system switch patterns.
3.2 Reinforcement Learning with ACPO
While the SFT process allows the model to align its outputs with the annotated reasoning tokens, explicitly capturing fast and slow thinking patterns, this approach may lead to overfitting and ten- dency to memorize fixed reasoning paths. To overcome this limitation, we introduce a reinforcement learning phase, optimizing the fine-tuned model achieve more flexible and adaptive cognition allocation through interaction and exploration. In the following sections, present the proposed ACPO method, including the online token length budget estimation and reward design that are specifically tailored for dynamic system switch.

---

3.2.1 Reward Design
In this work, adopt GRPO the underlying RL algorithm, extending its framework with customized reward structure to explicitly guide the model's 's reasoning behavior. Formally, during the ACPO optimization process, sample set of N candidate responses {yı V/2 .2...... ................................ . from the current policy model for each training query qi- For each response, we compute a composite reward that consists of three components: accuracy reward, online TLBB reward, and system pattern reward to encourage accurate and efficient reasoning with adaptive cognition allocation, as described below.
Accuracy Reward. For each sampled response we assess its correctness by comparing the generated answer with the ground-truth label in the training data. The accuracy reward is assigned as:
Race, {
+1,
if y is correct.
-1
otherwise.
Online TLB Reward. To guide the model in generating reasoning paths of appropriate length that align with the task difficulty, we introduce an online token length budget (TLB) reward. Unlike the offline approach in DAST, which requires extensive pre-sampling and pairwise preference construction, our method estimates the length budget Lisige online during training, without additional computational overhead. This achieved by leveraging the natural group-based sampling process in GRPO, allowing the sampling success rate p and token length budget budger to be directly calculated from the existing candidate responses, as defined in Eq. Given a sampled response y: with actual length Li we compute its TLB reward based on the deviation from the estimated Loadget- The online TLB score BILB, for each response is then defined as:
tanh( -de, if y; is correct, (An)) otherwise,
RILB,1
(5)
where A; Li-ten and the tanh function is used to smoothly bound the reward within (-1, 1). The Lisetge
TLB reward adapts the model's reasoning strategy to the difficulty of each task, promoting efficient fast thinking for casier problems while allowing longer and more deliberate reasoning for harder problems. Additionally, the online TLB estimation enables real-time adaptation and avoids reliance on static preference data, leading to better generalization across diverse tasks.
System Pattern Reward. To further encourage difficulty- system switch, success rate p from the TLB estimation as a proxy for task difficulty. For easier queries with higher p. the model is encouraged to allocate larger proportion of its reasoning path to fast thinking. In contrast, for harder queries with lower p. the model is incentivized to allocate more steps to slow thinking for more careful deliberation. Given sampled response ys, compute the proportions of fast and slow thinking tokens within the total reasoning sequence, denoted as Pinse, and Pslow,i respectively. The system pattern reward then defined
the sampling
Plast.s if > Ptures
Rehink. =
Pstow,is otherwise,
where Pthresh is a predefined complexity threshold separating casy and hard questions.
The final reward R; for each sampled response is computed as weighted combination of the three components:
max (Wase Ruse,t + Wlen RILB, + Wiiikk R 0.1), if ys is correct,
min (Wace Rawe, When RILB, + Wiink
Riiinkis
0.11, if ys is incorrect.
The weights Wase. Wien and Whinkk are hyperparameters that balance the importance of different reward signals during optimization. In order to ensure that the reward for correct responses remains strictly positive and the reward for incorrect responses remains strictly negative, we apply max and min operations to clip the final reward within the desired range. Based on the final reward, we leverage the GRPO objective in Eq. 2 to optimize the policy model.
5

---

Table 1: Evaluation results of ACPO three different reasoning models across the MATH 500 and AIME 2024 datasets. Bold fonts indicate the best performance for each reasoning model.
MATH 500
AIME 2024
Methods
Accuracy
# #Token
ACU
Accuracy
#Token
ACU
DeepSSet-RI--issill-eeenn
83.9
5708
0.98
28.9
16894
0.11
THINKPRUNE
82.9
2356
2.35
27.0
7574
0.24
ACPO-1.5B
81.0
1679
3.22
30.0
6670
0.30
DeepScaleR-1.5B-PP
87.8
3914
1.50
43.1
17206
0.17
L1-Exact
79.8
1044
5.09
16.7
1798
0.62
LI-Max
81.8
999
5.46
23.3
2230
0.69
92.8
3977
0.33
55.5
13254
0.06
SFT Shortest
91.8
2954
0.44
50.0
10757
0.07
SimPO Shortest
87.8
970
1.29
33.3
2737
0.17
SimPO DAST
92.6
2802
0.47
53.3
6337
0.12
ACPO-7B
91.6
1405
0.93
52.8
4520
0.17
DeepSeek-R ---issill---aaa-----
89.1
5003
0.22
42.9
16374
0.04
ACPO-8B
87.4
2232
0.49
43.3
7405
0.07
4 Experiment
4.1 Experimental Setup
Datasets and Evaluation Metries. We conduct training on the
Peeeed--rreiiw--iiie 226]
a mathematical dataset consisting of 40K question-answer pairs drawn from AIME, AMC, Omni- Math (27] and STILL (28). For evaluation,
three mathematical datasets: GSM8K 22)], AIME 2024, and MATH 500 [30] each test question, we generate 16 responses using sampling temperature of 0.6 and top-p value of 0.95, and compute pass @ to measure accuracy. We report the average number of tokens generated per response each dataset to assess reasoning efficiency. Moreover, we use Accuracy per Computation Unit (ACU) metric [3] to capture the balance between reasoning accuracy and efficiency, which is defined as
assess model performance
Accuracy
ACU
(8)
#Params x #Tokens
Baselines. We conduct experiments on -iseece--R---isii-------eeee Qwen-7B, and DeepSeck-RR D--issl-----ssslll-aaa------ (2) and compare ACPO against the following methods.
DeepSeek- RI-Dittill-
DeepScaleR--155--Preeiie 26 is model trained on DeepSeek- SeeeSSee--R--isttiie R1-Distill-Qwen. with GRPO. We include it as an important bascline in our evaluations.
THINKPRUNE (17) trains models with fixed token limit, pruning unfinished responses beyond this limit with zero reward. We include THINKPRUNE with 2k token length constraint trained on DeepSeek-N I-Distill-Qwen 1.5B for comparison.
L.1
18 trains models follow specified response lengths by introducing exact and maxi- mum length penalties in reinforcement learning, yielding two variants, L.I-xxatt and L.-Max, both trained on DeepScaleR-1.55 1.5B-Preview.
DAST 19 introduces an offline approach to estimate token length budgets through sampling and construct length preference data for SimPO [32] training. We include the comparison methods SFT Shortest,,, SimPO SimPP-Shhotest and SimPO_DASe from DAST in our experiments.
Implementation Details. In the cold start phase, we fine-tune the models for 3 epochs using 745 annotated samples with explicit reasoning tokens. For ACPO training, we adopt same hyperparameter settings as used in DeepScaleR- B----eeeeee Specifically, we use learning rate of 10-6, batch size of 128, and a maximum context length of 8K tokens during training. The models are trained for one epoch, and both the SFT and RL stages conducted using the VeRL

---

Table 3: Performance comparison between three reasoning models trained with GRPO and ACPO on the MATH 500 and AIME 2024 datasets.
MATH
AIME 2024
Methods
Accuracy
#Token
ACU
Accuracy
#Token
ACU
DeepSSek-RI--Dsiiil--eeee.
83.9
5708
0.98
28.9
16894
0.11
+GRPO
3098
1.82
29.0
12990
0.15
+ACPO
81.0
1679
3.22
30.0
6670
0.30
DiaSScckKR--isssil---ee----
92.8
3977
0.33
55.5
13254
0.06
+GRPO
92.5
3700
0,36
53.2
8577
0.08
+ACPO
91.6
1405
0.93
52.8
4520
0.17
DeepSeek- -
89.1
5003
0.22
42.9
16374
0.04
+GRPO
90.4
2172
0.52
43.3
8883
0.06
+ACPO
87.4
2232
0.49
43.3
7405
0.07
framework [33]] We set Pthresh 0.5 in Eq. and set the reward weights Wace 0.6, Wlen 0.3,
and Wihink = 0.1 Eq.
4.2 Main Results
The evaluation results of our method and the baselines are presented in Table | and Table
For more challenging datasets, such MATH 500 and AIME 2024 in Table ACPO achieves signifi- cant token reduction while maintaining competitive accuracy. For instance, on the AIME 2024 dataset, ACPO-1.5B reaches 30.0% accuracy with average token count of 6670, representing 60.5% reduction in token usage compared to 16,894 tokens required by DeepSeek-R1--issill-- 1.5.B. For MATH 500 dataset, ACPO reduces the to- ken count for DeepSeek-R11- from 3977 1405 with only slight accu- racy decrease. Although LI-Max and L1-Exact also achieve substantial token compression, they suffer noticeable accuracy drops compared to DeepScaleR- P.S--Prviiw. This demonstrates that ACPO can effectively shorten reasoning lengths without sacrificing too much accuracy. particularly complex reasoning tasks.
Table 2: Evaluation results of ACPO with the three reasoning models the GSM8K dataset.
Methods
Accuracy
#Token
ACU
R1-Distill-Qwen-I.. ACPO-1.5B
79.9
643
8.28
81.3
572
9.48
RI-Distill Qwen-7B
Distiill-Qwen-7B
86.5
445
2.78
ACPO-7B
88.3
413
3.05
R1-Distil--Llamaa ACPO-SB
82.9
1026
1.01
For simpler GSM8K dataset in Table 2 where the reasoning paths of DeepSeek- SoapSSeck-R--iisiie Ri--iiiill- Qwen-1.5B and 7B tend be less redundant, ACPO maintains the original length scale while achieving notable accuracy improvements. Specif- ically, ACPO-1.5B reduces average token usage from 643 to 572 while improving accuracy from 79.9% to 81.3%. Similarly, ACPO-7B achieves 88.3% accuracy with 413 tokens, outperforming the baseline in both accuracy and efficiency. However, for DeepSeek- RI-Distiil-LLama has relatively long reasoning paths with 1026 tokens, ACPO achieves effective compression by reducing the token count to 732 tokens with 3.8% accuracy improvement. These results highlight the adaptability of the online token length budget estimation, enabling selective compression that preserves necessary reasoning length while avoiding excessive compression.
86.7
732
1.48
which
4.3 Further Analysis
4.3.1
Ablation Analysis
To assess the impact of the reward design ACPO on both accuracy and reasoning length, we conduct ablation studies by training models with the GRPO reward setting, keeping all other parameters identical those in ACPO. Specifically, we evaluate the three models on the MATH 500 and AIME 2024 datascts: DeepSeek-RR -I---itillllieeeeeee 1.5BB, DeepSeck-RR -I--ttsi--------------------- Qwen-7B, and DeepSeek-RI.

---

3000
ACPO-1.59
LI-Mix
L1-Ex
01:6-
Fae Think Roisa
Slow Think Raii
1000
2.00
0.14
0.8
Think Ratio
0.12
0.925
00.5
Ascumnyy
1500
0.900
1000
0.05
9.875
0.04
0.2
0.825
0.0
000
10,000
Difficulty
Difficuity
Figure 2: Average response length and accuracy across different difficulty levels on MATH 500.
Figure 3: Average fast and slow thinking ratios across different difficulty levels on MATH 500.
Distill-Llama-8B, each trained for one epoch with GRPO and ACPO. We report accuracy, average token length, and ACU scores for comparison in Table 3
From the results, we observe that both GRPO and ACPO compress the reasoning length of the original models. In contrast, ACPO consistently achieves more significant compression, often exceeding 50% reduction in token length. For instance, it reduces the average reasoning length of PraSeek-R1--istil--PPe
on the MATH dataset to 1679 tokens, compared to 3098 tokens for GRPO, In terms of accuracy, GRPO shows slight advantage, especially on the MATH dataset. However, ACPO demonstrates comparable performance on the more challenging AIME dataset. For
ACPO reaches accuracy of 30.0% compared 29.0% for GRPO, effectively balancing accuracy and efficiency. While GRPO focuses solely on correctness, the reward design of ACPO jointly optimizes answer accuracy, reasoning length, and system switch, enabling a flexible coordination between correctness, efficiency, and cognitive allocation.
4.3.2 Difficulty Adaptability Analysis
To validate the difficulty adaptability of ACPO, we perform analyses on the MATH 500 dataset, which is divided into five difficulty levels. In Figure we compare the accuracy and average response length ACPO-1.5B, LI-Exact, and L1-Max. Unlike the LI approach, which applies uniform token constraints regardless of the difficulty of questions, ACPO demonstrates more adaptive length control strategy. It effectively shortens responses for simpler problems while preserving the necessary reasoning length for complex questions, resulting in minimal accuracy loss more challenging levels (e.g., level 4, level 5). This adaptive capability highlights the advantage our approach in difficulty- reasoning, aligning the token length budget more closely with problem complexity.
In Figure 3] we further analyze the fast and slow thinking ratios for ACPO-7B on the MATH dataset. It is observed that, as the problem difficulty increases, the proportion of fast thinking decreases while the slow thinking component increases. Notably, the proportion of slow thinking increases rapidly from difficulty level 1 to level 3, and then grows more gradually beyond level 3. This trend suggests that the model able to allocate more reasoning effort when solving more difficult problems, dynamically allocating cognitive resources based on task complexity. Such adaptive behavior aligns with the dual process theory of human thinking, demonstrating the effectiveness of our method in reasonable cognition allocation and dynamic system switch.
4.3.3 Case Study
In Figure 4 we present an example comparing the reasoning behavior of DeepSeek-RR -iieck-RR--istiee -Distilll Qwen-1.5B trained with GRPO and ACPO on number theory problem from the Math 500 dataset. The task requires finding the smallest positive integer multiple of 30 that can be written using only the digits 0 and which requires applying divisibility rules: the number must end in 0 and the sum of its digits must be divisible by 3. The model trained with GRPO adopts a cautious and exhaustive strategy, enumerating multiple candidate numbers (e.g., 30, 60, 90) step by step, even after identifying valid solution. This leads to lengthy reasoning path with 1555 tokens, filled with redundant self- verfiiiiiiin and repeated constraint checking. In contrast, the model trained with ACPO efficiently identifies the necessary constraints, rapidly filtering out invalid candidates and
8

---

What is the least positive integer muttiple of that can be written with only the digits and 2?
DeepSeek-Ai -ii-----ii---e-------- with ACPO
DeepSeek-A1 R1--iiiDiiiiii Dist-ii---iiiiiii Sawiil--wee----. 1.58 with GRPO
Kslow, thinkrOkkay, need find smallest positive integer maltipc of 20 that can be writtes using only the digits 0 and 2. Hmms, let's brenk this down. First, I know thet 30 is times 10, so any multiple of must be divisible by both and That means the number has to and with a because of the divisibility by 10,
find the smallest posiiiee integer multiple 30 that be writen using only the digits and 2. Hmm, let's break this down. First, know that 50 oqual to 3 multippied by 10, so any multiple of 50 must be multiple of means number the 10 part, and the sum of its digits has to be a multippl of bocouse the 3 part. end with 0 and the rest of the digits Us. Let me think about how to appreach this, Maybe can start by considering numbers made up 2s and Os that and with 0 and check if their digit sums are the smallest possible numbers.
Okhnnk-Okay,
and 10.
looking for
and with becouse
Next, the sumber must also be divisible
3. The rule
Alright, the number
only he
divisiiility by that the sum the digiis must be a malliple of
Since the number can unly have digits O and 2. need figure out divisible by
how many need so that
me tr with the smallest number of diglts, Starting with one digit: 2 But
divisible by Let's start
smallest multiple
of 3o is 50 itself. Is 30 made up only Os and 24? Well, it has 3 and (1, so no. So 30 is Nevt, let's 60, 60 has and O Again, also out How about 907 That's and Still not good Himm, mayhe nead diferently. Instedd of looking at mulliples of maybe I should construct numbers Since the number
isa't divisible 3, so that's Next, two digits: The
and Os. So 60
the digits is 2 00=2 2, which work. How about three digits? Let's try 220. The sum is 2 0- Nest, 2220. sum
visibllee by 3. 20 docan't
2 6, which divisible by 3. So 2220 should work. Killww think? Stast Wait, let me double-
scill divisible
only 2s
divisible
O, let's
consider numbers like 20, 220, 2220, etc. Let's check their divisibility by 3. Starting with 20: The sum of the digits is 2 0 2, 2 not divisible by 3, so 20 isn't multiple of Next, 220:
2220 divided by 30 is 74, which integor, So yes, 2220 a mulriple
only uses the digits 2 and D, don't think thume's a smaller number that lits these criteria bocnuse previous attempts fewer digits dichi't work Kiaast thinkes
Therefore, 2220 the smallest such number.
double-chock; 2220 divided by
30 is 74, which is an inieger. So yes, 2220 is multiple of 30, and it's made up only of 2s and Os. I think that's the answer. So, the least positive integer mnlltiple of 30 can written with only digits and 2220, <thinks
The least positive integer multiple of 30 that can be writtnn with 2220, samwwrr The positive integer
only digits
maltiple of 30 that can be writtes with only the digits O and 2
<answer- The least positive integer multiple of 30 that can be written with only the digits is lboodd (2220). <lanwer- Tokens)
lboced (2200)xaanweer <anwwer- (476 Tokens)
Figure 4: An case study comparing the reasoning process of DeepSeek- -1- RI-Distill Siriil--wen-Ila 1.5B trained with GRPO and ACPÓ in MATH 500 Dataset.
quickly converging to the correct answer in just 476 tokens. Notably, the model exhibits effective system switch, employing slow thinking during problem analysis and fast thinking for verification once solution is found, achieving substantial token savings especially in the verification stage by eliminating repetitive constraint checks. The system-aware reasoning tokens offer a transparent view of the model's cognitive process, enabling interpretable analysis of how reasoning behavior aligns with fast and slow thinking modes.
5 Related Work
Efficient Reasoning in LRMs. Large reasoning models (LRMs) like DeepSeek-R1 2 and OpenAl
ol 4 have demonstrated strong reasoning capabilities, but their long reasoning paths often suffer from overthinking, introducing redundant content and reducing inference efficiency [ [I]. To address this, one line of work focuses on supervised fine-tuning with concise, high-quality data to reduce the length of reasoning paths [1 15). For instance, TokenSkip [14 generates short reasoning chains through token importance sampling, while Self-Training [15] selects correct short reasoning paths through sampling. Another line of work seeks to improve efficiency through reinforcement learning by modifying reward functions to penalize excessively long reasoning paths [3] [7] Among these methods, length budgeting is a direct way to explicitly control reasoning length. For example, LI [18] introduces fixed length budget to penalize responses that exceed a predefined length during RL training. DAST 19 estimates task difficulty and token length budget through sampling, and constructs preference data for SimPO (32) training. In this work, we integrate online difficulty- aware length budgeting into reinforcement learning process, enabling real-time reasoning budget estimation and reward allocation for efficient reasoning.
[3]
Fast and Slow Thinking in LLMs. The dual process theory (20 describes two modes of human thinking: fast, intuitive thinking (System 1) and slow, deliberate thinking (System 2). Recent studies have investigated fast and slow thinking in LLMs, focusing on system switch based on task complexity or uncertainty. Specifically, System 1.x 34 adapts a controller and System- 1/2 planner to adjust reasoning systems based on task difficulty. FaST (35] develops a switching adapter that transitions between System 1 and System 2 for visual reasoning, depending on factors like uncertainty. HaluSearch (36) leverages model performance to generate supervised labels, enabling hierarchical dynamic switch between reasoning systems within MCTS. Dyna-Think (37 implements a training-free dynamic thinking mechanism, allowing the model to autonomously determine when to apply slow reasoning. In this work, we introduce system-aware reasoning tokens to explicitly

---

represent fast and slow thinking modes, and leverage ACPO with online token length budget to enable adaptive cognitive allocation and dynamic system switch based on task difficulty.
6 Conclusion
In this paper, we proposed ACPO, reinforcement learning framework designed to address the overthinking problem in LRMs through adaptive cognitive allocation and dynamic system switch. In our approach, we first introduce system-aware reasoning tokens to explicitly represent fast and slow thinking modes, making the model's cognitive process transparent and interpretable. Next, we proposed two-stage training strategy, first fine-tuning the model to establish the ability generate explicit thinking process. followed by reinforcement learning with ACPO to enhance adaptive cognition allocation. ACPO integrates online difficulty estimation and token length budget during training, guiding dynamic reasoning through a carefully designed reward function. Experimental results demonstrate that ACPO effectively reduces redundant reasoning without sacrificing much accuracy. Our work provides a flexible and interpretable framework for adaptive hybrid reasoning, supporting efficient and difficulty-aware cognitive processes in LRMs.
7 Limitation
Although ACPO effectively reduces redundant reasoning and enables adaptive cognitive allocation, it has several limitations. First, its online difficulty estimation and token length budget mechanisms rely on verifiable data, which may limit generalization to open-domain tasks. Future work should explore more generalizable estimation methods for broader applicability. Second, ACPO can introduce accuracy trade-offs despite improving reasoning efficiency, highlighting the need for more refined reward designs to better balance efficiency and correctness.
Acknowledgments
This work partially supported by National Natural Science Foundation of China under Grant No. 92470205 and 62222215, Beijing Natural Science Foundation under Grant No. L233008 and Beijing Municipal Science and Technology Project under Grant No. Z23 1231100010323000 Xin Zhao is the corresponding author.
References
[1] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303. 18223, 2023.
[2]
DeepSeek-Al. AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue. Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li. Jianzhong Guo. Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan. Runji Wang, R. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. CoRR, 010025011.19445 2025.
10

---

[3] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjuu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congoong Wang. Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang. Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jía Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang. Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye. Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma. Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei. Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuchai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang. Zhiqi Huang, Zihao Huang. Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with Ilms. CoRR, abs/2501 12599, 2025.
[4] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. Openai ol system card. arXiv preprint arXiv:2412.1677) 2024.
[5] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025.
[6] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408. 2024.
[7] Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 1830-1842, 2024.
[8] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 45:24824-24887 2022.
[9] Mengdi Liu, Xiaoxue Cheng, Zhangyang Gao, Hong Chang, Cheng Tan, Shiguang Shan, and Xilin Chen. Protinvtree: Deliberate protein inverse folding with reward reward-guided tree search. arXiv preprint arXiv:2506.0000 00925, 2025.
[10] Xingyu Chen, Jiahao Xu, Tian Liang. Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do NOT think that much for 2+3=? on the overthinking of ol-like llms. CoRR, abs/2412.21187,
2024.
[11] Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Bo Wang, Shimin Li, Yunhua Zhou, Qipeng Guo, Xuanjing Huang, and Xipeng Qiu. Scaling of search and learning: A roadmap to reproduce ol from reinforcement learning perspective. arXiv preprint
211:2442..11355 2024.
[12] Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, Peng Li, Wei Wei, Jing Shao, Chaochao Lu, Yue Zhang, Xian- Sheng Hua, Bowen Zhou, and Yu Cheng. A survey of efficient reasoning for large reasoning models:
Language, multimodality, and beyond. CORR, abs/2503.21614 2025.
[13] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu. Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Ben Hu. Stop overthinking: A survey on efficient reasoning for large language models. CoRR, abs/2503.16419, 2025.
[14] Heming Xia, Yonggi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable thain-of-thought compression in llms. arXiv preprint arXiv:2502. /2067, 2025.
[15] Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang, Yujin Kim, and Se- Young Yun. Self-training elicits concise reasoning in large language models. arXiv preprint 22X12 2025.

---

[16] Edward Yeo. Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning llms. arXiv preprint arXiv:2502.03373, 2025.
[17] Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of- of-thought of Ilms via reinforcement learning. arXiv preprint arXiv:2504.0122 012966 2025.
[18] Pranjal Aggarwal and Sean Welleck, LI: Controlling how long reasoning model thinks with reinforcement learning, arXiv preprint arXiv:2503. 2025.
[19] Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian. Dast: Difficult--dappiix slow-thinking for large reasoning models. arXiv preprint 2xxx:25033.0472. 2025.
[20] Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, 2011.
[21] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li,
Y. Wu, and Daya Guo. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024.
[22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1707.06347, 2017.
[23] Yixin Ye, Zhen Huang. Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint urXiv:2502.033877 2025.
[24] Wenkai Yang, Shuming Ma. Yankai Lin, and Furu Wei. Towards thinking-optimin scaling of test-time compute for Imm reasoning. arXiv preprint arXiv:2502. 1e..5002.880000 18080, 2025.
[25] OpenAl. GPT-4 technical report. CoRR, 00122303308774, 2023.
[26] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with 1.5b model by scaling rl. --w-------------------------- htw.:aaaaaaaaaao--o0999991666680 1tl--198881911466000000033000000 Satr---uuusssinn-----e--
RL- b75 n notion site/
2025. Notion Blog.
[27] Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong. Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang. Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models. CoRR, abs/2410.07988. 2024.
[28] Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. CoRR, abs/2412.004113, 2024.
[29] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. CoRR, 0t.s/2110.14160 2021.
[30] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt, Measuring mathematical problem solving with the math dataset. arXiv preprint 1 2021.
[31] Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: cennth-compressibbl chain-of -ffthuught tuning. arXiv preprint arXiv:2502.0960) 2025.
[32] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122235. 2024.
[33] Bytedance Seed MI.Sys. verl: Volcano engine reinforcement learning for llms. https
h/github.com//olll. httpaa/..oom/oooengnnnee/e/eee/e 2025.
12

---

[34] Swarnadeep Saha, Archiki Prasad, Justin Chih- Yao Chen, Peter Hase, Elias Stengel-Esking and Mohit Bansal. System- x: Learning balance fast and slow planning with language models. arXiv preprini arXiv:2407.144 144144 2024.
[35] Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu. Visual agents as fast and slow thinkers. arXiv preprint arXiv:2408.000 08862. 2024.
[36] Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Think more, hallucinate less: Mitigating hallucinations via dual process of fast and slow thinking. arXiv preprint arXiv:2501.00 306, 2025.
[37] Jiabao Pan, Yan Zhang, Chen Zhang, Zuozhu Liu, Hongwei Wang, and Haizhou Li. Dynathink: Fast or slow? a dynamic decision-makin framework for large language models. arXiv preprint arXiv:2407.01009 2024.
[38] An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan
Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024.
13

---

A Prompts for Explicit Thinking Annotation
to generate responses of varying reasoning lengths for candidate response sampling, and leverage GPT-4 for fine-grained comparison to annotate different thinking modes. The system prompts for these two components are provided below, where the system prompts for candidate sampling are adapted from TOPS (24]
In Section we prompt
System Prompts for Candidate Response Sampling
Low Reasoning Effort: You have extremely limited time to think respond to the user's query. Every additional second processing and reasoning incurs significunt resource cost, which could affect efficiency and etfectiveness. Your task is to prioritize speed without sacrificing essential clarity or accuracy. Provide the most direct and concise unswer pussible. Avoid unnecessary steps, reflections, veri prrfticaiion, refinements UNLLESS ABSOLUTELY NECESSARY. Your primary goal is deliver quick, clear and correct response.
High Reasoning Effort: You have unlimited time to think and respond to the user's question. There is no need to worry about reasoning time or associated costs. Your only goal is to arrive at reliable, correct final answer. Foel free to explore the problem from multiple angles, and various methods your reasoning. This includes reflecting on reasoning by trying different approaches, verifying sieps from different sspects, and rethinking your conclusions as needed. You are encouraged take the time to analyze the problem thorouhhly, reflect your reasoning prompily and all possible solutions, Only after deep, comprehensive thought process should you provide the final answer, casuring it is correct and well-supportd by your reasoning.
System Prompts for Response Comparison and Annotation
Given a problem, a short answer, and long answer, compure the short answer with the long answer and annolate the shoet following
short answer omits certain reasoning calculation steps these omilted steps considered fast thinking, and the correpunding parts the short answer should be enclosed within <fast think>-//fast thinks, If the short answer contains the same reasoning or calculation steps as the long answer, these parts considered slow thinking and should be enclosed within eslow thinks-c/llow thinks. Fast thinking parts typically involve intuitive judgments, skipped steps, or direct conclusions, whereas slow thinking parts involve full reasoning or calculations that shoet answer the appropriate <last think>-</fast think> and
answer based
present long
long answer. output <slow thinks-chlow think tags added.
align
#Problem#:
#I Hnng Answerdi:
#Short Answert:
#Annotated Answel::
B Experimental Setting
B.1 Training and Evaluation Details
In the SFT stage, the learning rate is 1 10 the batch size is 8, the number of epochs is 3. For evaluation, we use Qwen2.5 [38 tokenizer to calculate the number of tokens in the responses generated by each model for a fair comparison. All the experiments conducted on 16 NVIDIA A100 GPUs.
B.2 Baseline Details
We present descriptions of the three baseline methods SFT_Shortest, DOSPT from DAST in T We adopt their evaluation results as reported in the original DAST paper [19].
and
SFT Shortest: Supervised fine-tuning using only the shortest correct sampled response of each problem training data.
SimPO with contrastive instance pairs, which take the shortest correct sampled response of each problem positive instance and the longest as negative instance.
SimPO DAST:
SimPO_DAST: SimPO with contrastive instance pairs from Dpre constructed in DAST.
B.3 Prompt for ACPO Training
We present the prompts used in the ACPO training process below.
14

---

Prompt for ACPO Training
You helpful Assistant that provides well reasomed and detailed responses. You first think about the reasoming nal monologue and then provide the user with the auswer. Respond in the following format: <ilink>.../thinkk Withini-------eeeee..... clanserr.....> reasoning process, you
simple steps slowly complex steps, and put the thinking <slow thinks<s/slow, thinks and fast thinking content in <hast thinko-c/fast thinks. Let's think step by step and outpur the fi- answer within boxed J.
C License for existing assets
We list the licenses for the datasets and models used in our experiments.
C.1
Models
DeepSeek-RI-Distill-Peee Qwe.-...B: Apache 2.0 License.
DeppSSet-R---Disiii---ee----- Apache 2.0 License.
DeepSect-RI-Dssiii-LLam--B-B Llama3,1 License.
C.2
Datasets
The GSM8K 29
dataset: MIT License.
The MATH
30
dataset: MIT License.
The DeepSealcR-Previee-DDe
266 MIT License.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We made sure our abstraction and introduction accurately reflect paper's main contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include standalone limitation section in the Appendix 7 We have discussed both the methodological limitations and performance bottlenecks our approach.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yss]
Justification: We provide the experiment details and results in Section and Appendix We will also release our code upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification: Yes, we provide all the necessary code to reproduce every experiment men- tioned in the paper. We will also release our code publicly upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppullccguuudodeo Cotee//ddeeeSbmmmcma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, disclose all the training and test details in Section and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our evaluation experiment is conducted by sampling 16 responses and com- puting the pass@1 score demonstrate the performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method training technique for large language models which does not have negative societal impact that needs to be addressed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We list licenses for used datasets and models in Appendix
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The released code, data, and models well documented including details about training, license, limitations.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
21

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We describe the usage of LLMs in experiments in Section
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standaddd components.
Please refer to our LLM policy (https: for what should or should not be described.
22


Reviews:
[{"review_id": "y6TczSrkL6", "reviewer": "Reviewer_dgDW", "summary": "This paper introduces a novel method to teach large reasoning models (LRMs) how to incorporate 'fast' and 'slow' thinking adaptively while solving mathematical reasoning problems. A small dataset is curated using rejection sampling and LLM-as-judge to annotate correct, shorter (longer) responses as fast (slow) thinking examples. Subsequently, a language model is supervised fine-tuned for cold start to generate chain-of-thought with fast and slow thinking token markers. The ensuing RL training is primarily based on GRPO but includes two additional rewards: online token length budget that calculates the token budget adaptively from the GRPO sampled candidate answers and their success rate, and an additional reward to incentivize slow thinking for harder problems and fast thinking for easier problems (sample success rate is used a proxy for task difficulty). Three testbeds are used to evaluate the effectiveness of the proposed method.", "questions": "1. How do you interleave two modes in the response? Figure 1 tells that the mode annotation is per-instance and not per-token.\n\n2. Did you perform manual checking of the GPT generated annotation? Given the quality of the LIMO dataset, I doubt GPT-4 will be able to correctly identify all cases of reasoning nuances. What happens if the short and long responses are explicitly two different state-transition sequences?\n\n3. How do you differentiate between short and long (fast or slow) reasoning? Can you provide the length-wise distribution of the responses and where you put the decision boundary?\n\n3. The LIMO paper itself used the same testbeds and fewer training data. The accuracy numbers are much higher than all the numbers in this paper. I would like to have your comment on this.\n\n4. The dual-mode thinking is a debated hypothesis in cognitive science (see Strengths and Weaknesses). I would suggest the authors take into account and correspondingly position their contribution and limitations.\n\n5. How does the token budget affect the performance? Can you analyse cases where the baseline model (that used a longer reasoning trace) was able to solve the problem, but ACPO failed?\n\n6. Analysis of ACPO on distribution shift (e.g., GSM-plus [1]) compared to a standard (overthinking) model will significantly improve the quality of the paper.\n\n[1] Li, Q., et al. (2024). A comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n- Highly relevant research problem: Most LRMs suffer heavily from overthinking, leading to erroneous reasoning as well as high computational cost.\n\n- Interesting approach to solution: I really like the idea of interleaving slow and fast thinking. Personal opinion: Differentiating between fast or slow thinking in absolute terms of the problem (i.e., this problem requires system 1 thinking or that problem requires system 2 thinking) is a debatable topic in cognitive science (see [1] or [2], for example). The interleaving can be thought of as an alternative to such hard binaries. I would really appreciate if the authors put some thought into this direction to further strengthen the argument of the paper.\n\n- Cheap reasoning trace: Proposed method renders the model to use tokens in an efficient manner. This (along with similar papers in this direction) can provide cheaper effective alternatives.\n\n# Weaknesses\n\n- Occasional lack of clarity: Certain major details are missing (or, I could not find them effectively), for example, how they make the model to interleave two different modes within a single chain-of-thought when the data is per-instance, what happens (in the data curation phase) is the longer vs shorter reasoning traces explore two different paths of reasoning altogether (that leads to the same answer, but uses different ways or premises), why certain experiments are not performed with GSM8K, etc.\n\n- Narrow focus: The proposed method is designed and experimented with mathematical reasoning only, and no pointers are provided for if and how it can be adapted to other reasoning tasks.\n\n- Insufficient analysis of errors: While the limitation section is quite superficial in my opinion, there is no explicit analysis of erroneous cases.\n\n[1] Keren, G., & Schul, Y. (2009). Two is not always better than one: A critical evaluation of dual-system theories.\n\n[2] Melnikoff, D. E., & Bargh, J. A. (2018). The Mythical Number Two.", "was_revised": true, "final_justification": "Author's response could address my initial concern.", "rebuttal": "Thanks for your valuable comments. Our responses to your concerns are as follows:\n\n**Q1: Clarification on how fast and slow thinking modes are interleaved**\n\nThe Dual Process Reasoning in Figure 1 indeed corresponds to step-level thinking-mode annotations, not per-instance labels. Each response includes interleaved &lt;fast_think&gt; and &lt;slow_think&gt; tags to mark different reasoning steps, enabling fine-grained cognitive switching within a single trace. This interleaving behavior is learned through a two-stage process: supervised fine-tuning with explicitly annotated data and reinforcement learning with system-aware rewards. A concrete token-level example is shown in Figure 4. We will revise Figure 1 to clarify this more explicitly.\n\n**Q2：Annotation data quality**\n\nWe acknowledge that annotation quality is critical. In our pipeline, we use the same model with different prompts to generate short and long responses, which encourages differences in reasoning granularity while largely preserving the core solution structure.\n\nWe rely on GPT-4 to perform the annotation automatically and filter out any samples where GPT-4 fails to annotate due to structural mismatch or unclear reasoning trace. We spot-checked a subset of samples to verify that the generated tags aligned with our intended labeling protocol.\n\nWe appreciate the reviewer’s suggestion and will further review all annotated data to ensure consistency and quality.\n\n**Q3: Differentiation between fast and slow reasoning**\n\nIn our framework, the distinction between fast and slow reasoning is established during data construction by prompting the same model with different instructions. The short responses are encouraged to be concise and direct, while the long responses are prompted to be more thorough and elaborative.\n\nTo obtain step-level annotations, we compare the short and long response pairs using GPT-4. Reasoning steps shared by both responses are labeled as slow thinking, while steps present only in the long response are labeled as fast thinking. This approach does not rely on a fixed token-length threshold, but instead uses semantic comparison to determine reasoning granularity.\n\nBelow, we report the average token lengths of short, long, and annotated responses to provide a reference for their length distribution.\n\n| Response Type | Long Response | Short Response | Annotated Response | \n|----------|----------|----------|----------|\n|Tokens   | 7105 | 2683  | 3020 | \n\n**Q4: Comparison with LIMO performance**\n\nWe would like to clarify that the LIMO paper uses Qwen2.5-32B-Instruct as its backbone for supervised fine-tuning, while our experiments are conducted on smaller models (DeepSeek-R1-Distill-Qwen-1.5B, 7B, and DeepSeek-R1-Distill-LLaMA-8B). Given the significant gap in model capacity, the accuracy numbers are not directly comparable.\n\nAdditionally, our SFT phase does not use the original LIMO solutions, but the newly generated reasoning paths annotated with &lt;fast_think&gt; and &lt;slow_think&gt; tokens.\n\nTo isolate the impact of backbone, we fine-tuned DeepSeek-R1-Distill-Qwen-1.5B on the original LIMO dataset to serve as a fair comparison. As shown in the table below, the performance of the trained model is even lower than our ACPO results.\n\nFinally, we would like to emphasize that our method is not solely aimed at maximizing accuracy, but at achieving comparable performance with more efficient (i.e., shorter) reasoning traces.\n\n| Models | GSM8K | MATH 500 | AIME 2024|\n|----------|----------|----------|----------|\n|DeepSeek-R1-Distill-Qwen-1.5B| 79.9|83.9|28.9|\n|1.5B-limo-sft  | 76.2 | 74.6 | 23.3 | \n| ACPO-1.5B | 81.3 | 81.0 | 30.0 |\n\n**Q5: Controversies of Dual-Mode Thinking**\n\nWe are pleasantly surprised and grateful that the reviewer recognized our method as an alternative to the hard binary of fast and slow thinking. In fact, our step-level thinking-mode switching is designed to reflect the dynamic nature of the reasoning process, where fast and slow thinking are interleaved rather than strictly separated.\n\nRigidly dividing cognition into two distinct modes, or assigning a fixed mode to each problem, is a coarse approach. What inspires us from the dual process theory is not the binary classification itself, but the notion of dynamic cognitive adjustment during reasoning.\n\nWe will further clarify this point in the final version to better explain our motivation.\n\n**Q6: Impact of token budget on performance and failure cases**\n\nWhile ACPO achieves substantial token savings, we acknowledge that aggressive compression may occasionally lead to performance degradation.\n\nWe conducted case studies where the baseline model answered correctly but the ACPO-trained model failed. On these cases, baseline responses averaged 7691 tokens, while ACPO responses averaged 3999 tokens.\nThe baseline typically explored more intermediate steps and reasoning paths, while ACPO missed certain details due to insufficient exploration. These failures mostly occurred in more challenging cases where longer reasoning is more beneficial. However, our dynamic budget mechanism helps reduce some of these failures by allocating more tokens to harder problems.\n\nDue to space constraints, we will show the detailed case study in the final version to further clarify this point.\n\n**Q7: Analysis of ACPO on distribution shift**\n\nTo assess ACPO’s robustness under distribution shift, we evaluated both the baseline and ACPO-trained models on GSM-Plus.\n\nAs shown in the table below, ACPO maintains comparable accuracy while effectively reducing the reasoning length, demonstrating its generalization ability under distribution shift.\n\nModels | Accuracy|#Token|ACU |\n|----------|----------|----------|----------|\n| DeepSeek-R1-Distill-Qwen-1.5B | 62.62 |10939 | 0.38 |\n|ACPO-1.5B | 61.25 | 8141| 0.50 |\n\n**W1：Some detailed explanations**\n\nSome experiments were not conducted on GSM8K because it is relatively simple, and the compared baselines did not consider it as a target test set. Additionally, due to the lack of released models or code, or missing reported results, we were unable to include them for comparison. We will include GSM8K results once the corresponding baseline models become available.\n\n**W2: Generalization to other domains**\n\nWhile our original experiments focused on math reasoning tasks, we agree that broader evaluation is important. To this end, we conducted additional experiments on GPQA Diamond. As shown in the table below, ACPO reduces reasoning token usage while maintaining comparable accuracy, demonstrating its generalization potential beyond math.\n\n| Models | Accuracy|#Token|ACU |\n|----------|----------|----------|----------|\n| DeepSeek-R1-Distill-Qwen-1.5B | 33.8 | 22557 | 0.09 |\n|ACPO-1.5B | 32.3 | 11226| 0.19 |\n\nOur method is task-agnostic and can be extended to other domains involving complex reasoning. In future work, we plan to synthesize more fast/slow thinking data in general domains for supervised fine-tuning and ACPO training, to further enhance its applicability across a wider range of tasks."}, {"review_id": "Xdwkgnb8XC", "reviewer": "Reviewer_wub4", "summary": "This paper aims to train the LLM to think (or to reason) in two speeds, specifically fast thinking and slow thinking. Here, the fast thinking is for system 1 for easy reasoning problems (or easy reasoning stage, such as verification after answering), while slow thinking is for complex reasoning. To achieve this, the author proposes a model to think by using special tokens, namely \\<fast_thinking\\> and \\<slow_thinking\\>. To train this model, the author proposes SFT as a cold start by collecting slow and fast thinking processes, then uses online RL with a new reward that forces the model to think efficiently.", "questions": "See the weakness above", "limitations": "See the weakness above", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n(1) I think the problem is well-motivated, and the method is quite novel. I do like the special tokens, \\<fast_think\\> and \\<slow_think\\>. Also, having a cold start with SFT makes sense. \n\n(2) Overall, the writing is clear\n\n(3) The experiments are also well conducted. \n\nI do not find major weaknesses in this paper, but I do have some concerns that would be great if the authors could resolve them through the rebuttal.\n\n**Weakness**\n\n(1) I think the paper has only focused on the math domain, which is one limitation. I think it would be great if the authors could extend to coding (e.g., MBPP, Humaneval) or general domains (e.g., GPQA diamond). \n\n(2) Additionally, I believe it is worthwhile to analyze the idea on more challenging benchmarks. It is great that the authors have already considered AIME 2024, but also AIME 2025, and AMC 2023 would be worth it. The motivation behind this question is that slow thinking may appear more in challenging scenarios compared to easy ones (e.g., GSM 8K). \n\n(3) I quite like the idea of SFT warmup (i.e., Cold start). However, I could not find the effect of this cold start in the paper. Can the authors provide some analysis behind this?\n\n(4) This is not a weakness rather a question. Can the author provide the intuition behind the reward design? For instance, using negative reward for a wrong answer is not common. Is there any intuition behind this? \n\n(5) I think adding GRPO as a default baseline for all datasets will be required (i.e., in Table 2 the GRPO is missing). Also, when adding more datasets during the rebuttal, I kindly ask the authors to add GRPO as a baseline.\n\nOverall, I think the paper is well written and has the strength to be accepted. I kindly ask the authors to address the minor concerns that I have (I am willing to increase the score after the rebuttal).", "was_revised": true, "final_justification": "The rebuttal has well addressed my concern, thus increasing my score from 4 (to 5).", "rebuttal": "Thanks for your valuable comments. Our responses to your concerns are as follows:\n\n**Q1: Generalization to other domains**\n\nWe understand the reviewer’s concern regarding generalization. While our initial experiments focused on math reasoning, we additionally evaluated our method on the GPQA Diamond dataset. As shown in the table below, ACPO significantly reduces reasoning token usage while maintaining comparable accuracy, demonstrating its potential to generalize beyond math.\n\n| Models | Accuracy|#Token|ACU |\n|----------|----------|----------|----------|\n| DeepSeek-R1-Distill-Qwen-1.5B | 33.8 | 22557 | 0.09 |\n|ACPO-1.5B | 32.3 | 11226| 0.19 |\n\nOur method is task-agnostic and can be extended to other domains involving complex reasoning. Our current training data focuses primarily on math problems. Tasks like code generation pose additional challenges due to some formatting constraints. Nevertheless, we plan to synthesize fast/slow thinking data in these domains to enable supervised fine-tuning and ACPO training, further enhancing the method’s applicability to a broader range of complex reasoning tasks.\n\n**Q2: Evaluation on more challenging benchmarks**\n\nTo strengthen our evaluation on challenging benchmarks, we tested our method on AIME 2025 and AMC 2023. As shown below, ACPO achieves effective token compression on both datasets. Notably, it achieves higher compression on the simpler AMC 2023, and on the more difficult AIME 2025, it improves accuracy while reducing reasoning length, further demonstrating the effectiveness of our method in adapting reasoning length to task complexity.\n\n| Model                          |        AMC 2023         |                    |        |        AIME 2025        |                    |        |\n|-------------------------------|--------------------------|--------------------|--------|--------------------------|--------------------|--------|\n|                               | Accuracy                 | #Tokens            | ACU    | Accuracy                 | #Tokens            | ACU    |\n| DeepSeek-R1-Distill-Qwen-1.5B | 70.3                     | 10030              | 0.46   | 25.0                     | 18431              | 0.09   |\n| ACPO-1.5B                     | 69.2                     | 3270               | 1.41   | 30.0                     | 10690              | 0.19   |\n\n**Q3：Effect of SFT warmup**\n\nWe thank the reviewer for highlighting the cold-start phase. Its main purpose is not to boost accuracy, but to teach the model to emit &lt;fast_think&gt; and &lt;slow_think&gt; tokens in a consistent and semantically meaningful way. This structured initialization lays the foundation for the reinforcement learning phase.\n\nTo assess the impact of this phase, we report the model's accuracy before and after SFT. As shown below, the accuracy remains largely stable. We will clarify this point more explicitly in the future version.\n\n| Models | GSM8K | MATH 500 | AIME 2024|\n|----------|----------|----------|----------|\n|DeepSeek-R1-Distill-Qwen-1.5B  | 79.9|83.9|28.9|\n| +SFT | 80.4 | 83.0| 26.7|\n|DeepSeek-R1-Distill-Qwen-7B  | 86.5 | 92.8 | 55.5| \n| +SFT | 87.4 | 90.3| 53.3|\n|DeepSeek-R1-Distill-Llama-8B  | 82.9 | 89.1| 42.9|\n| +SFT |87.2 | 88.2| 42.0|\n\n**Q4: Intuition behind reward design**\n\nFor tasks like math problem solving, answer correctness is the primary objective, and any reasoning trace—no matter how efficient or well-structured—is ultimately invalid if it leads to the wrong answer.\n\nTo enforce this principle, we assign a mild negative reward to incorrect answers, ensuring that no positive reward is given unless the final prediction is correct. Using zero reward for incorrect answers can unintentionally result in a positive total reward if the reasoning trace scores well on length or structure, which contradicts the task goal.\n\nThis design encourages the model to first guarantee correctness, and then optimize for reasoning efficiency and cognitive control. We found this constraint important to prevent reward hacking and guide the model toward meaningful improvements.\n\n**Q5：GRPO baseline results**\n\nWe thank the reviewer for pointing this out. We will supplement Table 2 with the corresponding GRPO results and include GRPO as a baseline across all evaluated datasets in the updated version.\n\n| Models | Accuracy|#Token|ACU |\n|----------|----------|----------|----------|\n|GRPO-1.5B | 81.5 | 621| 8.72 |\n|GRPO-7B | 88.3 | 429| 2.94 |\n|GRPO-8B | 87.9 | 857| 1.28 |"}, {"review_id": "2hCdi8vPDI", "reviewer": "Reviewer_mRGf", "summary": "The authors of this paper find that Large Reasoning Models (LRMs) tend to \"overthink,\" producing redundant reasoning content regardless of task difficulty. This behavior leads to inefficiency on simple problems and strategic inadequacy on complex ones, where a fixed approach may fail. The root cause, according to the paper, is the models' lack of a mechanism to dynamically adjust their reasoning policy in response to task complexity. To tackle this challenge, the authors introduce the Adaptive Cognition Policy Optimization (ACPO) framework.", "questions": "In my view, a major significance of this research lies in its potential for saving computational costs during inference. Could the authors provide more analysis to quantify this advantage? For instance, it would be highly compelling to show that your method achieves comparable performance to previous approaches while using only a fraction (e.g., x%) of the reasoning tokens. This is particularly relevant as redundancy in the reasoning process is a core problem for current LRMs, which your method aims to solve.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The online Token Length Budget (TLB) reward is a well-designed mechanism. Its ability to adaptively adjust the reasoning length based on sample success rates is a notable contribution.\n\n2. The proposed method appears to have significant practical value, especially given the current landscape of large-scale deployment of Large Reasoning Models (LRMs).\n\nWeakness\n1. The generalizability of the proposed method is not well-established. The training dataset is based on LIMO, a high-quality dataset for complex mathematical reasoning, and the experiments are exclusively confined to this domain. The paper does not evaluate the approach on other types of reasoning tasks, such as commonsense reasoning. Consequently, it remains uncertain whether the method is effective in scenarios beyond mathematical problem-solving.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Our responses to your concerns are as follows:\n\n**W1: Generalization of ACPO**\n\nWe understand the reviewer's concern regarding generalization. While our original experiments focused on math reasoning tasks, we agree that broader evaluation is important. To this end, we conducted additional experiments on GPQA Diamond. As shown in the table below, ACPO reduces reasoning token usage while maintaining comparable accuracy, demonstrating its generalization potential beyond math.\n\n| Models | Accuracy|#Token|ACU |\n|----------|----------|----------|----------|\n| DeepSeek-R1-Distill-Qwen-1.5B | 33.8 | 22557 | 0.09 |\n|ACPO-1.5B | 32.3 | 11226| 0.19 |\n\nOur method is task-agnostic and can be extended to other domains involving complex reasoning. In future work, we plan to synthesize more fast/slow thinking data in general domains for supervised fine-tuning and ACPO training, to further enhance its applicability across a wider range of tasks.\n\n**Q1: Quantify the computational saving**\n\nWe thank the reviewer for highlighting this important aspect. One of the key motivations of our method is indeed to reduce overthinking and improve reasoning efficiency during inference.\n\nIn Table 1 and Table 2, we report Accuracy, Token Count, and ACU to demonstrate the effectiveness of our approach. Additionally, we compute the token compression ratio, defined as: \nCompression Ratio = 1 − tokens(ACPO) / tokens(Baseline), \nwhich quantifies how much token usage is reduced compared to baseline models.\n\nAs shown in the table below, ACPO consistently reduces the number of reasoning tokens across all datasets and backbones. Notably, the maximum compression of ACPO-1.5B reaches 70.58% on the MATH dataset. We will include additional quantitative analyses of token savings in the final version to better illustrate the computational efficiency gains enabled by ACPO.\n\n| Models | GSM8K|MATH 500|AIME 2024|\n|----------|----------|----------|----------|\n|ACPO-1.5B  | 11.04% | 70.58% | 60.52%|\n|ACPO-7B  | 7.19% | 64.67%| 65.89%|\n|ACPO-8B  | 28.65%| 55.38%| 54.77% |"}, {"review_id": "wsIckNyDhU", "reviewer": "Reviewer_Swif", "summary": "This paper proposes Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables large language models to dynamically switch between fast and slow reasoning modes using system-aware tokens. ACPO introduces an online difficulty-aware token length budget and combines it with a multi-part reward function to optimize reasoning efficiency. The two-stage training pipeline includes supervised fine-tuning followed by reinforcement learning with GRPO. Experiments on math benchmarks show improved token efficiency with competitive accuracy. The method aligns reasoning behavior with problem complexity, aiming to mitigate overthinking in LRMs.", "questions": "- The performance of the proposed method is worse than L1-max in MATH 500 and AIME 2024 datasets on AUC. Could the author explain why that happens?\n- How does the proposed reward function explicitly encourage longer slow thinking segments for more complex tasks during reinforcement learning, beyond what is learned during supervised fine-tuning? Could the authors show the distribution of the fast and slow thing token numbers under different question difficulties?\n- Can the authors provide ablation studies to demonstrate the individual contributions of each reward component and address potential reward hacking from the weighted combination? E.g., reward loss figures through training.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The dual-process reasoning formulation (fast thinking vs. slow thinking) provides an interpretable and cognitively grounded approach to reasoning in LLMs.\n- Results demonstrate a substantial reduction in token usage without significant accuracy loss, with improvements in ACU.\n\nWeakness:\n- There is no ablation study to prove the effectiveness of the proposed reward function. Using a weighted sum of multiple reward functions may easily lead to reward hacking. \n- The optimization objective is somewhat underspecified. While the SFT stage enforces distinct fast and slow thinking patterns, the RL phase does not directly reinforce the intended structural relationship (e.g., slow thinking should be longer than fast thinking), potentially weakening the cognitive alignment.\n- The proposed pipeline is only evaluated on two math datasets without any other task, e.g., code, reasoning gym agent, etc.\n- The performance of the proposed method is worse than L1-max in MATH 500 and AIME 2024 datasets on AUC. Could the author explain why that happens?", "was_revised": true, "final_justification": "Thanks for the response from the authors. My concerns are addressed. I vote for borderline accept now.", "rebuttal": "Thanks for your valuable comments. Our responses to your concerns are as follows:\n\n**Q1&W4: Performance of L1-Max**\n\nWe would like to clarify that both L1-Exact and L1-Max are trained on DeepScaleR-1.5B-Preview, which is a stronger model obtained by further tuning DeepSeek-R1-Distill-Qwen-1.5B using GRPO. As such, the performance of L1-Max benefits from additional training and a more capable backbone.\n\nIn contrast, our ACPO model are trained based on DeepSeek-R1-Distill-Qwen-1.5B, without the additional GRPO training step. Therefore, the comparison is not directly fair in terms of model capacity.\n\nTo reflect this, we report L1 baselines separately in Table 1 as reference points, rather than treating them as primary baselines.\n\n**Q2: Reward Function effects on fast and slow thinking behavior**\n\nIn the reward computation phase of ACPO, for queries whose estimated difficulty exceeds a predefined threshold, our system pattern reward includes a positive term proportional to the ratio of &lt;slow_think&gt; tokens in the response. This design explicitly encourages the model to allocate more slow thinking to harder questions during reinforcement learning. Conversely, for easier queries (below the threshold), the reward encourages a higher proportion of fast thinking.\n\nAs shown in Figure 3, we visualize the distribution of fast vs. slow thinking tokens across different difficulty levels in the MATH dataset. The results show a clear increase in the proportion of slow thinking as task difficulty increases, validating the intended behavior of our reward function.\n\n**Q3&W1: Ablation studies of reward**\n\nTo assess the individual contributions of each reward component, we conduct ablation studies by incrementally adding each reward term during training. The results on MATH 500 and AIME 2024 are summarized below.\n\n| Model | MATH 500|| |AIME 2024|| |\n|------------------------------|--------|--------|--------|--------|--------|--------|\n|               | Accuracy   | #Tokens | ACU| Accuracy | #Tokens | ACU     |\n| DeepSeek-R1-Distill-Qwen-1.5B| 83.9 | 5708 |  0.98  | 28.9 | 16894  | 0.11 |\n| +Accuracy Reward       | 84.5 | 3098  | 1.82   |  29.0 |  12990 |  0.15 |\n| + Accuracy + Online TLB Reward |80.7 |1635 | 3.29| 28.9| 6390| 0.30 |\n| + Accuracy + TLB + System Reward | 81.0|1679 |3.22 | 30.0| 6670 | 0.30 |\n\nFrom the results we can see that:\n1) The accuracy reward improves task performance while moderately reducing token usage. 2) Adding the online TLB reward significantly compresses reasoning length, though it may slightly affect accuracy due to more aggressive compression. 3) The system pattern reward helps recover some accuracy while maintaining efficient token usage.\n\nWe will include reward loss curves and additional analysis in the final version to provide a more comprehensive ablation study.\n\n**W2: The optimization objective of RL phase**\n\nWhile the SFT stage introduces token-level thinking modes, the RL phase reinforces the intended structural relationship through the system pattern reward. Specifically, this reward is positively correlated with the proportion of &lt;slow_think&gt; or &lt;fast_think&gt; tokens, depending on the estimated difficulty of the question.\n\nFor harder questions (above a predefined difficulty threshold), the model receives higher reward for allocating more &lt;slow_think&gt; tokens, promoting more deliberate and thorough reasoning. Conversely, for easier questions, the reward encourages a higher proportion of &lt;fast_think&gt; tokens to favor brevity and efficiency. This dynamic reward design guides the model to allocate cognitive effort adaptively based on task complexity.\n\nAs shown in Figure 3, the model learns to produce longer slow thinking segments for more difficult problems, and shorter reasoning traces for easier ones.\n\n**W3: Generalization to other domains**\n\nWe understand the reviewer’s concern regarding generalization. While our initial experiments focused on math reasoning, we additionally evaluated our method on the GPQA Diamond dataset. As shown in the table below, ACPO significantly reduces reasoning token usage while maintaining comparable accuracy, demonstrating its potential to generalize beyond math.\n\n| Models | Accuracy|#Token|ACU |\n|----------|----------|----------|----------|\n| DeepSeek-R1-Distill-Qwen-1.5B | 33.8 | 22557 | 0.09 |\n|ACPO-1.5B | 32.3 | 11226| 0.19 |\n\nOur method is task-agnostic and can be extended to other domains involving complex reasoning. Our current training data focuses primarily on math problems. Tasks like code generation and agent pose additional challenges due to the task constraints. Nevertheless, we plan to synthesize fast/slow thinking data in these domains to enable supervised fine-tuning and ACPO training, further enhancing the method’s applicability to a broader range of complex reasoning tasks."}]

Author rebuttal:

Meta-review: The domain of experiments limited but this is a nice idea and after the author response was a consensus accept.
Decision: Accept (poster)