ARM: Adaptive Reasoning Model
Authors: Siye Wu
Jian Xie
Yikai Zhang
Aili Chen
Kai Zhang
Yu Su
Yanghua Xiao
Venue: NeurIPS 2025 spotlight ()
Keywords: Large Language Models
Large Reasoning Models
Adaptive Reasoning
Primary area: applications
TL;DR: We propose Adaptive Reasoning Model (ARM), a reasoning model capable of adaptively selecting appropriate reasoning formats based on the task at hand.

Abstract: While large reasoning models demonstrate strong performance on complex tasks, they lack the ability to adjust reasoning token usage based on task difficulty. This often leads to the "overthinking" problem—excessive and unnecessary reasoning—which, although potentially mitigated by human intervention to control the token budget, still fundamentally contradicts the goal of achieving fully autonomous AI. In this work, we propose Adaptive Reasoning Model (ARM), a reasoning model capable of adaptively selecting appropriate reasoning formats based on the task at hand. These formats include three efficient ones—Direct Answer, Short CoT, and Code—as well as a more elaborate format, Long CoT. To train ARM, we introduce Ada-GRPO, an adaptation of Group Relative Policy Optimization (GRPO), which addresses the format collapse issue in traditional GRPO. Ada-GRPO enables ARM to achieve high token efficiency, reducing tokens by an average of $\sim$30%, and up to $\sim$70%, while maintaining performance comparable to the model that relies solely on Long CoT. Furthermore, not only does it improve inference efficiency through reduced token generation, but it also brings a $\sim$2$\times$ speedup in training. In addition to the default Adaptive Mode, ARM supports two additional reasoning modes: 1) Instruction-Guided Mode, which allows users to explicitly specify the reasoning format via special tokens—ideal when the appropriate format is known for a batch of tasks. 2) Consensus-Guided Mode, which aggregates the outputs of the three efficient formats and resorts to Long CoT in case of disagreement, prioritizing performance with higher token usage. All the resources will be released.

ARM: Adaptive Reasoning Model
Siye Wu
Jian Xie v
Yikai Zhang Aili Chen Kai Zhang Yu Su Yanghua Xiao
Shanghai Key Laboratory of Data Science
College of Computer Science and Artificial Intelligence, Fudan University
The Ohio State University Shanghai Academy of AI for Science
siyewu240m. fudan. edu. cn, xie. 17..1774oou.... edu, shawyhōfudan. edu cn
Project Page: https://teet armm wom..githubb..... 10/ arrm
Abstract
While large reasoning models demonstrate strong performance on complex tasks, they lack the ability adjust reasoning token usage based task difficulty. This often leads to the "overthinking" problem-excessive and unnecessary reasoning- which, although potentially mitigated by human intervention to control the token budget, still fundamentally contradicts the goal of achieving fully autonomous Al. In this work, we propose Adaptive Reasoning Model (ARM), reasoning model capable of adaptively selecting appropriate reasoning formats based on the task at hand. These formats include three efficient ones
Answer. Short CoT, and Code-as as well as more elaborate format, Long CoT. To train ARM. we introduce Ada-GRPO, an adaptation of Group Relative Policy Optimization (GRPO), which addresses the format collapse issue in traditional GRPO. Ada-GRPO enables ARM to achieve high token efficiency, reducing tokens by an average of ~ 30%, and up 70%, while maintaining performance comparable the model that relies solely on Long COT. Furthermore, not only does it improve inference efficiency through reduced token generation, but it also brings 2x speedup in training. addition to the default Adaptive Mode, ARM supports two additional reasoning modes: 1) Instruction Guidedd Mode, which allows users explicitly speciffy the reasoning format via special tokens-ideal when the appropriate format is known for batch of tasks. 2) Consensus- Guided Mode, which aggregates the outputs the three efficient formats and resorts to Long CoT in case of disagreement, prioritizing performance with higher token usage.
Introduction
The emergence of large reasoning models (LRMs) such as OpenAl-ol [19] and DeepSeek-RI [11] has led to unprecedented breakthroughs in gobblem-solving capabilities through test-time scaling [3: 57]. These models are designed to solve tasks using Long Chain-of-Thought Thought (Long CoT), generating more tokens to achieve better performance. However, because they are primarily trained on tasks requiring intensive reasoning, LRMs tend to apply Long CoT uniformly across all tasks, resulting in the so- called "overthinking" problem [4; 42). This issue refers to the excessive use of tokens for reasoning, which yields no performance gains and may introduce noise that misleads the model [52; 7].
While some efforts aim to reduce token usage in LRMs, they often rely on clear estimations of the token budget per task [1; 46] or require specialized, length-constrainedd model training [16]. In reality, such estimations are not always accurate, and a more desirable solution is for models to adaptively
PProject lead. Part of the work was done at Fudan University. "ooresponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

where are lilly to find hambburge? (A) restourant (8) pizza (C) piano
Dowm2.5-18'78. 7B/18
50
postiums integeec
General Model
Reasoning Model
AsN
+Ada-GRPO
+GRPO
1.
Restou nnt serves food: (A).
1. Ok, let's see. know that
1. Correct answer (A). Thrn TITHERT
Trye by Tikee..
there's amoser (A).
2. Let's tackle at step by step.
Think sooess
2. Ok, let's trokle this cuestion
Stop 1) Get the
Overthenung
Fin the
postiiue
Elsand product.
integers Altomattvely, could
Stop 3: lsee modular an thimbic...
number theory.. Wast, let me
there be.. Let mh thik again.
Shep 4: Try Semal valuee
vorifyy So according triee
problem
integers
dividee....
128
256
512
1024
2048
ATokens
(a) Comparison of model reasoning behaviors on easy and hard tasks.
(b) Accuracy vs. Token Cost.
Figure 1: (a) Comparison of reasoning behaviors across different models on easy and hard tasks, The General Model fails on harder tasks without elaborate reasoning. The Reasoning Model applies Long CoT across all tasks, causing the "overthinking" phenomenon. In contrast, our proposed ARM adapts its reasoning formats based on task difficulty, answering easy questions efficiently while adopting Long CoT for hard tasks. (b) Accuracy versus token cost for Qwen2.5 under different training strategies. "SFT", "+GRPO", and "+Ada-GRPO" refer to models trained with SFT, SFT+GRPO, and SFT+Ada- GRPO, respectively. "+Ada-GRPO" consistently outperforms the expected trade-off line between "SFT" and "+GRPO," demonstrating ARM'S superior fffectiveness-efffiiinne balance.
control their token usage based task complexity without human intervention. For example, for the first easy question in Figure la, answering directly is the ideal choice, whereas for the second, hard question, the use of Long CoT is precisely what is needed. Thus, Long CoT is not a "silver bullet"; selecting an appropriate reasoning format is essential to balance efficiency and effectiveness.
In this work, we propose Adaptive Reasoning Model (ARM), reasoning model capable of adaptively selecting reasoning formats based on task difficulty, balancing both performance and computational efficiency. ARM supports four reasoning formats: three efficient ones-Direct Answer, Shori CoT, and Code one elaborate format, Long CoT. In addition to its adaptive selection mechanism Mode, which allows explicit control over
(Adaptive Mode), ARM also supports Instruction-Guiie the reasoning format via special tokens, and a Consensus- Uunsensus-Guided Mode, which aggregates the outputs of the three efficient formats and resorts to Long CoTf in case of disagreement.
To train ARM, we adopt two-stage training framework. Stage 1, we apply supervised fine-tuning (SFT) to equip the language model with foundational understanding of four reasoning formats. In Stage 2, we introduce Ada-GRPO, an adaptation Group Relative Policy Optimization (GRPO) [38], which encourages efficient format selection while preserving accuracy as the primary objective. Ada-GRPO is designed to address two key issues: 1) The uniform distribution of reasoning formats regardless of task difficulty observed during the SFT stage; 2) The format collapse problem in GRPO, where Long CoT gradually dominates as training progresses, leading to the diminished use of other, more efficient formats. Extensive evaluations show that ARM trained with Ada-GRPO achieves comparable performance while using ~ 30% fewer tokens than GRPO (as shown in Figure lb), across both in-domain and of--ai- tasks in commonsense, mathematical, and symbolic reasoning. Furthermore, by leveraging the three more efficient reasoning formats in the roll-out stage, Ada-GRPO achieves approximately 2 training speedup compared to GRPO.
Our additional analysis further reveals that: 1) Adaptive Mode achieves a superior balance be- tween effectiveness and token efficiency by adaptively selecting suitable reasoning formats, while Instruction- Guidedd Mode performs well when the specified format is suitable for the task, and Paasensus-Guided Mode prioritizes performance the cost of higher token usage. 2) The choice of backbone model has limited impact on ARM'S performance when using base or Instruction-tunct models, which yield similar results; however, using the DeepSeek-R1-Disse performance on hard tasks due to the advanced reasoning capability distilled from the strong teacher DeepSeek-RR R11, but leads to worse performance on easy tasks despite increased token cost. 3) Length- cenalty-basee strategies for improving LRM efficiency suffer from performance degradation as the token budget decreases, whereas ARM maintains stable performance.
backbone improves
In summary, our contributions are three-fold: /) We propose ARM, a reasoning model that balances effectiveness and efficiency by adaptively selecting task-appropriate reasoning formats. Compared to

---

the model that relies solely on Long CoT. ARM achieves comparable performance while significantly reducing token cost, saving an average of 30% and up to 70%. 2) In addition to the default Adap- tive Mode, ARM also supports Instruction- format is appropriately specified, and Consensus-Guidee Guided Mode, which maximizes performance the cost of higher token usage. 3) We introduce Ada-GRPO, an adaptation of GRPO that addresses the format collapse problem and achieves a ~ 2x training speedup without compromising performance.
Mode, which performs well when the reasoning
2 Related Work
2.1 Reinforcement Learning for Improving Reasoning
Reinforcement Learning (RL) has demonstrated significant potential in enhancing the problem- solving abilities of large language models (LLMs) across various domains [33; 47; 20J. Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has gained substantial attention for ad- vancing LLM capabilities [21; 11; 25), resulting in the development of large reasoning models (LRMs) [53] such as OpenAI-o1 [19] and DeepSeck-RI [11]. Based simple rule-based rewards, RLVR algorithms such as Group Relative Policy Optimization (GRPO) [38] enable models to use Long Chain-of-Thouglh (Long CоT) [56; 18]. This facilitates deep reasoning behaviors, such as searching, backtracking, and verifying through test-time scaling [3: 57J. However, these models also suffer from significant computational overhead due extended outputs across all tasks, leading to inefficiency associated with the "overthinking" phenomenon [4; 35; 42]. Verbose and redundant outputs obscure logical clarity and hinder the model's ability to solve problems effectively [52: 7].
2.2 Efficiency in Large Language Models
Recently, many studies have focused on improving reasoning efficiency in LLMs. Some prompt- guided methods [12: 13; 54; 22] explicitly instruct LLMs to generate concise reasoning outputs by controlling input properties such as task difficulty and response length. Other approaches [14; 5; 39] explore training LLMs to reason in latent space, generating the direct answer without the need for detailed language tokens. Several techniques have also been proposed to reduce inference costs by controlling or pruning output length, either by injecting multiple reasoning formats during the pre-training stage [41] or by applying length penalties during the RL stage [45; 2: 1; 16]. Many of these methods aim to strike a trade-off between token budget and reasoning performance by shortening output lengths, often relying on clear estimations of the token budget for each task or requiring specialized, length-constrained model training. However, in reality, such estimations are not always accurate, and what we ultimately expect is for models to adaptively regulate their token usage based on the complexity of the task at hand. Therefore, in this work, we propose a novel training framework that enables models to adaptively select suitable reasoning formats for given tasks by themselves, optimizing both performance and computational efficiency.
3 Method
We propose Adaptive Reasoning Model (ARM), a reasoning model designed to optimize effectiveness and efficiency by adaptively selecting reasoning formats. Specifically, ARM is trained in two stages: 1) Stage 1: Supervised Fine-tuning (SFT) for Reasoning Formats Understanding: In this stage, use 10.8K diverse questions, each annotated with solutions in four distinct reasoning formats, to fine-tune the model and build a foundational understanding of different reasoning strategies. 2) Stage 2: Reinforcement Learning (RL) for Encouraging Efficient Format Selection: We adopt an adapted version of the GRPO algorithm, named Ada-GRPO, to train the model to be capable of selecting more efficient reasoning formats over solely Long CoT, while maintaining accuracy.
3.1 Stage 1: SFT for Reasoning Formats Understanding
In this stage, we leverage SFT as cold start to introduce the model to various reasoning formats it can utilize solve problems. These formats include three efficient reasoning formats Direct
'In preliminary experiments. models without SFT failed to distinguish between the four reasoning formats. like producing mixed outputs, wrapping a Short CoT response using the special tokens intended for Long CoT.
3

---

Answer, Short CoT, and Code, as well as the elaborate reasoning format Long CoT. We use special tokens (e.g.. <Code></Code>) to embrace thinking rationale. Specifically, 1) Direct Answer: This format provides a direct answer without any reasoning chain, making it the most efficient in terms of token usage. 2) Short CoT: This format begins with short reasoning and then provides an answer, which has been proved effective in mathematical problems [49]. 3) Code: This format adopts code-based reasoning. which has proven effective across a variety of tasks due to its structured process [50; 51; 24]. 4) Long CoT: This format involves more detailed, iterative reasoning process, thus incurs higher token usage. is suited for tasks requiring advanced reasoning capabilities, such as self-reffection and alternative generation, where those more efficient formats fall short [31; 11; 56].
3.2 Stage 2: RL for Encouraging Efficient Format Selection
After SFT, the model learns to respond using various reasoning formats but lacks the ability to adaptively switch between them based on the task (see Section 4.3 for details). To address this, we propose Adaptive GRPO (Ada-GRPO), which enables the model to dynamically select appropriate reasoning formats according to the task difficulty through a format diversity reward mechanism.
GRPO In traditional GRPO [38], the model samples a group of outputs O {or, 02. .og ) for each question q. where G denotes the group size. For each O₁, a binary reward r; is computed using sule-based reward function that checks whether the prediction pred matches the ground truth gt:
T2 I paneed(pt.precd
(I)
However, since traditional GRPO solely optimizes for accuracy, it leads, in our setting, to overuse of the highest-accuracy format while discouraging exploration of alternative reasoning formats. Specifically, if Long CoT achieves higher accuracy than other formats, models trained with GRPO tend to increasingly reinforce it, leading to an over-reliance on Long CoT and reduced exploration of more efficient alternatives. We refer to this phenomenon Format Collapse, which ultimately hinders the model's ability to develop adaptiveness. We further analyze this in Section 4.3.
Ada-GRPO We propose Ada-GRPO to address the format collapse issue. Specifically, Ada-GRPO amplifies the reward r, for less frequently sampled reasoning formats, preventing their disappearance and ensuring adequate learning. Formally, we scale the reward r; to r by:
r (t) rd.
(2)
(t F(oo) (ecay)(t),
(3)
(t)ay:(t) F(o:) + 0.5
(I-F(aa) F(oz))
[1+cos cos (
π (T.7)
(4)
G
G
where F(o,) denotes the number of times the reasoning format corresponding to o, appears within its group O, and represents the training step. (;(t) is a format diversity scaling factor that gradually decreases from G at the beginning of training 0) to at the end of training (t = T).
F(o.)
We introduce 0s (t) to extend GRPO into Ada-GRPO, enabling models to adaptively select reasoning formats. Specifically, or (t) consists of two components: 1) Format Diversity Scaling Factor F(o:)* To prevent premature convergence on the
format (i.e., format collapse to Long CoTT), we upweight rewards for less frequent formats to encourage exploration. 2) Decay Factor decay, (t): To avoid long-term misalignment caused by over-rewarding rare formats, this term gradually reduces the influence of diversity over time. For example, F(a,J might make the model favor a lower-accuracy format like Short CoT over Long CoT simply because it appears less frequently and thus receives higher reward. While such exploration is beneficial early in training, it can hinder convergence later. The decay mechanism mitigates this by promoting diversity initially, then shifting focus to accuracy again as training progresses. Refer to Appendix A for details of the decay factor.
G
Then the group advantage Á for all tokens in each output is computed based on the group of reshaped rewards r' {rr, T.. rro)=
ALa r/ mean((ri, ,.,))
(5)
std(fri, 12 rra))
4

---

Finally, we optimize the model by maximizing the following objective (see Appendix A for details):
lo
70(0r,u99,0,0 0r,ck)
JAdo-GRPO(0) =B P(Q): [or] Tomm (0/q)
min
Ai,e:
D
Jeald
(1, (,,ck))
i=
k=
clip
Aa (0.. (..4) 0c,ck)
1 x Arx
- -8 3KL
(TT || Mear] }}
(6)
4 Experiment
4.1 Experimental Setup
Model
To assess the effectiveness of our method across different model sizes, we select Qwen2.5-
Base-3B/7B/14B [55] as backbone models. We further examine models of the same family but fine-tuned on different datasets, specifically the Instruct [55] and PreppSek-RI-Disiie variants [11], which exhibit varying levels of base reasoning capabilities. detailed analysis given in Section 5.3.
Training Datasets Stage 1: We use AQuA-Rat [26] as the SFT dataset, as its answers can be naturally transformed into four distinct reasoning formats. In addition to the Direct Answer and Short CoT rationales provided with the dataset, we utilize GPT-4o [30] and DeepSeek-RI [11] to supplement the Code and Long CoT rationales, respectively. To ensure the quality of the generated rationales, we filter out those that lead to incorrect answers, resulting in a training set containing 3.0K multiple-choice and 7.8K open-form questions, each with four reasoning formats. Appendix B provides further details on the generation and filtering process. Stage 2: To prevent data leakage, we employ three additional datasets exclusively for the RL stage." 4 These datasets cover a range of difficulty levels, from relatively simple commonsense reasoning tasks to more complex mathematical reasoning tasks, including Commonsense(AA (CSQA) [44], GSM8K [6], and MATH [15], collectively comprising 19.8K verifiable question-answer pairs. Please refer to Appendix C for details of the datasets we use and Appendix D for implementation details.
Baselines In addition to backbone models, we compare ARM with models trained using alternative algorithms that may enable adaptive reasoning capabilities. Specifically, Qwen2.5srT refers the backbone models trained on the AQuA- dataset used in Stage 1. In this setting, we explore whether language models can master adaptive reasoning through straightforward SFT strategy. For Qwen2.5sFT-GRPo, we examine whether SFT models, further trained with GRPO, can better understand di different reasoning formats and whether this approach empowers them select appropriate reasoning formats based on rule-based rewards.
4.2 Evaluation
Evaluation Datasets To assess the models' reasoning capabilities, we select range of evaluation datasets, including both in-domain and out-of-domain samples. These datasets span commonsense, mathematical, and symbolic reasoning tasks. For commonsense reasoning, we include Common- senseQA (CSQA)[44) and OpenBookQA (0BQ))22)), which are easier tasks based on intuitive knowledge. For mathematical reasoning, we utilize SVAMP [34], GSM8K [6], MATH [15], and AIME'25 [9] to assess models' ability to solve complex mathematical problems that require advanced reasoning and strict logical thinking. For symbolic reasoning, we turn to Big-Bench-Bee
(BBH) [43],
a benchmark for evaluating models' structured reasoning ability to manipulate symbols according to formal rules. For further analysis, we group the evaluation datasets into three difficulty levels: commonsense tasks easy; mathematical and symbolic tasks medium; and AIME'25 as hard given its competition-leww difficulty.
Inference During inference, we set the temperature to 0.7 and top-p to 1.0. For all evaluation datasets, we use accuracy as the metric. addition to pass@ to reduce bias and uncertainty associated with single generation outputs and to enhance the robustness of the results [57], we further use majority @k (maj@k) k), which measures the correctness of the majority vote from k independently
'In preliminary experiments, we observed that using the same training data in both stages causes the model recite answers rather than reasoning during the RL stage, resulting in poor generalization.
5

---

Table 1: Performance of various models across evaluation datasets. "#Tokens" refers to the token cost for each model on each dataset. For each model, & = 1 corresponds to pass@1 1, and k = 8 corresponds to maj @8. When l = 8, the token cost is averaged over single output to facilitate clear comparison. "*" denotes in-domain tasks, while "#" denotes out-of-domain tasks. "A" represents the difference between ARM and Qwen2.5sPT-GRRM
calculated by subtracting the accuracy of Qwen2.5smT-GRPO from that of ARM, with the token usage expressed as the ratio of tokens saved by ARM compared to Qwen2.5sFT-ORBD with all settings based on l to ensure a stable comparison.
Arruracy (f)
#Tokens (D)
Models
Easy
Medium
Hand
Ang.
Eany
Mediun
Hurd
Avg.
(BQ4]
GSMEK)
MATHI
SVAMP,
BRI11
AIME
CSQAi
[BQ44]
(SMMKK
MATH()
SVAMP)
00:11
AIME"1.)
GPT-4o el prpresiiw
85.9
94.2
95.9
75.9
91.3
$4-7
10.0
76.8
192
165
257
663
1.56
278
984
359
85.5
95.5
04.2
$2.6
91.3
91.8
40.0
54.6
57%
493
445
1961
4859
0480
7919
1819
of mini high
84.7
95.0
96.9
97.7
94.0
92.2
96.7
94.0
502
259
139
1332
301
755
9550
1910
DecpSazk-VV
82.4
06.5
91.3
85.8
56.3
81.3
150
400
712
DeepSeek
83.3
38.9
1030
DS (Diilll-1 5B
$3.5
20,0
3425
DS Rl Dieill-B
64.9
77.4
90.0
93.6
90.3
72.1
40.0
75.5
792
925
574
3001
315
1468
12427
2707
DS kt D415
80.6
41.2
96.1
95.5
V2.7
$11.6
50.0
N1.5
816
750
625
2002
736
1292
11004
2545
DS RI Discill-2BB
83.2
94.5
93.5
93.0
92.0
$6.3
56.7
65.6
674
096
4.5
2.61
253
595
11276
2561
5-M0
25.5
38.4
76
355
Quen2
75.5
45.5
79.7
80.3
56.2
1590
1173
A
80LI
79.6
78.0
90.8
72.8
93.0
53.8
67
65.3
123
169
350
1006
246
430
3063
756
778
-0.2
-2:0
-0.5
-3.3
+0.3
-2.4
13:35
-71.35 13.3%
-66.0%
-55.215
61.415
-50.55
-5.155
-33.01
Quee? 5-711
76.7
62.0
75.6
$1.6
50.1
$1.00
51.7
62.0
2.3
60.4
56
64
53
156
376
370
99
181
183
767
260
247
Qwee25 7Byr
M13.9
80.8
7M
76.0
56.
48.2
53.0
136
348
1.66
245
374
373
347
Qwen2 7Bymmotr
83.1
64.3
16.7
1410
557
1133
3196
1173
ARA--18
86.1
61.4
16.7
73.0
413
756
A
+2.0
+1.2
-2.5
-1.4
-0.2
773.0%%
-5.45%
-60.15
-96.99 95L
-55.0%
47.9%
-52.55
Qweed 5-14B
70.9
83.8
83.4
00.2
92.3
44.9
68.4
60.2
91.7
B4.
67.4
56.8
3.3
1.3
61.3
71.0
55
56
50)
50
131
193
325
515
81
23
131
1.9
611
735
217
201
Quen25 14BBrr
81.8
85.0
83.0
01.4
62.6
37.4
8410
0L3
64.6
53.5
33
0
56.2
65.3
155
149
140
161
276
25R
152
140
254
2.43
527
232
238
Quen2 siio
85.8
M1.0
96.8
96.1
81.3
87.1
V17
77.0
745
2000
20.0
79.4
75.0
51N
S52
1810
1815
565
565
943
548
4031
1261
1336
ARX--148
85.3
92.5
91.1
65.
300
25.5
146
212
430
85%
85.6
957
72.
23.
76.7
145
910
159
415
355
sampled outputs. For inference on the three backbone models, we use an example with a short-cot- based answer within the prompt to guide the model toward specific answer formats while preserving its original reasoning capabilities as much as possible.
4.3 Main Results
Alongside our baselines, we include several state-of-the-art general models, including GPT-4o [30] and DeepSeek- V3 [27], as well as reasoning models ol -preview [31], 04-mini-high [32], and DeepSeek-RR R1 [11], along with several DeepSeek-RI RI-iisiill Qwen (DS-RI-Distill) models ranging from 1.5B to 32B [11]. We report our results in Table I, and we have the following findings:
Current reasoning models struggle with the "overthinking" problem, with smaller distilled models being more affected. We observe that all current reasoning models consume more than 500 tokens on easy commonsense tasks but do not always achieve corresponding improvements. For example, although DeepSeek-RI and DS-RI -Disttill-- use nearly 4x and 10x more tokens than their backbone models, DeepSeek-V3 and Qwen2.5-7B, they do not show significant improvement and even experience performance degradation, highlighting the "overthinking" problem. Additionally, we find that when comparing different sizes of tokens while delivering worse performance.
smaller models often require more
SFT only teaches models about formats, yet does not teach how to choose the appropriate formats based on the task. We observe that SFT models, across three sizes, show improvement on easy commonsense tasks but experience performance drops on medium and hard tasks. To investigate the cause, we conduct a deeper analysis of the reasoning formats selected during inference. Figure visualizes how models allocate the four reasoning formats across three difficulty levels. Specifically, we find that for models trained with SFT, their outputs are distributed almost uniformly across the reasoning formats, with the majority in Direct Answer and the least in Long CoT, regardless of task difficulty. As shown in Figure 2, the inappropriate selection of Direct Answer, which yields extremely low accuracy (35.2%) on medium tasks and significantly hinders the model's 's reasoning capabilities,

---

Direct
Short CoTT
Code
Long CoTT
SFT
36.1
24.9
2
25.9
24.3
12.2
40.0
23.3
26.7
10.0
GFT-GRPO
100.8
100.6
100.0
ARM
21.
27.8
41.0
20.0
56.7
20
40
60
80
100
20
40
60.
80
100
0
20
40
60
80
100
(a) Easy Tasks
(b) Medium Tasks
(c) Hard Tasks
Figure 2:
Format distribution by task difficulty with Qwen2.5-7B. The hatched areas indicate the percentage of correct answers that were generated using the selected reasoning format.
Table 2: Accuracy (Acc.) and token usage (Tok.) for the three reasoning modes supported by ARM-7B. In the Consensus-Guided Mode, the percentage of Long CoT usage indicates how often the model resorts to Long CoT when simpler reasoning formats fail to reach a consensus.
Easy
Medium
Hard
Avg.
ARM-7B
CSQAT
OBQA
GSM8K
MATH
SVAMP
BBH
AIME
25g
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Adaptive
86.1
136
84,4
159
89.2
305
73,9
889
92.0
218
61.4
401
16.7
3253
72.0
766
Instowerr
84.1
10
81.8
10
22.9
11
23.1
67.0
11
44.7
12
46.2
13
Tnstster car
81.3
33
77.4
35
85.0
124
70.9
633
86.7
66
49.7
101
10.0
2010
65.9
428
Instowie
84.4
140
81.6
147
84.2
285
65.9
559
88.3
182
57.9
344
10.0
1821
67.5
497
Instaon CuT
84.0
259
87.4
294
91.8
426
77.2
1220
94.3
340
66.9
660
20.0
4130
74.5
1047
Consensus
85.8
228
87.0
260
92.9
777
78.4
2281
95.7
433
66.4
1039
20.0
7973
75.2
1856
Long CoT Usage
12.9%
21.4%
79.8%
79.2%
36.3%
56.3%
100%
55.1%
finally leads decline in overall performance. This suggests that while SFT teaches models various formats, it fails to help them adaptively select appropriate ones based on the task, leading to an inability to choose more advanced formats as problem complexity increases.
GRPO does improve reasoning capabilities, but it tends to rely on Long CoT to solve all tasks. We observe that models trained with GRPO achieve significant improvements across all tasks, yet the token cost remains substantial, especially for the two easier tasks. Further analysis reveals that Long CoT' is predominantly used in the inference stage, shown in Figure 2. This behavior stems from the nature of GRPO (i.e., format collapse discussed in Section 3.2), where models converge to the format with the highest accuracy (i.c., Long CoT) early in training (~10 steps in our experiment). As a result, GRPO also fails to teach models how to select more efficient reasoning format based on the task. We provide more details of format collapse in Appendix E.
ARM is able to adaptively select reasoning formats based on task difficulty, while achieving comparable accuracy across all tasks compared to GRPO and using significantly fewer tokens. As shown in Table 1, across three different model sizes, all ARMS experience an average performance drop of less than 1 % compared to models trained with GRPO, yet they save more than 30% of the tokens. Specifically, ARM demonstrates a clear advantage on easy tasks, saving over 70% of tokens while maintaining comparable accuracy. This advantage extends to medium tasks well. For the more challenging AIME'23 25 task, ARM adapts to the task difficulty by increasingly selecting Long CoT, thereby avoiding performance degradation on harder tasks, with ARM- 14B even surpassing its counterpart
1 Figure 2 further confirms that ARM is able to gradually adopt more advanced reasoning formats and discards simpler ones as task difficulty increases. Moreover, as shown in Figure 1b, the line connecting "SFT" and "+GRPO" illustrates the expected trade-off, while *+Ada- Aada-GRPO" consistently lies above it, indicating better balance between effectiveness and efficiency of ARM. Additionally. ARM-7B achieves comparable performance to 1S-RI-Distill-7 while using only 27. 27.8% of the tokens on average. For a broader view of generalization across additional benchmarks, please refer to Appendix F.
4.4 Reasoning Mode Switching
ARM is capable of autonomously selecting appropriate reasoning formats (Adaptive Mode), while also supporting explicit guidance to reason in specified formats Enstruction-Guidee Mode) or through consensus between different reasoning formats (Consensus-GGiice Mode). Specifically, 1) Adaptive Mode: In this mode, ARM autonomously selects the reasoning format for each task, which is also the
7

---

Accuracy(%)
80
74.9
70.2
Adaptive
68.6
65.9
70.2
Instruction-Guidee Guided
67.5
75.2
74.5
T-Accuracy
1.0
Accuracc
q-f Tokens
1.01
1.0
T-# Tokens
0.96
60
0.9-
0.93
0.35
0.34
0.31
0.3
40
Direct
Short CoT
Code
Long CoT
TP
ARM
Figure 3: Accuracy comparison between ARM's Adaptive and Instruction cerpuctiionGuuded modes. The figure shows average accuracy across evaluation datasets, with Direct Answer applied only to commonsense and symbolic tasks, as does not appear in mathematical tasks in Adaptive mode.
Figure 4: Relative accuracy and token usage of different models compared to their backbone models on CSQA. "LI" denotes LI-Exact [1], and "TP" denotes THINKPRUNE [16]. "r-
Accuracy" and "r-#Tokens" are reported relative to each model's backbone after RL. training.
default reasoning mode if not specified this paper. 2) Instruction-Guidee Mode: In this mode, in the
specific token (e.g., <Long Col>) is provided as the first input, forcing ARM to specified format. 3) Consensus-Guided Mode: this mode, ARM first generates answers using the three simpler reasoning formats (i.e., Direct Answer, Short CoT, and Code) and checks for consensus among them. If all formats agree, the consensus answer is adopted as the final result. Otherwise, ARM defaults Long CoT for the final answer, treating the task sufficiently complex.
To evaluate the performance and effectiveness of the proposed reasoning modes, we conduct exper- iments across various evaluation datasets. Table 2 presents the results for ARM-7B. Specifically: 1) Adaptive Mode strikes a superior balance between high accuracy and efficient token us- age across all datasets, demonstrating its ability to adaptively select the reasoning formats. 2) Instruction-Guided Mode offers clear advantage when the assigned reasoning format is appropriate. For example, Direct Answer is sufficient for commonsense tasks, while Code, due to its structured nature, performs better on symbolic reasoning tasks compared to Direct Answer and Short CoT. Furthermore, Instrong CoT achieves better performance (74.5%) than the same-sized model trained on GRPO (73.2% in Table 1). This demonstrates that Ada-GRPO does not hinder the model's Long CoT reasoning capabilities. We further validate this by analyzing the reflective words used by ARM-7B and other hand, is performance-orienter requiring more tokens This mode leverages consensus across multiple formats to mitigate bias and uncertainty present in any single format, offering greater reliability, particularly for reasoning tasks that demand advanced cognitive capabilities, where simpler formats may fall short. This is evidenced by the fact that Long CoT is less likely to be used for easy tasks, but is highly likely to be selected for medium tasks and even used 100% of the time for the most difficult A AIME'25 task.
in Appendix G. 3) Consensus-Guided Mode, on the
achieve better performance.
5 Analysis
5.1
Effectiveness of Adaptive Format Selection
To verify that ARM'S format selection indeed adapts to the task at hand rather than relying random selection, we compare ARM'S Adaptive Mode with Instruction-Guided Mode. In Instruction- Guided Mode, the reasoning format is fixed and manually specified, providing a strong baseline to test whether adaptive selection offers real benefits over using a uniform format across tasks. We report the accuracy of both modes in Figure 3. We observe that the accuracy of the reasoning formats selected in Adaptive Mode higher than that in Instruction- Ciaruction-Guidedd Mode. Specifically, Adaptive Mode improves accuracy by 4.7% Direct Answer, by 2.7% on both Short CoT and Code, and yields slight improvement on Long CoT. These results confirm that ARM is not randomly switching formats but is instead learning select appropriate for each task. Further ablation results reasoning formats are presented Appendix H.
5.2 Comparison of Ada-GRPO and GRPO
We find that, compared to GRPO, ARM trained with Ada-GRPO achieves comparable performance on the evaluation dataset while achieving approximately speedup training time. To understand the source of this efficiency, compare the training dynamics of Ada- Ada-GRPO and GRPO across

---

Ada-GRPO
GRPO
7B AdaOGRPO
BB GRPO
14B Ada-GRPO
GRPO
Response Length
Aceurccy(%)
1000
2500
Time (6)
750
1500
50-
500
250
500
30
0
50
100
150
50
100
150
50
100
150
Step
Step
Step
Figure 5: Performance on the training set across different model sizes trained with Ada-GRPO and
GRPO. Except for the implementation of the algorithm, all hyperparameters are kept the same.
Qwen2.5-7B
Qwen2.5-2 1een2..-BBInstruuc
D--RI-DisilllQwen--B
#Tokens
90
Acourncy(%)
256
512
40
4K
75
75
20
2K
60
60
256
(a) Easy Tasks
(b) Medium
(c) Hard Tasks
Figure ARMS' performance across different backbones. Base and instruction-tumed models perform similarly, while DS-RI-Distill improves medium and hard tasks but struggles easy ones.
different model sizes, focusing on accuracy, response length, and training time, as shown in Figure 5, The results highlight the following advantages of Ada-GRPO: 1) Comparable Accuracy. Although Ada-GRPO initially lags behind GRPO in accuracy due to suboptimal reasoning format selection in the early training steps, both methods converge similar final accuracy across all model sizes. This demonstrates that Ada-GRPO does not compromise final performance. 2) Half Response Length. While GRPO uses Long CoT uniformly across tasks, Ada-GRPO adaptively selects reasoning formats based on task difficulty. Due to the length efficiency of Direct Answer, Short CoT, and Code, Ada-GRPO ultimately reduces the average response length to roughly half that of GRPO. 3) Half Training Time Cost. Since the majority of training time is spent response generation during the roll-out stage, reducing response length directly translates into lower time cost. As a result, Ada-GRPO achieves approximately
2 speedup compared to GRPO. Overall, Ada-GRPO maintains strong performance while significantly reducing computational overhead, underscoring its efficiency and reliability for training.
5.3 Comparison of Backbone Models
Beyond the base model, we further analyze the impact of different backbone models, including instruction-tune -tuneed and DS-RI ---ssilll variants. Figure reports accuracy and token usage across easy, medium, and hard tasks. We observe that base and instruction tuned models have highly similar performance. This suggests that RL effectively bridges the gap left by instruction tuning, enabling base models to achieve comparable performance, consistent with findings from previous work [20]. In contrast, the DS-RI-Disshe variant performs notably better on medium and hard tasks, benefiting from distilled knowledge from the stronger DeepSeek-R model, though at the expense of increased token cost. However, performs significantly worse on easy tasks, even with excessive token usage, resulting from the overthinking phenomenon. Additional discussion and case studies on the overthinking phenomenon are presented in Appendix and a complementary analysis of LLaMA-based backbones is included in Appendix J.
5.4 Comparison of ARM and Senngh--ennaly--o -Based Strategies
To examine whether previously proposed length penalty- based strategies- proven effectivee in com- plex ceasoning-remane effective for easier tasks, we evaluate two representative methods. LI [1] and THINKPRUNE [16], on the CSQA dataset. Since both methods are based on the DS-R1-Diss model, we ensure fair comparison by also evaluating the version of ARM trained on the same backbone. We report the relative accuracy and token usage of all three models compared to their respective backbone models Figure 4. When using the minimum allowed lengths specified the official settings of L.1 and THINKPRUNE both methods exhibit performance drops. In contrast, ARM maintains strong performance while using relatively fewer tokens, demonstrating ability to balance reasoning efficiency and effectiveness. Please see Appendix K for further discussion and details.

---

6 Conclusion
In this work, we propose Adaptive Reasoning Model (ARM), which adaptively selects reasoning formats based on task difficulty, ARM is trained with Ada-GRPO, a GRPO variant that addresses format collapse via a format diversity reward and achieves ~ 2x training speedup. Experiments show that ARM maintains performance comparable to the GRPO-trained model relying solely on Long CoT, while significantly improving token efficiency. Beyond the default Adaptive Mode, ARM also supports Instruction-Guided Mode, which excels when the format is appropriately specified, and Consensus- Guided Mode, which maximizes performance at higher token usage. By adopting the adaptive reasoning format selection strategy, ARM effectively mitigates the overthinking problem and offers novel, efficient approach to reducing unnecessary reasoning overhead.
References
[1] Pranjal Aggarwal and Sean Welleck, LI: Controlling how long reasoning model thinks with reinforcement learning, arXiv preprint arXiv:2503. 046677, 2025.
[2] Daman Arora and Andrea Zanette. Training language models to reason efficiently, arXiv preprint arXiv:2502.0446. 2025.
[3] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of-thhough for reasoning large language models. arXiv preprint wwXiv:2503.09500 2025.
[4] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like Ilms. arXiv preprint arXiv:2412.21187, 2024.
[5] Jeffrey Cheng and Benjamin Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint arXiv:2412.13317. 2024.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21100 10..144..8,, 2021.
[7] Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill? arXiv preprint arXiv:2504.065 2025.
[8] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-3666 361,-61,122 2021.
[9] Google. Aime problems and solutions, 2025. htttps::/attoofprollemsss///// com/ wiki/index php/ PII/AIME././.../......A.IIIIIA Problems soluutiioo
[10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407:21783, 2024.
[11] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arX1v:2501.12948, 2025.
[12] Simeng Han, Tianyu Liu, Chuhan Li, Xuyuan Xiong, and Arman Cohan. Hybridmind: Meta selection of natural language and symbolic language for enhanced Ilm reasoning. arXiv preprint Stti::2009.1988/. 2024.
[13] Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. -ake Ilm reasoning. arXiv preprint 12. 2024.
[14] Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint 27Xiv:2412.06769, 2024.
10

---

[15] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[16] Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of Ilms via reinforcement learning. arXiv preprint wwXiv:2504.01290 2025.
[17] Edward Hu. yelong shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL /tttps://operrevierr
[18) Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zere An open source approach scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24229, 2025.
[19] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
[20] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 2025.
[21] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu 3: Pushing frontiers open language model post-training. arXiv preprint arXiv:2411.1512 2024.
[22] Ayeong Lee, Ethan Che, and Tianyi Peng. How well do ilms compress their own chain-of- thought? token complexity approach. arXiv preprint wrXiv:2503.01444 2025.
[23] Chengshu Li, Jacky Liang. Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and brian ichter. Chain of code: Reasoning with a language model-augmentedd code emulator. In Forty-first International Conference on Machine Learning, 2024. URL htwph:tt///// /ttp::///opeeeeeeeeeeeeeee
net/ forumrrmmdeee www.mmmidiwwwtoolmasom
[24] Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condensing reasoning patterns via code input-output prediction. arXiv preprint arXiv:2502.07316, 2025.
[25] Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system to system 2: A survey of reasoning large language models. arXiv preprini 2rXiv:2502.17419, 2025.
[26] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation:
Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 158-167, 2017.
[27] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 2xXi::2412.19437, 2024.
[28] Yan Ma, Steffi Chern, Xuyang Shen, Yiran Zhong, and Pengfei Liu. Rethinking rl scaling for vision language models: transparent, from comm-ccaathh framework and comprehensive evaluation scheme. arXiv preprint arXiv:2504.0255 2025.
[29] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct clectricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, 2018.
[30] OpenAl. Hello GPT-4o, 2024. URL https://oopnaa..cm//nn////////l/

---

[31] OpenAI. Learning to reason with lmm,, 2024. URL http:////opeaii.ooiinieeeeee learning- to reason- rewserewwihh-hleeee------------- 11ms.
[32] OpenAI. Introducing openai o3 and o4-mini, 2025. URL http:::///opaaa.co//iii introducing- o3- and- 04- mini/.
[33] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[34] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, 2021.
[35] Xiao Pu, Michael Saxon, Wenyue Hua, and William Yang Wang. Thoughtterninator: Bench- marking, calibrating, and mitigating overthinking in reasoning models. arXiv preprint arXiv:2504.13300 2025.
[36] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505-3506, 2020.
[37] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proo q&a benchmark. In First Conference on Language Modeling, 2024.
[38] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[39] Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing thatout into continuous space via self-distillation. arXiv preprins arXiv:2502.21074, 2025.
[40] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409. 19256, 2024.
[41] DiJia Su, Sainbayar Sukhbaatar, Michael Rabbat, Yuandong Tian, and Qinqing Zheng. Dual- former: Controllable fast and slow thinking by learning with randomized reasoning traces. In The Thirteenth International Conference on Learning Representations, 2024.
[42] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025.
[43] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint urXiv: 10.00266, 2022.
[44] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqaa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4149-4158 2019.
[45] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025.
12

---

[46] Qwen Team. Qwen3 technical report, 2025. URL httpp::/giite www. CommqwenLM/Qeent
http://eeeeeciiaaaaegeeeeeeee pdf.
[47] Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reason- ing with reinforced fine-tuning, In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7601-7614, 2024.
[48] Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling, arXiv preprint ar arXiv:2506.20512, 2025.
[49] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thouge prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[50] Nathaniel Weir, Muhammad Khalifa, Linlu Qiu, Orion Weller, and Peter Clark. Learning to reason via program generation, emulation, and search. In The Thirty-eigtt Annual Conference on Neural Information Processing Systems, 2024. URL htttp:://pppereeeeeeeee wwwprreviieeeneeefrrreee net/forum?
id=te6VagJf6G.
[51] Jiaxin Wen, Jian Guan, Hongning Wang, Wei Wu, and Minlie Huang. Codeplan: Unlocking reasoning potential in large language models by scaling code-form planning. In The Thirteenth International Conference on Learning Representations, 2025. URL httpss:/oppenreviie
[52] Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. How easily do irrelevant inputs skew the responses of large language models? In First Conference on Language Modeling, 2024. URL
net wtt.:/eeetoooomdddd////////
[53] Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09% 2025.
[54] Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengeheng He. Chain of draft: Thinking faster by writing less. arXiv preprint 100.002502.18600, 2025.
[55] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1511 15115, 2024.
[56] Weihao Zeng. Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
[57] Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? survey test-time scaling in large language models. arXiv preprint 25Xi:25503.22335, 2025.
[58] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400-410, 2024.
13

---

Appendix
A Details of Ada-GRPO
A.1 Training Objective
Following GRPO [38], given query q and of responses {01, .2....,.0. oc) sampled from
the old policy Told- we optimize the policy model i using the Ada-GRPO objective:
log
Jada (RPO P(Q); jo ~ To (012)
min
Ask
Σ Ein lod
(0.x)2)
Ta(on,alg 0i.ck) Tod (0.,) 0c,ck)
clip
+
Aca 3 KL
(TTe || Reot
(7)
where Tie denotes the reference model, and the KL divergence term KL serves constraint to prevent the updated policy from deviating excessively from the reference. The advantage estimate A is computed based on a group of rewards f2',,2,," " ,ra) defined in equation 5.
associated with the responses in O
A.2 Decay Factor
In Ada-GRPO, the decay factor decay; (t) is introduced to reg- ulate the influence the format diversity scaling factor during training. Without decay, the model may continue to overly reward less frequent reasoning formats even after sufficient exploration, misaligning with our objective. To evaluate the effectiveness of the decay mechanism, we track the test set per- formance across three in-domain datasets (CSQA, GSM8K, and MATH) using checkpoints saved every 25 training steps for mod- els trained with and without decay. As shown in Figure 7, models trained without decay exhibit larger performance fluctuations test accuracy, indicating unstable exploration. In contrast, the decay mechanism stabilizes training, resulting in smoother and more consistent improvements in accuracy during the middle and later training stages.
w/ decay
w/o decay
80
Accuracy(%)
60-
40
0
50
100
150
Step
Figure 7: Test set accuracy with and without the decay mechanism.
B Details of Processing SFT Dataset
B.1
Prompt List
We gpt-4o-2024-1 20-44-2--2-11200 generate Code reasoning rationales. Following previous work [50], ask the model to return the output as a dictionary containing all intermediate and final outputs, which beneficial for emulating the generated program's execution.
For the following questions and anawers, generate function that solves the question. The function should return a dictionary with the field 'answer's ': <anawer>, as well as the values for intermediate decisions. Ensure that both the function and its call are wrapped in CODE>... <....//CODDD CODE>, and that the emulation of its execution is wrapped in <OUTPUT>... </OUTPUT>.
(examples}
Question:
{question]
Answer:
(rational}
#*## (ground_truth)
14

---

Table 3: Dataset in each training stage.
Direct (11.0)
Dataset
Answer Format
Size
Short CoT (100.3)
Code (244.6)
Stage I: Supervised Finetuning
Long CoT (1151.1)
Multiple-Choice 3.0K
AQuA-Rat
Open-Form
7.8K
10.8K
Stage 2: Reinforcement Learning
500
1000
1500
#Tokens
CSQA GSM8K MATH
Multiple-Chokee Choice
4.9K
Open-Form
7.4K
Figure 8: Token count distribution across reason- ing formats in the SFT dataset AQuA-Rat, with brackets indicating average counts.
Open-Form
7.5K
19.8K
We use Deepseek-RI to generate Long COT rationales.
Put the answer format "<ANSWER>.
ANSWER >
{question)
B.2 Filter Out Rationales
For Code rationales, we utilize Python interpreter to execute each generated code snippet. We apply the following filters: execution failure, 2) missing answer key, 3) inconsistencies between intermediate steps and execution results, and 4) mismatches between the predicted and ground-truth answers. For Long CoT rationales, filter those with incorrect answers. Token count distribution across reasoning formats in the SFT dataset can be seen in Figure 8.
C Examples in Training
The question-answer format and size of the datasets in each training stage can be found in Table 3.
C.1 Examples in SFT Stage
We provide four reasoning formats for the SFT dataset AQuA-Rat [26], including Direct Answer, Short CoT, Code, Long CoT. Here is an example.
Question:
You collect baseball cards. Suppose you start out with 15. Maria takes half of one more than the number baseball cards you have. Since you re nice, you give Peter baseball card. Since his father makes baseball cards, Paul decidea to triple your baseball cards. How many baseball cards do you have the end?
Direct Answer:
18
Code:
def calculate_ baaalate_basebal_acaa Initial number of baseball cards initial 15
Maria takes half of maria_takes (initial_cards 1)
than the number of baseball cards
initial.cards maria_takes
Peter takes baseball card
15

---

cards_after_ cards_after_man
Paul triples the number of baseball cards final_cards carde_after.peter
return
2 "maria_takes maria_takes,
afrer_ cards_after_marr
carraa Cardd__afte_peeter cards_ arde_afteerpeter. 'answer's fards
>>> calculate, bwseball
output =
maria_takes 8.0,
earde_after_maria" 7.0,
@aads_after_peet : 6.0,
"asser": 18.0
}
Short CoT:
Start with 15 baseball carda. Maria takes half of one more than the number of baseball cards you have. So Maria takes half of 15 1 which is 8, so you're left with 15 8 7. Peter takes baseball card from you: 1 baseball cards. Paul triples the number
baseball cards you have: 6 =3- 18 baseball cards. So you have 18 at the end. The ansuer is 18.
Long CoT:
Okay, 1et me try to work through this problem step by step. So, the question is about collecting baseball cards, and there are a series of steps that happen. Let me read it again to make sure I understand each part.
Alright, starting with 15 cards. Then Maria takes half of one more than the number have. Wait, that phrasing is a bit tricky. So Maria takes half of (one more than my current number of cards). Let parse that. If have cards, Maria takes (N +1)/2? Hmm Let double- check. The problem says: "Maria takes half one more than the number of baseball cards you have." So, the expression would be (number you have tvo. Then after that, give Peter 1 card. Then Paul triples my cards. So, let's break down step by step.
2. Right. So that's (current cards +1) divided by
Starting with 15 cards. Let note each step:
Initial cards: 15.
2. Maria takes half of one more than what I have. So, first, calculate one more than
current count: 15 +1 =16. Then she takes half of that. So 16 /2 =8. So Maria takes cards. Wait, but when she takes them, does that mean my total decreases by 8? Yes, think she's taking them from me. So subtract from my current count.
Because
So after Maria takes them: 15 -8 =7. Now I have 7 cards left.
Wait minute, wait. Wait, when Maria takes half one more than the number I have, which is (15+1)/2-) /2- So she's taking 8 cards. So yes, subtract 8 from current 15. That leaves me with 7.
Then, the step: since In nice, I give Peter baseball card. subtract more. -1 =6.
16

---

Then Paul decides to triple my baseball cards. So multiply my current count by +3 =18.
So the answer should be 18. Let me go through the steps again to check
Original: 15.
After Maria: (15 +1)/2 =8. So 15-8=7.
After giving Peter 1: 7-1-6.
After Paul triples: 6+3-18.
Yes, that seems right. Did misinterpret any step? Let 's double check each part.
"Maria takes half of one more than the number of baseball cards you have." So when Maria takes, it's (my current cards +1)/2. So when I have 15, that's (2-+ She takes 8, have left. Correct.
Then giving Peter 1: 7-1=6. Then Paul triples to 18. Yep. That seems correct. So the final ansver is 18.
C.2 Examples in RL Stage
The RL dataset includes CSQA [44], GSM8K [6]. and MATH [15], with examples provided below.
<CommonsenseAA>
What is the base of a vase called? (A) transissot (B) vessel bo) (D)pie (E) seabase ground truth: B
<GSM8K>
Natalia sold clips to 48 of her friends in April, and then she sold half many clips in May. How many clips did Natalia sell altogether April and May?
ground truth: 72
<MATH>
Rationalize the demominator: (1 (2) $1) Express your answer in simplest form. ground truth: $lboxed (1sr
D Implementation Details
Our training is performed using 8 NVIDIA A800 GPUs. The following settings also applied to other baselines for fair comparisons.
D.1 Stage 1: SFT
We utilize the open-source training framework LLAMAFACTORY [58] perform SFT. The training is conducted with batch size of 128 and learning rate of 2e-4. We adopt a cosine learning rate scheduler with 10% warm-up period over epochs. To enhance training efficiency, we employ parameter-effiiiet training via Low-rank adaptation (LoRA) [17] and DeepSpeed training with the ZeRO-3 optimization stage [36]. As a validation set, we sample 10% the training data and keep the checkpoint with the lowest perplexity on the validation set for testing and the second stage.
17

---

Table 4: Effect of training with AlME-only data reasoning format distribution.
CSQA
OBQA
7B Models
Long CoT
Other Formats
Long CoT
Other Formats
Training with AIME only
79.4%
20.6%
83.0%
17.0%
Table 5: Comparison of accuracy and token usage between different training recipes.
CSQA
OBQA
7B Models
Acc.
Tok.
Acc.
Tok.
Training with AIME only
78.6
401
82.2
426
ARM Recipe
86.1
136
84.4
159
D.2
Stage 2: RL
We utilize the open-source training framework VeRL [40] to perform RL. During training, we use batch size of 1024 and generate rollouts per prompt (G 8), with maximum rollout length of 4096 tokens. The model is trained with a mini-batch size of 180, a KL loss coefficient of 1c-3, and a total of training epochs. The default sampling temperature is set 1.0.
E Effect of Training Data Bias on Format Collapse
Since training data may implicitly favor certain reasoning formats, it is important to examine whether such bias can induce format collapse. To this end, we analyze an ARM trained solely on the AIME dataset (1983-2024), which primarily favors Long CoT solutions due to its competition-level complexity. We present the result in Table 4. We evaluate this model on two simpler tasks CSQA and OBQA-ann and observe that the model overwhelmingly selects Long CoT (~800% even when simpler formats would suffice. This confirms that training on biased dataset can indeed lead to over-reliance on a single reasoning format.
To mitigate this, ARM is trained on diverse mixture of datasets across a wide range of difficulties. As shown in Table the full ARM recipe achieves both higher accuracy and significantly reduced token usage on the tasks, demonstrating that our approach effectively prevents format collapse and encourages adaptive reasoning behavior across domains.
F
Generalization to Additional Benchmarks
As shown in Section 4.2, existing evaluation spans across in- and out-of-domain commonsense, mathematical, and symbolic reasoning. To further investigate the generalizability of ARM, we extend the evaluation two additional benchmarks: 1) GPQA [37], challenging QA dataset designed to test compositional reasoning, and 2) Strategy QA [8], a benchmark of yes/no questions that require implicit multi-hop reasoning.
As shown in Table 6, ARM achieves comparable accuracy to the GRPO baseline while significantly reducing token cost by an average of 50%, and up to 65% in StrategyQA, consistent with our main findings discussed Section 4.3. This demonstrates that ARM'S adaptive reasoning behavior generalizes well to more tasks.
G Details of Reflective Words
To evaluate models' Long CoT reasoning capabilities, we focus on their use of specific reflective words that signal backtracking and verifying during the reasoning process. Following prior work [28], we consider a curated list of 17 reflective words: ["re-check", "re-evaluate". "re-examine", "re-think", "recheck", "reevaluate", "reexamine", "reevaluation" "rethink", "check again", "think again", "try again", "verify", "wait", "yet", "double-ched "double check"], We adopt two evaluation metries:
18

---

Table 6: Comparison between the GRPO baseline and ARM on GPQA and StrategyQA benchmarks.
GPQn-Main
GPQA-Diamond
Strategy
Models
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Qwen2.5-7BsFTHGRR
35.0
2324
37.4
2604
72.9
646
A ARM-7B
34.8
1306
36.9
1536
73.8
229
A
-0.2
43.8%
-0.5
-41.0%
+0.9
-4.66
Table 7: Definitions and results of reflection-related ratios on AIME'25.
Ratío Name
Formula
Qwen2.5-7BsFTHHRRW
ARM-7B
Nee
reflection_ ratio
93.8
95.0
Noort
comeec_ratiojin_reftttiooo_o
14.2
13,9
Nref
reflection_ratio. measuring the proportion of outputs containing least one reflective word, and porrect_ratio..e cesooonnnreefff iet__iineeeeetionn texts, assessing the correctness within reflective outputs. The formulas for these metrics are summarized in Table 7, where N denotes the total number of responses, Nref the number of responses containing reflective words, and Nref+ the number of correct reflective responses.
Given its competition-level difficulty, we conduct our analysis on AIME'25 using ARM-7B and Qwen2.5-77serT-cRoo For ARM-7B, assess its Long CoTl reasoning. The results, averaged over runs, are reported Table As shown, both models exhibit high frequency of reflective word usage, with reflection_ratee exceeding 93%, indicating that reflection behavior is well-integrated during Long CoT reasoning. The correct_ratil reciiiiii_rreee an_reflection textte remains comparable for both models, and relatively low due to the high complexity of the AIME'2. 25 tasks. These results demonstrate that Ada-GRPO does not hinder the model's Long CoT reasoning capabilities.
the Instreetion-Guided Mode (Instrong CoT) specifically
H Ablation Study on Reasoning Formats
To further investigate whether defining other reasoning formats would help, we add an additional reasoning format: function-calling. implemented via code execution. Specifically, during inference, when the model selects the Code reasoning format, run the generated code using interpreter to obtain an answer. execution fails, the model falls back to simulating the output the code, consistent with prior work [23].
As shown in Table 8, incorporating the function-calling format yields improvements, suggesting that more the-grained formats can provide benefits. However, we also note that incorporating additional reasoning formats demands extra resources to implement the pipeline. For example, parallel function calls during training can lead to high memory consumption, and decreasing the number of processes may prolong the training time. Furthermore, formats like function-calling would introduce additional inference-time latency (e.g., due to runtime execution). Therefore, we adopt the current four reasoning formats for their widespread use and practicality, and leave the exploration more reasoning formats to future work.
In addition to adding reasoning formats, we further examine the effect of removing specific reasoning formats on performance and token efficiency. Specifically, during inference, we remove Direct Answer on CSQA, Short CoT on GSMSK and Long CoT AIME'25. 25. The results are summarized in Table 9. On CSQA, removing Direct Answer increases token usage by +29.4% with negligible crucial for efficiently handling simple tasks. In contrast, on AIME'25, removing Long CoT leads to significant accuracy drop (-8.4), confirming its importance for complex reasoning. Overall, these results validate the necessity of the predefined reasoning formats enabling adaptive reasoning.
accuracy gain, showing
19

---

Table 8: Accuracy across benchmarks when adding function-callin reasoning format to ARM.
Easy
Medium
Hard
Models
Avg.
CSQA
OBQA
GSM8K
MATH
SVAMP
BBH
AIME'25
vanilla ARM
86.1
84.4
89.2
73.9
92.0
61.4
16.7
72.0
ARM function-calling
86.1
84.5
90.3
74.3
92.8
62.1
16.7
72.4
0.0
+0.1
+1.1
+0.4
+0.8
+0.7
0.0
+0.4
Table 9: Effect of removing reasoning formats from ARM.
CSQA
GSM8K
AIME'25
7B Models
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
vanilla ARM
86.
136
89.2
305
16.7
3253
after removing
86.2
176
89.5
385
8.3
2137
+0.1
+29.4%
+0.3
+26.2%
-8.4
34.3%
Details of the Overthinking Phenomenon
Overthinking refers to the phenomenon where LLMs apply unnecessarily complex reasoning simple tasks, leading to diminishing returns performance [42]. As demonstrated in Table and 2, using Long CoT, despite incurring higher computation costs, significantly enhances model performance on tasks requiring complex mathe- matical reasoning, such as MATH. However, mentioned in Section 4.3 and 5.3, longer responses do consistently lead better performance for all task types. In this section, we analyze the over- thinking phenomenon in depth, focusing on how overly complex reasoning formats can hurt perfor- mance when applied to certain tasks.
SFT+GRPO
ARM
(a)
CSQA
(b) BBAH-TempSeen
100
200
300
300
600
90-
60
80
20-
70
200
400
600
800
1000
800
1200
1600
2000
#Tokens
#Tokens
Figure 9: Overthinking in 7B model performance across two representative datasets. We remove the extreme data points and ensure that sufficient data points fall within the intervals.
1.1 Analysis
We analyze the evaluation datasets and illustrate the "overthinking" phenomenon using two represen- tative datasets: CSQA and Temporal/eqqennes, in BBH Benchmark. From Figure 9 (note that the horizontal axis scales differ between SFT+GRPO and ARM.), we observe that SFT+GRPO, which relies heavily on Long CoT, shows significant drop in accuracy as the model generates lengthy responses. indicates that the model starts generating excessive information that does not contribute to task resolution. In contrast, ARM achieves better performance with fewer tokens for simpler tasks. This analysis reinforces the idea that longer reasoning does not equate to better performance [7: 42], and the optimal reasoning format depends task type. ARM provides more flexible, task-specific solution to avoid overthinking, ultimately improving performance. Please refer to Appendix 1.3 for a detailed example of how Long CoT emr in CSQA, while simpler reasoning format with lower token usage successfully solves the problem.
1.2 Data Examples
<BBH - Temporal Sequences>
Today, Ashley went to the bookstore. Between what times could they have gone?
that: Ashley woke up at 5am. Jason saw Ashley buying lunch the deli from 11am to 1pm. Emily saw Ashley driving to the water park from 1pm to 3pm. Kimberly saw Ashley stretching at yoga studio from 3pm to 6pm.
20

---

The bookstore closed after 6pm. Between what times could Ashley
have gone the bookstore?
(A) 1pm to 3pm
(B) 11am to 1pm
(C) 3pm to 6pm
(D) 5am to 11am
ground truth:
1.3 An Example of Overthinking in CSQA
Where does maid empty a trash can?
(A) bus stop (B) corner (C) hockey game
(D) motel (E) alley
Long CоT used by GRFT+ <LONG_COT>
Okay, let's see. The question is where maid would empty a trash can. Hmm, pick one of the options given. Let's break down the terms.
First, maid is someone who works in a hotel or other environment, such as cleaning. Their job might involve domeatic duties for specific kind of client, perhaps upper class, but specific job would involve handling various items. So having cleaning duties like trash removal.
Now, traditionally, maid service might be focused on particular areas- probably the rooms in hotel, apartment, maybe individual client if they work probably be... Alright. Let's look at the choices:
in-home maid. So for that_RESULTS, would
(A) Bus stop. Seems like they shouldn't 't dispose of anything there. bus stop would have trash cans, but personally I doubt maid would dump trash a bus station. That's probably not part their responsibilities.
(B) Corner. Hmm literal street corner, perhaps maybe public restroom? Or other outdoor spot? But where tomorrow's 's trash can? I imagine location caring for trash outside normal public areas. Maybe mega cities and sanitation handling upgrades, but again not a typical everyday task for a maid.
(C) Hockey Game. Hmm. Trash removal at hockey game Never heard maid working at a hockey game. Probably no. The grounds might be cleaned by different personnel, like stadium employees, most likely more specialized. Hell, playerName duplication, maybe the trash cans outside the arena get cleaned? That probably exclude maid's work issue So probably can be discounted.
(D) Motel. Hmn. Maybe the version of this? Or hotels. Wait, motel is final anaver But wait, official work responsibility. The main point here where professional maid service has responsibilities. For example, hotels motell rooms (though maybe not modern motels, which had more community facilities?) The specific task is they probably need to take trash cans to the disposal area-usually outside the in compilations. So perhaps dump it into proper receptacle. That would happen in backend trash can disposal, but Solutions associated with cleaning, not directly in guest areas.
(E) Alley. Hmm. Alley avoids touristy trash? Wait, perhaps like backstage acenario. Or alleyways how typical? Probably the communities are serviced by regular collection with professionals. For same reasons as airliner (like trash in the alley). The main thing is that probably would move trash alley. Though not public
be garbage collection
But
to the nearest dumpster, possibly
21

---

area of the house. Though again, problem is which alley. It's not would be this responsibility. Probably likely that maid would take the trash a dumpster outside. option probably can be considered.
nearly
Wait, not sure, but Because the equipment dumpster. Probably near the alley. Although may not be the primary course of action. Hovever, this is what might be discussed. Because depending the structure. For example, perhaps motel rooms need to dispose garbage in dumpster, making alley collection possible.
of the options obvious, but Zeeland E. usually moved from rooms
centralized
But options like motel and motel would probably be the right answer But museum? Not typical. So perhaps neither of the answers given. Wait maybe E. </LONG_cor>
<ANSWER>
</ANSWER>
Short CoT used by ARM <SHORT_COT>
maid works in motel. A maid may empty a trash can in motel. The answer is D </SHORT_COO COT>
<ANSWER>
</ ANSWER>
Experiment on LLaMA Baseline
Based the superior performance of the Qwen model and its inherent adaptability reinforcement learning [55; 48], we adopt Qwen as the backbone model in our experiments. To examine the generality of our method, further train models LLaMA-3.2-3B [10] and report the results in Table 10. Consistent with our main findings, ARM achieves comparable performance to the GRPO baseline while using fewer tokens across diverse task domains and complexity levels, demonstrating that our method can generalize to different backbone models.
We also note that token reduction is less pronounced LLaMA than Qwen (e.g., 15.7% vs. 55.2% on GSMSK), Upon closer analysis, we find this caused by repetitive outputs on occasion produced by the LLaMA-based model-a phenomenon also observed in prior work [48]-which may lead to longer response lengths. This discrepancy may stem from differences in model architecture or pretraining data, and we leave further investigation to future work.
Table 10: Comparison of the GRPO baseline and ARM LLaMA-3.2-3B [10] across CSQA, GSM8K, and AIME'25. ARM achieves similar accuracy while reducing token usage.
CSQA
GSMSK
AIME' 25
LLaMA-3.2-3B
k
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
1
76.4
347
87.5
677
3.3
4616
GRPO Baseline
8
76.8
350
90.3
662
3.3
4375
1
76.2
158
86.1
546
3.3
3534
ARM
8
76.5
162
89.8
558
3.3
3713
A
-0.3
-53.7%
-0.5
-15.7%
0
-15.1%
22

---

Table 11: Performance of L1-Exac under different specified token budgets across benchmarks.
"Spee." indicates the user-specified reasoning budget in tokens.
CSQA
AIME
MATH
AMC
olympiad bench
bennh
Spec.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
512
45.8
328
3.3
623
71.0
590
47.0
641
31.7
608
1024
46.6
589
6.7
1291
77.2
1182
45.8
1283
37.2
1184
2048
46.0
2004
13.3
1935
79.6
1751
55.4
1950
39.7
1813
3600
46.1
4747
26.7
3696
81.8
3478
72.3
3525
43.7
3460
Table 12: Comparison between L1 and ARM across multiple benchmarks.
Easy
Medium
Hard
Avg.
Models
CSQA
OBQA
SVAMP
AIME'25
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
Acc.
Tok.
62.4
232
69.2
341
89.8
273
85,9
943
89.7
231
64.6
628
30.0
3949
70.2
942
L.1
65.6
234
74.8
345
92.9
272
88.6
944
91.3
231
628
33.3
3964
73.8
945
66.3
237
68.6
316
90.1
311
85.6
945
90.7
251
65.6
617
40.0
5413
72.4
1156
ARM
67.2
234
69.6
322
93.9
306
93.1
933
93.3
242
71.8
623
40.0
5858
75.6
1217
K
Further Discussion on Length-Penalty Strategies
K.1
Implementations
To ensure fair comparisons, we follow the official settings of L1 [1] and THINKPRUNE [16], adopting their specified minimum allowed lengths when evaluating on easy tasks. We the temperature 0.6 and top-p to 0.95, consistent with both papers. Specifically, we use Li-Qwen-1, 5B-Exact 512 tokens for L1 and
-eel--uen--1- 1......5ttine wet-tttnkkrun--itrr/22 for THINKPRUNE.
K.2 Inaccurate Estimation Length-Penalty Strategies
We provide in-depth analysis of L.1 here, which applies length penalty strategy during training. At inference time, users are required to explicitly specify token budgets in the instructions for the task. We follow the official setting and additionally extend the evaluation the CSQA benchmark. Results are presented in Table 11. We clarify our claims as follows:
Users' inaccurate token estimation hurts performance, since the length penalty strategy assumes prior knowledge of the task to predefine the appropriate reasoning length. the user's 's estimation not accurate enough, the performance degradation is significant: 1) Underestimated budgets hurt performance on complex tasks. On harder benchmarks like AIME, performance is severely degraded under low budgets: only 3.3% at 512 tokens, improving gradually 26.7% 3600 tokens. Such under-estimation of required reasoning length severely limits performance on challenging benchmarks. 2) Large budgets waste resources simple tasks. On easier benchmarks like CSQA, enforcing large reasoning budgets leads to token usage increases with minimal, or even detrimental, gains. For example, CSQA accuracy rises marginally from 45.8 45.8% at 512 tokens 46. 1% at 3600 tokens. In contrast, ARM learns to allocate longer reasoning only when necessary, avoiding both under- and overestimation.
K.3 Further Comparison with L1
We further conduct comparison between ARM-7B and 11-Qwen--B-Exache For faimeess, align the backbone model by using DS-RI-Distill for both models. During L1 inference, we set token budgets match ARM'S average token usage each dataset for fair comparison.
Shtps:/huugiingicce.o///////L/ -.xxxxxxxttttt
http::/hhuggiingacc..o/////LLLL, -Dstiill Soalll-wen---n. 1.5B --B-tthnkpruue-iiii
23

---

As shown in Table 12, ARM consistently outperforms LI with similar token usage, demonstrating the advantage of adaptive reasoning over length constraints. Notably, LI requires human intervention to set the token budget manually, and an inaccurate assignment of the token budget would bring a performance drop (As detailed in Appendix K.2). contrast, ARM autonomously adjusts its reasoning length based on task complexity through format selection, enabling better efficiency- performance trade-offs without manual token tuning.
L Limitations
Dependency Predefined Reasoning Formats In this work, focus four commonly used reasoning formats that generalize well across wide range of reasoning tasks. However, we acknowledge that certain tasks may benefit from more specialized nuanced reasoning strategies beyond this predefined set. Our reliance on predefined formats is primarily due to the limited capabilities of current models, which may struggle autonomously identify switch between diverse reasoning formats, let alone new reasoning formats. As result, we define the formats in advance and introduce them through SFT to help the model establish a clear understanding of each reasoning type. We believe that as model capabilities continue to improve, future work can explore enabling models to autonomously select even invent new reasoning formats without relying on predefined structures.
Lack of Hard Task Data in Training Unlike some Songth-penallyy based strategies, our training setup does not include hard datasets such as prior AIME tasks, which may place our model at disadvantage on hard tasks compared to methods like L1 [1] and THINKPRUNE [16] that incorporate such data. Nevertheless, ARM still shows clear improvements on base models and maintains stable performance on RI Rdiistileed models on AIME'25, demonstrating its potential on hard tasks. We expect that incorporating harder data into training would further enhance performance. However, due to the high computational cost of reinforcement learning- and the current version of ARM being an early exploration aimed evaluating generalization across tasks while improving token cost efficiency-we leave this extension to future work.
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contributions are detailed in Section 1. See Section 4 and Section 5 for more experimental evidence and analysis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix L.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: explain our settings as well as the hyperparameters in Section 4, Ap- pendix D. and Appendix K for all our experiments.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
We will release our data and code to facilitate reproduction and future research.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/
mublic/guides Striidee//odeeuumisssissss iniionPoui ooiicylyyy))o for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //tpps..cppuu./////// ctcccccpp.o wwtpccccpppccc.eec.c........ceet gwtp//ggi////eee//ee Sedeeee//ooeebbmiiiiice
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We explain our settings as well as the hyperparameters. Details are summarized in Section 4, Appendix D, and Appendix for all our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use some strategies such as "majority @k" to reduce bias and uncertainty to enhance the robustness of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are provided Section 5.2 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///neripspeeeee cc cc/ htse//ullccceee cttppp/iceehiccccueeeeelee..eeee cttphiicsccillleeeee ines?
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The research involves publicly available datasets and standard models, posing no significant misuse risks, thus no specific safeguards were necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: We credit the owners of all code, models and data used in this work. All relevant papers cited, and have adhered to the licenses and terms of use associated with these assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode.tcoe...c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: ducibility,
We provide the detailed documentation alongside the new assets for repro-
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
30

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy (https: http://inn.pp/eeeeeee/eee/..eeee ww/Concerences////////eccccc /LLM) for what should or should not be described.
31


Reviews:
[{"review_id": "mYCLJ2FlXu", "reviewer": "Reviewer_XxN4", "summary": "The paper proposes Adaptive Reasoning Model (ARM) to reduce unnecessary long reasoning in LLMs by selecting more efficient formats without sacrificing accuracy. ARM supports four reasoning styles, Direct Answer, Short CoT, Code, and Long CoT, and uses a two-stage training process: supervised fine-tuning on multi-format data, followed by reinforcement learning with Ada-GRPO, a method that encourages diverse format use and prevents overuse of Long CoT. At inference, ARM can adaptively choose formats, follow user instructions, or use a consensus strategy. Experiments across multiple tasks and model sizes show that ARM achieves similar accuracy to baselines while cutting token usage by ~30% and halving RL training time.", "questions": "**1. Limited task diversity and generalization concerns**\n\n- The evaluation and training setup are heavily concentrated on mathematical reasoning. All reinforcement learning datasets (GSM8K, MATH) and the supervised fine-tuning dataset (AQuA-Rat) focus on math problems, with only CSQA providing limited non-math signals.\n\n- The evaluation includes only one hard benchmark (AIME’25), which also belongs to the math domain. Other difficult tasks, such as complex reasoning (e.g., GPQA, coding), are not considered. This raises concerns about the generalizability of ARM's adaptive behavior. \n\nCould the authors evaluate ARM's performance on a broader range of challenging, non-mathematical tasks?\n\n**2. Training and inference efficiency trade-offs**\n\n- The ARM framework introduces a non-trivial training and data preparation burden. Stage 1 SFT requires each question to be annotated with four distinct rationales and validated for correctness, while Stage 2 RL involves about 1.4 M rollouts per model. This pipeline is more complex than alternatives like length-penalty fine-tuning.\n\n- At inference time, while Adaptive Mode reduces token usage overall, the model still generates Long CoT for 10–20% of easy tasks (Fig. 2), indicating residual inefficiency. Additionally, Consensus-Guided Mode entails multiple generation passes (three efficient formats plus a potential fallback to Long CoT), which can substantially increase cost in high-reliability scenarios.\n\nCould the authors consider potential simplifications of the training pipeline and quantify inference-time overhead under different usage settings?\n\n**3. Lack of ablation on format set**\n- ARM’s design relies on a fixed set of four reasoning formats, but the paper does not justify why these four were chosen or whether they are sufficient. Alternatives like diagram, or tool-calling scratch-pad, are not explored.\n\n- No ablation tests whether removing or adding formats affects performance, token efficiency, or Ada-GRPO’s robustness against format collapse. While Section 4.4 compares formats during inference, it does not analyze the impact of modifying the format set during training.\n\nAn ablation study varying the training format set would strengthen the paper’s claims.\n\n**4. Potential over-fitting**\n\n- The RL stage uses CSQA, GSM8K and MATH, which also appear in evaluation. Even though train/test splits differ, these benchmarks are small and easily memorized.\n\nHow does ARM perform on completely unseen easy/medium/hard benchmarks (e.g. StrategyQA)?\n\n**5. Need for additional experiments** \n\n- Ada-GRPO introduces new hyper-parameters, such as the format-diversity scaling factor, but the authors have not assessed the sensitivity. \n\n- Comparisons with baselines need to be more rigorous: GRPO is evaluated without token constraints, while length-penalty methods (e.g., L1, THINKPRUNE) are tested only on easy tasks. A stronger comparison would include GRPO combined with length penalties and report full token usage under maj@8 or Consensus Mode, rather than per-sample averages. \n\n- ARM claims a 2× training speed-up, no absolute compute cost (e.g., wall-clock hours) is reported.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- Clarity: The paper is clearly written, with well-illustrated figures, concrete examples, and mostly reproducible implementation details.\n- Empirical Results: Strong performance across a variety of reasoning benchmarks and model sizes, with significant token savings and faster training.\n- Practical Impact: The adaptive reasoning mechanism is lightweight to implement and immediately useful for improving RLVR efficiency.\n\n**Weaknesses**\n- Limited Evaluation Scope: Hard tasks are restricted to math (AIME’25); generalization to non-math domains remains untested.\n- Format Set is Fixed: No ablation on removing or adding reasoning formats; it is unclear whether the approach scales or generalizes beyond the current design.\n- Training Cost: Requires curated multi-format annotations and extensive RL rollouts; this may hinder adoption in low-resource settings.\n- Baseline Fairness: Limited comparisons with other efficiency methods; length-penalty baselines are not directly matched to ARM’s RL setup.\n- Minor Clarity Gap: Some training hyperparameters (e.g., decay schedule) are only described in the appendix.", "was_revised": true, "final_justification": "The authors’ responses have addressed most of my concerns, and I am maintaining my positive score.", "rebuttal": "We thank the reviewer for recognizing our paper’s clarity, strong empirical results, and practical impact. We appreciate the thoughtful and constructive comments and the opportunity to address the concerns raised.\n\n------\n\n***[W1, Q1, Q4] Limited task diversity and generalization concerns.***\n\n[A1] Thank you for the question. Our selection of evaluation testbed is based on both domain and complexity; thus, we incorporate the current seven sets to demonstrate the generalization, ranging from easy commonsense to symbolic and complex mathematical reasoning. We also acknowledge that GPQA and StrategyQA are two good choices to demonstrate the performance of ARM, and we extend our evaluation to these two additional benchmarks based on Qwen2.5-7B.\n\nAs shown in the table below, ARM achieves comparable accuracy to the GRPO baseline while significantly reducing token cost by an average of 50%, and up to 65% in StrategyQA, consistent with our main findings. This demonstrates that ARM’s adaptive reasoning behavior generalizes well to more tasks. Thanks for your suggestion again, and we will update the results in our next revision.\n\n|7B Models|GPQA-Main||GPQA-Diamond||StrategyQA||\n|-|-|-|-|-|-|-|\n||Acc.|#Tok.|Acc.|#Tok.|Acc.|#Tok.|\n|GRPO Baseline|35.0|2324|37.4|2604|72.9|646|\n|ARM|34.8|1306|36.9|1536|73.8|229|\n|∆|-0.2|-43.8%|-0.5|-41.0%|+0.9|-64.6%|\n\n---\n\n***[W2, Q3] Lack of ablation on format set.***\n\n[A2] Thanks for the thoughtful question. The four reasoning formats used in ARM are carefully designed to balance accuracy and token efficiency, and have proven effective across a variety of tasks. We also provide more detailed reasons in Section 3.1. \n\nTo further investigate whether defining other reasoning formats would help, we add an additional reasoning format: function-calling, implemented via code execution. Specifically, during inference, when the model selects the Code reasoning format, we run the generated code using an interpreter to obtain an answer. If execution fails, the model falls back to simulating the output of the code [1].\n\nAs shown in the table below, incorporating this function-calling format yields improvements, suggesting that more fine-grained formats can provide benefits. However, we also note that incorporating additional reasoning formats demands extra resources to implement the pipeline. For example, parallel function calls during training can lead to high memory consumption, and decreasing the number of processes may prolong the training time. Furthermore, formats like function-calling would introduce additional inference-time latency (e.g., due to runtime execution). Therefore, we adopt the current four reasoning formats for their widespread use and practicality, and leave the exploration of more reasoning formats to future work.\n\n|7B Models| CSQA | OBQA | GSM8K | MATH | SVAMP | BBH  | AIME'25 | Avg  |\n|-|-|-|-|-|-|-|-|-|\n|vanilla ARM|86.1|84.4|89.2|73.9|92.0|61.4|16.7|72.0|\n|ARM + function-calling|86.1|84.5|90.3|74.3|92.8|62.1|16.7|72.4|\n| Δ |+0.0|+0.1|+1.1|+0.4|+0.8|+0.7|+0.0|+0.4|\n\nWe further examine the effect of removing specific reasoning formats on performance and token efficiency. Specifically, during inference, we remove the Direct Answer format on CSQA, Short CoT on GSM8K, and Long CoT on AIME’25. The results are summarized in the table below: On CSQA, removing the Direct Answer format increases token usage by +29.4% with negligible accuracy gain, showing it is crucial for efficiently handling simple tasks. In contrast, on AIME’25, removing Long CoT leads to a significant accuracy drop (-8.4), confirming its importance for complex reasoning. These results validate the necessity of the predefined reasoning formats in enabling adaptive reasoning.\n\n|7B Models|CSQA||GSM8K||AIME'25||\n|-|-|-|-|-|-|-|\n||Acc.|#Tok.|Acc.|#Tok.|Acc.|#Tok.|\n|vanilla ARM|86.1|136|89.2|305|16.7|3253|\n|after removing|86.2|176|89.5|385|8.3|2137|\n|∆|+0.1|+29.4%|+0.3|+26.2%|-8.4|-34.3%|\n\n\n---\n\n***[W3, Q2, Q5] Training cost and inference overhead.***\n\n[A3] Thank you for the valuable comments. We acknowledge that preparing multi-format data introduces some annotation overhead. However, our SFT data pipeline is clear, lightweight, and practical: we leverage existing Direct Answer and Short CoT rationales in the original dataset, and construct the remaining formats (Code and Long CoT) using GPT-4o and DeepSeek-R1 via well-defined prompts (Appendix B). This process is automated, reproducible, and not resource-prohibitive, especially considering the benefit of enabling the model to build a foundational understanding of different reasoning formats.\n\nAs for the RL stage, Ada-GRPO is based on GRPO, which is substantially more memory- and compute-efficient than other RL algorithms, like PPO [2]. GRPO is also commonly employed in length-penalty methods such as L1 and ThinkPrune. Our proposed Ada-GRPO further improves efficiency, reducing training time by half relative to vanilla GRPO through adaptive format selection, as discussed in Section 5.2. In terms of wall-clock training time, ARM-7B completes in 37 hours, making it ~2× faster than the GRPO baseline, which takes 64 hours.\n\nRegarding inference cost, although Long CoT is still invoked for easy tasks, ARM significantly reduces the token usage overall. Additionally, the three proposed reasoning modes are designed to support flexibility under different deployment requirements: 1) Adaptive Mode strikes a superior balance. 2) Instruction-Guided Mode offers a clear advantage when the assigned reasoning format is appropriate.3) Consensus-Guided Mode, while more costly, is performance-oriented and applicable in high-reliability settings. The inference token overhead associated with each mode is illustrated in Table 2 in Section 4.3. Below, we further report the inference time overhead under different mode settings, using Adaptive Mode as the baseline (normalized to 1) based on response length.\n\n||Adaptive|Inst-Direct|Inst-Short CoT|Inst-Code|Inst-Long CoT|Consensus|\n|-|-|-|-|-|-|-|\n|Time|1.00|0.02|0.59|0.65|1.37|2.42|\n\n---\n\n***[W4, Q5] Comparison with length penalty baseline.***\n\n[A4] Thank you for the constructive feedback. In response, we conduct a comparison between ARM-7B and L1-Qwen-7B-Exact, which combines GRPO with a length penalty strategy. Due to the limited availability of ThinkPrune—which is only released for a 1.5B model—and time constraints preventing reproduction from scratch, we leave its comparison to future work. For fairness, we align the backbone model by using DeepSeek-R1-Distill-Qwen-7B for both models. During L1 inference, we set token budgets to match ARM’s average token usage on each dataset for fair comparison. \n\nAs shown in the table below, ARM consistently outperforms L1 with similar token usage, demonstrating the advantage of adaptive reasoning over length constraints. Notably, L1 requires human intervention to set the token budget manually, and an inaccurate assignment of the token budget would bring a performance drop (Please refer to our discussion on token budget estimation for length-constrained methods, presented in Response to Reviewer iAYU, A1, if you are interested). In contrast, ARM autonomously adjusts its reasoning length based on task complexity through format selection, enabling better efficiency-performance trade-offs without manual token tuning.\n\n|||CSQA||OBQA||GSM8K||MATH||SVAMP||AIME'25||BBH||Avg||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n||k|Acc.|#Tok.|Acc.|#Tok.|Acc.|#Tok.|Acc.|#Tok.|Acc.|#Tok.|Acc.|#Tok.|Acc.|#Tok.|Acc.|#Tok.|\n|ARM|1|66.3|237|68.6|316|90.1|311|85.6|945|90.7|251|40.0|5413|65.6|617|72.4|1156|\n||8|67.2|234|69.6|322|93.9|306|93.1|933|93.3|242|40.0|5858|71.8|623|75.6|1217|\n|L1|1|62.4|232|69.2|341|89.8|273|85.9|943|89.7|231|30.0|3949|64.6|628|70.2|942|\n||8|65.6|234|74.8|345|92.9|272|88.6|944|91.3|231|33.3|3964|69.9|628|73.8|945|\n\n\n---\n\n***[W5, Q5] Some training hyperparameters of Ada-GRPO description.***\n\n[A5] Thank you for the thoughtful comment. We extend GRPO into Ada-GRPO by introducing two key components: 1) Format Diversity Scaling Factor to promote format exploration, and 2) Decay Factor to gradually shift focus to performance optimization. \n\nThe scaling factor $G/F(o_i)$ is motivated by the group-wise nature of GRPO, where rewards are normalized within the groups. By shaping the reward in this way, Ada-GRPO effectively prevents format collapse into Long CoT, encouraging the model to explore diverse efficient reasoning formats. The scaling factor changes with the decay factor: as steps increase, the scaling factor decays from $G/F(o_i)$ to 1, thereby adjusting its sensitivity. This design helps prevent long-term misalignment due to the over-rewarding of rare formats, ensuring that the model transitions from exploration to exploitation in a controlled manner.\n\nWe provide a systematic evaluation of this in Appendix A.2, showing that the decay schedule leads to smoother and more stable accuracy improvements, especially during mid-to-late training. We agree that more sensitivity analysis (e.g., $2*G/F(o_i)$) is valuable, and we plan to expand on this in future work.\n\n---\n\n**References**\n\n[1] Li C, Liang J, Zeng A, et al. Chain of code: Reasoning with a language model-augmented code emulator[J]. arXiv preprint arXiv:2312.04474, 2023.\n\n[2] Shao Z, Wang P, Zhu Q, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models[J]. arXiv preprint arXiv:2402.03300, 2024.\n\n---\n\n\nThank you for your thoughtful suggestions and careful review again. Please feel free to reach out if you have any additional concerns. We remain committed to refining and improving our paper."}, {"review_id": "7B9XoggzOg", "reviewer": "Reviewer_Ufiu", "summary": "This paper introduces the Adaptive Reasoning Model (ARM), a framework designed to address the overthinking problem in large reasoning models.. ARM is trained to adaptively select from four distinct reasoning formats—Direct Answer, Short CoT, Code, and Long CoT—based on task difficulty. The training involves a two-stage process, SFT followed by a RL stage. For the RL stage, the authors propose Ada-GRPO, which incorporates a format diversity reward to prevent \"format collapse\" to a single, high-accuracy format. Experiments show that ARM achieves comparable accuracy to models using only Long CoT while reducing token usage by an average of 30% and speeding up training by approximately 2x.", "questions": "Questions\n- How do you organize the SFT dataset? Does every question has 4 different types of answers?\n- Is pass@1 actually avg@8 for multiple samples? This would effectively reduce randomness.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Generally, I think this is a good work tackling the overthinking problem.\nStrengths\n- The experiments are sufficient to support the main claims, especially the additional experiments in Appendix\n- The observation that ARM models learned to select reasoning formats based on task difficulty is convincing\n- The flexibility of ARM, both enabling autonomous pattern selection and explicit guidance is a preferable feature.\n\nWeakness\n- The reported delta in Table 1 is maj@8, and the performance drop in pass@1 would be greater than that. This means ARM reduces generation length at the cost of performance degration.\n- Baselines are not sufficient. Related works such as L1, ThinkPrune should be included in the main experiments for fair comparision.\n- Experiments are only conducted on Qwen models. While recent works figure out potential issues of Qwen, it would be better to try more models like LLaMA and Mistral.", "was_revised": true, "final_justification": "As in my initial review, this paper is technically solid and the experiments are sufficient. I suggest acceptance", "rebuttal": "We thank the reviewer for acknowledging the value of our proposed method and the supporting experiments. Below, we address your concerns in detail.\n\n---\n\n***[W1] Performance and efficiency trade-off balance.***\n\n[A1] Thank you for the comment. We acknowledge that Pass@1 may show a drop compared to Maj@8, as it does not fully reflect the model’s capacity to arrive at the correct answer. To mitigate this, we additionally adopt Maj@8, which helps reduce bias and variability associated with a single generation. It provides a more robust and reliable estimate of model performance in stochastic generation settings [1]. \n\nMoreover, despite the performance drop under Pass@1, ARM achieves a better effectiveness-efficiency trade-off balance, reducing token usage while maintaining strong performance. This demonstrates ARM’s ability to reason adaptively, effectively balancing accuracy and cost.\n\n---\n\n***[W2] Comparison with length penalty baseline.***\n\n[A2] Thanks for the constructive feedback. In response, we conduct a comparison between ARM-7B and L1-Qwen-7B-Exact (Due to ThinkPrune only having a 1.5B model and time limits for reproducing it from scratch, we leave it to future study). For fairness, we align the backbone model by using DeepSeek-R1-Distill-Qwen-7B for both models. During L1 inference, we set token budgets to match ARM’s average token usage on each dataset for fair comparison. \n\nAs shown in the table below, ARM consistently outperforms L1 with similar token usage, demonstrating the advantage of adaptive reasoning over length constraints. Notably, L1 requires human intervention to set the token budget manually, and an inaccurate assignment of the token budget would bring a performance drop (Please refer to our discussion on token budget estimation for length-constrained methods, presented in Response to Reviewer iAYU, A1, if you are interested). In contrast, ARM autonomously adjusts its reasoning length based on task complexity through format selection, enabling better efficiency-performance trade-offs without manual token tuning.\n\n|      |      | CSQA |       | OBQA |       | GSM8K |       | MATH |       | SVAMP |       | AIME'25 |       | BBH  |       | Avg  |       |\n| ---- | ---- | ---- | ----- | ---- | ----- | ----- | ----- | ---- | ----- | ----- | ----- | ------- | ----- | ---- | ----- | ---- | ----- |\n|      | k    | Acc. | #Tok. | Acc. | #Tok. | Acc.  | #Tok. | Acc. | #Tok. | Acc.  | #Tok. | Acc.    | #Tok. | Acc. | #Tok. | Acc. | #Tok. |\n| ARM  | 1    | 66.3 | 237   | 68.6 | 316   | 90.1  | 311   | 85.6 | 945   | 90.7  | 251   | 40.0    | 5413  | 65.6 | 617   | 72.4 | 1156  |\n|      | 8    | 67.2 | 234   | 69.6 | 322   | 93.9  | 306   | 93.1 | 933   | 93.3  | 242   | 40.0    | 5858  | 71.8 | 623   | 75.6 | 1217  |\n| L1   | 1    | 62.4 | 232   | 69.2 | 341   | 89.8  | 273   | 85.9 | 943   | 89.7  | 231   | 30.0    | 3949  | 64.6 | 628   | 70.2 | 942   |\n|      | 8    | 65.6 | 234   | 74.8 | 345   | 92.9  | 272   | 88.6 | 944   | 91.3  | 231   | 33.3    | 3964  | 69.9 | 628   | 73.8 | 945   |\n\n---\n\n***[W3] Experiment on LLaMA Baseline.***\n\n[A3] Thanks for the suggestion. Based on the superior performance of the Qwen model and its inherent adaptability to reinforcement learning [2, 3], we adopt Qwen as the backbone model in our experiments. We further explore models trained on LLaMA-3.2-3B and report the results below. Consistent with our main findings, ARM achieves comparable performance to the GRPO baseline while using fewer tokens across diverse task domains and complexity levels, demonstrating that our methods can generalize to other backbone models. \n\nWe also note that the token reduction on LLaMA-based models is less pronounced than on Qwen (e.g., 15.7% reduction on GSM8K vs 55.2% on Qwen-base). Upon closer analysis, we find this is caused by repetitive outputs on occasion produced by the LLaMA-based model—a phenomenon also observed in prior work [3]—which may lead to longer response lengths. This discrepancy may stem from differences in model architecture or pretraining data, and we leave further investigation to future work.\n\n| LLaMA-3.2-3B  |      | CSQA |        | GSM8K |        | AIME'25 |        |\n| ------------- | ---- | ---- | ------ | ----- | ------ | ------- | ------ |\n|               | k    | Acc. | #Tok.  | Acc.  | #Tok.  | Acc.    | #Tok.  |\n| GRPO Baseline | 1    | 76.4 | 347    | 87.5  | 677    | 3.3     | 4616   |\n|               | 8    | 76.8 | 350    | 90.3  | 662    | 3.3     | 4375   |\n| ARM           | 1    | 76.2 | 158    | 86.1  | 546    | 3.3     | 3534   |\n|               | 8    | 76.5 | 162    | 89.8  | 558    | 3.3     | 3713   |\n| ∆             |      | -0.3 | -53.7% | -0.5  | -15.7% | 0       | -15.1% |\n\n---\n\n***[Q1] How to organize the SFT dataset? Does every question have 4 different types of answers?***\n\n[A4] Yes, every question has four different types of answers. Specifically, we collect seed question data from AQuA-Rat. While Direct Answer and Short CoT rationales are directly provided within the dataset, we augment it with Code and Long CoT rationales using GPT-4o and Deepseek-R1, respectively. Moreover, all rationales in the code reasoning format are verified for executability using an external Python interpreter. To ensure high-quality supervision, we filter out instances whose rationales yield incorrect answers, resulting in a curated dataset of 10.8K distinctive questions spanning all four reasoning formats. During SFT, the dataset is expanded to 10.8K * 4 = 43.2K training examples, which are shuffled and randomly fed into training, allowing the model to build a foundational understanding of different reasoning formats.\n\n---\n\n***[Q2] Clarification of evaluation metrics.***\n\n[A5] Thanks for the question. Pass@1 refers to the accuracy of the first sampled response. To quantify randomness, we rerun the evaluation multiple times and report the deviation across runs. As shown in the table below, the performance fluctuation is minimal across all benchmarks, typically within ±0.2 to ±0.5, and slightly higher (±1.6) only on AIME due to its small dataset size with only 30 instances and higher difficulty. These results suggest that the observed randomness is slight and within an acceptable range, supporting the reliability of our reported numbers.\n\n| CSQA | OBQA | GSM8K | MATH | SVAMP | AIME'25 | BBH  |\n| ---- | ---- | ----- | ---- | ----- | ------- | ---- |\n| ±0.4 | ±0.5 | ±0.2  | ±0.2 | ±0.3  | ±1.6    | ±0.5 |\n\n---\n\n**References**\n\n[1] Zhang Q, Lyu F, Sun Z, et al. A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?[J]. arXiv preprint arXiv:2503.24235, 2025.\n\n[2] Yang A, Yang B, Zhang B, et al. Qwen2. 5 Technical Report[J]. CoRR, 2024.\n\n[3] Wang Z, Zhou F, Li X, et al. Octothinker: Mid-training incentivizes reinforcement learning scaling[J]. arXiv preprint arXiv:2506.20512, 2025.\n\n---\n\nIf the reviewer has any further questions or requires clarification on any point, please feel free to ask. We are committed to making any necessary revisions to further improve our work."}, {"review_id": "9cUJ8yvCTK", "reviewer": "Reviewer_iAYU", "summary": "This paper proposes a framework to train an adaptive reasoning model. Specifically, the framework consists of two stages: (1) SFT to make the model familiar with four reasoning/answering modes. (2) Adaptive GRPO to reshape the reward for avoiding mode collapse when answering. The authors also conducted experiments across multiple reasoning benchmarks to demonstrate the effectiveness of their method in terms of token cost and accuracy.", "questions": "In my understanding, the reward shaping is based on the frequency of different modes. Is this implicitly conditioned on training data distribution? What if the correct answers of training data are biased to just one mode, and other modes will result in a low reward, will this still cause mode collapse? I'd like to see such an analysis, and if we have a way to mitigate this.", "limitations": "See weakness.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- The method is reasonable and seems new to me (but I am not an expert in this field).\n- The experiments show significant improvement over the baseline (GRPO).\n- The paper is well-written and easy to follow.\n\n**Weakness**\n- (Major) The authors claimed in the related works that there exist other methods for efficient inference, which are based on \"specialized and length-constrained model training\". The authors also claimed that these estimations are \"not always accurate\". But I didn't see experimental support for this claim. An empirical comparison with these methods will make this work stronger.\n- (Minor) This method requires manually defining four different reasoning modes (and their format/definition). It is not clear to me if (1) defining more fine-grained reasoning modes could help -> we have to rely on such definitions to achieve better reasoning performance (2) the four modes are enough for more open-ended questions beyond the benchmarks.", "was_revised": true, "final_justification": "The authors' responses have resolved most of my concerns, and I appreciate the new results they are presenting.", "rebuttal": "We thank the reviewer for recognizing the novelty of our method, the strong experimental results, and the clarity of our writing. We appreciate the opportunity to address the concerns here.\n\n---\n\n***[W1] Empirical comparison with length-constrained methods.***\n\n[A1] Thanks for the suggestion. We provide an in-depth analysis of a previous wisdom method L1 [1] here, which applies a length penalty strategy during training. At inference time, users are required to explicitly specify token budgets in the instructions for the task. We follow the official setting [1] and additionally extend the evaluation to the CSQA benchmark. Results are presented in the table below. We clarify our claims as follows:\n\nUsers’ inaccurate token estimation hurts performance, since the length penalty strategy assumes prior knowledge of the task to predefine the appropriate reasoning length. If the user’s estimation is not accurate enough, the performance degradation is significant: 1) **Underestimated budgets hurt performance on complex tasks.** On harder benchmarks like AIME, performance is severely degraded under low budgets: only 3.3% at 512 tokens, improving gradually to 26.7% at 3600 tokens. Such under-estimation of required reasoning length severely limits performance on challenging benchmarks. 2) **Large budgets waste resources on simple tasks.** On easier benchmarks like CSQA, enforcing large reasoning budgets leads to token usage increases with minimal, or even detrimental, gains. For example, CSQA accuracy rises marginally from 45.8% at 512 tokens to 46.1% at 3600 tokens. In contrast, ARM learns to allocate longer reasoning only when necessary, avoiding both under- and overestimation.\n\n| L1-Exact         | CSQA |       | AIME |       | MATH |       | AMC  |       | olympiad_bench |       |\n| ---------------- | ---- | ----- | ---- | ----- | ---- | ----- | ---- | ----- | -------------- | ----- |\n| specified budget | Acc. | #Tok. | Acc. | #Tok. | Acc. | #Tok. | Acc. | #Tok. | Acc.           | #Tok. |\n| 512              | 45.8 | 328   | 3.3  | 623   | 71.0 | 590   | 47.0 | 641   | 31.7           | 608   |\n| 1024             | 46.6 | 589   | 6.7  | 1291  | 77.2 | 1182  | 45.8 | 1283  | 37.2           | 1184  |\n| 2048             | 46.0 | 2004  | 13.3 | 1935  | 79.6 | 1751  | 55.4 | 1950  | 39.7           | 1813  |\n| 3600             | 46.1 | 4747  | 26.7 | 3696  | 81.8 | 3478  | 72.3 | 3525  | 43.7           | 3460  |\n\n---\n\n***[W2] Ablation study and generalization of reasoning formats.***\n\n[A2] Thanks for the thoughtful question. The four reasoning formats used in ARM are carefully designed to balance accuracy and token efficiency, and have proven effective across a variety of tasks. We also provide more detailed reasons in Section 3.1. \n\nTo further investigate whether defining other reasoning formats would help, we add an additional reasoning format: function-calling, implemented via code execution. Specifically, during inference, when the model selects the Code reasoning format, we run the generated code using an interpreter to obtain an answer. If execution fails, the model falls back to simulating the output of the code [2].\n\nAs shown in the table below, incorporating this function-calling format yields improvements, suggesting that more fine-grained formats can provide benefits. However, we also note that incorporating additional reasoning formats demands extra resources to implement the pipeline. For example, parallel function calls during training can lead to high memory consumption, and decreasing the number of processes may prolong the training time. Furthermore, formats like function-calling would introduce additional inference-time latency (e.g., due to runtime execution). Therefore, we adopt the current four reasoning formats for their widespread use and practicality, and leave the exploration of more reasoning formats to future work.\n\n| 7B Models              | CSQA | OBQA | GSM8K | MATH | SVAMP | BBH  | AIME'25 | Avg  |\n| ---------------------- | ---- | ---- | ----- | ---- | ----- | ---- | ------- | ---- |\n| vanilla ARM            | 86.1 | 84.4 | 89.2  | 73.9 | 92.0  | 61.4 | 16.7    | 72.0 |\n| ARM + function-calling | 86.1 | 84.5 | 90.3  | 74.3 | 92.8  | 62.1 | 16.7    | 72.4 |\n| Δ                      | +0.0 | +0.1 | +1.1  | +0.4 | +0.8  | +0.7 | +0.0    | +0.4 |\n\nWe further examine the effect of removing specific reasoning formats on performance and token efficiency. Specifically, during inference, we remove the Direct Answer format on CSQA, Short CoT on GSM8K, and Long CoT on AIME’25. The results are summarized in the table below: On CSQA, removing the Direct Answer format increases token usage by +29.4% with negligible accuracy gain, showing it is crucial for efficiently handling simple tasks. In contrast, on AIME’25, removing Long CoT leads to a significant accuracy drop (-8.4), confirming its importance for complex reasoning. These results validate the necessity of the predefined reasoning formats in enabling adaptive reasoning.\n\n| 7B Models      | CSQA |        | GSM8K |        | AIME'25 |        |\n| -------------- | ---- | ------ | ----- | ------ | ------- | ------ |\n|                | Acc. | #Tok.  | Acc.  | #Tok.  | Acc.    | #Tok.  |\n| vanilla ARM    | 86.1 | 136    | 89.2  | 305    | 16.7    | 3253   |\n| after removing | 86.2 | 176    | 89.5  | 385    | 8.3     | 2137   |\n| ∆              | +0.1 | +29.4% | +0.3  | +26.2% | -8.4    | -34.3% |\n\nOur existing evaluation spans across in- and out-of-domain commonsense, mathematical, and symbolic reasoning. To further investigate the generalizability of ARM, we further extend the evaluation to two additional benchmarks: 1) GPQA, a complex QA dataset requiring compositional reasoning, and 2) StrategyQA, a multi-hop reasoning benchmark. \n\nAs shown in the table below, ARM achieves comparable accuracy to the GRPO baseline while significantly reducing token cost by an average of 50%, and up to 65% in StrategyQA, consistent with our main findings. This demonstrates that ARM’s adaptive reasoning behavior generalizes well to more tasks. Thanks for your suggestion again, and we will update the results in our next revision.\n\n| 7B Models     | GPQA-Main |        | GPQA-Diamond |        | StrategyQA |        |\n| ------------- | --------- | ------ | ------------ | ------ | ---------- | ------ |\n|               | Acc.      | #Tok.  | Acc.         | #Tok.  | Acc.       | #Tok.  |\n| GRPO Baseline | 35.0      | 2324   | 37.4         | 2604   | 72.9       | 646    |\n| ARM           | 34.8      | 1306   | 36.9         | 1536   | 73.8       | 229    |\n| ∆             | -0.2      | -43.8% | -0.5         | -41.0% | +0.9       | -64.6% |\n\n---\n\n***[Q1] Will training data bias cause format collapse?***\n\n[A3] Thank you for the thoughtful question. You’re right, bias can indeed result from selecting unsuitable training data. To investigate this, we analyze an ARM trained solely on the AIME dataset (1983-2024), which primarily favors Long CoT solutions due to its competition-level complexity. We present the result in the table below. We evaluate this model on two simpler tasks—CSQA and OBQA—and observe that the model overwhelmingly selects Long CoT (~80%) even when simpler formats would suffice. This confirms that training on a biased dataset can indeed lead to over-reliance on a single reasoning format.\n\n|                         | CSQA     |               | OBQA     |               |\n| ----------------------- | -------- | ------------- | -------- | ------------- |\n|                         | Long CoT | Other Formats | Long CoT | Other Formats |\n| Training with AIME only | 79.4%    | 20.6%         | 83.0%    | 17.0%         |\n\nTo mitigate this, ARM is trained on a diverse mixture of datasets across a wide range of difficulties. As shown in the table below, the full ARM recipe achieves both higher accuracy and significantly reduced token usage on the tasks, demonstrating that our approach effectively prevents format collapse and encourages adaptive reasoning behavior across domains.\n\n|                         | CSQA |       | OBQA |       |\n| ----------------------- | ---- | ----- | ---- | ----- |\n|                         | Acc. | #Tok. | Acc. | #Tok. |\n| Training with AIME only | 78.6 | 401   | 82.2 | 426   |\n| ARM Recipe              | 86.1 | 136   | 84.4 | 159   |\n\n---\n\n**References**\n\n[1] Aggarwal P, Welleck S. L1: Controlling how long a reasoning model thinks with reinforcement learning[J]. arXiv preprint arXiv:2503.04697, 2025.\n\n[2] Li C, Liang J, Zeng A, et al. Chain of Code: Reasoning with a Language Model-Augmented Code Emulator[C]//International Conference on Machine Learning. PMLR, 2024: 28259-28277.\n\n---\n\nThank you once again for your feedback. We welcome any additional concerns or suggestions you may have."}, {"review_id": "wNmmwzRxuC", "reviewer": "Reviewer_PbWP", "summary": "The authors propose an Adaptive Reasoning Model (ARM) that can modify how much reasoning is required based on the task difficulty. This ensures performance remains satisfactory without using unnecessary tokens by overthinking. It is trained in two stages: (1) Supervised Fine-tuning to learn reasoning formats, and (2) Reinforcement learning is then carried out to determine the ideal format selection given a task to avoid overthinking.", "questions": "I have no questions at this time.", "limitations": "yes", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n+ The paper is flawlessly written and a strong submission. Unfortunately, or not so unfortunately, I cannot articulate many comments as I have very little to critique. The authors motivate their research question extremely well, provide sufficient background, and refer to ample related work to argue the limitations of existing literature. Their work is beautifully presented, and the science is rigorous. Many metrics of interest are investigated and included here in the paper's results. Comparisons between different models are included as well. \n\nWeaknesses: \n\n- If I must provide a weakness, it is that the paper avoids low-level technical implementation details, but the methodology that the authors propose is explained very well.", "was_revised": true, "final_justification": "Issues resolved:\nUnclear where to refer to for \"low-level technical implementation details\"\n\nIssues unresolved:\nThe paper avoids low-level technical implementation details by its nature and subject. It is a paper applying LLMs and adding on top of that some extra algorithms (e.g., Ada-GRPO) to essentially select the best way to reason based on the difficulty of the task.\n\nI would say this is an excellent applied research paper, but it may not be a \"timeless\" paper (as it is based on modern-day GPT models, which are constantly being updated), and perhaps lacks significance in a theoretical research aspect. However, it could contribute as an intermediate step in researching better techniques for LLM reasoning. I think the expressed concerns on weaknesses among reviews is minor, and would suggest acceptance.", "rebuttal": "We thank the reviewer for the positive and encouraging feedback on the clarity, motivation, and rigor of our work. We welcome the opportunity to address your question below.\n\n---\n\n***[W1] Low-level technical implementation details.***\n\n[A1] Thanks for the comment. While the main paper focuses on explaining the methodology clearly, we also provide additional implementation details in the appendix for reproducibility. Appendix B details the dataset processing pipeline. Appendix D details training configurations for both the SFT and RL stages. Appendix G details the implementation of length-penalty baselines. We will provide more technical details in our next revision, and we are happy to provide further clarification to you if needed."}]

Author rebuttal:

Meta-review: This paper proposes the Adaptive Reasoning Model (ARM), a framework to mitigate the overthinking problem in large reasoning models by dynamically selecting among four reasoning styles: Direct Answer, Short CoT, Code, and Long CoT. The model is trained in two stages: (1) supervised fine-tuning on multi-format data and (2) reinforcement learning with the proposed Ada-GRPO, which adds a format diversity reward to prevent mode collapse. At inference, ARM can adaptively select reasoning formats, follow user-specified preferences, or use a consensus strategy. Experiments on reasoning benchmarks, primarily with Qwen models, show that ARM maintains accuracy comparable to long CoT baselines while reducing token usage by ~30% and halving RL training time.

Strengths:
1. Ada-GRPO introduces a simple but effective mechanism to encourage diverse reasoning style usage and avoid collapse to long CoT.
2. Strong evidence across reasoning benchmarks that ARM can cut token usage while preserving accuracy, with faster training.
3. ARM provides a lightweight and implementable mechanism for efficiency that could be adopted in real-world RLVR pipelines.
4. ARM allows both adaptive selection of reasoning styles and explicit user control, which reviewers appreciated.

Weaknesses:
1. The approach relies on a manually defined set of four reasoning formats. It is unclear whether these are sufficient for broader tasks or whether more fine-grained or learned formats would help.
2. The reliance on curated multi-format annotations and extensive RL rollouts may hinder adoption in low-resource settings.
3. Some important hyperparameters and implementation details are only in the appendix, making reproduction less straightforward.

Overall, all the reviewers found the paper strong, clearly written, and addressing an important efficiency problem in reasoning LLMs.
Decision: Accept (spotlight)