AdaCtrl: Towards Adaptive and Controllable Reasoning via  Difficulty-Aware Budgeting
Authors: Shijue Huang
Hongru WANG
Wanjun Zhong
Zhaochen Su
Jiazhan Feng
Bowen Cao
Yi R. Fung
Venue: Accepted by TMLR ()

Abstract: With the advent of test-time scaling, Large Reasoning Models have achieved remarkable performance. However, the reinforcement learning process used to unlock these capabilities often leads to uncontrolled generation length, resulting in substantial computational overhead and unnecessary "overthinking" on simple tasks. Current methods either uniformly minimize reasoning tokens, thereby neglecting the necessity for more intricate reasoning on complex tasks, or employ precise token-level control, which often hinges on accurate difficulty estimation and suffers from unreliable model interpretation for nuanced instructions. To address these limitations, we introduce AdaCtrl,
a novel framework that can dynamically adjust its reasoning length based on the model’s self-assessed problem difficulty and also allow human-in-the-loop control of the budget to prioritize either efficiency or effectiveness. Specifically, we carefully develop a two-stage training pipeline: 1) Cold-start fine-tuning stage, where we first design explicit difficulty-aware tags (e.g., ``[Easy]'' or ``[Hard]'') to indicate difficulty of problems, and train the model on a curated dataset to align its reasoning behavior with these difficulty levels; and 2) Difficulty-aware reinforcement learning stage, which further refines the model’s adaptive reasoning behavior and calibrates its self-assessment of problem difficulty. In this way, AdaCtrl not only empowers the model to adaptively assess the difficulty of problem and adjust reasoning budget allocation, but also enables the user to explicitly control the desired reasoning mode by injecting the specific difficulty-aware tag.
Empirical results across four benchmarks show that, compared to different types of baselines, AdaCtrl effectively balances performance and computational efficiency, leading to performance improvements while dynamically reducing response lengths by up to 90%.

Published in Transactions on Machine Learning Research (04/2026)
AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficult-Aware Budgeting
Shijue Huang"
shijuc.buang@coonnnccss.tkk
The Hong Kong University of Science and Technology (HKUST)
Hongru Wang"
krwangWed..cm uk
University of Edinburgh
Wanjun Zhong
ujjhonng@conneet.usst
The Hong Kong University of Science and Technology (HKUST)
Zhaochen Su
zsubf@connect.us... ust.kk
The Hong Kong University of Science and Technology (HKUST)
Jiazhan Feng
sengioahan@ppku.com edu.cn
Peking University
Bowen Cao
bwcao edu.hk
The Chinese University Hong Kong
Yi (May) Fung!
yrfung@ust.th hk
MMSense Lab
The Hong Kong University of Science and Technology (HKUST)
Reviewed on OpenReview: https: // openreview. forun? id= 4d24.22002004
Abstract
With the advent of test-time scaling, Large Reasoning Models have achieved remarkable performance. However, the reinforcement learning process used to unlock these capabilities often leads to uncontrolled generation length, resulting in substantial computational over- head and unnecessary "overthinking" on simple tasks. Current methods either uniformly minimize reasoning tokens, thereby neglecting the necessity for more intricate reasoning on complex tasks, or employ precise token-level control, which often hinges on accurate difficulty estimation and suffers from unreliable model interpretation for nuanced instructions. To address these limitations, we introduce AdaCtrl, a novel framework that can dynamically adjust its reasoning length based on the model's self-assessed problem difficulty and also allow human-in-the-llop control of the budget to prioritize either efficiency effectiveness. Specifically, we carefully develop two-stage training pipeline: 1) Cold-start fine-tuning stage, where we first design explicit difficulty-aware tags (c.g., "[Easy]" or "[Hard]") to indicate difficulty of problems, and train the model on a curated dataset to align ils reasoning behav- ior with these difficulty levels; and 2) Difficulty-auis aware reinforcement learning stage, which further refines the model's adaptive reasoning behavior and calibrates its self-assessment of problem difficulty. In this way, AdaCtrl not only empowers the model to adaptively assess the difficulty of problem and adjust reasoning budget allocation, but also enables the to explicitly control the desired reasoning mode by injecting the specific difficulty-wware tag. Empirical results across four benchmarks show that, compared to different types of
"Equal contribution.
i "ooreeppondig author.

---

Published in Transactions on Machine Learning Research (04/2026)
baselines, AdaCtrl effectively balances performance and computational efficiency, leading to
performance improvements while dynamically reducing response lengths by up to 90%."
AdaChi Tronierr
Qwn225 Instrrct
RI-S7T-LL
+
CoM--Snn-Sr
25
Hard
Adaptive
20
Accuracy (%)
R1-SFI-RL
15
RI-SFT
Easy
Stad-Start-SFT
S1.1
SnwstuetInstrrut
5
0
5000
10000
15000
20000
25000
Average Reasoning Length (Tokens)
Figure 1: Accuracy-Lentth Pareto Frontier on AIME 2025. AdaCtrl constructs a dominant frontier (blue line) that sits to the upper-left of all baselines, indicating higher accuracy for given token budget.
1 Introduction
With the emergence of test-time scaling (Snell et al., 2024; Muennighoff et al., 2025; Huang et al., 2025; Balachandran et al., 2025; Zhang et al., 2025; Jiang et al., 2026), large reasoning models such as Deepseek RI (DeepSeek-AT et al., 2025) and OpenAl ol (OpenAI, 2024) have demonstrated superior performance across variety of tasks by thoroughly exploring various reasoning paths prior to generating final answers. However, this capability often comes cost: the Reinforcement Learning (RL) optimization used to push reasoning boundaries inevitably leads to uncontrolled growth of generation length. Models tend to discover that generating longer and more redundant reasoning paths can speculatively maximize accuracy rewards (DeepSeek-AL et al., 2025), even when such depth is unnecessary. Consequently, they inevitably introduce significant inference overhead and lead to unnccessary overthinking on simpler problems (Chen et al., 2025; Sui et al., 2025). For instance, even when presented with easy and straightforward questions like "Evaluate (64))" these models still tend engage in lengthy chain-of-thought (Wei al., 2022), unnecessarily employing advanced meta-reasoning skills such as planning, reflection, and verification (Ryan et al., 2016; Li et al., 2025; Wang et al., 2025a). Such reasoning behavior, while beneficial for complex queries, incurs excessive latency and computational costs, negatively affecting user experience (Wang et al., 2025c).
Recent efforts have explored several ways to improve the reasoning efficiency and mitigate overthinking issue. Some approaches aim to minimize reasoning length across all questions, regardless of their actual complexity (e-g., easy or hard), by enforcing conciseness through fine-tuning (Munkhbat al., 2025; Ma et al., 2025) or reinforcement learning (Arora & Zanette, 2025; Aggarwal & Welleck, 2025). However, this universal compression strategy may sacrifice reasoning quality truly complex tasks that require deeper analysis. On the other hand, some methods attempt to precisely control the token budget for each question, soeking more efficient yet accurate reasoning process (Nayab et al., 2025; Muennighoff et al., 2025; Xu et al., 2025). Nevertheless, such fine-grained control often relies on accurate difficulty assessments and exceptional capabilities of underlying LLMs (e.g., precise instruction -ooltoioooo..ooiong....... which can be inflexible and brittle in practice. It is inherently challenging to determine the optimal token budget beforehand, and models may struggle to reliably interpret or execute fine- ine-grained instructions. These limitations underscore the need for a unified framework that. is both adaptive and easily controllable based task complexity.
To this end, propose AdaCtrl, a novel framework to 1) allow the model to dynamically adjust their reasoning effort based on self-assessed difficulty (i.e,, adaptive reasoning mode); and 2) enable users to specify
'Qur code available https://gitee httpt/ttww.ceee/e/e//n/tttt////e
2

---

Published in Transactions on Machine Learning Research (04/2026)
Problem: Evaluate (642.
Auswer: 6
Difficully-Awer Aware Tags [Fasy]
Usnt-Cantrol
Efractvnness
Efficercy ?
User-Controllable Difficulty-wware Tags
[Hard]
Effachwense f
Efficuncy +
Model-Adaptive [Fasy]
Difficulty-wware Tags [[ard]] ?
Efficincy
Efeccinnnc
Response (252 tukems):
Response (1164 tokens):
Response (386 tokens):
[Eary] To evaluate logx(64)) we
determine the
[[ard]] Okay, I need evaluate (og=(64). Hmm, logarithms can sometimes be tricky, but remember that
[Easy] evaluate log.(64). need deternine power to which the base must be raised to ubtain 64.
power to which the base 2 must he mise to obtain 64.
Let's break this down step by step.
logarithms are about finding exponents. Let me recall... IF have (og,() (a) = c. that means 6 a, So in this case. (og2(64) would meam 2" 64.
break down step step.
1. Idceniity the reltionnhip: We ueed to find x such that 2 64,
Express 64 as a power of We know that 2 64. This can be verified calculating 2 22 422
Express 64 a power 2: know that 2 2
o...
2 = 6 2 32
64
4.2 2" B 2² 2 = 32 2 64 3. Determine the
Wait, rightt 2 - 64, So that means 6. Therefore,
2. Use the property of logarilhos Since 2 64, we
exposeet: From the above calculations, we see
(64g
take the loguritim hase both sides: log,(2")
64
But let me double-chock to make sure
I didn't skip step.
(ngg(64)
Conclusion: Since
follows (64)
Mayhe should write step by step.
Simplity lett-hand side using the power nile
6. Therefore, the final answrr is (6).
logarihmm: The power rule states loga(a')
Alternatively, didn't remember powers of off top of my head, maybe I could break it down. Let me see.
(a), Applying this rule, we get: log:(29) 6ingz(2)
Simplify further using the fact that (0g,(6) Since
Another way think about is using legurithm properties.
(2g;(2) 1, we have: 6 (2)) = 6 1 Therefore, (og;(64) 6. The final answer is: (boxed
But maybe that's overcomplicating
Thus, the value (64; (6).
Given the same problem, AdaCtrl supports three reasoning modes: the easy mode offers concise answers with less tokens; the hard mode delivers extensive responses with more tokens; and the adaptive mode dynamically allocates reasoning budgets according to the problem complexity.
Figure 2:
reasoning mode according to difficulty of problem (i.c., controllable budget allocation). As shown in Figure 2, given a qucry, AdaCtrl offers three reasoning modes. Two of these modes are manually specified to allow users to control the reasoning budget: (i) Fasy, which prioritizes efficiency and provides concise responses to any question; (ii) Hard, which aims for higher effectiveness by elaborating the full reasoning process and delivering detailed information. In addition, an adaptive mode automatically adjusts the reasoning cffort based on the complexily of the input query, achieving balance between effectiveness and efficiency without manual intervention. To achieve this, we start by curating dataset that covers both easy and hard subsets and insert corresponding indicator tags (i.e., "[Easy]" or "[Hard]") prior to the model's responses. Then we utilize cold-start fine-tuning to empower the model with foundation capabilities to estimate the complexity of a question and allocate reasoning budgets accordingly, rendering self-aware difficulty estimation. Moreover, we adapt difficulty-awie -aware reinforcement lcarning framework with carcfully designed rewards to calibrate its self-assessment of problem difficulty (i.e., difficulty estimation calibration reward) and to refine the model's adaptive reasoning behavior (i.e., difficulty-wwar length reward).
Experimental results on four benchmarks demonstrate that AdaCtrl significantly improves the tradc-off belween effectiveness and efficiency. It perporrss most. baselines across four datasets while efficiently managing the reasoning budget. Specifically, compared with the standard SFT + RL baselines, AdaCtrl achieves accuracy improvements of up to 10.14%, while roducing response length by as much as 91.04%. Further analysis demonstrates that AdaCtrl offers effective human-in-the-looe controllability via explicit difficulty-wware tags, enables accurate difficulty estimation during reinforcement learning, and maintains robust performance under hyperparameter variation. Overall, the contributions are as follows:
We introduce AdaCtrl, a unified framework for adaptive and controllable reasoning that supports dynamic trade-offs between efficiency and performance, allowing the model to estimate the difficulty of problem and adjust the reasoning mode itself, and also the user to specify the desired reasoning mode to mect diverse needs in practice.
We present two-stage training paradigm that integrates cold-start fine-tuning and diffieulty-aware reinforcement learning together to foster awarenesssssss of problem difficulty and supports difficulty-aware budget allocation, considering the differences and dynamics of model capabilities.
Empirical results on four benchmark datasets demonstrate that AdaCtrl succossfully enhance adaptivity and controllability via explicit difficulty- awaree aware tags. Further analysis reveals that AdaCtrl serves as an effective difficulty estimator, and accurately enables difficulty-aware budget allocation.
3

---

Published in Transactions on Machine Learning Research (04/2026)
2
Related Work
Reasoning Efficiency via Supervised Fine-Tuning. While LLMs achieve impressive performance on complex tasks by generating elaborate multi-step reasoning chains (Dubey et al., 2024; Su et al., 2025), this capability can lead to excessive verbosity and computational overhead for simpler queries. This "overthinking" phenomenon has motivated research into improving reasoning efficiency (Qu et al., 2025a; Wang et al., 2025c; Liu et al., 2025a)). One prominent strategy involves Supervised Fine-Tuning (SFT) to guide models towards more concise reasoning. Some SFT works focus training with inherently shorter reasoning paths. For example models learn adherence to loken budgets through specific prompling during data generation (Han et al., 2024). Others distill concise paths from best-of-N sampling (Munkhbat et al., 2025) or fine-tune models to omit intermodiate steps for samples where the model is already confident (Yu et al., 2024). Another SFT direction compresses existing reasoning chains. Kang et al. (2024) employ GPT-4 (Achiam et al., 2023) as a compressor then fine-tune a model on these long-to-short CoT mappings. L.Mski (Liu et al., 2024) induces step-skipping behavior under step constraints. SPIRIT-FT (Cui et al., 2025) identifies critical reasoning steps using perplexity guide for pruning. TokenSkip (Xia et al., 2025) analyzes token importance within CoT outputs for rollable compression. These SFT methods reduce length but often enforce a general conciseness ill-suited for complex problems and typically lack self-assessment of difficulty or user budget control.
Reasoning Efficiency via Reinforcement Learning. RI. offers another significant avenue for optimizing reasoning efficiency, building upon its success in developing doep reasoning capabilities in models like DeepSeck- Coder (Guo et al., 2025). Many RL approaches incorporate explicit length-based rewards to encourage conciseness alongside accuracy. Some methods link generation length to task difficulty or directives within the prompt: DAST (Shen et al., 2025) adapts CoT length to problem complexity via reward shaping, while LCPO (Aggarwal & Welleck, 2025) controls length using prompt-specified targets. Other techniques normalize length rewards against baselines, as seen in O1-Pruner (Luo et al., 2025), the per-prompt normalization by Arora& & Zanette (2025), and the Kimi 1.5 report (Team, 2025). Yeo et al. (2025) proposed a cosine reward to manage length effoctively, also highlighting the "length hacking" problem where models artificially extend reasoning. Beyond explicit length rewards, alternative RL stralegies include mela-RL for test-Lime optimization (Qu et al., 2025b), utility maximization for budget awareness (Yu et al., 2025b), preference optimization with heuristics (Chen et al., 2025), and mitigating GRPO's bias towards longer trajectories (Liu et al., 2025b). However, these methods generally lack the explicit user control over reasoning depth offered by AdaCtrl's difficulty-aware tags and do not prioritize training for self-awareness of problem difficulty. Our two-stage SFT-RL framework uniquely addresses these aspects, enabling both autonomous and user-influenced reasoning budgets.
3 Method
In this section, we demonstrate the design of our proposed AdaCtrl, which includes: (1) Cold-start fine-tuning that provides initialization for difficulty estimation and difficult-awaare budget adjustment: (2) Difficulty- aware reinforcement learning framework that. boosts model's capabilities on response length control and difficulty estimation. The entire framework is demonstrated in Figure 3.
3.1 Cold-Start Fine-Tume Tuning
This stage primarily focuses on equipping models with the ability to adhere to output formats that include difficultyawware tags (e.g., "[Easy]" and "[Hard]")2 and to control response length accordingly. To curate suitable training data for this purpose, we directly select both easy and hard problems from the DeepMATH dataset (He et al., 2025), which provides difficulty annotations for each problem. For easy problems {g Gy..,, qh 1, we utilize the instruction model, M, to generate concise response, while for hard ones (o) 12 ..., gm }, strong large reasoning model R is employed to generate reasoning trace. Then we filter
We do not consider more fine-grained category order to maintain ease control and ensure reliability.
4

---

Published in Transactions on Machine Learning Research (04/2026)
Large Language Models (e.g. QwenZ (-B--nntuct)
Short Solution
Easy Problems
SFT
[Easy] (Hard
Hard
Large Reasoning Models (ogg, Deepspok-R1)
Long Solution
1q So Pt.
Problems
Stage One: Cold-Start Fine-Tuning
[Easy]
Pi
y1
A
[Hard]
P2
Y/2
70
a
B
A2
...
[Easy]
Pm
Y'm
1
Am
Outcome Reward
Difficulty Esimation
Diflicult--wware Length Reward
Tag
Reasoning Answer
Reward
Advantage
Estimated Difficulty: [Easy]
Optimization
Stage Two: Difer Reinforcement Learning
Figure 3: AdaCtrl comprises a two-stage training pipeline: the cold-start finetuning first utilizes both short and long reasoning trajectories to establish basic budget awareness; then difficult--wware reinforcement learning framework is utilized to calibrate problem difficulty estimation and develop adaptive reasoning strategies.
the solutions based on answer correctness for both subsets:
D {(af, Ip. pl)lll) uil = M(q)), De) =1}
(1)
D = {(ah, [p. Ip. sel = R(a,") I. I(vi, 9:) 1}.
(2)
where [re,9] is the model-generatedd response that includes reasoning process ps and the predicted answer yi, Ús is the ground truth answer of corresponding samples, and I[(] 0.) represents that the model response [Pi, si] is correct with the verification of answer ûi- Then by prepending the "[Fasy]" tag for response in De and "[Hard]" tag for those in Dd, we construct a dataset D = {gg [t,,Pi, Pi, bil)} for cold-start training, where & is the difficult-awware tag. After the cold-start finctuning, the model learns to adhere to the specified format and generate solutions based on its estimated difficulty of each problem, such as allocating more reasoning tokens for hard problems enable deeper and more diverse reasoning paths. Notably, this also provides greater controllability for non-expert users, who can easily guide the model's behavior by using simple tags like "[Easy]" and "[Hard]"
3.2 Difficult-Awarr Reinforcement Learning
After cold-start fine-tuning, we adapt Group Relative Policy Optimization (GRPO) (Shao et al., 2024) as the reinforcement learning (RL) algorithm because it produces multiple rollouts that can be naturally utilized to estimate difficulty of given question from the perspective of the trained model. Specifically, we carefully design three types of rewards to guide the difficulty-wware reinforcement learning optimization:
Outcome Accuracy Reward. The rule-based outcome accuracy reward has been widely utilize in RL training (Guo et al., 2025; Aggarwal & Welleck, 2025; Arora & Zanette, 2025; Yu et al., 2025a; Wang et al., 2025b), which evaluates the correctness of generated response:
1.0
I(yi, 0:) =1
ro(y))
(3)
11.
otherwise
To ensure more reliable verification, we explicitly require the model to present its final answers in specified format (i.e,, within \boxed(()).
5

---

Published in Transactions on Machine Learning Research (04/2026)
Difficulty Estimation Calibration Reward. Accurately estimating problem difficulty is fundamental aspect of diffieulty-aware budgeting. However, since the difficulty of a problem can vary significantly across different models, and obtaining reliable, model-specific difficulty labels is also challenging. To address this, we design a reward function that loverages GRPO's multiple rollouts to estimate a golden difficulty during online training in a natural and effective manner. Specifically, we calibrate the estimated difficulty of the model using the frequency of rollouts that leads a correct answer. We label question as easy the frequency of accurate rollouts exceeds pre-defined threshold 6, or it is labeled as hard. Moreover, during the rollout process, the model is expected to generate correct difficulty entire response to match the its capability, so the reward function is designed based on the matches between the generated difficult--waaee tag 4 and estimated tag label & that is determined by multiple rollouts:
tag at the beginning of the
1.0
((t,,,)= =1
(Bx) =
0.0
I(tt,,;) (i),
(4)
-1.0 I cannot be found ys
Diffiullt-aware Length Reward. Different from previous related works that encourage the model to generate concise responses for all problems (Arora Zanette, 2025; Aggarwal & Welleck, 2025), we hope to encourage such behavior only for easy problems and maintain long thinking capabilities for better tackling hard problems. This helps prevent over-optimization, where the model may fail to engage in deeper thinking when necessary. To prevent unnecessary overthinking, we design difficulty-aware length reward that encourage concise responses when the generated dificuuty-aware tag t, is " "[Eay]]:
(1():) = [ 1.0 1-cos((1) L.)))
t, [Fasy] otherwise
(5)
0.0
where i is the j-th rollout length for problem qi, and L2 is the max length in the rollout group of problem q:- We leverage the monotonicity of the cosine function within a specific domain (i.e., 0 T). This function is easy to tune and provides smooth behavior, as evidenced by a recent study (Yeo et. al., 2025). Therefore, the difficult--waare length reward assigns lower score as the generated response length increases.
Overall Reward and Objective. To cnhance the model's adaptive reasoning capabilities and calibrate its self-evaluation of problem difficulty during online training, the overall reward in the reinforcement learning process is computed by integrating the three specifically designed rewards:
r(u) ro(ss) + a (r((:) + B ri(ss)
where a and B are syper-parameltes During optimization, we sample collection of problems that covers both easy and hard problems from DeepMATH (He et al., 2025) to allow the model to learn dynamic strategies for different types of problems. Following Yu et al. (2025a); He et al. (2025), the policy model Ta is optimizod through the following objective:
lo:1
Te(oi, g, 0,0))))) (0)A Tos g, gi
(o) =
(11)
min
0i
10)1-1,14 e, A.t) 3e DKL DKL (moll ].
To(or, 0),2c) clip
TTaa (oi,e q, oi,
where is the group size, o, is the rollout response, A., is the advantage of the i-th response calculated by normalizing rewards the group, and 8 e are papre-paramcter... Through this reinforcement learning process, models can more effectively assess problem difficulty relative their own capabilities and develop adaptive reasoning strategies accordingly.
6

---

Published in Transactions on Machine Learning Research (04/2026)
4 Experiment
4.1 Experimental Setup
Model and Datasets. We adopt Qwen2.5-7B-Instruee and Owven..5-14B-Instrucc (Qwen et al., 2025) as the backbone model for training. During the cold-start fine-tuning phase, the training problems are drawn from the DeepMATH dataset (He et al., 2025), which assigns each problem a difficulty level ranging from to 9. We categorize problems with difficulty levels of or below as easy, and those above 5 hard. For easy problems, the backbone model is employed to generate concise responses. In contrast, for hard problems, we incorporate extensive long-form reasoning trajectories generated by Deepseek R1 (DeepSeek-Al et al., 2025). After filtering these trajectories using ground-truth answers, we construct a cold-start SFT dataset comprising 8K instances, which includes 4K with short and 4K with long reasoning chains. To support dificulty-awar aware reinforcement learning, we further sample an additional 30K examples from DeepMATH that comprise 10K easy and 20K hard problems and are distinct from those in the cold-start fine-tuning dataset.
Baselines. To assess the effectiveness of AdaCtrl, compare it against several baselines that share the same backbone model. These include: (1) Base Model: the unmodified base instruction-tunet model (i.e. 1 Qwen2.5-14B-B 1-14-Innstruc):: (2) $1.1: reasoning-enhanee
model obtained by further training the base model on the S1-1.1K dataset (Muennighoff et al., 2025); (3) RI-SFT: a baseline model fine-tuned base model on a curated dataset that includes problems sourced from the cold-start SFT dataset along with all responses generated by Decpseck R1; (4) R1-SFT-RL: a reinforcement lcarning model that trained (3) with outcome accuracy rewards on the aforementioned 30K RL dataset; (5) Cold-Starts model fine-tuned based model on the constructed cold-start SFT dataset; (6) Cold-Start-RI: reinforcement learning model based on (5), trained using outcome accuracy rewards the same 30K RL dataset. (7) ARM (Wu et al., 2025), reasoning model capable of adaptively selecting appropriate reasoning formats based on the task at hand.
Training Details. For cold-start fine-tuning, we employ the ms-swift framework (Zhao et al., 2024), using a learning rate of 1e-5 and a batch size of 8. For difficulty-waar reinforcement learning, we adopt the VeRL (Sheng et al., 2025) framework, with all RI. experiments conducted under unified setting. Specifically, we follow DAPO (Yu et al., 2025a) to eliminate KL divergence. The policy model is optimized using the AdamW optimizer with a learning rale of 1e-6, a batch size of 256, and a micro-batch size of 32. And we set the value of a and 3 both as 0.5, and the value of d as 0.625 during training. During the rollout phase, 16 responses are sampled per prompt, and the maximum generation length is set to 24K tokens. All experiments are conducted on NVIDIA H800 GPUs.
Baseline Prompt
Please reason step by step answer the following Math Problem, and put your final answer in the format of (boxed/answer)
Difficult-AAwer Aware Prompt
Answer the following math problem, judge the difficulty (Fasy (Hady/Hard) of given problem and start your response with format: [difficulty hore], and put your final answer in the format \boxed (boxed/answer). (answer).
Evaluation Settings. We evaluate method four mathematical datasets that span both easy and challenging problems: AIME2024 (Art of Problem Solving, n.d.), AIME2025 (Art of Problem Solving, n.d.), MATH500 (Hendrycks et al., 2021), and GSM8K (Cobbe et al., 2021). The first two datascts consist of more challenging Math Olympiad-style problems,, whereas the latter two primarily contain simpler, grade-school level problems, with GSM8K being the easiest among them. Since AIME2024 and AIME2025 each contain only 30 samples, we report the average performance over 8 independent runs for these two datasots. All evaluations are conducted using consistent inference 0.8. In this paper, we use baseline prompt to evaluate all baselines that do not incorporate difficulty-aware
set to a temperature of 0.7 and top-p value of
7

---

Published in Transactions on Machine Learning Research (04/2026)
Table 1: Main results of AdaCtrl's adaptive mode. We compute two metrics: Acc. (%) is average accuracy and Len, the average generated tokens.
Model
AIME2024
AIME2025
MATH500
GSMSK
(%))
Len. 4
Acc. ) f
Len.
Acc.(%) 1
Len. J
Acc.(%) t
Len.
Qwen2. 5-1-55--------
3-IB-Instuct
Swen2.5-7BB-o Instrrr
11.25
1805.60
7.08
1174.06
73.00
628.91
91.58
272.93
S1.1-78
16.67
19022.60
18.33
18302.72
68.80
5824.47
91.89
1838.12
RI-SFT-7B
12.08
20767-36
13.33
20184.36
62.00
8103.41
87.34
3336.81
Cold-Start-SFTT-77
11.25
5553.00
7.92
6237.49
71.00
805.78
90.22
345.44
ARM-7
16.70
3253.00
73.90
889.00
89.20
305.00
R1-SFT-RI-7B
21.25
18778.92
17.50
17924.35
8421.57
88.93
3896.76
Cold-Start-RI-7B
18.33
16911.63
14.58
15941.76
73.00
3797.60
90.67
369.94
AdaCtrl-7B
21.25
16889.50
19.17
15749.08
74.00
3195.69
90.98
349.34
Qwen2. B-I4D-Instruct
1ren2.5-44BInnstrue
11.67
1043.20
10.42
1136.26
73.60
568.28
93.86
215.42
$1.1-14B
32.92
17278.03
29.58
16167.50
76.00
4393.17
95.15
1460.26
RI-SFT-14B
19.58
20485.99
20.42
18447.68
65.80
6473.98
90.67
2883.76
Cold-Start-SFT-11B
12.92
4734.60
12.50
6431.17
76.20
929.48
93.63
287.72
ARM-14B
20.00
3871.00
779.0
903 00
02.50
294.00
RI-SFT-RL-14B
24.17
18549.26
23.33
18294.04
70.60
6397.75
91.05
2814.69
Cold-Start-RL-14BB
27.50
17722.25
24.17
17088.44
75.60
4657.38
94.01
378.39
AdaCtrl-14B 34.58
15173.21
25.83
14476.93
79.20
3209.84
94.09
316.12
reasoning. Furthermore, we design a simple difficult--wware prompt, derived from the baseline prompt, to support our proposed approach, as detailed in Box 1 for baseline prompt and Box 2 for difficulty-wware prompt. To quantify the effectiveness and efficiency of models, we report two metrics: accuracy (Acc.) and the number of tokens generated in the response (Ten.))
4.2
Main Results
The main results presented in Table 1. We can have the following obscrvations:
AdaCtrl Effectively Balances Performance and Reasoning Budget. AdaCtrl demonstrates com- petitive overall performance, oulperforming most baselines across four datasets while efficiently managing the reasoning budget. Specifically, compared to the RT. baseline R1-SFT-RT, based Qwen22.577B-Instrucc AdaCtrl-7B achieves comparable accuracy
AIME2024 and further improves accuracy by 1.67% on AIME2025, 7.20% on MATH500, and 2.05% on GSMSK. At the same time, it substantially reduces response lengths by 10.06%, 12.14%, 62.05%, and 91 31.0% on these datasets, respectively. Under the 14B setting, AdaCtrl-14B increases accuracy by 10.41%, 2.5%, 8.6%, and 3.04% on AIME2024, AIME2025, MATH500, and GSM8K, while compressing the reasoning budget by 18. 18.20% 20.92%, 49.83%, and 78%77% on the correspond- ing datasets. Similar trends are also observed when comparing AdaCtrl to the S1.1-7B and S1.1-14B baselines. Furthermore, AdaCtrl consistently surpasses the utility-based ARM baseline, particularly complex tasks (e.g., AIME2025), by effectively scaling up reasoning length to avoid the capability degradation observed in ARM due to excessive length penalization. Overall, these results suggest that AdaCtrl can adaptively allocate the reasoning budget according to problem difficulty, thereby effectively balancing reasoning efficiency and effoctivencss.
Cold-start Fine-tuning Provides an Effective Foundation for Adaptive Budgeting. Unlike R1- SFT, which is trained solely on long-form reasoning traces from Deepseek R1, our cold-start fine-tuning strategy (i.e., Cold-Start-SFT) incorporates a combination of both concise and extended reasoning trajectories. This diverse training regimen enables the model to acquire more efficient reasoning strategics, resulting substantial reductions response length by 73.26%, 69.10%, 90.06%, and 89.65% for AIME2024, AIME2025, MATH500, and GSMSK, respectively, the 7B model, and by 76.88% 54.14%, 85.64%, and 90.02%

---

Published in Transactions on Machine Learning Research (04/2026)
Table 2: Comparison of adaptive and controlled reasoning modes in AdaCtrl. The easy mode reduces reasoning tokens but at the cost of lower performance, making it suitable for fast-response scenarios. Hard mode uses more tokens and yields better results. The adaplive mode automatically estimates problem
difficulty to balance effectiveness and efficiency.
Model
AIME2024
AIME2025
MATH500
GSM8K
Len. 4
Acc.(%) t
Len. 4
A%c. 1
Len.
Acc.(%) f
Len. J
AdaCtrl-7B (Adaptive)
21.25
16889.50
19.17
15749.08
74.00
3195.69
90.98
349.34
AdaCtrl-7B (Easy)
14.58
1652.42
10.00
896.14
70.80
652.76
90.75
314.49
AdaCtrl-7B (Iladd)
21.67
17562.94
22.08
16114.87
71.20
5960.15
92.57
2058.13
AdsCtrl-14B (Adeptive)
34.58
15173.21
25.83
14476.93
79.20
3209.84
94.09
316.12
AdaCtrl-14B (Easy)
15.83
1394.78
11.25
985.35
75.40
578.37
94.31
284.83
AdaCtrl-14B (Iard)
36.25
15004.55
26.25
141564.21
74.40
5321,25
93.33
2598.28
the 14B model. Furthermore, when models initialized from the cold-start checkpoint undergo additional reinforcement learning (i.e., Sold-Start-RL), they demonstrate superior budget control compared to those initialized from R1-SFT (Le,,RI-SFT-RT), yielding additional reductions in response length of 9.94%, 11.06%, 54.91%, and 91.04% across the same datasets on the 7B model, and 4.46%, 6.61%, 27. 27.20% and 86.56% on the 14B model. These results underscore the critical role of the cold-start fine-tuning phase in establishing robust foundation for effective adaptive budgeting in downstream tasks.
Our Reward Design Enhance Both Reasoning Effectiveness and Efficiency. Building upon the cold-start fine-tuning model, AdaCtrl outperforms Cold-Start-RL, which applies reinforcement learning based solely on outcome accuracy by achieving notable accuracy improvements of 2.92%, 4.59%, 1.00%, and 0.31% on ATME2024, AIME2025, MATH500, and GSMSK, based on with the 7B model, and improvements of 7.08%, 1.66%, 3.6,, and 0.08% with the 14B model with much less token consumption. These accuracy gains are attributable to our reward function design, which incorporates both difficulty estimation calibration and difficuly-awarr length adjustments. By introducing these additional reward signals, our approach enables the model to more accurately assess problem complexity and allocate computational resources accordingly, thereby achieving a more refined balance between reasoning effectiveness and efficiency.
Human-in-the-Loog Controllability of AdaCtrl. AdaCtrl introduces difficulty-wware tags that serve as prerequisite tokens during generation, offering explicit control over response length. To assess the controllability of this mechanism, we emulate user intent by specifying either the "[Fasy]" or "[Hard]" tag, which correspond to simplified and complex reasoning modes, respectively. As presented in Table 2, the experimental results demonstrate that our approach affords effective control over the reasoning budget. Compared to the adaptive reasoning mode, where the model autonomously determines the problem's difficulty, enforcing the "[Easy]" mode consistently leads to reduced performance across all four datasets. However, it also achieves substantial decrease in response length by 90. 90.22% and 94.31% on the more challenging AIME2024 and AIME2025 datasets for the 7B model, and by 90.81% and 93.19% for the 14B model, respectively. Conversely, the " "[Har]]" mode enhances performance on most datasets and markedly increases response length, with gains of 86.51 % and 489.15% on the simpler MATH500 and GSM8K datascts under the 7B sctting, and increases of 55% and 721.93 12.9 under the 14B setting. These findings indicate that AdaCtrl enables precise, human-in-the-loop control over reasoning budgets.
4.3 Analysis
Generalizability across Model Familics To demonstrate the generalizability of AdaCtrl beyond the model. The results, summarized
Qwen architecture, we exlended our evaluation to the in Table 3, confirm that our proposed two-stage training pipeline effectively transfers to different model familics. As shown in Table 3, AdaCtrl consistently maintains its adaptive length control advantages on the Llama architecture. Compared to the R1-SFT-RL baseline, AdaCtrl achieves higher accuracy (c.g., 9.17% vs 7.08% on ATMF2024) while significantly reducing token consumption simpler tasks (e.g., 881.69
9

---

Published in Transactions on Machine Learning Research (04/2026)
Table 3: Performance comparison on -lama-3.1-8B-Istttee AdaCtrl consistently demonstrates adaptive
length control advantages, achieving higher accuracy than basclines while significantly reducing token consumption simpler tasks, proving the method's strong generalizability.
Model
AIME2024
AIME2025
MATH500
GSMSK
Len. 4
Aec.(%) 1
Len. 4
Ace.(%) f
Len.
Acc.(%)
Len.
-lama-3.1-8B-Instree
3.58
10901.07
2.08
10383.25
46.80
4203.35
85.11
1319.00
+ RI-SFT
3.33
19595.04
3.33
19040.31
48.40
8792.70
83.78
3387.50
+ Cold-Start-SFT
2.92
15811.69
2.08
17247.47
43.20
5327.22
83.02
641.50
R1-SFT-RL 7.08
19377.87
5.83
18026.00
53.20
11140040
83.09
4371.49
+ AdaCtrl
9.17
16630.41
6.25
16361.67
55.60
6397.70
87.49
881.69
Table 4:
Hyper-parameter Analysis.o controls the reward weight for difficuly-cctimation calibration, while 3 corresponds the reward weight for the difficulty-aree aware length objective. The parameter ö serves as a predefined threshold for distinguishing between easy and hard samples.
AdaCtrl-7B
AIME2024
AIME2025
MATH500
GSMSK
(%))(
Len. 4
Acc.(%) t
Len.
Acc.(%) 1
Len. 4
Acc.(%) t Len.
0.5, 0,5,8 0.625
21.25
16889.50
19.17
15749.08
74.00
3195.69
90.98
349.34
a 1.0,8 1.0,8 8 0.625
23.75
14762.78
20.00
13986.64
73.80
3134.37
90.30
325.04
0.5, f = = = 0.625
22.08
15616.19
18.75
16116.17
73.80
2836.52
91.13
324.10
1.0,3 8 = 8= 0.625
21.67
17843.66
22.08
16684.28
73.40
3727.77
91.66
365.03
& 0.5, 3 0.5,8 0.5
22.92
13790.53
20.42
14040.20 20
74.00
2888.39
90.60
388.10
0.5, 8 0.5, 8 0.375
20.00
13385.01
20.00
13689.74
74.60
2181.96
90.60
323.27
vs 4371.49 on GSM8K). Although the absolute performance constrained by the differing mathematical capabilities of the base model compared to Qwen, these results confirm that our framework effectively enables diffiuutyy-wwareness and budget control across different model architectures.
Hyper-parameter Analysis. To assess the robustness of our proposed method, we conduct two groups of hyperparameter analyses, as summarized in Table 4. Specifically, we examine: (1) the weights a and 3, which correspond to the difficulty-estimattin calibration reward and the difficulty-aware length reward, respectively; and (2) the difficulty threshold 8. For reward weights analysis, we investigate the effect of varying the reward weights across different combinations {1 0.5 0.5, 1, 0.5 1, : 1 0.5). We observe that accuracy remains largely stable while oulput length exhibits slight variations. For example, on AIME2024, accuracy stays within 21-23% as the response length ranges from 14K to 17K tokens. Similarly, on GSMSK, accuracy consistently falls between 90 91%, while the response length varies from 325 365 tokens. For difficulty threshold analysis, we further sweep the difficulty threshold over {0.625, 0.5,, 0.375). A smaller value of 8 categorizes more samples as Fasy, leading to shorter outputs while maintaining comparable accuracy. For instance, on MATH500, the response length decreases from 3195.69 to 3888.39 and then to 2181.96 lokens, while accuracy remains around 74%. These results collectively demonstrate the robustness of our proposed approach under different hyperparameter settings.
Ablation Study, To further assess the effectiveness of the proposed reward functions, we conducted ablation studics by individually removing the difficulty estimation calibration reward ry and the difficulty-aware length reward rı- As shown in Table 5, the removal of either reward leads to noticeable decline in performance across all four datasets. These results demonstrate that the combined use of all designed rewards is essential for achieving better optimization.
AdaCtrl Serves as Good Difficulty Estimator. To better assess the difficulty estimation capability of AdaCtrl, we analysis the proportion of difficult-awaare tags generated by AdaCtrl-7B across four datasets during reinforcement learning (RL). As illustrated in Figure 4, we first observe that, at the initial stage,
10

---

Published in Transactions on Machine Learning Research (04/2026)
Table 5: Ablation study based on 7B model. ry the cifficulty-essiiaatio calibration reward and 77 is the diffiulty-awaree length reward.
Acc.(%
AIME2024
AIME2025
MATH500
GSM8K
AdaCtrl-7B
21.25
19.17
74.00
90.98
w/o ry w/o rt
17.08 (-4.17%6
16:25 (-2.92%)
72.20 (-1.80%)
89.92 (1.06%)
15.42 (-5.83%)
16.67 (-2.50%)
68.60 (-5.4%)
90.60 (-0.88%)
Easy
Hard
Easy
Hard
Easy
Hard
Easy
Hard
100%
100%
100%
100.0%
Easy: 81.67%
Easy: 75.42%
80%
Hard: 18.33%
80%
Hard: 24.58%
99.9%
60%
60%
Easy: 99.209%
Hard: 0.80%
99.8%
Kass: 99. 20%
Hard: 0.80%
40%
Easy: 1.25%
40%
Easy: 1.67%
40%
Easy: 76. 20%
99.7%
Easy; 99. 85%
20%
Hard: 98.75%
Hord: 98.33%
20%
Hord: 23.80%
99.6%
Hard: 0.15%
0%
0%
0%
99.5%
80
160
240
400
180
500
640
80
160
240
52000
400
400
500
640
80
160
240
320
400
480
560
640
80
160
240
320
400
450
560
640
steps
S
AIME2024
(b) AIME2025
(c) MATH500
(d) GSM8K
Figure 4: The proportion dynamics of difiiuuly-aware tags across different datasets during reinforcement learning.
the model tends to classify most samples in all datascts as casy. This is likely because the cold-start SFT primarily teach the model to explicitly generate self-aware difficulty tags in the expected format, rather than to accurately assess problem difficulty from its own perspective. However, following our designed diffeuult-awarro RL, AdaCtrl predominantly classifies the majority of problems the AIME2024 and AIME2025 datasets as hard. These datasets consist of challenging math Olympiad- problems. In contrast, in the MATH500 dataset, which contains a mixture of easy and difficult problems (the majority of which relatively solvable by current large language models), the model identifies 76.2% of problems as casy. For GSM8K, the simplest dataset among the four, over 99% of problems are calegorized as easy, which also accounts for the superior performance of AdaCtrl-14B (Easy) on GSM8K as reported in Table 2. These results align with the actual difficulty levels of the datascts and demonstrate that AdaCtrl develops a robust capability to estimate problem difficulty through RL.
AdaCtrl Facilitates Accurate Difficult-Aware Budgeting. To further investigate the adaptive difficulty-waar budgeting capabilities of AdaCtrl, we analyze AdaCtrl-7B's responses on the MATH500 dataset, which provides difficulty level annotations for each problem. As illustrated in Figure 5 (a), AdaCtrl generates progressively longer and more elaborate responses as the difficulty level increases from to 5, ranging from approximately 0.3K to 6K tokens. This trend indicates that AdaCtrl can accurately regulate its reasoning budget based on its self-assessed estimation of problem difficulty, thereby enabling automatic and adaptive allocation of computational resources.
Qualitative Analysis of Reasoning Patterns notable observation from Table that AdaCtrl (Hard) generates response lengths approximately 20% shorter than the RI-SFT-RI. bascline, despite both being trained on similar long-form reasoning trajectories. This raises the question of whether this reduction comes at the cost of missing critical reasoning steps. Our empirical results negate this concern: MATH500, AdaCtrl (Hard) outperforms R1-SFT-RL (71.20% vs. 66.80%) while using significantly fewer tokens (5,960 vs. 8,421). We attribute this efficiency to a "conciseness transfer' effect introduced by our mixed-data cold-start. strategy, which regularizes the model against non-functional verbosity.
To further investigate the quality of the generated logic, we conducted a fine-grained analysis of reasoning primitives on 100 sampled traces using Gemini-3-Pro. As visualized in Figure 6, AdaCtrl exhibits a richer diversity of reasoning patterns (28 distinct types) compared to the baseline (22 types). While AdaCtal retains
11

---

hypothesis Published in Transactions on Machine Learning Research (04/2026)
S1,1778
S1.1-7B-PE
Adactrl-BB
Adactrl-7B(Fasy)
8000
7000
6255..7
6400
6000
1600
5000
4000
3517,54
400
3000
100
1838.06
2000
AIME2024
AIME2025
MATH500
GSMSK
1117.71
19022.66
18302.72
5824,47
1000
362,88
S1.1-7B-PE
18026.17
16662.39
$360.42
1608.80
0
Adactrl-7B
16889.50
15749.08
3195.69
349.34
Level Level 2 Level 3 Level 4 Level 5
Adactrl-7B(Easy)
1652.42
896.14
652.76
314.49
(a) Length Distribution
(b) Controllability comparison
Figure (a) The length of response in different difficulty levels of problems in MATH500, where higher levels indicate more challenging problems; (b) Controllability comparison of AdaCtrl and prompting-base approach.
E
Exploration Backtracking . PPlanning
Reflection Decomposition simplification Planning Verification Exp Toration Senoralization
siplificat
Backt
DecosposiTion
senel aubzattin
Figure 6: Reasoning Pattern Diversity Analysis. Comparison of reasoning primitives extracted from R1-SFT-
RL (Teft) and AdaCtrl-Hard (Right).
foundational steps like 'Decomposition' and 'Reflcction, it demonstrates emergent capabilities such as A Algorithuicc Thinking", 'Heuristic Reasoning", and "Self-Correction' that are less prominent in the baseline. This indicates that AdaCtrl produces denser, higher-quality reasoning chains, effectively "purifying" the logic by eliminating redundant loops while preserving and even enhancing the depth of reasoning.
Controllability Analysis. Previous approaches often rely on vanilla prompting methods to achieve token-level control over reasoning budgets. To assess the controllability of AdaCtrl, compare it with prompting-based baseline that uses the $1.1-7B model as the backbone and follows Nayab et al. (2025) to augment user prompts with the instruction: 'Limit the length of the answer to 500 tokens." The response lengths generated by AdaCtrl in both easy and adaptive modes, as well as those produced by S1.1-7B and its prompting based budget control variant, are reported in Figure (b). We can observe that, through the prompting-base based based method (i.e., S1.1-7B-PE) explicitly restricting outputs 500 tokens, only yields reductions in response length of only 5.25%, 8.96%, 7.97%, and 12.48% on the AIME2024, AIME2025, MATH500, and GSMSK datasets, respectively, falling significantly short of the targeted 500-token limit. We set to 1000 tokens. These results suggest that
also observed similar trends when the length constraint achieving precisc, fino-grained control over output length is challenging due to highly dependent on model's instruetion-followiig capabilities, and that current models may struggle to reliably interpret and execute such granular prompts. In contrast, the easy mode AdaCtrl achieves substantially greater compression of reasoning budgets, reducing response lengths by 90.22%, 94.32%, 79.57%, and 9.98% across the same datasets. This demonstrates superior controllability, which can be primarily attributed to AdaCtrl's mixed fine-tuning strategy and its design of a diffieulty-aware length reward. For comprehensive evaluation,
12

---

Training Dynamics During RL. To further in- vestigate how the model learns to allocate adaptive reasoning budgets, we analyze both performance trends and budget dynamics of AdaCtrl-7B throngh- out the reinforcement learning (RL) training process. Published in Transactions on Machine Learning Research (04/2026)
75
21225
20
74
91.5
20
18
18
16
19.17
Acc. (%)
73
Ape: (%)
90.98
Aoc.
16
Acc.
14
72
90.5
90.20
71
90
12 11.25
10
10
70
89.5
8
69
89
10
20
30
60 70 80
10
20
50
60
70
0
10
20
40
50
60
70
10
Steps
Steps
Steps
Steps
20000
4000
500
19000
16000
14000
18000
12000
16889.5 5
17000
15000
13000
15749.08
3500
3000
2500
3195.69
5 350
450
400
345.44
349.34
10000
8000
11000
9000
2000
1500
300
6000
7000
1000
5553
5000
02337.9
805.78
0 10
20
30
40
50
60 70 80
0. 10
20
30
40
50 60 70 80
0. 10
20
30
40
50 60 70 80
0
10 20
30
80 Steps
40
50
60
70
Steps
Steps
Steps
(a) AIME2024
(b) AIME2025
(c) MATH500
(d) GSM8K
Figure 8: Training dynamics accuracy and response length across different datasets during reinforcement learning.
we further provide a detailed accuracy-offiiinny tradc-off comparison between AdaCtrl and prompt-based baselines Appendix D.
Length Distribution of Generated Responses. To further demonstrate the budget allocation capa- bility of AdaCtrl, analyze the response length distribution of AdaCtrl-7B across all inference re- sults from the evaluated benchmarks. Specifically, we compute the distributions separately for samples categorized as easy and hard by the model predicted tags. As shown in Figure 7, AdaCtrl clcarly differen- tiates between easy and hard problems. Notably, the response lengths for easy problems are concentrated within a relatively narrow range. These findings indicate that our approach achieves effective and accurate budget control guided by the self-assessed diffeully-awaro tags.
10a
Easy
Hard
Sample Counts
10a
10¹
10
0
5000
10000
15000
20000
25000
Tokens
Figure 7: The length distribution of response generated by AdaCtrl-7B.
As illustrated in Figure 8, the model exhibits upward trends in performance across all four datascts, suggesting a progressive enhancement in reasoning capabilities. Regarding budget dynamics, we observe distinct patterns across datasets. On AIME2024, AIME2025, and MATH500, the average response length initially increases rapidly during carly training steps, then gradually docreases and stabilizes at level longer than that before RL training. While for GSM8K, the response length remains relatively stable and close to that observed before RL training.
13

---

Published in Transactions on Machine Learning Research (04/2026)
These findings suggest that the reasoning budget allocation learned during cold-start fine-tuning is insufficient for more complex problems, such as those in ATME2024, ATMF2025, and MATH500. Consequently, the model adjusts its budget dynamically in response to actual problem difficulty during RL phase. In contrast, for the comparatively simpler GSM8K datasct, the model is already capable of effectively allocating minimal budgets after cold-start Line-tuning, indicating its ability to distinguish and handle easier problems without. requiring significant adjustment.
5 Conclusion
In this work, propose adaptive and controllable reasoning framework designed to mitigate the problem of overthinking while granting users explicit control over computational resources. To this end, we introduce AdaCtrl that supports both dynamic reasoning budget allocation and user-directed budget adjustments. Our approach utilizes a two-stage training pipeline that combines cold-start fine-tuning with difficulty-aware reinforcement learning. Experiments conducted on four benchmark datasets demonstrate that AdaCtrl effectively allocates reasoning budgets based on self-assessed problem difficulty, leading to performance improvements while dynamically reducing response lengths by 10%-90% This enables flexible trade-offs between efficiency and performance. Furthermore, AdaCtrl unlocks the potential of human-in -eee-ii-tttttoooo control towards reasoning budgets according to tailored needs.
Broader Impact Statement
AdaCtrl enables adaptive and user-controllable allocation of reasoning budgets in large language models, aiming to mitigate unnecessary overthinking while preserving performance harder queries. By reducing response length when extensive reasoning is not needed, it can lower inference cost and latency, which may also reduce the environmental footprint of large-scalc deployment. The difficulty-wware tags provide a lightweight human-in-the-llop mechanism to steer efficiency-effeeee effectiveness trade-ofTs in practical applications. As with most LLM research, there is a general dual-use risk; however, since this work primarily changes how computational effort allocated rather than enabling new capabilities, we expect the incremental risk introduced by this method to be limited.
Limitations and Future Work
Several limitations of our study should be noted. First, to keep the control interface lightweight and reliable, AdaCtrl currently uses a binary difficulty schema (i.e., [Easy] vs. [Hard]), which may be less expressive for intermediate cases. Second, our human-in-the--log, mochanism instantiated as tag- bag-basod control, which provides simple steering but does not yet capture richer user intent such preferences over verbosity, latency, explanation style. Third, although we introduce explicit calibration to improve self-assessed difficulty, the estimated difficulty can still be imperfect in some cases, which may lead to occasional over- under-allocation of reasoning effort.
Future work will explore finer-grained and hierarchical control signals beyond binary tags to enable smoother and more flexible budget allocation while preserving robustness. We will also extend human-in-the-loop control beyond tag-only steering by incorporating richer feedback signals (e.g., preference- or correction-basel guidance) to better align budgeting behavior with user intent. Additionally, we plan to further improve difficulty estimation calibration (c.g., by leveraging more informative uncertainty signals from rollouts and training dynamics), strengthening the consistency between self-assessed difficulty and actual solvability.
Acknowledgments
This research was supported in part by BYD (Grant BYD26EG05), as well as by the Frontier Technology Research for Joint Institutes with Industry Scheme (Grant WF..26FG002.
14

---

Published in Transactions on Machine Learning Research (04/2026)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arxiv preprind arXiv:2303.0877 0,7777 2023.
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement. learning, 2025. URL httpa////arxxi..gg/bbb////// 04697.
Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025. URL https: harxiv.org/abs//522244442 04463.
Art of Problem Solving. Aime problems and solutions. http:p//aarttpoooleeeelllrrrerer pap/AIME_Prrbloutt _and_____________eaaaee___ Sonuddontiions, n.d. Accessed: 2025-05-22.
Vidhisha Balachandran, Jingya Chen, Lingjaa Chen, Shivam Garg, Neel Joshi, Yash Lara, John Langford, Besmira Nushi, Vibhav Vineet, Yue Wu, and Safoora Yousefi. Inference-time scaling for complex tasks: Where we stand and what lies ahead, 2025. URL https: /etps://arxiv.. httpaaaiivv....aaaaa/aaaaa 20g//bb//554.0000 00294.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwi He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like Imms, 2025. URL https://aarat www.aaiwwi...aaaaaaaaaaaeaoeeeee wrg//bs////aaa//a/ 21187.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL
abs/2110..... 14168.
Yingqian Cui, Pengfei He, Jingying Zeng, Hui Liu, Xianfeng Tang, Zhenwei Dai, Yan Han, Chen Luo, Jing Huang, Zhen Li, et al. Stepwise perplexity- guiddd refinement for efficient chain-of thiouggh reasoning in large language models. aixiv preprint or 250: 1eeee.11200 13260, 2025.
DeepSeck-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghus Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li,, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia Hc, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xic, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscek-rl: Incentivizing reasoning capability in lmss via reinforcement learning, 2025. URL https: /tttpp:///rxii.../ wwwpriii.org//base.ae wtg//bs/////...../// 119.//111111.....11111
15

---

Published in Transactions on Machine Learning Research (04/2026)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprinl 2rXiu2407..2188. 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Pciyi Wang, Xiao Bi, et al. Deepsock-r1: Incentivizing reasoning capability in llms via reinforcement learning. Xiv preprint arXiv:2501.12945 2025.
Tingxu Han, Chunrong Fang, Shiyu Zhao, Shiqing Ma, Zhenyu Chen, and Zhenting Wang. Token-budget-aware Ilm reasoning. Xiv preprinl or 12. 2024.
Zhiwei Hc, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath- 103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning, 2025. URL
httpziii.rr/abb//55..11111 11456.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math datasct, 2021. URL attps://aras http://rivv.orgaaaa/aaaaaae org/abs/2103 htt/aaaa/2113.0337//
Shijue Huang, Wanjun Zhong, Deng Cai, Fanqi Wan, Chengyi Wang, Mingxuan Wang, Mu Qiao, and Ruifeng Xu. Empowering self-learning of llms: Inner knowledge explicitation as catalyst. Proceedings of the AAAT Conference on Artificial Intelligence, (51233):2110-24114 Apr. 2025. doi: 10.1609
URL
https://ojs aaai maaii.rg//a. werg//ndx..../ phtp/AAA//articll//veerr//
Guanyu Jiang, Zhaochen Su, Xiaoye Qu, and Yi R. Fung. Xskill: Continual learning from experience and skills in multimodal agents, 2026. URL https://arxiv.org// org/abs/2003. 12056.
Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter sainn-of-hhouhtt without compromising effectiveness. arXiv preprint arXi::2412. 11664, 2024.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhiwei Li, Bao-Long Bi, Ring-Rui Mei, Junfeng Fang, Zhijiang Guo, Le Song, and Cheng-Tin Liu. From system 1 to system 2: A survey of reasoning large language models, 2025. URL /ttts://arxxxr hwtp:/oorg//b//////eee 17419.
Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, and Yi R. Fung. Costbench: Evaluating multi-turn cost-optimal planning and adaptation in dynamic environments for 1lm tool-use agents, 2025a. URL https:
02734.
Tengxiao Liu, Qipeng Guo, Xiangkun Hu, Cheng Jiayang, Yue Zhang, Xipeng Qiu, and Zheng Zhang, Can language models learn to skip steps? arXiv preprint arXiv:2411.0188 1185555, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min cww/sailms//a/eeeeeecca..oss Lin. Understanding rl-zero-like training: A critical perspective. httpss//gitttb..ommaaalsg/ understand-r1-zerd 2025b.
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: harmoniir fine-tuning for ol-like reasoning pruming, orXie preprint arXiv:2501.12570 2025.
Xinyin Ma, Guangnian Wan, Rumpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length-compressibbe chain-of-thought tuning, 2025. URL
09601.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL
https://arxiv. httr::///axxiv.orrg/a/////// arg/abs/2501.15 19393.
Tergel Munkhbat, Namgyu Ho, Scohyun Kim, Yongjin Yang, Yujin Kim, and Se- elicits concise reasoning in large language models, 2025. URL https://arxis //arxiv. wttpt/iww.oor/aa//////eee org/abs//552..211222
Sc- Young Yun. Self-training
16

---

Published in Transactions on Machine Learning Research (04/2026)
Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. Concise thoughts: Impact of output length on 1lm reasoning and cost, 2025. URL
https://arxiv.. www.iii.orggaab//407..oo 19825.
OpenAl. Introducing openai o1, September 2024. URL https://openai.com///
Accessed: 2025-05-21.
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, et al. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. ar preprint arXiv:2503.0 21614, 2025a.
Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhuidinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning, 2025b. URL https: //axxiv org/ htt/ab//2550.000//
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: /ttp///aaxvvvviveeeei/eeeeeawavr oraiii.orrg/eee owggabs/////.ooooooeooooooe /a////ab/222.1155 15115.
Tony Ryan, Shu Wen Tay, P. D. Ryan, and C. Anthony Ryan. Systems and 2 thinking processes and cognitive reflection testing in medical students. Canadian Medical Education Journal, 7:c97 - e103, 2016. URL https: //api. swwanticschollattt..eeeeeeeeaaee httope///pp..eennnccccchooo.rr org/CorpusDDD. 11436274.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL /ttpp:///rrivv wwwlii..rrr/bbba hetpabb//422..0552 03300.
Yi Shon, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian. Dast: Diffcully-sdaapliee slow-thinking for large reasoning models, 2025. URL https: wtt.:o////b///...0////0
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Compuler Systems, EuroSys '25, pp. 1279 1297. ACM, March 2025. doi: 10.1145/ 3689031.3696075. URL http://dx: /oo/xxxxiiooooo//ooooooo//o.oo.. org/10. 1145/3689031 3696075.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https: /ttpxv/xivv.v///////////////.av. hegg/bbs/2ed 03314.
Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, al. Openthinking: Learning think with images via visual tool reinforcement. learning. arx preprint arvvv:250 08617, 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models, 2025. URL https://arxiz: http:///aaxxv..//a///////e/ wttp///////a...oo.o///// 16419.
Kimi Team. Kimi k1.5: Scaling reinforcement learning with 1mss, 2025. URL 12599.
wrw.:..g///...... abs/011
Hongru Wang, Deng Cai, Wanjun Zhong, Shijue Huang, Jelf Z. Pan, Zeming Liu, and Kam-Fai Wong. Self- reasoning language models: Unfold hidden reasoning chains with few reasoning catalyst. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (ods.), Findings of the Association for Computational Linguisties: ACL 2025, pp. 3578-567 5596, Vienna, Austria, July 2025a. Association for Computa- tional Linguistics. ISBN 979-8-891 76--255-5.. URL https://acllanthology org//2/2 findings-acl. 291/
17

---

Published in Transactions on Machine Learning Research (04/2026)
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Acting less is reasoning more! teaching model to act efficiently, 2025b.
URL https://arxiv. httpa////aaxi...gaaaba////// wttp:///..........////.// 14870.
Rui Wang, Hongru Wang, Boyang Xue, Jianhui Pang, Shudong Lau, Yi Chen, Jiahao Qiu, Derek Fai Wong, Heng Ji, and Kam-Fai Wong. Harnessing the reasoning economy: A survey of efficient reasoning for large language models. orXiv preprinl ar urXiv:2503.24377 2025c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural in information processing systems, 35:24824-24533 2022.
Siye Wu, Jian Xie, Yikai Zhang, Aili Chen, Kai Zhang, Yu Su, and Yanghua Xiao. ARM: Adaptive reasoning model. In NeurIPS 2025 Workshop on Efficient Reasoning, 2025. URL https://openrevleet wtw.///reeee///uun id=FCcQ7o0wds.
Hemning Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable chain-of- thought compression in llms. arXiv preprint arXiv:2502. 12067, 2025.
Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less, 2025. URL https://arxil arg/abs/2500 18600.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thoughh reasoning in Ilms, 2025. URL https://arres http:////ardwooo//////////// 2532203333....
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1, 2024. URL https:
//arxiv. wrg/abs/////eeoooooggooeeee 22007000023
Qiying Yu, Zheng Zhang, Ruofci Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingiun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-souree IIm reinforcement learning system scale, 2025a. URL https: https://aaxti //axxiv org/ htrggabb/2503... 14447
Zishun Yu, Tengyu Xu, Di Jin, Karthik Abinav Sankararaman, Yun Hc, Wenxuan Zhou, Zhouhao Zeng, Eryk Helenowski, Chen Zhu, Sinong Wang, et al. Think smarter not harder: Adaptive reasoning with inference aware optimization. Xixiv preprint 2rXiv:2501. 17974, 2025b.
Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufci Wang, Niklas Mucnnighoff, Irwin King, Xue Liu, and Chen Ma. A survey on test-time scaling in large language models: What, how, where, and how well?, 2025. URL https:
org/abs/2500 24235.
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingiun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swiftra scalable lightweight infrastructure for fine-tuning, 2024. URL attps://arxx: www..ogg///aa.e...... httg/bb//240..0556 05517.
18

---

Published in Transactions on Machine Learning Research (04/2026)
Appendix
A Analysis of Difficulty Estimation Capability
To further quantify the reliability of this difficulty estimation and address concerns regarding potential false negatives (i.e., identifying hard problems easy), we evaluated the tagging accuracy of AdaCtrl. Ground-truth difficulty was determined by the pass rate of 16 parallel rollouts (problems solved >> 10 times were labeled Easy). As shown in Table 6, AdaCtrl achieves remarkable precision (> 92% on AIME and > 96% on GSM8K) across both sizes. This indicates that the difficulty estimation calibration reward (ry) effectively aligns the model's assfssssesm with its actual capabilities, making 'first-token commitment' a robust decision mechanism rather than a brittle constraint.
Table 6: Difficulty Tagging Accuracy (%) of AdaCtal across benchmarks. High accuracy indicates robust.ness against mode collapse and false negatives.
Model
AIME2024
AIME2025
MATH500
GSM8K
AdaCtrl-7B
93.33
94.58
87.82
96.58
AdaCtrl-14B
92.08
95.42
89.47
96.40
B Detailed Analysis of Performance Trade-offs in Forced Modes
In the main text, observed that mixed-difficulty datasets like MATH500, the forced Hard mode (71.20%) yields slightly lower accuracy than the Adoptive mode (74.00%). To investigate the underlying causes, we conducted a fine-grained error analysis tracking samples where correctness 'flipped' between the Adaptive mode (specifically when autonomously selected the Easy tag) and the forced Hard mode.
Table 7: Error analysis of forcing Hard mode on MATH500. "Gain" denotes samples corrected by switching from Easy to Hard, while "Loss" denotes samples that were correct Fasy mode but failed in Hard mode.
Model
Gain (Easy x + Hard v)
Loss (Easy v > Hard x)
Overthinking Aualysis within "Loss"
AdaCtrl-7B
27 samples
47 samples
32/47 cases initially derived the correct answer but drifted into error due forced verbosity.
AdaCtrl-14B
18 samples
38 samples
27/38 cases failed due to similar redundancy- induced errors.
As illustrated in Table 7, the forced Hard mode indeed demonstrates the potential to raise the performance ceiling, evidenced by the 27 'Gain' samples where extended reasoning successfully solved complex problems that failed in the concise mode. However, this benefit is currently outweighed by the 47 'Loss' cases. Crucially, our error analysis reveals that these failures do not stem from lack of capability. Taking AdaCtrl-7B as an example, human validation confirms that in 32 out of the 47 'Loss' cases (approx. 68%), the model had initially derived the correct answer but was compelled to drift into incorrect revisions due to 'forced verbosity.' This indicates that the 'intrinsic capability" of the Hard mode actually higher than its final accuracy suggests, but compromisod by 'thinking noise" such as unnocessary sclf-doubt or hallucinatory verification on simpler queries. While this noise is detrimental on mixed datasets, the dynamic shifts entirely on genuinely challenging benchmarks like AIME, where deep reasoning is indispensable and leads to significant gains (e.g., boosting 7B accuracy to 22.08%). This contrast underscores the critical value of AdaCtrl's Adaptive mode: by autonomously identifying problem difficulty, it intelligently navigates this trade-off, successfully filtering out the overthinking noise on simpler tasks while preserving the capacity to deploy extensive reasoning budgets when truly needed.
Regarding the comparison between the Easy mode and the vanilla model, we emphasize that our method maintains comparable efficiency and performance simple tasks while delivering superior robustness on
19

---

Published in Transactions on Machine Learning Research (04/2026)
challenging ones. On GSM8K, the response length is remarkably similar (314 vs 273 tokens) with competitive accuracy (90.75% vs 91.58%). However, complex benchmarks where vanilla models typically struggle, our Easy mode actually outperforms the vanilla baseline: for instance, on AIME 2024, it achieves 14.58% accuracy compared to Vanilla's 11.25%, and on AIME 2025, it reaches 10.00% vs 7.08%. This highlights the core value of our framework: offers the flexibility to secure competitive results with merely 1/10th of the standard reasoning budget (via Easy mode), while retaining the capacity to dynamically expand reasoning length by 10x to unlock maximum performance. This wide dynamic range allows AdaCtrl to navigate the performance-cost trade-off more effectively than any static baseline.
C Analysis of Baseline Prompting Fairness
To ensure rigorous comparison and decouple the effects of prompt engineering from our training methodology, we conducted an additional experiment to assess the fairness of using different prompls for baselines and AdaCtrl. The "Baseline Prompt' used in our main evaluation was modeled after the official system prompt recommended for the Qwen-Math series, ensuring optimal native performance for the baselines.
We applied the 'Diffieully-Aware Prompt" (used in AdaCurl) to the vanilla Qwen2.5-Instucl baseline to determine if performance gains were attributable to the prompt structure itself. As shown in Table 8, replacing the official baseline prompt with the difficulty-aware prompt leads to noticeable drop in accuracy across most benchmarks. For instance, the 14B model's accuracy on AIME2024 falls from 11.67% to 4.58%. We attribute this degradation to semantic interference, as the vanilla model lacks the internalized alignment to map difficulty tags to appropriate reasoning budgets. Consequently, the extra instructions function as out-of-disttibuttin noise rather than valid control signals. This oulcome confirms that the standard 'Baseline Prompt' represents the most robust and favorable configuration for the vanilla model, validating the fairness of our original experimental sctup.
Table 8: Comparison of Baseline Prompt vs. Difficulty-Aware Prompt on vanilla Qwen2.5-Instrr models. Applying the difficult-aware prompt to the baseline results in performance degradation, confirming that the baseline prompt is the fairer and stronger setting for comparison.
Model
AIME2024
AIME2025
MATH500
GSM8K
(%)
Len. +
A%c.(%)
Len. 4
Acc. 1
Len. 4
Aoc.(%) 1
Len. +
Qwen2.5-7B-Be (Baseline Prompt)
11.25
1805.60
7.08
1174.06
73.00
628.91
91.58
272.93
Owen2.5-7B-Iin (Diff-Aware Prompt)
10.42
1715.62
1402.52
67.60
538.29
87.34
237.83
Qwen2.5 5- 1-D---Iss (Baseline Prompt)
11.67
1043.20
10.42
1136.26
73,60
568,25
93.86
215.42
Qwen2.5. InB-Ins (Diff- Aware Prompt)
4.58
830.68
5.00
686.01
73.00
409.86
85.75
188.47
D -fccury Trade-off in Prompt-based Controllability
Table Comparison of accuracy and length between Prompt-based Control (S1.1-7B-PE) and AdaCtrl. The prompt-base method fails to achieve genuine budget control (e.g., >18k tokens on ALME) and underperforms AdaCtrl in afficiency-accuracy ratio.
Model
AIME2024
AIME2025
MATH500
GSM8K
(%))
Len.
A%c.(%)
Len. 1
A%c. t
Len.
Acc.(%)
Len.
S1.1-7B-PE
17.50
18026.17
16.67
16662.39
70.80
5360.42
90.83
1608.80
AdaCul-7B (Easy)
14.58
1652.42
10.00
896.14
70.80
652.76
90.75
314.49
AdaCul1-B (Adaptive)
21.25
16889.50
19.17
15749.08
74.00
3195.69 69
90.98
349.34
In Section 4.3, we demonstrated that prompt-based basclines (e.g., S1.1-7B-PE with instructions 'limit length") fail to achieve effective token reduction. To further assess the trade-off between controllability and
20

---

Published in Transactions on Machine Learning Research (04/2026)
effectiveness, supplemented the accuracy data for this baseline and compared it directly with AdaCtrl's Easy and Adaptive modes.
As shown in Table 9, relying solely on prompting not only fails in length control but also yields an inferior afficinccy-cceuracy ratio. For instance, on GSMSK, while S1.1-7B-PE achieves 90.83% accuracy, it consumes 11,608 tokens. In contrast, AdaCtrl (Fasy) matches this accuracy (90.75%) with only 314 tokens, achieving 5x efficiency improvement. On challenging tasks like AIME 2024, although S1.1-7B-PE shows higher accuracy than the Easy mode, its length balloons to over 18,000 tokens, indicating total failure of budget control. Meanwhile, AdaCtrl (Adaptive) significantly outperforms the prompt baseline (21.25% 17.50%) with reduced token usage, demonstrating superior Pareto frontier.
21


Reviews:
[{"review_id": "DErozOugje", "reviewer": "Reviewer_8nVw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\nThe paper introduces AdaCtrl, a framework designed to address the computational inefficiency and \"overthinking\" issues prevalent in Large Reasoning Models. The authors argue that while deep reasoning enhances performance on complex tasks, it incurs unnecessary overhead on simpler problems.\n\n**Pros**\n1. **Novel Reward Design:** Introduces a \"Difficulty-Aware Length Reward\" ($r_l$) that only penalizes verbosity when the problem is tagged as [Easy], preserving reasoning depth for hard tasks.\n2. **Effective Pipeline:** Demonstrates that the \"Cold-Start Fine-Tuning\" stage is critical; without it, RL alone fails to learn efficient budgeting strategies.\n3. **User Controllability:** Provides option for human control via explicit [Easy] and [Hard] tags, allowing users to manually prioritize efficiency or performance.\n\n**Cons**\n1. **Questionable Premise:** The motivation relies on \"overthinking\" phenomenon, which data suggests may be an artifact of distillation rather than an inherent flaw.\n2. **Limited Evaluation:** Experiments are restricted to a single model family (Qwen), limiting generalizability.\n3. **Performance Anomalies:** Unexplained degradation in forced \"Hard\" modes and inefficiencies compared to vanilla baselines on easy tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper presents a technically interesting solution to the problem of adaptive compute; however, there are several concerns regarding the framing of the problem and the robustness of the experimental analysis that need addressing.\n1. **Misleading Premise Regarding \"Overthinking\" in Base Models:** The paper motivates the method by claiming LLMs suffer from substantial computational overhead due to overthinking. However, for example, Table 1 shows that the both vanilla base model is actually concise and highly-accuracte for some easier dataset already, including GSM8k and MATH. The \"overthinking\" issue (generating ~20k tokens) only appears in the distilled baselines (R1-SFT). Therefore, the paper is essentially targeting a problem introduced by the distillation process itself, rather than an inherent flaw in the base LLM.\n2. **Lack of experiment from other model family:** The main result comes from Qwen models only, and it should contain other family of models for stating to solve a broadly exist problem\n3. **Potential \"Mode Collapse\" at Inference Time:** The framework requires the model to commit to a difficulty tag ([Easy] or [Hard]) at the very first token of generation. The paper does not analyze the recoverability of this decision. If the model incorrectly tags a hard problem as [Easy], it is effectively locked into a concise budget and likely fails. An analysis of \"False Negatives\" (Hard problems tagged Easy) is missing to understand the failure modes of this rigid tagging system.\n4. **Performance Degradation in Forced \"Hard\" Mode:** Table 2 reveals that on the MATH500 dataset, the forced [Hard] mode achieves lower accuracy (71.20%) than the Adaptive mode (74.00%). Intuitively, allocating more budget should maintain or improve performance unless the reasoning quality degrades. The author does not provide a clear explanation on this. Furthermore, the [Easy] mode has 1: lower accuracy, 2: higher average token usage than the vanilla mode. I think an explanation is also necessary in this part.", "audience_yn": "Yes", "audience_explanation": "The paper addresses the challenge of computational inefficiency in Large Reasoning Models, making it relevant to researchers focused on efficient inference and reinforcement learning.   \nFurthermore, the proposed framework for adaptive and controllable budgeting offers practical insights that would be valuable to people seeking to deploy reasoning models in resource-constrained environments.", "requested_changes": "The author should address concerns mentioned in \"Cons\" section above, and answer the following questions, for strengthen the work.  \n**Questions**\n1. Base on my use experience, the vanilla qwen model is actually not affected by overthinking, and this is also reflected in Table 1. I think the author need to compare the response length of the common correct subset between the vanilla Qwen model and other method, including AdaCtrl.\n2. Why is the average response length of AdaCtrl in forced *Hard* mode noticeably shorter (~20%) than the R1-SFT and R1-SFT-RL baselines, despite both being trained on similar 'hard' reasoning trajectories? Does the inclusion of *Easy* samples in the training data cause a 'conciseness transfer' that regularizes the lengthy reasoning patterns even when the *Hard* tag is active? If so, does this reduction in length come at the cost of missing specific reasoning steps present in the pure R1 baselines?\n3. Models like deepseek-distilled-qwen-7b is actually suffered from overthinking and backtracking a lot. Can this method use to mitigate this phenomenon for a heavy RLed model?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yk1srsRBpk", "reviewer": "Reviewer_BBA3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "These paper proposes a novel two-stage training paradigm designed to achieve both adaptive (model-driven) and controllable (user-driven) reasoning budgeting. Cold-start SFT aligns the model to generate concise or extensive responses based on prepended difficulty tags ([Easy] or [Hard]) using a curated mix of short and long CoT data. Difficulty-Aware RL utilizes GRPO with a specialized reward system, including a Difficulty Estimation Calibration Reward and a Difficulty-aware Length Reward. Experiments demonstrate that AdaCtrl can reduce response lengths while maintaining or even improving accuracy on math benchmarks like AIME2024 and GSM8K.\n\n**strengths**\n1. Cleverly uses the relative pass rates of multiple rollouts in GRPO as a \"golden difficulty\" signal to train the model's self-awareness.\n2. Offers flexibility by allowing the model to self-assess difficulty while also providing a \"human-in-the-loop\" mechanism to explicitly steer the reasoning budget via tags.\n3. Shows significant improvements over baselines.\n\n**weaknesses**\n1. The model must decide the difficulty at the very first token. This \"judgment-before-thought\" might fail on \"trap\" questions that appear easy but require deep reasoning to solve.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides strong empirical data across four major math benchmarks. The results clearly show that AdaCtrl achieves significant token reduction while maintaining or improving accuracy compared to SFT and RL baselines. Figure 4 and Figure 6 effectively visualize how the model learns to differentiate difficulty levels and adjust its \"thinking budget\" accordingly.", "audience_yn": "Yes", "audience_explanation": "With the rise of reasoning LLMs, the researchers in LLMs and RL is highly focused on the efficiency of test-time scaling. AdaCtrl addresses a bottleneck of these models: the \"overthinking\" problem. The strategy of using RL to calibrate \"model self-awareness\" (difficulty estimation) is also a topic of significant interest.", "requested_changes": "1. **Analysis of Reward Competition (Critical)**: There is a potential logical conflict between the Difficulty-aware Length Reward and the Difficulty Estimation Calibration Reward. Since length penalties are only applied when the model generates the [Easy] tag, the model has a strong incentive to \"cheat\" the system by labeling easy problems as [Hard] to bypass the length constraint and maximize reward. Please discuss whether this \"reward hacking\" (avoiding penalties by mislabeling) was observed during training.\n2. **Baseline Prompting Fairness and Ablation (Critical)**: The current evaluation compares AdaCtrl (using a \"Difficulty-Aware Prompt\" that instructs the model to judge difficulty) against baselines using a standard \"Baseline Prompt.\" It is well-documented in LLM literature that simply adding meta-cognitive instructions (like \"judge the difficulty\") can alter a model's activation paths and performance, regardless of the underlying RL training. Please conduct an additional experiment where the baseline is given a similar difficulty-aware prompt.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O2ehGxwXGH", "reviewer": "Reviewer_nv4F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Paper summary**\n\n- Method:\n    - They propose a framework that dynamically adjusts the reasoning length based on the model’s self-assessed problem difficulty.\n    - their framework consists of two stages:\n        - cold start finetuning on curated datasets with explicit difficulty tags. The dataset they used is DEEPMATH that includes difficulty annotations from 1 to 9 which they map to easy (1-5) and hard (6-9).\n        - reinforcement learning to calibrate model’s assessment of problem difficulty. They use GRPO and optimize three rewards:\n            - outcome accuracy reward\n            - difficulty estimation calibration reward\n            - difficulty-aware length reward\n    - They experiment on Qwen2.5-instruct 7B and 14B as base models.\n- Training data:\n    - to generate reasoning outputs for training data, the base models are used for easy problems while DeepSeek R1 is used for difficult problems. the finetuning data includes 4K easy and 4K hard instances (cold-start dataset).\n    - for RL, they use 10K easy and 20K hard instances for training (RL dataset).\n- Baselines:\n    - base models: Qwen 2.5-instruct 7B and 14B\n    - S1.1: a reasoning model obtained by finetuning the base model on S1-1.1K dataset (no description of the data is provided).\n    - R1-SFT: a model obtained by finetuning the base models on a curated dataset similar to the cold-start dataset described before, but with all answers generated using DeepSeek R1.\n    - R1-SFT-RL: an RL model obtained by training R1-SFT model on the RL dataset using the outcome accuracy reward.\n    - Cold-Start: a finetuned base model on the cold-start dataset.\n    - Cold-Start-RL: an RL model obtained by training Cold-Start model on the RL dataset using the outcome accuracy reward.\n- Evaluation data:\n    - GSM8K (easy)\n    - MATH500 (easy)\n    - AIME2024 (hard)\n    - AIME2025 (hard)\n- Results:\n    - Compared to the tested baselines, AdaCtrl provides the best balance between performance and reasoning budget.\n    - finetuning on their cold-start dataset is better than finetuning on DeepSeek R1 responses only.\n    - training with the three rewards achieves better efficiency effectiveness balance than training on the outcome accuracy reward only.\n    - AdaCtrl provides user controllability over reasoning budget.\n    - AdaCtrl is robust to hyperparameter changes in the weights of the reward functions and the predefined threshold of easy and hard samples.\n    - AdaCtrl is better at length control than prompt-based controllability.\n\n**Strengths**\n\n- The approach is simple and clearly explained.\n- I appreciate the extent of the ablation studies on different components of the pipeline including the training data, the RL step, the reward functions and their hyperparameters.\n\n**Weaknesses**\n\n- I find that the result section is focusing more on the efficiency of AdaCtrl and less on its effectiveness. For example:\n    - explicitly setting the difficulty tags in the controllability results in Table 2 has more effect on the length than on the accuracy. The easy mode results in much worse performance than the adaptive mode, while the hard mode does not improve much over the adaptive mode. This raises the question of the benefit of controllability to the accuracy.\n    - In the controllability analysis in Figure 4(b), the paper focuses on response length comparison only. I would like to see the effect of prompt-based controllability on answer accuracy as well to better compare the trade-off.\n- the study is missing quantitative comparisons to other SFT approaches (distilling concise paths from best-of-N sampling, omitting steps, compression, pruning steps) and RL approaches (length optimization through prompting, utility maximisation, heuristics) described in the related work section. It makes it hard to place this work within the literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Other that the balance between efficiency and effectiveness, the main claims in the paper are supported by exhaustive experiments.", "audience_yn": "Yes", "audience_explanation": "It is a relevant paper to anyone working on dynamic reasoning with LLMs, controllability, and efficient SFT and RL algorithms for LLMs. However, as mentioned before, it is difficult to quantitatively compare it to other approaches that try to solve the same problem.", "requested_changes": "### \n\n- **Critical**\n    - See weaknesses above.\n\n- **recommended to strengthen the work**\n    - besides Table1, it would be nice to present the accuracy-length trade-off in a Pareto frontier, as the numbers in the table are hard to read.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)