THE END OF MANUAL DECODING: TOWARDS TRULY END-TO-END LANGUAGE MODELS
Authors: Zhichao Wang
Dongyang Ma
Xinting Huang
Deng Cai
Tian Lan
Jiahao Xu
Haitao Mi
Xiaoying Tang
Yan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: dynamic decoding
instruction-based Control
truly end-to-end
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduces AutoDeco to dynamically generate sampling parameters which improves LLMs' performance with almost no added latency. Crucially, it enables the model can understand natural language commands and actively steer its own decoding parameters.

Abstract: The "end-to-end" label for LLMs is a misnomer. In practice, they depend on a non-differentiable decoding process that requires laborious, hand-tuning of hyperparameters like temperature and top-p. This paper introduces AutoDeco, a novel architecture that enables truly "end-to-end'' generation by learning to control its own decoding strategy. We augment the standard transformer with lightweight heads that, at each step, dynamically predict context-specific temperature and top-p values alongside the next-token logits. This approach transforms decoding into a parametric, token-level process, allowing the model to self-regulate its sampling strategy within a single forward pass.

Through extensive experiments on eight benchmarks, we demonstrate that AutoDeco not only significantly outperforms common decoding strategies but also achieves performance comparable to an oracle-tuned baseline derived from "hacking the test set"—a practical upper bound for any static method. Besides, we demonstrate an emergent capability for instruction-based decoding control: the model learns to interpret natural language commands (e.g., ''generate with low randomness'') and adjusts its predicted temperature and top-p on a token-by-token basis, which may open a new paradigm for steerable and interactive LLM decoding.

Published as a conference paper at ICLR 2026
THE END OF MANUAL DECODING: TOWARDS TRULY OND-TT-ENDI LANGUAGE MODELS
Zhichao Wang
Dongyang Ma
Xinting Huang²
Deng Cai
Tian Lan²
Jíahao Xu²
Haitao Mi²
Xiaoying Tangli4 Yan Wang"
School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)
Tencent AlLab
Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen), Shenzhen, China *Guangdong Provincial Key Laboratory of Future Networks of Intelligence, Shenzhen, China zh ahichaowang www.kkkkkkkkkkk.......eeee edu. dongyarggma@eencent com tangxi wwwpxiccynngccce www.:nneeuucuccc.c.r..mee edu.cn
com
ABSTRACT
The "end-to-end" label for LLMs is a misnomer. In practice, they depend on a non-differentiable decoding process that requires laborious, hand-tuning of hyper- parameters like temperature and top-p. This paper introduces AutoDeco, a novel architecture that enables truly "end-to-end" generation by learning control its own decoding strategy. We augment the standard transformer with lightweight heads that, each step, dynamically predict context-specific temperature and top-p values alongside the next-token logits. This approach transforms decoding into parametric, token-level process, allowing the model to self-regulate its sampling strategy within single forward pass.
Through extensive experiments on eight benchmarks, demonstrate that Au- toDeco not only significantly outperforms common decoding strategies but also achieves performance comparable an oracle-tuned baseline derived from "hack- ing the test set" practical upper bound for any static method. Besides, we demonstrate an emergent capability for instruction-baser decoding control: the model learns to interpret natural language commands (e.g., "generate with low randomness") and adjusts its predicted temperature and top-p on a token-by-token basis, which may open a new paradigm for steerable and interactive LLM decoding. Our model and code are available at: https: /tt::///iitt /qgttub..../..//////////.....h com 2ackks17/ Autooeco
INTRODUCTION
LLMs have become the de-facto standard in NLP. yet the quality of their generated text hinges on surprisingly manual and heuristic process: the selection of decoding hyperparameters. Parameters like temperature, top-p, and top-k must be carefully chosen through a task-dependent process of manual sweeps and post-hoe filtering (Shi et al. 2024). This not only incurs significant computational and human costs but also profoundly impacts the final output's creativity, diversity, and factual correctness, undermining the promise of a truly "end-to-end" system.
This reliance on static, hand-tuned parameters creates fundamental bottlenecks. Firstly, the search for an optimal configuration widely acknowledged as laborious process because the ideal settings are highly task-dependent; commercial API providers like DeepSeek, for instance, explicitly recommend different temperature settings for distinet application scenario However, this problem, runs even deeper: single static configuration is inherently suboptimal because the ideal level of stochasticity varies dramatically within a single generation. For instance, a model might need high creativity to
Equal Contribution
1 Corresponding Author

---

Published as a conference paper at ICLR 2026
End-to-End Decoding
Topep lead
sigmoid
0.93
Sampling
("temperature", 0.72,
Next Token
"top_ 0.93)
Temp head
sigmoid
Manual Decoding
Input Prefix
Soher
Sampling
Layer
Next
Taken
("temperature"; 0.4,
"top_p": 0.85)
Figure overview of our proposed end-to-end decoding architecture compared to manual decoding. Our method dynamically predicts temperature and top-p values from the model's s hidden states for each generation step. In contrast, manual decoding (bottom) relies on single set of static, predefined hyperparameters for the entire sequence generation.
explore initial reasoning paths but high precision to deliver the final answer. This on-the-fly control is, by design, impossible for current LLMs achieve natively. Consequently, the prevailing static decoding paradigm is a solution as inefficient it is ineffective, forcing one-size-fits-all strategy onto a problem that demands dynamic adaptation.
In this paper, we propose AutoDeco, a novel architecture that creates truly "end-to-end" language model capable of controlling own decoding process. illustrated in Figure we augment the standard transformer with lightweight, dedicated prediction heads. At each decoding step, these AutoDeco heads leverage the model's current hidden state to dynamically predict the optimal sampling parameters for the next token. This seamlessly integrates hyperparameter selection into the model's forward pass, creating segflregul inference pipeline that adds nearly-zero latency.
We validate approach by integrating AutoDeco into major model families, including Qwen, Llama, GPT, and DeepSeek, requiring only a brief fine-tuning process of 400 steps. Across eight distinct benchmarks, the results striking: AutoDeco not only consistently outperforms standard default decoding settings but also matches or surpasses the performance of meticulously expert- guided tuning (an oracle-tuned baseline derived from "hacking the test set") hyperparameters. An important secondary benefit of our architecture is the observed capacity for instruction- based decoding control, which is learned unexpectedly during the "end-to-end" optimization. When prompted with a meta- instruucittttoon like, "Please ensure that the diversity of your output is low," the model immediately responded by lowering its average predicted temperature and top-p values by 0. I and 0.09, respectively. This demonstrates that AutoDeco does not merely automate tedious process; it endows the model with a new, intuitive way to interpret and act on user intent.
Our contributions four-fold: (i) We propose AutoDeco, novel and lightweight architecture, along with an efficient strategy to train its prediction heads, that makes LLM generation truly "end-to-end" by dynamically predicting decoding parameters at each step. (ii) We demonstrate through extensive experiments that AutoDeco consistently matches or exceeds the performance of expert-guided tuning, static hyperparameters across eight benchmarks and multiple model families. (iii) We demonstrate for the first time that an LLM's 's decoding can be controlled by natural language. (iv) We release a set of AutoDeco heads, trained on the most widely adopted open-source models, providing the community with streamlined, drop-in solution for immediate deployment.
AUTODECO
The foregoing discussion raises two fundamental questions that frame the core inquiry of this work:
First, how train the AutoDeco heads without any token-level "ground-truth" labels for the optimal temperature and top-p values? Second, how can these predictions integrated into inference without adding computational latency? This section details our solutions to both.
2

---

Published as a conference paper at ICLR 2026
Original Probability Distribution
1.0
Top
Hard Cup-Oft Tipp Prots
Top (A
Mark Value (m)
Probaiiiity
0.15
6.6
0.4
010
0.05
Cut-Off
Diferenttacle Tap Mosk
Threshold (As0.6)
6.0
0.00
0.6
02
0.4
0.6
10
10
20
50
Cumulative Probability (c)
Token Index (Sorted)
(a) Top-p mask function.
(b) An example of top-p sampling probability.
Figure 2: Comparison of the differentiable "soft" top-p sampling (decay steepness 30) with the standard hard-cutoff method. (a) illustrates the standard hard-cutoff mask, which has a non- differentiable step, against proposed smooth and differentiable "soft" mask. (b) shows the effect of applying both masks to an example original probability distribution, where the "soft" mask method produces differentiable probability distribution suitable for "end-to-end" training.
In Section 2.1 we will introduce our training strategy and explain how we train both heads in an "end-to-end" manner. Then, in Section 2.2 we will walk through our inference process. The AutoDeco modifies the model's final output probabilities internally-4 design that adds absolutely no extra latency. The result is model that can be used almost exactly like standard one, requiring only "I "I-line-change" a user's code to unlock its dynamic decoding capabilities.
2.1 TRAINING STRATEGY
The central challenge in training AutoDeco is the absence of token-level "ground-truth" labels for sampling parameters. A natural approach would be to optimize the temperature and top-p heads directly from the final cross-entropy loss of the generated tokens. However, this path obstructed by the standard top-p sampling algorithm. Its "hard cutoff" retainnng only the smallest set of tokens whose cumulative probability exceeds threshold is a non-differentiable operation, which severs the gradient flow from the loss back to the top-p head.
To overcome this, we introduce novel, differentiable "soft" top-p mechanism that is used during training, enabling fully "end-to-end" optimization strategy. Traditional top-p sampling methods assign a probability of zero all tokens beyond the top-p threshold, while our approach is different: for tokens that fall outside the top-p threshold, apply differentiable weight scaling. The further token is from the threshold, the more its probability is scaled down, eventually approaching zero.
The following is the training data stream:
1. Temperature-Scalle Probabilities: First, we scale the predicted logits to compute the initial probability distribution using the predicted temperature T.
softmax
2. Differentiable Mask Generation: After sorting the probabilities p and calculating their cumu- lative sum c, we generate "soft mask" m (sorted) This is done in single step that combines the thresholding and decay logic:
(sorted)
exp (-a: ReLUUC- -)))
Here, a is hyperparameter that controls the steepness of decay. As shown in Figure 2a this formulation ensures that for tokens inside the nucleus (where < PP) the ReLU term zero, resulting in mask value of 1. For tokens outside, the mask value smoothly decays towards zero their cumulative probability further exceeds P.

---

Published as a conference paper at ICLR 2026
3. Final Probability Distribution: The "soft mask" (unsorted to match the original vocabulary order) is applied to the initial probabilities, and the result is re- re-normalized to form the final, differentiable distribution p:
P
E(pōm)+
where e is a small constant for numerical stability, In Figure 2b we provide an example with vocabulary size of 50 to illustrate how the model's predicted probability distribution changes after the application of our "soft" top-p sampling. As the probability of the token exceeding P decreases gradually and differentially, the "soft" top-p sampling becomes the final piece of the puzzle for the AutoDeco's "end-to-end" training.
Training. As the entire process differentiable, gradients from the standard cross-entropy loss are backpropagated to simultancously update the parameters of both the temperature and top-p heads, allowing the model to learn its own optimal, context-specific decoding strategy by directly optimizing for the final task objective.
Theoretically, these two heads could be trained from the pre-training stage. However, in this paper, we build upon a pre-trained LLM, freezing its base parameters and solely training the AutoDeco heads. While training these heads on SFT data provides a strong baseline, we find that applying some certain de-biasing operations to the data can further enhance model performance and robustness:
Easy-Token Masking. For many tokens, the base model's greedy prediction already matches the ground-truthh truth. These "easy" tokens often yield an optimal temperature T, near zero, biasing the head to be overly conservative. To mitigate this, we randomly mask the training loss for large fraction (e.g.. 60%) of these positions, forcing the model to learn from more challenging and informative examples.
Dynamic Fine-Tunine Tuning, Conversely, naive fine-tuning approach can cause the temperature head to predict unexpected large values for uncertain tokens. We incorporate Dynamic Fine- Tuning (Wu et al., 2025), which re-weights the training loss to focus on tokens where the model has reasonable prior. This teaches the head to apply high temperatures more judiciously in situations calibrated uncertainty, rather than being skewed by outlier signals.
2.2 INFERENCE: DYNAMIC DECODING
At the heart of AutoDeco lies a design optimized for efficiency. By seamlessly integrating all dynamic adjustments into the model's 's standard forward pass, it avoids any separate, costly computational steps. This architecture results in a negligible latency overhead, typically adding only 1 1-2% to the total generation time. As illustrated in Figure T. the process for each token generation step is follows:
1. Compute Hidden State: The base LLM computes the final hidden state he
2. Predict Decoding Parameters: In parallel, the standard 1m. mea computes the logits while the AutoDeco heads predict the dynamic parameters. The temperature is predicted directly from the hidden state. Crucially, the top-p head then uses both the hidden state and the just-predicted temperature as input:
f temp temp-head(h)) head(h:),
P. top-p top-p-head(h, ft).
This micro-dependency, shown as dashed arrow in Figure allows for more nuanced interplay between the two parameters.
3. Internal Probability Modification: The model immediately uses the predicted 7, and P. to internally rescale and filter the logits, producing final, synamically-adjjssei distribution.
Latency and Simplicity. The AutoDeco heads (simple 2-layer MLPs) add negligible computational overhead compared to the massive transformer layers. This internal architecture results in only 1-2% additional latency and makes usage incredibly simple, and ensures seamless integration, allowing an AutoDeco-enables model to serve as drop-in replacement for its standard counterpart, requiring no modifications to the user's existing generation logic.
4

---

Published as a conference paper at ICLR 2026
3 EXPERIMENTS
We conduct extensive experiments to validate AutoDeco, structuring our evaluation around its core contributions performance, efficiency, and surprising capability that emerged as a byproduct.
In Section 3.2.1
we demonstrate the superior performance AutoDeco. It not only substantially outperforms standard, non-expert decoding baselines (Greedy Search and Default Sampling) but also matches or even slightly surpasses the performance of optimal static hyperparameters found through exhaustive expert-guided tuning.
Following this, Section 3.2.2 we analyze its practical efficiency and confirm that AutoDeco introduces minimal computational burden, with marginal latency increase of 1-2% and negligible memory footprint.
A noteworthy finding is presented in Section 3.3 the emergent capability of AutoDeco to interpret natural language commands to dynamically steer its own generation style. This development is a significant step towards more intuitive and controllable AI.
3.1 EXPERIMENTAL SETUP
To demonstrate broad applicability, we select representative models from three of the most popular open-source model families. All AutoDeco heads are trained top of the official pre-trained checkpoints of these models:
Models.
Ilama-3. L. --w.oooooo--annn------------- (Beroovich et al. 2025): A general-purpose model from the widely-used Llama family, developed by Nvidia (hereinafter Jama--emotronn-8B)
RI-Distill-Qwen-7BBB
al. 2025): A distilled model from the Qwen family developed by
DeepSeek, known for its strong reasoning capabilities.
2025): : An advanced MoE architecture instruct
(non-thinking) model from Qwen. (hereinafter Owen3-30B-Instruch)
2ww.://-----222---iiiiinnnn----0 ing model from Qwen. (hereinafter Qwen 235B 2-5---3inTB-inng Thinking)
2025): An advanced MoE architecture Think-
OpenAA--PTT-SS--288/// et al.
2025): A MoE model with 20B parameters released by OpenAI. The reasoning effort is set to medium by default.
More models, including DeepSeek- D V3.1-Terminus (with multi-token prediction) (DeepSeck-A) 2024 and results can be found in the Appendix
Datasets.
The models are trained focused domain and evaluated on wide range of tasks to test for generalization.
Training Data: The AutoDeco heads are trained on a specialized dataset of reject sampling trajectories. These trajectories were generated by sampling solutions from all the base models on problems from the DeepMath- 103K dataset (He et al. 2025).
Evaluation Benchmarks: We evaluate on a diverse suite of eight benchmarks, split into two categories to assess both in-domain and out-of-domain performance:
In-Domain (Math): AIME 25).. BRUMO25, HMMT25 (Balunović al. 2025 and
BeyondAIME (ByteDance-Seed) 20255
-Out-of- (General Tasks): GPQA-Diamond (Rein et al. 2024 and MMLU-Pro
[Wang al. 2024b) (QA). LiveCodeBenchV66 (Jain et al.) 2024) (Code), and IFEval Zhoou
et al. 2023 (Instruction Following).
We evaluate AutoDeco against two standard, non-expert decoding strategies: Greedy Search and Default Sampling (T = 1.0, P 1.0). Furthermore, to establish practical upper bound, we also compare against an Expert-Guided Tuning. crucial to note that
Baselines and Evaluation.
We focus these recent, hard benchmarks mitigate the risk of data leakage issues in older datasets.
5

---

Published as a conference paper at ICLR 2026
Table 1: @ass@ accuracy on mathematical reasoning benchmarks. AutoDeco consistently outper- forms both Greedy Search and Default Sampling methods across various models.
Model
Method
AIME
BRUMO25
HMMT25
BeyondAIME
Average
Greedy Search
55.00
60.00
26.67
37.00
44.67
Maima-NemotronSPi
Default Sampling
51.24:11.31
60.68411.1
30.89:1..29
32.65:0.51
43.86
AutoDeco (Ours)
56.07:11.12
63.23:11.04
34.37
34.51:10.54
47.05
Greedy Search
40.00
43.33
20.00
24.00
31.83
R1-Distill-Qwae Qweill
Default Sampling
43.62:1:1.06
51.38:4:1.88
22.50:1:.888
24.87:10.72
35.59
AutoDeco (Ours)
47.57
53.93
2.746.10.76
10.95
38.24
Greedy Search
65.00
70.00
36.67
45,00
54.17
Qwen3-30B-Inse
Default Sampling
66.95:11.44
70.99:11.1
44.69:10811
47.02:4:.33
57.41
AutoDeco (Ours)
68.22.10.85
10.36
43.15:11.10
47.15
57.47
Greedy Search
80.00
83.33
63.33
$3.00
69.92
Qwen3-235B Thinking
Default Sampling
80.76:+:0.43
82.40++0.69
63.41+1.20 1.20
50.05++0.56
69.16
AutoDeco (Ours)
82.79.10.72
84.14
66.07:11.11
$1.88
71.22
Greedy Search
56.67
73.33
$6.67
37.00
55.92
OpenAT-GPT-OSS-2009
Default Sampling
69.65-t1..3
7..50++1.11
50.91+22.76
47.11++0.23
60.79
AutoDeco (Ours)
7..2
10.21
53.00:11.02
47.32.10.37
62.06
Table 2: Pass 1 accuracy general-domain benchmarks. AutoDeco shows exciting generalization performance across General QA, Code Generation, Instruction Following tasks.
Model
Method
GPQA-Diamond
MMLU-Pro
LiveCodeBenchV6
IFEval
Average
Greedy Search
51.01
$2.00
19.17
71.53
48.43
Llama-Nemotron-BB
Default Sampling
44.93
54.00
21.22
65.25
46.35
AutoDeco (Ours)
50.52
55.64
21.68
71.02
49.72
Greedy Search
37.87
47.20
14.98
33.24
RI--Dsttll-Qwae
Default Sampling
47.41
47.65
15.40
32.35
35.70
AutoDeco (Ours)
48.91
50.75
15.46
33.90
37.26
Greedy Search
65.86
78.00
47.75
83.73
68.84
Owen3-30B-Insmme
Default Sampling
76.25
48.52
81.52
AutoDeco (Ours)
69.96
78.38
49.80
82.81
70.24
Greedy Seurch
77.78
81.00
77.25
31,98
67.00
Owen3-235B-Thinkeee
Default Sampling
80,81
80.33
77.47
31.61
67.56
AutoDecc (Ours)
81.13
79.20
78.10
32.90
67.83
Greedy Search
59.60
67.00
69.69
29.94
56.56
OpenAI-GiPTOOSS.2U0
Default Sampling
70.15
AatoDeco (Ours)
66.48
69.12
71.25
30.84
59.42
this expert-tuned baseline is oracle setting, it involves finding the optimal statie hyperparameters by tuning on the test set-a process that is infeasible in real-world applications.
Our primary metric is Pass accuracy, estimated via oversampling with 128 samples per problem (with 8 random seeds, 16 samples per seed). The maximum generation length is set to 32768.
3.2
MAIN RESULTS
We present our main findings separately for mathematical reasoning and open-domain question answering to provide a clear and detailed view of AutoDeco's performance across different domains.
3.2. PERFORMANCE
In-Domain Performance. As shown in Table AutoDeco consistently demonstrates a performance boost compared to Greedy Search and Default Sampling. For instance, on Homma-Nemotron-BB achieves average score of 47.05. substantial average improvement of nearly 2.8 absolute points over Default Sampling and Greedy Search.
One may notice that the performance gain from AutoDeco is less pronounced Qwen3-30B-Ar Instruct-2507 compared to other models. This may stem from Qwen3-30B-ASS A3B-Instruct-2500 con-thinking-modee produces answers that are significantly shorter than the other models.

---

Published as a conference paper at ICLR 2026
1amm-311--ee N-Nestrronn Tempenaure Turknn
Liama-31 Menetror--Bee Tapo
Liara Merr surretron--a $1.01
Liaru-Neradinn Caan---eetttr---
51.39
03.07
50
DRIN0 Score
$3.00
01
93.12
62.76
BRUNN Scan
googo Flanord
50
60
63.05
44.50
ma.0
45.46
62.01
42.18
Semperature (Tp-pp 1.0)
(lemperature 0.00
C4
Temperature (Top-p
52
1.0)
(Temperature
Expart-Cuided Tuning
AutoDeco
AutoOecc Expert Gulddd Tuning
AutaDece Expert-Guioe Guided Tuning
(a) Maamm-Nemottoo-8BB with AutoDeco.
31.8911 Owen Bercenture Terig
5a. 54.15
63.64
1410
$8.25
BRUNND Seore
$4.50
P20 VoLn
47
00
40.06
4127
47.66
47.41
49.38
48.86
64,70
40,00
40.17
42,46
20.00
Temperature TTp-pp
Top-p (Tempereture
64
Terperrture (op--
32
Top (Temperature 0.6)
Espert-Quided luning
Turing
Autolaco Expart-Guided luning
(b) R1-Dissill-Qwen-7B with Autoleco.
Figure 3: Expert-Guided Tuning Comparison with Search Interval of 0.1. Temperature adjusted first (setting top-p to 1.0), and the selection is made based the best performance of temperature to conduct the search for top-p. AutoDeco achieves competitive performance without requiring any prior empirical tuning or domain-specific expert knowledge.
Consequently, the sensitivity of task accuracy to variations in sampling parameters is substantially lower, a trend that is further demonstrated by the results in Table 2
Out-of-Domain Generalization. More strikingly, despite being trained exclusively on mathe- matical reasoning, AutoDeco demonstrates powerful zero-shot generalization to diverse set of out-of-domain tasks (Table consistently secures the highest average scores across general QA, code generation, and instruction following. This strong performance reveals two interesting patterns.
First, the magnitude of improvement remarkably consistent across domains. For example, on RI-Distill-Qwen-7BB AutoDeco improves the average score general tasks by 2.8 points over Default Sampling and Greedy Search a gain even matching that seen in the math domain. This suggests that the benefits of dynamic decoding fundamental and not tied to a specific task type.
Second, AutoDeco shows an ability to dynamically balance deterministic and stochastic strategies. On general tasks, Default Sampling not always better than Greedy Search (e.g., on 8B for GPQA-Diamon and IFEval). In these cases, AutoDeco learns to predict more deterministic, low-temperature parameters, allowing match exceed the performance of the stronger greedy baseline. Conversely, when stochasticity is beneficial, it raises the temperature to outperform Default Sampling.
The above findings suggest that AutoDeco is not simply learning "what" to generate, but rather the fundamental "meta-skill how" to generate text effectively. By training on high-signal domain like mathematics, it learns universal principles for balancing exploration and exploitation. We also show that AutoDeco has the ability adapt different task requirements (Stability vs. Creativity) in Appendix
9 We will further discuss this in Sec. Sec.3.3), and this finding challenges the conventional assumption that adaptive decoding requires broad, task-matched supervision, and instead points toward a more efficient, modular paradigm for real "end-to-end" controllable generation.
Pass @1 Performance. Some recent works (Yue et al. 2025 Chen et al. 2025) have highlighted potential trade-off in the training of reasoning models, where achieving superb pass @ accuracy can come at the expense of performance on pass @ (for > 1). To investigate this, we present an extended evaluation of method pass @ l (R: 16, 32, 64) accuracies.
7

---

Published as a conference paper at ICLR 2026
With encouraging results, find that the absolute improvements delivered by AutoDeco at higher k-values are consistent with, and at times even slightly greater than, those observed at pass 1. We show the numerical results in the Appendix[]
is important to note that for any given model. pass@ accuracy is inherently much higher than pass@ I accuracy. Consequently, it is obvious that securing absolute performance gains becomes substantially more difficult, and a similar absolute improvement at pass @66 translates to much larger relative error reduction, compared to pass @ I. For example, on the OpenAl-GPT-OSS-200 model, we observe that the performance gains from AutoDeco are consistent across different A: values in pass @ evaluations. More importantly, this consistent absolute gain translates to a significantly larger impact in higher-accuracy (when k is large) scenarios. The relative error reduction dramatically increases from 3.2% at pass 30 % at pass 64. This demonstrates that as the task becomes easier for the baseline model (i.e., the error rate decreases at high k:), the performance gains from our method become even more significant.
Comparison with Expert-Guided Tuning. In real-world applications, developers often undertake a laborious tuning process to find task-specific, optimal static hyperparameters. To assess how AutoDeco compares this best-case scenario, we simulate an expert with an unfair advantage: access to test-set oracle. As shown in Figure we first perform fine-grained search to find the optimal static temperature on the test set, and then, using that temperature, find the optimal top-p. This process represents the practical upper bound for any static decoding strategy.
The results are striking. AutoDeco's 's single-pass performance is nearly identical to this oracle-tuned baseline, with the performance gap consistently less than one point across all models and datasets. Given that the Expert- Guided Tuning relies on "hacking the test set", a process impossible in any real-world scenario where the test data is unknown, we can confidently assert that AutoDeco is effectively superior to any feasible expert-tuning strategy in practice.
Furthermore, the figure highlights the fundamental limitation of static decoding: parameters are extremely task-dependent. For instance, Slama-Nemotron requires drastically different settings for BRUMO25 (f = 0.8, P = 0.6) versus GPQA- Diamond (T' = 0.3, P = 0.6). However, in real-world scenarios, model developer has way switch hyperparameters based on the user's 's query type. AutoDeco elegantly solves this problem. By achieving near-oracle performance automatically and on-the-fly for any task, it provides the optimal and, frankly, only practical solution for developers seeking robust, high-performance generation across diverse user inputs.
the optimal hyper-
Compatibility with Advanced Decoding Algorithms. In practice, using only the model for next- token prediction might be inefficient. For industrial and production-level deployment and usage, speculative decoding is a very common and advanced technique, such as multi-token prediction (MTP) (DeepSeek-Al Al 2024) that is widely used in the state-of-the-art models of various fami-
lies OwenTeem 2025)
Our AutoDeco is fully compatible with speculative decoding mechanisms:
Theoretically, speculative decoding (such as MTP, which is currently applied in production-level models like DeepSeek-V3.1 Qwen3-Next) is fundamentally still of the form of autoregression. Sampling parameters such as Temperature and Top-p are widely and continuously employed within the speculative decoding framework to control the diversity and quality of the final accepted tokens.
To empirically confirm this scamless integration, conducted MTP experiment the advanced LLM DeepSeck-V3.11 -3.r--ernns.. It shows that our method does not require any specific adaptation and will not break the process of speculative decoding. (Please refer to Appendix D for detailed results.)
Ablation Study. A natural question is what role the temperature and top-p heads play individually. To isolate their effects, we evaluate on AIME using R RI-Dissill--wen-7B to conduct an ablation study, with the results presented in Figure The most striking finding is the remarkable effectiveness of each component in isolation. Using either the temperature head or the top-p head alone achieves an average performance gain of approximately 3-3.5 absolute points over the Default Sampling baseline.
8

---

Published as a conference paper at ICLR 2026
Table FLOPs, Memory Usage and latency (Ik tokens) across various prompt length for RI-Distill- Qwen-7B with/without temp head and top-p head.
Metries
Method
2k
8k
16k
24k
Default Sampling
2.89e+13 13
4.34e+13
7.23e+13
13.03e+13
24.61e+13
36.19e+13
FLOPs
AutoDeco (Ours)
2.89e+13
4.34e+13
7.24c+13
13.03e+13
24.62e+13
36.20e+13
Default Sampling
18.23
18.86
18.93
19.72
22.11
25.76
Latency (8)
AutoDeco (Ours)
18.84
19.10
19.43
20.03
22.36
26.05
Default Sampling
15546
16032
17130
19098
23182
27262
Memory (MB)
AutoDeco (Ours)
15550
16036
17134
19102
23183
27266
This result is highly significant. It demonstrates that sub- stantial improvements decoding do not require a sophis- ticated architecture. A single, lightweight prediction head sufficient to dramatically outperform standard static decoding methods.
48-
AIME Score
course, while each head is powerful on its own, our results also confirm that the full AutoDeco model, with
both heads, yields the best performance. They provide complementary benefits, allowing for even finer-grained control over the generation process to achieve optimal results.
Defiutt
AutoDecc bump AutoDooo top
AutoCeco
Figure 4: Ablation study AutoDeco architecture designs. Joint optimization achieves the highest AIME Score.
3.2.2 EFFICIENCY
critical advantage of AutoDeco is its computational efficiency. To quantify this, evaluated its overhead against Default Sampling across three key metrics, with results summarized Table
The analysis shows that the additional computational burden is minimal. The FLOPs virtually identical to the baseline, and the memory footprint increases by mere 4 MB, an insignificant amount for modern hardware. The impact latency is also negligible. This overhead remains consistently low regardless of prompt length, adding a consistent overhead of 0.29-0.6 s/k tokens, which translates an average relative increase of just 1.7%.
These results empirically validate that AutoDeco is lightweight enhancement, When considering the substantial performance gains and the convenience automatic, task-agnostic hyperparameter tuning demonstrated in Sec. I.2.1 this minor computational cost becomes trivial. AutoDeco thus presents highly practical solution, offering significant benefits for negligible price.
The analysis regarding training efficiency can be found in the Appendix g
3.3 EMERGENT CONTROL OF DECODING VIA NATURAL LANGUAGE
Beyond outperforming static methods, we observe that AutoDeco acquires a crucial capability: learns map abstract, high-level commands (such instructions for diversity certainty) directly control enables the model to
to its internal decoding parameters. This learned instruction- dynamically respond user intent regarding the desired generation style, marking a significant step towards truly end-to-end and controllable generation. For this phenomenon, we conducted an in-depth evaluation and discussion, which are detailed in the Appendix 7
RELATED WORKS
The process of generating text from a language model, known as decoding, critical step that significantly influences the quality of the Wutn et al. 2025 Shi et al. 2024). One directly related work is Adaptive Decoding (Dhuliawala et al. (2024). which also introduces learned decoding head. Their method focuses predicting temperature and utilizes reinforcement learning (RL) for training. Our work differs two main aspects: 1) we learn to control both temperature and top-p. and 2) employ fully differentiable pipeline that allows for direct "end-to-end" training from

---

Published as a conference paper at ICLR 2026
the next-token prediction loss, instead of RL. The other existing decoding strategies can be broadly categorized into deterministic, stochastic sampling, and model-based approaches, most of which traditionally rely static, predefined configurations.
Deterministic Decoding. Deterministic methods produce a single, reproducible output for a given input. The most fundamental of these is Greedy Search, which selects the token with the highest probability at each step. Another classic one beam search, which maintains "beam" of most probable partial sequences to explore a larger search space (Sutskever et al.] 2014: Gravess 2013).
However, both of them are known favor dull, high-frequency phrases (Vijayakumar et al. 2016).
this results in their good performance on Machine Translation and QA tasks, but not suitable for open-ended generation tasks. A more recent line of deterministic methods, Contrastive Search(Su & Collier 2022 Su et al. 2022), directly optimizes for open-ended generation quality by penalizing tokens that are too similar to previous tokens, effectively mitigating the degeneration problem.
Stochastic Sampling. To inject diversity, stochastic sampling methods are essential. These methods sample from the model's output probability distribution, which is typically modulated by some hyperparameters. However, unrestricted sampling can produce incoherent text. To counter this, truncation methods were developed. Top-K sampling(Fan et al. 2018) restricts the sampling pool to the & most likely tokens, while the more adaptive Nucleus Sampling (oootsamon et al.) selects the smallest set of tokens whose cumulative probability exceeds a threshold Despite their power, as our introduction highlights, finding the optimal configuration for these hyperparameters is a non-trivial, task-dependen manual process (Shi et al.) 2024).
Model-Based Decoding, To gain more fine-grained control over generation, third category of methods modifies the model's output distribution using external signals or auxiliary models. Early examples include Plug-and-Play Language Models, which leverage attribute models to steer generation towards desired topies (Dathathri et al.). More recently, Contrastive Decoding uses a smaller "amateur" model steer larger "expert" model away from generic text (Li al. 2023; Chuang et al.) 2023). Similarly, Speculative Decoding utilizes a faster "draft" model to generate sequences of tokens that are then verified by the larger model, significantly accelerating inference (Leviathan et al. 2023 Chen et 2023)). There is also an art to verification methods (Liu ctt al., 2025). While they are effective, they still operate under a fixed algorithmic framework: the choice of the "guidance model" itself acts another form of hyperparameter. For example, in contrastive decoding and speculative decoding, the authors suggest that using a smaller LM of the same architecture the guidance model yields the best results.
Despite this rich landscape of research, a fundamental limitation persists: all these methods rely on static decoding strategy. Whether it's fixed algorithm (like Beam Search) or a fixed set of hyperparameters, this "one-size-fits-all approach is inherently suboptimal. In contrast, AutoDeco proposes paradigm shift. Instead of relying on fixed hyperparameters or predefined heuristics, empower the model to dynamically control its own stochasticity at each generation step.
5 CONCLUSION AND FUTURE WORK
In this work, we challenged that the "end-to-end" label for LLM is a misnomer. We introduced AutoDeco, a truly "end-to-end" architecture that empowers models to dynamically control their own decoding strategy. By learning to predict token-level temperature and top-p values, AutoDeco transforms decoding from manual, static process into dynamic, self-regulating pipeline.
Our extensive experiments reveal three key contributions. First, AutoDeco demonstrates remark- able generalization, consistently outperforming standard decoding methods across diverse models and tasks, even matching oracle-tuned baselines without any task-specific tuning. Second, this performance is achieved with negligible computational overhead, making it practical, drop-in in cnhancement for any transformer- based model. Additionally, we discovered an intriguing emergent capability: AutoDeco learss to interpret natural language commands to steer its own generation style, a foundational step towards more intuitive human-Al interaction.
Future Work. Our immediate future work involves jointly training the base model with AutoDeco. We believe this will address current limitations like imprecise prompt-based control and data biases-bor likely consequences of frozen backbone- thereby enabling more granular control over generation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported in part by the Guangdong Basic and Applied Basic Research Foundation under Grant No. 2025A1515012968, in part by Shenzhen Science and Technology Program under Grant No. JCYJ200408111133 13550000,,, in part by the National Natural Science Foundation of China under Grant No. 62001412, part by Shenzhen Stability Science Program 2023, in part by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 2022B1211 2010000),)) and in part by the Shenzhen Key Lab Crowd Intelligence Empowered Low-Carbon Energy Network
(Grant No. ZDSYS20220006000000000
STATEMENTS
ETHICS STATEMENT
The authors of this paper have read and agree to abide by the ICLR Code of Ethics. We believe that this work does not raise any significant ethical concerns. Our research did not involve experiments with human subjects. nor did it process sensitive personal data. All datasets used in study publicly available. We foresee no direct negative societal impacts from the methods and potential applications presented in this work.
REPRODUCIBILITY STATEMENT
We committed to ensuring the reproducibility of our research. We have provided comprehensive experimental details in the main paper and Appendix (6] including dataset preprocessing proce- dures, model architecture specifications, full training details, and all hyperparameter configurations. Furthermore, we will make our source code and model checkpoints publicly available.
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss- & gpt-oss-20b model card. arXiv preprint ar arXiv:2508.1099 2025.
Mislay Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Math- arena: Evaluating ilms uncontaminated math competitions, February 2025. URL https //matharena ai/
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemotron: Efficient reasoning models. arXiv preprini (5Xx::2505500999 2025.
ByteDance-SSed Seed. Beyondaime: Advancing math reasoning evaluation beyond high school olympiads. co/
2025.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Sean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013/8, 2023.
Zhipeng Chen, Xiaobo Qin, Youbin Wu. Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass @k training for adaptively balancing exploration and exploitation of large reasoning models, 2025. URL hittps://art wwwpxiv..org/aaee wwtp://ggg.g.....///////
10751
Yung-Sung Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2023.
Sumanth Dathathri. Andrea Madotto, Janice Lan, Jane Hung. Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference Learning Representations.
11

---

Published as a conference paper at ICLR 2026
DeepSeek-AL. Deepseek-v3 technical report, 2024. /ttL:t//////xx..or//a//as//// 19437
Shehzaad Dhuliawala, Ilia Kulikov, Ping Yu, Asli Celikyilmaz, Jason Weston, Sainbayar Sukhbaatar, and Jack Lanchantin. Adaptive decoding via latent preference optimization. arXiv preprint arXiv:2411.09661, 2024.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 889-898, 2018.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint 130.v0300.00850 2013.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501. 122.4 2025.
Zhiwei He, Tian Liang. Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, al. Deepmath-103k: A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504. 2025.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint, 2024.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pp. 19274-19286. PMLR, 2023.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In The 61st Annual Meeting The Association For Computational Linguisties, 2023.
Xiaoyuan Liu, Tian Liang, Zhiwei He, Jiahao Xu, Wenxuan Wang, Pinjia He, Zhaopeng Tu, Haitao Mi, and Dong Yu. Trust, but verify: selfiveriiicatioon approach to reinforcement learning with verifiable rewards. 2025. URL httppp/.//aaai.....//a/////....1
QwenTeam. Qwen3 technical report, 2025, URL http:/ts///aax.x.or/////a//
09388
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. GPQA: graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL Bttl:ttttt////eereeee/ Bet/forum?id=T1e
Chufan Shi, Haoran Yang. Deng Cai, Zhisong Zhang. Yifan Wang. Yujiu Yang. and Wai Lam. A thorough examination of decoding methods in the era of llms. arXiv preprint 2rXiv:2402.069255 2024.
Yixuan Su and Nigel Collier. Contrastive search is what you need for neural text generation, arXiv preprint arXiv:2210.141: 2022.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. Advances in Neural Information Processing Systems, 35: 11548-21566 2022.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv: 1610000000 2016.
12

---

Published as a conference paper at ICLR 2026
Haoran Wang, Xiongxiao Xu, Philip S Yu, and Kai Shu. Beyond tokens: A survey de- coding methods for large language models and large vision-language models. April 2025. doi: http:///70ccccmii111777900000000
ftcchxxlv.1449330..0 007.5999000.07799966/
Noah Wang, Zy Peng. Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. Rolellm: Benchmarking, eliciting. and enhancing role-playing abilities of large language models. In Findings of the Association for Computational Linguistics: ACI. 2024, pp. 14743-14777, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574. 2024b.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: reinforcement learning perspective with reward rectification, 2025.
05629
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity Ilms beyond the base model? arXiv preprint arXiv:2504. 00044.13832 13837, 2025.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follooinng evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
13

---

Published as a conference paper at ICLR 2026
CONTENTS OF THE PAPER
Introduction
2
AutoDece
2.1
Training Strategy
2.2
Inference: Dynamic Decoding
3
Experiments
3.1
Experimental Setup
3.2
Main Results
3
Performance
3.2.2
Efficiency/
3.3
Emergent Control of Decoding via Natural Language
4
Related Works
Conclusion and Future Work
10
Experimental Setup
14
In-depth discussion Instruction-based decoding control
15
8
Supplementary Discussion of Efficiency
16
9
Supplementary Experimental Results
16
10
Declaration LLM usage
18
EXPERIMENTAL SETUP
Training. For the training of all models with AutoDeco framework, employed consistent hyperparameter configuration to ensure fair comparison. To efficiently manage memory and scale our experiments, utilized the DeepSpeed library with the ZeRO Stage optimization. The specific training settings are detailed below:
Training Framework:
DeepSpeed (ZeRO Stage 3) for DeepSeek-RI-De RI-Distill-Qwen- and Megatron for the MoE model Qwen3-30B-A3B Instruct-2507, -35ee33-23BBA22BBB- Thinking- OpenAA-GPT--ss-22000 OpenAL-GPT-Oss- 120B, and DeepSeek-V3.
Hardware: 32 GPUs for 1...teek--V..-Temmiie 8 GPUs for the others.
Batch Size: A per-device batch size of with gradient accumulation steps, resulting in an effective global batch size of 32.
Optimizer: AdamW
Learning Rate: 10-6
Max Token Length: 32768.
Decay Steepness a: 30.
For each task, we calculated the Pass@1 through oversampling (16 times). To ensure the results are solid, do 8 runs each experiment with different seeds. For all the tasks, our maximum generation length is 32768.
Datasets. Our experimental configuration is detailed as follows:
MMMU-Pro: We used a comprehensive and evenly distributed "lite" subset for evaluation to ensure a balanced assessment across all subject areas.
14

---

Published as a conference paper at ICLR 2026
Prompi: Design unique drink for each emotion (joy, anger, sadncss,
Prompt: :... I hope the answers be innovative and diverse.
Prompt :... hope the answers can be as certain as possible.
1.0
Valte
0.
appexcxce, sawery expelicne...
oppertixe, sonery experience....
apperonce simnry experienoe
Figure 5: An Emergent Phenomenon. This figure shows the token-level T/P predictions for the same prompt under three conditions, observed without any targeted training. (Left) Baseline: The model's default dynamic / values. (Middle) High-Diversity Command: The model sponta- neously elevates its T/P predictions. (Right) Low-Diversity Command: The model spontaneously suppresses its / P predictions.
Table 4: Quantitative Impact of Diversity Commands on Predicted Decoding Parameters (N=100).
Command
Avg. Temp.
A Temp.
Consistency (T)
Avg. top-p
top-p
Consistency (P)
Baseline (No Cmd)
0.59
0.84
Low Diversity
0.48
0.11 0.07
99% 90%
0.75
0.09 0.05
99%
High Diversity
0.66
0.89
96 %
LiveCodeBench: The V6 version of the dataset was used. The evaluation window for this benchmark initiated September 1, 2023, and included all subsequent data.
Others: All the others selected datasets were processed using their full sets.
7
IN-DEPTH DISCUSSION ON INSTRUCTION-BASHR DECODING CONTROL
The instruction-basee decoding control capability is indeed an unexpected yet interesting phenomenon discovered during experiments.
Figure provides a qualitative demonstration of this capability. On the left, a creative prompt to "Design unique drink for each emotion" elicits dynamic but baseline set of temperature and top-p values (solid lines). In the middle panel, when we append the command, "I hope the answers can be more innovative and diverse," the model's response immediate and visible: the predicted T and P values (dotted lines) are consistently elevated above the baseline, effectively "turning up" its own creativity. Conversely, on the right, the command "I hope the answers can be certain as possible" causes the model to autonomously suppress its T and P predictions, "turning down" its randomness to favor more deterministic outputs. To our knowledge, this the first demonstration of an LLM directly translating natural language intent for creativity and certainty into its internal sampling parameters on a token-by-token basis.
To verify that this is not an anecdotal result, we conducted large-scale quantitative analysis. We prepended commands for "high" or "low" diversity to a set of 100 questions and aggregated the results, presented in Table The data confirms the effect is systematic and robust. The "low diversity" command prompted substantial drop in average temperature from 0.59 to 0.48 with remarkable 99% consistency across all questions. The "high diversity" command triggered similarly consistent increase in both temperature and top-p, proving that the model has learned a generalizable mapping from abstract language to its internal generation mechanies.
However, we do not yet have a conclusive understanding of this phenomenon theoretically. We will continue advance this.
15

---

Published as a conference paper at ICLR 2026
2.00
GPT-Oss-200
1.75
Lame Nernotron--8
1.50
R1-Distil-Qwen-76
1.25
0.50
0.25
0.00
200
250
Training Stop
Figure 6: AutoDeco's training curves on all models. Training loss curve across models. The loss converges effectively, indicating resource- frrenndly training of AutoDeco.
Table Pass 16 accuracy on mathematical reasoning benchmarks. Comparing Default Sampling with the AutoDeco method.
Model
Method
AIME
BRUM025
HMM125
BeyondAIME:
Average
Default Sampling AutoDeco (Ours)
77.93
87.98
61.98
57.36
71.31
Jlama-Nemotron-88
80.64
87.63
65.76
60.11
73.54
Default Sampling
71.50
78.16
51.84
54.21
63.93
RI-Distill-Qwen.
AutoDeco (Ours)
74.04
80.08
58.60
54.88
66.90
Default Sumpling AutoDeco (Ours)
88.05
90.81
66.57
69.87
78.83
Qwen3-30B-Instuuct
88.94
91.15
66.39
69.55
79.01
Default Sampling
91.44
93.63
85.94
70.55
85.39
Qwen3-235B-TThinking
AutoDeco (Ours)
92.00
95.81
85.69
70.00
85.88
Default Sampling
91.42
97.10
86.37
77.32
88.05
OpenAI-GPT-OSS-2220
AutoDeco (Ours)
91.48
97.54
88.43
79.05
89.13
8 SUPPLEMENTARY DISCUSSION OF EFFICIENCY
Training Efficiency. Given AutoDeco's superior decoding performance and minimal deployment overhead, natural question arises: What is the cost of endowing language model with this adaptive decoding capability? Remarkably, the answer is negligible. AutoDeco is resource-efficien, general- purpose solution for adaptive decoding optimization. Our experiments reveal two key practical advantages:
Label-free supervision: AutoDeco eliminates the need to pre-compute or invoke any external optimization modules to generate supervision signals (e.g., temperature top-p labels) for fine-tuning.
Data efficiency: We show the training curves of all models in Figure 6 and AutoDeco achieves strong performance within about only 6K training samples and 400 steps, making it effortlessly be integrated into any pre-trained L.MSs.
9 SUPPLEMENTARY EXPERIMENTAL RESULTS
Adaptability Stability and Creativity. We selected the eight benchmarks in the main paper primarily because they are widely recognized, standard benchmarks that are frequently featured in the technical reports of leading LLMs.. Furthermore, AutoDeco has proven effective in common open-ended generation scenarios like role-playing, enabling to eliminate manual decoding without performance degradation. To demonstrate this, we also evaluated AutoDeco's 's effectiveness on RoleLLM (Wang al. 2024a) benchmark:
Model: C0ren3-30B-A3B A3B-IInstruct-2007
16

---

Published as a conference paper at ICLR 2026
Table Pass @ 32 accuracy on mathematical reasoning benchmarks. Comparing Default Sampling with the AutoDeco method.
Model
Method
AIME
BRUMO25
HMMT25
BeyondAIME
Average
Default Sampling
81.04
91.09
68.89
61.19
75.55
Llama-Nemotron-8B
AutoDeco (Ours)
84.15
91.57
71.54
64.72
78.00
Default Sampling
74.24
80,91
60.65
59.59
68.85
RI RI-Dishill-Qwen-BB
AutoDeco (Ours)
77.12
81.84
66.88
59.88
71.43
Default Sampling
90.22
93.33
71.03
74.37
82.23
Qwen3-30B-Insee
AutoDeco (Ours)
90.40
94.35
70.88
74.00
82.41
Default Sampling
91.80
95.15
88.96
73.50
87.35
Dwen3-235B-Thinnknng
AutoDeco (Ours)
92.20
97.32
89.00
73.42
87.99
Default Sampling
92.91
98.
89.19
81.20
90.36
OpenAI-GPT-OSS-202
AutoDeco (Ours)
93.55
99.14
91.95
84.36
92.25
Table 7: P accuracy on mathematical reasoning benchmarks. Comparing Default Sampling with the AutoDeco method.
Model
Method
AIME
BRUMO25
HMMT25
BeyondAIME
Average
Default Sampling
83.53
93.02
76.11
65.14
79.45
Llama-Nemotrrn-SBB
AutoDeco (Ours)
87.80
92.41
74.79
68.87
80.97
Default Sampling
77.75
81.44
66.25
65.48
72.73
RI-Distill-Qwen-7B
AutoDeco (Ours)
80.64
83.93
74.50
65.96
76.26
Default Sampling
91.15
95.54
74.31
78.65
84.91
Qwen3-30B-Inee
AutoDeco (Ours)
92.30
97.94
74.53
77.94
85.68
Default Sampling
91.67
96.26
91.24
77.19
89.09
Qwen2-235B-Thinking
AutoDeco (Ours)
92.50
98.33
90.64
77.02
89.62
OpenAl-GPT-OSS-20R
Default Sampling AutoDeco (Ours)
94.10
99.17
89.97
85.30
92.1
95.22
100
93.97
88.81
94,50
Table 8: The WinRate of AutoDeco / fixed decoding parameters RoleLLM.
WinRate
T 0, O,Top-p=0
T' 0.8, Top pp = 0.95
0.9, Top p= 0.95
AutoDeco
54.65/45.35
54.32 /45.68
53.82/44 /3.11/
52.21 /47.79
Table 9: The average predictions AutoDeco in different tasks.
AIME
Creative Tasks
AutoDeco
T = 0.61, 0.93 f 1.18, 0.88
Evaluation: We employed the advanced Claude-Sonnet-3.7 to LLM Judge to compare generations using AutoDeco's dynamic predictions against generations using several optimized fixed-parameter settings.
As demonstrated Table AutoDeco consistently outperforms all optimized -xed-paaammtte settings in these head-to-head comparisons.
We conducted an additional comparison on Qwen3-235B-A22B -a..222--ihinn between two distinct generation regimes: Mathematical Reasoning (which favors relatively high consistency/stabiiit)) and creative task (which benefits from higher variability/crcatiiityy..
Math Results: Average results of AIME sampling.
Creative Tasks: Average results of 20 distinct prompts. (e.g., "Please write story about cat" and "Help me plan trip to Europe.")
17

---

Published as a conference paper at ICLR 2026
Table 10: @ass@ aceuracy on Synthetic-100. AutoDeco consistently outperforms both Greedy
Search and Default Sampling methods
Model
Greedy Search
Default Sampling
AutoDeco (Ours)
DeepSeck-671B
56.94
57.76
58.47
DeepSeck-685B (with MTP)
57.12
57.89
58.40
The average predictions shown in Table demonstrates the ability of AutoDeco to adapt to different task requirements.
Pass@ Performance. by AutoDeco higher k-values are consistent with, and times even slightly greater than, those observed at pass@1 1.
With encouraging results, we find that the absolute improvements delivered
important to note that for any given model, pass@ accuracy is inherently much higher than pass@ I accuracy. Consequently, it is obvious that securing absolute performance gains becomes substantially more difficult, and a similar absolute improvement at pass @64 translates to much larger relative error reduction, compared to pass @ 1. For example, on the OpenAl-GPT-OSS-2000 model, we observe that performance gains from AutoDeco are consistent across different A values in pass @ l evaluations. More importantly, this consistent absolute gain translates to significantly larger impact in higher-accuracy (when is large) scenarios. The relative error reduction dramatically increases from 3.2% % at pass @ 30 % pass aass@64, @64. This demonstrates that the task becomes easier for the baseline model (i.e., the error rate decreases at high A:), the performance gains from our method become even more significant.
DeepSeek-Vas V3.1-Terminus The performance gains achieved on production-level models are our main focus. We have also conducted performance evaluations on industrial-grade SOTA large models such as DeepSeek-VV.. V3.1 V3.1-Terriiuu-671 (DeepSeek-A] 2024).
Due to the deployment and sampling pressure of the ultra-large thinking model, we combined all the evaluated benchmarks an average basis and created tiny synthetic benchmark consisting of 100 evaluation questions (named Synthetic-100) for evaluating AutoDeco.
Our method is fully compatible and integrates seamlessly with speculative decoding mechanisms. Taking MTP as an example, it is fundamentally still of the form of autoregression. Sampling parameters such as Temperature and Top-p are widely and continuously employed within it control the diversity and quality of the final accepted tokens.
To empirically confirm this seamless integration, conducted MTP experiment the advanced LLM DeepSeek-V33 -3. 13e.mnurms and demonstrate the superiority of AutoDeco in Table I0] It proves that using AutoDeco will not disrupt the MTP process of the model. Critically, find that the sampling parameters temperature and top-p are still of vital importance and do not diminish as the size and capability of the model increase. AutoDeco help users achieve descent performance with the least effort.
10 DECLARATION OF LLM USAGE
The LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research.
18

---

Published as a conference paper at ICLR 2026
Table Pass@) accuracy on mathematical reasoning benchmarks. AutoDeco consistently outper- forms both Greedy Search and Default Sampling methods across various models.
Method
AIME
BRUMO25
HMMT225
BeyondAIME
Average
Greedy Scarch
$5.00
60.00
26.67
37,00
44.67
Jlama-Nemotron---e
Default Sampling
51.24=1.31
60.68+1.21
30,89=11.29
32.65=+0.51
43.86
AutoDeco (Ours)
5..07-1.12
63.23++1044
34.37=1.33
34.51+0.54
47.05
Greedy Search
40.00
43.33
20.00
24.00
31.83
I--issill-Qwen:
Default Sampling
43.62-1.06
11.38.11.08
22.50.10.88
24.87.10.72
35.59
AutoDeco (Ours)
47.57-1.20
53.93++.31
2..46=0.76
26,98-+0.75
38.24
Greedy Search
70.00
36.67
45.00
54.17
Qwen3-30B-Inse Instruccc
Default Sampling
66.95_1.44
70.99111.1
44.69-0.81
47.02.10.33
57.41
AutoDeco (Ours)
68.22-0.85
71.36:10.83
43.15=11110
47.15
57.47
Greedy Search
80.00
83.33
63,33
$3,00
69.92
(wen3-235B-Thinkke
Default Sampling
80.76=0.43
82.40=0.669
63.41=1.20
50.05+00.56
69.16
AutoDeco (Ours)
82.79-0.7 0.72
84.14+11.27
66.07=1.100
51.88+0062
71.22
Greedy Search
73.33
56.67
37.00
55.92
OpenAl--PT-OSS--00
Default Sampling
69.65-1.33
15.50:11.11
50,91-22.76
47.11-10.23
60.79
AutoDeco (Ours)
72.72-1.22
75.21:10883
53,00=11002
47.32:10.37
62.06
Greedy Search
78.50
66.67
66.67
50,00
65.46
OpenAI-GPT-OSS--200
Default Sampling
78.33-0.72
78.23+00991
63.17
53.45-+0.799
68.30
AutoDeco (Ours)
78.65-0.46
78.55::1.35
62.90=1.19
53.24:10.63
68.34
Table 12: Pass@1 accuracy on general-domain benchmarks. AutoDeco shows exciting generalization performance General QA, Code Generation, and Instruction Following tasks.
Model
Method
GPQA-Diamond
MMLU-Pro
LiveCodeBenchVa
Average
Greedy Search
51.01
52.00
19.17
48.43
Llama-Nemotroa-88
Default Sampling
44.93
54.00
21.22
65.25
46.35
(Ours)
55.64
49.72
Greedy Search
37.87
47.20
14.98
32.90
33.24
RI -I-Distill-Qwen-TB
Default Sampling
47.41
47.65
15.40
32.35
35.70
AutoDeco (Ours)
48.91
50.75
15.46
37.26
Greedy Seurch
65.86
78.00
47.75
83.73
68.84
Owen3-30B-A31B. A.B0-------sm
Default Sampling
69.82
76.25
48.52
81.52
69.03
AutoDeco (Ours)
69.96
78.38
49,80
82.81
70.24
Greedy Search
77.78
81.00
77.25
31.98
67. .0
Owen3-2358-Thinle
Default Sampling
80.81
80.33
77.47
31.61
67.56
AutoDeco (Ours)
81.13
79.20
78.10
32.90
67.83
Greedy Search
59.60
67.00
69.69
29.
$6.56
OpenAl-GPTTOSS--208
Default Sampling
68.00
70.15
AstoDeco (Ours)
66.48
69.12
71.25
30.84
59.42
Greody Search
71.21
75.00
73.93
32,72
63.22
OpenAL-GPTO0S--1000
Default Sampling
70.20
76.00
74.19
63.23
AutoDeco (Ours)
70.24
76.50
74.30
32.90
63.49
19


Reviews:
[{"review_id": "KQ6q6h7W7o", "reviewer": "Reviewer_XeNy", "summary": "The paper introduces AutoDeco, a novel architecture designed to make large language models (LLMs) truly end-to-end by eliminating the need for manual decoding hyperparameter tuning (e.g., temperature, top-p). Traditional decoding strategies rely on static, hand-tuned parameters that must be manually adjusted for different tasks or even different parts of a generation. AutoDeco addresses this by augmenting the transformer with lightweight “decoding heads” that dynamically predict temperature and top-p values at every generation step.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Novelty and Conceptual Contribution: The paper identifies and addresses a fundamental yet overlooked bottleneck in LLM deployment, the manual, non-differentiable decoding process.\n\nAutoDeco reframes decoding as a learnable and parametric component within the model itself, offering a principled step toward fully end-to-end generation.", "weaknesses": "1. While the emergent instruction-following behavior is a highlight, the explanation for why this arises is mostly empirical. A deeper analysis (e.g., probing whether linguistic cues correlate with latent space adjustments) would strengthen this claim.\n\n\n2. Most benchmarks are reasoning or QA-oriented. It would be valuable to test AutoDeco on creative writing, dialogue, or long-form generation, where decoding choices play a larger role. Human evaluation or qualitative examples of improved text quality would strengthen the practical impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "miLQgCKUpt", "reviewer": "Reviewer_zLkm", "summary": "This work addresses an important practical issue in deploying large language models (LLMs): the need to manually tune decoding hyperparameters, which breaks the “end-to-end” workflow. To mitigate this, the authors propose AutoDeco, a method that augments standard transformer architectures with lightweight prediction heads that output context-dependent temperature and top-p values alongside next-token logits during decoding. Experiments across multiple benchmarks demonstrate that the approach can yield improved generation quality and adaptability.", "questions": "1. Can this framework be extended to other decoding parameters, such as top-k or repetition penalty?\n\n2. In practical deployments, how should developers express or configure user-level preferences (e.g., consistency vs. creativity)? More discussion or empirical analysis would clarify how the method adapts to varying application requirements, especially in relation to the concern raised above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem is timely and relevant. Removing manual tuning of decoding hyperparameters can significantly improve the practicality and usability of LLM-based systems.\n\n2. The pseudo-label generation strategy for supervision is clever and helps circumvent the lack of direct ground-truth hyperparameter labels.", "weaknesses": "1. A key concern is that the model is trained to further increase the likelihood of the reference text. Since both pre-training and downstream fine-tuning typically already optimize for the likelihood of the ground-truth sequence, this additional adjustment may risk overfitting or reduce robustness in more open-ended generation settings.\n\n2. While dynamic prediction of decoding hyperparameters is appealing, different applications may require different behavior. For example, customer support systems typically favor stability and consistency, whereas brainstorming or creative writing tasks may benefit from higher variability. It is unclear whether a single learned mechanism can generalize effectively across such diverse usage scenarios without explicit control, and it may limit adaptability when user preferences change.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q1oMiI0KXi", "reviewer": "Reviewer_tpjS", "summary": "The paper introduces a method to learn the sampling hyperparameters (temp, top-p) end to end via finetuning, avoiding manual task-specific tuning. They also identify and give ways to solve challenges that arise when trying to do this naively. In the end, they find they can basically match task-specific manual tuning (the oracle upper bound).", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The idea is interesting and to my knowledge novel. \n- The problem space is richer than meets the eye (eg. how do you get training data for supervised training is surprisingly nontrivial). \n- The results are fairly convincing (Fig2). \n- They show this doesn't hurt performance (Fig3).", "weaknesses": "- In practice, nobody does pure autoregression in real world LLM usage at production-scale. Everyone uses speculative decoding of some sort, and it's not clear to me whether this sampling scheme permits that or breaks it. I would want to see an explanation or formal argument/construction for how speculative decoding would work when the target model has an AutoDeco head to be convinced this would not break speculation. Because if it does, then in practice it will never be used, which would be a major limitation. I think it can be made to work, though, it just needs maybe some explanation of how it would all come together. \n- The idea and science is good, but writing is mediocre in my opinion. The contributions section repeats the second contribution twice, omitting I imagine the third contribution (presumably the \"emergent instruction-following\"), the plots are not consistently formatted well (axes in Fig4 impossible to read), the name \"AutoDeco\" is neither catchy nor informative in my opinion, etc. \n- The \"emergent ability\" is not as surprising as the authors seem to think it is. You're training on a base model that conditions on language by construction, so I would expect this behavior as the language conditioning affects the model representations fed into the AutoDeco head. Maybe de-emphasize this, or figure out an alternative framing? But this is minor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jiUcoIPKX1", "reviewer": "Reviewer_8WUM", "summary": "The paper introduces AutoDeco, a framework that aims to make language model decoding fully end-to-end by enabling models to predict their own decoding hyperparameters (temperature and top-p) dynamically at each generation step. The authors augment standard transformers with lightweight MLP heads that output token-specific decoding parameters, trained using pseudo-labels derived from optimization on ground-truth tokens. The approach claims to remove the need for manual or oracle-style hyperparameter tuning while maintaining negligible computational overhead.", "questions": "Please see the weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel framing of decoding as a learnable, differentiable process. Treating temperature and top-p as learnable functions of context rather than fixed hyperparameters is an appealing conceptual shift that aligns with recent trends toward self-regulating and adaptive LLM inference.\n2. Simple yet effective architecture. The addition of two MLP heads is computationally cheap and can be easily integrated into existing transformer architectures, which improves the practicality and reproducibility of the approach.\n3. The observation that the model can interpret natural-language modifiers like “low randomness” to adjust temperature/top-p values is intriguing and opens a new line of research in interpretable controllability.", "weaknesses": "1. My concern is that it is not clear to me how they obtain labels for training prediction of temperature and top_p value per token. It is unclear how the argmax over continuous T > 0 is solved, what constraints or search ranges are used, and how noise in logits affects these derived labels. It is better to provide a simple example to explain this process. \n2. The experiments only compare against Greedy and Default Sampling (and an oracle). Missing are modern decoding methods such as Contrastive Search [1], Contrastive Decoding [2].\n\n[1] https://arxiv.org/pdf/2210.14140\n[2] https://arxiv.org/abs/2309.09117", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)