ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
Authors: Feijiang Han
Xiaodong Yu
Jianheng Tang
Delip Rao
Weihua Du
Lyle Ungar
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Attention Mechanisms
Training-free Methods
Inference-time Optimization
Model Interpretability
Unsupervised Learning
Attention Sink
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce ZeroTuning, a training-free method that enhances LLM performance by tuning attention to the initial token, a simple yet powerful and universal control point.

Abstract: Token-level attention tuning -- a class of training-free methods including Post-hoc Attention Steering (PASTA) and Attention Calibration (ACT) -- has emerged as a promising approach for improving frozen LLMs via interpretable interventions. However, these methods rely on auxiliary heuristics to identify important task-specific tokens, which can introduce bias and limit applicability when token importance is ambiguous or when optimized kernels make attention maps inaccessible. We propose a simpler alternative: intervening only on the initial token (e.g., <BOS> in LLaMA). We theoretically show that adding lightweight biases to this token’s attention logits systematically shifts and reshapes downstream attention patterns -- an effect amplified by its natural role as an attention sink. Empirically, we find that this tuning can improve LLM performance and better elicit pretrained knowledge, with stronger effects in early layers and distinct scaling preferences across attention heads. Building on these findings, we introduce ZeroTuning, a training-free method that improves LLM performance by applying head-specific attention adjustments to the initial token, requiring no parameter updates. We present two variants: a supervised mode that calibrates on validation examples, and an unsupervised mode that directly minimizes output entropy. ZeroTuning requires no KV-cache or decoding changes and is kernel-agnostic (works with SDPA and FlashAttention). It requires only four lines of modification to standard \texttt{LlamaAttention} code, achieves gains across 15 datasets, and outperforms prior, more complex methods. For example, on Llama-3.1-8B, it yields relative improvements of 19.9% on classification, 4.5% on question answering, and 2.1% on dialogue. ZeroTuning also works out of the box with quantized inference and maintains its improvements as context length increases. Our work provides a lightweight tool for inference-time improvement, advancing both optimization and interpretability. Our code and runnable demo are available at https://anonymous.4open.science/r/ZeroTuning.

Published as a conference paper at ICLR 2026
ZEROTUNING: UNLOCKING THE INITIAL TOKEN'S POWER TO ENHANCE LARGE LANGUAGE MODELS WITHOUT TRAINING
Feijiang Han
Xiaodong Yu
Jianheng Tang² Lyle Ungar!
Delip Rao
Weihua Du
'University of Pennsylvania 2MD
*Peking University *Carnegie Mellon University
ABSTRACT
Token-level attention tuning a class of training-free methods including Post-hoc Attention Steering (PASTA) and Attention Calibration (ACT)-has emerged as a promising approach for improving frozen LLMs via interpretable interventions. However, these methods rely on auxiliary heuristics to identify "important" task- specific tokens, which can introduce bias and limit applicability when token im- portance is ambiguous or when optimized kernels make attention maps inaccessi- ble. We propose a simpler alternative: intervening only the initial token (e.g.. <BOS> in LLaMA). We theoretically show that adding lightweight biases to this token's attention logits systematically shifts and reshapes downstream attention patterns- an effect amplified by its natural role as an attention sink. Empiri- cally, we find that this tuning can improve LLM performance and better elicit pretrained knowledge, with stronger effects in early layers and distinct scaling preferences across attention heads. Building on these findings, we introduce Ze- roTTuning training-free free method that improves LLM performance by applying head-specific attention adjustments to the initial token, requiring no parameter updates. We present two variants: supervised mode that calibrates on valida- tion examples, and an unsupervised mode that directly minimizes output entropy. Zerinuning requires no KV-cache decoding changes and is kernel-agnostic (works with SDPA and FlashAttention). requires only four lines of modifi- cation to standard LlamaAttention code, achieves gains across 15 datasets, and outperforms prior, more complex methods. For example, on Llama-3. 3.1BBB it yields relative improvements of 19.9% classification, 4.5% question swering, and 2.1 1% on dialogue. Zero'l Tuning also works out of the box with quan- ZeroTuninn
tized inference and maintains its improvements as context length increases. Our work provides a lightweight tool for inference-time improvement, advancing both optimization and interpretability. Our code and runnable demo are available 4oper. sctence/r/Zetotun..
INTRODUCTION
Training-free ffre methods have been widely explored to enhance the performance of Large Language Models (LLMs) at inference time. Among these, token-level attention tuning has emerged as a par- ticularly promising direction, offering interpretable way to steer model behavior by modifying attention distributions without any parameter updates. Unlike fine- tuning (Hu et al. 2021 Dettmers al. (023) prompt engineering (Wei al. [022 Wang al. 2022 ?), which largely treat LLMs as black boxes, attention tuning provides a transparent mechanism for guiding the model's focus. Methods such as Post-hoe Attention Steering (PASTA (Zhang et al. 2023a). AutoPASTA (Zhang et al. 2024a)) and Attention Calibration (ACT) YYu et al. (024) have demonstrated the potential of this approach, in some cases outperforming prompting-bass
techniques on complex tasks such as open-domain question answering /Zhang et al. (024a). Similar principles have also been ap- plied to vision-language models to mitigate hallucinations by re-weighting attention toward image
tokens (Liu et al. 2024b; Zhu et al. 2024 Wei & Zhang 2024).
"Corresponding author. feijhan seahaa@ssess.uucn..dee

---

Published as a conference paper at ICLR 2026
However, the efficacy of these methods is fundamentally constrained by their reliance external, often heuristic, mechanisms for identifying task-specific "important" tokens. This dependency not only introduces the risk of bias (e.g., amplifying misleading cues) but also limits applicability when token importance is ambiguous or when optimized attention kernels make attention maps inaccessi- ble. This limitation motivates fundamental question: Is it possible enhance model performance by tuning universal, task-agnostic token, thereby bypassing the need for fragile, task-specific to- ken identification?
In this paper, we show that the answer is yes. The key is not added complexity, but rather leveraging ubiquitous yet often overlooked architectural artifact: the initial token (e.g.. <BOS> in LLaMA). While its tendency to function an "attention sink" is well documented (Xiao et al. [023] Kaul et al. 2024 Gu et al., 2024, Barbber et al. 2025). its potential for performance enhancement has remained largely untapped.
Blacs Trmutrrarttonn/s toddla) def meeodddd"". lumyet lngers
+19.9%
+4.3%
+2.1%
Robaove tmprroo Co Coein (90
Standard arrecttion wnight axtrrtattio
actrumeighco P.softmo (araeCCeecchaaee www.hhhwtttaaoorrr...oo
+.3..0%
+2.8%
2. our [Deatasing] Nothoc
Jayer
Loyure:
reighte, lmnyst Leadl
aualiny Testie
lee cormalies
R1
+6.2%
+3.6%
wwwraaiisefattrr..o reigitall, Carget, Saadal) p-t, 4im--1)
Consuce uttention
acteooutse::
seiigtt, oolie_otattee)
Limma 2- 2-130-3000000 Chat
+1.0%
catrring Fted SNAATTTTTIO cods
Thes Closetfication
Multiple Chotos
Convernation
Figure Overview of the Zero Tuningg method and its effectiveness. Left:
Our method requires only a few lines of code to scale the initial token's 's attention within the model's forward pass. Right: This simple intervention yields significant and consistent improvements across variety of LLMSs and tasks.
Empirically, find that steering only the initial token's attention consistently improve LLM performance across diverse tasks, often matching or surpassing prior attention-steering approaches that rely tuning multiple, task-specific tokens.
To understand why this simple intervention works, we proceed in three steps: we first provide a the- oretical characterization of how scaling the initial token reshapes attention (Sec. 3.T]. then connect the induced redistribution to changes in systematic eтrоr patterns and output entropy (Sec. and finally localize where the control most effective by dissecting its impact across layers and heads
(Sec. 3.3 3.5
Insights. Steering the BOS token provides universal, training-free method to improve LLMs: modulates attention sharpness, reduces output entropy, mitigates understanding biases, and concentrates the effect in early layers and specific heads.
Building on these findings, we introduce ZeroTuning (Figure D simple and training-free method that recalibrates the initial token's 's attention to boost LLM performance without task-specific token identification (Sec. 3.6). We present two variants: a supervised mode that maximizes accuracy on labeled validation set, and an unsupervised mode that minimizes output entropy. Across 15 bench- marks, ZeroTuning yields substantial gains on models including 1amma-3.--BB-Insee 13B-Instruct Qwen-2-7B, and Deepseck-R1-14B For example, it improves Llama-3. by a relative 19.9% on classification and 4.5% on question answering, and increases its MT-Bench score from 7.804 to 7.966. The method is robust across long contexts, few-shot settings, quantiza- tion, and prompt variations.
Llama-2-
RELATED WORK
Our work is situated the intersection of two active research areas: inference-time attention tun- ing and the mechanistic understanding of initial tokens. A growing body of work has shown that modifying token-level attention inference time enhance the performance of both LLMs and
2

---

Published as a conference paper at ICLR 2026
VLMs (Yu al. 2024; Zhang et al. 2023a Liu et al. vailing methods like PASTA Zhang et al. 2023a) and Auto-PASII (Zhang et al. 2024a), which identify and up-weight key tokens, or ACT Yu all 2024), which down-weight non-initial sink tokens, fundamenally rely on heuristics to identity task-specific tokens. This reliance limits their universality and introduces potential biases. Concurrently, another line of research has focused explaining why the initial token often becomes an "attention sink" (Xiao et al. (023). attributing it to architectural biases and its role stabilizing anchor (Barbero et al. 2025 Gu al. 2024). While these studies provide a crucial understanding of what the phenomenon is, the question of how to ac- tively and elegantly harness it for performance gains remains largely unexplored. Our work bridges this gap. We shift the focus from task-specific token identification to a universal, task-agnostic con- trol point, and move from passive observation of the initial token to practical tuning framework that leverages its unique properties. Detailed related work is provided in Appendix A
2024b Wei & Zhang, 2024). However, pre-
3 UNVEILING THE POWER OF THE INITIAL TOKEN
In this section, formalize how scaling the initial token's attention reshapes the attention dis- tribution, then empirically demonstrate why this token uniquely effective and analyze how the effect varies across layers and heads. These findings motivate our ZeroTuning methodology. Unless otherwise specified, all experiments use Llama-3. --8--Instuuct, see Section 4.1 for setup details.
3.1 FORMALIZING THE TUNING PROCESS
In decoder-only Transformer, autoregressive generation for a sequence = [arg .1....,......................... 21-1 RdXT relies on causal self-attention. At timestep T, the query derived from the final token repre- sentation æT-1 attends to all preceding token representations (including itself) as keys, producing an attention distribution:
T-1 Σ
a = ao "0o00),;..,.........
-T-, where a+
and
a 1.
Here, dg is the attention score assigned to the initial token, while d1.... 0T-1 correspond to subse- quent tokens. To control the influence of xg. we introduce a tuning factor > 0 to scale its attention and re-normalize:
700
a; for I,...,T 1, a'
d =
where the normalization constant = 740 + E a; = (7 1)ae +
This rescaling preserves the relative proportions among all non-initial tokens:
D
a
for i 2 1,
(3)
Σ a'
2j=1
D
j=
aj
but compresses or expands their differences as
de d
01 aj
0. aj
for 2 1.
(4)
(7 1)00
Intuitively, 7 > 1 amplifies oo, flattening the remaining distribution, while y 1 suppresses an- sharpening it. The magnitude of this effect is governed by the initial token's own attention weight, ag- We define E as the change in the attention difference between any two non-initial tokens i and j:
by 1 1/a0
Eam = 1(0b - -a)) -
(a;=- [= -
la; asl
1)a0 + = la; - azl
(5)
(7
(7 1)a0 +
To analyze how i i. varies with ag, we take the partial derivative with respect to ag:
Faniis
la; aglly 11 ((7 1)a0 + 1)
Dag
3

---

Published as a conference paper at ICLR 2026
Since la, a,1ly 2 0 and the denominator ((> 1) + 1)2 = D2 0, the derivative is non-negative. Thus, s is monotonically non-decreasing in ag; in the non-trivia triviai case (7 * and 0; # az), it is strictly increasing. A detailed proof and visualization
in Appendix
Key insight: the larger the initial token's s attention (ag). the more effectively it can act as a lever to control the attention distribution. Since initial tokens known to be attention sinks (Barbero al. 2025 they are natural control points for this tuning process.
3.2
THE UNIQUE IMPORTANCE OF THE INITIAL TOKEN
00.00
00.00
75:00
45.00 NC
c0.50
74.00
50.00
38.00
30.00
70.00
36.00
20.00
Scating Tector
Scalig
Sca my
SST-2
(h) BoolQ
(c) LogiQA
Figure 2: Impact of attention scaling factor on different token positions across three tasks: (a) SST- (b) BoolQ, and (c) LogiQA. Modifying the initial token's attention consistently yields significant accuracy improvements, often surpassing adjustments to other tokens.
Given the special role of the initial token, we first investigate a key empirical question: (a) Does tuning its attention positively impact performance on downstream tasks? (b) Is this position more effective and influential than others? To investigate, we conduct a controlled experiment in which uniformly scale the attention scores of single token position across all heads and layers ing a scaling factor 7, We evaluate the resulting performance on three downstream tasks: SST-2, BoolQ, and LogiQA. For comparison, we repeat the same procedure for other positions, including the second, third, middle (LT'/2),, and final tokens. As shown in Figure 2 tuning the initial token consistently yields the largest and most stable gains across tasks. Interestingly, the best direction depends on the task: SST-2 benefits from up-scaling (y > 1), while BoolQ and LogiQA improve with down-scaling (7 < 1).
Previous work has identified the initial token attention sink that helps prevent over-mixing of information during autoregressive generation (Gu et al., 2024 Barbero al. 2025). Our results extend this understanding by showing that tuning the initial token reshapes attention over subsequent tokens and improves downstream performance. We interpret this effect from two perspectives.
1. Correcting Biases. Tuning coun- teracts reasoning failures driven by pre- trained attention biases. We observe
1.32
honing 20
a consistent pattern: (a) Up-scaling (y > 1) benefits tasks requiring holis- tie context integration. flattens atten- tion over non-initial tokens, reducing over-reliance on misleading local cues. For example, in SST-2 (Appendix TT) models can fixate on isolated negative words while missing positive context; increasing initial-token attention pro- motes a more global read and corrects such errors. (b) Down-scaling (y 1) benefits tasks requiring sharp focus critical evidence. It increases the relative contrast among non-initial tokens, helping the model localize salient spans in diffuse contexts. In long-context BoolQ, reducing initial-token attention can sharpen focus the text segment that contains the answer.
Earry
-14
soitig Bocks
Scaling Booker
(a) SST-2
(b) LogiQA
Figure 3: A strong inverse correlation between accuracy (orange) and next-token prediction entropy (blue).
2. Reducing Predictive Uncertainty. The tuning process can be viewed through the lens of out- put entropy. a proxy for model uncertainty. As illustrated in Figure B a clear inverse correlation emerges: the scaling factor that minimizes entropy consistently aligns with the factor that maxi-
4

---

Published as a conference paper at ICLR 2026
mizes accuracy. This suggests method better unlocks the model's pretrained knowledge, leading to more confident and correct predictions.
3.3 LAYER WIRE ANALYSIS OF INITIAL TOKEN SCALING
To understand how this effect propagates, we apply scaling selectively to different layer groups
Following prior work layer functionality in Transformers (Jin et al. 2024 Zhang al. 2024b); we partition the 32 layers of 1-ama-3.1-8B-Intttee
into shallow (Layers 1-10), middle (Layers
11-21), and deep (Layers 22-31). We then scale the initial token within each group and evaluate six tasks: BoolQ, SST-2, SST-5, MR, LogiQA, and MathQA. Based on Section 3.2
we use [0, for
BoolQ and LogiQA, and [1, 2] for the remaining tasks.
01.00
50.00
78.00
8 43.00
92.00
79.00
74.0
71.00
47.00
70
71.00
70.00
66.00
120
04
05
3.5
67
035
00
10
(a) SST-2
(h) BoolQ
(c) SST-5
01.50
36.5
4.00
9250
42.00
3050
22.00
X1
3450
03.00
03.00
30.00
13 23
11
(d) MR
(c) LogiQA
(f) MathQA
Figure 4: Accuracy trends when scaling the initial token's attention across different layer groups: shallow (Layers 1-10), middle (Layers 11-21), and deep (Layers 22-31). Different depths exhibit consistent accuracy trend with varying magnitudes.
As shown in Figure 4 scaling induces similar trends across depths, and tuning all layers jointly typically yields the best performance. However, the gains vary magnitude: tuning shallow and middle layers usually helps more than tuning deep layers.
Prior studies have found that early and middle layers mainly support representation learning and knowledge integration, while deep layers focus on task-specific reasoning over aggregated features argue that the tuning process more effectively
(Chen al., 2024; Jin et al. 2025). Therefore, reshapes the representational space in shallow and middle layers, promoting better downstream per- formance and reducing uncertainty.
3.4 ANALYZING THE ROLE OF THE INITIAL TOKEN ACROSS ATTENTION HEADS
Unlike layers, attention heads operate in parallel and contribute independently via concatenation, so their responses to initial-token scaling may differ. To test this, we scale the initial token in each head individually with 7 1.5 and evaluate on SST-2 and MMLU. For comparison, we also consider (i) no scaling 1). (ii) uniform up-scaling across all heads (? 1.5), and (iii) uniform down-scaling
across all heads (? 0.6).
As shown in Figure heads respond heterogencously to initial-token amplification. We call head up-effective if scaling improves performance, and down-effective otherwise. The mix of these head types varies across datasets, which helps explain why uniform scaling behaves differently: SST-2 has more up-effective heads and benefits from up-scaling, whereas MMLU has more down-effective heads and thus prefers down-scaling.
These results align with prior studies showing that attention heads specialize into distinct functional roles during pretraining (Zheng al. 2024b Guo et al. 2024), such global retrieval, structural
5

---

Published as a conference paper at ICLR 2026
05.00%
3
22.
75.00%
14
12.00%
24
Accurras
87
65.00%
Up-Cifective llacs
Donn@fflccie heads
Vanill Saso me Rote
56.00%
Hote 9.6
Head
Head
(a) SST-2
(b) MMLU
Figure 5: Accuracy of scaling the initial token's attention in individual heads using 7 1.5 across (a) SST-2, (b) BoolQ, (c) MMLU, and (d) MathQA. Results reveal heterogeneous behavior among heads, motivating head-specific tuning strategies.
parsing, option discrimination, and negation sensitivity. We propose that these functional differ- ences may explain the variable impact initial token attention scaling, with some heads supporting broad global reasoning and others focusing on salient tokens. This interpretation requires further exploration in future work.
3.5
EVALUATING HEAD-SPECIRIS SPECIFIC TUNING STRATEGIES
70.00
80.00
67.50
65.00
Accureat acy (%6
60.00
62.50
40.00
20.00
55.00
52.50
50.00
2.0
3.0
4.0
5.0
0.0
1.0
2.0
ing Factor
Scaling
(a) SST-2
(b) MMLU
Figure 6: Accuracy comparison of different tuning strategies (a) SST-2 and (b) MMLU. Head- specific tuning (UP, DOWN) consistently outperforms uniform scaling, validating the importance of accounting for head-level behavior.
Given the diversity in head responses, we investigate whether head-specific tuning offers greater effectiveness than uniform tuning. Specifically, we compare four strategies: (i) uniform scaling of all heads (ALL), (ii) scaling only up-effective heads (UP), (iii) scaling only down-effective heads (DowN), and (iv) hybrid strategy (UP+DowN) that scales up-effective heads to fixed optimal value and tunes down-effective ones.
As shown in Figure 6 head-specific tuning (UP, DOWN) yields higher accuracy and faster conver- gence compared uniform scaling (ALL). Notably, UP is most effective when y > while DoWN excels when 7 < 1. Interestingly, the UP+DOWN strategy does not outperform UP or DowN indi- vidually, possibly due to the concatenative nature of attention heads and suboptimal joint scaling.
3.6 ZEROTUNING METHODOLOGY
Building on these findings, we propose ZeroTuning. which improves LLM performance by apply- ing head-specific scaling to the initial token without identifying task-specific tokens. ZeroTunnin

---

Published as a conference paper at ICLR 2026
has three steps: (i) head behavior profiling, (ii) selective rescaling, and (iii) renormalization. For calibration (steps (i)-, we introduce both supervised and unsupervised variants.
Supervised Calibration Consistent with standard practices in inference-time adaptation (Yu et al.
2024 Zhang et al. 2023a; 2024a), our primary approach utilizes labeled calibration set (c.g.,
validation examples) to perform:
1. Head Behavior Profiling: Assessing each attention head's 's sensitivity to the initial token's attention scaling. A head classified as up-effective if increased attention improves accu- racy, and down-effective otherwise.
2. Selective Rescaling:
Applying scaling factor identified by searching for the value that maximizes accuracy on the calibration set, exclusively to the dominant head type (i.c., the most numerous group).
Unsupervised Calibration via Entropy Minimization To mitigate the reliance on labeled data, propose a novel unsupervised calibration strategy. Based our finding that model's output entropy strongly correlates with its accuracy [Section[3.2), this variant does not require any labeled calibration or validation set. Instead, it identifies the optimal heads and scaling factor y by mini- mizing the average next-token prediction entropy over batch of unlabeled inputs. Crucially, the unlabeled inputs can be obtained in two practical ways: (i) in an offline setting, by running Zero- Tuning on a held-out, unlabeled corpus from the same domain; or (ii) in test-time adaptation style, by performing entropy-based search over the current batch of test-time queries. We provide a more detailed analysis of this unsupervised variant in Appendix E
Finally, Renormalization softmaxx to maintain a valid distribution. For optimized attention implementations (e.g., SDPA, FlashAttention) where direct score modification is infeasible, ZeroTuning applies scaling to the query or key states; Appendix GanndixxGshae shows this yields similar effects.
shared by both variants: we re-normalize the scaled scores (via
4
EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETUP
Our evaluation includes four recent LLMs with distinct attention implementations (Llama-3.1-8B.
Llama-2-13B, Qwen-2-7B, and DeepSeck-RI-1444 RI-14BB.. We test performance across 15 datasets span- ning three categories: Text Classification,
Caltiple-Chhice QA, and Multi-Round Conversation. We benchmark ZeroTuning against three methods: (1) Vanilla inference; (2) ACT (Yu et al.) 2024). a sink-token down-scaling method and (3) Auto-PASTA (Zhang et al. (0244)), an LLM-guided key- token up-scaling method. All experiments are conducted in zero-shot setting with greedy decoding for fair comparison. For our supervised variant and the baselines, hyperparameters are calibrated on a fixed validation set. A detailed description of all models, datasets, basclines, and implementation specifics is provided in Appendix
4.2 OVERALL PERFORMANCE OF ZEROTUNING
For fair and direct comparison with existing supervised baselines, we focus on supervised Zero- Tuning in our main experiments.
Text Classification We first evaluate ZeroTuning on various text classification datasets using dif- ferent LLMs, shown in Table [ Despite tuning only a single token, ZeroTuning consistently outperforms baselines and methods that require tuning more tokens. With Llama-3.1-88 Instruet it achieves an average improvement of +11.7%% over vanilla, with peaks of +22.00% on SUBJ and + 18..0% on SST-2. It outperforms AutoPASTA by an average of 7.71%. On Qwen-2-7B, ZeroTTun- ing gains +3.00%, and on Deepseck-Rl-145 improves by +4.20%, with a notable increase of +11.20% TREC.
Domain-Speccfic Multiple Choice Next, we evaluate Zeronuning on common domain-specific multiple-choice datasets under various settings, as shown in Table 2 For Llama-3, 1-8---..-8--Insttucc
Since ACT explicitly manipulates attention maps, we only evaluate it on Llama-3.1 1-B8888---sstrrec

---

Published as a conference paper at ICLR 2026
Performance Comparison Classification Tasks Across Models. The best performance in each dataset is bolded and the ZeroTuning method is highlighted in gray.
Table 1:
Datasets
Model
Method
SST2
SSTS
MR
BoolQ
CB
TREC
SUBJ
Avg.
Vanilla
73.20
45,40
89.20
69.60
82.14
14.00
44.60
59.59
ACT
85.00
43.80
90.80
58.60
82.14
15.80
44.60
60.11
Llama-3.1-888 3.1--8B-nstttt
Auto-PASTA
89.60
47.20
91.40
72.60
83.93
16.00
45.40
63.73
ZeroTuning
91.60
52.00
92.00
82.40
89.29
26.20
66.60
71.44
Vanilla
78.80
45.40
72.40
85.00
78.50
12.60
13.00
55.10
ACT
Qwen-2-7B (SDPA)
Auto-PASTA
89.00
47.00
77.70
85.00
89.29
14.00
57.00
65.57
ZeroTTuning
89.60
47.20
87.40
86.40
85.71
26.60
54.40
68.19
Vanilla
91.20
49.40
89.20
83.40
89.29
20.80
50.40
67.67
ACT
Deepseek-Rie Ri-sseek-R--4BB (Flash)
Auto-PASTA
92.00
52.20
89.80
83.40
92.86
22.60
50.40
69.04
Zero Tuning
93.00
51.20
90.20
88,00
92.86
32.00
55.80
71.87
Table 2: Performance Comparison of Multiple- Choice Tasks Across Models.
Model
Method
Datasets
MMLU
AQUA
MathQA
LogiQA
CQA
PIQA
ARCC
Avg.
Vanilla
67.40
25.69
33.60
39.40
77.60
83.60
84.62
58.84
ACT
67.60
29.64
33.60
38.00
77.60
83,00
84.62
59.15
Auto-PASTA
67.00
31.23
35.20
40.40
78.20
84,60
84.62
60.18
Zero Zuningging
68.80
20.43
36.60
42.80
80.40
85.40
85.95
61.48
Vanilla
69.80
36.76
45.00
78.80
ACT
Qwen-2-7B (SDPA)
Auto-PASTA
69.80
39.13
39.20
45.00
82.60
85,40
86.96
64.01
ZeroTuuning
39.92
$1.80
Vanilla
66.60
38.74
38.20
27.80
78.20
84.20
86.62
60.05
ACT
Deepseek-R1 11-11eck--114BB (Flash)
Auto-PASTA
66.60
38.74
28.20
78.20
86.62
60.31
ZeroTTuning
70.00
39.13
39.80
35.60
78.60
85.00
87.29
62.20
it increases the average accuracy by +2.64%, with gains of +3.40% on LogiQA and +1.40% on MMLU. Qwen-2-7B gains +1.74%, and Reepseck-RI-14B gains +2.15%, with outstanding +7.80% on LogiQA.
Multi-Round Conversation We further demonstrate ZeroTuming's effectiveness multi-round conversations using MT-Bench (Zheng et al. (2024)), with results in Table 3 Zero
Table 3: MT-Bench Performance Scores for Multi-Round Conversation Across Models
Midel
Fext Tuza
Sccond Tira
Averge
im- proves the average score by 0.162 points (7.966 vs. 7.804). For Llama-2-133 1-a--22-1----C 13B-Chat, it achieves 0.266 points gain (6.916 vs. 6.650), showing its effectiveness in interactive settings
For -lama-3.1-$B-nnttree
spi4
Ail-ZerTTuning Gpe-35-uube
(-4.020)
5.075
chadfeimuuee
1.00
7,105
clauboct
1.50
7.690
Liane vanilla
$,263
7.583
7,804
Lhamae 2- 2410-CChttZZeeTToinn
7.108 (-10043)
6.225 (-11.47)
6,916 (44.2266)
Liume 2. 138 Chaz vonilla
7.003
5,235
6.050
4.3 UNSUPERVISED ZEROTUNING
103
We validate fully unsupervised variant of Ze- roTuniigg which removes the need for labeled calibration data by minimizing the model's av- erage output entropy. Appendix L provides a detailed empirical validation, including visual comparisons, additional analysis, and a break- down of error patterns. As shown in Figure 7 this entropy-guided method is highly competi- tive with its supervised counterpart, extending applicability to label-scarce scenarios.
Ven
ZersBiiig (Sapenstodi
83
Zartiliing (nmspentted
Acturaty
5512
5313
Bou10
MPL
Figure Performance comparison of Vanilla, pervised, and Unsupervised ZeroTuning on Llama- 3.1-8B-Instruct.
8

---

Published as a conference paper at ICLR 2026
5 FURTHER ANALYSIS AND KEY FINDINGS
Robustness Across Diverse Conditions. ZeroTuning demonstrates remarkable stability. It main- tains strong performance gains even when faced with: (1) long contexts padded with irrelevant dis- tractors, where it stabilizes information flow better than the vanilla model (Appendix[]) [T; (2) few-shot scenarios, where consistently improves dostruction-folloimng and reduces invalid outputs (Ap- pendix I); (3) significant prompt variations, including missing altered instructions (Appendix and (4) low-pre pew-preciiion- 4-bit and 8-bit quantization, where it partially mitigates the associated accu- racy degradation (Appendix N).
Practicality and Methodological Choices. Our method is not only robust but also highly prac- tical. We theoretically and empirically confirm that tuning key states provides viable, kemel- agnostic alternative to direct attention score manipulation, proving effective in optimized envi- ronments like FlashAttention (Appendix G). Furthermore, ZeroTTuning is adaptable to resource- constrained settings, delivering gains even with a minimal, search-free scaling approach (Ap- pendix H). We also analyze key methodological choices, showing that tuning moderate subset of heads (40%-70%) is optimal, providing a clear and efficient configuration (Appendix (
Boundaries of Efficacy. Finally, analyze the method's boundaries and potential side effects. We quantitatively demonstrate that Zero Tuning excels at correcting a model's uncertain errors but cannot override high-confidence mistakes rooted in flawed pretrained knowledge. We also charac- terize the negative effects of extreme tuning, which provides a clearer picture of the method's 's oper- ational limits (Appendix Dj. This positions our method as powerful tool for unlocking model's latent knowledge, rather than substitute for fine-tuning. Intriguingly, we also find that within safe operational range, the scaling factor can modulate output diversity in a manner analogous to temperature, but with the unique ability to alter the rank-ordering of logits and thereby correct errors that temperature scaling cannot fix. This aligns with our experiments on temperature tuning, where pure temperature adjustment fails improve benchmark performance, while ZeroTuning yields consistent gains substantially.
6 FURTHER DISCUSSION
6.1 EFFICIENT CALIBRATION AND SEARCH
In our current implementation, the offline search for the scaling factor 7 and head selection with Llama-3.1 1.1-8-31-8BB 3.B takes roughly 4-5 minutes on single GPU, and the inference-tima overhead is neg- ligible. We also highlight an interesting empirical observation: the clear U-shaped entropy curve identified in Appendix E suggests that more efficient search strategies, such as Bayesian optimiza- tion or simple rule-based early stopping, could further reduce calibration cost. We view this promising direction for future optimization.
6.2 ZEROTUNING AND SUPERVISED TIN-TUNING
Supervised fine-tuning (SFT) remains a strong and versatile way to improve LLMs: by updating parameters, it can simultaneously adjust task knowledge, instruction following, output formatting. and various biases. However, this also means SFT is an indirect optimization over many intertwined behaviors and typically requires more data, compute, and careful hyperparameter tuning.
Zero Tuning takes complementary approach. Instead of changing parameters, it directly adjusts the attention pattern on the initial token, targeting the specific failure modes identified our analysis (e.g., format traps and biased attention over context tokens). This makes the intervention more localized and training-free while still being compatible with SFT.
To study this relationship, Turininng use the same 500 labeled examples. We fine-tune the model with a lightweight LoRA setup (rank 4, 3 epochs) and compare four configurations:
controlled experiment BoolQ where both SFT and supervised

---

Published as a conference paper at ICLR 2026
Method
BoolQ Accuracy
Vanilla
69.60
Vanilla + SFT (LoRA)
81.20
Vanilla + Zero' ZeroTuning (Ours)
82.40
Vanilla + SFT + ZeroTuning (Ours)
83.60
In this limited setting, the training-free Zero ZeroTuning slightly outperforms our small-scale LoRA SFT, and applying Zero Tuning on top of the SFT model yields the best performance. While stronger fine- tuning setups (more data, higher rank, more epochs) may implicitly learn to adjust the initial-token attention a similar way, we hope our observations provide a useful lens for understanding how SFT optimizes attention patterns and inspire future work on combining maining-free and training-bant baasd approaches.
CONCLUSION
This work provides systematic analysis of how scaling the initial token's attention affects Trans- former's attention distribution and downstream accuracy. Building on this analysis, we introduce ZeroTuning, training-free inference-time method that calibrates single, task-agnostic token. Across a wide range of tasks and models, Zero'T Tuning improves performance over vanilla inference and is competitive with (and often stronger than) prior inference-time tuning baselines that require manipulating additional tokens or task-specific heuristics. Zero Tuning supports both supervised cal- ibration (via small labeled set) and unsupervised calibration (via entropy minimization), and can be implemented across different optimized attention kernels. Overall, our results highlight the initial token as a simple but effective control point for lightweight, deployment-friendly model adaptation.
ACKNOWLEDGMENTS
We extend special thanks Prof. Surbhi Goel from the University of Pennsylvania for her valuable advice on the theoretical aspects of this research.
REFERENCES
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannanch Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation-based for- malisms. arXiv preprint arXiv: 2019.
Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Razvan Pascanu, al. Why do Ilms attend the first token? arXiv preprint arXiv:2504.02732. 2025.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tianxiang Chen, Zhentao Tan, Tao Gong, Yue Wu, Chu, Bin Liu, Jieping Ye, and Nenghai Yu. Llama slayer 8b: Shallow layers hold the key to knowledge injection, 2024. URL https:
02330
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1905.10044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 05457, 2018
10

---

Published as a conference paper at ICLR 2026
Catre-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang. Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025. URL BtRhhtps:: arxiv. org/ hbtp///as/255.. 12948
Tim Dettmers, Artidoro Pagnoni. Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. ArXiv, httbb//335.1331.11 2023. PRLAttpss/app...eaaaiccco org/Corpus TD: 258841328
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Xiangming Gu, Tianyu Pang, Chao Du Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang. and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410. 10781, 2024.
Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena llms. arXiv preprint arXiv:2410.138. 13335, 2024.
Feijiang Han. Beyond detection: A comprehensive benchmark and study on representation learn- ing for fine- fine-graines grained webshell family classification. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI-26), 2026. URI. /tttps://arxi...rtt org/ abs 22122 05288
Feijiang Han, Jiaming Zhang, Chuyi Deng, Jianheng Tang, and Yunhuai Liu. Can Ilms handle webshell detection? overcoming detection challenges with behavioral function-aware frame- work. In Second Conference on Language Modeling (COLM 2025), 2025. URL https:
Feijiang Han, Hengtao Cui, Licheng Guo, Zelong Wang, and Zhiyuan Lyu. Read before you think: Mitigating Ilm comprehension failures with step-by-step reading. In Proceedings of the IEEE
11

---

Published as a conference paper at ICLR 2026
International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026a. URL https://arxii/: otw.ii..org//bb//2555....
Feijiang Han, Zelong Wang. Bowen Wang, Xinxin Liu, Skyler Cheung. Delip Rao, Chris Callison- Burch, and Lyle Ungar. Latex2layout: High-fidelity, scalable document layout annotation pipeline for layout detection. In Proceedings of the AAAI Conference on Artificial Intelli- gence (AAA/-26), 2026b. Latex2layout.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.0965. 2021. URL htttht:t///apppt
semant www.mnaatccccha......mmmm
23345580009
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng. Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng. Kaize Ding, et al. Exploring concept depth: How large language models acquire knowledge and concept at different layers? arXiv preprint arXiv:2404.0777 07066, 2024.
Mingyu Jin, Qinkai Yu. Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengman Du, and Yongfeng Zhang. Explor- ing concept depth: How large language models acquire knowledge and concept at different layers? In Owen Rambow, Leo Wanner, Marianna Apidianaki. Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.). Proceedings of the 31st International Conference Computa- tional Linguistics, pp.
Abu Dhabi, UAE January 2025. Association for Computational Linguistics. URL https: /aa/aaaaanntoloooooooe org/ 2025. co ing -aan-maiinn. 1
Prannay Kaul, Chengcheng Ma, Ismail Elezi, and Jiankang Deng. From attention to activation: Unraveling the enigmas of large language models. In The Thirteenth International Conference on Learning Representations, 2024.
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. Logiqa an improved dataset for logical reasoning in natural language understanding. 1EEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2947-2962, 2023.
2.0-an
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao. Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.0/244 2024a.
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: training-free freee method for alleviating hallucination in lvims. In European Conference on Computer Vision, pp. 125-140. Springer, 2024b.
Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, and Mu Li. Attention calibration for transformer in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2288-1298, 2021.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. arXiv preprint cx/0409058, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cx/0506075, 2005.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
12

---

Published as a conference paper at ICLR 2026
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arw.v:1111.0000000000 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet. Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL httpt::/ttssattttt
abs/ 2307 09288
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed H. Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203. 11171, 2022. URL https:://api.
2475595222
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language mod- els. ArXiv, abs/2201.11903,
2022.
URL attps::/api.. https///ppp..emmmatccccccohooo.o
CorpusID:24611121
Jinfeng Wei and Xiaofeng Zhang. Dopra: Decoding over-accumulation penalization and re- allocation in specific weighting layer. In Proceedings the 32nd ACM International Conference on Multimedia, pp. 7065-7074, 2024.
G Xiao, Y Tian, B Chen, et al. Efficient streaming language models with attention sinks. arXiv preprint arX9v:2309. 17453, 2023.
An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu. Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://arxxt
org/abs/////oooooooooooo/////oar /200s/2400.o0 10671
Yu, Z Wang, Y Fu, et al. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406. 15765, 2024.
Q Zhang, C Singh, L Liu. et al. Tell your model where to attend: Post-hoc attention steering for llms. arXiv preprint arXiv:2311.02262. 2023a.
Qingru Zhang, Xiaodong Yu, Chandan Singh, Xiaodong Liu, Liyuan Liu, Jianfeng Gao, Tuo Zhao, Dan Roth, and Hao Cheng. Model tells itself where to attend: Faithfulness meets automatic attention steering, 2024a.
Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. Investigating layer importance in large language models. arxiv preprint XXX.:2449..138.... 2024b.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023b.
13

---

Published as a conference paper at ICLR 2026
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024a.
Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Mingchuan Yang, Bo Tang, Feiyu Xiong, and Zhiyu Li. Attention heads of large language models: A survey, 2024b. URLhttps:
//axxiv.org/abt wtt./////p///...////////// 03752
Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucina- tions in large vision-language models via image-biased decoding. ArXiv, abs/2402. 18476, 2024. URL https:: //:://a /api. semanticscholas ar ottg/coppus11::2250111...
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
A.1
ATTENTION TUNING
Token-level attention tuning typically aims to increase attention to critical input tokens decrease attention to less informative tokens. [u et al. (2021) proposes a mask perturbation method to ad- just attention weights for key tokens, thereby improving translation quality. Zhang et al. (2023a) introduce PASTA, which allows manual designation of important tokens during inference. This is extended by AutoPASTA (Zhang et al. 2024a). which uses LLMs to autonomously identify salient tokens and increase attention to them. In contrast, ACT (Yu et al.
2024) reduces attention to seman- tically trivial sink tokens and redirects it to meaningful content. Similar strategies have been applied to VLMMs to mitigate hallucinations. PAI (Liu et al.) 2024b) enhances attention to image tokens at inference time to counteract text-dominant bias. IBD (Zhu et al.) 2024) and OPERA (Wei Zhang 2024) further refine this idea by prioritizing visual information or penalizing overconfident summary tokens. While effective, these methods depend on identifying task-specific tokens, which may intro- duce bias (e.g., overemphasizing misleading tokens) and limit applicability when token importance unclear or attention maps are unavailable. In contrast, our method focuses on a task-in insariavtr initial token, removing the need for costly token identification, and can be easily applied by tuning key states.
A.2 THE MAGIC OF THE INITIAL TOKEN
Recent studies highlight the significance of the initial token, especially through the lens of the at- tention sink phenomenon, where it draws substantial attention despite low semantic content. [iio]] al. (2023) show that preserving such tokens is critical for maintaining performance in sliding win- dow attention. Kaul et al. (2024) attribute this effect to softmax normalization and causal masking, while Gu al. (2024) and Barbero et al. (2025) identify architectural biases that amplify attention to the initial token, including key-query alignment and LayerNorm effects. Functionally, the ini- tial token is hypothesized to serve as a stabilizing "no-op" anchor, enhancing robustness to prompt variations (Barbero et al., 2025). It has been leveraged in applications such as long-context mod- eling (Zhang et al.J 2023b; Xiao et al. 2023), but also poses challenges for quantization due to its high attention weight (Dettmers et al. 2023 Liu et al. 2024a). While previous work has identified the structural and functional importance of the initial token, its potential as a target for attention tuning remains underexplored. In this work, we provide detailed analysis of attention tuning of the initial token across layers and heads, demonstrating its consistent influence across different tasks. Our approach bridges the gap between the these lines of research by proposing a novel method that advances interpretable attention tuning.
B DETAILED EXPERIMENTAL SETUP
Models, Tasks, and Datasets. Models: We evaluate ZeroTuning on four LLMS with distinct at- tention implementations:
1-B--..--8----srruee (Grattafiori et al., 2024) and Llama-2-13B 1-1BB-hh (Tovrron et al. (023) with eager attention, Qwen-2-7B (Yang et al. 2024h with SDPA attention, and ReepSeck-R1-14B (DeepSeck-AI et al. 2025) with Flash attention P Tasks and Datasets; Our experiments encompass three task types across 15 datasets: (1) Text Classification and Reasoning, including SST-2 (binary sentiment classification) (Socher et al. 2013), SST-5 (fine-grained senti- ment analysis) (Socher et al.) 2013). MR (movie review polarity detection) (Pang & Lee| 2005).
SUBJ (subjectivity classification) (Pang & Lee, 20004), TREC (question type classification) (Li & Roth 2002), CB (commitment detection) (De Marneffe et al. 2019). and BoolQ (boolcan question answering) (Clark al. 2019: (2) Comain-Specific Multple-Choice, including MMLU (cross- domain knowledge testing) (Hendrycks et al. 2020). AQUA (math word problems) (Zheng et al. 2024a), MathQA (algebraic reasoning) (Amini al. 2019). LogiQA (logical reasoning) (Liu al, 2023]. CQA (commonsense reasoning) (Talmor et al. 2018). PIQA (physical commonsense QA)
(Bisk et al., 2020), and ARCC (scientific reasoning) (Clark et al. 2018); and (3) Multi-Round Con- versation, using MT-Bench (Zheng al.) 2024a).
'Eager. SDPA, and Flash are official attention implementations in modern Transformer libraries. Eager computes the full attention map; SDPA uses PyTorch's efficient API to select the optimal implementation; Flash relies on fused CUDA kernels from the FlashAttention library.

---

Published as a conference paper at ICLR 2026
Baselines and Evaluation Metrics. Baselines: We benchmark ZeroTuning against three baselines: (1) vanilla inference, which performs standard inference without any modifications; (2) ACT (Yu al. 2024). which identifies mone-initial sink tokens using attention score threshold and reduces their attention weights; and (3) Auto-PASTA (Zhang et al. 2024a), which leverages an L.M to locate important tokens and enhance their attention weights. Evaluation Metries: We assess perfor- mance using accuracy for text classification and multiple-choice tasks. For the multi-round con- versation task, we report average quality scores as evaluated by GPT-4, following the methodology outlined in Zheng et al. 2024a).
Implementation Details. All experiments are implemented in PyTorch using the Hugging Face Transformers library. We use a zero-shot setting with greedy decoding for consistency across all methods. For our supervised variant and the baselines, we use fixed validation set of 500 randomly selected samples (seed 42) for calibration. For Zero Turing, we tune the top 40% of identified heads unless otherwise specified. For ACT, we use the official hyperparameter (3 0.4), and since requires explicit attention maps, we only evaluate on Llama-3. I-8B-Instruct Prompts for all tasks and baselines are detailed Appendix O
C THEORETICAL ANALYSIS OF TUNING EFFICACY VIA THE INITIAL TOKEN
This appendix provides a formal proof for the claim made Section 3.1 that the tuning effect's magnitude is governed by the initial token's 's attention weight ag.
Proposition 1. For any given scaling factor + 1 and any two non-initial tokens i.j 2 1 with unequal initial attention weights (0: + aj}. the magnitude of the tuning effect on their attention difference is strictly monotonically increasing function of the initial token's 's attention weight, og-
We aim to show that the tuning effect, Fam,i.j- is a monotonically increasing function of on by prov- ing its partial derivative with respect to ag is positive. Recall the definition of the effect magnitude from Eq. equation 5
17 1 |00
Eaitt ( = =aa aj
(7 1)ag +1
Taking the partial derivative of Eai with respect to ag, we treat the term la:-a; ||> - 1| as a constant factor:
|a:- -ayll7 11.
a0
(8)
Das
Oag
(7 1)a0 +
1. ((7 1)ao + 1) ao (7 1)
|0: -
((7 - 1)a0 + 1)2
|0, - -jily - 1|. [(7 1)00 + 1)2
(10)
= D2, is the square of the normalization constant and strictly positive for any valid probability distribution. Therefore, the derivative &Eat....
The term la, -a,-a;l is non-negative. The denominator, ((y- 1 )ag + =
2 0.
Dao
Furthermore, for any non-trivial case where the tuning factor is active (y * 1) and the attention weights not uniform (a, * aj for some i) the derivative is strictly positive. This proves that Fast is a strictly monotonically increasing function of an- Consequently. a larger initial attention weight provides more powerful lever for modulating the attention distribution. This theoretical result is visually corroborated by Figure
2

---

Published as a conference paper at ICLR 2026
0.5
8,
2)
Attention Difference
0.4
0.8
0.3
Sharp
Smooth
(May Damage LLM)
Excessive Tuning
0.2
0.1
0.0
2
4 Scaling Factor
8
10
Figure 8: Visualization of the tuning effect attention differences |a a'Sl function of the scaling factor y for various initial attention weights ag {0, 0.2, 0.4, 0.6, 0.8). The plot demon- strates that a higher initial attention weight (e.g., the red curve) leads to significantly stronger response to changes y. We identify three primary regimes: sharpening (y < 1). where attention differences are amplified; smoothing (y > 1), where differences are diminished; and regions of excessive tuning (e.g.. + 0 or 1), which may degrade performance.
D DEEPER ANALYSIS OF POTENTIAL FAILURES AND NEGATIVE EFFECTS
To provide comprehensive understanding of ZeroTuning, we analyze its operational boundaries and potential negative side effects when pushed to its limits.
Boundaries of Efficacy: Unlocking Latent Knowledge vs. Correcting Factual Errors. Our analysis reveals a key insight into ZeroTuning's mechanism: primarily unlocks and disambiguates model's latent knowledge, rather than correcting deeply ingrained factual errors. To test this, quantitatively analyzed Zero'uning's corrective power as function the vanilla model's initial prediction confidence. We partitioned the set of SST2 incorrect predictions into "uncertain errors" (vanilla softmax confidence < threshold) and "certain errors" (confidence 2 threshold) and evalu- ated our method's performance on each group.
Corrected Accuracy vs. Certainty Threshold
Conected "Cartain
Corrected Accuracy Eners"
0.8
Accuracy within Subset
0.6
0.4
0.0
Cartainty
Figure 9: Corrected accuracy initially incorrect samples as function of the vanilla model's confidence (certainty threshold).
The results, shown in Figure 9 provide strong empirical validation. ZeroTuning is highly effective the uncertain error population (blue line), correcting over 80% of mistakes where the vanilla
3

---

Published as a conference paper at ICLR 2026
model's confidence below 0.5. Conversely, its ability to fix certain errors (red line) decays sharply as the base model's confidence increases, dropping to near-zero on predictions where the model was already confidently wrong. This confirms clear operational boundary: Tuming excels at resolving low-confidence mistakes by refining the model's focus, but it is not designed to overwrite high-confidence knowledge learned during pretraining.
This finding positions ZeroTTniig not as a replacement for fine-tuning, but as a powerful, com- plementary inference-time technique. It assists a model in better leveraging its existing, albeit sometimes uncertain, knowledge. The potential for synergistic interaction between ZeroTuniig and parameter-effiieet fine-tuning methods like LoRA remains promising avenue for future research.
Negative Effects of Extreme Tuning. We also investigated the effects of applying extreme scaling factors (7), far outside the optimal range. These experiments reveal predictable failure modes that further illuminate the role of the initial token:
Overly-suppressed attention (y - O): When the initial token's attention is excessively reduced, the model's output often becomes degenerative. We observe tendency for the model to enter repetitive loops, outputting a single answer (e.g.. "TTrue") without any of the semantic elaboration or reasoning present in the vanilla output. This suggests that minimal level of attention to the "sink" token is necessary to maintain generative stability.
Overly-amplifted attention (y > 1): Conversely, when the initial token's 's attention is excessively high, it can disrupt the model's ability to follow complex instructions. By absorbing too much of the attention budget, the initial token appears to prevent other, more task-relevant tokens from receiving the focus they need, leading incomplete or non- compliant answers.
Interestingly, within a reasonable range, moderate tuning of the initial token's 's attention can produce effects analogous to adjusting the temperature parameter in decoding. It can modulate the diver- sity of the output, encouraging the model to explore different perspectives or generate more varied responses, without the repetitive downsides of extreme scaling.
However, our method is fundamentally more powerful for error correction. Temperature scaling acts on the final logits = just before the softmax, calculating the probability of the i-th token as P; -oftmax(2z//TT /T)) Since dividing by a positive temperature T does not change the relative order of the logits (i.e., arg max; (2x) arg max;(2:/T) temperature scaling cannot alter the outcome of greedy decoding. In contrast, ZeroTuning operates at the attention level, optimizing the model's internal representations. This process produces entirely new set of output logits, 2', which can have a different rank ordering. It is therefore possible for the originally predicted token arg max; be incorrect, while the new prediction arg max, becomes correct, enabling error correction.
E UNSUPERVISED ZEROTUNING: ANALYSIS AND RESULTS
This section details the unsupervised variant of Zero which eliminates the need for labeled calibration set by leveraging the model's 's output entropy as proxy for performance.
We begin with a visual analysis establish the core principle behind this approach. Figure I0 plots three key metrics against the attention scaling factor 7 the average next-token entropy. its logarithm, and the final task accuracy. The plots reveal a compelling and consistent pattern across all datasets. The average entropy curve (left subplot) exhibits a distinct U-shape, identifying clear scaling factor that minimizes the model's predictive uncertainty. Critically, the trough of these entropy curves aligns remarkably well with the peak, or near-peak plateau, in the accuracy curve (right). This strong visual correlation provides powerful evidence that minimizing entropy can serve as robust, unsupervised signal for identifying high-peeffrmance region for
Guided by this insight, we quantify the effectiveness of an unsupervised approach where we select the 7 that minimizes entropy the unlabeled test set. compares its performance against the vanilla baseline and the supervised ZeroTuning variant. The results demonstrate that the unsuper- vised method is remarkably effective, achieving average score of 67.02 hiihhly competitive with the supervised result of 67.52 and a substantial improvement over the vanilla baseline's 59.00. No- tably, on LogiQA, the entropy-guided method even slightly outperforms its supervised counterpart.
4

---

Published as a conference paper at ICLR 2026
(a) SST-5
(b) BoolQ
330
330
MMLU
Tate vt Estrgy
Tale Eatrey
Fam dedeamay
0.00
3.0
4.50
4.3
(d) LogiQA
Figure 10: Visualizing the relationship between the scaling factor y and three key metrics: average entropy (left), log-entropy (middle), and task accuracy (right). Across diverse datasets, the entropy minimum consistently aligns with region of high accuracy, validating entropy as a strong signal for unsupervised tuning.
This quantitative validation confirms that unsupervised ZeroTuning is powerful and practical alter- native, transforming our method into a versatile tool that can be deployed without any specifcc labeled data.
5

---

Published as a conference paper at ICLR 2026
Table Performance comparison of Vanilla, Supervised, and Unsupervised ZeroTuning on Llama- 3.1-8B-Inse Insttruct The best performance in each column is bolded.
Method
SST-2
SST-5
BoolQ
MMLU
LogiQA
Avg.
Vanilla
73.20
45.40
69.60
67.40
39.40
59.00
Zero (Supervised)
91.60
52.00
82.40
68.80
42.80
67.52
ZeroTuniig (Unsupervised)
90.20
51.80
80.80
68.80
43.40
67.02
E. DEEPER ERROR ANALYSIS FOR UNSUPERVISED ZEROTUNING
A key finding of our analysis is that while minimizing entropy on individual samples can be mislead- ing, minimizing the average entropy across a dataset robustly identifies optimal tuning parameter. This suggests Unsupervised ZeroTTuning corrects for systemic, dataset-level biases rather than iso- lated prediction errors. To understand this phenomenon, we analyze how the method behaves different error types for SST2.
The Unreliable Entropy Landscape of Uncertain Samples. We first examine "uncertain" sam- ples, where the vanilla model's s confidence in its top-choice token is low (c.g., Ptop < 0.5). For these samples, the entropy landscape is often deceptive and contains two primary "traps":
Effect of Scaling Factor (y) on Model Output "the lower your expectations the more you 'Il enjoy it
Sentence:
Trajectory
Titce
Tea Trice aieald Cesall tu (eri)
Enerapy Trleccory Lagits
1.6
Croppy nincetarrs
Scaling Factor
Figure 1I: An uncertain sample where the correct answer is "negative". The vanilla model incor- rectly outputs "positive". For < 1, the model outputs invalid format. For y >> 1, it first amplifies the probability of the initial incorrect answer before flipping to the correct one. The entropy minima misleadingly located in the regions of the invalid and incorrect answers.
1. The Format Trap (y 1): When reducing the initial token's attention, the model enters a "local-focused" mode. For uncertain samples, this can amplify attention on unintended tokens, leading to a violation of task instructions. In this mode, the model often outputs tokens outside the constrained answer space, such as "neutral" or "The", instead of the required "positive" or "negative". As 7 decreases below the model's confidence this wrongly formatted token can increase, creating a misleading drop in entropy.
2. The Bias Amplification Trap (y > 1): When increasing the initial token's 's attention, the model enters "global-integrative" mode. For an initially incorrect, uncertain sample, this

---

Published as a conference paper at ICLR 2026
Effect of Scaling Factor (v) on Model Output Sentence: "a good piece of work more often than not:
Protabiitty Trojetory
To Kince a Yye fre"
To Rows alradd
Goeall be teri
Eneropy Tajectooy Loght Distribution
Entrog (nncertar
Scaling Factor (y)
Figure 12: A more uncertain sample where the correct answer is "positive". The vanilla model incorrectly outputs "neutral" (an invalid format). As increases, the model first transitions valid but incorrect answer ("negative") before finally flipping to the correct answer ("positive"). The entropy landscape exhibits multiple local minima corresponding the invalid, incorrect, and correct answers, making direct judgment based on the global minimum unreliable.
often induces "competing peaks" phenomenon. Both Figure and 12 illustrate cases where the model first amplifies an existing bias, leading a deep entropy well correspond- ing to an incorrect answer. Only with further increases in does the model's interpretation "flip" to the correct one. Figure 12 shows a more complex cascade: after overcoming the format trap, the model first falls into bias trap (incorrectly predicting "negative" due to the word "not") before finally settling on the correct answer ("positive"), In both cases, the global entropy minimum is sought, it may lock onto amplified bias formatting error, creating a trap.
The Stabilizing Behavior of Certain Samples. In contrast, "certain" samples (where Ptop 2 0.5) exhibit more predictable behavior and act as a stabilizing force during average entropy minimization. We identify two sub-types:
Recalcitrant Certain Errors: As shown in Figure 13 if the model is confidently wrong but its answer is within the valid format (e.g., predicting "negative" for a positive sentence), its semantic conviction is strong. In this case, modulating 7 reinforces this conviction, leading to a stable, uni-modal probability peak. The error is not corrected. This suggests the model's relevant pretrained knowledge is already strongly activated, albeit towards an incorrect conclusion. These samples act as stable "ballast" the collective average.
As shown in Figure 14 if the model is confidently wrong because it produced an out-of-format token (e.g.. "neutral"), the error is rooted in a misun- derstanding of task constraints/instruttion. not deep semantic conviction. Here, the model's pretrained knowledge is "locked". Modulating ? (specifically, increasing it) helps the model refocus on the instructions, "unlocking" its latent knowledge and guiding it to the correct, in-format answer. In these cases, the entropy minimum correctly corresponds to the right answer.
Correctable Certain Errors:
7

---

Published as a conference paper at ICLR 2026
Effect of Scaling Factor [yl on Model Output Sentence: courage and complicity auachwitz herrowing drama that tries to the unspeakable.
Purabllt Tejustory el Top Saanhaal Tiban
Entroyy ectory loght Ditritutioo
Sciling
Figure 13: A recalcitrant certain sample. The vanilla model (*) predicts an incorrect but in-format answer (" ("negative") with high confidence. Modulating " only reinforces this conviction without an answer flip.
Effect of Scaling Factor (y) on Model Output Sentence: 'the movie dawn dead crossed with john carpenter 's ghosts mars, ghoulish
big
Probabilty Vrajectory Top Raneed "blan
Prpssiiiyy
Teajectory Logits Distriburion
2.4
Scaling
Figure 14: correctable certain sample. The vanilla model (x) predicts an out-of-format answer ("neutral"") with high confidence. Increasing guides the model to overcome the format error and output the correct answer ("negative").
Why Collective Signal Works Better? The visualizations in Figure 15 reveal a key insight: while individual uncertain samples exhibit highly irregular and sometimes multi-peake probability/accu- racy trajectories as y varies, the aggregate signal constructed from many samples becomes smooth, stable, and strongly aligned with true accuracy improvement.
8

---

Published as a conference paper at ICLR 2026
(a) Uncertain Population (p (1.5) (b) Certain Population (p.> 0.5)
All Samples (Aggregated)
Figure 15: Population-level probability and accuracy trajectories. Gray lines represent individual
sample trajectories. Red: average probability of the top token. Blue: average accuracy.
We observe two complementary mechanisms:
Statistical Cancellation for 7 < 1. Uncertain samples tend to fall into the format trap and display artificially low entropy for invalid outputs, whereas certain samples consistently lose confidence when decreases. When averaged, these opposing trends cancel out, pro- ducing a flat and stable population-level curve. This prevents the collective entropy signal from being dominated by pathological samples.
Signal Reinforcement for 1. In this region, both populations- different behaviors at the individual level-exhibit shared upward trend in accuracy as y increases. This alignment creates resonant effect: the aggregated probability and accuracy curves become smoother and strongly monotonic, reliably pointing toward the globally beneficial regime of y 1.
their very
Together, these effects explain why Unsupervised Zero Tuning succeeds when optimizing the aver- age entropy. The population average filters out idiosyneratic traps (format errors, bias amplifica- tions) and amplifies the underlying structural signal that reflects the model's latent reasoning and instruction-followiiii capabilities. This makes the minimum of the average-entropy curve over y a trustworthy indicator of the optimal tuning direction.
F
LLM OUTPUT EXAMPLE
F.1
EXAMPLES OF SST-2 DATASET
Example 1:
Classify the sentence into one of the following sentiments: positive or negative. Sentence: "hardly masterpiece, but it introduces viewers good charitable enterprise and some interesting real people."
Sentiment:
negative. Explanation: The sentence describes the movie as "hardly a masterpiece", which a negative description. The positive aspects men- tioned in the sentence are secondary to the negative description. Therefore, the overall sentiment of the sentence is negative.
Vanilla Output:
Zero Turingg Output (y 4): positive. Explanation: The word "good" in the sentence has a positive connotation, and the word "interesting" also has a positive connotation. This indicates that the overall sentiment of the sentence is positive. The word "hardly" is a negative word, but in this context, it is used to express mild criticism, it does not dominate the overall sentiment of the sentence. Therefore, the sentence is classified as having positive sentiment. Ground Truth: positive
Example 2:
Classify the sentence into one of the following sentiments: positive or negative. Sentence: "generally, clockstoppers will fulfill your wildest fantasies about being

---

Published as a conference paper at ICLR 2026
different kind of time traveler, while happily killing 94 minutes." Sentiment:
Vanilla Output: negative. Explanation: Although the sentence mentions "wildest fantasies," " which can be interpreted as positive, the overall tone is negative due the phrase "happily killing 94 minutes." Thii phrase implies that the movie is a waste of time and that the viewer will lose 94 minutes of their life watching it. This negative connotation outweighs the potential positive interpretation of "wildest fantasies," leading to a classification as negative sentiment.
ZeroTuning Output ( 4): positive. Reasoning skill: Emotion Recognition And Sentiment Analysis. The sentence is classified as positive because it uses words with positive connotations such as "wildest fantasies" and "happily", which suggest sense of excitement and enjoyment. The phrase "fulfill your wildest fantasies" implics that the experience being a clockstopper is desirable and thrilling. Additionally, the word "happily" reinforces this positive sentiment. Ground Truth: positive
Example 3:
Classify the sentence into one of the following sentiments: positive negative. Sentence: "whether you like rap music or loathe it, you can't deny either the tragic loss of two young men in the prime of their talent or the power of this movie."
Sentiment:
Vanilla Output: neutral. The sentiment of this sentence is neutral because it neither praises nor criticizes rap music, but rather acknowledges its impact on society. The sentence also expresses a sense of sadness and loss, but factual and matter-of-fact way, rather than with strong emotion. The word "tragic" is a strong word, but it is used to describe fact, rather than to express a personal opinion or feeling. Overall, the tone of the sentence is informative and objective, rather than emotional or biased
Zero Tuning Output (y 2): positive. Reasoning Skill: This question requires the ability to identify the sentiment of a sentence, which is a fundamental skill Emotion Recognition and Sentiment Analysis. The sentence presents neutral statement about rap music, but the overall tone is positive due to the mention of the "tragic loss" and the "power" of the movie, which suggests sense of admiration and respect. The correct answer requires the reader to analyze the sentence and recognize the underlying sentiment, and classify the sentence positive or negative. Ground Truth: positive
F.2
EXAMPLES OF BOOLQ DATASET
Example 1:
Read the text and answer the question by True or False
Text: Hawaii Five-O (2010 TV series, season 8) The eighth season of the CBS police procedural drama series Hawaii Five-O premiered on September 29, 2017 for the 2017-18 television season. CBS renewed the series for 23 episode eighth season on March 23, 2017. However, on November 6, 2017 CBS ordered an additional episode for the season and did the same again on February 8. 2018 bringing the count to 25 episodes. The season concluded on May 18, 2018. The eighth season ranked #18 for the 2017-18 television season and had an average 1l million viewers. The series was also renewed for ninth season. Question: will hawaii five o have a season 8?
Answer:
Vanilla Output: False. The text says the series has already concluded its 8th season. (Note: this question is not about whether the series will be renewed for
10

---

Vanilla Output: False, Serein is rain falling from a cloudless sky. Therefore, there are clouds, but they are not visible. Serein is not rain falling from com- pletely clear sky. Therefore, it is not possible for it to rain if there are clouds. Zero Tuning Output (y = 0.3): True. (According to the text, this type rain is called "serein" and falls from cloudless sky.) False. (This is common misconception, but itš not true that caní rain if there are no clouds.) (According Published as a conference paper at ICLR 2026
9th season, but whether it will have 8th season, which it has already done). (Note: the series was renewed for a 9th season, but that is not relevant to the question).
ZeroTuning Output ( = 0.3): True. (It is already confirmed by CBS). The season premiered on September 29, 2017 and concluded May 18, 2018. It was renewed for a 23 episode eighth season on March 23, 2017 and had 25 episodes in total. (The extra two episodes were ordered on November 6, 2017 and February 8, 2018). The series also ranked 18 for the 2017-18 television season and had an average of 11 million viewers.
Ground Truth: True
Example 2:
Read the text and answer the question by True False
Text: Serein (meteorology)- Serein in French refers to rain falling from a cloud- less sky. This sort of rain is said to take the form of a fine, light drizzle, typically after dusk. The name derives from French serein, meaning "serene", or "clear" (as in unclouded). An alternative etymology is from Old French serain, evening. Question: can it rain if there are no clouds?
Answer:
the text, this type of rain is called "serein" and it falls from cloudless sky.) Ground Truth: True
G
THE EFFECT OF TUNING DIFFERENT MATRICES
80.00
80.00
60.00
Accuracy (%5
Accuracy (%)
40.00
20.00
20.00
Altention Score
0.00
Kay State
0.00
Vini Baveline
4.0
1.0
1.5
Scaling Factor
Scaling Factor
(a) SST-2
(b) BoolQ
30.00
40.00
Accur acy (86)
25.00
Accuracy Accuracy (56
20.00
20.00
10.00
10.00
0.00
0.5
1.0
1.5
2.0
2.5
3.0
0.5
1.0
1.5
2,0
2.5
3.0
ing Factor
Scaling Factor
(c) AQUA
LogiQA
Figure 16: Accuracy of tuning the initial token's attention scores and key states over (a) SST-2, (b)
BoolQ, (c) AQUA, and (d) LogiQA.

---

Published as a conference paper at ICLR 2026
In certain scenarios where the attention map not explicitly computed, is challenging influence the final representation by modifying the attention weights. Therefore, we consider tuning the key query states an alternative approach. As illustrated in the Figure 16 observe that within an appropriate scaling range, tuning the key state exhibits a similar trend to tuning the attention score. However, find that directly tuning the key states is sensitive: when the sealing factor is too small or too large, the performance of the LLM drops sharply, while tuning the attention score results in more stable performance.
We now analyze the theoretical differences between applying the scaling factor to the attention scores versus the key states. To begin, we revisit and extend the attention weight formulation from Section 3.1 For sequence of length T, the attention weight for token is given by:
04 exp(zi)
(11)
Imd (xp(2m)
where 2; denotes the logit for token i, given by:
q k.
2j dd
(12)
with q E RR as the query vector, k; E Rd the key vector for token i, and de as the dimension- ality of the key vectors. Note that ag corresponds to the initial token, and E/= a, = 1.
Tuning the Attention Score As derived in Section 3.1 when tuning the attention score, the dif- ference between the attention weights of non-initial tokens i.j 2 1 becomes:
dg
04
0x
(13)
D
(7 1 aa +
Next, expand og, as and aj follows:
df a'
d/ = (7 1)a0 +1
0j aj
exp(2:) exp(2;)
Σ Itd exp(zx) (7 - Ei-o exp(=u) +
exp(so)
exp(zi) exp(2))
(14)
(v exex((2)) + Eho exp(z*)
Tuning the Key State Now, consider scaling the key state by y, i.e., Kn =7ka- This changes the logit for the initial token:
2/6 9 ( 7/20:
(15)
Vde
while the logits for other tokens remain unchanged: 2 = for i 2 1. The tuned attention weights are then:
exp(2f) Σ exp(2))
cxp(2,)
d
for i 2 1.
(16)
exp(>20) + ET-L exp(2,)'
The attention difference for non-initial tokens i,j 2 1 is derived as:
12

---

Published as a conference paper at ICLR 2026
(xp)
cxp(2))
df - -d- exp(720) + Σk=1 exp(x*)
exp(1z0) + Ek=1 exp(#*)
exp(z:) - exp(zj) (z))
(17)
exp(7=0) +E exp(=k)
The denominator in equation 14ficcudss the linear term (7 - 1) exp(20) of y, whereas the denom- inator in equation 17 contains the exponential component exp(y=o). This indicates that tuning the attention weights results in smoother effect, while tuning the key states has more abrupt impact.
Beyond the effect attention differences, the stability disparity. The final representation o' is a weighted sum of value vectors, o' = Σ a.v, The structure of these weights de dictates the stability of o'.
analyze the final representations to understand
Representation from Tuning Attention Scores. The output representation is a convex combina- tion of value vectors, as the normalized weights af sum to one. Specifically:
attn ( 1) +
yag
Vo + Σ ( 1)a0 +1 Vi.
(
ai
(18)
Crucially, the coefficients of v; are smooth rational functions of 7/ This ensures that the output representation Jam changes smoothly and its magnitude remains bounded by the magnitudes of the value vectors. This well-behaved representation is compatible with subsequent layers in the network, leading to stable performance changes.
Representation from Tuning Key & Query States. This method also produces a convex combi- nation. However, its stability undermined by the exponential nature of the weights:
Okey -( (oxp((za)) 7'
exp(2;) (2:) Z'
where Z' exp(122) Σ exp(z)). (19)
Vo
Vas
The instability arises from the exponential sensitivity of the leading coefficient to the scaling factor y. Due to the exp(720) term, the weight applied to Vo grows exponentially with y. For large values of y. this exponential amplification causes the initial token's attention weight to rapidly saturate towards 1, forcing the weights all other tokens towards 0. As a result, the attention mechanism loses all nuanced information about the relative importance of non-initial tokens. The output okyy ceases to be meaningful synthesis of context, instead collapsing to approximately Vo-
Even though the magnitude of okyy bounded, the information-impoorrsshe representation fed to subsequent layers cripples the model's reasoning ability, causing the observed sharp drop in aceu- racy.
H PERFORMANCE UNDER RESOURCE CONSTRAINTS
Computational constraints are common in real-world applications and can limit the feasibility of head classification and parameter optimization in LLMs. To investigate how well Zeronuning adapts to such conditions, we define three resource constraint levels based on available computational re- sources:
Level 0: Severely limited resources that prevent both head classification and parameter search.
Level Moderately limited resources that allow parameter search but not head classifica- tion.
Level 2: Ample resources that support both head classification and parameter search.
We evaluate Zeronnning's performance across these levels using the LLaMA-3.1-4 model. At Level 0, apply fixed scaling factors (7 2 and 0.6) to all attention heads, reflecting
13

---

Published as a conference paper at ICLR 2026
dataset-specific scaling preferences as explored in Section 3.2 Additionally, we introduce hybrid
approach at Level O, which selects the best-performing y for each dataset. At Level 1, ZeroTuning uses uniform scaling across all heads with optimized y. At Level 2, it classifies attention heads, scales only the over-mixing or under-mixing heads, and searches for the optimal
Table 5: Performance ZeroTuning Under Different Resource Constraints.
Mahhed
Classfication
Clus.
Muitiple Choice
Arg. MC
SST2
SST.5
MR
BoolQ
CB
TRBC
SUBI
MELLU
AQUA
MabiqN
LogiA
CO...
FIQ/
ARCC
Vasilla
73.20
45,40
59.20
69:60
52.14
14.00
44.00
59.73
67.40
25.69
33.60
29.40
17.60
83.60
84.52
58.54
Level (-
2)
0.8)
(Myhnd))
5395
17:30
Level
89:00
91
21
20.04
83,80
84.62
Lovel
91:60
52.00
92.00
82.40
69.29
26.30
66.00
71.44
68.80
30.43
36.60
42.80
60.40
85.40
65.95
61.48
As shown in Table [ ZeroTuniin consistently improves performance across all resource levels. Even at Level O, where resources are tightly constrained, the hybrid approach delivers steady gains over vanilla inference. These improvements become more substantial at Levels I and 2, where additional resources enable parameter optimization and head classification. Specifically, compared the vanilla baseline, ZeroTuning increases average accuracy on text classification tasks by 3.63 percentage points at Level O (Hybrid), 4.86 percentage points at Level 1, and 11.71 percentage points Level 2. For multiple-choice tasks, the corresponding gains are 0.37, 1.65, and 2.64 percentage points, respectively.
I SENSITIVITY TO DIFFERENT CONTEXT LENGTHS
To investigate how the distance between the initial token and task-relevant tokens affects model be- havior, we evaluate the sensitivity of Zero Tuning under varying context lengths. Specifically, we insert task-irrelevant tokens between the initial token and the original input to artificially extend the context. This allows us to isolate the impact of increased token distance attention and perfor- mance.
We conduct experiments using Llama-3.1-88 3.1.-..1-BBBB- ---tuc-nnttrutt and apply ZeroTuning with the same set of heads and scaling factors used in the previous base (non-extended) context setting. This ensures that any performance change is due solely increased context length rather than re-optimized tuning parameters.
As shown in Table 6 the performance of vanilla LLMS consistently degrades as context length increases, likely due to disrupted information mixing caused by the inserted tokens. In contrast, Zero remains robust across all tested context lengths, yielding consistent and often significant improvements even under extreme cases of context extension. These results suggest that tuning the initial token's attention can effectively stabilize information flow, even when relevant content is pushed further away in the input sequence.
J ROBUSTNESS ACROSS FEW-SHOT SCENARIOS
Few-shot learning has become widely adopted approach to improve the performance of LLMs by providing small number of in-context examples, enabling adaptation to specific tasks with minimal data (Brown et al.) 2020). Building on previous zero-shot evaluations, we now evaluate the robustness of Zero Tenonuning in 1-shot and 2-shot scenarios across four datasets: SST-5, BoolQ, MMLU, and AQUA. ensure consistency, we fix the randomly selected examples, maintain the selected head and scaling factor throughout the experiments.
The results in Table 7 show that ZeroTuning consistently outperforms the vanilla baseline across both 1-shot and 2-shot settings. In the I-shot scenario. ZeroTuning achieves average accuracy improvement of 1.85% over the vanilla model, with notable gains of 2.0% on BoolQ (82.40% vs. 80.40%) and 1.8% SST-5 (49.40% 47.60%). In the 2-shot scenario, the average improvement increases to 3.08%, with significant 7.12% increase on AQUA (32.81% vs. 25.69%) and 2.0% on SST-5 40% vs. 50.40%). Notably, ZeroTuning in the zero-shot setting outperforms vanilla few-
14

---

Published as a conference paper at ICLR 2026
Table 6: Impact of Context Length on Zuningning Performance.
Dataset
Method
Extra Context Length
Average
100
200
300
Vanilla
73.20
68.40
59.20
32.00
58.20
SST-2
ZeroTuning
91.60
89.20
87.40
85.40
88.40
Diff
18.40
20.80
28.20
53.40
30.20
Vanilla
69.60
68.60
67.60
68.60
68.60
BoolQ
ZeroTuning
82.40
81.80
81.40
81.20
81.70
Diff
12.80
13.20
13.80
12.60
13.10
Vanilla
39.40
36.60
36,20
35.80
37.00
LogiQA
Zeronuning
42.40
43,00
41.00
41.00
41.85
Diff
3.00
6.40
4.80
5.20
4.85
Vanilla
83.60
82.20
81.20
80.60
81.90
PIQA
ZeroTuning
85,40
83.80
83.20
82.80
83,80
Diff
1.80
1.60
2.00
2.20
1.90
shot baselines, achieving higher accuracy without the additional context overhead and contextual biases introduced by in-context examples.
Our results also highlight the following key findings:
1. ZeroTuning improves LLM performance, even when few-shot learning does not benefit the base model. Most datasets show improvements with few-shot learning, likely due to clearer patterns and better output formatting. However, some datasets, like MMLU, experience performance drops, possibly due to increased confusion from the examples. Despite this, ZeroTuning still leads consistent performance gains.
2. Similar to Few-Shot Learning, Zero'Tuning reduces invalid responses from LLMs, indicat- ing improved instruction following. For instance, in the SST-2 dataset, LLMs sometimes output incorrect responses like "neutral" in zero-shot settings when they should respond with "positive" or "negative". Few-shot learning helps the model understand the expected format, improving accuracy. Interestingly, Zero ZeroTuning Tuning also reduces these errors, suggest- ing that helps the model better understand task-relevant information.
Table 7: Comparison of Vanilla and ZeroTuning Performance Across Few-Shot Learning Scenarios.
Shot
Method
SST-5
BoolQ
MMLU
AQUA
Average
Vanilla
45.4
69.6
67.4
25.7
52.0
0-Sh Shot
Zero Zening
52.0
82.4
68.80
30.4
58,40
Diff
6.6
12.8
1,4
4.7
6.4
Vanilla
47.6
80.4
61.8
28.1
54.5
1-Shot
ZeroTuning
49.4
82.4
63.4
30.0
56.3
Diff
1.8
2.0
1.6
1.9
1.8
Vanilla
50.4
83.4
64.4
25.7
$6.0
2-Shot
ZeroTuning
52.4
85.0
66.0
32.8
59.1
Diff
2.0
1.6
1.6
7.1
3.1
K IMPACT OF DECODING STRATEGIES
Decoding strategies play a crucial role in shaping the output behavior LLMs, and can influence performance across tasks. We evaluate the robustness of ZeroTuning over three strategies: Top- k Sampling, Top-p Sampling, and Beam Search, using Llama-3. 1-8B on MMLU and SST-2, with results in Table
15

---

Published as a conference paper at ICLR 2026
Table 8:
Performance Comparison Across Decoding Strategies with and without Zero Tuning on
MMLU and SST-2.
Dataset
Method
Top-k Sampling
Top-p Sampling
Beam Search
Average
Vanilla
63.80
63.80
63.00
63.53
MMLU
ZeroTuning
65.80
66.00
65.20
65.67
Diff
2.00
2.20
2.20
2.13
Vanilla
64.40
66.60
66.60
65.87
SST-2
Zeronuning
89,20
89.60
89.20
89.33
Diff
24.80
23.00
22.60
23.47
Across all decoding strategies, ZeroTuning consistently improves over the vanilla baseline. On MMLU, it yields performance gains of 2.0% with Top-k, 2.2% with Top-p, and 2.2% with Beam Search, resulting in an average improvement 2.1%. %. On SST-2, the improvements are even more substantial: 24.8% with Top-k, 23.0% with Top-p. and 22.6% with Beam Search, with an average gain of 23.5%.
L THE EFFECT OF DIFFERENT NUMBERS OF HEADS
As shown Figure 18 we observe that tuning an appropriate proportion of attention heads leads the best performance. Specifically, Figure 18h presents results the SST-2 dataset, where tune the up-effective heads, while Figure I8p reports performance on the MMLU dataset with the down-effective heads. Across both datasets, we find that tuning moderate proportion of heads (lypically between 40% and 70%) achieves the highest accuracy. In contrast, tuning too few or too many heads tends degrade performance, suggesting that selective head tuning key to effective inference-time adaptation.
70.00
20%
20%
40%
40%
87.50
70%
70%
85.00
100%
100%
Accuracy (%)
Accuracy
82.50
66.00
80.00
64.00
77.50
62.00
72.50
70.00
60.00
1.0
1.2
2.0
0.2
0.4
0.8
1.0
Scaling Factor
Scaling Factor
(a) SST-2
(b) MMLU
Figure 17: Accuracy of tuning different proportions of heads. (a) SST-2: tuning up-effective heads; (b) MMLU: tuning down-effeccive heads.
16

---

Published as a conference paper at ICLR 2026
M SENSITIVITY TO PROMPT VARIATIONS
Prompts play a crucial role in guiding LLM behavior and typically consist of three components: Instructionl (task guidance), Question (the actual query), and Instruction2 (output format spec- ification). To evaluate the robustness of ZeroTuning under prompt perturbations, we perform ex- periments on the LLaMA-3.1-SB model using MMLU and SST-2 under three prompt formats: Full Prompt (Instructionl + Question + Choices + Instruction2), Drop Instructionl, and Modify Instruc- tion2. Detailed prompt examples are provided in Appendix O
As shown in Table 9 Zero'Tuning consistently improves performance across all prompt configu- rations, and maintains strong performance even when key instructions are modified or omitted, demonstrating its distinctive ability to regulate and adapt to prompt variations. On MMLU, the performance gains range from 1.2% to 2.4%, with an average improvement 1.7%. On SST-2, the gains are more substantial, ranging from 24.8% to 26.2%, with an average improvement of 25.3%.
Table 9: Effect of Prompt Variations on Performance with and without ZEROTUNING.
Prompt Format
Method
MMLU
SST-2
Average
Vanilla
67.40
64.40
65.90
Full Prompt
ZeroTuning
68,80
89.20
79.00
Diff
1.4
24.8
13.1
Vanilla
66.80
64.40
65.60
Drop Instructionl
ZeroTuning
68.00
89.20
78.60
Diff
1.2
24.8
13.0
Vanilla
61.80
61.80
61.80
Modify Instruction2
ZeroTuning
64.20
88.00
76.10
Diff
2.4
26.2
14.3
N THE EFFECT OF DIFFERENT QUANTIZATION CONFIGURATIONS
As shown in Figure 18 we observe:
(a) Quantizing to 8-bit results in only slight accuracy decrease compared to 16-bit, while 4-bit quantization leads to a significant accuracy decrease. However, by appropriately tuning attention to the initial token, find that the best accuracy with 8-bit quantization becomes comparable that of the 16-bit model on the SST-2 and BoolQ datasets. This suggests that our method can partially compensate for the performance loss caused by quantization.
(c) The accuracy trends across different quantization levels are largely similar. This consistency may offer useful insights for future work, for instance, searching for optimal parameters using low- precision models and transferring them to higher-precision models.
57.50
22.00
55
65.00
35
57
25:00
55.00
6:08
1.0
23
2.0
2.5
33
33
40
35:02
0.2
25
08
1-2
23
14
51:00
02
73
0.
1.3
12
1.5
3.5
20
(a) SST-2
(b) BoolQ
(c) MMLU
Figure 18: Accuracy when tuning under different quantization configurations.
17

---

Published as a conference paper at ICLR 2026
O PROMPTS USED FOR EACH DATASET
Here, we list all the prompts we used in this paper on different datasets:
For multiple-choice task, the following prompt:
Prompt for Muutiple-Choics Tasks
Generate the correct answer to the following question.
<Question>>
<choice 1>
<choice 2>
<choice 3>
Answer:"
For text classification, different prompts for different datasets.
Prompt for SST-2
"Classify the sentiment of the user's message into one of the following categories: 'positive" or 'negative'.
Sentence: <sentence>>>
Sentiment:
Prompt for SST-5
"Classify the sentiment the user's message into one the following categories: 'terrible', 'negative", 'neutral', "positive", or 'ggeat'. Sentence: <sentence>>
Sentiment:
Prompt for MR
"Classify the sentiment of the movie's review into one the following categories: 'positive' or 'negative".
Review: <sentence>> Sentiment:
Prompt for TREC
"Classify the given questions into the following categories: 'Description', 'Entity', 'Expres- sion', 'Person', 'Number', or 'Location'. Question: <sentence>>
Type:
Prompt for CB
"Read the following paragraph and determine if the hypothesis is true. Premise: <premise> Hypothesis: <hypothesis>>
Answer:
Prompt for BoolQ
"Read the text and answer the question by True or False. Text: <passage>> Question: <question>>?
Answer:
18

---

Published as a conference paper at ICLR 2026
Prompt for SUBJ
"Classify the input into one of the following categories: subjective or objective. Input: <text>
Category:
P PROMPT FOR KEY TOKENS IDENTIFICATION
Prompt for key tokens identification
Below is question. Please extract the key content words phrases from the question that are crucial understanding and answering correctly. These are typically the nouns, verbs, adjectives, multi-word expressions that define the subject, action, relation in the question. Output your selection as a Python list, where each element is a word or a phrase enclosed in quotes. For example, for the question 'What is the boiling point of water?", the key words might be ['boiling point', 'water'].
Question: {question)
Key Words:
19


Reviews:
[{"review_id": "WAiBC5KvOP", "reviewer": "Reviewer_LeHa", "summary": "The paper introduces ZeroTuning, a training-free approach that steers large language models by scaling the attention weight of the initial token (BOS). The authors argue this operation “monotonically controls” the entropy of downstream attention, and propose both supervised and unsupervised (entropy-minimizing) calibration procedures. Empirical results across 15 NLP benchmarks and multiple model families suggest measurable gains (e.g., +19.9% on classification, +4.5% on MC-QA), with additional analyses on layer and head behavior showing BOS as a dominant control point.", "questions": "Please refer to weakness section above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **Simple and broadly applicable idea**. The notion that adjusting only the first token’s attention can improve diverse tasks is conceptually elegant and easy to integrate, requiring no retraining or architectural modification.\n\n- **Comprehensive empirical coverage**. The paper evaluates across multiple datasets, models, and settings (few-shot, quantized, SDPA/FlashAttention), providing reasonable evidence of generality.\n\n- **Clarity and presentation**. The paper is clearly written and well-structured, with intuitive figures and concise mathematical derivations that make the mechanism easy to follow. Readers can quickly understand the motivation and implementation.", "weaknesses": "- **Theoretical over-reach**. The claim that BOS scaling “monotonically controls attention entropy” lacks formal proof; the derivation only handles pairwise attention differences, not entropy. This gap weakens the conceptual basis of the unsupervised variant.\n\n- **Transductive unsupervised tuning**. The unsupervised version minimizes entropy on test inputs, while baselines are not given equivalent unsupervised access, overstating generalization gains.\n\n- **No statistical robustness**. All results appear single-seed, with no confidence intervals or variance reporting. For small gains (1–3 %), the significance is uncertain.\n\n- **Head profiling risks data-set-specific overfitting; selection rules are ad-hoc**. Heads are labeled by measured response to γ and then the “dominant head type” is scaled; implementation tunes the top 40% of identified heads. Multiple-testing control, stability across resamples, and cross-dataset transfer of head labels are not demonstrated.\n\n- **Confounding with generic decoding calibration.** Reported gains may largely reflect generic logit/decoding tweaks rather than an attention-specific effect. The paper notes BOS scaling behaves like temperature, yet no matched unsupervised baselines (e.g., temperature, label-bias/length penalties) are tuned on the same unlabeled inputs. Add these controls and report invalid-output rates to isolate a genuine attention-level contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "94I1vfReca", "reviewer": "Reviewer_LN7k", "summary": "The paper introduces ZeroTuning, a training-free method to improve large language model (LLM) performance by adjusting the attention logits of the initial token (such as, \\<BOS>\\). The approach selectively scales attention heads associated with this token, requiring no parameter updates and only minimal code changes. The authors provide both supervised and unsupervised variants—respectively calibrated on validation data or optimized via output entropy minimization. Theoretically, they show that modifying the initial token’s logits can monotonically regulate the downstream attention entropy due to its inherent role as an attention sink. Empirically, ZeroTuning demonstrates consistent gains across three different model types (Llama, Gwen, Deepseek) and on several downstream tasks and maintains compatibility with various attention kernels (suchSDPA, FlashAttention) and quantized inference setups.", "questions": "- Including analysis for the model variants in section 3 would show the generalization of the claims.\n\n- It would be helpful to compare it with other task specific tuning methods such as  SFT / adapter and to show if it works well with them as well?\n\n- Minor presentation issues, a typo in Figure 5 (“shadow” label instead of \"shallow\").", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "In short: the paper identifies a control lever in large language models (LLMs) - the initial token (such as \\<BOS>\\) - and shows how modulating its attention yields performance gains. The method is practically appealing since it is lightweight (just a few lines of code to scale attention) and kernel-agnostic. \n\n\n-  Provides experimental Analysis on - how scaling the attention weight of the initial token affects the downstream distribution of attention among other tokens, experiments showing that tuning the initial token produces larger improvements than tuning other token positions, showcase layer-wise and head-wise analyses, showing how the effect varies across shallow/middle/deep layers and across individual attention heads.\n\n- Demonstrates broad empirical gains across multiple LLMs in the experiment section (such as Llama-3.1-8B, Llama-2-13B, Qwen-2-7B, DeepSeek-R1-14B) and a variety of downstream tasks (classification, QA, conversation).\n\n- The paper contribution is in interpretability/mechanistic understanding of LLMs (why the initial token works as a control point and how attention patterns propagate). \n- The writing is clear in explaining both the motivation (limitations of previous attention-tuning methods that rely on heuristics) and the method’s design.", "weaknesses": "Following are some limitations I see in the paper:\n\n - Limited model generalization: most of the analysis and findings in section 3 reply on just one model, Llama-3.1-8B-Instruct, raising concern that some of those effects may be model-specific. I would suggest to add some findings for the other models tried as well - Qwen or Deepseek, to show generality.\n\n - Huge hyperparameter tuning overhead: This method introduces a huge number of hyper parameters to tune - task specific tuning, layer wise tuning, head specific tuning. This limits its practical applicability.\n\n- Unclear task selection & possible training overlap: The paper lacks explanation for why specific evaluation tasks were chosen or do they cover diverse range of SFT based downstream tasks? I would suggest to clarify task selection rationale and check for the overlap/lack of task specific data in the pertaining and if the gains are correlated with that.\n\n- Weak justification for “bias correction” claim (Sec. 3.2): the claim that scaling (y > 1) “corrects bias” lacks empirical evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wXQtsoflnz", "reviewer": "Reviewer_Vcie", "summary": "This work introduces ZeroTuning, a training-free attention-tuning method that modifies only the initial (sink) token (e.g., BOS). By scaling that token’s attention logits with a single factor, the method adjusts the entropy/sharpness of downstream attention while preserving the relative proportions among non-initial tokens. Notably, the scaling factor that minimizes entropy empirically aligns with the factor that maximizes accuracy. Building on this, the authors provide both supervised and unsupervised (entropy-minimization) calibration variants, implemented in a kernel-agnostic way (compatible with SDPA/FlashAttention) by altering attention maps only. Extensive experiments show consistent gains across 15 datasets, with ZeroTuning outperforming prior, more complex approaches.", "questions": "1. What is the overhead for the supervised profiling process? Could you please clarify the GPU hours it will cost?\n\n2. For the unsupervised method which identifies the optimal heads and scaling factor γ by minimizing the average next-token prediction entropy, what is the actual cost? Will this significantly make the inference slow?\n\n3. For the long context generation tasks, do we need to adjust γ dynamically as well? If not, will using the adaptive values harm the performance?\n\n4. Have you ever tried to make γ as a learnable parameter? Because if it can be learned during some fine-tuning process, we can make it work in the inference stage without any overhead and more adaptable. \n\n5. What will ZeroTuning perform on math reasoning tasks and reasoning models? Will this method still demonstrate performance improvement?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work theoretically and empirically demonstrates that the initial tokens can function as a reliable controller for the attention dynamics, which is also strongly related to the next-token prediction entropy. Moreover, the systematic head-wise and layer-wise initial token scaling analysis provides more insights and reliable motivation for the proposed ZeroTuning method. \n\n2. The proposed plug-and-play attention adjustment, ZeroTuning, is simple yet effective and well-motivated both empirically and theoretically. Its supervised and unsupervised calibration variants are easy to implement and attention kernel-agnostic. Overall, its analysis and effectiveness inspires a fresh look at the role of the initial sink token in shaping attention.\n\n3. This extensive experiments demonstrate that ZeroTuning can achieve consistent gains across different models and downstream tasks including text classification, domain-specific multiple choice and multi-round conversation, outperforming the previous methods including PASTA and ACT with negligible engineering overhead. \n\n4. The writing is clear and easy to follow. The paper is well structured.", "weaknesses": "1. While Table 5 shows that ZeroTuning improves even with fixed γ and scales with more search, the paper does not quantify the time/energy required for Level-0/1/2 nor its trade-off with accuracy. \n\n2. γ is calibrated per dataset, and its robustness to distribution shifts and mis-specified γ is unclear, and the cost of head classification is not reported as well.\n\n3. Because ZeroTuning controls attention by scaling the initial sink token, its effect can fade or fluctuate over very long contexts and in streaming with KV-reuse, and the same γ induces different sharpness across sliding windows, causing calibration drift or confidence oscillations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HkqwMoLqo0", "reviewer": "Reviewer_Nnho", "summary": "This paper introduces ZeroTuning, a training-free method to improve the performance of frozen LLMs by making a simple yet powerful intervention: applying lightweight, head-specific attention adjustments only to the initial token (e.g., the <BOS> token).", "questions": "The paper mentions that ZeroTuning requires only \"four lines of code,\" but could the authors quantify its runtime impact? How does the inference latency and memory footprint compare to the vanilla model, and how does this overhead scale with context length?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Training-Free & Parameter-Efficient**: It improves model performance without updating any weights.\n\n**Theoretical Grounding**: The method is supported by a theoretical insight linking the initial token's attention to the entropy of the entire attention distribution, providing a principled foundation for the intervention.", "weaknesses": "**Sensitivity in Alternative Implementations**: The paper shows that tuning the key/query states directly (as a kernel-agnostic alternative) leads to a much sharper and more sensitive performance drop outside the optimal range compared to tuning attention scores, making it a less stable implementation choice.\n\n**Calibration Overhead**: The supervised variant requires a labeled validation set (500 examples in their setup) to calibrate the optimal scaling factors. Although the unsupervised variant (using entropy minimization) is a significant contribution, the paper's main results and comparisons are based on the supervised mode, which still incurs a data and computation cost for calibration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)