Distribution-Aligned Decoding for Efficient LLM Task Adaptation
Authors: Senkang Hu
Xudong Han
Jinqi Jiang
Yihang Tao
Zihan Fang
Yong Dai
Sam Kwong
Yuguang Fang
Venue: NeurIPS 2025 poster ()
Keywords: large language models
task adaptation
parameter-effcient fine-tuning
LLM decoding
Primary area: deep_learning

Abstract: Adapting billion-parameter language models to a downstream task is still costly, even with parameter-efficient fine-tuning (PEFT). We re-cast task adaptation as output-distribution alignment: the objective is to steer the output distribution toward the task distribution directly during decoding rather than indirectly through weight updates. Building on this view, we introduce Steering Vector Decoding (SVDecode), a lightweight, PEFT-compatible, and theoretically grounded method. We start with a short warm-start fine-tune and extract a task-aware steering vector from the Kullback-Leibler (KL) divergence gradient between the output distribution of the warm-started and pre-trained models. This steering vector is then used to guide the decoding process to steer the model's output distribution towards the task distribution. We theoretically prove that SVDecode is first-order equivalent to the gradient step of full fine-tuning and derive a globally optimal solution for the strength of the steering vector. Across three tasks and nine benchmarks, SVDecode paired with four standard PEFT methods improves multiple-choice accuracy by up to 5 percentage points and open-ended truthfulness by 2 percentage points, with similar gains (1-2 percentage points) on commonsense datasets without adding trainable parameters beyond the PEFT adapter. SVDecode thus offers a lightweight, theoretically grounded path to stronger task adaptation for large language models.

Distribution-Aligned Decoding for Efficient LLM Task
Adaptation
Senkang 12:+, Xudong Han Jinqi Jiang', Yihang Tao Zihan Fang 1.2 Yong Dai Sam Tak Wu Kwong", Yuguang Fang 1.2.1
Hong Kong JC STEM Lab of Smart City, City University of Hong Kong, 3 UUniversity of Sussex, "Huazhong University of Science and Technology, Fuudn University, Lingnan University
Esenkang.foress yihang. tommy, zihanfang3-c) xh2180sussex. uk. (annijjjan6677@mmmm samkwong@ln. edu. hk, my Fanggaci tyu.
cityyu.eue edu. hk, com, daiyongya@outllook com, edu.hk wyw.u.ityu.edu.h.kk
Abstract
Adapting billion-parameen parameter language models to downstream task is still costly, even with parameter-efficiee fine-tuning (PEFT). We re-cast task adaptation as output-distribution alignment: the objective is to steer the output distribution toward the task distribution directly during decoding rather than indirectly through weight updates. Building on this view, introduce Steering Vector Decoding (SVDecode), a lightweight, PEFT-compatible, and theoretically grounded method. We start with short warm-start fine-tune and extract task-aware steering vector from the Kullback-Leibler (KI.) divergence gradient between the output distribution the warm-started and pre-trained models. This steering vector is then used guide the decoding process to steer the model's output distribution towards the task distribution. We theoretically prove that SVDecode is first-order equivalent the gradient step of full fine-tuning and derive globally optimal solution for the strength of the steering vector. Across three tasks and nine benchmarks, SVDecode paired with four standard PEFT methods improves multiple-choice accuracy by up to percentage points and open-ended truthfulness by percentage points, with similar gains (1-2 percentage points) on commonsense datasets without adding trainable parameters beyond the PEFT adapter. SVDecode thus offers lightweight, theoretically grounded path to stronger task adaptation for large language models.
Introdcution
Large language models (LLMs) [1, 2. 4, are pivotal in AI, marking early steps towards artificial general intelligence (AGI). They excel in tasks like language understanding, generation, and transla- tion, transforming natural language processing with their grasp of context and human intent. Models like DeepSeek-RI [4] and OpenAl ol [5] demonstrate strong reasoning and multimodal capabilities, respectively. Specialized models such as EmoLLMs [6], LMDrive [7], and AnomalyGPT [8] address specific downstream tasks like affective instructions, autonomous driving, and anomaly detection. Despite their capabilities, LLMs are resource- intnueeeeeteesivee often requiring hundreds millions to billions of parameters. For instance, training a LLaMA-7B model demands at least 58 GB of memory [9], which is beyond the capacity of consumer-grade hardware like the NVIDIA RTX 4090 with 24GB, limiting their broader applications.
"qqual contribution Correspnnding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To tackle this challenge, trramieer-efficient fine-tuning (PEFT) [10, 11, 12, 13, 14, 15] has emerged as key area of progress in modifying LLMs with minimal computational and GPU memory demands. This approach focuses on updating a few trainable parameters to significantly reduce the memory footprint while enhancing the performance on downstream tasks. For example, additive fine-tuning such as prompt tuning [16] and adapter methods [17] incorporates a small set of trainable parameters while maintaining the original pre-trained parameters unchanged. Selective fine-tuning such as Diff' Pruning [18] chooses subset of the model's existing parameters to undergo updates during training. Reparameterization methods such as LoRA [15] restructure the model's parameters to achieve efficient low-rank representations.
However, while PEFT methods effectively reduce the cost of training adaptation, the adaptation process itself is still primarily viewed through the lens of modifying model weights to change the model's 's output distribution to match the task-specific target distribution, which requires backward passes, optimizing states, and multiple training epochs.
Why do We Still Chase the Weights? The end goal of adaptation is not to adjust internal tensors. is to shift the model's 's output distribution so that Po(y x)) aligns with the task-specific target. Current PEFT methods achieve this indirectly: they adjust weights in the hope that the logits will follow. However, this indirect approach leads to three practical issues: 1) training still scales linearly in model size and data epochs: 2) weight updates can have unpredictable, non-local effects on token probabilities; and 3) a fixed PEFT hyper-paramenn
often fails to transfer across tasks and domains.
A Distribution-Firs, Perspective. To answer this question, we propose shift in perspective, rethinking task adaptation not just as a weight-update problem but fundamentally as a process of aligning the model's output distribution with the task-specific target distribution. We argue that adaptation can be achieved more directly and efficiently by manipulating the output distribution during the decoding phase itself.
Steering Vector Decoding (SVDecode). To achieve this goal, we present Steering Vector Decoding (SVDecode), innovative, efficient, and PEFT- CEFT-compatible method for task adaptation. SVDecode begins with a short warm-start fine-tuning phase to obtain the warm warm-started model, whose output distribution is closer to the task-specific target compared to the pre-trained model. Then we can capture the task-specific direction from the differences between output distributions of the warm- started model and the pre-trained model. Specifically, these two distributions, and then use the negative gradient of the KL divergence to construct the steering signal. Next, this signal is mapped from the distribution space to the logit space to avoid simplex geometry violation and yields a task-aware steering vector that tells us which tokens need more (or less) probability mass and by how much. Additionally, confidence-aware constraints are applied to the steering vector to ensure its robustness and stability. Finally, the steering vector is used to adjust the model's logits at each step during decoding, effectively steering the generation process towards the desired task behavior. Because the vector is applied during decoding, no additional backward pass is required, and the method is compatible with any existing PEFT methods.
first compute the KL divergence between
Our contributions are summarized as follows: 1) We rethink LLM task adaptation from the per- spective of output distribution alignment. 2) We propose the SVDecode method, which leverages negative gradients of KL divergence between distributions to construct task-aware steering vectors for decoding-tinn adaptation. 3) We provide theoretical analysis, linking SVDecode to traditional PEFT methods and derive an analytical solution for the optimal steering strength. 4) We demonstrate through extensive experiments on various tasks and models that SVDecode, when combined with standard PEFT techniques, consistently improves performance while maintaining computational efficiency.
2 Rethinking LLM Task Adaptation from the Perspective of Output Distribution Alignment
Large language models (LLMs) define a conditional output distribution over tokens or task labels. parameterized by 6, as Po(y 2) Softmax (fo (z)), where y = fo( is the logits vector produced by the model for input x. Fine-tuning adapts the model to a downstream task by updating such that the model's output distribution better aligns with the task-specific target distribution. Specifically, given downstream dataset Drask = ((r,, y1) ) fine-tuning adjusts 0 by minimizing the loss
function on the dataset Drask. The standard fine-tuning objective is the negative log- likeeloolooo (NLL):
2

---

(Fr( -((e.)) [log Po x)]-
(I)
This is exactly the cross-entropy between the model's output distribution and the empirical one-hot distribution of the correct tokens. Minimizing this encourages the model to assign higher probability to the correct token each position. In the special case where each yi is single label (e.g. for classification). this formula reduces to log Polys æt), the usual cross-entropy for single-label prediction.
Theorem 1. The NLL objective in Eq. (1) is equivalent to minimizing the expected Kullback-Leible (KL) divergence between the empirical label distribution Praek(y x)) and the model's 's output distribution:
(FT BEND KL (Prsak(y)
Po =))]
where Prasi (y) x)) typically a delta fanction centered on the ground-truth label.
Proof. Let (y y | x) = 5g (v) be the empirical distribution over labels for input 2i where Sy, (v) if y = Yi and otherwise. The KL divergence between the empirical distribution and the model's predicted distribution is defined as:
Penst (1 (x) Po x))
KL. (Pruste a:) Poly x))
-E
Praste (y (x) log
(3)
NEY
where {81 12.... } the vocabulary set of the model. Since Prack(y) x)) is a delta function, only the true label = Yi contributes:
KL (Pussly x4) (y m.)) log Polyi x;) log Po xz).
1
(4)
Taking the expectation over all samples in the dataset yields:
ERND KL (Prssk(y x)) || Po(y
x))]
- log Po x4)]
(5)
which matches the definition of the average negative log- lig-likeddooo as in Eq. (1). Hence, the NLL objective is equivalent to minimizing the expected KI, divergence between the task label distribution and the model's output distribution.
Distributional Interpretation. From the output distribution perspective, fine-tuning reshapes the model's belief (v(lx) over the output space to align more closely with the true task-specific behavior. The output distribution (o(2z) resides on the probability simplex 1 [19], and fine-tuning can be seen as shifting the model's position on this simplex toward the optimal region defined by the task. Minimizing the KL divergence from the empirical distribution emphasizes increasing the probability mass on the correct label without penalizing overconfidence in incorrect predictions. This yields learning dynamic that is both efficient and focused.
3 Method
In this section, we will introduce the details of the proposed method. As shown Fig. the proposed method includes two steps. The first step is to construct the steering vector, which is the core of the proposed method. It includes several steps, warm-start, KL gradient as steering signal, logit-space projection, and confidence-ane vector decoding, which leverages the steering vector to steer the model's output distribution with the optimal steering strength for different tasks. The detailed algorithm is shown in Algorithm 1.
steering vector constraint. The second step is task-aware steering

---

Specific Direction
Steering Voctor
Pre-tramm Outout
ned Outrut
Becoding (SVD)
Dittibution
Dirbbution
Warm harled Output
Warmetante Oulpo
Distribution
Distribution
Task Distribution
Task Distribution
Steering Vector Constructioin
(b) Steering Vector Decoding
Figure 1: Illustration of the framework of our proposed SVDecode. includes two steps: (a) steering vector construction and (b) task-aware steering vector decoding. After the decoding with the steering vector, we can see the warm-started model's 's output distribution steered towards the task-specific target distribution, thereby enhancing the performance of the model on the downstream task.
3.1 Steering Vector Construction
Warm-Start, In order to construct the steering vector, we first need to know the task task-specific direction of the steering vector. Specifically, given pre-trained LLM with the parameter 0, the model defines conditional probability Po xx) over output text y given input r. y = (y² yy is sequence II Po x, y where y <t denotes the sequence previous tokens, x the input tokens. The model's prediction for each token is usually given by softmax layer producing (o x, 5t) over the vocabulary at that position.
of T tokens, this typically factorizes autoregressively as: Po | x)
Given a downstream dataset Drask {( yı Ji then we warm-start the model by fine-tuning one epoch in Drask or part of the dataset. This warm-start process can leverage different parameter- efficient fine-tuning strategies, such as additive fine-tuning, selective fine-tuning, and reparametriza- tion fine-tuning discussed Section A.1.
KL Gradient as Steering Signal. After the warm-start process, the model's conditional output distribution can be formulated (v z)) where is the updated parameters, and we believe that the model's output distribution (v x) is close to the task-specific target distribution Prask x) compared with the pre-trane pre-trained distribution (y 2) since the warm-started model's training loss decreases and the test accuracy increases.
Then we can leverage the KL divergence to measure the difference between the pre-trained distribution Po(y æ) and the warm-start distribution (s r). Before we do this, we need to know that the KL divergence is not symmetric, i.e., (L(M N) f KL(N||M unless the two distributions are identical (in which case both are 0). If we use KL KL Pa x)) to measure the difference, it means that assume that the pre-trained model knows more about the task than the warm- started model, and we want to steer the model's output distribution towards the pre-trained model, which is not what expect. On the contrary, if difference, it means that we assume that the warm-started model knows more about the task than the pre-trained model, and we want to steer the model's 's output distribution towards the task sask-spec target distribution. Therefore, we use the following KL divergence to measure the distributional difference:
x)]l Po(y x)) measure the
Po ②) Po
KL (2) x)) Σ Poly xx) log BEy
(6)
construct the steering vector. First, we need to compute the gradient of the KL divergence with respect to Po(y x). denoted as gp, which V Vp KL KL (y x) Pely x)). For clarity, let Po Pa(y x), P. Pa(y x). Then KL(P, || Po) = E, , (Po log Pe P& log Po). We compute the gradient with respect to Pa (i.e., V Pr KT. by taking the partial derivative of KL with respect each Pa:
After obtaining the KL divergence, we can
aKL
Po.. Pa.m
(Po.g: log Po.y Po.y log Pa.y.) ) = log
+1
OP JPo.y
aPo.

---

Therefore, the gradient VP, K || Po) is a vector where each component corresponds to the partial
derivative with respect specific py²
Po.y
PPa.v2
Po Po.y ael
+1
Kp.KL(PP = log
+ 1, log
+ ,1....,,og
(8)
Po.e
P0.02
Powwa
The meaning of this gradient is that indicates how we should adjust P. to reduce the KL divergence: 1) For a token where Po.y. > Po.g.. the gradient component log(Pa...//)) Po...))))) + is positive, suggesting we should decrease the probability of this token; 2) For token y/ where Po... < Pa.. the gradient component is negative, suggesting we should increase the probability of this token. In other words, this gradient points to the direction of returning to the pre-trained distribution Po. Conversely, if we the negative of this gradient our steering vector, it represents the direction of task-specific knowledge that the warm-started model has acquired relative to the pre-trained model. This task-aware steering vector captures the distributional shift needed to adapt the pre-traiee pre-trained model to the specific downstream task.
Logit-Space Projection. We can leverage the negative gradient of the KL divergence with respect to the output distribution, rr, (L || Po). as a task-aware steering vector. This gradient points to the direction that decreases the divergence between the fine-tuned model and the pre-trained model, and thus encodes the local task aas--specfiic adjustment direction the distributional space.
The simplest approach is to directly apply this vector to adjust the decoding distribution:
P(y 2) (1 (11))) p1) Poly + (-Vr,KL(ZZ Po))
(9)
This aims to move Pe. closer to the task-optimal distribution Puzsk along the steepest descent direction of KI. divergence. However, this method introduces several practical issues: 1) Normalization Constraint: Since (y x) is probability distribution over the vocabulary, the adjusted distribution P must satisfy Σ, Ply| x) = Directly adding the gradient vector may violate this constraint, requiring techniques such Lagrangian optimization or projected gradient descent to ensure nor- malization. 2) Numerical Stability: The gradient involves logarithmic terms log Pa (v)) PP((), which can be numerically unstable when Po (y) or Po(y) are close to zero To mitigate this, one may apply clipping (e.g.. minimum threshold) or smoothing techniques (e.g., adding e) to stabilize the computation. 3) Simplex Geometry Violation: The KL gradient defined in the Euclidean tangent space of the probability simplex. Without proper geometric projection. applying this vector may lead to invalid probability values, such as negative entries or totals not summing to one.
Therefore, while the KL gradient in the probability space provides an informative direction for reducing divergence from the pre-trained distribution, its direct application in decoding is hindered by constraints and numerical issues. To resolve this, we shift to the logit space, where the model is parameterized and unconstrained. By leveraging the chain rule, we can project the KL gradient from probability space into logit space via the softmax Jacobian matrix:
iigiiss = J p- KL(P, Pb)) = (diag(Pa) - -PPP.) (-log
Po Po
-1)
(10)
This projected vector &iogits serves as a task-aware logit delta, which can be added to the original logits before softmax:
S 2Φ pa Stogits, P = Softmax(25)
(11)
This approach preserves the normalization constraint by construction and enables fine- tne-grained control of task-specific adaptation in the model's output distribution.
Confidence-Awan Steering Vector Constraint. Although the projected task-aware logit delta óiogits captures the KL gradient direction in logit space, it can still be dominated by false positive tokens-tokens that are not semantically relevant but receive large KL gradients due to numerical instability (e.g., when (o()) is extremely small). To mitigate this, we introduce a confidence-aware filtering mechanism to suppress the influence of low-confidence tokens.
We define the confidence of each token y E V at a decoding step as its predicted probability under the task-adapted model s(y) = (v | x). Let y"
arg maxyev (o(y) ar) denote the most likely token. Then we introduce a threshold a E (0, 1) to retain only the confident tokens which have a probability greater than times the probability of the most likely token. The binary mask T(y) is defined as:
1(y) 1 (P)())
Pv))))
(12)
5

---

We now mask the logit delta by element-wise applying the confidence mask and penalty:
Grogits( (v) [vy) Singits (v) + (1 I(()) X,
(13)
where À a constant penalty term (e.g.. À = 0, 1, or a small negative value). This constraint ensures that only confident (high-probabilityy) tokens contribute to the task steering vector, while suppressing noise from low-probability regions that are numerically unstable or semantically irrelevant.
3.2 Task-Aware Steering Vector Decoding
Logit Adjustment. In the decoding process, we first compute the logits ( for each token y E V using the task-adapted model Po(y æ). Then, we apply the task-aware steering vector with the confidence mask constraint to steer the logits towards the task-specific direction. The adjusted logits for decoding then:
2y(y) zạ(y) f (io zι(y) + p. (I(y) Srogits (v) + (1- (1-1(y)) A)
(14)
where le E R is a scalar to control the strength of the steering vector. Finally, we apply the softmax function to the adjusted logits to get the adjusted distribution of the output tokens:
P(y x) Softmax (z)
(15)
After we get the adjusted distribution,, we can leverage different decoding strategies to generate the final output tokens, such as greedy decoding, beam search, and top-k sampling.
Optimal μl as Newton Step. The value of μ is an important hyperparameter that controls the strength of the steering vector. If μu is too small, the steering vector will have little effect the decoding process. If is too large, the steering vector will dominate the decoding process, and the model will be more likely produce incorrect results. Previous works fixed μ for all tasks, but here we can derive the optimal for each task. Specifically, denoting the distribution of the downstream task as Prast (y x)) we want the distribution of the model's 's output as (y x)) to be as close as possible to Praste (y x). that is: finding a f such that the final distribution approximates the task
label distribution as closely as possible.
To derive μt, we first expand the KL divergence around μ = 0 to obtain the second-order Taylor series:
(I (ppp) (L + / (V (LPusk((() 6z) 12
(16)
where H(6.]] 6% V2 (T.(Prasteele (1))) 0- is the quadratic form of the Hessian. To find the optimal
step length μ that minimizes (L Px)), we ignore the constant zeroth-orde term and the higher- order terms O(u²), and consider only the first two orders. Then take the derivative of these two terms with respect to μ and set it to zero:
dy ( ( (Vz, KT. task (lpo), 0=)) + -=²H((6)) = 0.
d
(17)
Solving for n. we get:
KL(P (ennk) pa)), 5z)
task
1
(18)
H(
which is the exact Newton step. For one-hot ground-truie truth label y", the task distribution is Ptask (y) 1y=y", and the gradient of the KL divergence is V., KL (L llpo) = Ps Ey', where ey- is the one-hot basis vector for y². Substituting this into the expression for μ²' gives:
1 (ps eyr, dz)
(19)
H[82]
This derivation shows that Jt" is the negative ratio of the linear term to the quadratic term in the Taylor expansion. The exact Newton step requires computing the Hessian H(02). However, computing the full Hessian is expensive. We therefore adopt the common Gauss-Newton approximation [20] H|8z] se || (which is exact for a quadratic loss function), yielding
(ey. Po. 8.)
1
(20)
18_
1 +
where small (e.g., 10-12) prevents division by zero when | 12 is tiny. Finally, we can calculate a global optimal j by averaging the token-level j=" over a calibration dataset. The detailed derivation and algorithm can be found in Appendix C and D.

---

Table Experimental results on multiple-choice task in TruthfulQA and open-ended generation task in TruthfulQA. %TxI stands for % IIfo in TruthfulQA.
Multiple- Choice (%)
Open-Ended Generation (%)
Model
Method
MC1
MC2
MC3 f
Avg-
%Truth
%Info f
&T=1 f
Avg.
Prompt Tuning
29.88
43.02
19.22
30.71
28.04
32.32
24.39
28.25
SVDecode
28.66
44.47
21.79
31.64
28.66
33.70
25.34
29.23
IA3
40.85
47.28
27.51
38.55
32.31
32,93
28.65
31.30
SVDecode
42.19
55.67
34.04
43.97
34.15
33.87
29.87
32.63
Qwen2.5-1.55
P-Tuning v2
33.54
45.28
23.45
34.09
31.70
33,53
27.44
30.89
SVDecode
33.54
48.41
25.96
35.97
32.32
32.32
28.05
30,90
LoRA
50.61
55.55
34.81
46.99
49.39
43,90
40.85
44.71
SVDecode
52.94
61.41
34.95
49.77
50.00
44.52
42.68
45.73
Prompt Tuning
51.95
49.34
35.17
45.49
64.02
62.19
56.10
60.77
SVDecnde
53.25
62.16
35.45
50.29
65.24
62,80
57.92
61.99
IA3 SVDecode
47.56
50.36
31.89
43.27
52.44
55.48
48.78
52.23
46.07
57.04
31.99
45.03
54.26
55.48
50,00
53,25
Qwen2.5-7B
P- P-Tuning v2 SVDecode
46.95
50.23
33.08
43.42
62.19 19
67.07
59, 14
62.80
48.78
59.35
35.09
47.74
64,63
67.68
60.97
64.43
LoRA
49.39
51.31
32.82
44.51
54.89
49.39
46.34
50.21
SVDecode
50.61
58.33
34.47
47.80
55.48
50.61
46.95
51.01
Prompt Tuning
35.37
43.11
22.43
33.64
36.58
32.32
28.55
SVDecnde
29.61
55.06
30.64
38.44
37.90
33.54
28.66
33.37
IA3
34.76
45.83
24.85
35.15
43.90
47.56
39.63
43.70
SVDecode
30.49
54.73
31.89
39.04
44.51
40.23
LLaMA3.1-8B
P. Tuning v2
38.41
46. 14
25.91
36.82
48.17
48.78
42.07
46.34
SVDecode
31.71
49.52
25.97
35.73
48.78
50.12
43.68
47.53
LoRA
46.34
49.12
33,20
42.89
51.21
44.51
41.63
45.78
SVDecode
48.17
60.17
35.07
47.80
51.82
45.12
42.68
46.54
4 Experiments
4.1 Experimental Setup
Tasks and Datasets. In order to evaluate the performance of our method, we consider three tasks:
1. Maltiple-Choice Tasks. For multiple-choice and open-ended generation tasks, evaluate on the TruthfulQA dataset [21], which is a benchmark designed to measure a model's tendency to generate truthful answers to questions. We consider three metrics in this task: MC1, MC2, and MC3. The detailed definitions of these metrics are shown in Appendix E.3.
2. Open-Ended Generation Tasks. For open-ended generation tasks, we also evaluate on the TruthfulQA dataset [21]. We consider four metrics in this task: Truthfulness, Informative- ness, Truthfulness & Informativeness:. The detailed definitions of these metries shown in Appendix E.3.
3. Commonsensse Reasoning Tasks. For commonsense reasoning tasks, we leverage eight datasets including BoolQ [22]. PIQA [23], SIQA [24], HellaSwag [25], WinoGrande [26], ARC-easy [27], ARC-challenge [27] and OBQA [28], and we leverage accuracy as the metric. The implementation details are shown in Appendix E.5.
Base Models and PEFT Methods. We consider four latest pre-trained LLMs:: Qwen2.5-1.5B [2]. Qwen2.5-7B [2]. LLaMA3-8B [1], and LLLMA3.1--88 [1] as the base models. In addition, we leverage four PEFT methods to incorporate method: LoRA [29], P-Tuning v2 [30], Prompt Tuning [16], and IA3 [31]. We eloborate the implementation details in Appendix E.
4.2 Main Results
Multiple-Choice Tasks. Table shows that our approach consistently outperforms baseline PEFT methods. For 1555, SVDecode with LoRA improves scores from 46.9 46.99% 49.77%. For Qwen2.5-7B, SVD.e, SVDecode with Prompt Tuning increases scores from 45,49% to 50.29%. For

---

Table 2: Experimental results on commonsense reasoning tasks. We evaluate different PEFT methods and our proposed SVDecode method on Qwen2.5-7B and LLaMA3. 1-BB.
Model
Method
BoolQ
PIQA
SIQA
HellaS.
WinoG:
ARC-e
ARC-c
OBQA
Avg.
LoRA
59.12
85.71
68.57
78.10
58.79
91.00
82.57
79.77
75.45
+ SVDecode
60.09
86,97
70.13
79.23
59.67
93.33
85.62
81.43
77.06
IA3
71.23
86.61
75.41
89.05
67.22
88,00
81.60
81.54
80.08
Qwen2.5-7B
+ SVDecode
72.69
87.23
76.72
90.31
68.41
92.67
85.12
82.07
81.90
Prompi Tuning
64.00
86.58
67.54
73.30
60.64
83.28
72.02
68.36
71.97
+ SVDecude
65.67
87.21
67.79
75.42
62.35
84.05
72.68
69.67
73.11
P- P-Tuning v2
$9.65
83.67
69.00
78.66
59.00
92.32
81.65
79.18
75.39
+ SVDecode
60.71
84.10
71.36
79.72
59.48
92.60
82.33
81.04
76.42
LoRA + SVDecode
74.18
83.21
79.56
95.00
87.92
91.86
83.67
88.52
85. ,49
74.74
84.10
80.31
95.48
88.65
92.45
83.98
89,43
86.14
IA3
69.84
83.67
68.22
85.33
69.00
87.83
73.90
78.01
76.97
LLaMA3.1-8B
+ SVDecode
70.32
84,20
68.75
69.29
88.10
74.66
78.77
77.52
Prompt Tuning
67.64
80,33
64.67
79.58
62.34
83.57
70.33
74.26
72.84
SVDecode
68.35
82.00
65.00
80,39
63.07
84.63
71.00
75.41
73.73
P- Tuning
65.33
81.55
66.30
82.42
64.48
87.40
73.56
73.80
74.35
SVDecode
66.12
82.65
67.54
83.58
65.67
87.68
74.32
75.17
75.34
37.
150
375
150
FELL
D1)
141 MELS
560
50
LLaMA3.1-8D
Qwen2.5-7B
Figure 2: Ablation study logit-space projection. 'w/" means with logit-space projection, 'w/o" means without logit-space projection. 'Prompt' means Prompt Tuning, and 'P-T' means P-Tuning v2. We conduct the ablation study multiple-choice tasks.
LLaMA3. SVDecode with LoRA boosts scores from 42.89% to 47.80%. Despite occasional MCI score drops, MC2 and MC3 improvements ensure overall better performance, highlighting SVDecode's 's effectiveness in enhancing truthful answer selection.
Open-Ended Generation Tasks. Table shows that our approach improves performance across all datasets. For Qwen2.5-1.55 1.552, SVDecode with LoRA increases the score from 44.71 % to 45.73%. For Qwen2.5-7B SVDecode with P-Tuning v2 raises the from 62.80% to 64.43%. For ALaMA3.1-8B. SVDecode with LoRA boosts the score from 45.78% 46.54%. This demonstrates SVDe- code's 's effectiveness in enhancing model responses.
Table 3: Study on the absence of confidence-aware constraint. 'w/" means with and 'w/o" means without confidence-awant constraint. The PEFT method is LoRA.
Qwen2.5-7B
&Truth
%Info
%
55.48
50.61
46.95
Commonsense Reasoning Tasks. Table 2 shows that our approach consistently improves the accuracy of all PEFT baselines across multiple commonsense reasoning datasets and models. The improvements observed for every method and model, demonstrating the effectiveness and generalizability of our approach in adapting commonsense reasoning tasks.
w/o
0.02
0.01
0.00
4.3 Ablation Study
Logit-Space Projection Ablation Study. Figure illustrates the impact of logit-space projection on model performance. The study compares results with and without logit-space projection across multiple-choice tasks, highlighting differences in accuracy for I.LMMA.1-SRP and Qwen2.5-7B models. From the figure, we can that without logit-space projection, the performance of the

---

Table 4: failure example of the absence of confidence-aweee aware steering vector constraint. 'w/o' means without confidence-awaid aware steering vector constraint, "w/" means with confidence-aware steering vector constraint.
Question
What fundamental principle of human psychology is established by Amy Cuddy's a power poses experiments?
w/o
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Answer
w/
Amy Cuddy's power poses experimenta established the principle that adopting expansive, powerful posturea can increase feelings of power and confidence.
Table 5: Integrating SVDecode with four basic decoding strategies, Greedy Search, Beam Search, Top-p sampling, and Top-k sampling, where Beam-4 indicates using 4 beams. We evaluate proposed SVDecode method on Qwen2.5-7B.
Model
Method
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg.
Greedy
59.12
85.71
68.57
78.10
58.79
91.00
82.57
79.77
75.45
+VDDecode
60.09
86.97
70.13
79.23
59.67
93.33
85.62
81.43
77.06
Beam-4
61.45
88.53
70.45
79.66
60.54
92.17
85.26
82.80
77.61
Qwen2.5-7B
+VDecode
62.16
89.31
71.82
80.71
61.12
94.19
87.10
84.26
78.83
Top-p 59.87
85.80
69.24
78.30
59.13
91.15
82.70
79.80
75.75
SVDecode
60.79
87.00
70.13
79.82
59.89
93.40
85.69
81.47
77.27
Top-k 60.12
86.11
69.76
78.75
59.64
91.63
83.15
80.24
76.
+SVDecode
60.93
87.10
70.35
79.90
60.36
93.56
86.31
81.90
77.55
method drops in all metrics and across all PEFT methods, some of them even drop 10% in accuracy. These results indicate that logit-space projection is crucial for the performance of the method.
Ablation Study Confidence-
Steering Vector Constraint. Table 4 presents qualitative example of the absence of confidence-aware steering vector constraint. As shown in the table, without the confidence-awar constraint, the model generates repetitive and meaningless sequences of exclamation marks, indicating a complete loss of control in the generation process. Table presents the results on the absence of confidence-aware constraint. As shown in the table, without the confidence-awand constraint, the model is failed to generate a meaningful and controlled response. These results indicate that the confidence-aware steering vector constraint is crucial and indispensable for the proposed method.
Study the Influence of the Warm-Start Steps. In this section, we study the influence of the warm-start steps the performance of the proposed method. From Figure 3, we can that method continuously outperforms the warm-started model. In addition, we ob- serve interesting phenomenon that after the warm-started model converges after 5 epochs, our method still continues to improve the perfor-
Figure 3: Analysis of warm-star steps. The task is multiple-chicee task, the PEFT method is LoRA, and the base model is LLaMA3.1-88
5834
Average Accuracy (9)
SVD
56.00
55
5224
52.10
30.22
mance of the warm-started model.
45
Integrated With Different Basic Decoding Strategies. Table 5 presents experimental re- sults for commonsense reasoning tasks, exam- ining the integration SVDecode with various decoding strategies including Greedy Search, Beam Search, Top-p sampling, and Top-k sampling. The results demonstrate that incorporating SVDecode consistently enhances the performance of fine- tuned LLMSs commonsense reasoning datasets, irrespective of the underlying decoding approach used.
Epoch
5 Conclusion
In this paper, we have re-framed L.MM task adaptation as a problem output-distribution alignment. Building on this perspective, we have introduced Steering Vector Decoding (SVDecode), lightweight,

---

PEFT-compatible method that can consistently improve performance across wide range of tasks and model sizes via adjusting the decoding distribution by the task-specific steering vector with a global optimal steering strength. In addition, we have proved the equivalence between SVDecode and the gradient step fine-tuning, thereby grounding the method the classical optimization theory. In summary, SVDecode offers lightweight, theoretically grounded, and empirically validated path toward stronger LLM task adaptation, bridging the gap between gradient- based fine-tuning and decoding-tiiee control of model behavior, and demonstrating that shifting distributions, not weights, can be the shortest route to better performance.
6
Acknowledgement
The research work described in this paper was conducted in the JC STEM Lab of Smart City funded by The Hong Kong Jockey Club Charities Trust under Contract 2023-0108. The work was supported in part by the Hong Kong SAR Government under the Global STEM Professorship and Research Talent Hub. The work of S. Hu was supported in part by the Hong Kong Innovation and Technology Commission under InnoHK Project CIMDA.
References
[1] Ac@Meta. Llama 3 Model Card. 2024.
[2] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, and et al. Qwen2 Technical Report. arXiv preprint arXiv:2407.10671 2024.
[3] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, and et al. DeepSeek-V3 Technical Report, 2025. arXiv preprint 194447241..19437.
[4] DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, and et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. arXiv preprint MxXxv:2501.12945
[5] OpenAL, Aaron Jaech, and et al. OpenAl ol System Card, 2024. arXiv preprint arXiv:2412.16720
[6] Zhiwei Liu, Kailai Yang, Qianqian Xie, Tianlin Zhang, and Sophia Ananiadou. EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysis. In Pro- ceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, page 5487-5496, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704901. doi: 10.1 142.1444/7775588667555555
[7] Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven I. Waslander, Yu Liu, and Hongsheng Li. LMDrive: Closed-Loop End-to-End Driving with Large Language Models. In 2024 IERE/CV Conference on Computer Vision and Puttern Recognition (CVPR), pages 15120-15130, 2024. doi:
10. 1..10999VRR52233.222..00
[8] Zhaopeng Gu, Bingke Zhu. Guibo Zhu. Yingying Chen, Ming Tang, Jinqiao Wang. AnomalyGPT: detecting industrial anomalies using large vision-language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium Educational Advances in Artificial Intelligence, AAAI244AAAT24FAAAT AAAI Press, 2024, ISBN 977-1-57735-887-9 doi: 10. (0.1609/aaii.v333 v3813. 27963.
[9] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: memory-efficient LLM training by gradient low-rank projection. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
[10] Senkang Hu, Yanan Ma, Yihang Tao, Zhengru Fang, Zihan Fang. Yigin Deng, Sam Kwong, and Yuguang Fang. Task- Aware Parameter-Efficient Fine- Tuning of Large Pre-Trained Models at the Edge, 2025.
[11] Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shanghang Zhang. Gradient-basse Parameter Selection for Efficient Fine- Tunnng, In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 28566 28577, Seattle, WA, USA, June 2024. IEEE. ISBN 979-8-3503
8-303--5300----- 5300-66.--
doi: 10. 12L1109/CVVV22233222440000022
10

---

[12] Yizhe Xiong, Hui Chen, Tianxiang Hao, Zijia Lin, Jungong Han. Yuesong Zhang, Guoxin Wang, Yongjun Bao, and Guiguang Ding. PYRA: Parallel Yielding Re-activation for Training- Inference Efficient Task Adaptation. In Ales Leonardis. Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision ECCV 2024, volume 15067, pages 455 4733. Springer Nature Switzerland, Cham, 2025. ISBN 108-3-031-726722 978-3-031-72677 6. doi: 10. 12110007978---000 3-031-72673 72673-62227 6_25, Series Title: Lecture Notes in Computer Science.
[13] Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable Sparse Fine-Tuning for Cross- Lingual Transfer. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1778- 1796. Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10.18653/v1/ 2022.acl-long. 125.
[14] Haoyu Hc, Jianfci Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Scnsitivity- Visual Parameter- Efficient Fine-Tuning In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11791-11801, Paris, France, October 2023. IEEE, ISBN 979-8-3503-0718-44 doi: 10.1 N0.1109/CCC510000 2023.01086.
[15] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low- Rank Adaptation of Large Language Models. In International Conference on Learning Representations. 2022.
[16] Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter--ifficien Prompt Tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pages 3045- 3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 16653/v1/2021.0 1e0.021emnlp--ae m4...243.
[17] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim. Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lec. LLM-Adapters: An Adapter Family for Frramcter-Effiiiee Fine-Tuning of Large Language Models. In Houda Bouamor, Juan Pino, and Kalika Bali, aditors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2554,276,, Singapore, December 2023. Association for Computational Linguistics. doi: (0.18853/v112000 1/t03//1/22222// hwtp2/222...mmmmmmmmee emnlp-main.11..
[18] Demi Guo, Alexander Rush, and Yoon Kim. Parameter-
Transfer Learning with Diff Pruning. In Chengging Zong. Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4884 4896, Online, August 2021. Association for Computational Linguisties. doi:
122.22..cllloooo...1 lo8
[19] Karl Heinz Borgwardt. The simplex method: a probabilistic analysis, volume 1. Springer Science & Business Media, 2012.
[20] F. Dan Foresee and M.T. Hagan. Gauss-newton approximation to bayesian learning. Proceedings of International Conference on Neural Networks (ICNN'97), volume 3, pages 1930-1935 vol.3, 1997. doi: 10. N11111/ICNN1999..0 614194.
[21] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Mcasuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 3214-3252 Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10.18653/v1/ 2022.acl-long.229.
[22] Christopher Clark, Kenton Lee. Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ:
Exploring the Surprising Difficulty of Natural Yes/No Questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis, Minnesota, 2019, Association for Computational Linguistics. doi: 10. 1653/v
[23] Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Phys- ical Commonsense Natural Language. In The Thirty-Fourd Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secome Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-1 2020, pages 1432-7439 AAAI Press, 2020.
11

---

[24] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense Reasoning about Social Interactions. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing TEMNLP-IJCNLP. pages 4463 4473, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v 1/D1// 1/D19-1454.
[25] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi, HellaSwag: Can Machine Really Finish Your Sentence? In Anna Korhonen, David Traum, and Lluís Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-48000 Florence. Italy, 2019. Association for Computational Linguistics. doi: 10. 11663//1111911777
[26] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In The Thirty- Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Phirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8740. AAAI Press, 2020.
[27] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge, 2018.
[28] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujüi, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391 Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 11886//1/D11-11111
[29] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang. and Weizhu Chen. IORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS. 2022.
[30] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 61-68, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 118633v1/20222 -ch-sotr....
[31] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-effiiient fine-tuning is better and cheaper than in-context learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NTPS '22, Red Hook, NY. USA, 2022. Curran Associates Inc. ISBN 9781713871088.
[32] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Strameter-Efficiien Fine- Tine-Tuning for Large Models: A Comprehensive Survey, 2024, arXiv preprint arXiv:2403. 14608.
[33] Neil Houlshy, Andrei Giurgiu, Stanislaw Jastrzehski, Bruna Morrone, Quentin De I Larousiilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Efficieet Transfer Learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 09-15 Jun 2019.
[34] Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6998-7010, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 118853/v122233emmm/mm
[35] Baohao Liao, Yan Meng. and Christof Monz. Parameter- Efficcent Fine-Tuning without Introducing New Latency. Anna Rogers. Jordan Boyd-Graber. and Naoaki Okazaki. editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 4242- 4260, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.1865.10 11163//v12222... acl- acl-
[36] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoye.. Intrinsic Dimensionality Explains the Effective- ness Language Model Fine-Tuning In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference on Natural Language Processing (Volune 1: Long Papers), pages 7319 7328, Online, August 2021. Association for Computational Linguistics. doi: 10. 10..8853/v Laco://l-oo..o.oooooooogooo 1o8......66.
12

---

[37] Yin Cui. Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large Scale Fine-Grained Categorization and Speeif-Speccfic Transfer Learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 4109-4118, 2018. doi: 1091 2018.00432.
[38] Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data Valuation using Reinforcement Learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pages 10842-10851. PMLR, 13-18
Jul 2020.
[39] Anh Tran, Cuong Nguyen, and Tal Hassner. Transferability and Hardness of Supervised Classification Tasks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pages 1395-1405, 2019. doi: 10.100//CC..2001000000
[40] Cuong Nguyen. Tal Hassner, Matthias Seeger, and Cedrie Archambeau. LEEP: A New Measure to Evaluate Transferability of I. Learned Representations, In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7294-7305. PMLR, 13-18 Jul 2020.
[41] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the Effectiveness of Parameter- Partiieter-Eficciettt Fine-Tuning. Proceedings of the AAAI Conference Artificial Intelligence, 37(11):1779--12800 June 2023, ISSN 2374-3468, 1159. doi: 10. ww..6699aaai.v31... 26505.
[42] Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to Sequence Learning with Neural Networks. In 7. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 27, Curran Associates, Inc., 2014.
[43] Angela Fan, Mike Lewis, and Yann Dauphin. In Tryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-899 Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 18653/v1/P18- 1082.
[44] Ari Holtzman, Jan Buys, Li Du. Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[45] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner. Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive Decoding: Open-ended Text Generation as Optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 12286- -23112, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653/v1/20233 aaclaallllnsse lonel-loo.....
[46] Ariel Gera, Roni Friedman, Ofir Arviv, Chulaka Gunasekara, Benjamin Sznajder. Noam Slonim, and Eyal Shnarch. The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational I. Linguistics ( (Volume I: Long Papers), pages 10406-10420, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653/ (01p5//v1/202...0 acl- acl-long..500
[47] Yung-Sun Sung Chuang. Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. DoLa: De- coding by Contrasting Layers Improves Factuality in Large Language Models. In The Tweifth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[48] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Qizhe Xie. Self-Evaluation Guided Beam Search for Reasoning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
[49] Cheng Han Chiang and Hung-yi Lee. Can Large Language Models Be an Alternttiv Human Evaluations? Anna Rogers, Jordan Boyd-Graber, Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607-15631, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 16653//110023.ccllmaa
[50] Cheng- Han Chiang and Hung-yi Lee. A Closer Look into Using Large Language Models for Automatic Evaluation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8928 8942, Singapore, December 2023. Association for Computational Linguistics. doi: 10.11853//1/2022 http://////222...innnnemeemmemm findings-emmll..59
13

---

[51] Xinhao Xu, Hui Chen, Zijia Lin, Jungong Han, Lixing Gong, Guoxin Wang, Yongjun Bao, and Guiguang Ding. TaD: A Plug-and-Plas Play Task-Aware Decoding Method Better Adapt LLMs Downstream Tasks. Proceedings of the Thirty- Thindnnttnemtnnn Joint Conference on Artificial Intelligence, pages 6587-6588 6596, Jeju, South Korea, August 2024, International Joint Conferences on Artificial Intelligence Organization, ISBN 978-1 1 1566792-04-1,
doi: 0..24963/jeai.22227772
14

---

Contents
Introdcution
2
Rethinking LLM Task Adaptation from the Perspective of Output Distribution Align- ment
3
Method
3.1 Steering Vector Construction
3.2 Task-Aware Steering Vector Decoding
4
Experiments
4.1
Experimental Setup
4.2 Main Results
4.3
Ablation Study
8
5 Conclusion
Acknowledgement
10
A Related Work
16
A.1 Parameter-Efficien Fine- MininT
16
A.2 LLM Task Adaptation
16
A.3 LLM Decoding Strategies
17
Mathematical Analysis: Equivalence Between SVDecode and Fine- TenenTunig
17
B.1 Notation and Preliminaries
17
B.2 Fine-Tuning Gradient Logit Space
18
B.3 One SVDecode Step in Logit Space
19
B.4
First-Order Equivalence Theorem
19
B.5 Conditions for Higher-Order Equivalence
20
One-Token Derivation of Optimal Steering Strength
20
C.I Setup.
20
C.2 First-Order Taylor Expansion of KL
20
C.3 Optimal Step Length (Newton Approximation)
20
D One-Token and Dataset-Leevl Derivation of the Offline Steering Strength
21
D.I Per-token optimal strength u4,,
D.2 From Tokens to Global Constant p
D.3 Extension sequences (T > 1)
22
Experiment Implementation Details
23
E.1 Implementation Details of SVDecode
23
15

---

E.2
Hyperparameters for PEFT Methods
23
E.3
Evaluation Metrics
23
E,4
Implementation Details on Multiple- Choice Tasks and Open- Open-Ended Generation Tasks
26
E.5
Implementation Details on Commonsense Reasoning Tasks
26
E,6
Details about DeepSeek- DeepSeck-V3-0324 V3-0324 Evaluation
26
More Experiment Results
27
F.I
More Results on Commonsense Reasoning Tasks
27
F2 Comparison with Other Decoding Adaptation Methods
27
F3 The Influence of the Parameter in the Confidence-ANee Aware Constraint.
27
G Limitations and Future Work
28
H Practical Impact
28
A Related Work
A.1 Parameter-Efffiient Fine-Tunm Tuning
As language models continue increase in scale, traditional full fine-tuning approaches have be- come increasingly resource-intensive. Parameter-efficien fine-tuning (PEFT) emerges as a practical alternative to address these computational constraints [11, 12, 13, 14, 15]. According to [32], PEFT techniques generally fall into three distinct categories: 1) Additive Fine- Fine-Tuming Tuming, which incorporates limited set of trainable parameters while maintaining the original pre- pre-trained parameters unchanged. Notable examples in this category include Adapter modules [33] and Prompt Tuming, which inte- grates learnable soft prompts into the input. Though effective, these approaches typically introduce additional computational demands during inference. 2) Selective Fine-Tuning, which focuses on updating only carefully chosen subset the model's existing parameters [18, 34,35) 35]. This strategy employs binary mask M to selectively determine which parameters undergo updates during training, thereby avoiding the introduction new parameters. 3) Reparameterization, which restructures the model's parameters achieve efficient low-rank representations [14, 15, 36]. A prominent example LoRA [15], which factorizes weight updates into products of smaller matrices, facilitating compact storage of task-specifi adaptations. SPT [14] enhances performance by combining sparse tuning with LoRA techniques, achieving leading results in visual PEFT applications. Nevertheless, many existing approaches still lack sufficient aws-enssssssss in their parameter selection mechanisms.
A.2 LLM Task Adaptation
Researchers have developed diverse approaches adapt pre-trained LLMSs for downstream tasks by investigating optimal configurations of pre-training data, model architecture, and weight selection. For instance, Cui et al. [37] leverage Earth Mover's Distance to select the top K most relevant classes from the source domain for task-specift pre-training. Yoon et al. [38] apply reinforcement learning techniques determine appropriate weights for source domain classes. Other approaches assess model transferability to target domains by examining inter-class covariance relationships between source data and target classes, or by analyzing conditional cross-entropy between source and target labels [39, 40]. More recent research focuses on identifying which pre-trained model weights to fine-tune while keeping others frozen [11, 14, 41]. Zhang et al. [11] introduced gradient-based parameter selection (GPS), method that uses gradients to identify optimal tunable weights. Fu et al. [41] developed second-order approximation method (SAM) that approximates the Hessian matrix in the loss function's 's second-order Taylor expansion to enhance weight selection precision.
16

---

A.3 LLM Decoding Strategies
Decoding methods play pivotal role in shaping the output characteristics of large language models, determining both the fluency and creativity of generated text. The most basic Greedy Search approach has been observed frequently fall into repetitive patterns due to its myopic selection strategy. More advanced methods like Beam Search [42] have addressed this limitation by exploring multiple potential sequences simultaneouslyy, though increased computational cost. Compared greedy search's 's tendency for repetitive outputs and beam search's limited diversity, Top-k sampling [43] introduces diversity by sampling from a fixed-size set of most probable tokens, while Top-p sampling [44] further improves adaptability by dynamically adjusting the candidate set based on probability mass. Moreover, the introduction of Contrastive Decoding (CD) [45] marks a breakthrough by utilizing comparative analysis between models of different scales to improve generation quality. Building on this foundation, Anticipative Contrastive Decoding (ACD) [46] introduced layer-wise contrastive mechanisms within a single model, while Decoding by Contrasting Layers (DoLa) [47] has advanced the field further through dynamic layer selection algorithms. Guided Decoding (GD) [48] uses the model's ' self-evaluation score as a criterion to control the quality of each step and demonstrates higher consistency and robustness in multi-step reasoning. These decoding approaches, however, predominantly emphasize optimizing pre-trained LLM performance without accounting for model transformations that occur during fine-tuning. As a result, they inadequately leverage the task-specific adaptations acquired through fine-tuning processes, which often leads to suboptimal performance gains or even degradation when implemented with fine-tuned LLMs on downstream applications. On the contrary, our method focuses LLM task adaptation via decoding with task-aware steered vectors, thereby enhancing the performance of LLMs in downstream tasks.
B Mathematical Analysis: Equivalence Between SVDecode and Fine- Tine-Tuning
In this appendix, we prove that a first-order Steering- Vector- Dector-Decoding (SVDecode) step is equivalent, in expectation, to one parameter-update step of makiiuu--lkkelihon fine-tuning.
B.1 Notation and Preliminaries
Let =0(x) E RIV be the logits produced by the pre-trained LLM for input and po(y x) Softmax 20 (x)). After a short warm-start fine-tuning, the parameters are o, giving logits 24 and distribution P Softmax(2Ø).
For downstream dataset Drack = ((1)) the standard fine-tuning objective is the expected negative log- likelihoodd (NLL):
(FT( - -(sra))D Plog x)],
(21)
which is equivalent to (Prask (Ipe) up to an additive constant, where Prask is the empirical one-hot distribution.
KL-Gradient Steering Signal.
Section 3.1 derives the task-aware direction in probability space as
gp v (T((lelpo)) - [log( Po 1].
(22)
po
which is then projected into logit space via the softmax Jacobian:
J(ps) = diag(ps) - P.P..
(23)
Thus, the logit-space steering vector is:
6, = =J(ps) gp = (diag(ps) - PoPP)
-log( Po 1).
(24)
pa
17

---

Algorithm Twsk-Aware Steering Vector for LLM Decoding
Require:
1: Pre-trained LLM with parameters 0: ())l)
2: Downstream task dataset Drask { y4 (.)::.
3: Confidence threshold a E (0, 1] for token filtering.
4: Penalty value À (typically -oo) for low-confidence tokens
Ensure: Task-adapted decoding strategy with steered logits
5: Stage 1: Warm-Start Fine-tuning
> Initialize task-specific parameter distribution
6: Split Drask into training set Dirain and calibration set Dealib
7: Initialize task-specific parameters Φ 0
Start from pre-trained parameters > Typically epoch sufficient
8: Fine-tune model on Drain to obtain Ø
9: function
Po. Pa)
Get base model probabilities: pe - Po (x)
11:
Get fine-tuned model probabilities: pe Pa(-1z)
12:
Stage 2: Steering Vector Construction
Capture task-specific direction Measure distribution mismatch
13:
Compute KL gradient: 9P +- - -log(pp/pp) +
14:
Compute softmax Jacobian: ((ps) + diag(ps) PoPo
15:
Transform probability gradient logit space Stage 3: Apply Confidence-Aware Constraint & Filter noise and focus on high-confidence
Project to logit space: 8_ ((ps) gr
16:
tokens
17:
Identify most likely token: y" + arg Pax
18:
Get threshold probability: Pthresh a (y"
19:
Create confidence mask: I(v) + 1 (pp(y) 2 Pthresh)
D Binary mask for tokens above
threshold
20:
Apply mask: (v) + -( + (1- (())) AD Apply penalty to low-confidence tokens > Return filtered steering vector
21:
return 8,
22: end function
23: Stage 4: Calculate Optimal Steering Strength
Calibrate μ using labeled data 24: Compute global steering constant f from calibration dataset Dealib & See Algorithm 2 for details
25: Stage 5: Decoding with Steering Vector Generate task-specific outputs at inference time
26: function STEERDECOD ING(I, Pa. Pa. (t)
27: Initialize generated sequence: y
28:
for each decoding step until completion do
29:
Get current context: Z2 (Ir)
30:
Compute task adapted logits: ZΦ.t + Logits of Po(+)z))
31:
8., < CONFUTEESTEERRNCVEECTOR
Po. Po)
Adjust logits: 2Φ.t t- 2 + p..8 88,
Apply steering
33:
Compute adjusted distribution: p. + Softmax(2
34:
Sample/select next token y/. according to chosen decoding strategy
35:
Append to sequence: <- [y, yc] end for
36:
37:
return y end function
Complete generated sequence
38:
39:
return SteERDeCODING function
>> Return configured decoding function for inference
B.2
Fine-Tuning Gradient in Logit Space
For a single training pair (I, y²) the NLL loss
((=) logr(y" xx)
ex Σg e
log (Softmax(2g)) 2y-
-log
(25)
-ny + log Σ 30
Its gradient w.r.t. logits is
V p C
(26)
18

---

where ey- is the one-hot vector for the correct token. Taking the dataset expectation:
V LrT = (x. Ptask ]-
(27)
Because Prask is one-hot, this is the steepest-deseent direction that moves Ps toward the true task distribution.
Derivation of the Gradient To derive Vzl = Cy²; we compute the partial derivative of {(2) with respect to each logit mg:
Partial derivative of - If k y/,, then This can be written as -le lks where [ey-] is the k-th component of the one-hot vector Cyr
81 ((=,,-) -1. If + y", then
8 0(-a,))
Beu
dzz
chhh 2. Partial derivative of log Ey c.v: The derivative Eyew = Pk. where p = Softmax(2)) (2).
Combining these, the partial derivative is:
df(=) pk le
(28)
02k
Thus, the gradient vector is given by Eq. 26.
B.3 One SVDecode Step in Logit Space
SVDecode perturbs logits during decoding by
z 2s + 62,
p = Softmax(3),
(29)
where 8 is given in Eq. 24 and μ E R is a scalar strength (estimated in Appendix C). Because the perturbation is before the softmax, is always valid probability distribution.
B.4 First-Order Equivalence Theorem
Theorem Let x fixed and pol- |
x) be the warm-started distribution. For any label
distribution g()) and any strength denote the KL divergence after one SVDecode step by
K(H) = (Sfttmaa0 + +(d2)).
(30)
Then we have
aK((p)
(31)
0p ===
In particular, i Prust and pæ is obtained by any fine-tuning algorithm that has converged to (p)(g/ppp)--- 0), then (k( / O. Hence infinitesimal SVDecode
stationary point (V KL. [Pp)
step leaves the fine-tuning objective unchanged up ((p2).
Proof. By the chain rule,
Ok(μ) OK. 0 ak
OK
(26 [ō²)
8,
Ōμ D5 Op
0z
Ōμ
05
(32)
<V () 5z).
where p Softmax(2), == 2 n. de. When μ 0, p Pv. we have
OK((u)
-V:KTT 0μ ==0
== 6z).
(33)
(Ve, (L(g)pp)), (2))
which gives the same result as Eq. 31. If ps is at stationary point of KL(gll:), the inner product vanishes. A full Taylor expansion shows K(μ)) KC(0) + (2²))
19

---

Interpretation. Eq. 31 states that the first-order of the KL objective reacts to an SVDecode step exactly as it reacts to gradient step of fine-tuning. Therefore SVDecode and fine-tuning are locally equivalent in the space of output distributions, even though one edits logits at decode time while the other edits weights during training.
B.5 Conditions for Higher-Order Equivalence
Theorem 2 guarantees local equivalence. Exact global equivalence holds when (i) the steering direction lies in the span of the fine-tuning gradient subspace across all imputs, and (ii) j follows the continuous- tieerooou---eeeeee ordinary differential equation ju(t) = 17 μ² (t) for learning-rate 1). In practice we use single discrete step per token, which is sufficient to capture the empirical gains reported in Section 4.
Therefore, SVDecode can be viewed executed logit space with provable first-order equivalence but without the memory or time overhead of pack-propagation. This theoretical link explains why combining SVDecode with any PEFT method consistently improves performance while preserving efficiency.
on-the-fly proxy for gradient step of fine-tuning,
C One-Token Derivation of Optimal Steering Strength Jr
C.1 Setup.
Let 2g E | be the warm-started logits for an input x and let Pa = Softmax (z²). Denote by óg the task-aware steering vector obtained from the probjian-projected KL gradient Eq. 10. At decode time we can form perturbed logits:
Zn = 2Φ + p. 82, Pp = Softmax(zz))
(34)
Our goal is to choose scalar strength that locally reduces the true objective (L (lpp)) as much as possible, while keeping the computation lightweight.
C.2 First-Order Taylor Expansion of KL
Because a single decoding step is small, expand the KI. divergence around 1 O:
(T.(Puuslllpa) ST.(Ptskkllpg + μ <Ves (Pteck [loo); 5z) + 2 H[S.] + 0(1²), (35)
linear
where H[S.] = 5 22 2g KL, (p&) 6g is the quadratic form of the Hessian.
C.3 Optimal Step Length (Newton Approximation)
To find the optimal step length μt, that minimizes KT.(Ptakk (lp,), ignore the constant zeroth-order term and the higher-order terms O(12), and consider only the first two orders:
f(u) = - (V (L IIpo), 5z + 2 H[[6)
(36)
Then, we take the derivative of f11) with respect to i and set it to zero:
d dy 1(1) (V KL(KLLLas tack (pp). 8z) + [H 0.
(37)
Solving for μs we get:
V KL Peask P&). 0z)
n²
(38)
H[._]
which is the exact Newton step. For one-hot ground-truth label y the task distribution is Prask = 1 [v=y")> }. and the gradient of the KI. divergence is formulated as follows which can be recalled from
Eq. 26:
V. 24 (task (pt) Ps Ey"
(39)
20

---

where ey- is the one-hot basis vector for y". Substituting this into the expression for μ² gives:
1r (p& - Cy-, dz) H|62]
(40)
This derivation shows that the optimal Jr" is the negative ratio of the linear term to the quadratic term the Taylor expansion. The exact Newton step requires computing the Hessian H[6z]. How- ever, computing the full Hessian is expensive. We therefore adopt the common Gauss-Newton approximation H[ōz] ss ||óz|12 (which is exact for a quadratic loss), yielding
(cy- Pa. 6z) +
(41)
16
where small (e.g. 10-12) prevents division by zero when ||6,|12 is tiny.
shift (eg- - PP) onto the steering direction 8g; the scalar ratio tells us how far to move along d₂ so that the first-order drop in KL is maximal. If ||6=||2 < e we fall back to small default jmmin (e.g. 10-4) or simply skip steering for that token.
Interpretation. Eq. 41 projects the desired probability-
D One-Token and Dataset-Level Derivation of the Offline Steering Strength ji
derive here a two-stage procedure: 1) compute the per-token optimal strength μ on labelled calibration split (training validation set), and 2) aggregate these values into single, task-specifie constant fi that reused for all decoding steps at test time. The detailed algorithm is shown in Algorithm 2.
D.1 Per-token optimal strength 14,,
For sentence and position let 28,1. RIVI be the warm-started logits, Ps (oftmax(22gs.))) and y E V the ground-truth token. The Jacobian-projected KL-gradieet steering vector 82,, is given by Eq. 10.
Notation.
Local KL Objective.
We seek scalar that decreases
(L, 1|oftmmxxxxt + μ &
(42)
where Eyr, the one-hot target distribution.
Gauss-Newton Step. first-order Taylor expansion around μ 0 combined with the Gauss-
Newton Hessian approximation [|ōz, 11 yields the optimal step length:
(ewi, Po.itt 62,,)
(43)
115., 11 +
where c is a small constant to prevent division by zero. This is identical in form to Eq. 41.
D.2 From Tokens Global Constant j
Because the calibration split provides the true labels, we can evaluate Eq. 43 for every token whose prediction is made by the warm-started model. Let S denote this collection of indices (i, t). The simplest unbiased estimator is the arithmetic mean:
Σ
15,5-
(44)
S
(1,1)S
However, if the distribution of pt, is heavy-tailed, we can replace the mean by the median or trimmed mean:
i = median (1 or i
Σ Mi.
(45)
S,
(i,))S
21

---

Algorithm 2 Computing the Global Steering Constant jz
Require:
1: Pre-trained LLM (y(l))
2: Warm-started (fine-tuned) LLM Pa(u(æ)
3: Task-specific labeled dataset Dealib = (:) for calibration
4: Confidence threshold a
5: Small constant to prevent division by zero
Ensure: Task-specific global steering constant jı
6: function CONPUUTTTOKENTTTERRIVVVEVVTETTTT
p6)
7:
Compute KL-gradient: gP +- [pog(pp/ pp) + 1]
8: Compute softmax Jacobian: (p&) + diag(ps) PoPs
9: Project to logit space: 8g - (pp) gp
10: return 8²
end function
12:
function COOPP//TCOOOFIEEEEEEEAAAEEEESSSS
P&r a,,
13:
Identify most likely token: y + ma Po (y)
14:
Create confidence mask: 1(y) < 1(pp()) a (p
Apply mask: (g) + I(y) (v) (1 I(y))
16:
return _
17:
end function
18: function COMPUTETOxEEwIISEMM(C. Yr. Ps. Po)
19:
Initialize empty set S +- / to collect token-level M7, values for each token position / in sequence do
20:
21:
Get model logits: Z..i. Logits of Pa for (Ii, yi,ss) Compute probabilities: Po.i. + Softmax(Za...)
22:
Get probabilities from base model: pe Po(g/21,,1.4
24:
Get ground truth token: VL, + 31[t]
25:
Create one-hot distribution: Eyr, + OneHot(y;,,)
26:
8e. -onprrrEDeeeSTTTTT...oovorrroooD
27:
8 + CONPOTTCOOWDDDDINGAAAVVEEEEETEET
Po,i, -00)
28:
Compute optimal token-level strength:
(Ev;, Po..... )s..))
29:
PT, 16_,, 113++
Add to collection: S -u(((tt,tt.)
31: end for
32: return
33: end function
Initialize empty collection
35: for each sample (z,, 9:) in Dealib do
36: S. - COMPUTTETOKKI..SSSMO(.. yo. Ps. Po)
37: S
38: end for
39: Compute mean steering strength: + (s) Eesn
40:
return
where S, = {i, t) : |AC, ml < <T is a central r-trimmed subset around the median m. In all our experiments we adopt the plain mean formulated in Eq. 44, which already works well.
This scalar j not only captures the dominant shift dictated by the task distribution but also preserves the key advantage of SVDecode: run-time optimisation loop and per-token label needed.
D.3 Extension to sequences (T >>> 1)
The derivation in Appendix D.I treats one decoding step in isolation. For an autoregressive sequence ........94 yr) the joint likelihood factorises Po(y æ) I Po.c æ, yer), the sequence- y (V1....,
22

---

level KL objective is
T (Ltataskl1a = -E (e lIp..). 1=1
Because each term depends only on its local logits 20. the first-order "Newton in n" argument extends verbatim: the optimal global strength that minimises the quadratic approximation of the total KL is
(cxz Potes 622)
Pi.T
(46)
||ōzz 112 Te
which is the token-wise numerator and denominator from Eq. 43 summed over t. The Gauss-Newton Hessian remains block-diagonal, so cross-time Jacobian terms cancel in the same first-order limit.
E
Experiment Implementation Details
E.1
Implementation Details of SVDecode
In this section, we provide the implementation details of the SVDecode method. It is summarized in the following Table
Table 6: Implementation Details of SVDecode.
Parameter
Value/Setting
Warm-start Steps (Epochs)
or in Confidence-aware Constraint À in Confidence-aware Constraint
0.1 -inf Greedy Search
Default Decoding Strategy
E.2
Hyperparameters for PEFT Methods
In this section, provide the hyperparameters for the PEFT methods used in the experiments, including LoRA, IA3, Prompt Tuning, and P-Tuming v2. The hyperparameters are summarized in Table
E.3 Evaluation Metrics
In order to evaluate the performance of our method on multiple-choice tasks, we consider MC1, MC2, and MC3. MC1 measures the accuracy single-best-asweer questions, MC2 measures the accuracy multiple-correctanswwee questions based on picking any correct answer as the top choice, and MC3 normalized total probability assigned all correct answers multiple-correct-anse questions, measuring overall preference for the true set. Here we provide the mathematical formulations for the MC1, MC2, and MC3 metrics in TruthfulQA.
Consider multiple-choice question g with possible answer choices. Let A, = {a1, a2. ax) be the set of answer choices, and C2 C A, be the subset of correct answer choices. Let I, AA C2 be the subset incorrect answer choices. Let P(a. (q) be the probability assigned by the language model to answer choice for question q. Typically, these probabilities are normalized using softmax over all choices for question q, Σ P(a.l4) 1. In addition, let II(-) be the indicator function, which is if the condition inside is true, and 0 otherwise. Let @best (g) = arg max,, P(a: (q) be the answer choice assigned the highest probability by the model for question q. (We assume ties are broken consistently, c.g., randomly or by picking the first). Then the metrics are defined as follows:
MC1 (Single-True Accuracy): This metric calculated only over the subset of questions QMe1 S where there is exactly correct answer (i.c.. |Cyl 1). It measures the
23

---

Table 7: Hyperparameters for PEFT Methods. Here, Prompt means Prompt Tuning, P-7 means
P-Tuning v2.
Parameter
LoRA
IA3
Prompt
P-T
LoRA Rank
8
LoRA a
16
LoRA Dropout
0.1
Num Virtual Tokens
20 False
20
Prefix Projection
Encoder Hidden Size
128
Encoder Num Layers
Target Modules
q_rroj. (Qwen/llama: k k_proj, v v_proj. o_proj)
v_proj q projj
q_proj, k_proj, v_proj.
o_proj, down_proj.
up up_proj (Ilama) /__rro_.
k p_roo, v_proj. p_pooj, fel, fc2 (other)
Feedforward Modules
down_proo) up_proj
(Ilama) / fcl, fc2 (other)
Learning Rate
5e-5
5e-5
5e-5
5e-5
Epochs
1
Train Batch Size
1
1
Eval Batch Size
2
2
Max Seq Length
512
512
512
512
FP16
True
True
True
True
fraction of these questions where the model assigns the highest probability to the single correct answer. defined as:
MC1
I[abest (g) E C.y)
(47)
1Qacil
qEQNC
2. MC2 (Multi-True Accuracy): This metric is typically caleulated over all questions Q (or designated subset QMC2/3 that includes both single- and multi-true questions, where Cgl > 1). It measures the fraction of questions where the model assigns the highest probability any of the correct answers. It is defined
MC2 =
Σ 9EQ
T(abeat (1) E Cy))
(48)
Q
3. MC3 (Multi-True Normalized Probability): This metric is calculated over the same set of questions MC2 (Q QMC2/3)- For each question, it calculates the of probabilities assigned to all correct answers. The final score is the average of these sums over all questions. It is defined as:
MC3
Q Σ gEQ a.EC,
Σ
P(arla)
(49)
To evaluate the performance of our method open-ended generation tasks, consider Truthfulness, Informativeness, and Truthfulness Informativeness. Unlike the multiple-choice metrics (MC1, MC2, MC3) which calculated directly from model output probabilities, the metrics for the generation task (Truthfulness. Informativeness, Truthfulness & Informativeness) rely on external judgments of the generated answers. These judgments typically binary (0 or 1) and often come from human evaluators or trained classifier models. In our experiments, we use DeepSeek-V3-0324 V3-0324 [3] the external judge.
Consider question and the generated answer agen Let Jr (Agen (g)) E {o, 1] be the judgment function for Truthfulness. It returns 1 if the 'answer' is judged truthful, and 0 otherwise. Let

---

Table 8: The data template of each dataset used create commonsense reasoning data for parameter efficient tinitt.ning..
Dataset
Fine-tuning Data Template
Please answer the following question with true or false, question: [QUESTION]
BoolQ
Answer format: true/false
the correct answer is [ANSWER]
Please choose the correct solution to the question: [QUESTION]
Solution1: [SOLUTIION_]
PIQA
Solution2: (SOLUTION 2]
Answer format: solution 1/solution2
the correct answer is [ANSWER]
Please choose the correct answer to the question: [QUESTION]
Answerl: [ANSWER_]
Answer2: [ANSWER
SIQA
Answer3: [ANSWER
Answer format: answerl /answer2/nswerr
the correct unswer is [ANSWER]
Please choose the correct ending complete given sentence: [ACTIVITY JACTIVITY_LABEL]: [CONTEXT]
Endingl: [ENDING_]]
Ending2:
[ENDING 2]
HellaSwag
Ending3:
[ENDING_] 3]
Ending4:
[ENDING_L]
Answer format: ending 2eddingg//odding2n2dnnnneenXiei
the correct answer is [ANSWER]
WinoGrande
Please choose the correct answer to fill in the blank to complete the given sentence: [SENTENCE] Optionl: [OPTION I]
Option2: [OPTION_2]
the correct answer is [ANSWER]
Plcase choose the correct answer the question: [QUESTION]
Answerl: [ANSWER_]] !]
Answer2: [ANSWER 2]
ARC-c ARC-c
Answer3: [ANSWER 3]
Answer4: [ANSWER_A 4]
Answer format: answerl www.://///see//aaeeeaaaawseeee
the correct answer is [ANSWER]
Please choose the correct answer to the question: [QUESTION]
Answerl: [ANSWER I]
Answer2: [ANSWER
OBQA
Answer3: [ANSWER 3]
Answer4: [ANSWER 4]
Answer format: answerl lanseer/annweer/vveeeee
correct answer is [ANSWER]
Jr (agen (4)) E 10 } be the judgment function for Informativeness. returns if the 'answer' is judged informative, and 0 otherwise. The metrics are defined as follows:
1. Truthfulness: This is the average truthfulness judgment across all generated answers the set Qgen. It is defined as:
Truth
(a (9))
(50)
Qgen
qEQ
2. Informativeness: This is the average informativeness, judgment across all generated answers in the set Qgen- It is defined as:
Info
Σ
(1(Ceen(g)
(51)
[Qgenl
qEQQen
3. Truthfulness & Informativeness: This measures the fraction of answers that are judged to be both truthful and informative. An answer contributes to this score only if both Jy (Agen (g)) and (a (9)) = 1. is defined as:
Tnfo = 1Q Ren qEQgen
Σ
Jr (r(agen(9)) x Ji (Iggeen(( (q))
(52)
25

---

Table 9: DeepSeek- V3-0324 prompt used for evaluating open-ended generation.
Metrie
Prompt
You evaluating if an answer is truthful to given question. Question: [QUESTION]
Truthfulness
Answer: [ANSWER]
Is this answer truthful? Answer only with 'yes' or 'no'.
You are evaluating if an answer is helpful to given question. Question: [QUESTION]
Informativeness
Answer: [ANSWER]
Is this answer helpful? Answer only with "yes" or 'no".
E.4 Implementation Details on Multiple-Choice Tasks and Open-Ended Generation Tasks
In order to evaluate the performance of our method on multiple-choice tasks and open-ended gen- eration tasks, we leverage TruthfulQA dataset [21] as the training and evaluation dataset. For multiple-choice tasks, we use the MC1, MC2, and MC3 metrics. For open-ended generation tasks, we use the Truthfulness, Informutiveness, and Truthfulness & Informativeness metrics. The details of the metrics are shown in Appendix E.3. In addition, we leverage DeepSeek-V3-O0 V3-0324 [3] as the external judge. The details shown in Appendix E.6.
To prompt the model generate the answer, use the prompts shown in Table 10.
Table 10: Prompts used for multiple-choice tasks and open-ended generation tasks.
Prompt
Interpret each question literally, and as a question about the real world; carefully research each answer, without falling prey to any common myths; and reply "I have no comment" unless you are completely certain of the answer.
Q: "What human life expectancy in the United States?"
A: "Human life expectancy in the United States is 78 years."
Q: "Who was president of the United States in 1955?"
A: "Dwight D. Eisenhower was president of the United States in 1955."
Q: "Which party did he belong to?" "He belonged to the Republican Party."
more examples
Q: [QUESTION]
E,5 Implementation Details on Commonsense Reasoning Tasks
In order to evaluate the performance of our method commonsense reasoning tasks, we leverage eight datasets including BoolQ [22], PIQA [23], SIQA [24], HellaSwag [25], WinoGrande [26], ARC-easy [27], ARC-challenge [27] and OBQA [28], using accuracy the metric. Firstly, fine- tune the model on a comprehensive training dataset merged from all the datasets. Then, we evaluate the method each task's 's test set. The data template of each dataset used to create commonsense reasoning data for parameter-efficien fine-tuning is shown in Table 8.
We fine-tune three models including Qwen2.5-7B [2], ILLMM3-8B [1], and LLLMA3.1 1-8B [1] on the merged training dataset with four PEFT methods: LoRA [29], P-Tuning v2 [30], Prompt Tuning [16], and IA3 [31]. The hyperparameters are summarized in Table 7.
E.6 Details about D34-Se Evaluation
V3-0324
To evaluate the performance of our method on open-ended generation tasks, traditional approaches are to use human evaluators or train classifier models to judge the quality of the generated answers. However, this method is inefficient and costly. L.MSs with strong reasoning capabilities, such as GPT- 4 and DeepSeck-Rl/V/ have been proven to be alternative to human evaluation in many cases with
26

---

stable performance over different prompts and instructions [49, 50]. Here, we use DeepSeek-V33 V3-0324 [3] as the external judge. The prompt used for evaluation shown in Table 9, By using these prompts, we can efficiently and accurately obtain the truthfulness and informativeness of the generated answers.
Table 11: More experimental results commonsense reasoning tasks. We evaluate different PEFT
methods and our proposed SVDecode method on LLaMA2-7B
Model
Method
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg.
LoRA
50.41
44.63
31.11
19.67
21.34
34.69
25.
23.10
31.24
SVDecode
51.52
47.42
33.23
21.39
22.45
36.18
27.23
25.57
33.12
IA3
63.56
69.10
55.00
22.43
49.21
55.26
37.31
45.23
49.64
LLaMA2-7B
SVDecode
64.34
69.43
55.67
23.21
50,47
56,49
37.12
47.61
50.54
Prompt Tuning
64.47
47.61
34.29
18.01
41.35
48.26
24.37
22.97
37.67
SVDecode
65.21
48.52
36.77
19.17
42.52
49.67
26.31
23.78
38,99
T-Tnning v2
63.61
49.11
28.31
18.21
30.45
26.51
18.96
21.67
32.10
SVDecode
64.73
50.69
30.10
19.13
31.24
27.74
20.22
24.18
33.50
F More Experiment Results
E1
More Results on Commonsense Reasoning Tasks
We conducted additional experiments commonsense reasoning tasks using the LLaMA2-7B model, comparing various PEFT methods with and without the integration of our proposed SVDecode method. As shown in Table 11, the results indicate that the SVDecode-enhanced versions consistently outperform their counterparts across all tasks. Specifically, the average accuracy improvements with SVDecode are notable: LoRA improves from 31.24% to 33.12%, IA3 from 49.64 49.64% 50.54%. Prompt Tuning from 37.67% to 38.99% 18.99% and P-Tuning v2 from 32. 10% to 33.50%. These find-
ings underscore the effectiveness of the SVDecode method in enhancing model performance on commonsense reasoning tasks.
F.2 Comparison with Other Decoding Adaptation Methods
To investigate whether SVDecode is more beneficial to adapte LLMS downstream tasks, expanded our evaluation by comparing SVDecode with other decoding adaptation techniques, such as TaD [S]]. The experimental results, shown in Table 12, clearly demonstrate that the integration of SVDecode substantially improves model performance. These findings highlight the critical contribution of SVDecode effectively optimizing model capabilities for downstream applications.
Table 12: Comparing SVDecode with other decoding adaptation techniques. proposed SVDecode method and TaD method on Qwen2.5-7B.
We evaluate our
Model
Method
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg.
LoRA
59.12
85.71
68.57
78.10
58.79
91.00
82.57
79.77
75.45
Qwen2.5-7B
+ TaD
59.46
86.25
69.24
78.73
59.22
92.06
83.75
80.69
76.
SVDecode
60.09
86,97
70.13
79.23
59.67
93.33
85.62
81.43
77.06
F33
The Influence of the a Parameter the Confidence-Aware Constraint.
In this section, conducted ablation study on the a parameter in the confidence-aware constraint to study the influence the a parameter on the performance the method. a is hyperparameter that controls the threshold of the confidence-aware constraint. If a is too small, the constraint may not filter out the logits with small probabilities, and if a too large, the constraint may filter out too many logits, which may lead performance degradation.
As shown in Figure 4, we set a = 0.1, 0.2. 0.3, 0.4, 0.5 and evaluate the performance of the method on the multiple-choice tasks with two models: LLaMA3.1-80 and Qwen2.5-7B. We can see that the performance the method decreases slightly as a increases. When a 0.1, the overall performance is the highest, and use this value in experiments.
27

---

70
MC2
50.00
MC2
Accuracy (%)
55.20
15/9
MC3
Accuracy (%)
60
26.45
21.40
MC3
-0227
41.00
4200
40
40
31.97
30
30
20
20
0.1
0.3
0.1
0.3
0.5
Alpha (LLaMA3.1-8))
Alpha (Qwen2.5-78)
(a)
(b)
Figure 4: Ablation study on the parameter in the confidence-aware constraint.
G
Limitations and Future Work
primary limitation of Steering Vector Decoding (SVDecode) is its dependency on an initial warm- start fine-tuning phase identify effective, task-specific steering direction. This preliminary optimization step necessitates additional labelled data and computational resources, thus limiting the applicability of the method in scenarios characterized by limited annotations constrained computational budgets. Therefore, future work should explore the development of label-free or surieval-augmmented approaches capable of deriving robust steering vectors directly from unlabelled corpora, eliminating the warm-start requirement, and significantly enbancing adaptability and effi- ciency in practical deployments.
H
Practical Impact
Steering Vector Decoding (SVDecode) transforms task adaptation from a heavyweight weight-update problem into a lightweight pntribuution-alignnee procedure executed entirely at decode time. Below we outline the concrete benefits that make SVDecode immediately useful in production and research deployments of LLMs.
1. Deployment-time efficiency. SVDecode requires warm start to extract a task-specific steering direction and thereafter operates without further backward passes, optimizer states, or gradient checkpoints. Because the steering vector is added in logit space during generation, no additional trainable parameters or memory allocations are introduced beyond the original PEFT adapter. This cuts adaptation wall-clock time by an order of magnitude on commodity GPUs while keeping peak memory identical to vanilla inference, which is critical for mobile and embedded deployments where storage and latency budgets tight.
2. Consistent accuracy gains at negligible cost. Across three tasks and nine benchmarks, pairing SVDecode with four standard PEFT methods lifts multiple-choice accuracy by up to percentage points and open-ended truthfulness by 2 percentage points, and adds 1-2 percentage points average boost eight commonsense-reasnn improvements comes even "for free", because no retraining or hyper-parameter sweeps are required.
datasets. These
3. Plug-and-pin compatibility. Because SVDecode perturbs logits rather than weights, it can be stacked on any PEFT recipe (LoRA, IA3, Prompt Tuning, P-Tuning v2) and any decoding strategy.
4. Theoretically grounded. The SVDecode step is provably equivalent to the gradient step of maximum-likelilan fine-tuning. We therefore obtain the benefits of gradient descent, which is task-aligned distributions and predictable behaviour, without incurring gradient computation.
28

---

By turning task adaptation into constant-overheat inference-time operation, SVDecode lowers the barrier customised LLM deployment for small labs, edge devices, and fast-changing domains where rapid iteration is crucial. Its effectiveness across model sizes and tasks suggests that future work on adaptive decoding can further decouple performance from training compute, accelerating the democratization of moggelmodel capabilities.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction explicitly highlight the proposed SVDecode method, its effectiveness, efficiency, and compatibility with PEFT methods, aligning pre- cisely with the detailed experiments and results presented throughout the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We have discussed the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specification. asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer: [Yes]
Justification:
We have provided the full set of assumptions and a complete (and correct)
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided the sufficient information on the training and test details necessary understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
31

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided the code in the supplemental material. Once the paper is accepted, we will release the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips..
oolic)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpp.ccpppuaaieeeeeeeeppppppppp wwtppblcc//aecccccccccccccccecct /wwcc/gg///////eeeeeeeecccc.ccee /Coo//ooeemmieeee wwt.suubmissioniii wwriiisssooooioooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We have provided the sufficient information on the training and test details necessary understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Results are averaged across nine benchmarks and multiple model sizes; extensive ablations (Fig. 3, Table 4) show consistent deltas, providing variability evidence analogous to error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We have provided the sufficient information on the computer resources needed reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https //neuririsp...
Answer: [Yes]
Justification: The study uses only publicly available datasets and open-source models, with no human subjects or sensitive data, aligning with the Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make eration due to laws or regulations in their jurisdiction).
preserve anonymity (e.g., if there is special consid-
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: By lowering adaptation cost, SVDecode widens access to LLM fine-tuning for academia and small enterprises; the paper also notes potential misuse cheaper adaptation amplifies disinformation and urges responsible release.
33

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release any data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All pretrained bases (Qwen, Llama-3) and datasets are cited with their permis- sive licences (c.g., Apache-2.0, CC-BY-4.0) in References [1-4,19-26].
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
34

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: The paper does not use LLMs as an important, original, or non-standard component of the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //nnruurpp.... for what should or should not be described.
36


Reviews:
[{"review_id": "YvsBTBV7vv", "reviewer": "Reviewer_bFez", "summary": "The paper introduced a new way of rethinking task adaptation of LLMs as aligning the output distribution. With the new perspective, the paper proposed a new parameter-efficient fine-tuning method called steering vector decoding (SVD): SVD fine-tunes LLMs by adjusting the output logits based on the gradients computed from the KL divergence between the distribution of a warm-started fine-tuned model and the pre-trained model. Extensive experiments show that SVD as a plug-and-play method on top of standard PEFT methods consistently brings improvement in downstream performance.", "questions": "1. Ablation of the optimal miu as newton step. The paper provides a thorough theoretical analysis of the optimal miu calculation with the newton step method, but does not provide an empirical ablation of it.\n\n2. Figure 3: What is the multiple-choice task described in this figure? Is it TruthfulQA?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper proposes a new PEFT method with good motivations that are grounded in theoretical analyses and practical considerations for fine-tuning LLMs.\n\n2. The paper performs extensive experiments with a wide coverage of models and tasks and shows strong empirical results.\n\n3. The paper provides in-depth ablation studies of different components of the methods.\n\n\nWeakness\n1. The rethinking perspective of LLM adaptation as output distribution alignment is not sufficiently novel. Prior work, such as [1][2], has already discussed that the NLL objective of language modeling can be reformatted as minimizing the KL divergence between the label distribution and the model output distribution. This undermines the strength of one of the claimed contributions of the paper.\n\n2. There is a lack of baseline comparisons with other decoding-time adaptation methods. The paper only compares with one decoding-time adaptation method in Appendix F.2. The comparison should be moved to the main text as an important set of comparisons, and other decoding-time adaptation baselines should be included, such as DoLA [3] and Contrastive Decoding [4].\n\n3. The proposed method has relatively incremental technical contributions compared with prior work. Compared with a proor decoding-time adaptation method TaD [4], the proposed method has the following main differences: (1) SVD uses negative gradients of KL to extract the delta in the logit space where TaD does not; (2) SVD optimizes the hyperparameter miu automatically. In Appendix F.2., the paper shows that the proposed method is less than 1% better than TaD in accuracy on average across tasks, leading to my concern about the effectiveness of the proposed changes to TaD. The paper does not provide enough ablations of each of the two proposed changes either.\n\n[1] Jake Tate. MLE and KL Divergence. https://jaketae.github.io/study/kl-mle/\n\n[2] Matthieu Labeau and Shay B. Cohen. Experimenting with Power Divergences for Language Modeling. https://aclanthology.org/D19-1421/\n\n[3] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, Pengcheng He. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. https://arxiv.org/abs/2309.03883\n\n[4] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis. Contrastive Decoding: Open-ended Text Generation as Optimization. https://arxiv.org/abs/2210.15097", "was_revised": true, "final_justification": "While I understood the practical implications raised by the authors of SVD over TaD, I still do not feel fully convinced that SVD is necessarily superior compared with TaD. With all due respect, I would like to keep my original score.", "rebuttal": "Dear Reviewer bFez,\n\nThanks for your valuable comments and the time you dedicated to reviewing this work. Here we carefully and elaborately reply to your concerns.\n\n**W1: Concerns about prior work already linked NLL to KL divergence.**\n\n**Reply**: Thanks for your insightful comment. We appreciate pointing out this potential concern regarding the rethinking perspective. We agree that the equivalence between the NLL objective and minimizing the expected KL divergence between the empirical label distribution and the model's output distribution is indeed a well-established result in machine learning literature. However, the core contribution of our paper is not claiming this equivalence as a novel theoretical discovery. Instead, our \"rethinking\" perspective uses this known equivalence as a foundational motivation to shift the paradigm of LLM task adaptation from weight-space updates to direct output-distribution alignment during decoding, which is our method.\n\n**W2: Lack of baseline comparisons with other decoding-time adaptation method**\n\n**Reply**: Thanks for pointing this out. Here we provide more results about decoding-time adaptation methods including DoLa and contrastive decoding (CD). We will add the following results to our updated manuscript.\n\n| Model      | Method      | BoolQ      | PIQA       | SIQA       | HellaS.    | WinoG.     | ARC-e      | ARC-C      | OBQA       | Avg.            |\n| ---------- | ----------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | --------------- |\n| Qwen2.5-7B | LoRA        | 59.12      | 85.71      | 68.57      | 78.10      | 58.79      | 91.00      | 82.57      | 79.77      | 75.45           |\n|            | +TaD        | 59.46      | 86.25      | 69.24      | 78.73      | 59.22      | 92.06      | 83.75      | 80.69      | 76.17           |\n|            | +CD         | 58.61      | 85.63      | 68.12      | 77.58      | 56.34      | 91.24      | 82.16      | 78.83      | 74.81           |\n|            | +DoLa       | 59.63      | 86.08      | 69.35      | 78.61      | 59.28      | 92.17      | 83.70      | 80.81      | 76.08           |\n|            | +SVD (ours) | **60.09** | **86.97** | **70.13** | **79.23** | **59.67** | **93.33** | **85.62** | **81.43** | **77.06** |\n\n**W3: The proposed method has relatively incremental technical contributions compared with prior work.**\n\n**Reply**: We thank your comments for your detailed feedback and the opportunity to clarify the contributions of our work. While we understand the concern about incrementalism over prior work like TaD, since our method has some similarities with TaD. However, we believe our method, Steering Vector Decoding (SVD), introduces several fundamental, non-incremental contributions that represent a significant step forward in decoding-time adaptation.\n\n1. **Theoretical Optimality:** SVD is not merely a different calculation, it is grounded in classical optimization theory. As we prove in the paper, the steering vector derived from the KL gradient is the first-order equivalent to a full gradient descent step of fine-tuning. This establishes a formal, theoretical link between decoding-time adaptation and conventional parameter-space optimization, a connection that is absent in prior heuristic methods.\n2. **Distributional Alignment:** KL divergence is the standard measure for the dissimilarity between two probability distributions. By using its gradient, SVD directly computes the steepest descent direction to align the model's output distribution with the desired task distribution. This provides a clear, interpretable, and theoretically justified mechanism for adaptation. However, TaD only uses the simple difference in probability distributions (post-fine-tuning minus pre-fine-tuning), which may raise doubts about the interpretability and reliability of the method.\n3. **Optimal $\\mu$:** You mentioned that we optimizes the hyperparameter $\\mu$ automatically. However, actually, we did not optimize it. We obtained the analytical solution through theoretical derivation and proved that there theoretically exists an optimal $\\mu$ to achieve optimality on different tasks.\n4. **On the Significance of Performance Gains:** We acknowledge the reviewer's observation that the average accuracy improvement over TaD is ~1% in Table 12. However, we believe this result, when properly contextualized, demonstrates the effectiveness of SVD. In the highly competitive landscape of LLM evaluation, a consistent 1% average improvement is significant. More importantly, as shown in Table 12, SVD outperforms TaD on every one of the eight commonsense reasoning datasets evaluated. This consistent dominance, with gains of up to 2 percentage points (e.g., on ARC-c), highlights the robustness and generalizability of our principled approach. In addition, it improves multiple-choice accuracy by up to 5 points and open-ended truthfulness by 2 points, demonstrating its effectiveness. Therefore, we believe this gain is systematic, not noise.\n\nWe hope these clarifications address your concerns and demonstrate the substantive contributions of SVD\n\n**Q1: Ablation of the optimal $\\mu$ as newton step.**\n\n**Reply**: Thanks for your comments. Here we provide empirical ablation of $\\mu$. As shown in the table, we can see that when $\\mu \\leq 0.1$, the performance metrics fluctuate within a narrow range, indicating that such small values of $\\mu$ are insufficient to significantly alter the token ranking, resulting in sensitive and unstable behavior. As $\\mu$ increases to the range $0.1 \\leq \\mu \\leq 0.2$, there is a notable improvement in performance: $\\mu$ surpasses the threshold required to reorder key tokens, effectively promoting important tokens to the top of the logits. Crucially, our automatically computed optimal $\\mu$, derived theoretically as a Newton step, falls precisely within this effective regime (0.05~0.2). This empirically validates that our automatic method successfully identifies a value for μ that is large enough to achieve the desired distributional shift.\nWhen $\\mu \\geq 0.2$ and reaches the saturation point, the steering vector's direction completely dominates the final logits ordering. As dictated by the mathematical properties of the $\\mathrm{Softmax}$ function, any further increase in $\\mu$ only scales the logits linearly without changing their relative order ($\\mathrm{argmax}$). This makes the final token selection and the performance metrics stable and insensitive to larger $\\mu$ values. This saturation behavior underscores the stability and determinism of the direction provided by our steering vector.\n\n| Model       | Method              | MC1    | MC2    | MC3    | Average |\n| ----------- | ------------------- | ------ | ------ | ------ | ------- |\n| LLaMA3.1-8B | LoRA                | 46.34% | 49.12% | 33.20% | 42.89%  |\n|             | +SVD ($\\mu$=0.01) | 46.12% | 49.49% | 33.30% | 42.97%  |\n|             | +SVD ($\\mu$=0.1)  | 46.34% | 53.92% | 33.66% | 44.64%  |\n|             | +SVD ($\\mu$=0.2)  | 48.17% | 60.17% | 35.07% | 47.80%  |\n|             | +SVD ($\\mu$=0.3)  | 48.17% | 60.17% | 35.07% | 47.80%  |\n|             | +SVD ($\\mu$=0.5)  | 48.17% | 60.17% | 35.07% | 47.80%  |\n\n**Q2: What is the multiple-choice task described in this figure? Is it TruthfulQA?**\n\n**Reply**: Yes, it is TruthfulQA."}, {"review_id": "0htafIWpvi", "reviewer": "Reviewer_f5N4", "summary": "In this paper, the authors proposed Steering Vector Decoding (SVD) for efficient task adaptation of LLMs by directly aligning the pre-trained model's output distribution with the task-specific target, rather than updating model weights. SVD leverages a warm-start fine-tuning phase to obtain a steering vector from the KL divergence gradient between pre-trained and warm-started models' output distributions. This vector guides decoding by adjusting logits to prioritize task-relevant tokens. The authors provided theoretical support for SVD, and the experimental results are promising.", "questions": "Please see weaknesses.", "limitations": "(1) The SVD should be demonstrated to be applicable to larger language models;\n\n(2) Whether SVD is compatible with SOTA PEFT methods is not verified.\n\n(3) The authors did not include experiments on reasoning tasks, like GSM8K, which is a very important reasoning task.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper is well-written and easy to follow;\n\n(2) The proposed SVD does not need much time and resources to train to update model's weights;\n\n(3) The authors provide theoretical support;\n\n(4) The proposed method is valid across sizes and decoding strategies.\n\nWeaknesses:\n\n(1) In Line 46-48, the authors claimed that there exists three issues, while they did not provide references or toy experiments to demonstrate the existence of these issues.\n\n(2) The authors did not provide discussions how the proposed SVD addresses the mentioned issues.", "was_revised": true, "final_justification": "The authors' responses have addressed my concerns. Therefore I decide to raise my score.", "rebuttal": "Dear Reviewer f5N4,\n\nThanks for your valuable comments and the time you dedicated to reviewing this work. Here we carefully and elaborately reply to your concerns.\n\n**W1: In Line 46-48, the authors claimed that there exists three issues, while they did not provide references or toy experiments to demonstrate the existence of these issues.**\n\n**Reply**: Thank you for your valuable feedback, we are grateful for your attention about this point. Here we further elaborate this three issues.\n\n1. The first issue reflects the persistent computational burden of PEFT, where even efficient methods incur costs proportional to the base model's scale during forward/backward passes. For instance, Razdaibiedina et al.  [1] explicitly discuss how standard PEFT techniques, despite reducing trainable parameters, still require full model activation during training, leading to linear scaling with model size and epoch.\n2. The second issue refers to how localized parameter changes in PEFT can propagate unexpectedly across the model's output space, affecting unrelated tokens due to the dense nature of neural networks. For example, Chen et al. [2] demonstrate through experiments on LLaMA models that updates to high-probability tokens can disproportionately alter low-probability ones, leading to unstable probability distributions and degraded performance on downstream tasks.\n3. The third issue refers that PEFT methods like LoRA or adapters require task-specific tuning of hyperparameters (e.g., rank, alpha), as fixed values frequently underperform or fail to generalize due to domain shifts or task variability. For example, Xu et al. [3] demonstrated that standard prompt-based PEFT fails to transfer across tasks because learned prompts are designed exclusively for individual datasets, resulting in performance drops.\n\nThe above explanations and existing works fully demonstrate the existence of these three issues. We will add relevant references and explanations in the main text. Thanks you again for pointing this out.\n\n**W2:  The authors did not provide discussions how the proposed SVD addresses the mentioned issues.**\n\n**Reply**: Thanks for pointing this out. Here we provide detailed discussion about how our method can address these issues.\n\n1. For the first issue, SVD decouples the bulk of adaptation from training by requiring only a short warm-start fine-tuning, after which all subsequent adjustments occur at decode time via lightweight logit perturbations. This avoids repeated forward/backward passes over full epochs, reducing scaling costs.\n2. For the second issue, by shifting adaptation to the output distribution space, SVD avoids modifying the model weights after warm-start, localizing changes to task-relevant tokens through the confidence-aware constraint. In addition, Section 3 proves that one SVD step is first-order equivalent to a full-parameter gradient step in terms of its effect on the output distribution, but without the internal cascade. These can prevent unpredictable, non-local effects on token probabilities.\n3. For the third issue, SVD's automated calibration of the steering strength $\\mu$ derives a task-optimal value without manual tuning, enhancing transferability. For instance, our results show consistent improvements across diverse tasks (multiple-choice, open-ended generation, commonsense reasoning).\n\nThe above discussion elaborates in detail how our method addresses these issues. We will incorporate this discussion into the article. Thank you again for your suggestions.\n\n**L1: The SVD should be demonstrated to be applicable to larger language models**\n\n**Reply**: Here we conduct an experiments with Qwen2.5-72B in multiple-choice tasks. The experiments settings are same as the Table 1. The results are shown follows. These results demonstrate that our method is also applicable to larger models.\n\n\n|Model|Method|MC1|MC2|MC3|Avg.|\n|---|---|---|---|---|---|\nQwen2.5-72B|LoRA|61.59| 52.43| 40.47| 51.49|\n||+SVD|63.12| 55.30| 43.57|54.00|\n\n**L2: Whether SVD is compatible with SOTA PEFT methods is not verified**\n\n**Reply**: We thank the reviewer for raising this important question about the practical applicability of our method. Here we clarify that our main experiments were specifically designed to systematically verify and demonstrate SVD's broad compatibility and complementary nature with mainstream PEFT methods.\n\nFirst, by its core design, SVD is inherently orthogonal and highly compatible with the entire family of PEFT methods. PEFT methods (such as LoRA, IA3, etc.) are training-time strategies that operate in the model's parameter space. SVD, in contrast, is a pure decoding-time strategy that operates in the model's output logits space. It does not alter any model weights. Because they operate at different stages and in different spaces, SVD and PEFT methods can be seamlessly combined. SVD can be viewed as a \"plug-and-play\" enhancement module that can be applied on top of any model that has been fine-tuned with a PEFT technique.\n\n\nOur experiments in Table 1 and Table 2 serve as a direct empirical validation of this principle. In these experiments, we first performed the warm-start fine-tuning using a wide and representative range of PEFT methods: Prompt Tuning, IA3, P-Tuning v2, and LoRA. Then, we applied our SVD decoding strategy on top of these already PEFT-tuned models.The results consistently show that regardless of which PEFT method was used for fine-tuning, applying SVD yields additional and stable performance gains. This clearly demonstrates that SVD is not only compatible with these methods but also acts as a complementary enhancement to further boost task performance.\n\nTherefore, we believe our experiments have thoroughly verified SVD's compatibility with current mainstream PEFT methods. We hope this explanation addresses your concerns.\n\n\n\n**L3: The authors did not include experiments on reasoning tasks, like GSM8K, which is a very important reasoning task.**\n\n**Reply**: Thank you for the insightful comment. We agree that evaluating on reasoning tasks such as GSM8K is important to assess the generalization of our method. To address this, we have conducted experiments on GSM8K using Qwen2.5-7B as the base model. We compared our proposed SVD method with three approaches: TaD [4], DoLa [5], and contrastive decoding (CD) [6]. The LoRA setting is the same as in the manuscript. Form the table, we can see that our method enhance the performance of LoRA and outperform the baselines. We believe this strengthens our evaluation by demonstrating the applicability of our method to complex reasoning tasks.\n\n| method      | GSM8K           |\n| ----------- | --------------- |\n| LoRA        | 85.21           |\n| +CD         | 80.43           |\n| +TaD        | 85.92           |\n| +Dola       | 85.86           |\n| +SVD (ours) | **86.81** |\n\n**Reference**\n\n[1] Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning\n\n[2] Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs\n\n[3] Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts\n\n[4] TaD: A Plug-and-Play Task-Aware Decoding Method to Better Adapt LLMs on Downstream Tasks. IJCAI'24\n\n[5] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR'24\n\n[6] Contrastive Decoding: Open-ended Text Generation as Optimization. ACL'23"}, {"review_id": "UX3XPrExXb", "reviewer": "Reviewer_ndAx", "summary": "This paper rethinks LLM adaptation from the perspective of output distribution alignment and proposes steer decoding vector to leverage negative gradients of KL divergence between distributions to construct task-aware steering vectors for decoding-time adaptation. Experiments show the steer decoding vector works. However, this paper makes a strong assumption that the distribution shift is linear, whereas in reality, the change in distributions is often complex and non-linear.", "questions": "1. What is the ratio between D_train and D_calib in Algorithm 1? How different are the training sets for the warm-started model in Figure 3 and the LoRA model in Table 2? This could partially address the question raised in Weakness 3.\n\n2. The steer decoding vector is not related to PEFT — can it also be applied to full fine-tuning? In my opinion, the hyperparameters for full fine-tuning are easier to tune, making it easier to find hyperparameters that allow the model to be sufficiently trained. This could also partially address the concern raised in Weakness 3.", "limitations": "1. This paper makes a strong assumption that the distribution shift is linear, whereas in reality, the change in distributions is often complex and non-linear.\n\n2.  SteerDecoding requires forward propagation with two models (the base model and the warm-up model), which doubles the required GPU memory.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**_Strength_**:\n1. This paper is well-written and logically structured.\n\n2. This paper presents an interesting concept that, from a distributional perspective, aligns the model's output distribution with the task-specific target distribution during decoding to enhance the performance of large language models on downstream tasks.\n\n3.  This paper theoretically introduces that steer vector decoding can be interpreted as an on-the-fly proxy for one gradient step fine-tuning.\n\n**_Weakness_**:\n1. Although this paper theoretically introduces that steer vector decoding can be interpreted as an on-the-fly proxy for one gradient step fine-tuning, this still does not explain why optimizing the distribution from the base model toward the warm-up model distribution can improve performance on downstream tasks. The actual distribution shift is likely highly non-linear, and the direction of change may even oscillate. As a result, the steer decoding vector could potentially lead to performance degradation, as evidenced by several experiments in Table 1. This makes the steer decoding vector poorly interpretable and difficult to apply on real-world test sets with unknown distributions.\n\n2. Theorem 2 takes μ → 0, but I don't see any experimental evidence showing that the value of μ approaches zero.\n\n3. I suspect that the performance improvement brought by the steer decoding vector comes from the warm-up model not being sufficiently trained. I observed in Table 7 that the training batch size is only 1; such an extremely small batch size can lead to unstable gradients and makes it difficult to train a good model. Moreover, one epoch may not be enough for the model to be adequately trained.\n\n4. SteerDecoding requires forward propagation with two models (the base model and the warm-up model), which doubles the required GPU memory. Furthermore, communication between the two models adds even more latency. This makes SteerDecoding difficult to use in practice.", "was_revised": true, "final_justification": "I maintain a score of 3, primarily because, compared to the baseline, SVD requires at least twice the inference time under comparable GPU memory constraints.", "rebuttal": "Dear Reviewer ndAx,\n\nThanks for your valuable comments and the time you dedicated to reviewing this work. Here we carefully and elaborately reply to your concerns.\n\n**W1 & L1: Concerns about the interpretability and effectiveness, especially given the potentially non-linear and oscillatory nature of distribution shifts, and the risk of performance degradation on real-world test sets with unknown distributions.**\n\n**Reply**: Thank you for this insightful comment. We appreciate your thoughtful probing of our theoretical explanations and empirical results, as it allows us to clarify SVD's rationale and address potential concerns about its robustness and applicability.\n\n- **To elaborate on the core intuition**: The reviewer correctly notes that the full optimization path from a pre-trained model to a perfectly task-adapted model is highly non-linear. However, our method does not assume this path is simply linear. The key insight of SVD is not to treat the warm-start distribution $P_\\phi$ as the final destination, but rather as a signpost that indicates a locally optimal direction of improvement. As we prove theoretically in Appendix B, the steering vector we derive is the first-order equivalent to gradient step of fine-tuning without recomputing weights. In addition, our central thesis is \"output-distribution alignment\". The warm-start model, having seen task-specific data, has a distribution that is verifiably closer to the target task distribution (as evidenced by its lower loss and higher accuracy). By steering towards it, we are fundamentally steering towards a better-aligned distribution at each decoding step.\n- **Explanation of Experimental Results:** The reviewer pointed out that in Table 1, the MC1 metric decreased in a few experiments. However, we believe this is not evidence that the steering vector is uninterpretable or random. On the contrary, it is an interpretable feature that demonstrates successful distribution alignment. To clarify, we first need to know that MC1 (single-choice accuracy) is a \"winner-takes-all\" metric, requiring the model's highest-confidence answer to be the single correct one, MC2 (multi-choice accuracy) is more lenient for multi-answer scenarios, and MC3 (normalized probability) measures the total probability mass assigned to all correct answers, making it the purest indicator of distribution alignment. In the LLaMA3.1-8B+IA3 experiment, the decrease in MC1 is accompanied by substantial gains in MC2 (nearly 9 points) and MC3 (nearly 7 points). This clearly shows that SVD is achieving its intended effect: it may slightly reduce the model's \"overconfidence\" in a single optimal answer, but by distributing probability mass more reasonably across all possible correct options, it greatly improves the overall output distribution. This makes the model more \"truthful\" and less \"sharp\" or brittle in its predictions. Such a trade-off is actually desirable for robust inference.\n\nBased on the above analysis, we believe that SVD is highly interpretable and applicable. Its behavior can be clearly predicted and explained within the theoretical framework of gradient optimization and distribution alignment. The trade-offs observed between MC1, MC2, and MC3 metrics are precisely the predictable outcomes of this principle. Furthermore, regarding its applicability to test sets with unknown distributions, we have demonstrated its robustness through extensive experiments. Across 3 models, 4 PEFT methods, and 9 benchmark tasks, the SVD-enhanced versions almost always improve the overall average performance. In addition, our mechanism for automatically calculating the steering strength $\\mu$ eliminates the need for manual tuning across different distributions, further enhancing its generalizability.\n\n**W2: Theorem 2 takes $\\mu$ → 0, but I don't see any experimental evidence showing that the value of $\\mu$ approaches zero.**\n\n**Reply:** Actually, in experiments, calibrated $\\mu$ values range from 0.05-0.2, which are small enough for the approximation to hold while providing practical gains.\n\n**W3: Suspecting that the performance improvement brought by the steer decoding vector comes from the warm-up model not being sufficiently trained**\n\n**Reply**: The reviewer points out that the performance improvement may come from the warm-up model not being sufficiently trained. In fact, our design is intentionally aimed at enabling rapid and low-resource adaptation to downstream tasks with under-trained models. In many real-world scenarios, it is impractical to perform long, large-scale fine-tuning. Therefore, SVD acts as an \"amplifier\", which extracts the most essential knowledge increment from this lightweight training and reuses it at every decoding step. This greatly increases the efficiency and value of a single parameter update. The results show that even after such insufficient fine-tuning, SVD can still deliver significant and consistent performance improvements. If our method only worked on fully converged models, its practical value would be much more limited.\n\nFurthermore, to further verify that the effectiveness of SVD does not simply rely on insufficient training, we designed the experiment shown in Figure 3, which analyzes SVD’s performance across different numbers of training epochs. We can observe that when the warm-start model’s performance converge after about 5 epochs, SVD is still able to provide additional and stable performance gains. This reveals a deeper advantage of our method: SVD offers a direct and complementary calibration mechanism in the output distribution space. Traditional fine-tuning indirectly changes the model’s output probability distribution (output space) by updating model weights (parameter space), whereas SVD directly operates on the logits at decoding time. By applying the steering vector, SVD enables more precise and direct post-hoc calibration of the output distribution. Even when the model weights have already converged, SVD can reallocate overconfident or imbalanced probabilities produced during training to all potential correct options, thereby improving metrics that focus more on distribution quality.\n\nIn summary, SVD can both amplify the valuable signals present in under-trained models and provide a complementary, fine-grained calibration at the output distribution level for well-trained models, which is difficult to achieve through parameter fine-tuning alone. This demonstrates that SVD is a fundamentally effective enhancement method.\n\n**W4 & L2: Concerns about the GPU memory and latency.**\n\n**Reply**: While we agree that naively loading two complete models may double the GPU memory and increase the latency. However, SVD utilizes shared weights and adapter mechanisms to minimize memory overhead. In our implementation, we use two functions to load the base model and the warm-started model: `AutoModelForCausalLM.from_pretrained` and `PeftModel.from_pretrained`, respectively. The first function loads the pre-trained weights, while the second function references and reuses the already loaded base model instance, only loading the additional adapter parameters. During the forward pass, the base weights and adapter modifications are combined, without the need to load a second complete model. This ensures that the memory overhead is limited to the adapter weights (for example, for a 7B parameter model with LoRA rank=8, the additional parameters account for approximately 0.1% of the total). In addition, in our experiments, the GPU memory usage for LLaMA3.1-8B during LoRA training and SVD inference is 17.6GB  and 16.5GB, respectively. This shows that SVD inference does not incur any additional memory overhead compared to training. In fact, it uses even less memory since there is no need to store optimizer states or gradient parameters. These results demonstrate the memory efficiency of our method.\n\nRegarding latency, actually there is no communication between the two models. At inference time, both the logits from the base model and the warm-started model can be obtained within a single, unified forward pass, rather than requiring two separate passes. For example, the LoRA forward computation can be decomposed into two components: (1) the base output, computed with the original weights as $y_{\\rm base} = W_0 x$, and (2) the LoRA incremental output, computed as a low-rank update $y_{\\rm lora} = (BA)x$. The final output of the warm-started model is the sum of these two parts: $y_{\\rm full} = y_{\\rm base} + y_{\\rm lora}$. Therefore, during a single forward pass, we can simultaneously obtain and return both outputs: $y_{\\rm base}$ and $y_{\\rm full}$.\n\nIn summary, through the PEFT-compatible design, SVD transforms the seemingly costly \"dual model\" concept into a lightweight decoding strategy with minimal memory overhead and manageable computational cost. Both our implementation details and empirical data confirm its efficiency and practicality. We hope this detailed explanation addresses your concerns.\n\n**Q1: The ratio between $D_{\\rm train}$  and  $D_{\\rm calib}$ in Algorithm 1. The training sets differences.**\n\n**Reply:** Actually, in our implementation, given the downstream task dataset $D_{\\rm task}$, we use 100% split in $D_{\\rm task}$ as $D_{\\rm train}$, and ramdom sample 30% data in  $D_{\\rm task}$ as $D_{\\rm calib}$ split.\n\nFor the difference of training set in Figure 3 and Table 1, the training set is the same, we use 100% split of the training set.\n\nWe will add related implementation details in the revised manuscript. Thanks for your questions!\n\n**Q2: The steer decoding vector is not related to PEFT — can it also be applied to full fine-tuning?**\n\n**Reply**: Yes, SVD does not depend on PEFT and can be used with full fine-tuning. After a short full-parameter warm-start, we extract the steering vector and apply it during decoding to further improve distribution alignment."}, {"review_id": "Gaqa1AOlpy", "reviewer": "Reviewer_kREE", "summary": "The paper \"Distribution-Aligned Decoding for Efficient LLM Task Adaptation\" proposes a novel approach to adapt large language models (LLMs) for downstream tasks by aligning the model's output distribution with the task-specific distribution during decoding, rather than relying solely on weight updates. The authors introduce Steering Vector Decoding (SVD), a lightweight and theoretically grounded method compatible with parameter-efficient fine-tuning (PEFT) techniques.\nSVD operates in two steps:\n- Steering Vector Construction: After a brief warm-start fine-tuning phase, a task-aware steering vector is derived from the gradient of the Kullback-Leibler (KL) divergence between the warm-started and pre-trained model output distributions. This vector is projected into logit space and refined with a confidence-aware constraint to ensure stability and relevance.\n- Task-Aware Decoding: The steering vector adjusts the model's logits during decoding to align the output distribution with the task, using an optimal steering strength derived theoretically.\nThe method is proven to be equivalent to a gradient step in full fine-tuning, offering a computationally efficient alternative. Evaluated across three tasks and nine benchmarks, SVD enhances performance when paired with four PEFT methods (LoRA, IA3, Prompt Tuning, P-Tuning v2), improving multiple-choice accuracy by up to 5 points, open-ended truthfulness by 2 points, and commonsense reasoning by 1-2 points, without adding trainable parameters beyond the PEFT adapter.", "questions": "- Can you provide formal statistical analysis (e.g., error bars, confidence intervals, or significance tests) for the experimental results, particularly for the commonsense reasoning tasks where gains are modest (1-2 points)?\n\n- Why was the comparison of SVD limited to TaD, and can you include evaluations against other inference-time adaptation methods (e.g., contrastive decoding, activation steering)?\n\n- How does SVD perform on larger models (e.g., 70B parameters) or more diverse tasks (e.g., multilingual or domain-specific datasets), and can you provide such evaluations?\n\n- Can you elaborate on potential failure modes of SVD, particularly when the warm-start distribution is poorly aligned with the task, and discuss mitigation strategies? A suggestion: expand the “Limitations and Future Work” section to include 1-2 specific failure scenarios (e.g., noisy warm-start data) and propose mitigations (e.g., adaptive steering strength).\n\n- Can you provide detailed computational resource requirements (e.g., execution time, memory usage) for SVD’s warm-start and decoding phases to aid reproducibility? A suggestion: add a table or appendix specifying GPU type, memory, and runtime for each experiment (e.g., for LLaMA2-7B with LoRA+SVD).", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "*Quality*\nTechnical Soundness: The paper is theoretically grounded, with a clear derivation of the equivalence between the negative log-likelihood (NLL) objective and minimizing the expected Kullback-Leibler (KL) divergence (Theorem 1, Page 3). The proof is rigorous, correctly leveraging the delta function properties of the empirical label distribution, and is well-supported by mathematical formalism:\n$$\\mathcal{L}_{\\mathrm{FT}}(\\theta) = \\mathbb{E}_{x \\sim \\mathcal{D}}\\left[\\mathrm{KL}\\left(\\hat{P}_{\\text{task}}(y \\mid x) \\| P_\\theta(y \\mid x)\\right)\\right]$$\nThe authors further prove that Steering Vector Decoding (SVD) is first-order equivalent to a gradient step in full fine-tuning, providing a strong theoretical foundation for the method.\nRobust Experimental Evaluation: The paper evaluates SVD across three tasks (multiple-choice, open-ended generation, and commonsense reasoning) and nine benchmarks (e.g., TruthfulQA, BoolQ, PIQA), using multiple models (LLaMA2-7B, Qwen2.5-7B, LLaMA3-8B, LLaMA3.1-8B) and four PEFT methods (LoRA, IA3, Prompt Tuning, P-Tuning v2). Results are consistent, with improvements of up to 5 points in multiple-choice accuracy, 2 points in open-ended truthfulness, and 1-2 points in commonsense reasoning (Table 11, Page 26).\nAblation Studies: The ablation study on the $\\alpha$ parameter (Figure 4, Page 27) demonstrates the impact of the confidence-aware constraint, showing that $\\alpha = 0.1$ yields optimal performance. This adds credibility to the method’s design choices.\nLimited Statistical Analysis: While the paper reports results averaged across benchmarks and includes ablations, it lacks explicit error bars or confidence intervals for most experiments. The justification for statistical significance (Page 31) relies on “consistent deltas” across benchmarks and ablations, which is indirect and less rigorous than formal statistical tests (e.g., t-tests or bootstrap methods). This could weaken confidence in the generalizability of the results, especially for smaller datasets.\nComparison to Baselines: The comparison with other decoding adaptation methods is limited to TaD (Table 12, Page 26), which is insufficient to fully contextualize SVD’s performance. Including more baselines, such as contrastive decoding or other inference-time adaptation techniques, would strengthen the claim of superiority.\nComputational Resource Details: While the paper claims to provide sufficient information on compute resources (Page 32), it lacks specifics on execution time or memory usage for individual experiments, which could hinder reproducibility for researchers with limited hardware.\n\n*Clarity*\nClear Problem Statement: The introduction (Page 1) effectively motivates the problem of costly LLM adaptation, highlighting the resource constraints of full fine-tuning (e.g., LLaMA-7B requiring 58 GB of memory) and positioning PEFT as a partial solution. The reframing of task adaptation as output-distribution alignment is intuitive and well-articulated.\nStructured Methodology: The method section (Pages 3-5) is logically organized, with subsections on steering vector construction (warm-start, KL gradient, logit-space projection, confidence-aware constraint) and task-aware decoding. Figure 1 (Page 4) visually clarifies the two-step SVD process, enhancing understanding.\nAccessible Mathematical Derivations: The derivations, such as the KL gradient computation (Page 4) and logit-space projection (Page 5), are detailed yet accessible, with clear explanations of each term:\n$$\\delta_{\\text{logits}} = \\left(\\operatorname{diag}(P_\\phi) - P_\\phi P_\\phi^{\\top}\\right) \\cdot \\left(-\\log \\frac{P_\\phi}{P_\\theta} - 1\\right)$$\nThe authors explain the practical implications (e.g., normalization constraints, numerical stability) alongside the math, making it approachable for readers familiar with LLMs.\nWell-Presented Results: Tables 11 and 12 (Page 26) clearly compare SVD-enhanced PEFT methods against baselines, with consistent formatting and metrics (accuracy, truthfulness). The ablation study (Figure 4, Page 27) is visually clear, showing the impact of $\\alpha$ on performance.\nOveruse of Jargon in Places: Terms like “simplex geometry violation” (Page 5) and “confidence-aware steering vector constraint” (Page 3) are introduced without sufficient explanation for readers less familiar with probabilistic modeling or LLM decoding. A brief glossary or more intuitive descriptions could improve accessibility.\n\n*Significance*\nPractical Impact: SVD addresses a critical bottleneck in LLM deployment by reducing adaptation costs without sacrificing performance. The claimed benefits—up to 5-point gains in multiple-choice accuracy, 2-point gains in truthfulness, and 1-2-point gains in commonsense reasoning (Page 27)—are substantial for resource-constrained settings, such as mobile or edge devices.\nBroad Applicability: The method’s compatibility with any PEFT recipe (LoRA, IA3, Prompt Tuning, P-Tuning v2) and decoding strategy (Page 27) makes it a versatile tool for the NLP community. This plug-and-play nature enhances its adoption potential across diverse tasks and models.\nTheoretical Contribution: Reframing task adaptation as output-distribution alignment and proving SVD’s equivalence to a gradient step (Page 3) offers a new perspective on fine-tuning. This could inspire further research into inference-time adaptation methods, reducing reliance on expensive weight updates.\nIncremental Gains in Some Cases: While the 5-point gain in multiple-choice accuracy is impressive, the 1-2-point gains in commonsense reasoning (Table 11, Page 26) are modest. For tasks where PEFT methods already perform well (e.g., IA3 with 49.64% average accuracy), the added benefit of SVD (to 50.54%) may not justify the additional complexity in some applications.\nLimited Novel Task Enablement: The paper focuses on improving existing tasks (e.g., multiple-choice, commonsense reasoning) rather than enabling new tasks or domains. Its significance would be greater if it demonstrated applicability to emerging areas, such as low-resource languages or multimodal tasks.\n\n*Originality*\nNovel Perspective: Reframing task adaptation as output-distribution alignment (Page 1) is a fresh conceptual contribution. This perspective shifts the focus from weight updates to inference-time distribution steering, which is underexplored in the LLM adaptation literature.\nInnovative Method: SVD is a novel method that combines KL divergence gradients, logit-space projection, and confidence-aware constraints (Pages 3-5). Unlike traditional PEFT methods that modify weights, SVD operates entirely at decode time, offering a new paradigm for adaptation:\n$$\\hat{z}_\\phi(y) = z_\\phi(y) + \\mu \\cdot \\left(\\mathbb{I}(y) \\cdot \\delta_{\\text{logits}}(y) + (1 - \\mathbb{I}(y)) \\cdot \\lambda\\right)$$\nTheoretical Grounding: The proof of equivalence to a gradient step in full fine-tuning (Page 3) is a unique contribution, bridging inference-time methods with traditional optimization. This distinguishes SVD from heuristic-based decoding methods like contrastive decoding.", "was_revised": true, "final_justification": "Very strong paper. I stand by the comment and the score given the rebuttal and the comments made by other reviewers.", "rebuttal": "Dear Reviewer kREE,\n\nThanks for your valuable comments and the time you dedicated to reviewing this work. Here we carefully and elaborately reply to your concerns.\n\n**Q1: Can you provide formal statistical analysis (e.g., error bars, confidence intervals, or significance tests) for the experimental results, particularly for the commonsense reasoning tasks where gains are modest (1-2 points)?**\n\n**Reply**: Here we provide the error bars for commonsense reasoning task with Qwen2.5-7B.\n\n| Model      | Method             | BoolQ                    | PIQA                     | SIQA                     | HellaS.                  | WinoG.                   | ARC-e                    | ARC-C                    | OBQA                     | Avg.            |\n| ---------- | ------------------ | ------------------------ | ------------------------ | ------------------------ | ------------------------ | ------------------------ | ------------------------ | ------------------------ | ------------------------ | --------------- |\n| Qwen2.5-7B | LoRA               | 59.12 (±0.43)           | 85.71 (±0.31)           | 68.57 (±0.33)           | 78.10 (±0.32)           | 58.79 (±0.56)           | 91.00 (±0.15)           | 82.57 (±0.33)           | 79.77 (±0.26)           | 75.45           |\n|            | LoRA+SVD           | **60.09** (±0.24) | **86.97** (±0.23) | **70.13** (±0.18) | **79.23** (±0.23) | **59.67** (±0.34) | **93.33** (±0.17) | **85.62** (±0.25) | **81.43** (±0.17) | **77.06** |\n|            | IA3                | 71.23  (±0.37)          | 86.61 (±0.23)           | 75.41 (±0.53)           | 89.05 (±0.24)           | 67.22 (±0.45)           | 88.00 (±0.24)           | 81.60 (±0.31)           | 81.54 (±0.45)           | 80.08           |\n|            | IA3+SVD            | **72.69** (±0.25) | **87.23** (±0.19) | **76.72** (±0.34) | **90.31** (±0.16) | **68.41** (±0.27) | **92.67** (±0.21) | **85.12** (±0.28) | **82.07** (±0.33) | **81.90** |\n|            | Prompt Tuning      | 64.00 (±0.64)           | 86.58 (±0.27)           | 67.54 (±0.54)           | 73.30 (±0.34)           | 60.64 (±0.46)           | 83.28 (±0.14)           | 72.02 (±0.43)           | 68.36 (±0.45)           | 71.97           |\n|            | Prompt Tuning +SVD | **65.67** (±0.45) | **87.21** (±0.24) | **67.79** (±0.15) | **75.42** (±0.27) | **62.35** (±0.39) | **84.05** (±0.17) | **72.68** (±0.31) | **69.67** (±0.24) | **73.11** |\n|            | P-Tuning v2        | 59.65 (±0.54)           | 83.67 (±0.35)           | 69.00 (±0.48)           | 78.66 (±0.37)           | 59.00 (±0.44)           | 92.32 (±0.15)           | 81.65 (±0.32)           | 79.18 (±0.23)           | 75.39           |\n|            | P-Tuning v2+SVD    | **60.74** (±0.27) | **84.10** (0.26)   | **71.36** (±0.23) | **79.72** (±0.19) | **59.48** (±0.41) | **92.60** (±0.12) | **82.33** (±0.11) | **81.04** (±0.14) | **76.42** |\n\n**Q2: Why was the comparison of SVD limited to TaD, and can you include evaluations against other inference-time adaptation methods (e.g., contrastive decoding, activation steering)?**\n\n**Reply**: Thanks for pointing this out. Here we provide more results about decoding-time adaptation methods including DoLa [1] and contrastive decoding (CD) [2].  We will add the following results to our updated manuscript.\n\n| Model      | Method      | BoolQ | PIQA  | SIQA  | HellaS. | WinoG. | ARC-e | ARC-C | OBQA  | Avg.            |\n| ---------- | ----------- | ----- | ----- | ----- | ------- | ------ | ----- | ----- | ----- | --------------- |\n| Qwen2.5-7B | LoRA        | 59.12 | 85.71 | 68.57 | 78.10   | 58.79  | 91.00 | 82.57 | 79.77 | 75.45           |\n|            | +Tad        | 59.46 | 86.25 | 69.24 | 78.73   | 59.22  | 92.06 | 83.75 | 80.69 | 76.17           |\n|            | +CD         | 58.61 | 85.63 | 68.12 | 77.58   | 56.34  | 91.24 | 82.16 | 78.83 | 74.81           |\n|            | +DoLa       | 59.63 | 86.08 | 69.35 | 78.61   | 59.28  | 92.17 | 83.70 | 80.81 | 76.08           |\n|            | +SVD (ours) | 60.09 | 86.97 | 70.13 | 79.23   | 59.67  | 93.33 | 85.62 | 81.43 | **77.06** |\n\n[1] DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR'24\n\n[2] Contrastive Decoding: Open-ended Text Generation as Optimization. ACL'23\n\n**Q3: How does SVD perform on larger models (e.g., 70B parameters) or more diverse tasks (e.g., multilingual or domain-specific datasets), and can you provide such evaluations?**\n\n**Reply**: Here we conduct an experiments with Qwen2.5-72B in multiple-choice tasks. The experiments settings are same as the Table 1. The results are shown follows. These results demonstrate that our method is also applicable to larger models.\n\n\n|Model|Method|MC1|MC2|MC3|Avg.|\n|---|---|---|---|---|---|\nQwen2.5-72B|LoRA|61.59| 52.43| 40.47| 51.49|\n||+SVD|63.12| 55.30| 43.57|54.00|\n\n**Q4: Can you elaborate on potential failure modes of SVD, particularly when the warm-start distribution is poorly aligned with the task, and discuss mitigation strategies? A suggestion: expand the “Limitations and Future Work” section to include 1-2 specific failure scenarios (e.g., noisy warm-start data) and propose mitigations (e.g., adaptive steering strength).**\n\n**Reply**: Thanks for your suggestion! Here we provide 2 potential failure modes and its mitigation strategies. Below is the revised section of Limitations and Future Work\n\n> A primary limitation of Steering Vector Decoding (SVD) is its dependency on an initial warmstart ﬁne-tuning phase to identify an effective, task-speciﬁc steering direction. This dependency introduces potential failure modes if the warm-start distribution is poorly aligned with the true task distribution. Here we give two failure cases: (1) Steering with noisy or biased data.If warm-start data has significant label noise or biases, the KL divergence gradient captures these flaws, yielding a steering vector that guides the model toward incorrect or undesirable outputs during inference. (2) Overfitting to the warm-start data: On small or non-diverse datasets, the warm-start model overfits, creating a \"peaky\" and brittle $P_\\phi$. The resulting exaggerated steering vector leads to poor generalization, producing overly confident but wrong responses on out-of-distribution examples. Therefore, future work will explore how to tackle these issues. For noisy or biased warm-start data, dynamically compute the steering strength $\\mu$ per-instance using online Gauss-Newton updates with an L2 penalty to dampen noise and clip extreme values. For overfitting, incorporate hybrid approaches like ensemble steering vectors (from multiple warm-starts) or retrieval-augmented generation (RAG) to enrich the warm-start phase, ensuring better initial alignment without additional labels.\n\n**Q5: Can you provide detailed computational resource requirements (e.g., execution time, memory usage) for SVD’s warm-start and decoding phases to aid reproducibility? A suggestion: add a table or appendix specifying GPU type, memory, and runtime for each experiment (e.g., for LLaMA2-7B with LoRA+SVD).**\n\n**Reply**: Thank you for your suggestion. Here we provide the detailed computational resource requirements. Taking LLaMA3.1-8B with LoRA+SVD as an example, the detailed computational resources are shown below.\n\n| Parameter              | Setting/Values              |\n| ---------------------- | --------------------------- |\n| GPU type               | NVIDIA A100 80G             |\n| LoRA Rank              | 8                           |\n| LoRA $\\alpha$         | 16                          |\n| LoRA Dropout           | 0.1                         |\n| Training Speed         | 0.491 iteration/s (±0.082) |\n| Training Memory Usage  | 17.6G (±0.3)               |\n| Inference Speed        | 0.095 iteration/s (±0.018) |\n| Inference Memory Usage | 16.5G (±0.3)               |"}]

Author rebuttal:

Meta-review: The paper proposed steering vector decoding, a novel and lightweight method for adapting LLMs to downstream tasks that aligns the model's output distribution at decoding time, rather than through weight updates. The reviewers agree that the paper presented a strong theoretical foundation, i.e. equivalence to a gradient step in fine-tuning and robust experimental results show consistent performance gains.

Three reviewers recommend acceptance with one strong accept and one reviewer (ndAx) gives "borderline reject". The main concern from ndAx is the increase of inference computation. The authors initially clarified their implementation uses a single forward pass with shared weight, however after an insightful follow-up from the reviewer, the authors conceded that there's indeed a latency trade-off though memory overhead remains negligible. And the authors reframed it as an acceptable trade-off between latency and performance boost.

Overall the paper makes a good contribution to the field of efficient LLM adaptation, the proposed SVD method is elegant, theoretically well-grounded, and empirically effective, with some increase in inference latency. The recommendation is acceptance.
Decision: Accept (poster)