SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models
Authors: Jianyi Zhang
Da-Cheng Juan
Cyrus Rashtchian
Chun-Sung Ferng
Heinrich Jiang
Yiran Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Primary area: natural_language_processing
TL;DR: Self Logits Evolution Decoding (SLED) improves LLM factuality through better decoding.

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities, but their outputs can sometimes be unreliable or factually incorrect. To address this, we introduce Self Logits Evolution Decoding (SLED), a novel decoding framework that enhances the truthfulness of LLMs without relying on external knowledge bases or requiring further fine-tuning. From an optimization perspective, our SLED framework leverages the latent knowledge embedded within the LLM by contrasting the output logits from the final layer with those from early layers. It then utilizes an approximate gradient approach to enable latent knowledge to guide the self-refinement of outputs, thereby effectively improving factual accuracy. Extensive experiments have been conducted on established benchmarks across a diverse range of model families (LLaMA 2, LLaMA 3, Gemma) and scales (from 2B to 70B), including more advanced architectural configurations such as the mixture of experts (MoE). Our evaluation spans a wide variety of tasks, including multi-choice, open-generation, and adaptations to chain-of-thought reasoning tasks. The results demonstrate that SLED consistently improves factual accuracy by up to 20\% compared to existing decoding methods while maintaining natural language fluency and negligible latency overhead. Furthermore, it can be flexibly combined with other decoding methods to further enhance their performance.

SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models
Jianyi Zhang', Da-Cheng Juan', Cyrus Rashtchian", Chun-Sung Ferng", Heinrich Jiang",
Yiran Chen'
Duke University, 2 Google Research Project Website
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities, but their outputs can sometimes be unreliable or factually incorrect. To address this, we introduce Self Logits Evolution Decoding (SLED), novel decoding framework that enhances the truthfulness of LLMs without relying on external knowledge bases or requiring further fine-tuning. From an optimization perspective, our SLED framework leverages the latent knowledge embedded within the L.M by contrasting the output logits from the final layer with those from early layers. It then utilizes an approximate gradient approach to enable latent knowledge to guide the self-refinement of outputs, thereby effectively improving factual accuracy. Extensive experiments have been conducted on established benchmarks across diverse range of model families (LLaMA 2, LLaMA 3, Gemma) and scales (from 2B to 70B), including more advanced architectural configurations such as the mixture of experts (MoE). Our evaluation spans a wide variety of tasks, including multi-choice, upen-generating and adaptations to offin-of-thought reasoning tasks. The results demonstrate that SLED consistently improves factual accuracy by up 20% compared to existing decoding methods while maintaining natural language fluency and negligible latency overhead. Furthermore, it can be flexibly combined with other decoding methods to further enhance their performance.
Introduction
Large Language Models (LLMs) have achieved remarkable breakthroughs recent years, demon- strating exceptional performance across various domains [1, 2, 35, 36, 44, 47, 48). How- ever, a significant challenge associated with LLMs is their tendency hallucinate or distort the truth, resulting in outputs that are not factual [15, 17, 65). This issue of hallucination undermines the reliability and trustworthiness of LLMs in practical applications. popu- lar strategy for improving the LLM factuality involves refining the decoding process [43, 53]. Decoding focuses on how the model selects the next to- ken during the generation process, which significantly influence the factual accuracy of the output. The decod- ing methods
Factuality Distribution
be cost-effective since (a) they do not rely on external knowledge and (b) no additional train- ing is required. Furthermore, decoding methods be synergistically combined with other techniques aimed at improving the LLM factuality, such retrieving infor- mation from external knowledge bases [24, 25], various fine-tuning strategies for better alignment [46, 48], semble learning methods [10].
Implicit Learning
Explicit Training
LLM Latent Knowledge
Factuality Decoding
LUM Output Distribution What the tells
Inferance
What the knows
Figure 1: Factuality decoding overview.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

X False
True
Richmand
Surrey
Vansownn
Victoria
Richmond
Surrey
Vancauver
Victoria
cotorer
sotmner
Original logits: logitsy
Self-evolved logits: logits logitsy - a- (KL( Intent Pugitss)
latens to estimate
N-th layer (final layer)
the real-worid distribution the self-evolution of So
Contrast logits uncover latent knowledge
24th Layer
Earty-oxit
logity24
LLM
16th Layer
loggts::
logits
Latent knowledge P latent
8th layer
Early-exit
togtts,
The capital of British Columbia province is
Figure 2: Illustration our Self Logits-Evolution Decoding (SLED) workflow.
Recent studies [22, 26, 42, 50] suggest that LLMs sometimes have learned the factual content based on extensive pretraining or fine-tuning, although they fail to produce the correct answer when user queries the model. This has inspired the development of several factuality decoding methods [7, 26, 27, 64] to reveal what the model implicitly "knows." Figure 1 summarizes the underlying mechanism of these factuality decoding methods. The LLMs output distribution is derived by applying the softmax function to the output logits from the final layer. During the training phase, this distribution is optimized based on the real-world factuality distribution represented by the training dataset. However, during the inference phase, "what the LLM tells" might still contain factual errors, which implies a discrepancy between the output distribution and the real-world factuality distribution. While the real-world distribution remains inaccessible during the inference phase, the model's 's latent knowledge ("what the model knows") may have implicitly learned some factual content correctly during the training phase [22, 50]. Therefore, a key challenge for factuality decoding strategies lies in effectively harnessing the latent knowledge embedded within LLMs to refine the output distribution (logits) during inference.
To address this challenge, we propose Self Logits Evolution Decoding (SLED), novel factuality decoding approach that leverages the latent knowledge within LLMs by contrasting the final layer's logits with early layers' logits. During the decoding process, as LLMS progress from early to final layers, they progressively incorporate factual information stored in each layer into the output. SLED tracks this evolution process to uncarth the latent knowledge within LLMs, and enables the "self- evolution" of the output distribution further to align it more closely with real-world facts. Furthermore, our approach recognizes that the latent knowledge within LLMs, while valuable, may not always be perfect. Therefore, instead of simply replacing the original outputs with this latent knowledge, SLED integrates it into the original logits through an operation similar to "single-step gradient descent" the output logits during the inference time. This operation minimizes the Kullback-Leible (KL) divergence between the latent knowledge distribution and the output distribution, effectively balancing the two and mitigating potential drawbacks such as overfitting or biased outputs. Figure 2 illustrates the SLED workflow, highlighting how SLED optimizes the output logits, leading to a more factual output distribution. We evaluate SLED on various LLMs (e.g., LLaMA [48], LLaMA 3 [1]. Gemma [31]) and benchmarks to demonstrate its state-of- -fth--rrt performance in layer-wise contrastive decoding methods. In summary, our main contributions are:
We propose SLED,, a novel decoding method that aligns LLMs outputs with factual knowl- edge without requiring external knowledge base or fine-tuning data.
We conduct extensive experiments across range of LLMs, with varying configurations and scales. The results demonstrate that SLED consistently improves factual accuracy various tasks and benchmarks, including multiple-choice, open-ended generation, and chain-of- -fhouugh-thh reasoning tasks.
SLED be flexibly integrated with other factuality decoding methods to enhance their effectiveness further.
We provide new interpretable perspective for understanding layer-wise contrastive decoding methods, paving the way for further developments in factuality decoding.

---

7B
26
31
138
30
12
24
16
30
20
22
24
266
28
30
32
34
36
30
60
10 15
25
30
40
50
55
69
65
75
80
Layer Index
Figure 3: We analyze the next-token predictions of three LLaMA-2-base models using the logits from each layer individually. This analysis is performed 200 true claims from the FACTOR dataset. The results verify that the logits distribution at the final layer is closer to the real-world distribution than all the early layers in terms of KL divergence.
2 Self Logits Evolution Decoding
large language model, equipped with layers and a vocabulary [V,, ..,,.... val. typically generates text in the next-token prediction fashion. For each given prefix, the model computes the logits at the final (N-th) layer, logils N A (f(1,)), ((,.)), (2..)).... (a,N)), which are obtained by applying a linear transformation to the hidden states of the final layer, projecting the nigh-dimensional hidden state vectors onto the space of the vocabulary size. Subsequently, the output distribution P Plogstay the final (N-th) layer for the next token is derived by applying softmax function on the logits,
P Plogits x (P(1.N)) (VC...))....PP...N.) = sof Imar (logils N /T),
where r is the temperature parameter. Therefore, for each P(i,N) (1 L d). we have
P(i.)) = exp(( where S = Σ exp(((,.N)/T).
Similarly, we can also derive the logits from early layers by applying the same linear transformation mentioned above their hidden states. For any early layer n (n < N), we denote its logits as logits, A ((d.))) and the corresponding distribution as Plogits, A (P(1,a).... P(d.nn).
2.1 Logits Evolution
To improve factual accuracy, it is crucial that the correct token receives a higher value of logits n ensure higher probability value P(:.)) in the output distribution Piogite From mathematical perspective, this means aligning the model's output distribution Piogits closely with the real-worde factuality distribution Preat. Specifically, we can formulate this goal as optimizing the following loss function L regarding the logits:
C(logits) KL(P real, Plogits), ) where logits (f1, La). Plagits softtamat (logits/7 /T)
(1)
We describe the above optimization as Logits Evolution. Interestingly. the training of LLMs also aims minimizing the divergence (typically the KL divergence, as the training loss function is often the cross-entropy loss) between the ground truth Preat and the output distribution Piogits, During the training phase, the logits evolution driven externally by the real-world distribution Prest presented in the training dataset, and the corresponding solution is logits logils x- However, Preat is not accessible during the inference phase. To address this challenge, SLED utilizes the model's latent knowledge estimate Preat and enables "self-evolution" of the logits. We denote the estimation as Piatent and the self logits evolution can achieved by the following gradient-descent operation:
logits n logits n V Vogiss KL(P latents Plogitsy)
The parameter a, termed the Evolution Rate, governs the magnitude of adjustments applied to logits y in the direction of the gradient V iogits K [iPiatent: Ptogits,)) ). the following Section 2.2 and 2.3, discuss how derive the Ptatent as the estimation of the real- world distribution Preet-

---

2.2 Estimate Preat by Tracking the Logits Evolution Direction throughout Layers
The core principle of our method involves leveraging the difference between each early layer's logits and the final layer's s logit, logits,, logits, n to approximate the gradient of K (Prral: P Plogits) at logits = logits,,. ,.. Then we estimate Preul based on this approximation.
This is inspired by a new perspective of interpreting the training phase of L.MMs as the evolution of logits described in Problem As mentioned above, the solution derived by the training phase the final layer's s logits logits = logits N- since the final layer's logits N directly engage with the real-world distribution Preat through the loss function in training. This implies that we can generally consider the final logits logits N to be a better solution than the logits from an early layer logits, n. with demonstrate
L (Peat, Plogits.) < ((Preat Plogits..). We present some examples Figure this. Based on this discussion, if we contrast the final layer's logits with the early layer's ss logits, we can consider the direction (orientation) of logits, n logits n can upproximately align with the direction of the gradient V "logittKLL L. Plogats logits= To further verify this motivation, we calculate the cosine similarity between logits, thousands of tokens across different models in Figure 7. We find that the majority of these values are positive, which means that the directions of these two vectors are close.
logils and Viogits, K I.(Preat: Ptogits,) for
Hence, for each early layer n, we propose to maximize the following function of cosine similarity and derive the p(n) to estimate the Preat. latent
p(n) Platent
arg max (CosSim(logiaas logilss -logils V Iogits, KL(P, Plogits,) ).0)
P
2.3 Achieving the Self Logits Evolution in Three Phases
Based on the above analysis, for each early layer n using the gradient approximation in Section 2.2. Subsequently, we apply a weighted average on (P(n) iatent across all early layers < N to derive Platent, which serves as the final estimation of the real-world distribution. Finally, we apply Pintent in Equation 2 to facilitate the self-evolution of logits n, thereby derive the updated logits, logits N.
introduce the procedures of SLED: First, we estimate P(n) latent
direction
logits, - -logils n
V legiss,,
KL(Prral reat, Piogits,)
Plo))
latent
Phase
Platent
Phose
logits n
Estimnate
Ensembble
Self-evolution in Eig
Phase 1: An exhaustive search for exact solution to the complex optimization problem (Equation 3) is computationally impractical. We can reduce the solution space by the following. Suppose the real-world factuality distribution dictates that the next word to be generated is the i-th token vs from the vocabulary V. Thus Preat = Pee, where Pe. represents a standard basis vector (one-hot vector) with the i-th component set to I and all other components set to O. Then, can simplify the aforementioned optimization problem by limiting the solution space to {Pe, dd Ji=o and decide which token i should be selected. The corresponding gradient when P = Pa, has the following formulation. Proposition The gradient of [L((Pe, Plogits) at logits = logits, is:
V togits, K (T((Pe,) c,3 Plogits.)) = (Piogits_ Pe,))/T \/T (P(1.n):--.. (((..n))----PPPP.) P(t.n) -1....,((d,))) /T
We calculate the cosine similarity between the gradient V logits, KL(Pe,, .Plogste.) ( ) and the difference logits,, logits, x for each token in the vocabulary V. Then we select the Pe,. of which the gradient is closest to logils, n according to the following criterion
latent" Mathematically, this involves selecting i* logils n as the estimation p(m)
1 = arg max m where vi; (n) max (CosSim(Logiie - logits N. Plogits. - -) 15i5d
(n)
and adopting p(n) latent Pe.. as the "hard estimation" of Preat- Drawing from the concept of hard and aurtherextend extend to the "soft estimation",
soft targets in label smoothing and knowledge distillation, we further
((n)
intent = (m] (n) m (n)
,...,,.. m / where m, (m (2)) and m (m) - m (n)
We square {m7 (m) } to moderately amplify their differences. Prior studies prove that soft targets usually offer stronger generalization capabilities, more information, and more robustness to noise than hard targets [13, 34, 45, 59, 62]. Hence, we adopt the soft estimation in lieu of the hard estimation.

---

Question (The ground truth
hours overtime, her
rate
first
are x
f-2
[-1
fo
fr
fz.
fa
works each week is $10. She also receives an overtime pay of times her regular hourty rate. Eliza worked for 45 hours this week, earnings this weak?
values across layers
Z
F Fingitess
LLaMA-2-7B -B-Chat
pau laoet
0.150
Eliza's regular hourly $10, For hours, hours
earnings are
0.125
overtime, her earnings are 510 $50, So her total earnings for the week are $400 $50 $450. The answer $450, (Wrong)
y
0.100
P
0.050
LLaMA-2-7B--o SAMAA---B-Chat-- SLED
Eliza's as regular hourly $10. For hours. her earnings are 610 $400. For hours avertime, her
(01
0.025
$10 $60, $60 - $460. The answer $460. (Correct)
pib)
0.000
16
total eaminns
( Iatent
#Layer
Figure 4: An example from GSM8K demonstrating SLED's mechanism. SLED derives the estima- tions (x)
latent by contrasting final layer's logits logits n with early layers' logits {logits,). We list the token with the highest probability value from the Platent for different early layers. As shown, SLED
P(x) downplays incorrect tokens by assigning lower weights s(x) to the corresponding latent' Conversely, if the estimation is correct, the weights are relatively larger. The parameter evaluation scale is set to 2.
p(a) latent across all layers by computing a weighted average of the set (pla) and adopt it as the final estimation of the Platent:
Phase 2: We ensemble
Plstent
(()) latents where s(() = m
n=0 m())
n=
This estimation suggests that the weight ((n) of certain layer n will be larger if the corre- sponding gradient approximation logits,, logits N is more closely aligned with the gradients (oigits, L Pioguts,,) }} for the tokens in the vocabulary. This in turn amplifies the influence of layer the final estimation, which is a desirable effect in our method. Figure 4 demonstrates that SLED can downplay incorrect tokens based on the gradient alignment. One can further validate that for each component m, in the final estimation Piatent A (m1, m2,..., ma), the following relationship holds: m, En=o m(") /(En E ) This property simplifies the description in Algorithm 1.
Phase 3: Applying Platent in Equation enables us derive the gradient necessary for steering the self-evolution on the final layer's 's logits logits
Proposition 2. The gradient of K
P Plogits) logits logits x is:
V logits, , K L(Pittennt P Plogits)) ) = (P Plogits - Plotent )/T = (P(L.N) ....... P(d,N) -ma) /T
Then derive the self-evolved logits logits n
logils x A ((..)) Lr,,)), where E(,N) ((.N) a(p(NN) mz)/T. (5)
2.4 Computational Complexity and Design Decisions
For each layer, computing CosSim(logits,, logits n: Plogits, Pe.) for every token v; in the vocabulary V needs ((d)) operations. To reduce the computational complexity, we select only subset Vi,- where the token E Vr, has the top-k highest logits in the final layer. In this scenario, we only initiate the self-evolution in Equation of the logits corresponding to these top-k tokens. For the remaining tokens, which have lower probabilities, their logits are adjusted to a very lower numerical value, e.g., 1000. This strategy significantly reduces the computational complexity, while maintaining focus on the most relevant tokens. We name the parameter k, as Evolution Scale, since determines the number of top- cap-prrbbbblity tokens active for self-evolution.
Q 2:1: Why SLED contrast the final layer with all the early layers, instead of picking one premature layer to contrast based JSD?
DoLa a selects a subset early layers to form a candidate set. Then calculates the Jensen- Gensen-Shannon Divergence (JSD) between the final layer and each layer in this set. Their strategy is to choose the
5

---

Algorithm Self Logits Evolution Decoding
1: Initialization: LLM with N layers, inpuls, evolution rate a, evolution scale k: > 0. 11 < 0, temperature parameter T, and the one-hoe vectors {Pe: defined in Section 2.3.
2: Feed the imputs into the LLM to obtain the logits logits, x. = (f(I..).... (dd,)) and probabilities
Plogits, = (P(..)):-... P(d,,)) = sof tmor( logits,, /T) at each layer n, where n S N.
3: Identify the tokens with the lar-esa values in logits, x and denote their indices by In.
4: for each early layer rt, (n < N) do
5: Compute differences for top-k logits logits, - logits N.
(m) [max (CosSim( (logits, logits N: Plogsts, - -P.)),)] , 1x- 6: Calculate m;
7: end for
> m!
8: Compute weighted average m, = 2 Eeery m) across different layers for each i E LR-
9: for each i from to d do
10: Set ((I,N) (i, (P((,,)) mz) if E Is else Set (i,N = 11 <
11: end for
12: Output: The self-evolved logits are logils N = (l(1,,), _aN,) (a.. ,.............)
layer with the highest JSD as the premature layer, and the chosen layer will be contrasted with the final layer to update probabilities. Obviously, if this strategy reasonable, a larger candidate set should lead to better choice of the premature layer and, consequently, enhanced overall performance. However, a paradoxical finding from their experimental results, which our tests also confirm in the discussion in Section 3.5, is that a larger candidate set for DoLa leads to decreased performance. Specifically, when the candidate set for Dol aa ranged from 0 to 32 layers for ---M----7B-- BBBaase, the performance inferior compared that selecting a good candidate set remains a challenge when applying DoLa. In contrast, our method does not face this concern as it applies an ensemble approach to all early layers. It is also important to note that our method works well even when only contrasting the final layer with part of the early layers, as demonstrated in Section 3.5 and B. proving the robustness of our approach.
smaller of 0 to 16 layers. This fundamental flaw indicates
@ 2.2: Why not use Ptatent directly as the model's output distribution?
It is crucial to understand that Platent is merely an estimation of the real-world distribution based on the model's latent knowledge instead of the exact Preat. Consequently, relying solely on Plotent- similar to DoLa, la might lead to inaccuracies, as the latent knowledge can be imperfect. The original logits logits, n are still important as they are refined directly by real-world data during training. The evolution rate a in Equation 2, serves to balance this trade-off, enabling a reciprocal enhancement between Piatent and the orig original logils x. More ablation studies are provided Section 3.5 and B.
Q 2.2: Considering that SLED adopts logils, logils n as the estimation of the gradient, why not directly apply it in Equation 2?
is important to note that while logits, n - logits, n is unconstrained, the gradients estimated in Equation 2 (c.g., P(1.N) mi..... : ma) are constrained within [-1, 1]. Thus, direct substi- tution could lead to mismatch in magnitudes and might also introduce unexpected noise. Proper normalization and subsequent aggregation of estimations from different layers precisely what method addresses in Section 2.2 and 2.3. Further analysis is provided in Section B.
Experiments
As a novel layer-wise contrastive decoding approach, we first benchmark SLED against the state- of-the-art approach DoLa [7] across diverse range of model families (LLaMA 2, LLaMA Gemma) and model scales (from 2B to 70B). including the more advanced mixture of experts (MoE) architecture, as detailed in Section 3.2 and 3.3. The results showcase notable factuality improvements across a variety of tasks, including multi-choice, reasoning tasks. Then, in Section 3.4, we integrate our method with other established factuality decoding techniques, illustrating that SLED can further enhance their performance. In Section 3.5, we further conduct in-depth studies on mitigating the repetition issue, layer selection, various parameter settings, and latency overhead to gain more comprchensive insights into SLED's performance. We
generrener,,onn and adaptations to chain-of chain-of-thought

---

Table Comparison on LLaMA 2 model family. The best results are in bold for each dataset/metric. SLED outperforms DoLa and vanilla greedy decoding.
TruthfulQA (MC)
TruthfulQA (Open-Ended)
CoT
Motho&Method
FACTOR
MC1
MC2
MC3
&Truth
&Info
%T*
%Rcject
SiQQ
GSM8K
LLaMA-2-7B-Base
33.17
59.42
31.78
58.15
32.80
90.09
23.99
8.45
60.96
14.03
+DoLa
32.56
63.03
30.57
62.49
35.74
95.23
32.31
2.57
60.61
14.71
+SLED (ours)
34.15
62.57
31.89
67.27
55.81
94.61
52.87
0.12
61.31
15.01
LLaMA-2-7B-Chal
35.62
57.46
32.07
56.78
59.24
78.95
38.68
17.50
63.67
21.08
+DoLa
33.41
61.93
30.35
56.65
58.02
87.03
45.78
13.10
64.32
21.00
+SLED (ours)
37.08
63.86
32.90
64.70
67.07
88.13
55.69
11.02
64.67
21.15
LLLMA-2-11B-BBa.
33.69
62.75
31.74
63.69
31.21
91,55
23.26
7.96
66.07
28.66
+DoLa
29.25
62.13
30.29
57.08
37.58
92.41
30.11
7.47
65.55
18.88
+SLED (ours)
34.15
63.62
31.89
70.91
38.31
94.85
33.29
5.02
66.81
29.34
LLLMA-2-11B-
36.47
63.05
32.77
62.06
60.34
86.54
47.12
13.59
69.87
36.47
+DoLa
34.52
63.24
31.48
58.08
60.22
90.33
51.16
9.67
67.90
34.57
+SLED (ours)
37.09
63.75
32.60
67.50
63.65
95.23
58.87
5.26
69.96
36.54
LLaMA-2-70B-BBn
33.66
61.10
32.33
72.78
55.45
62.55
18.48
36.74
75.20
56.33
+DoLa
26.93
60.33
29.42
61.92
60.95
70.62
32.07
17.72
73.45
43.37
+SLED (ours)
35.13
64.92
33.52
77.49
82.99
43.70
13.10
75.20
$7.09
LLaMA-2-70B-CChn
35.98
64.18
32,99
69.07
49.57
81,27
31.33
29.13 13
77.25
54.59
+DoLa
31.58
54.40
32.31
58.28
61.44
77.97
39,90
21.28
74.41
49.05
+SLED (ours)
38.31
66.71
34.66
73.98
62.55
84.70
47.74
14.98
77.38
54.81
also extend our analysis with additional ablation studies and results across more benchmarks in Section B and D in the Appendix, and provide several examples of generated text as the qualitative study in Section
3.1 Experimental Setup
Benchmarks We compare our method with baselines on several multiple-choiee and open- endee generation tasks, For multiple-choice question tasks, we use the TruthfulQA [29] and FACTOR (Wiki) [33] datasets to assess the LLMs' factuality in short-answer/long paragraph scenario, respectively. For open-ended generation tasks, we adopt TruthfulQA [29] and tasks involving chain-of-thought reasoning [52]: Strategy [12] and GSM8K [8].
Models & Baselines We evaluate the performance of SLED on six LLaMA-2 models [48] [7B,,1B,70BB (-B,70B)-BB --Bsee, (3B, (113B,00BBB 70BB70BBB-hhht J-hat), four LLaMA-3 family models [1] (18B,70B)-Buse (13,70J-ITT,, two Gemma models (2B,7B), two MoE models (Mixtral-8 7B, Mixtral-8x7B: IT) [18]. We adopt the following baselines: 1) standard decoding (greedy decoding or sampling depending on the tasks), 2) Dol.a [7]. 3) Inference Time Intervention (ITT) [26), 4) Activation Decod-
ing (AD) [4]. 5) Contrastive Decoding (CD) [27]. and 6) Seatuce-then-oontaee Decoding (ICD) [64].
Metrics We adopt the factual accuracy evaluation implemented in [7] for multiple-choice tasks and chain-of-thought reasoning tasks. For the open-ended generation task TruthfulQA, follow the evaluation procedure in [7, 29], using finetuned-GPPT--juu informativeness, and rejection rate of generated outputs respectively.
to measure the truthfulness,
3.2 Evaluation on Broad Range LLM Benchmarks
Multiple-Choices Tasks The objective of these tasks is to employ decoding methods that enable LLMs assign higher probabilities to correct completions/answers over incorrect alternatives. We demonstrate the effectiveness of SLED for both Short-Answe Factuality on the TruthfulQA and Long-Paragraph Factuality on the FACTOR dataset. For both DoLa and our SLED, we contrast the results from the final layer against all preceding layers. We randomly sample approximately 5% of the data for validation regarding parameter selection. The results, as shown in Table 1, indicate that SLED achieves superior outcomes in almost all metrics across six LLaMA-2 models. Notably, SLED
7

---

Table 2: Using SLED with other LLM families also improves the factuality.
TruthfulQA
TruthfulQA
Model
FACTOR
Model
FACTOR
MC1
MC2
MC3
MC1
MC2
MC3
LLLMA-3-8B
64.33
33.78
63.00
32.59
Mixtral-8x7B
71.41
35.13
49.98
34.17
+DoLa
68.04
33.29
63.35
32.16
+DoLa
58.28
32.44
35.91
33.68
+SLED (ours)
68.67
35.13
64.09
32.50
+SLED (ours)
74.92
35,86
57.26
32,96
LLaMA-3-8B-IT
$9.49
38.92
68.16
36.50
Mixtral-8xx -B-Ira
70.51
37.94
62.51
35.25
+DoLa
61.06
35,86
65.30
33.78
+DoLa
56.15
32.19
39.17
33.76
+SLED (ours)
67.17
42.23
69.03
37.97
+SLED (ours)
75.55
41.73
68.52
37.70
LLaMA-3-70B
78.72
35.62
65.66
34.18
Gemma-2B
50.87
23.38
37.16
17.42
+DoLa
77.56
33.29
64.83
32.81
+DoLa
32.93
26.07
48.97
26.55
+SLED (ours)
80.83
37.58
66.19
34.11
+SLED (ours)
57.05
25.21
50.20
26.94
LLaMA-3-70B-IT
73.95
44,80
70.29
41.02
Gemma-7B
60.42
31.58
47.63
22.75
+DoLa
71.51
38.43
68.70
35.21
+DoLa
36.07
25.21
43. 14
26.13
+SLED (ours)
76.85
48.35
74.03
43.16
+SLED (ours)
65.56
32.31
49.88
25.22
achieves better performance under the MC1/MC3 metrics on TruthfulQA, which are more sensitive to fluctuations and pose greater challenge. For long sentences FACTOR, our method shows improvements over baselines by 5-13%. These results not only underscore the benefits of our method for factuality but also demonstrate its robustness across different lengths of text.
Open-Ended Generation Tasks In open-ended settings, we prompt the model to generate answers for the same questions from TruthfulQA, following the settings outlined [29, 7, 27]. In Table 1, we compare the performance of six LLaMA-2 - models using standard greedy decoding, (greedy) DoLa, and (greedy) SLED.. All the generated answers are then evaluated by a fine-tuned GPT-3 model for both truthfulness and informativeness scores. Considering that 100% truthful score can be easily achieved by simply responding with have no comment," which would result in 0% informative score and thus is not very useful, we have introduced additional metrics- Truth x Info and the rejection ratio &Reject- -to demonstrate that SLED is mutual-gains approach to achieve better both truthful and informative scores. We have improved the overall &rruth x Info scores by 3-20% across different models and reduced the rejection ratio by up to 95%. These enhancements demonstrate that method effectively avoids the 'reeection pitfall,' making it more helpful.
Adaptation to ohain-of-thought Reasoning Tasks Although the Strategy QA and GSMSK tasks are also open-ended and require factual accuracy, the primary focus here is to evaluate how different decoding methods adapt to the Chain-of-Thought (COT) approach for handling complex reasoning tasks. We maintain repetition penalty of 1, as will discuss the repetition flaws associated with DoLa in Section 3.5. Strategy demands multi-hop reasoning, and as shown in Table 1, our method boosts accuracy across six models, whereas DoLa generally worsens it without repetition penalty. GSM8K, a benchmark for math word problems that require arithmetic reasoning, also shows consistent accuracy improvement with SLED in 7B, 13B and 70B models.
3.3 Evaluation Across Diverse LLM Configurations
As discussed above and shown in Table 1, our method, SLED, demonstrates strong generalization capabilities across the LLaMA-2 model family, proving robust from 7B to 70B model sizes. In Table 2, we further showcase SLED's s impressive performance on the more recent LLaMA-3 family models, both at 8B and 70B sizes, in terms of long paragraph factuality and short answer factuality. Interestingly, SLED also applicable to different pre- treitrained models, such as Gemma at both 2B and 7B sizes, and can even be adapted to the increasingly popular Mixture of Experts (MoE) architectures. These results confirm the exceptional adaptability of our method across various LLM configurations.
3.4 Evaluation on Integrating SLED with Other LLM Factuality Decoding Methods
SLED exclusively focuses on contrasting differences between layers without altering other parts of the model. Thus, remains compatible with other techniques that incorporate additional strategies or utilize auxiliary models. This compatibility allows SLED to be seamlessly integrated into existing
8

---

Table 3: Comparison of decoding strategies TruthfulQA datasets. SLED can also be seamlessly combined with other decoding strategies improve performance further.
Model
LLaMA-2-7B-bass
LLaMA-2-7B-chat
AD
AD +SLED
AD +DoLa
AD +SLED
ITI +SLED
ICD
Method
AD
AD
ITI
ICD
+DoLa
+SLED
MCI
32.80
25.58
33.29
35.37
33.41
36.23
36.60
43.33
46.32
46.87
MC2
59.59
39.06
62.55
58.14
50.31
63.15
65.62
65.75
69.08
72.09
MC3
31.05
17.89
31.80
31.84
23.15
32.23
34.89
37.66
41.25
43.64
Model
ILaMA-2-13B-BB
I.LMA-2-13B-chhm
AD +DoLa
AD +SLED
CD
AD +DoLa
AD
CD +SLED
Method
CD
AD
+SLED
+SLED
MCI
33.90
24.72
33.90
30.11
33.78
36.84
34.72
36.35
28.
36.47
MC2
62.93
37.74
63.69
50.31
63.22
63.75
50.42
64.83
54.87
64.93
MC3
31.61
17.66
31.38
28.18
32.21
32.69
23.83
32.85
29.75
33.39
Table 4: Accuracy of LLaMA 2 13B Base on StrategyQA with Varying Repetition Penalties
Metric
Method
1.02
1.04
1.06
1.08
1.1
1.2
2
DoLa
65.55
65.98
66.37
65.98
65.59
66.37
67.16
66.64
Accuracy(%)
SLED (Ours)
66.81
69.39
68.51
68.47
67.07
65.72
60.87
54.75
DoLa
7.63
7.19
6.45
5,98
5.50
5.10
3.73
2.05
Eppetition-4(%)
SLED (Ours)
3.73
2.45
1.89
1,36
1.05
0,69
0.20
0.10
DoLa
2.16
2.04
1.66
1.37
1.12
0.89
0.23
0.03
Repetition-Sen(()
SLED (Ours)
0.88
0.39
0.10
0.02
0.03
0
0
0
methods, enhancing factuality further without the need for modifications to SLED. We integrate SLED with the following approaches: ITI, AD, CD and ICD. Table 3 shows that SLED leads to accuracy improvements from 1% to 12% across four LLaMA-2 models.
3.5 Ablation Studies and Analysis
Mitigating Repetition Issues Table demonstrates that our method, SLED, effectively addresses significant issue in DoLa: repetitive content in open-ended generation tasks. Our approach outperforms DoLaa without the need for excessive repetition penalty. While a slight increase the repetition penalty further enhances the performance of method, excessive penalties, such as 1.1, tend to degrade it. This suggests that SLED does not inherently require heavy adjustments for repetition issues. In contrast, DoLa's performance improves with higher penalties (e.g.. 1.1, 1.2, 2), indicating a more critical need for addressing repetitive content. We also employ two intuitive metrics, Repetition-4 and Repetition-Sen, research [55]. Regardless the repetition penalty imposed, our method consistently exhibits lower repetition rates. Table 7 includes some examples of generated text illustrate this further.
gauge the severity of repetition issues, following prior
Layer Selection As discussed in Section 2.4, how to choose a good candidate set is still para- doxically difficult task when applying DoLa. Our method does not exhibit this issue. Instead of selecting single premature layer from the candidate set like DoLa, SLED contrasts the final layer with layers in the candidate set and then ensembles the results. Figure shows that setting a larger candidate set, such as all the 32 layers for LLaMA-2-77 Base, yields better performance than focusing solely on either the first (0, 16) or second half [16, 32). This implics that our layer-wise contrast approach captures more useful information in more scientific manner. Furthermore. tests confirm the robustness of our method even when the candidate set is minimal, such as a single layer, consistently demonstrating strong performance. Our settings mirror those of DoLa.
Parameter Analysis We next investigate the impact of parameters evolution rate and evolution scale à: - on the performance SLED using subset of the FACTOR dataset. We test evolution rates from (0.01, 0. 1, 2, 5, 10) and evolution scale values from {5, 10, 20, 50). Without extreme

---

Figure 5: Evaluating using different premature layers for SLED and DoLa on 10% subset of the GSM8K dataset. Contrasting all layers for SLED better than using only the first half [0, 16) or the second half [16, 32), Hence, there are no improvements for SLED from strategic layer subset selection.
13B
20
(0.15)
DeLa [0:32]
(0.35)..
Duys 00:407
Ourr (0,20)
(single)
Sroadd
(1.22))
Accuracy
244
Grondy
22.5
DoLa (0,20)
Greedy
20.0
Greedy
Ours (singl))
Ours (sigle)
(single)
[8.10]
17.5
Ours [0,20]
Ours [16,32]
Oura (20,40)
Gola (0:40)
Dela (20,40)
15.0
DoLa (16.32)
12.5
15
20
30
35
20
25
30
40
Premature Layer
Premature Layer
Figure 6: WE explore the impact of evolution scale and rate based on the factual accuracy of a subset of the FACTOR dataset. (G: Greedy, D: DoLa)
Evolution Rate
Evolution Rate
Evolution Rate
Evalution Rate
Evalution Scale
41.5
61.5
+1.
615
59.5
53.0
67.0
66.5
67.5
69.0
63.0
51.0
54.0
$4.0
56.0
57.5
56.0
49.0
54.0
54.0
56.0
57.5
$6.0
49.0
da
66.5
64.5
67.5
60.5
10
70.5
69.0
10
65.0
66.0
64.5
10
65.0
64.5
66.0
64.5
05.3
64.5
64.5
57.5
68.3
63.5
59.3
71.0
05.0
$6.0
59.0
60.5
58.5
58.0
36.0
59.0
$8.5
S21Gi
56(0)
$4.51D]
64(c)
SSCD)
S7(0)
56,510)
+110]
(a) LLaMA 2 7B Base
(b) LLLMA 2 7B Chat
(c) LLLMA 2 13B Base
(d) LLaMA 2 13B Chat
evolution rates (e.g.. 10), method performs well, confirming its robustness. As analyzed in our methodology and Eq. 2, the evolution rate balances the logit distribution (Px) with the latent knowledge distribution (Piatent). A lower evolution rate works better for larger models (13B) and chat models as their logits already better represent real-world distributions.
Latency Our method, SLED, does not incur significant latency overhead. The latencies presented in Table demonstrate that our method, SLED, just increases the decoding time of Dol.a la by factors ranging from 0.1 % to 10%. Notably, even with an atypical setting such as evolution scale = 100, which is seldom used, the increase remains around 10%. The latency for DoLa and SLED is much higher compared to the vanilla greedy decoding because we set all early layers as candidate layers set for both DoLa and SLED for fair comparison.
Table 5: Latency (ms/token) comparison across different configurations. (ES: evolution scale)
Model
Greedy
DoLa
SLED (ES=5)
SLED (ES=20)
SLED (ES=50)
SLED (ES=100)
LLaMA-2-7B
23.64
29.93
30.41
31.15
32.70
34.63
LLaMA-2-13R
30.41
39.57
39.61
41.14
43.30
45.09
LLaMA-2-70B
82.63
136.42
138.33
140.24
143.12
148.85
4 Conclusion
We introduced Self Logits Evolution Decoding (SLED), which is a new method to improve accuracy and factuality without requiring external knowledge (e.g.. RAG) or fine-tuning (e.g.. SFT). The key idea is to optimize the output logits based on the LLMs' latent knowledge to improve factuality during inference. On several datasets, SLED achieved the SOTA results, improving over the vanilla decoding and DoLa. We also show that SLED does not increase the inference time significantly and that it can combined with other factuality decoding methods. For future work, would be interesting to combine SLED with supervised fine-tuning methods, e.g., to adapt to other domains.
10

---

Acknowledgment
This work was done when Jianyi Zhang was an intern at Google Research. In addition, Jianyi Zhang and Yiran Chen disclose the support from grants NSF CNS-2112562 and ARO W911NF-23-2-0224 We thank area chair and reviewers for their valuable comments.
References
[1] @ Meta. Llama 3 model card. 2024. URL WWR..ttts///ggiibbbcoo///// wawp://mmeaaaaaaaaaaaaaaaaaaaaaa allaammlaaaaa #lob/main/MODEL_CRRR md.
[2] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprini arXiv:2305. 10403, 2023.
[3] Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language mod- els in retreeedl-augmeented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754-17762, 2024.
[4] Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. arXiv preprint arXiv:2403.01 2024.
[5] Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. Lift yourself up: Retrievallaugmmened text generation with self-memory. Advances in Neural Information Processing Systems, 36, 2024.
[6] Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017.
[7] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.oo// wwwpiiiwwwweeeeoeeeeeeeee wttes//foommmid-hhhoea
He. Dola:
[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4/68, 2021.
[9] Yujuan Ding, Wenqi Fan, Liangbo Ning. Shijie Wang. Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meets Ilms: Towards retrieval -atreeeal-uummented large language models. arXiv preprint arXiv:2405.06211 2024.
[10] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2024. URL https: / //openreview. net/
[11] Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16477-16508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18665/////223.accl// URL. https: httppp::///aaaaaaaa htt/a/aaaaannooooy.yoooooooo.lae wrog/////022 acl- long. 910.
[12] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
[13) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge a neural network. arXiv preprint
2(7), 2015.

---

[14] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint Xii::1904.09751, 2019.
[15] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qian- glong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232. 2023.
[16] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint 17X1::7111.0944 2017.
[17] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1 2023.
[18] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mixtral of experts, 2024. URL https: aariiv www...rggggoo.... absaabbs/111// 04088.
[19] Hadi S Jomaa, Josif Grabocka, and Lars Schmidt- Thieme. Hyp-rl: Hyperparameter optimization by reinforcement learning, arXiv preprint arXiv: 1527, 2019.
[20] Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, and Cyrus Rashtchian. Sufficient context: A new lens on retrieval augmented generation systems. arXiv preprint arXiv:241 1.04411.0677,, 2024.
[21] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.0355| 2017.
[22] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan. Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield- Doddss, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.0522/, 2022.
[23] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Woi Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguisties, 2019.
[24] Deren Lei, Yaxi Li, Mengya Hu, Mingyu Wang, Vincent Yun, Emily Ching, Eslam Kamal, et al. Chain of natural language inference for reducing large language model ungrounded hallucinations. arXiv preprint arXiv: 10.03951, 2023.
[25] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmentt generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9999 9474, 2020.
[26] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems, volume 36, pages 41451-41530. Curran Associates, Inc., 2023. URL httpss//scc///r/eeeeiiing neurips. 81133909997322660000770066666669
fww.fiilesppppl httppperff.leeaappp//2////////// Paper Conf Conference. pdf.
Sooper-Conferente
[27] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097 2022.
12

---

[28] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.055 05463, 2023.
[29]
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, edi- tors, Proceedings of the 60th Anmual Meeting of the Association for Computational Lin- guisties (Volume 1: Long Papers), pages 3214-3252, Dublin, Ireland, May 2022. Associ- ation for Computational Linguistics. doi: 10. (6655//1/20222aalloong.... // //aclanthology-
URI. https:
artele2..llooo.22. 229.
[30] Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm, 2019.
[31] Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L. Sifre, Morgane Riviere, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L'eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Am' elie H'eliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Cl'ement Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christine Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Lespiau, Jeff' Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katic Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev Nithum Thain, Olivier Bachem, Oscar Chang. Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L. Smith, Sebastian Borgeaud, Sertan Gir- gin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vladimir Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Brian Warkentin, Ludovic Peran, Minh Giang, Cl'ement Farabet, Oriol Vinyals, Jeffrey Dean, Koray Kavukcuoglu. Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Ke- nealy. Gemma: Open models based on gemini research and technology. ArXiv, 0bs/2203.00825, 2024. URL https: / /apii. semanti www.oolar.cl.r..r...coocraaaa..e org/ Corpus ID: : 268379206.
[32] Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. Evolving deep neural networks. In Artificial intelligence the age of neural networks and brain computing, pages 269-287. Elsevier, 2024.
[33] Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton- Brown, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. arXiv preprint arXiv:2307.06908, 2023.
[34] Rafael Müller, Simon Korbblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[35] OpenAI. Introducing chatgpt. https://opene com/ www//////////////////o/ooooeeeae wtw.t///oa/aaggt///////.mm//// November 2022.
Novembee
[36] OpenAl. GPT-4 Technical Report. arXiv e-prints, art arXiv:2303.08774, March 2023. doi: 104445500axxxw22303.87777
[37] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 15:27730-277441 2022.
[38] Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934 2023.
[39] Xin Qi and Bing Xu. Hyperparameter optimization of neural networks based on q-learning. Signal, Image and Video Processing, 17(4):1669-1676 2023.
13

---

[40] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[41] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International conference machine learning, pages 2902-291 202229911: 1. PMLR, 2017.
[42] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint 0rXiv:2206.05802, 2022.
[43] Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujuu Yang, and Wai Lam. A thorough examination of decoding methods in the era of Ilms. arXiv preprint arXiv:2402.06925, 2024.
[44] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 2Xi1:23122.1800, 2023.
[45] Christian Thiel. Classification on soft labels is robust against label noise. In Ignac Lovrek, Robert J. Howlett, and Lakhmi C. Jain, editors, Knowledge-Basse Intelligent Information and Engineering Systems, pages 65-73, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540-85563-7.
[46] Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine- tuning language models for factuality. In The Twelfth International Conference Learning Representations, 2024. URL https://openrreieer.tet wwwpennrvieeeeeeeeeeeeeee wtw.orrmmmmtooeteeettmmm www.rrmmidwwP22a.2 2d=WW222PPP4KKK
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288. 2023.
[49] Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju. Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10. 1609/aaai. w30.// 13211.12340, URL https://ojs aaai. wwwi.arr/aaaaaaaaa.....aa.waaaaa org/index.cr phtp AAA.//rticl///iee///////aeea///t 12340.
[50] Chenguang Wang. Xiao Liu, and Dawn Song. Language models are open knowledge graphs. arXiv preprint arXi::2010.11967 2020.
[51] Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, and Yiran Chen. Coreinfer: Accelerating large language model inference with semantics-inspired adaptive sparse activation, 2024. URL https: / /ttp//axivvvv.///////e////.aavvt org/ abs///ass///./////////////////// 18311.
[52] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: : /opnrevveewwweeeeeoeoeeeee net/ Mi_mmiddd__iMMMBBB___.
[53] Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, 1li Kulikov, and Zaid Harchaoui. From decoding meta-generation: Inference-time algorithms for large language models. arXiv preprint arXiv:2406. 16838, 2024.
[54] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pages 681-688 2011.
14

---

[55] Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. Advances Neural Information Processing Systems, 35:3082-3099 -095, 2022.
[56] Haoran Yang, Deng Cai, Huayang Li, Wei Bi, Wai Lam, and Shuming Shi. A frustratingly simple decoding method for neural text generation. arXiv preprint arXiv:2305.12675 2023.
[57] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL /ttps:///axii.....
abs//809 09600.
[58] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Selfrrewarding language models. arXiv preprint arXiv:2401.10020 2024.
[59] Chang-Bin Zhang, Peng-Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng. Delving deep into label smoothing. IEEE Transactions on Image Processing, 30: 1894-5996,20 2021.
[60] Jianyi Zhang, Ruiyi Zhang, Lawrence Carin, and Changyou Chen. Stochastic particle- optimization sampling and the non-asymplotic convergence theory. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Arti- ficial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1877-1887. PMLR, 26-28 Aug 2020. URL https:/ httpss///prooe p/tp://d//occcessss/ooo mlr, .rress/ v108/2haaa 2000//2hng20d... html.
[61] Jianyi Zhang, Yang Zhao, and Changyou Chen. Variance reduction in stochastic particle- optimization sampling. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Ma- chine Learning Research, pages 11307-11316. PMLR, 13-18 Jul 2020. URL https: //proceedings. mlr. press//119/2hang2aacc html.
[62] Jianyi Zhang. Aashiq Muhamed, Aditya Anantharaman, Guoyin Wang. Changyou Chen, Kai Zhong, Qingjun Cui, Yi Xu, Belinda Zeng, Trishul Chilimbi, and Yiran Chen. ReAugKD: Retrieval-augmente knowledge distillation for pre-trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1136. Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653/v/20233. 97. URL https: / aalaaaaoonnooooo..oooooooo orgg/0222 acl short.97.
[63] Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Yufan Zhou, Guoyin Wang, and Yiran Chen. Towards building the federated gpt: Federated instruction tuning, 2024. URL https: htap::///rxivv org/ at////aa///2/// 05644.
[64] Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint arXiv:2312. 157100 2023.
[65] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ocean: survey on ballucination in large language models. arXiv preprint 2rXiv22309.01219 2023.
[66] Yang Zhao, Jianyi Zhang, and Changyou Chen. Self-adversarially learned bayesian sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5893-5900, 2019.
[67] Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. In Inter- national Conference on Learning Representations, 2017. URL https httpp:://ppppee httppp::///ppeeeeeee //tppnrevieeeeeee//e net/ forum?id=r 1Ue8Hcg...
15

---

A Related Work
There have been many advances in improving training and inference to develop better out-of-the-bol LLMs [47, 48, 1, 44, 36, 28, 63, 51, 20J. Unfortunately, LLMs still suffer from hallucinations and producing non- non-factual text. This has led researchers to develop many methods to improve factuality.
Retrieval, Fine-tuning, and Preferences. Many techniques use additional knowledge graphs or fine-tuning data to increase factuality by updating the model parameters for this goal. One method is Retrieval-Augne Augmented Generation (RAG) to use external knowledge to improve generation [3.5, 9, 25]. Another option is to use post-generation retrieval and editing for improving attribution [11]. Other directions that use additional training or preference data are supervised fine- fine-tuning (SFT) [38, 46],
RLHF [37], DPO [40] or self-rewarding [58]. Complementary to these approaches, wish to improve the LLM output distribution directly without needing any additional data.
Decoding and Factuality Decoding For each prefix, the LLM generates a probability distribution for the next token on a fixed vocabulary list, and a decoding method determines how the next token is derived based on the estimated distribution. Decoding methods were initially developed to enhance the fluency and coherence of text generation, such as Beam Search (BS), which maintains the & most probable sequences at each time step. Common decoding methods also include Diverse Beam Search (DBS) [49], Contrastive Decoding [27]. Top-p Sampling [14] and so
Recently, the potential of decoding has extended beyond merely improving text readability, with some factuality decoding methods being proposed. These methods modify the generation process to focus on truthful statements rather than unsupported claims during the inference phase, aiming to reduce hallucinations. Notable recent works include Inference-Time Intervention (ITI) [26], Induced- Contrastive Decoding [64], Decoding by Contrasting Layers (DoLa) [7] and so on. ITI adjusts model activations during inference by following learned directions across limited number of attention heads to improve truthfulness. Some researchers have extended previous Contrastive Decoding [27] methods to improve factual accuracy, such as Frustratingly Easy Model Decoding [56] and Induced- Contrastive Decoding [64], leveraging differences between expert and amateur models. Most closely related to work is DoLa, which also employs contrasting logits from different layers. However, significant distinctions exist: Firstly, our method diverges in how to utilize those differences between logits to extract latent knowledge. Secondly, whereas DoLa directly substitutes the original output distribution with the latent knowledge distribution, our approach recognizes potential inaccuracies in this estimated distribution and adopts gradient descent within optimization framework integrate the model's latent knowledge with the original output.
Limitations. As we continue to refine our approach, several aspects of our method can be further developed and enhanced. Our method, SLED, achieves better factuality at the cost of operating slightly slower. Ideally, we could improve the output logits without incurring any computational cost compared to performing inference experimental results support the superiority of SL.D on multiple datasets. Parameter optimization using Bayesian methods [54, 30, 60, 61, 66], evolutionary algorithms [16, 32, 41] or reinforcement learning [67, 6, 19, 39] might also lead to more robust performance. would also be ideal to back up our results with more theoretical analysis of SLED.
the base LLM model. Another aspect is that currently,
B Additional Analysis and Ablation Studies
Justification on the Gradients Approximation of SLED in Section 2.2 To further sup- port our method's mechanism, which utilizes logils, logils y approximate the gradient of LL Plogits) at logits = logits,,. n. we manually calculate the Cosine Ammilarity(logits. logits n. V Viogit KL (loggse)loogis=== density function for different models. We find that the majority of these values are positive, demon- strating that the directions of these two vectors are very close. Hence, our gradient approximation strategy in Section 2.2 is reasonable.
) among thousands of tokens and layers. We plot the
Further Ablation Studies for Section 2.4 We design the following two ablation studies to support our claims in Section 2.4. The first study, referred to as 'Ablation 1', directly employs Platent as
16

---

3.5
Llama-2-7B-chat
Llama-2-13B-chat
3.0
Llama-2-70B-chat
2.5
Llama-3-8B-Instrucc
Density
2.0
Llama-3-70B-Instuuct
Mixtral-8x7B-Instrud
1.5
1.0
0.5
0.0
-1.00
-0.75
-0.50
-0.25 0.00 0.25
0.50
0.75
1.00
Cosine Similarity
Figure We collect 10k pairs of (logits,, -ortss,,loggee -logis N: V Vlgits, LL Plogits.)) )) on different tokens in FACTOR and different early layers. We calculate their cosine similarity values and draw the density function for each L.M.. Most of the pairs have positive Cosine similarity values, which verifies that the approximation strategy of SLED is reasonable.
the output distribution as discussed in Q 2.2. The second study, denoted as 'Ablation 2', involves directly scaling the differences, {logits,, logits J, to constrain their magnitudes within [-1, 1]. Then, we simply average these scaled differences across different layers and apply them to Equation 2 as mentioned Q 2.3. The results presented Table 6 demonstrate that the design our SLED reasonable.
FACTOR
TruthfulQA
MC1
MC2
MC3
LLaMA-2-7B-CL + Ablation 1
63.59
25.21
51.09
26.25
+ Ablation
62.73
33.66
39.83
31.47
+ SLED
65.16
37.08
63.86
32.90
LLaMA-2-13B-ChM Ablation
66.70
27.05
52.72
28.46
+ Ablation 2
66.29
37.33
45.00
31.98
+ SL.ED
67.06
37.09
63.75
32.60
Table 6: Performance comparison of ablation studies and SLED on FACTOR and TruthfulQA.
C
Qualitative Studies
We present some examples from the Strategy dataset in Table 7 to illustrate that our method addresses the repetition issue of DoLa.
D Further Results from Open-ended Generation Task Benchmarks
We have conducted additional experiments on more realistic open-ended generations datasets, Hot- PotQA [57], Natural Question (NQ) [23], TriviaQA [21]. We adopt the Exact Match(EM) and the Fl score. Different from the setting the Section we adopt (0, 2, 4, 6, 8, 10, 12, 14) as candidate layers for LLaMA 2 7B Chat model and [0, 2, 4, 6, 8, 10, 12, 14, 18 as candidate layers for LLaMA 2 13B Chat model for both DoLa and SLED. Our method still has robust performance across different datasets and metrics.
17

---

Q:
Can you buy a fair trade laptop?
A laptop is a portable computer. A portable computer is a handheld computer. A handheld computer is palmtop computer. A palmtop computer is PDA. A PDA is personal digital assistant. A personal digital assistant a PDA. A PDA is a handheld computer. A handheld computer is palmtop computer. A palmtop computer is PDA. A PDA is a personal digital assistant. A personal digital assistant is a PDA. Thus, a PDA is PDA. A PDA is PDA. A PDA is PDA. A PDA is PDA. A PDA is PDA. A PDA is a PDA
DoLa
Fair Trade is a certification for coffee. Laptops are not coffee. Thus, you cannot buy a fair trade laptop. So the answer no.
SLED
Q: Can you see the moon in Wembley Arena?
Wembley Arena is in London, England. The moon is a celestial body. and the moon is in the sky. The moon is in the sky, and Wembley Arena is in London, England. London, England is Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon in the sky. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena
DoLa
Wembley Arena is in London. London in the Northern Hemisphere, and the moon is in the Southern Hemisphere. Thus, you cannot see the moon in Wembley Arena. So the answer no.
SLED
Does Thiago Moises May 13 2020 submission move hypothetically hurt Achilles?
Thiago Moises is MMA fighter. He 1.73 m tall. and Achilles is about 00000000000000000000000000000000
DoLa
Thiago Moises's submission was rear naked choke, which is a blood choke. A blood choke can cause a fighter pass out. Thus, Thiago Moises's 's submission could hypothetically hurt Achilles. So the answer is yes.
SLED
Table We present three examples of the generated text from LLaMA-2-13B- on Strategy dataset. SLED method can mitigate the repetition issue
E Implementation Details
We conducted all experiments utilizing NVIDIA A100 GPUs and implemented our method based on the following repositories: DoLa', AD2, and ICD'. For decoding responses from the LLMs on TruthfulQA, StrategyQA, and GSM8K, we employed greedy decoding. The models were operated with 16-bit floating-point precision and a batch size of 1. For the LLaMA models sized 7B, 13B, and 70B, utilized I, 1, and 3 GPUs respectively. Cross-GPU inference, involving model weight sharding, was facilitated by the Hugging Face Accelerate package".
18

---

Table 8: Performance comparison on HotPotQA, Natural Question (NQ) and TriviaQA.
HotpotQA EM
NQ
TriviaQA EM
Model
F1
EM
F1
F1
LLaMA 2 7B Chat
19.6
20.1
21.8
20.4
44.4
44.3
DoLa
20.4
21.3
23.5
21.5
45.2
45.3
SLED (ours)
20.9
21.5
24.4
22.2
47.6
46.3
Llama 13B Chat
23.8
21.7
33.1
28.9
63.0
60.9
DoLa
24.5
23.2
33.1
28.9
63.2
61.5
SLED (ours)
25.0
24.5
34.6
31.6
63.3
62.2
Regarding the details in Section 3.4, we evaluate the 7B-chat model for ITT, as the checkpoint is publicly available. Combining ITI with SLED results in better performance compared to using ITI alone. AD employs an entropy-based metric to measure the 'sharpness' of in-context hidden states and incorporates it into the decoding process. Combining AD with SLED surpasses both the original AD and its combination with DoLa across four model types. For CD, we have conducted experiments in two distinct configurations: (i) the LLaMA 2 13B base model is contrasted with that of the Llama 7B base model, and (ii) the LLaMA 2 13B chat model and the LLaMA 7B chat model are compared. Applying SLED to the larger models (13B) boosts performance beyond vanilla CD. ICD contrasts a trustworthy 7B model with fine-tuned, untrustworthy 7B model, and again, applying SLED on the trustworthy 7B model improves factual accuracy further.
Additional Results of DoLa
Table 9 presents some additional results of DoLa across various benchmarks. Specifically, DoLa in Table 9 selects a subset of early layers candidates for calculating the jensen-Shanned Diver- gence (JSD) instead of using all layers. For example, for the LLaMA 7B Chat model, layers [0, 2, 4, 6, 8, 10, 12, 14) are designated candidate layers. Notably. a specific trick implemented in DoLa omitting the post-softmax step on logits for the TruthfulQA multiple-choice task to enhance accuracy. This trick is not applied to the vanilla greedy decoding in Table 9. In contrast, for the results presented in our Tables 1, 2, and 3, this technique is also been applied to vanilla greedy decoding to ensure fair comparison.
Table 9: The Performance of DoLa Across Various Benchmarks
Model
TruthfulQA
GSMSK
StrQA
MC1
MC2
MC3
LLaMA-2-7B-Base
28.40
43.39
20.52
14.03
60.96
+ DoLa
31.21
62.12
29.73
14.63
60.74
LLaMA-2-13B--asse
29.01
44.27
20.71
28.66
66.07
DoLa
29.38
63.95
33.63
28.81
66.59
LLaMA-2-70B-Basse
37.70
53.60
27.36
56.33
75.20
+ DoLa
27.05
60.26
31.64
56.94
74.93
LLaMA-2-7B-Chat
33.66
51.29
24.91
21.08
63.67
+ DoLa
33.29
60.86
29.77
20.55
64.37
LLaMA-2-13B-CHE
35.37
53.31
26.71
36.47
69.87
DoLa
31.95
62.44
31.23
35.79
69.48
LLaMA-2-70B-Chal
37.33
56.33
27.94
54.59
77.25
+ DoLa
31.33
54.48
34.43
54.44
76.86
'Theee results provided by Yung-Sung Chuang.
19

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
This paper introduces Self Logits Evolution Decoding and shows that improves the factual accuracy benchmark datasets. This is the main contribution and is reflected in the abstract and introduction.
Justification:
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in Section A.
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper introduces a new decoding algorithm for large language models, and validates its performance empirically on benchmark datasets.
Guidelines:
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The experiment settings described in Section and the parameters listed in Section 3.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use open source LLMs for all experiments, as well as baselines that have publicly available implementations. We do not use any confidential data or libraries for our experiments.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiment settings are described in Section 3.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We also follow the settings of existing work in this area for our experiments for more consistent comparison.
8. Experiments Compute Resources
20

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We use publicly available models and datasets, and hence, the inference time is dominated by their computational costs and implementations, which are well documented.
Guidelines:
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://eerrippppeee httpr/ppp.p.pppl.ccccccceeeeeeee
Answer: [Yes]
Justification: The experiments in this paper are conducted on public benchmark datasets. The algorithm proposed in this paper does pose new safety, security, other societal risks.
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper is about the factuality of LLMs and the experiments show that clearly these LLMs are not ready for critical applications requiring high accuracy answers.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The decoding algorithm introduced this paper does not pose new risks as long as it is used on reasonably safeguarded model.
Licenses for existing assets
Question: Are the creators original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all papers and creators used our studies.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We do not use crowdsourcing.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
21

---

Answer: [NA]
Justification: This paper does not involve crowdsourcing, nor research with human subjects.
22


Reviews:
[{"review_id": "sht2Zl5MGl", "reviewer": "Reviewer_Mgbd", "summary": "It introduces a novel decoding strategy named Self-Evolution Decoding (SED) aimed at enhancing the reliability and truthfulness of Large Language Models (LLMs). Unlike methods that depend on external knowledge bases or additional fine-tuning, SED is an intrinsic optimization technique that capitalizes on the self-evolution of LLMs' hidden states. The method refines the output during inference, akin to continued training, which improves accuracy and interpretability without sacrificing natural language fluency.", "questions": "Please see weaknesses.", "limitations": "While SED has been tested on multiple datasets, there may be a need for further evaluation on an even broader range of datasets to ensure the method's generalizability.\n\nThe paper does not provide a rigorous optimization analysis of SED, which could help understand why it leads to more factual outputs.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The SED method is an original contribution that tackles the issue of factuality in LLMs by introducing a new decoding approach. This is a novel way to enhance outputs without relying on external data or model retraining.\n\nThe concept of optimizing an implicit objective function using the self-evolution of LLMs is creative and presents a new angle for improving model outputs during inference.", "weaknesses": "While the empirical results are positive, there is no enough theoretical analysis provided to support the method's effectiveness.\n\nThe paper does not report error bars or measures of statistical significance, which are important for understanding the variability of the results.\n\nSED may introduce additional computational overhead during inference, which could be a concern for real-time applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer,**\n\nThank you so much for taking the time to provide your feedback. Your comments and suggestions are invaluable to us, especially regarding our methodologies. We appreciate this opportunity to address your concerns.\n\n> **\"SED may introduce additional computational overhead during inference, which could be a concern for real-time applications.\"**\n\nThank you for raising this concern. We have benchmarked our method against standard decoding and Dola by measure the latency (ms/token) across different configurations on different size of models. 0ur findings indicate that our approach increases computational time by less than 10%. \n\n| Model | DoLa   | SED (topk=5) | SED (topk=20) | SED (topk=50) |\n|----------------|--------|--------------|---------------|---------------|\n| LLaMA-2-7B     | 29.93  | 30.41        | 31.15         | 32.70         |\n| LLaMA-2-13B    | 39.57  | 39.61        | 41.14         | 43.30         |\n| LLaMA-2-70B    | 136.42 | 138.33       | 140.24        | 143.12        |\n\nThis minimal increase is due to:\n\n- **Optimized Operations:** Most operations, including the calculation of cosine similarity, are accelerated using PyTorch, which is crucial for maintaining low computational overhead.\n- **Vectorized Operations:** By extensively utilizing vector operations, we reduce reliance on for-loops, thereby enhancing computation speed.\n\nAlthough there is an increase in computational cost with a larger top-k, our parameter analysis in Figure 5 shows that a large top-k, such as 50, is not necessary. Thus our approach maintains a acceptable computational overhead, making it feasible for real-time applications.\n\n> **\"While the empirical results are positive, there is not enough theoretical analysis provided to support the method's effectiveness.\"**\n\nWe are grateful for your advice and plan to enhance our analysis to better understand why SED improves the factual accuracy of a Large Language Model (LLM). The principal insight is that pre-trained LLMs exhibit variations in token distributions across different layers, particularly when comparing the output layer with the earlier layers. We have discovered that contrasting the early layers with the final layer can yield a more factual distribution over specific tokens.\n\nWe provide a demonstration in the attached PDF to further reveal how the SED mechanism benefits from this approach.\n\n1. By contrasting the final layer with each of the early layers, we estimate an inner distribution:\n   $$\n   \\bar{t}^{(n)} = \\frac{1}{w^{(n)}} (\\bar{t}^{(n)}_1, \\bar{t}^{(n)}_2, ..., \\bar{t}^{(n)}_i, ..., \\bar{t}^{(n)}_d)\n   $$\n   This is more precise and can be demonstrated by comparing Figure 1(a) and Figure 2(a) in the attached PDF. Our analysis in Section 2.4, Question 1, also delves deeper into this aspect. For most early layers, the estimated inner distribution tends to assign a higher probability to the correct tokens. However, DoLa's estimates are imprecise, leading to a heavy reliance on the selection of candidate layers. As shown in Figures 2(a) and 2(b), DoLa's choice to contrast the final layer with the zeroth layer results in both incorrect and correct tokens having the same probability.\n\n2. For different layer estimates, when we ensemble different layers, we do not simply average them. Instead, our SED method determines weights by calculating the cosine similarity, identifying layers with imprecise estimates. Thus, their influence is diminished, as illustrated in Figures 1(a) and 1(b), where the weights are reduced for layers that misestimate the inner distribution.\n\n3. Our approach integrates the inner distribution with the original distribution, as opposed to DoLa's method, which simply replaces the original distribution with the inner one. This integration is discussed in Section 2.4, Question 2.\n\n> **\"While SED has been tested on multiple datasets, there may be a need for further evaluation on an even broader range of datasets to ensure the method's generalizability.\"**\n\nThank you for this valuable suggestion. We have conducted additional experiments on more realistic open-ended generations datasets, HotPotQA, Natural Question (NQ), TriviaQA (Tri). We adopt more evaluation metric, Exact Match(EM) and the F1.\n\n| Model           | HotpotQA EM | HotpotQA F1 | NQ EM | NQ F1 | Trivia EM | Trivia F1 |\n|-----------------|:-----------:|:-----------:|:-----:|:-----:|:---------:|:---------:|\n| Llama 2 7B chat | 19.6  | 20.1  | 21.8  | 20.4  | 44.4   | 44.3   |\n| + DoLa          | 20.4    | 21.3  | 23.5  | 21.5  | 45.2  | 45.3  |\n| + Sed(ours)     | **20.9**    | **21.5**    | **24.4** | **22.2** | **47.6**  | **46.3** |\n| Llama 2 13B chat| 23.8        | 21.7    | 33.1  | 28.9  | 63.0      | 60.9      |\n| + DoLa          | 24.5    | 23.2   | 33.1  | 28.9  | 63.2      | 61.5   |\n| + Sed(ours)     | **25.0**    | **24.5**    | **34.6** | **31.6** | **63.3**  | **62.2** |\n\n). The results show that our method improves the performance in more realistic open-ended generation tasks.\n\n\n\nSincerely,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wRXsKl5rZl", "reviewer": "Reviewer_12rF", "summary": "This paper introduces Self-Evolution Decoding (SED), a novel decoding strategy aimed at enhancing the factual accuracy of large language models (LLMs) without the need for external knowledge bases or additional fine-tuning. SED optimizes the outputs of LLMs by refining the logits from the final layer through the inherent self-evolution of hidden states, effectively reducing hallucinations and refocusing the probability mass on factual responses. Evaluations on several benchmarks, including TruthfulQA and FACTOR, demonstrate that SED outperforms existing methods like DoLa, achieving up to a 10% improvement in factual accuracy. The method is also compatible with other factuality-enhancing techniques, further boosting their effectiveness. While the empirical results are promising, the paper notes some computational overhead and calls for further theoretical analysis to better understand the mechanisms behind SED's success.", "questions": "Could you provide more insights or theoretical justifications for the functioning and effectiveness of SED? Are there any specific properties of the inner knowledge distribution (P_{inner}) that you believe contribute to the success of SED?", "limitations": "Computationally inefficiency is the biggest limitation of the proposed method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- __Novelty__: The paper presents a novel decoding strategy, SED, which improves the factual accuracy of large language models without requiring additional fine-tuning or external knowledge bases. This approach fills a crucial gap in the existing methodologies for improving the reliability and truthfulness of LLM outputs.\n\n- __Comprehensive evaluation__: The effectiveness of SED is validated across multiple benchmarks such as TruthfulQA, FACTOR, StrategyQA, and GSM8K. The results show that SED outperforms existing methods like DoLa and other baseline strategies, demonstrating significant improvements in factual accuracy and overall performance.\n\n\n- __Compatibility__:  SED is its compatibility with other factuality-enhancing methods. The paper demonstrates how SED can be integrated with methods like Inference-Time Intervention, Activation Decoding.", "weaknesses": "- __Lack of theoretical analysis__: While the empirical results are robust, the paper lacks a rigorous theoretical analysis to explain why SED improves the factual accuracy of LLMs. A better understanding of the underlying mechanics and theoretical justification for the approach would strengthen the contribution.\nExperimental Reproducibility and Statistical Significance:\n\n\n\n- __Computation efficiency__: Although SED improves factual accuracy, this comes at the cost of increased computational complexity compared to methods like DoLa. The paper mentions that SED operates slightly slower, which could be a drawback for applications requiring real-time performance. Further benchmarking on computational costs and scalability would be useful for assessing the practical applicability of SED.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "28smDOeicT", "reviewer": "Reviewer_Xg4n", "summary": "In this work, the authors present a decoding method called Self-Evolution Decoding (SED) to enhance factuality. During the decoding process, SED first estimates the “inner knowledge distribution,” representing the knowledge the model “knows,” by analyzing the difference between the top-layer logits and intermediate layers. This estimated inner knowledge distribution is then used to adjust the LLM output logits, steering the model’s behavior towards greater factuality. Experiments across several datasets and LLMs demonstrate that SED significantly improves the LLM’s factuality.", "questions": "1. The notion of logits \\mathcal{L} is confusing, which is normally used to represent loss function.\n\n2. Notation is not consistent. In line 69, P refers to the probability distribution, where in line 71 P_L denotes the logits distribution. Is this a typo?\n\n3. Equation 2 is confusing, I initially thought it represented L^n - L^N introduced in Line 90, but there is no description of this equation. I spent quite a while understanding that it is just a derivation of the gradient of the KL function and is not related to line 90.\n\n4. In Line 107, \"In this context\" and \"In this formulation\" duplicate.", "limitations": "The authors acknowledge that the approximated P_inner is not perfect, but they do not provide sufficient experimental results to support this claim. Including more detailed experiments that explore the accuracy and limitations of this approximation would strengthen their argument.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality: The approach of narrowing the gap between model generation and internal knowledge is innovative.\n\nSignificance: Enhancing the factuality of LLMs is a crucial research problem, and the positive results demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The motivation for this method is somewhat problematic. In section 2.2, the authors claim that the initial layers predominantly encode “lower-level” information while the later layers capture more “semantic” information. However, this does not imply that their difference is a good approximation of the gradient of KL divergence. If you believe this approximation is accurate, why not directly apply this approximated gradient to Equation 3?\n\n2. The experiments primarily focus on task performance but lack an investigation into how this method works. For example, in lines 145-147, the authors claim that the approximated P_inner cannot be directly used as it is not perfect. I believe presenting experimental results would be more convincing than a verbal analysis.\n\n3. The paper is not well-structured, and the concepts are confusing. Please refer to the questions below for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you so much for taking the time to provide your feedback. Your comments and suggestions are invaluable to us, especially regarding our methodologies and presentations. We appreciate this opportunity to address your concerns.\n\n> **\"However, this does not imply that their difference is a good approximation of the gradient of KL divergence. If you believe this approximation is accurate, why not directly apply this approximated gradient to Equation 3?\"**\n\n**First Issue: Suitability of the Approximation**\n\nWe are grateful for your insightful comments and are eager to further explain this aspect. Regarding why we interpret the difference $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$ as a suitable approximation for the gradient of the KL divergence, the core reason lies in our claim that the KL divergence from the real distribution to $\\mathcal{P}_{\\mathcal{L}}^{(N)}$  is smaller than that to $\\mathcal{P}_{\\mathcal{L}}^{(n)}$. Thus, $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$ serves as an approximate gradient.\n\nWe can verify this claim directly because:\n$$\nKL(P_{real} \\parallel P_{L}) = CE(P_{real}, P_{L}) + H(P_{real})\n$$\n\nHere, CE represents cross-entropy and H denotes entropy. By comparing the cross-entropy across different layers, as illustrated in Figure 3 (which was unfortunately not referenced in our original manuscript), we find that the final layer exhibits a smaller loss value compared to earlier layers. This observation makes sense because, as discussed, the final layer directly engages with real-world labels through cross-entropy during training, making it more accurate. Additionally, as you mentioned, the final layer contains more “semantic” information, making it closer to the real-world labels.\n\nThus, $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$ is a good approximation of the gradient of $KL(P_{real} \\parallel P_{L})$. When estimating the gradient of $KL(P_{inner} \\parallel P_{L})$, as per Equation 3, this gradient should be close to the $KL(P_{real} \\parallel P_{L})$ to benefit the decoding. Hence, we utilize $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$ as the source for appromximation.\n\n**Second Issue: Application to Equation 3**\n\nRegarding why we do not directly apply $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$ in Equation 3, it is important to consider that while $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$ is unconstrained, the gradients estimated in Equation 2 (e.g., $p_1 - t_1, p_2 - t_2, ..., p_i - t_i, ..., p_d - t_d)$ are constrained within 1. Thus, direct substitution could lead to a mismatch in magnitudes. Proper normalization and subsequent aggregation of estimations from different layers are precisely what our method addresses in Section 2.3. Our approach does not naively scale to normalize; it provides a more interpretable and computationally efficient method, aligning the directions of the gradients rather than their magnitudes to address their different scales.\n\nTo further address your concerns, we include a new ablation study by directly scaling the $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$, and simply averaging these scaled differences across different layers, a method we denote as \"ablation1\" in the following table\n\n> **\"The authors claim that the approximated P_inner cannot be directly used as it is not perfect. I believe presenting experimental results would be more convincing than a verbal analysis.**\n\nThank you so much for your suggestion. We include the corresponding ablation study following your suggestion and denote it as \"ablation2\". In this study, we directly use $P_{inner}$\n\n|                             | Factor    | TrufulQA(MC1) | TrufulQA(MC2) | TrufulQA(MC3) |\n|-----------------------------|-----------|---------------|---------------|---------------|\n| llama2-7B-chat + ablation1  | 62.73     | 33.66         | 39.83         | 31.47         |\n| + ablation2                 | 63.59     | 25.21         | 51.09         | 26.25         |\n| + SED                       | **65.16** | **37.08**     | **63.86**     | **32.90**     |\n| llama2-13B-chat + ablation1 | 66.29     | 37.33         | 45.0          | 31.98         |\n| + ablation2                 | 66.70     | 27.05         | 52.72         | 28.46         |\n|  + SED                      | **67.06** | **37.09**     | **63.75**     | **32.60**     |\n\n\n> **\"The notion of logits \\(\\mathcal{L}\\) \"**\n\nWe appreciate this suggestion. To avoid confusion, we will replace it in our revision. \n\n> **\"In line 69, Is this a typo?\"**\n\nWe apologize for this inconsistency. Yes, it was a typo. In our revision, we will clarify that \\(P_L\\) in line 71 indeed refers to the probability distribution derived from the logits, maintaining consistency throughout the document.\n\n> **\"Equation 2 is confusing\"**\n\nWe apologize for any confusion caused by Equation 2. We have removed the subscript \\(L\\) from \\(\\mathcal{P}_{\\mathcal{L}}\\) to clarify that it represents a general probability distribution, not specifically linked to 'logits' as previously implied. Additional explanations will be added to elucidate that this equation is a derivative of the gradient of the KL divergence function and is unrelated to the discussions around line 90.\n\n> **\"In Line 107, 'In this context' and 'In this formulation' duplicate.\"**\n\nThank you for pointing out the redundancy. We will revise this part to enhance clarity and avoid duplication.\n\nLastly, we would like to express our heartfelt gratitude for the time and effort you have dedicated to reviewing our paper. We deeply appreciate your guidance and assure that all new results and findings will be included during our revision. Thank you!\n\nSincerely,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uE7QZYqBi8", "reviewer": "Reviewer_1uAz", "summary": "This work proposes self-evolution decoding, a method to improve LM's factuality without using external knowledge or fine-tuning data. Specifically, the differences between each layer’s logits and the final layer’s logits are utilized to approximate the gradient, which are further used to estimate the inner knowledge of LMs. Finally, the estimated \"inner\" distribution is utilized to adjust the model's final outputs. Experiments on various tasks show the effectiveness of the proposed method.", "questions": "- I'm wondering if there can be good methods to evaluate LM's factuality in more realistic open-ended generation tasks, which are the main scenarios where we hope to improve LM's factuality and consistency.", "limitations": "Limitations have been discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally easy to understand.\n- The proposed method is shown to provide good results on a variety of tasks.", "weaknesses": "- I think some of the approximations might be questinable. First, in Section 2.2, I'm not sure whether it is reasonable to use the logit differences to estimate the gradient: logits are unconstrained, while the scale of the gradients are constrained within 1. Moreover, in Section 2.3, I'm not sure why the estimations are broken-down for each item in the vocabulary and why the weights for different layers are directly aggregated and normalized with cosine similarties. More ablation studies should be provided to verify some of these choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer\n\nThank you so much for taking the time to provide your feedback. Your comments and suggestions are invaluable to us. We appreciate the opportunity to address your concerns regarding the approximation used in our approach, and we are also including additional results to support our methodology.\n\n> **\"Q: logits are unconstrained, while the scale of the gradients are constrained within 1\"**\n\nThank you for your insightful comment regarding the unconstrained nature of the logits $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$ and the constrained nature of the gradients $(p_1 - t_1, p_2 - t_2, ..., p_i - t_i, ..., p_d - t_d)$. We appreciate the opportunity to address your concern. \n\nIn fact, this is an important consideration in our methodology. Our approach does not rely on the magnitudes of $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$; rather, it utilizes this difference to approximate the direction of the gradient vector.  Specifically, we do not employ the expression: \n$$\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)} \\approx (p_1 - t_1, p_2 - t_2, ..., p_i - t_i, ..., p_d - t_d)$$\nto directly estimate $(t_1, t_2, ..., t_i, ..., t_d)$.\n\nInstead, our method involves increasing the cosine similarity between $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$ and the vector $(p_1 - t_1, p_2 - t_2, ..., p_i - t_i, ..., p_d - t_d)$:\n$$CosineSimilarity[\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}, (p_1 - t_1, p_2 - t_2, ..., p_i - t_i, ..., p_d - t_d)]$$\nto estimate $(t_1, t_2, ..., t_i, ..., t_d)$.\n\nThis approach aligns the directions of the gradients rather than their magnitudes, thereby addressing the issue of their different scales.\n\n\n> **\"Q: why the estimations are broken-down for each item in the vocabulary\"**\n\nThank you for your inquiry. We are happy to clarify your concerns. Given that the vocabulary of a typical large language model (LLM) often exceeds 10,000 items (d > 10k), the  $(t_1, t_2, ..., t_i, ..., t_d)$ we need to estimate constitute a high-dimensional vector. Estimating this vector in its entirety for each vocabulary item simultaneously would involve significant computational overhead.\n\nBy breaking down the estimation process, we can focus only on the possible words in the vocabulary (that have the top-k highest probability in the original output) and then estimate their corresponding $t_i$ one by one. This selective approach reduces computational costs, as well as avoids the noise introduced by less significant words.\n\nTherefore, this breakdown is strategically designed to minimize computational load. For the sake of conciseness of the presentation, we move the discussion of computational considerations to Section 2.4 and still use d instead of k here.\n\n>**”Q: why aggregated and normalized”**\n\n**Normalization** is applied individually to each layer $n$ as described in line 115. This is necessary because the vector components:\n$(\\bar{t}^{(n)}_1, \\bar{t}^{(n)}_2, ..., \\bar{t}^{(n)}_i, ..., \\bar{t}^{(n)}_d)$ do not inherently sum to one. Therefore, we should normalize these components to achieve an estimation, $\\bar{t}^{(n)}$, of the inner distribution for each layer n. \n\n**Aggregations** are conducted across the $\\{\\bar{t}^{(n)}\\}$ for all the layers.\n\nContrary to direct aggregation, after obtaining a normalized $\\bar{t}^{(n)}$ for each layer, the aggregation process is not straightforward. The weights $w^{(n)} = \\sum_i^d (\\bar{t}^{(n)}_i)$ in line 116 indicate the degree to whichthe logits $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$ and the normalized vectors $(p_1 - \\bar{t}^{(n)}_1, p_2 - \\bar{t}^{(n)}_2, ..., p_i - \\bar{t}^{(n)}_i, ..., p_d - \\bar{t}^{(n)}_d)$ are well-aligned in terms of cosine similarity. A larger weight suggests that the estimations from $\\bar{t}^{(n)}$ are more reliable, and thus, we assign greater weight in the aggregation process to such layers. Conversely, layers with smaller weights are assigned lesser importance in the aggregation.\n\nWe include a new ablation study to support our proposed methods following your suggestion. In one such study, we deviated from our established processes by crudely scaling the $\\mathcal{L}^{(n)} - \\mathcal{L}^{(N)}$, and simply averaging these scaled differences across different layers, a method we denote as \"ablation1.\" It bypasses the steps of breakdown, normalization, and aggregation that we have discussed above. The results show that our method achieves better result.\n\n|                             | Factor    | TrufulQA(MC1) | TrufulQA(MC2) | TrufulQA(MC3) |\n|-----------------------------|-----------|---------------|---------------|---------------|\n| llama2-7B-chat + ablation1  | 62.73     | 33.66         | 39.83         | 31.47   |\n| + SED        | **65.16** | **37.08**     | **63.86**     | **32.90**     |\n| llama2-13B-chat + ablation1 | 66.29     | 37.33         | 45.0          | 31.98     |\n| + SED       | **67.06** | **37.09**     | **63.75**     | **32.60**     |\n\n>**\"Q: more realistic open-ended generation tasks\"**\n\nThank you for this valuable suggestion. We have conducted additional experiments on more realistic open-ended generations datasets, HotPotQA, Natural Question (NQ), TriviaQA (Tri). We adopt more evaluation metric, Exact Match(EM) and the F1.\n\n| Model           | HotpotQA EM | HotpotQA F1 | NQ EM | NQ F1 | Trivia EM | Trivia F1 |\n|-----------------|:-----------:|:-----------:|:-----:|:-----:|:---------:|:---------:|\n| Llama 2 7B chat | 19.6  | 20.1  | 21.8  | 20.4  | 44.4   | 44.3   |\n| + DoLa          | 20.4    | 21.3  | 23.5  | 21.5  | 45.2  | 45.3  |\n| + Sed(ours)     | **20.9**    | **21.5**    | **24.4** | **22.2** | **47.6**  | **46.3** |\n| Llama 2 13B chat| 23.8        | 21.7    | 33.1  | 28.9  | 63.0      | 60.9      |\n| + DoLa          | 24.5    | 23.2   | 33.1  | 28.9  | 63.2      | 61.5   |\n| + Sed(ours)     | **25.0**    | **24.5**    | **34.6** | **31.6** | **63.3**  | **62.2** |\n\n). The results show that our method improves the performance in more realistic open-ended generations tasks.\n\nSincerely,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We express our gratitude towards all reviewers for their time reviewing our submission and providing constructive feedbacks. Along with our rebuttal, we are including a PDF file containing figures that further support our analysis of why the SED method is effective. This additional material aims to provide a deeper understanding of the mechanisms underlying SED's efficacy.

Sincerely,
Authors

Meta-review: SUMMARY

The paper introduces a new decoding method for generating text from LLMs. The main idea is to modify the final output logits which are used for generation by adding in an additional term that seeks to optimize early-exit logits .

REASONS TO ACCEPT

The reviewers agree that the paper provides an original contribution, and that the new decoding method is particularly interesting because it doesn't require a separate knowledge base or additional finetuning to improve an LM's access of its internal knowledge at generating-time. The reviewers also note the impact of the problem being tackled (enhancing factuality in LM generations) and that the proposed method can be used in conjunction with other factuality methods, making it even more useful.

REASONS TO REJECT

Two of the reviewers point out that the proposed method ought to be backed by theoretical analysis of why it works. One reviewer requested further benchmarking on computational costs and scalability of the method. Multiple reviewers mentioned not understanding aspects of the method. They show their method improves answering ability on TruthfulQA and CoT.

CONCLUSION

As a note to the authors, the need for significant modifications is a valid reason to reject a paper, and it is well within Xg4n's purview as a reviewer to suggest that a submission that needs significant modifications to pass the bar for publication at NeurIPS ought to undergo an additional round of reviews. However, in this case, the changes made by the authors are sufficiently close to the original paper that I believe this should not be grounds for rejection. If the modifications and clarifications requested by reviewers are incorporated into the paper, I think it warrants acceptance to NeurIPS.
Decision: Accept (poster)