Title: SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Models
Abstract: Large language models (LLMs) have demonstrated remarkable capabilities, but their outputs can sometimes be unreliable or factually incorrect. To address this, we introduce Self Logits Evolution Decoding (SLED), a novel decoding framework that enhances the truthfulness of LLMs without relying on external knowledge bases or requiring further fine-tuning. From an optimization perspective, our SLED framework leverages the latent knowledge embedded within the LLM by contrasting the output logits from the final layer with those from early layers. It then utilizes an approximate gradient approach to enable latent knowledge to guide the self-refinement of outputs, thereby effectively improving factual accuracy. Extensive experiments have been conducted on established benchmarks across a diverse range of model families (LLaMA 2, LLaMA 3, Gemma) and scales (from 2B to 70B), including more advanced architectural configurations such as the mixture of experts (MoE). Our evaluation spans a wide variety of tasks, including multi-choice, open-generation, and adaptations to chain-of-thought reasoning tasks. The results demonstrate that SLED consistently improves factual accuracy by up to 20\% compared to existing decoding methods while maintaining natural language fluency and negligible latency overhead. Furthermore, it can be flexibly combined with other decoding methods to further enhance their performance.

SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, but their outputs can sometimes be unreliable or factually incorrect. To address this, we introduce S elf L ogits E volution D ecoding (SLED), a novel decoding framework that enhances the truthfulness of LLMs without relying on external knowledge bases or requiring further fine-tuning. From an optimization perspective, our SLED framework leverages the latent knowledge embedded within the LLM by contrasting the output logits from the final layer with those from early layers. It then utilizes an approximate gradient approach to enable latent knowledge to guide the self-refinement of outputs, thereby effectively improving factual accuracy. Extensive experiments have been conducted on established benchmarks across a diverse range of model families (LLaMA 2, LLaMA 3, Gemma) and scales (from 2B to 70B), including more advanced architectural configurations such as the mixture of experts (MoE). Our evaluation spans a wide variety of tasks, including multi-choice, open-generation, and adaptations to chain-of-thought reasoning tasks. The results demonstrate that SLED consistently improves factual accuracy by up to 20% compared to existing decoding methods while maintaining natural language fluency and negligible latency overhead. Furthermore, it can be flexibly combined with other decoding methods to further enhance their performance.

1 Introduction

Large Language Models (LLMs) have achieved remarkable breakthroughs in recent years, demonstrating exceptional performance across various domains [1, 2, 35, 36, 44, 47, 48]. However, a significant challenge associated with LLMs is their tendency to hallucinate or distort the truth, resulting in outputs that are not factual [15, 17, 65]. This issue of hallucination undermines the reliability and trustworthiness of LLMs in practical applications. A popular strategy for improving the LLM factuality involves refining the decoding process [43, 53].

Decoding focuses on how the model selects the next token during the generation process, which can significantly influence the factual accuracy of the output. The decoding methods can be cost-effective since (a) they do not rely on external knowledge and (b) no additional training is required. Furthermore, decoding methods can be synergistically combined with other techniques aimed at improving the LLM factuality, such as retrieving information from external knowledge bases [24, 25], various fine-tuning strategies for better alignment [46, 48], or ensemble learning methods [10].

Real-world

Factuality

Distribution

LLM

Output

Distribution

LLM

Latent

Knowledge

What the LLM knows

What the LLM tells

Inference

The capital of British Columbia province is

8th layer

16th layer

24th layer

N

-th layer (final layer)

Vancouver

Victoria

Richmond

…

…

…

Early-exit logits:

Early-exit logits:

softmax

softmax

Early-exit logits:

Latent knowledge

𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

Contrast logits to

uncover the latent

knowledge

False

True

…

…

Utilize

𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

to estimate

the real-world distribution

and steer the self-evolution

of

𝑵

𝟐𝟒

𝟏𝟔

𝟖

Original logits:

𝑵

Self-evolved logits:

*

𝑵

= 𝒍𝒐𝒈𝒊𝒕𝒔

𝑵

-𝜶⋅𝜵𝑲𝑳 𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

, 𝓟

𝑵

LLM

…

Vancouver

Victoria

Richmond

Recent studies [22, 26, 42, 50] suggest that LLMs sometimes have learned the factual content based on extensive pretraining or fine-tuning, although they fail to produce the correct answer when a user queries the model. This has inspired the development of several factuality decoding methods [7, 26, 27, 64] to reveal what the model implicitly "knows." Figure 1 summarizes the underlying mechanism of these factuality decoding methods. The LLMs' output distribution is derived by applying the softmax function to the output logits from the final layer. During the training phase, this distribution is optimized based on the real-world factuality distribution represented by the training dataset. However, during the inference phase, "what the LLM tells" might still contain factual errors, which implies a discrepancy between the output distribution and the real-world factuality distribution. While the real-world distribution remains inaccessible during the inference phase, the model's latent knowledge ("what the model knows") may have implicitly learned some factual content correctly during the training phase [22, 50]. Therefore, a key challenge for factuality decoding strategies lies in effectively harnessing the latent knowledge embedded within LLMs to refine the output distribution (logits) during inference.

To address this challenge, we propose S elf L ogits E volution D ecoding (SLED), a novel factuality decoding approach that leverages the latent knowledge within LLMs by contrasting the final layer's logits with early layers' logits. During the decoding process, as LLMs progress from early to final layers, they progressively incorporate factual information stored in each layer into the output. SLED tracks this evolution process to unearth the latent knowledge within LLMs, and enables the 'selfevolution' of the output distribution further to align it more closely with real-world facts. Furthermore, our approach recognizes that the latent knowledge within LLMs, while valuable, may not always be perfect. Therefore, instead of simply replacing the original outputs with this latent knowledge, SLED integrates it into the original logits through an operation similar to 'single-step gradient descent' over the output logits during the inference time. This operation minimizes the Kullback-Leibler (KL) divergence between the latent knowledge distribution and the output distribution, effectively balancing the two and mitigating potential drawbacks such as overfitting or biased outputs. Figure 2 illustrates the SLED workflow, highlighting how SLED optimizes the output logits, leading to a more factual output distribution. We evaluate SLED on various LLMs (e.g., LLaMA 2 [48], LLaMA 3 [1], Gemma [31]) and benchmarks to demonstrate its state-of-the-art performance in layer-wise contrastive decoding methods. In summary, our main contributions are:

We propose SLED, a novel decoding method that aligns LLMs outputs with factual knowledge without requiring an external knowledge base or fine-tuning data.

We conduct extensive experiments across a range of LLMs, with varying configurations and scales. The results demonstrate that SLED consistently improves factual accuracy on various tasks and benchmarks, including multiple-choice, open-ended generation, and chain-of-thought reasoning tasks.

SLED can be flexibly integrated with other factuality decoding methods to enhance their effectiveness further.

Weprovide a new interpretable perspective for understanding layer-wise contrastive decoding methods, paving the way for further developments in factuality decoding.

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

0

2

4

6

8

10

Loss

7B

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

34

36

38

40

0

2

4

6

8

10

Loss

13B

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

75

80

Layer Index

0

2

4

6

8

10

12

Loss

70B

2 Self Logits Evolution Decoding

A large language model, equipped with N layers and a vocabulary V = [ v 1 , v 2 , . . . , v d ] , typically generates text in the next-token prediction fashion. For each given prefix, the model computes the logits at the final ( N -th) layer, logits N ≜ ( ℓ (1 , N ) , ℓ (2 , N ) , . . . , ℓ ( d, N ) ) , which are obtained by applying a linear transformation to the hidden states of the final layer, projecting the high-dimensional hidden state vectors onto the space of the vocabulary size. Subsequently, the output distribution P logits N at the final ( N -th) layer for the next token is derived by applying softmax function on the logits,

where τ is the temperature parameter. Therefore, for each p ( i, N ) (1 ≤ i ≤ d ) , we have

Similarly, we can also derive the logits from early layers by applying the same linear transformation mentioned above to their hidden states. For any early layer n ( n < N ), we denote its logits as logits n ≜ ( ℓ (1 , n ) , . . . , ℓ ( d, n ) ) and the corresponding distribution as P logits n ≜ ( p (1 , n ) , . . . , p ( d, n ) ) .

2.1 Logits Evolution

To improve factual accuracy, it is crucial that the correct token v i receives a higher value of logits N to ensure a higher probability value p ( i, N ) in the output distribution P logits N . From a mathematical perspective, this means aligning the model's output distribution P logits N closely with the real-world factuality distribution P real . Specifically, we can formulate this goal as optimizing the following loss function L regarding the logits :

We describe the above optimization as Logits Evolution . Interestingly, the training of LLMs also aims at minimizing the divergence (typically the KL divergence, as the training loss function is often the cross-entropy loss) between the ground truth P real and the output distribution P logits N . During the training phase, the logits evolution is driven externally by the real-world distribution P real presented in the training dataset, and the corresponding solution is logits = logits N . However, P real is not accessible during the inference phase. To address this challenge, SLED utilizes the model's latent knowledge to estimate P real and enables "self-evolution" of the logits. We denote the estimation as P latent and the self logits evolution can be achieved by the following gradient-descent operation:

The parameter α , termed the Evolution Rate , governs the magnitude of adjustments applied to logits N in the direction of the gradient ∇ logits N KL ( P latent , P logits N ) . In the following Section 2.2 and 2.3, we discuss how we derive the P latent as the estimation of the real-world distribution P real .

2.2 Estimate P real by Tracking the Logits Evolution Direction throughout Layers

The core principle of our method involves leveraging the difference between each early layer's logits and the final layer's logit, logits n -logits N to approximate the gradient of KL ( P real , P logits ) at logits = logits n . Then we estimate P real based on this approximation.

This is inspired by a new perspective of interpreting the training phase of LLMs as the evolution of logits described in Problem 1. As mentioned above, the solution derived by the training phase is the final layer's logits logits = logits N , since the final layer's logits N directly engage with the real-world distribution P real through the loss function in training. This implies that we can generally consider the final logits logits N to be a better solution than the logits from an early layer logits n , with KL ( P real , P logits N ) < KL ( P real , P logits n ) . We present some examples in Figure 3 to demonstrate this. Based on this discussion, if we contrast the final layer's logits with the early layer's logits, we can consider the direction (orientation) of logits n -logits N can approximately align with the direction of the gradient ∇ logits KL ( P real , P logits ) | logits = logits n . To further verify this motivation, we calculate the cosine similarity between logits n -logits N and ∇ logits n KL ( P real , P logits n ) for thousands of tokens across different models in Figure 7. We find that the majority of these values are positive, which means that the directions of these two vectors are close.

Hence, for each early layer n , we propose to maximize the following function of cosine similarity and derive the P ( n ) latent to estimate the P real .

2.3 Achieving the Self Logits Evolution in Three Phases

Based on the above analysis, we can introduce the procedures of SLED: First, we estimate P ( n ) latent for each early layer n using the gradient approximation in Section 2.2. Subsequently, we apply a weighted average on {P ( n ) latent } across all early layers n < N to derive P latent , which serves as the final estimation of the real-world distribution. Finally, we apply P latent in Equation 2 to facilitate the self-evolution of logits N , thereby derive the updated logits, ˜ logits N .

Phase 1: An exhaustive search for an exact solution to the complex optimization problem (Equation 3) is computationally impractical. We can reduce the solution space by the following. Suppose the real-world factuality distribution dictates that the next word to be generated is the i -th token v i from the vocabulary V . Thus P real = P e i , where P e i represents a standard basis vector (one-hot vector) with the i -th component set to 1 and all other components set to 0. Then, we can simplify the aforementioned optimization problem by limiting the solution space to {P e i } d i =0 and decide which token i should be selected. The corresponding gradient when P = P e i has the following formulation.

Proposition 1. The gradient of KL ( P e i , P logits ) at logits = logits n is:

∇ logits n KL ( P e i , P logits n ) = ( P logits n -P e i ) /τ = ( p (1 , n ) , . . . , p ( i, n ) -1 , . . . , p ( d, n ) ) /τ (4) We calculate the cosine similarity between the gradient ∇ logits n KL ( P e i , P logits n ) and the difference logits n -logits N for each token in the vocabulary V . Then we select the P e i ∗ of which the gradient is closest to logits n -logits N as the estimation P ( n ) latent . Mathematically, this involves selecting i ∗ according to the following criterion

and adopting P ( n ) latent = P e i ∗ as the "hard estimation" of P real . Drawing from the concept of hard and soft targets in label smoothing and knowledge distillation, we further extend it to the "soft estimation",

We square { ¯ m ( n ) i } to moderately amplify their differences. Prior studies prove that soft targets usually offer stronger generalization capabilities, more information, and more robustness to noise than hard targets [13, 34, 45, 59, 62]. Hence, we adopt the soft estimation in lieu of the hard estimation.

Eliza's rate per hour for the first 40 hours she

works each week is $10. She also receives an

overtime pay of 1.2 times her regular hourly

rate. If Eliza worked for 45 hours this week,

how much are her earnings for this week?

Eliza's regular hourly rate is $10. For 40 hours,

her earnings are 40 x $10 = $400. For 5 hours of

overtime, her earnings are 5 x $10 = $50. So

her total earnings for the week are $400 + $50

= $450. The answer is $450. (Wrong)

Eliza's regular hourly rate is $10. For 40 hours,

her earnings are 40 x $10 = $400. For 5 hours of

overtime, her earnings are 5 x $10 x 1.2 = $60.

So her total earnings for the week are $400 +

$60 = $460. The answer is $460. (Correct)

𝒕

!𝟐

𝒕

!𝟏

𝒕

𝟎

𝒕

𝟏

𝒕

𝟐

𝒕

𝟑

𝓟

𝟑𝟐

1

0

=

$

5

0

𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

𝟑𝟏

1

0

=

1

2

2

𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

𝟐𝟔

1

0

x

1

.

2

𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

𝟐𝟏

1

0

x

1

.

2

𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

𝟏𝟔

1

0

x

1

.

2

𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

𝟏𝟏

1

0

x

1

.

2

𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

𝟔

1

0

x

1

.

2

𝓟

𝒍𝒂𝒕𝒆𝒏𝒕

𝟏

1

0

x

1

.

2

For 5 hours of overtime, her earnings

are 5 x

𝒕

!𝟐

𝒕

!𝟏

𝒕

𝟎

𝒕

𝟏

𝒕

𝟐

𝒕

𝟑

LLaMA-2-7B-Chat + SLED

LLaMA-2-7B-Chat

Question (The ground truth is $460)

#Layer

𝒔

𝒏

values across layers

Phase 2: We ensemble P ( n ) latent across all layers by computing a weighted average of the set {P ( n ) latent } and adopt it as the final estimation of the P latent :

This estimation suggests that the weight s ( n ) of certain layer n will be larger if the corresponding gradient approximation logits n -logits N is more closely aligned with the gradients {∇ logits n KL ( P e i , P logits n ) } for the tokens in the vocabulary. This in turn amplifies the influence of layer n on the final estimation, which is a desirable effect in our method. Figure 4 demonstrates that SLED can downplay incorrect tokens based on the gradient alignment. One can further validate that for each component m i in the final estimation P latent ≜ ( m 1 , m 2 , . . . , m d ) , the following relationship holds: m i = ∑ N n =0 m ( n ) i / ( ∑ N n =0 ∑ d j =1 m ( n ) j ) . This property simplifies the description in Algorithm 1.

Phase 3: Applying P latent in Equation 2 enables us to derive the gradient necessary for steering the self-evolution on the final layer's logits logits N .

Proposition 2. The gradient of KL ( P latent , P logits ) at logits = logits N is:

Then we can derive the self-evolved logits ˜ logits N

2.4 Computational Complexity and Design Decisions

For each layer, computing CosSim ( logits n -logits N , P logits n - P e i ) for every token v i in the vocabulary V needs O ( d 2 ) operations. To reduce the computational complexity, we select only a subset V I k , where the token v i ∈ V I k has the topk highest logits in the final layer. In this scenario, we only initiate the self-evolution in Equation 2 of the logits corresponding to these topk tokens. For the remaining tokens, which have lower probabilities, their logits are adjusted to a very lower numerical value, e.g. , -1000 . This strategy significantly reduces the computational complexity, while maintaining focus on the most relevant tokens. We name the parameter k , as Evolution Scale , since it determines the number of top-probability tokens active for self-evolution.

Q 2.1: Why SLED contrast the final layer with all the early layers, instead of picking one premature layer to contrast based on JSD?

DoLa selects a subset of early layers to form a candidate set. Then it calculates the Jensen-Shannon Divergence (JSD) between the final layer and each layer in this set. Their strategy is to choose the

Algorithm 1 Self Logits Evolution Decoding

1: Initialization: LLM with N layers, inputs , evolution rate α , evolution scale k > 0 , η ≪ 0 , temperature parameter τ , and the one-hoc vectors {P e i } defined in Section 2.3. 2: Feed the inputs into the LLM to obtain the logits logits n = ( ℓ (1 , n ) , . . . , ℓ ( d, n ) ) and probabilities P logits n = ( p (1 , n ) , . . . , p ( d, n ) ) = softmax ( logits n /τ ) at each layer n , where n ≤ N . 3: Identify the tokens with the topk largest values in logits N and denote their indices by I k . 4: for each early layer n , ( n < N ) do 5: Compute differences for topk logits logits n -logits N . 6: Calculate m ( n ) i = [ max ( CosSim ( logits n -logits N , P logits n -P e i ) , 0 )] 2 , i ∈ I k . 7: end for 8: Compute weighted average m i = ∑ N n =1 m ( n ) i ∑ N n =1 ∑ j ∈ I k m ( n ) j across different layers for each i ∈ I k . 9: for each i from 1 to d do 10: Set ˜ ℓ ( i, N ) = ℓ ( i, N ) -α τ ( p ( i, N ) -m i ) if i ∈ I k else Set ˜ ℓ ( i, N ) = η ≪ 0 . 11: end for 12: Output: The self-evolved logits are ˜ logits N = ( ˜ ℓ (1 , N ) , . . . , ˜ ℓ ( i, N ) , . . . , ˜ ℓ ( d, N ) ) .

layer with the highest JSD as the premature layer, and the chosen layer will be contrasted with the final layer to update probabilities. Obviously, if this strategy is reasonable, a larger candidate set should lead to a better choice of the premature layer and, consequently, enhanced overall performance. However, a paradoxical finding from their experimental results, which our tests also confirm in the discussion in Section 3.5, is that a larger candidate set for DoLa leads to decreased performance. Specifically, when the candidate set for DoLa ranged from 0 to 32 layers for LLaMA-2-7B-Base, the performance was inferior compared to a smaller set of 0 to 16 layers. This fundamental flaw indicates that selecting a good candidate set remains a challenge when applying DoLa. In contrast, our method does not face this concern as it applies an ensemble approach to all early layers. It is also important to note that our method works well even when only contrasting the final layer with part of the early layers, as demonstrated in Section 3.5 and B, proving the robustness of our approach.

Q 2.2: Why not use P latent directly as the model's output distribution?

It is crucial to understand that P latent is merely an estimation of the real-world distribution based on the model's latent knowledge instead of the exact P real . Consequently, relying solely on P latent , similar to DoLa, might lead to inaccuracies, as the latent knowledge can be imperfect. The original logits logits N are still important as they are refined directly by real-world data during training. The evolution rate α in Equation 2, serves to balance this trade-off, enabling a reciprocal enhancement between P latent and the original logits N . More ablation studies are provided in Section 3.5 and B.

Q 2.2: Considering that SLED adopts logits n -logits N as the estimation of the gradient, why not directly apply it in Equation 2?

It is important to note that while logits n -logits N is unconstrained, the gradients estimated in Equation 2 (e.g., p (1 , N ) -m 1 , . . . , p ( d, N ) -m d ) are constrained within [ -1 , 1] . Thus, direct substitution could lead to a mismatch in magnitudes and might also introduce unexpected noise. Proper normalization and subsequent aggregation of estimations from different layers are precisely what our method addresses in Section 2.2 and 2.3. Further analysis is provided in Section B.

3 Experiments

As a novel layer-wise contrastive decoding approach, we first benchmark SLED against the stateof-the-art approach DoLa [7] across a diverse range of model families (LLaMA 2, LLaMA 3, Gemma) and model scales (from 2B to 70B), including the more advanced mixture of experts (MoE) architecture, as detailed in Section 3.2 and 3.3. The results showcase notable factuality improvements across a variety of tasks, including multi-choice, open-generation, and adaptations to chain-of-thought reasoning tasks. Then, in Section 3.4, we integrate our method with other established factuality decoding techniques, illustrating that SLED can further enhance their performance. In Section 3.5, we further conduct in-depth studies on mitigating the repetition issue, layer selection, various parameter settings, and latency overhead to gain more comprehensive insights into SLED's performance. We

also extend our analysis with additional ablation studies and results across more benchmarks in Section B and D in the Appendix, and provide several examples of generated text as the qualitative study in Section C.

3.1 Experimental Setup

Benchmarks We compare our method with baselines on several multiple-choice and open-ended generation tasks. For multiple-choice question tasks, we use the TruthfulQA [29] and FACTOR (Wiki) [33] datasets to assess the LLMs' factuality in short-answer/long-paragraph scenario, respectively. For open-ended generation tasks, we adopt TruthfulQA [29] and tasks involving chain-of-thought reasoning [52]: StrategyQA [12] and GSM8K [8].

Models & Baselines We evaluate the performance of SLED on six LLaMA-2 models [48] ({7B,13B,70B}-Base, {7B,13B,70B}-Chat), four LLaMA-3 family models [1] ({8B,70B}-Base, {8B,70B}-IT), two Gemma models (2B,7B), two MoE models (Mixtral-8 × 7B, Mixtral-8 × 7BIT) [18]. We adopt the following baselines: 1) standard decoding (greedy decoding or sampling depending on the tasks), 2) DoLa [7], 3) Inference Time Intervention (ITI) [26], 4) Activation Decoding (AD) [4], 5) Contrastive Decoding (CD) [27], and 6) Induce-then-Contrast Decoding (ICD) [64].

Metrics We adopt the factual accuracy evaluation implemented in [7] for multiple-choice tasks and chain-of-thought reasoning tasks. For the open-ended generation task on TruthfulQA, we follow the evaluation procedure in [7, 29], using 'finetuned-GPT3-judge's to measure the truthfulness, informativeness, and rejection rate of generated outputs respectively.

3.2 Evaluation on a Broad Range of LLM Benchmarks

Multiple-Choices Tasks The objective of these tasks is to employ decoding methods that enable LLMs to assign higher probabilities to correct completions/answers over incorrect alternatives. We demonstrate the effectiveness of SLED for both Short-Answer Factuality on the TruthfulQA and Long-Paragraph Factuality on the FACTOR dataset. For both DoLa and our SLED, we contrast the results from the final layer against all preceding layers. We randomly sample approximately 5% of the data for validation regarding parameter selection. The results, as shown in Table 1, indicate that SLED achieves superior outcomes in almost all metrics across six LLaMA-2 models. Notably, SLED

achieves better performance under the MC1/MC3 metrics on TruthfulQA, which are more sensitive to fluctuations and pose a greater challenge. For long sentences in FACTOR, our method shows improvements over baselines by 5-13%. These results not only underscore the benefits of our method for factuality but also demonstrate its robustness across different lengths of text.

Open-Ended Generation Tasks In open-ended settings, we prompt the model to generate answers for the same questions from TruthfulQA, following the settings outlined in [29, 7, 27]. In Table 1, we compare the performance of six LLaMA-2 models using standard greedy decoding, (greedy) DoLa, and (greedy) SLED. All the generated answers are then evaluated by a fine-tuned GPT-3 model for both truthfulness and informativeness scores. Considering that a 100% truthful score can be easily achieved by simply responding with 'I have no comment,' which would result in a 0% informative score and thus is not very useful, we have introduced additional metrics-%Truth × Info and the rejection ratio %Reject -to demonstrate that SLED is a mutual-gains approach to achieve better both truthful and informative scores. We have improved the overall %Truth x Info scores by 3-20% across different models and reduced the rejection ratio by up to 95%. These enhancements demonstrate that our method effectively avoids the 'rejection pitfall,' making it more helpful.

Adaptation to Chain-of-thought Reasoning Tasks Although the StrategyQA and GSM8K tasks are also open-ended and require factual accuracy, the primary focus here is to evaluate how different decoding methods adapt to the Chain-of-Thought (COT) approach for handling complex reasoning tasks. We maintain a repetition penalty of 1, as we will discuss the repetition flaws associated with DoLa in Section 3.5. StrategyQA demands multi-hop reasoning, and as shown in Table 1, our method boosts accuracy across six models, whereas DoLa generally worsens it without a repetition penalty. GSM8K, a benchmark for math word problems that require arithmetic reasoning, also shows consistent accuracy improvement with SLED in 7B, 13B and 70B models.

3.3 Evaluation Across Diverse LLM Configurations

As discussed above and shown in Table 1, our method, SLED, demonstrates strong generalization capabilities across the LLaMA-2 model family, proving robust from 7B to 70B model sizes. In Table 2, we further showcase SLED's impressive performance on the more recent LLaMA-3 family models, both at 8B and 70B sizes, in terms of long paragraph factuality and short answer factuality. Interestingly, SLED is also applicable to different pre-trained models, such as Gemma at both 2B and 7B sizes, and can even be adapted to the increasingly popular Mixture of Experts (MoE) architectures. These results confirm the exceptional adaptability of our method across various LLM configurations.

3.4 Evaluation on Integrating SLED with Other LLM Factuality Decoding Methods

SLED exclusively focuses on contrasting differences between layers without altering other parts of the model. Thus, it remains compatible with other techniques that incorporate additional strategies or utilize auxiliary models. This compatibility allows SLED to be seamlessly integrated into existing

methods, enhancing factuality further without the need for modifications to SLED. We integrate SLED with the following approaches: ITI, AD, CD and ICD. Table 3 shows that SLED leads to accuracy improvements from 1% to 12% across four LLaMA-2 models.

3.5 Ablation Studies and Analysis

Mitigating Repetition Issues Table 4 demonstrates that our method, SLED, effectively addresses a significant issue in DoLa: repetitive content in open-ended generation tasks. Our approach outperforms DoLa without the need for excessive repetition penalty. While a slight increase in the repetition penalty further enhances the performance of our method, excessive penalties, such as 1.1, tend to degrade it. This suggests that SLED does not inherently require heavy adjustments for repetition issues. In contrast, DoLa's performance improves with higher penalties (e.g., 1.1, 1.2, 2), indicating a more critical need for addressing repetitive content. We also employ two intuitive metrics, Repetition-4 and Repetition-Sen, to gauge the severity of repetition issues, following prior research [55]. Regardless of the repetition penalty imposed, our method consistently exhibits lower repetition rates. Table 7 includes some examples of generated text to illustrate this further.

Layer Selection As discussed in Section 2.4, how to choose a good candidate set is still a paradoxically difficult task when applying DoLa. Our method does not exhibit this issue. Instead of selecting a single premature layer from the candidate set like DoLa, SLED contrasts the final layer with all layers in the candidate set and then ensembles all the results. Figure 5 shows that setting a larger candidate set, such as all the 32 layers for LLaMA-2-7B-Base, yields better performance than focusing solely on either the first [0 , 16) or second half [16 , 32) . This implies that our layer-wise contrast approach captures more useful information in a more scientific manner. Furthermore, our tests confirm the robustness of our method even when the candidate set is minimal, such as a single layer, consistently demonstrating strong performance. Our settings mirror those of DoLa.

Parameter Analysis We next investigate the impact of parameters - evolution rate α and evolution scale k -on the performance of SLED using a subset of the FACTOR dataset. We test evolution rates from { 0 . 01 , 0 . 1 , 1 , 2 , 5 , 10 } and evolution scale values from { 5 , 10 , 20 , 50 } . Without extreme

0

5

10

15

20

25

30

Premature Layer

8

10

12

14

16

18

20

Greedy

Ours [16,32)

Ours [0,16)

Ours [0,32)

Ours (single)

DoLa [0,32)

DoLa [16,32)

DoLa [0,16)

7B

Greedy

Ours (single)

Ours [0,16)

Ours [16,32)

Ours [0,32)

DoLa [0,16)

DoLa [16,32)

DoLa [0,32)

0

5

10

15

20

25

30

35

40

Premature Layer

12.5

15.0

17.5

20.0

22.5

25.0

27.5

30.0

32.5

Greedy

Ours [20,40)

Ours [0,20)

Ours [0,40)

Ours (single)

DoLa [20,40)

DoLa [0,20)

DoLa [0,40)

13B

Greedy

Ours (single)

Ours [0,20)

Ours [20,40)

Ours [0,40)

DoLa [0,20)

DoLa [20,40)

DoLa [0,40)

5 61.5

0.01

61.5

0.1

61.5

1

61.5

2

59.5

5

53.0

10

10 65.5

64.5

67.5

68.0

60.5

50.0

20 66.5

66.5

66.0

67.0

64.5

51.5

50 65.5

64.5

64.5

64.5

57.5

41.5

52(G)

58(D)

Evolution Rate

Evolution Scale

(a) LLaMA 2 7B Base

5 67.0

0.01

66.5

0.1

67.5

1

69.0

2

63.0

5

51.0

10

10 70.0

70.5

70.5

72.0

69.0

54.5

20 72.5

72.0

71.5

72.5

68.0

48.5

50 68.5

69.5

69.5

71.0

65.0

45.5

54.5(D)

64(G)

Evolution Rate

(b) LLaMA 2 7B Chat

5 54.0

0.01

54.0

0.1

56.0

1

57.5

2

56.0

5

49.0

10

10 65.0

64.5

66.0

67.5

64.5

56.5

20 62.0

62.5

65.0

63.5

65.0

48.0

50 58.0

56.0

59.0

60.5

58.5

45.0

55(D)

57(G)

Evolution Rate

(c) LLaMA 2 13B Base

5 54.0

0.01

54.0

0.1

56.0

1

57.5

2

56.0

5

49.0

10

10 65.0

64.5

66.0

67.5

64.5

56.5

20 62.0

62.5

65.0

63.5

65.0

48.0

50 58.0

56.0

59.0

60.5

58.5

45.0

56.5(D)

61(G)

Evolution Rate

(d) LLaMA 2 13B Chat

evolution rates (e.g., 10), our method performs well, confirming its robustness. As analyzed in our methodology and Eq. 2, the evolution rate balances the logit distribution ( P N ) with the latent knowledge distribution ( P latent ). A lower evolution rate works better for larger models (13B) and chat models as their logits already better represent real-world distributions.

Latency Our method, SLED, does not incur significant latency overhead. The latencies presented in Table 5 demonstrate that our method, SLED, just increases the decoding time of DoLa by factors ranging from 0.1% to 10%. Notably, even with an atypical setting such as evolution scale = 100 , which is seldom used, the increase remains around 10%. The latency for DoLa and SLED is much higher compared to the vanilla greedy decoding because we set all early layers as candidate layers set for both DoLa and SLED for a fair comparison.

4 Conclusion

We introduced Self Logits Evolution Decoding (SLED), which is a new method to improve accuracy and factuality without requiring external knowledge (e.g., RAG) or fine-tuning (e.g., SFT). The key idea is to optimize the output logits based on the LLMs' latent knowledge to improve factuality during inference. On several datasets, SLED achieved the SOTA results, improving over the vanilla decoding and DoLa. We also show that SLED does not increase the inference time significantly and that it can be combined with other factuality decoding methods. For future work, it would be interesting to combine SLED with supervised fine-tuning methods, e.g., to adapt to other domains.

Acknowledgment

This work was done when Jianyi Zhang was an intern at Google Research. In addition, Jianyi Zhang and Yiran Chen disclose the support from grants NSF CNS-2112562 and ARO W911NF-23-2-0224. We thank area chair and reviewers for their valuable comments.

References

AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/ blob/main/MODEL_CARD.md .

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 , 2023.

Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 17754-17762, 2024.

Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. arXiv preprint arXiv:2403.01548 , 2024.

Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. Lift yourself up: Retrieval-augmented text generation with self-memory. Advances in Neural Information Processing Systems , 36, 2024.

Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv , abs/1706.03741, 2017.

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=Th6NyL07na .

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Yujuan Ding, Wenqi Fan, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meets llms: Towards retrieval-augmented large language models. arXiv preprint arXiv:2405.06211 , 2024.

Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2024. URL https: //openreview.net/forum?id=QAwaaLJNCk .

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Anna Rogers, Jordan BoydGraber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 16477-16508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.910. URL https://aclanthology.org/2023.acl-long.910 .

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics , 9:346-361, 2021.

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2(7), 2015.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 , 2019.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232 , 2023.

Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846 , 2017.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys , 55(12):1-38, 2023.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https://arxiv.org/abs/2401.04088 .

Hadi S Jomaa, Josif Grabocka, and Lars Schmidt-Thieme. Hyp-rl: Hyperparameter optimization by reinforcement learning. arXiv preprint arXiv:1906.11527 , 2019.

Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, and Cyrus Rashtchian. Sufficient context: A new lens on retrieval augmented generation systems. arXiv preprint arXiv:2411.06037 , 2024.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints , art. arXiv:1705.03551, 2017.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 , 2022.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics , 2019.

Deren Lei, Yaxi Li, Mengya Hu, Mingyu Wang, Vincent Yun, Emily Ching, Eslam Kamal, et al. Chain of natural language inference for reducing large language model ungrounded hallucinations. arXiv preprint arXiv:2310.03951 , 2023.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459-9474, 2020.

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inferencetime intervention: Eliciting truthful answers from a language model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 41451-41530. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 81b8390039b7302c909cb769f8b6cd93-Paper-Conference.pdf .

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097 , 2022.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463 , 2023.

Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URL https: //aclanthology.org/2022.acl-long.229 .

Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm, 2019.

Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L. Sifre, Morgane Riviere, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L'eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Am'elie H'eliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Cl'ement Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vladimir Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Brian Warkentin, Ludovic Peran, Minh Giang, Cl'ement Farabet, Oriol Vinyals, Jeffrey Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. ArXiv , abs/2403.08295, 2024. URL https://api.semanticscholar.org/CorpusID:268379206 .

Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. Evolving deep neural networks. In Artificial intelligence in the age of neural networks and brain computing , pages 269-287. Elsevier, 2024.

Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. arXiv preprint arXiv:2307.06908 , 2023.

Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems , 32, 2019.

OpenAI. Introducing chatgpt. https://openai.com/blog/chatgpt/ , November 2022.

OpenAI. GPT-4 Technical Report. arXiv e-prints , art. arXiv:2303.08774, March 2023. doi: 10.48550/arXiv.2303.08774.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934 , 2023.

Xin Qi and Bing Xu. Hyperparameter optimization of neural networks based on q-learning. Signal, Image and Video Processing , 17(4):1669-1676, 2023.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International conference on machine learning , pages 2902-2911. PMLR, 2017.

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802 , 2022.

Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, and Wai Lam. A thorough examination of decoding methods in the era of llms. arXiv preprint arXiv:2402.06925 , 2024.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Christian Thiel. Classification on soft labels is robust against label noise. In Ignac Lovrek, Robert J. Howlett, and Lakhmi C. Jain, editors, Knowledge-Based Intelligent Information and Engineering Systems , pages 65-73, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540-85563-7.

Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Finetuning language models for factuality. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=WPZ2yPag4K .

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), Apr. 2018. doi: 10. 1609/aaai.v32i1.12340. URL https://ojs.aaai.org/index.php/AAAI/article/view/ 12340 .

Chenguang Wang, Xiao Liu, and Dawn Song. Language models are open knowledge graphs. arXiv preprint arXiv:2010.11967 , 2020.

Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, and Yiran Chen. Coreinfer: Accelerating large language model inference with semantics-inspired adaptive sparse activation, 2024. URL https://arxiv.org/abs/2410.18311 .

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/ forum?id=_VjQlMeSB_J .

Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to meta-generation: Inference-time algorithms for large language models. arXiv preprint arXiv:2406.16838 , 2024.

Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681-688, 2011.

Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. Advances in Neural Information Processing Systems , 35:3082-3095, 2022.

Haoran Yang, Deng Cai, Huayang Li, Wei Bi, Wai Lam, and Shuming Shi. A frustratingly simple decoding method for neural text generation. arXiv preprint arXiv:2305.12675 , 2023.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600 .

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020 , 2024.

Chang-Bin Zhang, Peng-Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng. Delving deep into label smoothing. IEEE Transactions on Image Processing , 30: 5984-5996, 2021.

Jianyi Zhang, Ruiyi Zhang, Lawrence Carin, and Changyou Chen. Stochastic particleoptimization sampling and the non-asymptotic convergence theory. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 1877-1887. PMLR, 26-28 Aug 2020. URL https://proceedings.mlr.press/ v108/zhang20d.html .

Jianyi Zhang, Yang Zhao, and Changyou Chen. Variance reduction in stochastic particleoptimization sampling. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 11307-11316. PMLR, 13-18 Jul 2020. URL https: //proceedings.mlr.press/v119/zhang20ac.html .

Jianyi Zhang, Aashiq Muhamed, Aditya Anantharaman, Guoyin Wang, Changyou Chen, Kai Zhong, Qingjun Cui, Yi Xu, Belinda Zeng, Trishul Chilimbi, and Yiran Chen. ReAugKD: Retrieval-augmented knowledge distillation for pre-trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 1128-1136, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. acl-short.97. URL https://aclanthology.org/2023.acl-short.97 .

Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Yufan Zhou, Guoyin Wang, and Yiran Chen. Towards building the federated gpt: Federated instruction tuning, 2024. URL https://arxiv.org/abs/2305.05644 .

Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint arXiv:2312.15710 , 2023.

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 , 2023.

Yang Zhao, Jianyi Zhang, and Changyou Chen. Self-adversarially learned bayesian sampling. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 5893-5900, 2019.

Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations , 2017. URL https://openreview.net/ forum?id=r1Ue8Hcxg .

A Related Work

There have been many advances in improving training and inference to develop better out-of-the-box LLMs [47, 48, 1, 44, 36, 28, 63, 51, 20]. Unfortunately, LLMs still suffer from hallucinations and producing non-factual text. This has led researchers to develop many methods to improve factuality.

Retrieval, Fine-tuning, and Preferences. Many techniques use additional knowledge graphs or fine-tuning data to increase factuality by updating the model parameters for this goal. One method is Retrieval-Augmented Generation (RAG) to use external knowledge to improve generation [3, 5, 9, 25]. Another option is to use post-generation retrieval and editing for improving attribution [11]. Other directions that use additional training or preference data are supervised fine-tuning (SFT) [38, 46], RLHF [37], DPO [40] or self-rewarding [58]. Complementary to these approaches, we wish to improve the LLM output distribution directly without needing any additional data.

Decoding and Factuality Decoding For each prefix, the LLM generates a probability distribution for the next token on a fixed vocabulary list, and a decoding method determines how the next token is derived based on the estimated distribution. Decoding methods were initially developed to enhance the fluency and coherence of text generation, such as Beam Search (BS), which maintains the k most probable sequences at each time step. Common decoding methods also include Diverse Beam Search (DBS) [49], Contrastive Decoding [27], Top-p Sampling [14] and so on.

Recently, the potential of decoding has extended beyond merely improving text readability, with some factuality decoding methods being proposed. These methods modify the generation process to focus on truthful statements rather than unsupported claims during the inference phase, aiming to reduce hallucinations. Notable recent works include Inference-Time Intervention (ITI) [26], InducedContrastive Decoding [64], Decoding by Contrasting Layers (DoLa) [7] and so on. ITI adjusts model activations during inference by following learned directions across a limited number of attention heads to improve truthfulness. Some researchers have extended previous Contrastive Decoding [27] methods to improve factual accuracy, such as Frustratingly Easy Model Decoding [56] and InducedContrastive Decoding [64], leveraging differences between expert and amateur models. Most closely related to our work is DoLa, which also employs contrasting logits from different layers. However, significant distinctions exist: Firstly, our method diverges in how to utilize those differences between logits to extract latent knowledge. Secondly, whereas DoLa directly substitutes the original output distribution with the latent knowledge distribution, our approach recognizes potential inaccuracies in this estimated distribution and adopts gradient descent within an optimization framework to integrate the model's latent knowledge with the original output.

Limitations. As we continue to refine our approach, several aspects of our method can be further developed and enhanced. Our method, SLED, achieves better factuality at the cost of operating slightly slower. Ideally, we could improve the output logits without incurring any computational cost compared to performing inference on the base LLM model. Another aspect is that currently, our experimental results support the superiority of SLED on multiple datasets. Parameter optimization using Bayesian methods [54, 30, 60, 61, 66], evolutionary algorithms [16, 32, 41] or reinforcement learning [67, 6, 19, 39] might also lead to more robust performance. It would also be ideal to back up our results with more theoretical analysis of SLED.

B Additional Analysis and Ablation Studies

Justification on the Gradients Approximation of SLED in Section 2.2 To further support our method's mechanism, which utilizes logits n -logits N to approximate the gradient of KL ( P real , P logits ) at logits = logits n , we manually calculate the Cosine _ similarity ( logits n -logits N , ∇ logits KL ( P real , P logits ) | logits = logits n ) among thousands of tokens and layers. We plot the density function for different models. We find that the majority of these values are positive, demonstrating that the directions of these two vectors are very close. Hence, our gradient approximation strategy in Section 2.2 is reasonable.

Further Ablation Studies for Section 2.4 We design the following two ablation studies to support our claims in Section 2.4. The first study, referred to as ' Ablation 1 ', directly employs P latent as

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Cosine Similarity

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Density

Llama-2-7B-chat

Llama-2-70B-chat

Llama-3-8B-Instruct

Llama-3-70B-Instruct

Mixtral-8x7B-Instruct

the output distribution as discussed in Q 2.2 . The second study, denoted as ' Ablation 2 ', involves directly scaling the differences, { logits n -logits N } , to constrain their magnitudes within [ -1 , 1] . Then, we simply average these scaled differences across different layers and apply them to Equation 2 as mentioned in Q 2.3 . The results presented in Table 6 demonstrate that the design of our SLED is reasonable.

C Qualitative Studies

We present some examples from the StrategyQA dataset in Table 7 to illustrate that our method addresses the repetition issue of DoLa.

D Further Results from Open-ended Generation Task Benchmarks

We have conducted additional experiments on more realistic open-ended generations datasets, HotPotQA [57], Natural Question (NQ) [23], TriviaQA [21]. We adopt the Exact Match(EM) and the F1 score. Different from the setting in the Section 3, we adopt [0 , 2 , 4 , 6 , 8 , 10 , 12 , 14] as candidate layers for LLaMA 2 7B Chat model and [0 , 2 , 4 , 6 , 8 , 10 , 12 , 14 , 18] as candidate layers for LLaMA 2 13B Chat model for both DoLa and SLED. Our method still has robust performance across different datasets and metrics.

Q: Can you buy a fair trade laptop?

DoLa A laptop is a portable computer. A portable computer is a handheld computer. A handheld computer is a palmtop computer. A palmtop computer is a PDA. A PDA is a personal digital assistant. A personal digital assistant is a PDA. A PDA is a handheld computer. A handheld computer is a palmtop computer. A palmtop computer is a PDA. A PDA is a personal digital assistant. A personal digital assistant is a PDA. Thus, a PDA is a PDA. A PDA is a PDA. A PDA is a PDA. A PDA is a PDA. A PDA is a PDA. A PDA is a PDA ...

SLED Fair Trade is a certification for coffee. Laptops are not coffee. Thus, you cannot buy a fair trade laptop. So the answer is no.

Wembley Arena is in London, England. The moon is a celestial body, and the moon is in the sky. The moon is in the sky, and Wembley Arena is in London, England. London, England is on Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon is in the sky. The moon is in the sky, and Wembley Arena is on Earth. Wembley Arena is on Earth, and the moon is in the sky. The moon is in the sky,

DoLa and Wembley Arena is on Earth. Wembley Arena

SLED Wembley Arena is in London. London is in the Northern Hemisphere, and the moon is in the Southern Hemisphere. Thus, you cannot see the moon in Wembley Arena. So the answer is no.

Q: Does Thiago Moises May 13 2020 submission move hypothetically hurt Achilles?

DoLa Thiago Moises is a MMA fighter. He is 1.73 m tall, and Achilles is about 0.00000000000000000000000000000000000000000000000000000000000000000000...

SLED Thiago Moises's submission was a rear naked choke, which is a blood choke. A blood choke can cause a fighter to pass out. Thus, Thiago Moises's submission could hypothetically hurt Achilles. So the answer is yes.

Table 7: We present three examples of the generated text from LLaMA-2-13B-Base on StrategyQA dataset. SLED method can mitigate the repetition issue.

E Implementation Details

We conducted all experiments utilizing NVIDIA A100 GPUs and implemented our method based on the following repositories: DoLa 1 , AD 2 , and ICD 3 . For decoding responses from the LLMs on TruthfulQA, StrategyQA, and GSM8K, we employed greedy decoding. The models were operated with 16-bit floating-point precision and a batch size of 1. For the LLaMA 2 models sized 7B, 13B, and 70B, we utilized 1, 1, and 3 GPUs respectively. Cross-GPU inference, involving model weight sharding, was facilitated by the Hugging Face Accelerate package 4 .

Regarding the details in Section 3.4, we evaluate the 7B-chat model for ITI, as the checkpoint is publicly available. Combining ITI with SLED results in better performance compared to using ITI alone. AD employs an entropy-based metric to measure the 'sharpness' of in-context hidden states and incorporates it into the decoding process. Combining AD with SLED surpasses both the original AD and its combination with DoLa across four model types. For CD, we have conducted experiments in two distinct configurations: (i) the LLaMA 2 13B base model is contrasted with that of the Llama 2 7B base model, and (ii) the LLaMA 2 13B chat model and the LLaMA 2 7B chat model are compared. Applying SLED to the larger models (13B) boosts performance beyond vanilla CD. ICD contrasts a trustworthy 7B model with a fine-tuned, untrustworthy 7B model, and again, applying SLED on the trustworthy 7B model improves factual accuracy further.

F Additional Results of DoLa

Table 9 presents some additional results of DoLa across various benchmarks. 5 Specifically, DoLa in Table 9 selects a subset of early layers as candidates for calculating the Jensen-Shannon Divergence (JSD) instead of using all layers. For example, for the LLaMA 2 7B Chat model, layers [0 , 2 , 4 , 6 , 8 , 10 , 12 , 14] are designated as candidate layers. Notably, a specific trick implemented in DoLa is omitting the post-softmax step on logits for the TruthfulQA multiple-choice task to enhance accuracy. This trick is not applied to the vanilla greedy decoding in Table 9. In contrast, for the results presented in our Tables 1, 2, and 3, this technique is also been applied to vanilla greedy decoding to ensure a fair comparison.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This paper introduces Self Logits Evolution Decoding and shows that it improves the factual accuracy on benchmark datasets. This is the main contribution and is reflected in the abstract and introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Section A.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper introduces a new decoding algorithm for large language models, and validates its performance empirically on benchmark datasets.

Guidelines:

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experiment settings are described in Section 3 and the parameters are listed in Section 3.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use open source LLMs for all experiments, as well as baselines that have publicly available implementations. We do not use any confidential data or libraries for our experiments.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment settings are described in Section 3.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We also follow the settings of existing work in this area for our experiments for a more consistent comparison.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We use publicly available models and datasets, and hence, the inference time is dominated by their computational costs and implementations, which are well documented.

Guidelines:

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The experiments in this paper are conducted on public benchmark datasets. The algorithm proposed in this paper does not pose new safety, security, or other societal risks.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper is about the factuality of LLMs and the experiments show that clearly these LLMs are not ready for critical applications requiring high accuracy answers.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The new decoding algorithm introduced by this paper does not pose new risks as long as it is used on a reasonably safeguarded model.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all papers and creators used in our studies.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not use crowdsourcing.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] Justification: This paper does not involve crowdsourcing, nor research with human subjects.

Peer reviews:
comments: SUMMARY:
This work proposes self-evolution decoding, a method to improve LM's factuality without using external knowledge or fine-tuning data. Specifically, the differences between each layer’s logits and the final layer’s logits are utilized to approximate the gradient, which are further used to estimate the inner knowledge of LMs. Finally, the estimated "inner" distribution is utilized to adjust the model's final outputs. Experiments on various tasks show the effectiveness of the proposed method.

STRENGTHS:
- The paper is generally easy to understand.
- The proposed method is shown to provide good results on a variety of tasks.

WEAKNESSES:
- I think some of the approximations might be questinable. First, in Section 2.2, I'm not sure whether it is reasonable to use the logit differences to estimate the gradient: logits are unconstrained, while the scale of the gradients are constrained within 1. Moreover, in Section 2.3, I'm not sure why the estimations are broken-down for each item in the vocabulary and why the weights for different layers are directly aggregated and normalized with cosine similarties. More ablation studies should be provided to verify some of these choices.

QUESTIONS:
- I'm wondering if there can be good methods to evaluate LM's factuality in more realistic open-ended generation tasks, which are the main scenarios where we hope to improve LM's factuality and consistency.

LIMITATIONS:
Limitations have been discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present a decoding method called Self-Evolution Decoding (SED) to enhance factuality. During the decoding process, SED first estimates the “inner knowledge distribution,” representing the knowledge the model “knows,” by analyzing the difference between the top-layer logits and intermediate layers. This estimated inner knowledge distribution is then used to adjust the LLM output logits, steering the model’s behavior towards greater factuality. Experiments across several datasets and LLMs demonstrate that SED significantly improves the LLM’s factuality.\n\nSTRENGTHS:\nOriginality: The approach of narrowing the gap between model generation and internal knowledge is innovative.\n\nSignificance: Enhancing the factuality of LLMs is a crucial research problem, and the positive results demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. The motivation for this method is somewhat problematic. In section 2.2, the authors claim that the initial layers predominantly encode “lower-level” information while the later layers capture more “semantic” information. However, this does not imply that their difference is a good approximation of the gradient of KL divergence. If you believe this approximation is accurate, why not directly apply this approximated gradient to Equation 3?\n\n2. The experiments primarily focus on task performance but lack an investigation into how this method works. For example, in lines 145-147, the authors claim that the approximated P_inner cannot be directly used as it is not perfect. I believe presenting experimental results would be more convincing than a verbal analysis.\n\n3. The paper is not well-structured, and the concepts are confusing. Please refer to the questions below for more details.\n\nQUESTIONS:\n1. The notion of logits \\mathcal{L} is confusing, which is normally used to represent loss function.\n\n2. Notation is not consistent. In line 69, P refers to the probability distribution, where in line 71 P_L denotes the logits distribution. Is this a typo?\n\n3. Equation 2 is confusing, I initially thought it represented L^n - L^N introduced in Line 90, but there is no description of this equation. I spent quite a while understanding that it is just a derivation of the gradient of the KL function and is not related to line 90.\n\n4. In Line 107, \"In this context\" and \"In this formulation\" duplicate.\n\nLIMITATIONS:\nThe authors acknowledge that the approximated P_inner is not perfect, but they do not provide sufficient experimental results to support this claim. Including more detailed experiments that explore the accuracy and limitations of this approximation would strengthen their argument.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Self-Evolution Decoding (SED), a novel decoding strategy aimed at enhancing the factual accuracy of large language models (LLMs) without the need for external knowledge bases or additional fine-tuning. SED optimizes the outputs of LLMs by refining the logits from the final layer through the inherent self-evolution of hidden states, effectively reducing hallucinations and refocusing the probability mass on factual responses. Evaluations on several benchmarks, including TruthfulQA and FACTOR, demonstrate that SED outperforms existing methods like DoLa, achieving up to a 10% improvement in factual accuracy. The method is also compatible with other factuality-enhancing techniques, further boosting their effectiveness. While the empirical results are promising, the paper notes some computational overhead and calls for further theoretical analysis to better understand the mechanisms behind SED's success.\n\nSTRENGTHS:\n- __Novelty__: The paper presents a novel decoding strategy, SED, which improves the factual accuracy of large language models without requiring additional fine-tuning or external knowledge bases. This approach fills a crucial gap in the existing methodologies for improving the reliability and truthfulness of LLM outputs.\n\n- __Comprehensive evaluation__: The effectiveness of SED is validated across multiple benchmarks such as TruthfulQA, FACTOR, StrategyQA, and GSM8K. The results show that SED outperforms existing methods like DoLa and other baseline strategies, demonstrating significant improvements in factual accuracy and overall performance.\n\n\n- __Compatibility__:  SED is its compatibility with other factuality-enhancing methods. The paper demonstrates how SED can be integrated with methods like Inference-Time Intervention, Activation Decoding.\n\nWEAKNESSES:\n- __Lack of theoretical analysis__: While the empirical results are robust, the paper lacks a rigorous theoretical analysis to explain why SED improves the factual accuracy of LLMs. A better understanding of the underlying mechanics and theoretical justification for the approach would strengthen the contribution.\nExperimental Reproducibility and Statistical Significance:\n\n\n\n- __Computation efficiency__: Although SED improves factual accuracy, this comes at the cost of increased computational complexity compared to methods like DoLa. The paper mentions that SED operates slightly slower, which could be a drawback for applications requiring real-time performance. Further benchmarking on computational costs and scalability would be useful for assessing the practical applicability of SED.\n\nQUESTIONS:\nCould you provide more insights or theoretical justifications for the functioning and effectiveness of SED? Are there any specific properties of the inner knowledge distribution (P_{inner}) that you believe contribute to the success of SED?\n\nLIMITATIONS:\nComputationally inefficiency is the biggest limitation of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
It introduces a novel decoding strategy named Self-Evolution Decoding (SED) aimed at enhancing the reliability and truthfulness of Large Language Models (LLMs). Unlike methods that depend on external knowledge bases or additional fine-tuning, SED is an intrinsic optimization technique that capitalizes on the self-evolution of LLMs' hidden states. The method refines the output during inference, akin to continued training, which improves accuracy and interpretability without sacrificing natural language fluency.

STRENGTHS:
The SED method is an original contribution that tackles the issue of factuality in LLMs by introducing a new decoding approach. This is a novel way to enhance outputs without relying on external data or model retraining.

The concept of optimizing an implicit objective function using the self-evolution of LLMs is creative and presents a new angle for improving model outputs during inference.

WEAKNESSES:
While the empirical results are positive, there is no enough theoretical analysis provided to support the method's effectiveness.

The paper does not report error bars or measures of statistical significance, which are important for understanding the variability of the results.

SED may introduce additional computational overhead during inference, which could be a concern for real-time applications.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
While SED has been tested on multiple datasets, there may be a need for further evaluation on an even broader range of datasets to ensure the method's generalizability.

The paper does not provide a rigorous optimization analysis of SED, which could help understand why it leads to more factual outputs.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-15