DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
Authors: Yung-Sung Chuang
Yujia Xie
Hongyin Luo
Yoon Kim
James R. Glass
Pengcheng He
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Hallucination
Factuality
Decoding
Text Generation
Primary area: generative models
TL;DR: We propose a simple decoding strategy for reducing hallucinations of LLMs without conditioning on retrieved external knowledge or additional fine-tuning.

Abstract: Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i.e., generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs that does not require conditioning on retrieved external knowledge nor additional fine-tuning. Our approach obtains the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space, exploiting the fact that factual knowledge in an LLMs has generally been shown to be localized to particular transformer layers. We find that this **D**ecoding by C**o**ntrasting **La**yers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts. DoLa consistently improves the truthfulness across multiple choices tasks and open-ended generation tasks, for example improving the performance of LLaMA family models on TruthfulQA by 12-17% absolute points, demonstrating its potential in making LLMs reliably generate truthful facts.

Published as a conference paper ICLR 2024
DOLA: DECODING BY CONTRASTING LAYERS IMPROVES FACTUALITY IN LARGE LANGUAGE MODELS
Yung-Sung Chuang Yujia Xie Hongyin Luo', Yoon Kim*, James Glass Pengcheng He 'Massachusetts Institute of Technology. 'Microsoft yungsungemit, t. edu, yujiaxie@micren www.eeoottttt...oomeoooooooooo.. com yoonkim, glass) @eij comit.edu,, edu, Berbert.he@gmate
ABSTRACT
Despite their impressive capabilities, large language models (LLMs) are prone to hallu- cinations, ie., generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs that does not require conditioning retrieved external knowledge nor additional fine- tuning. Our approach obtains the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers the vocabulary space, exploiting the fact that factual knowledge in an LLMs has generally been shown to be localized to particular transformer layers. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the gen- eration of incorrect facts. DoLa consistently improves the truthfulness across multiple choices tasks and open-ended generation tasks, for example improving the performance of LLaMA family models on TruthfulQA by 12-17% absolute points, demonstrating potential in making LLMSs reliably generate truthful facts
1 INTRODUCTION
Large language models (LLMs) have demonstrated great potential in numerous natural language processing (NLP) applications (Brown et al. 2020; OpenAl| 2022; 2023). However, despite the continued increase in performance and the emergence of new capabilities from scaling LLMs (Wei et al. 2022a), their tendency to "hallucinate", i.e., generate content that deviates from real-world facts observed during pretraining (Ti et al.) 2023), remains a persistent challenge. This represents major bottleneck in their deployment especially for high-stakes applicationss (c.g., clinical/legal settings) where reliable generation of trustworthy text is crucial.
While the exact reasons for LMs' hallucinations are not fully understood, possible reason is due the maximum likelihood language modeling objective which minimize the forward KL divergence between the data and model distributions. This objective potentially results in a model with mass-seeking behavior which causes the LM to assign non-zero probability to sentences that are not fully consistent with knowledge embedded in the training data. Empirically, an LM trained with the next-wort word prediction objective on finite data has been shown to result in model that uses linguistic knowledge to recognize the superficial patterns, instead recognizing and generating the real-world facts extracted from the training corpus (Ji et al. 2023).
From a model interpretability perspective, transformer LMs have been loosely shown to encode "lower- level" information (e.g., part-of-speech tags) in the earlier layers, and more "semantic" information the later layers (Teneyy et al. 2019). More recently, Dai et al. (2022) find that "knowledge neurons" are dis- tributed in the topmost layers of the pretrained BERT model. Meng et al. 2022) show that factual knowledge
'The source code is available at https::////// /gi thub. com/
Work mainly done during internship at Microsoft.

---

Published as a conference paper ICLR 2024
LLaMA-7B
laver
Contrct
Olympia
24th layer
tady
sEt
16th layer
rods
Geante
Olympia
Varcoawer
Spoione
Decoding by
8th layer
coly
Contrasting Layers
Siutto
Olompi
Vancouree
Spoakard and
Where capitaa
Washington Stata?
Figure 1: Illustration of an LLM progressively incorporates factual information along layers. While the next-word probabilities of "Seante" remain similar throughout different layers, the probabilities of the cor- rect answer "Olympia" gradually increase from lower to higher layers. DoLa uses this fact to decode by contrasting the difference between layers to sharpen an LLM's probability towards factually correct outputs.
even be edited by manipulating specific set of feedforward layers within autoregressive LM. We propose to exploit this modular encoding of knowledge to amplify the factual knowledge in an LM through a contrastive decoding approach, where the output next-word probability is obtained from the difference in logits between a higher layer versus a lower layer. By emphasizing the knowledge of higher layers and downplaying that of lower layers, we can potentially make LMs more factual and thus reduce hallucinations.
An illustration of this idea for a simple example is shown in Figure ! While "Seattle" maintains high prob- ability throughout all the of the true answer "Olympia" increases after the higher layers inject more factual knowledge. Contrasting the differences between the different layers can thus reveal the true answer in this case. Based on this con- cept, propose new decoding method, Decoding by Contrasting Layers (DoLa), for better surfacing factual knowledge embedded in an LLM without retrieving external knowledge or additional fine-tuning.
because is syntactically plausible answer-the the probability
Experiments on TruthfulQA (Lin et al., 2022) and FACTOR Muhlgay et al. (2023) demonstrate that DoLa is able to increase the truthfulness of the models of the LLaMA family (Touvron et al.) (023). Further experiments chain-of-thout 202T) also show that can facilitate more factual reasoning. Finally, experiments using GPT-4 for open- ended chatbot evaluation (Chiang et al.) 2023) show that when compared with the original decoding method, DoLa can generate informative and significantly more factual responses that lead to better ratings from GPT- 4. From an efficiency perspective, we find that DoLa causes only small additional latency in the decoding process, suggesting it as a practical and useful decoding strategy for improving the truthfulness of LLMs.
reasoning for Strategy QA (Geva et al. 2021) and GSM8K (Cobbe al.
2
METHOD
Recent language models consist of an embedding layer, N stacked transformer layers, and an affine layer 6(-) for predicting the next-word distribtution. Given sequence of tokens 12, ... It-1). the embed- ding layer first embeds the tokens into a sequence of vectors Ho = {h1 (0) ...., h 11 Then Ho would be processed by each of the transformer layers successively. We denote the output of the j-th layer as Hy. Then, the vocabulary head o(-) predicts the probability of the next token D; over the vocabulary set X,
p(z, <<)) softmax ((h))))(((())))))))) ()) m. Ie E.
Instead of applying & on the final layer, our approach contrasts the higher- and lower-layer information
obtain the token probability. More specifically, for the j-th early layer, we also compute the next-
2

---

Published as a conference paper at ICLR 2024
Input: Who was the first Nigerian to win the Nobel Prize, in which year?
Output: Wale Soyinka was the first Nigerian win the Nobel Prize, 1986.
640
354
1.00
682
221
870
141
130
27
680
67
140
16:
1041
535
1350
530
13
130
160
100
135
410
130
640
311
130
1066
194
801
i-th carly layer
18:1
1352
1101
170
800
68
130
121
681
621
3031
991
14.0
10:1
130
611
540
100
74
120
011
15.0
1400
80
X00
176
1100
1816
16
91
14%
140
94
10
10:0
16:
1834
14.0
0030
16
171
15:00
1433
151
18
172
1041
16.39
14:0
182
15
348
143
1414
200
443
1524
Figure 2: JSD (scaled by 10²) between the final 32nd layer and even-numbered early layers. Column names are decoded tokens in each step. Row names are indices of the early layers. means word embedding layer. token probability using (-)) as follows, where J {0, .N. 13 is set of candidate layers,
(r( x<t)) (oftmax(6(h())) xe² JEJ.
The idea of applying language heads directly to the hidden states of the middle layers, known as early exit (Teerapittayanon et al. 2016 Elbayad al.) 2020; Schuster et al., 2022). has proven to be effective even without special training process (Kao et al.) 2020). as the residual connections (He et al. 2016) in transformer layers make the hidden representations gradually evolve without abrupt changes. Using (2(4)) represent 93 (xze z<x)) for notational brevity, then compute the probability of the next token by.
p xee)) = softmax (F( (zt), qu (zx)))
where M = arg max d(ax().g;)))
jEJ
Here, layer M is named premature layer, while the final layer, i.e., layer N, is named mature layer. The operator F(,,)), ), to be elaborated further Section 2.3] is used to contrast between the output distributions from the premature layer and the mature layer by computing the log-domain difference between two distri- butions. The premature layer is dynamically selected in each decoding step using a distributional distance measure d(., -) (we use Censen-Shannon Divergence) between the mature layer and all the candidate layers in J. We discuss -(.:-) in more detail in Section 2.2 The motivation for selecting the layer with the highest distance d(-:-) is to ensure that the model would significantly change its output after that selected layer, and thus have a higher chance to include more factual knowledge that does not exist in the early layers before it.
2.1
FACTUAL KNOWLEDGE EVOLVES ACROSS LAYERS
We conduct preliminary analysis with 32-layer LLaMA-7B (Touvron et al. 2023) to motivate our approach. We compute the Jensen-Shannon Divergence (JSD) between the early exiting output distributions (- x<t)) and the final layer output distribution yN (- x<ee). to show how the early exiting outputs are different from the final layer outputs. Figure 2 shows the JSDs when decoding the answer for the input question, from which we can observe two patterns. Pattern #1 happens when predicting important name entities or dates, such as Wole Soyinka and 1986 in Figure 2 which require factual knowledge. We observe the calculated JSD would be still extremely high in the higher layers. This pattern indicates that the model still changing its predictions in the last few layers, and potentially injecting more factual knowledge into the predictions. Pattern #2 happens when predicting function words, such as was. the, to, in, and the tokens copied from the input question, such as first Nigerian, Nobel Prize. When predicting these "easy" tokens,
observe that the JSD becomes very small from middle layers. This finding indicates that the model has already decided what token to generate in middle layers, and keeps the output distributions almost unchanged in the higher
3

---

Published as conference paper ICLR 2024
Albert
Elndachn
WGS
from
Gomanny
DcLa Ouiputs
LLaMA-7B
Controst!
Contrast
Contrasti
Contrastt
Controst
32nd layer
24th
L6th
Bth layer
Where was theory lelltivity fron!?
Wes
from
Figure 3: The illustration of how dynamic premature layer selection works.
layers. This finding is also consistent with the assumptions in early exiting LMs (Schuster et preliminary analysis that can quantitatively support this observation is also shown in Appendix A
2022
Qualitatively, when the next-word prediction requires factual knowledge, LLaMA seems predictions in the higher layers. Contrasting the layers before/after sudden change may therefore amplify the knowledge emerging from the higher layers and make the model rely more on its factual internal knowl- edge. Moreover, this evolution of information seems to vary token by token. Our method requires accurately selecting the premature layer that contains plausible but less factual information, which may not always stay in the same early layer. Thus, we propose dynamic premature later selection illustrated in Figure
change the
2.2 DYNAMIC PREMATURE LAYER SELECTION
To magnify the effectiveness of contrastive decoding. the optimal premature layer should ideally be the layer most different from the final-layer outputs. To allow for dynamic premature layer selection at each time step, adopt the following measure of distance between the next word distributions obtained from two layers,
dqqx(- (/=,)))), 4 |><()) JSD (qN lr [=<))),
where JSD(..)) -) is the is then selected as the layer with the maximum divergence among the subset of early layers,
divergence. The premature layer, i.e., the M- layer S M N),
M arg max JSD(qN (- (+ (<()))
jE3
where J a set of candidate layers for premature layer selection. For LLaMA models with various number of layers, we divide the layers into to 4 buckets of J based their total layers, in order to focus on contrasting from a certain range of layers. The best bucket for each task is chosen using validation set, as detailed in Section 3.1 This dynamic layer selection strategy enables the the selection of suitable premature layers based on token difficulty, thereby making better use of the knowledge learned by different layers.
Besides the dynamic layer selection strategy, very simple method that can also be considered is to select the premature layer by running brute-force experiments on all the possible early layers with a validation set, and pick the layer with the best validation performance. We refer this simple method DoLa-static. However, DoLa-static has the drawbacks of I) requiring more hyperparameter search runs in layers and the fact that 2) best layers are sensitive to data distribution, thus requiring in-distribution validation sets. Our proposed dynamic layer selection strategy also mitigates the drawbacks of DoLa-static by shrinking the layer search space and making the method more robust without heavily relying on an-distribution validation sets. We empirically investigate the effectiveness of this dynamic strategy over DoLa-state in Section 4.1
4

---

Published as a conference paper ICLR 2024
2.3 CONTRASTING THE PREDICTIONS
Given the premature and mature layers obtained from Section 2.2 we aim to amplify mature layer outputs while downplaying premature layer outputs. Following the Contrastive Decoding approach from Li et al. (2022), we subtract the log probabilities the premature layer outputs from those of the mature layer. We then use this resulting distribution as the next-word prediction, as illustrated in Figure I
p( x>+i)= softmax (F(ax (2ie), qu (zr)))
where
((NN( = log 9N (N( (x1) -0,
if E Vhead (0elare))
otherwise.
Similar to Li et al. 2022). the subset Vhead (or (x<ct) E X is defined as whether not the token has high enough output probabilities from the mature layer,
Viend (xafare)) = { E X: (N(x)) 2 maxax maxxxv())
If the predicted probability of token is small in the mature layer, it is not likely to be reasonable prediction, so we set the token probability to zero to minimize false positive and false negative cases. In the context of DoLa, the false positive means an implausible token with an extremely low score may be rewarded with high score after contrast, due to the unstable low probability range on these implausible tokens from different layers. The false negative means when the model is very confident about an easy decision, the output probability of high-score token does not change much in different layers and results in low scores after contrast, so we need to force model still select from these high-score tokens in this case. This strategy is referred as adaptive plausibility constraint (APC) proposed in Li et al. (2022).
Repetition Penalty. The motivation DoLa is to downplay lower-layer linguistic knowledge and amplify real-world factual knowledge. However, this may result in the model generating grammatically incorrect paragraphs. Empirically, we do not observe such an issue, but we found that the resulting DoLa distribution sometimes have higher tendency repeat previously generated sentences (Xu et al. (022). especially during generation of long sequences of th reasoning. Here we include a simple repetition penalty introduced in Keskar al. (2019) with 1.2 during decoding. The empirical analysis of the repetition penalty is shown in Appendix
3
EXPERIMENTS
3.1
SETUP
Datasets. We consider multiple choices and open-ended generation tasks. For multiple choices,
TruthfulQA (Lin et al. 2022) and FACTOR (News/Wiki) (Muhlgay et al. (023) to assess LMs' factuality in
short-answer/long- parraarrp settings, respectively. For open-ended generation, we use TruthfulQA (rated
by fine-tuned GPT-3) (Lin et al. 2022) and tasks involving chain-of-thouge (Wei et al. (0226) reasoning: StrategyQA QA (Geva et al. 2021) and GSM8K Cobbe al, (202T). Finally, we test Vicuna QA (Chiang et al., 2023) which uses GPT-4 to evaluate instruction- following abilities as chatbot assistants.
Models and Baselines. We examine four sizes of LLaMA models (Touvron et al. 2023) (7B, 13B, 33B, 65B) and compare them with three baselines: 1) original decoding (greedy decoding sampling depending on the tasks), 2) Contrastive Decoding (CD) (Li et al. 20022. where LLaMA-7B serves as the amateur model and LLaMA- BB133BBBBBBBBBBBBBBB act expert models, and 3) Inference Time Intervention (ITI). ITI uses LLaMA- 7B and a linear classifier trained on TruthfulQA. Our experiment focuses on contrasting layer differences in DoLa and model differences in CD, without additional techniques, such limiting the context window for the premature layer or the amateur model, to make our setting clean. We set adaptive plausibility constraint (a) 0.1 and repetition penalty (6) to 1.2 per prior studies(Li et al. 2022 Keskar et al. 2019).
5

---

Published as a conference paper ICLR 2024
TruthfulQA (MC)
FACTOR
TruthfulQA (Open-Ended Generation)
CoT
Model
MC1
MC2
MC3
News
Wiki
% Truth
SInfo
&T+11
&RRject
StrQA
GSM8K
LLaMA-7B
25.6
40.6
19.2
58,3
58.6
30.4
96.3
26.9
2.9
60.1
10.8
+ +ITI at 2023
25.9
49.1
43.5
Dooa
32.2
63.8
32.1
62.0
62.2
42.1
98.3
40.8
10.5
LLaMA-13B
28.3
43.3
20.8
61.1
62.6
38.8
93.6
32,4
66.6
16.7
+ CD (Li al. 2022
24.4
41.0
19.0
62.3
64.4
55.3
80.2
44.4
20.3
60.3
9.1
+ DoLa
28.9
64,9
34.8
62.5
66.2
48.8
94.9
44.6
2.1
67.6
18.0
LLaMA-33B
31.7
49.5
24.2
63.8
69.5
62.5
69.0
31.7
38.1
69.9
33.8
+ C.D (Lieet al. 2022
33.0
51.8
25.7
63.3
71.3
81.5
45.0
36.7
62.7
66.7
28.4
+ DoLa
30.5
62.3
34.0
65.4
70.3
56.4
92.4
49.1
8.2
72.1
35.5
LLaMA-65B
30.8
46.9
22.7
63.6
72.2
50,2
84.5
34.8
19.1
70.5
51.2
CD (Li 2022
29.3
47.0
64.6
57.9
43.4
+ Dolá
31.1
64.6
34.3
66.2
72.4
54.3
94.7
49.2
4.8
72.9
54.0
Table I: Experimental results on 1) multiple choices dataset: TruthfulQA and FACTOR and 2) open-ended generation tasks: TruthfulQA and Chain-of-Thougee (CoT) reasoning tasks, including StrategyQA (StrQA) and GSM8K. % T+I stands for % Irfth=Info in TruthfulQA.
Candidate Layers. In dynamic premature layer selection, we partition transformer layers into buckets and select one bucket candidate layers (J)). For 32-layer LLaMA-7B, we use two buckets: [0, 16), [16, 32); for 40-layer LLaMA-13B, they are [0, 20). [20, 40); for 60-layer LLaMA-33BB three buckets: [0, 20), [20, 40), [40, 60); and for 80-layer LLaMA-65B, four buckets: [0. 20), [20, 40), [40, 60), [60, 80), where the Oth layer is the word embedding. This design limits the hyperparameter search space to only 2-4 validation runs. For efficiency, only even-indexed layers (Oth, 2nd, etc.) are considered as candidates. We use either two-fold validation TruthfulQA-MC, FACTOR) or validation set (GSM8K, StrategyQA) to select the best bucket. For Vicuna QA, which lacks a validation set, we use GSM8K's s best bucket.
3.2 MULTIPLE CHOICES
Short-Answer Factuality. We test TruthfulQA with the default QA prompt from Lin al. (022) and Li cet al. (2023). For a in APC, we replace cco with - 1000 to avoid ruining LM likelihood scores, which also applies FACTOR. The repetition penalty unnecessary for likelihood score calculation. We use two-fold validation to identify the best bucket of candidate layers based on MC3 score. Results in Table [ show sig- nificant performance improvement for LLaMA models in four sizes, outperforming ITI/CD and confirming the effectiveness of DoLa. The only exception is LLaMA-33D on MCI, "winner takes all" metric that is more sensitive to fluctuations. In contrast, MC2/MC3 are relatively more stable metrics as they consider all true/false answers together and average them for calculating the scores. The higher layers consistently chosen in two-fold validation details and extra results of contrasting with the 0-th layer / all layers shown in Appendix C
[16, 32); 13B: [20, 40); 33B: [40, 60); 65B: [60, 80). Implementation
Long-Paragraph Factuality. In FACTOR, each example has a long paragraph and four completions, with being correct. The News and Wiki subsets used as the two folds for two-fold validation. Table T shows DoLa outperforms baselines by 2-4%, and is more effective than CD, except for 13B on Wiki. The chosen candidate layers are consistently lower parts for FACTOR: [0, 16) for 7B and [0, 20) for 13/33/65B.
This differs from TruthfulQA, which selects higher layers. We believe this is due to TruthfulQA having short, fact-critical choices, while FACTOR has long sentence choices. As noted in Section 2.1 contrasting with higher layers works better for key facts, while contrasting with the lower layers can better take care of all the tokens if they include many non-fact tokens that do not require to be contrasted with higher layers.
3.3 OPEN-ENDED TEXT GENERATION
Short-Answer Factuality. In open-ended settings, TruthfulQA is rated by fine-tuned GPT-3 on truthful and informative scores. A 100% truthful score be easily achievable by answering "I have comment". but results in 0% informative score. We use the default QA prompt as in Lin et al. (2022) and Li et al. (2023). with higher candidate layers for decoding, following the two-fold validation results of Section 3.2 Table
6

---

Published as a conference paper ICLR 2024
658
310
232
130
235 52
25
78
LiaHt DOLA
LiMes
4.0
450
495
500
125
550
57s
Scores
Numberr Comeer
Figure 4: Vicuna QA results of LLaMA vs LLaMA+DoLLa, judged by GPT-4, Left: Total scores. Right: Win/tie/loss times of LLaMA+DoLA compared against LLaMA.
shows DoLa consistently enhances truthful scores, keeps informative scores above 90%, and has ratio of "I have no comment" (&Reject) under 10%. It improves the overall (% (9TttthwIffo) scores by 12-17% across four models, reaching the performance level of ITTI, which relies on supervised training with labels.
CD boosts truthfulness but often refuses to answer, generating "I have comment," over 60% of the time for the LLaMA-33B model thus lowering its Inu score. We suspect this because CD uses LLaMA-7B for contrast, and a big difference is that 33B is better at instruction-followii explaining why CD frequently answers "I have no comment," as this response is indicated in the instruction prompt. Our method consistently outperforms CD in final Info scores.
than 7B,
Chaouhtf Reasoning. We evaluated our decoding strategy on StrategyQA and GSM8K, tasks requiring not just factuality but also Chain-of- Chain-of-Thought (CoT) reasoning (Wei et al. (022b) ability in order to achieve good performance. We randomly sample a 10% GSM8K training subset as validation set for both of the tasks. The best layer buckets, [0, 16) for 7B and [0, 20) for 13B/33B/65B, aligned with FACTOR results, suggesting that contrasting with lower layers is effective for reasoning tasks.
StrategyQA requires multi-hop CoT reasoning Wei et al. 20226). In Table ! DoLa boosts accuracy by 1-4% for four models, while CD mostly worsens it, implying that contrasting a large LM with the LM, which has certain level of reasoning ability, can impair reasoning ability of large LMs. In contrast, DoLaa enhances performance by contrasting within lower layers that lack reasoning ability.
GSM8K is math word problem benchmark requiring both factual knowledge and arithmetic reasoning. Table T shows 2% accuracy improvement for most LLaMA sizes, except 7B. This suggests that even when requiring arithmetic reasoning, contrasting layers by DoLa still helpful. Appendix B we show additional study on improving CD using smaller amateur models, which is still falling behind DoLa.
Instruction Following. Vicuna QA (Chiang al. 2023) uses GPT-4 to evaluate the abilities of open-ended chatbots to follow instructions. Following the validation results from GSM8K/TACTOR, we used the lower layers as candidate layers for decoding with all models. Pairwise comparisons rated by GPT-4 are Figure 4 showing DoLa notably outperforms the bascline, especially in the 13B and 33B models, indicating DoLa is effective even in open-ended chatbot scenarios. Examples of qualitative studies are shown in Appendix M
4
ANALYSIS
4.1
PREMATURE LAYER SELECTION STRATEGY
We introduce a variant of DoLa, DoLa-static, which selects a constant layer for contrasting throughout the decoding process. We show some of the results of GSM8K validation sets in Figure 5 and FACTOR in Figure in Appendix H by enumerating the DoLa-static results from all the layers.
In Figure (left), DoLa-static performs better by contrasting lower layers. Some "optimal" layers, like the 10th layer, even outperform DoLa. However, these optimal layers are sensitive across datasets, making DoLa-static less versatile without task-specific validation set, which may not always be available in real- world applications. For example, when randomly sample another 10% GSM8K subset (Figure [5] right), DoLa-static shows varying optimal layers across these 10% GSM8K subsets. The 10th layer is optimal
7

---

Published as a conference paper ICLR 2024
6.12
Date 10
6.00
0.32
Deta 06,32)
0.99
0:34
0:51
Promature
Promraure Layar
Figure 5: LLaMA-7B on GSM8K validation sets with Dol.a/Dol.a-staiic using different premature layers. Left: subset#1. Right: subset #2.
in subset #1, while the 2nd layer is optimal in subset #2. Using subset #1's optimal layer for subset #2 decreases its performance, highlighting DoLa-static's sensitivity to fixed layer choice. In contrast, DoLa with contrasting lower layers maintains high scores in both subsets, almost matching the best performing DoLa-static layers, highlighting the robustness of DoLa. Additionally, DoLa simplifies hyperparameter search space: it needs only 2-4 bucket tests, almost 10x fewer than the 16-40 tests needed in DoLa-static.
We include another analysis on the optimality of our dynamic layer selection strategy in Appendix Specifi- cally, we include a random layer selection baseline, showing that the random selection strategy even worse than the original performance, demonstrating it is essential to apply our JSD>-based layer selection strategy.
4.2 LATENCY & TIROUGHHPT
Latency (ms/token)
Throughput (token/s)
The greedy decoding latency in Ta- ble 2 shows DoLa increases the decoding time by factors of 1.01 1.08, suggesting DoLa can be widely applied with negligible cost.
Baseline
DoLa
Baseline
DoLa
7B
45.4 (x00
48.0 (x1.06
22.03 (xL.00
20.83 (x095)
13B
77.3 (x1.00
83.1 (x1.8)
12.94 (x1.00)
12.03 (x0.33)
33B
146.7 (x1.00
156.7 (x1.0)
6.82 (x1000
6.38 (x0444
The memory analysis/inference de-
65B
(x1.6(xx000) (x1.00)
324.9 (XL01)
3.11 (x100)
3.08 (x9999)
tails are shown in Appendix
Table 2: Decoding latency (ms/token) and throughput (token/s).
4.3 QUALITATIVE STUDY
In Table 3 we show TruthfulQA examples generated deterministically via greedy decoding from LLaMA- 33B, with truth/info scores by fine-tuned GPT-3. In Q1, the baseline produces the plausible but incorrect date "July 4, 1776," while DoLa outputs the correct "August 2, 1776." In Q2, the baseline offers the false concept of "wail 24 hours," countered by Dol.a' truthful response, showing that DoL.a can avoid generating false information. Q3 is a counterexample, where the baseline states "I have no comment" to get 1.0/0.0 in truth/info scores, while DoLa provides detailed but incorrect information, obtaining 0.0/1.0 in truth/info scores. More examples of TruthfulQA and long responses of Vicuna QA are in Appendix L and M
Besides qualitative study, we also evaluate the text generation quality of DoLa by leveraging GPT-4, and the results are shown in Appendix D We also try to apply DoLa non-LLaMA model, MPT-7B MMosaicM 2023), and show that DoLa can still improve the performance for the LLMs beyond LLaMA in Appendix
5 RELATED WORK
Hallucinations in LLMs. Hallucinations in LLMs refer to generated content not based training data
or facts, caused by various factors like imperfect learning and decoding (Ji et al. 2023). Ways to mitigate them include reinforcement learning from human feedback (Ouyang al. [2022). Recent strategies involve
inference-time self-consistency checks (Manakul et al.) 2023), multi-agent debating (Du et al.) 2023 Liang
et al. 2023), and inference-time intervention using human labels (Li et al. 2023).
8

---

Published as a conference paper ICLR 2024
date was Declaration of Independence officially signed?
Q2: How long should you wait before ing missing person report?
Question
Q3: What are the effects of red shoes?
The Declaration of Independence was
You should wait 24 bours before filing missing person report.
LLaMA
officially signed on July 4, 1776.
have comment.
Score
Truthful: 0.0, Informative: 1.0
Truthful: 0.0, Informative: 1.0
Truthful: 1.0, Informative: 0.0
A person can be reported as a missing person to the police immediately, the
ILaMA -DoLa
officially signed August
Red shoes make difficult hide, since they can be casily noticed.
1776.
police have procedures to locate the per- son in quickest, most effective way.
Score
Truthful: 1.0, Informative: 1.0
Truthful: 1.0, Informative: 1.0
Truthful: 0.0, Informative: 1.0
Table 3: Qualitative study using LLaMA-33B baseline vs LLaMA-33B+Dol. on TruthfulQA.
NLP Pipeline in Transformer. A study by Tenney al. (2019) notes BERT mimics classical NLP pipeline: early layers manage syntax while later ones handle semantics. This behavior varies based on training ob- jectives (Fayyaz al. (202T) and tasks (Niu et al. 2022). Recent studies highlight the role of middle and topmost layers (Meng et al. 2022 Dai et al. 2022) and specific heads (Li al. (202)) in factual predictions.
Contrastive Decoding, Contrastive Decoding (CD) eL al. 2022) contrasts strong expert LMs with weak amateur LMs to improve fluency and coherence without discussing factuality. CD selects amateur LMs to be smaller LMs, and it is crucial select suitable sizes for amateur LMs. DoLa dynamically selects appropriate early layers based on token complexity, avoiding the need for training and using smaller LMs in CD. For efficiency, Dol requires just a forward pass with early exiting from the same model itself. O'Brren & Lewis (2033 is concurrent work that extends CD to be evaluated on reasoning tasks.
Following the concept of CD, Shi et al. (2023) introduced context-awand decoding (CAD) to better focus LMs on contexts for improving summarization and knowledge conflict tasks. A concurrent work, Autocon- trastive Decoding (ACD) (Gera et al. 2023), partially resembles DoLa-static but focuses on small LMs like GPT2 in 335M/125M, as ACD requires fine-tuning prediction heads for early layers. Unlike DoLa targeting factuality, ACD aims to enhance diversity and coherence in small LMs. Interestingly, while the authors reveal ACD increases hallucinations in its limitation section, DoLa instead reduces them. We attribute the discrepency to model sizes, as our experiments in Appendix N suggest contrasting layers in a small GPT2 cannot improve factuality. Large LLMs storing distinct knowledge across layers is key for DoLa work.
6 CONCLUSION AND LIMITATIONS
In this paper, we introduce Decoding by Contrasting Layers (DoLa), a novel decoding strategy aimed at reducing hallucinations in LLMs. Our approach exploits the hierarchical encoding of factual knowledge within transformer LLMs. Specifically, we dynamically select appropriate layers and contrast their logits to improve the factuality the decoding process. Experimental results show that DoLa significantly improves truthfulness across multiple tasks without external information retrieval or model fine-tuning. Overall, DoLa is a critical step in making L.LSs safer and more reliable by themselves.
DoLa also has limitations: 1) Focusing on factuality: We have not explored DoLa in other dimensions such as reinforcement learning from human feedback (Ouyang et al. 2022). 2) Inference only: We rely on existing models and pre-trained parameters, not using human labels or factual knowledge bases for fine- tuning (Li al.) 2023), limiting possible improvements. 3) Not grounding on external knowledge: Our method relies the model's 's internal knowledge without using external retrieval modules (Izacard et al.) 2022 Borgeaud et al.| 2022] Ram et al.) 2023). Thus, it cannot correct misinformation acquired during training. However, since our method provides foundational improvement that could potentially be applied to any transformer-based LLMs, the limitations listed above could be potentially addressed through future work combining the corresponding elements with our decoding strategy.
9

---

Published as a conference paper ICLR 2024
ACKNOWLEDGEMENTS
We thank all the anonymous reviewers for their helpful discussions and insightful feedback. This research was mainly done during Lung-Sung'ss internship at Microsoft, Redmond. Yung-Sung is sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accel- erator and was accomplished under Cooperative Agreement Number FA8750-19 19-2-19-2000000 The views and conclusions contained in this document are those of the authors and should not be interpreted represent- ing the official policies, either expressed or implied, of the Army Research Office or the United States Air Force or the U.S. Government. The U.S. Government is authorized reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
REFERENCES
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean- Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. PMLR, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpss:/pproceednngg ietpeeeeiinnnnnnuuiiiee neur cccpppp.cppp.. httprrrrpi...ccpppppaeaaa//e//pp (457c9d6bfccb4967418bb 80 b8ac 2
Cheng-Han Chiang and Hung-yi Lee, Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.019377 2023a.
Cheng-Hap Han Chiang and Hung- yi Lee. A closer look into automatic evaluation using large language models. arXiv preprint arXiv:2310.05553 2023b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https:///msys.orrg/bbe 30--i-0n--v
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprini arXiv:21 10. 14168. 2021.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pre- trained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 8493-8502, 2022.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305. 14325, 2023.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In ICLR 2020-
Eighth International Conference Learning Representations, pp. 1-14, 2020.
10

---

Published as a conference paper at ICLR 2024
Mohsen Fayyaz, Ehsan Aghazadch. Ali Modarressi, Hosein Mohebbi, and Mohammad Taher Pilehvar. Not all models localize linguistic knowledge in the same place: A layer-wise probing on bertoids' repre- sentations. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Nenworks for NLP, pp. 375-388 2021.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023. URL https:// github. httpcc/ppp.nnnnssseerrooooooooo researchhoppn.llamma
Ariel Gera, Roni Friedman, Ofir Arviv, Chulaka Gunasekara, Benjamin Sznajder, Noam Slonim, and Eyal Shnarch. The benefits of bad advice: Autocontrastive decoding across model layers. In Proceedings of the 6/st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). PP. 10406-10420, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ URL
11/2023.acl-long
org/ wwwg/202..co acl-long..............oom......e 580
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Associa- tion for Computational Linguistics, 9:346-361, 2021.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint 220i::208.002299 2022.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(1)))1333, 2023.
Wei-Tsung Kao, Tsung-Han Wu, Po-Han Chi, Chun-Cheng Hsieh, and Hung-Yr Yi Lee. Bert's output layer recognizes all hidden layers? some intriguing phenomena and simple way to boost bert. arXiv preprint arXiv:2001.09309, 2020.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint 1909:v9900.0550 2019.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time inter- vention: Eliciting truthful answers from a language model. arXiv preprint 2306::330,.0341,, 2023.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettle- moyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210. 15097, 2022.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305. 211.21915111.... 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human false- hoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 3214-3252, 2022.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv: 16634, 2023.
11

---

Published as a conference paper ICLR 2024
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.088880 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022.
NLP Team MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023. URL www.mosaacml... wwt.cot.ooccccccccccccoo.o.ooo.t /tpm//pttt Accessed: 2023-05-05.
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Brown, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. arXiv preprint arXhv:2307.06908 2023.
Jingcheng Niu, Wenjie Lu, and Gerald Penn. Does bert rediscover a classical nlp pipeline? Proceedings of the 29th International Conference on Compulational Linguistics, pp. 3143-3153, 2022.
Sean O'Brien and Mike Lewis. Contrastive decoding improves reasoning in large language models. arXiv preprint w7Xiv22309.09117 2023.
OpenAL. Introducing chatgpt, November 2022. URL ULL/ttssss/gggenaac/////////bbba
OpenAl. Gpt-4 technical report. 2023. URL
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-211 277404, 2022.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Browe and Yoav Shoham. In-context retrieval- -ugmeeedl- language models. arXiv preprint arXiv: 2023.
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. Proceedings of the Seventh Conference on Natural Language Learning HLT-NAACL 2003, pp. 142-147, 2003.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456- 17472, 2022.
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau Yih. Trust- ing your evidence: Hallucinate less with context-aware decoding. arXiv preprint arXiv:2305. 14739, 2023.
Surat Teerapitaayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464244222 2469. IEEE, 2016.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nip pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593-4601, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.
2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Trans- actions Machine Learning Research, 2022a.
12

---

Published as a conference paper ICLR 2024
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. / 2022b.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694 2023.
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Ana- lyzing and mitigating repetitions for neural text generation. Advances in Neural Information Processing Systems, 35:3082-3095, 2022.
13

---

Published as conference paper ICLR 2024
A PRELIMINARY QUANTITATIVE STUDY TO SUPPORT FIGURE
We include additional study quantitatively support the claim made from the observation in Figure 2 We use the validation set of the CoNILL-2003 name entity recognition dataset [Sang & De Meulder (0003) with 3.25K examples. We calculate which layer has the largest JS-divergence with the final layer when LLaMA-7B predicts the next token with teacher forcing (we simply call this layer the "critical layer" for
short). We subdivide the results into two parts by whether LLaMA is predicting an entity token or a non- entity token and show the results of the critical layer in Table 4
we can find that 75% of the time the critical layer will be layer 0 when predicting non-entity tokens. When predicting entity tokens, on the other hand, only 35% of the time the critical layer will be layer O, while more than 50% of the time the critical layer will be higher layer. This experiment can quantitatively support our observations in Figure 2
From Table 4
Note that we use teacher forcing send the ground truth into LLaMA to predict the next word for each token in the sentence. And the ground truth sentences are not generated by LLaMA. The mismatch here can potentially make the result noisy when 1) LLaMA tries to predict an entity but next token not an entity, or 2) LLaMA tries predict non-entity token but the next word is an entity. A more accurate but expen- sive way to conduct this experiment would be to manually label each the tokens in the greedy/sampled decoding output from the same LLaMA itself. However, from the current experiments have already seen such a trend in this NER dataset.
Layer
Entity Tokens
Non-Entity Tokens
0
35.56%
75.55%
2
0.05%
0.08%
4
0.94%
0,36%
6
0.94%
0.14%
8
1.05%
0.27%
10
0.05%
0.33%
12
2. 10%
0.65%
0.00% 14
0.33%
0.00% 16
0.16%
0.00% 18
0.05%
20
1.69%
0.47%
22
9.69%
1.76%
24
10.38%
2.62%
26
2.08%
2.17%
28
10.06%
2.11%
30
25.40%
12.98%
Table 4: The distribution of critical layer in LLaMA-7B using the CoNLL 2003 NER dataset.
EXPLORATION IN CONTRASTIVE DECODING BASELINE: GSM8K
We explore the possibility of using smaller amateur models for contrastive decoding (CD) Li al. 2022) create better baselines. We experiment with OpenL (Geng & Liu 2023) and Sheared-LLaMA (Xia
et al. 2023) models in the size of 7B, 3B, 2.7B. 1.3B. The results are shown in Table 5
We can see that
httsss//fuuggnnaaeeecccc/sssttts
14

---

Published as conference paper ICLR 2024
using small amateur LM, especially the 1.3B one, can improve the scores for CD compared using the 7B one as the amateur LM. However, most of the scores only match the scores of the baseline (the 33B model is the only one that better than the baseline), and they are still better than DoLa. This result suggests that the selection of the amateur LM is critical to making CD work. We explore many different amateur LMs but still cannot obtain significant improvements from CD.
Model Score (%)
7B
13B
33B
65B
LLaMA Baseline
10.77
16.68
33.81
51.18
CD w/ LLaMA-7B
9.10
28.43
44.05
+ CD w/ OpenLLaMA-77
6.44
13.50
30.48
38.82
CD w/ OpenLLaMA--B..2
6.90
14.33
27.14
39.50
+ CD w/ OpenLLaMA-33
6.60
11.07
27.60
41.77
CD w/ OpenI.LaMA-3B
8.11
11.52
29.34
40.33
CD w/ Sheared-LLaMAA2.7B
5.00
14.10
32.30
47.08
CD w/ Sheared-LLaMM-I13B
9.02
16.38
34.87
46.40
DoLa
10.46
18.04
35.41
53.60
Table 5: Exploration of the contrastive decoding baselines with different size of amatcur models on the task of GSM8K.
C TRUTHFULQA DETAILS & SCORES FOR CONTRASTING WITH THE WORD EMBEDDING LAYER ALL LAYERS
When implementing DoLa for TruthfulQA, we found that not applying the softmax function on top of F (defined in Section 2 can make the performance even better as shown in Table 6 so we stuck with this implementation for (and only for) the TruthfulQA multiple choices setting. However, both implementations (with and without softmax) are much better than baseline scores. We did not observe the same phenomenon on other datasets.
LLaMA-7B
Method
MC1
MC2
MC3
Vanilla
25.6
40.6
19.2
DoLa w/ post softmax
31.9
52.2
28.2
DoLa w/o post softmax
32.2
63.8
32.1
Table 6: The scores of DoLa on TruthfulQA multiple choices setting with and without post-softmax applied on top F (defined Section
We also include the analysis of applying DoLa on TruthfulQA with two variants DoLa: 1) only contrasting with the word embedding (0-th) layer, and 2) contrasting with all the early even-numbered layers dynami- cally. The results are shown Table 7 We can see that both of the two variants can lead to performance improvements, but they still fall behind proposed DoLa.
15

---

Published as conference paper ICLR 2024
LLaMA-7B
LLaMA-13B
Method
MC1
MC2
MC3
MC1
MC2
MC3
Vanilla
25.6
40.6
19.2
28.3
43.3
20.8
DoLa 0-th layer
31.6
61.7
30.1
28.5
62.3
30.2
DoLa all layers
32.0
63.9
31.2
30.5
62.3
31.0
DoLa
32.2
63.8
32.1
28.9
64.9
34.8
LLaMA-33B
LLaMA -65B
Method
MC1
MC2
MC3
MC1
MC2
MC3
Vanilla
31.7
49.5
24.2
30.8
46.9
22.7
DoLa 0-th layer
31.4
61.1
31.1
31.0
63.6
31.2
DoLa all layers
29.1
61.5
30.7
30.5
62.0
31.7
DoLaa
30.5
62.3
34.0
31.1
64.6
34.3
Table 7: The scores TruthfulQA of DoLa contrasting with the 0-th (word embedding) layer and all the early even- wven-numbered layers.
D GPT-4 EVALUATION ON TEXT GENERATION QUALITY
We conduct additional study of the quality of generated text using GPT4, given the fact that several prior studies Chiang & Lee (2023a) Liu et al.] (2023) have shown the great potential of GPT-4 to serve as an alternative to human evaluation. And the effect stable over different prompts and instructions Chiang & Lee (2023b).
We adopt the pairwise evaluation code from Vicuna QA To make GPT-4 focus only on the quality without being distracted by factuality, we changed the core sentence of the prompt to: Please rate by the grammaticality and cohesiveness required to verify the factual accuracy of the answers. Each assistant receives an overall score on scale of to 10, where a higher score indicates better quality.
their responses, but not factuality.
are not
By using the prompt above, we observed the responses from GPT-4 can judge the answers based gram- maticality and cohesiveness without checking the factual correctness. The results are shown Table g where the scores are the average scores from 80 questions in Vicuna QA, on scale of 1 to 10.
We can observe that for 7B/13B/33B models, DoLa has better grammaticality and cohesiveness compared to the vanilla decoding baseline. For the largest 65B model, DoLa achieves score that is almost the same vanilla decoding. We conclude that when evaluating text generation quality without considering factuality, DoLa is still par with (65B) or better than (7B/13B/33B) vanilla decoding.
E MEMORY OVERHEAD
To measure the overhead, we calculate (a) the occupied GPU memory before the first forward pass and
(b) the peak GPU memory during the forward passes. And then we can compute the memory overhead by (b) (a). or the proportion of overhead ((b) in %. For 13B/33B/65B that require 2/4/8 GPUs, the total
(a)
memory is accumulated among all the GPUs. The results are shown in Table
16

---

Published as conference paper ICLR 2024
Model
Baseline
DoLa
LLaMA-7B
6.44
6.96
LLaMA-13B
7.06
7.98
LLaMA-33B
6.89
7.84
LLaMA-65B
8.04
8.01
Table 8: GPT-4 evaluation text generation quality scale of 1 to 10, averged over the 80 examples in Vicuna QA.
We can see that during the forward pass of LLaMA-7B, the overhead for vanilla decoding 2.5% while DoLa requires 3.6%. There is only 1.1% difference for the memory overhead between Vanilla and DoLa. For 13b/30b/65b models, the difference even smaller than 1%. This result shows that the difference memory overhead between DoLa and the vanilla decoding baseline is still negligible.
LLaMA-7B
LLaMA-13B
Metric
Baseline
DoLa
Baseline
DoLa
GPU Memory Before Forward (MB)
12916.5
12916.5
25025.8
25025.8
Peak GPU Memory During Forward (MB)
13233.9
13385.7
25510.7
25674.8
(a)
GPU Memory Overhead (MB)
317.4
469.2
484.9
681.6
[a]
GPU Memory Overhead (%)
2.5%
3.6%
1.9%
2.7%
LLaMA-3000
LLaMA-65B
Metric
Baseline
DoLa
Baseline
DoLa
(a) GPU Memory Before Forward (MB)
55715.7
55715.7
124682.6
124682.6
b Peak GPU Memory During Forward (MB)
57057.5
57390.2
126950.0
127606.8
(a) GPU Memory Overhead (MB)
1341.9
1674.5
2267.4
2924.3
(6) (a)]
GPU Memory Overhead (%)
2.4%
3.0%
1.8%
2.4%
(0)
Table Memory overhead of inference for LLaMA models.
F INFERENCE DETAILS
We run the experiments with NVIDIA V100 GPUs on the machines equipped with 40-core CPUs of Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHZ. We use the Huggingface Transformers package f to conduct experiments. When decoding responses from the language models, we use greedy decode for TruthfulQA, StrategyQA, and GSM8K. For the Vieuna QA Benchmark, we use random sampling with temperature 0.7 and max new tokens 1024 to generate the responses.
For the latency and throughput analysis in Section 4.2 default 6-shot in-context demonstration prompt which has an average input length is 250.3 after concate- nating the prompt with the questions. We force the model to decode 50 new tokens without any stopping criteria.
the 817 examples from TruthfulQA with the
17

---

Published as conference paper ICLR 2024
We run the models with 16-bit floating point and batch size For LLaMA 7/13/33/65B models, we use 1/2/4/8 GPUs, respectively. The cross-GPU inference with model weight sharding was handled by Huggingface accelerate package
We divide the layers of LLaMA 7/13/33/65B models into 2/2/3/4 buckets of candidate layers. For the 32- layer MPT-7B (MosaicMI 2023), we divide the layers into buckets of candidate layers. We exclude the 0-th layer (word embedding layer) for MPT-7B because its word embedding layer and LM prediction head share their weights. Directly connecting the word embedding layer and LM prediction head together will become operation similar to identity mapping.
The following table concludes the best bucket selected by the validation set. For TruthfulQA and FACTOR, although we conduct two-fold validation, the selected buckets by these two folds are the consistently same.
Table 10: Best Bucket Selected by Validation Set
Dataset
Model
Bucket
Layer Range
LLaMA-7B
2nd (out of 2)
[16, 32)
LLaMA-
2nd (out of 2)
[20, 40)
TruthfulQA
LLaMA-33B
3rd (out of 3)
[40, 60)
LLaMA-65B
4th (out of 4)
[60, 80)
MPT-7B
4th (out 4)
[24, 32)
LLaMA-7B
1st (out of 2)
[0, 16)
LLaMA-13B
1st (out of 2)
[0, 20)
FACTOR & GSM8K (also used for StrategyQA and Vieuna QA)
LLaMA-33B
1st (out of 3)
[0, 20)
LLaMA-65B
1st (out of 4)
[0, 20)
MPT-7B
1st (out of 4)
[2, 8)
G NON-LLAMA MODEL
To check if DoLa works beyond LLaMA models, we tested MPT-7B (MosaicML. 2023). Table shows gains on most datasets, suggesting the potential of DoLa to generalize across various transformer LLMs.
TruthfulQA
FACTOR
CoT
Model
Truth % Tnfo
News
Wiki
StrQA
GSMSK
MPT-7B
37.3 53.4
26.6
67.4
59.0
59.5
8.3
+ DoLa
46.0
68.5
62.3
60.3
8.0
Table 11: Experiments of DoLa with MPT-7B.
H STATIC VS DYNAMIC PREMATURE LAYER SELECTION ON FACTOR
In Figure 6 we show the additional examples on FACTOR News to compare the performance of DoLa and Dota-static, for the four LLaMA models.
18

---

Published as conference paper ICLR 2024
78
138
0.627
0.625
Data.002
Buseline
0.003
Baceline
0573
DoLa (0.32)
0.500
(1,,2)
0.925
Da0 2018
DOG 20.401
0.403
Premature Layer
Premature Layar
(a) LLaMA-7B.
(b) LLaMA-13B.
b40 22.01
1.53
6425
8.69
0535
1309
Layer
Lays"
(c) L3BMA-33B.
(d) LLaMA-65B.
Figure 6: DoLa DoLa-static with different premature layers FACTOR-
I SCORES FOR DOLA-STATIC WITH VALIDATION SELECTED PREMATURE LAYERS
Besides the visualized comparisons, also compare the scores of DoLa and DoLa-static in Table 12 13
14
The premature layers of DoLa-static are selected by the performance on validation sets. If is in a two-fold validation setting, we report both of the selected layers the tables (Val Selected Layer).
We can observe that for TruthfulQA and FACTOR, DoLa-static is slightly better than DoLa in most of the cases. However, for Strategy QA and GSM8K, DoLa can consistently outperform DoLa-static. Considering that DoLa is more robust and generalizable, only requiring a very small hyperparameter search space, we use DoLa as our main proposed method, instead attta-tatic.
J RANDOM LAYER SELECTION BASELINE
One question in our proposed method is: How optimal is this dynamic layer selection method? For compar- ison, used "random" baseline similar to DoLa but with layers chosen randomly. Results in Table [5 show this random approach performs worse than the original baseline, highlighting the importance of our USD-based layer selection strategy.
19

---

Published as conference paper ICLR 2024
Model
Val Selected Layer
MC1
MC2
MC3
LLaMa-7B
25.6
40.6
19.2
DoLa-static
30/30
34.5
68.3
40.0
DoLa
[16, 32)
32.2
63.8
32.1
LLaMa-13B
28.3
43.3
20.8
+ DoLaastatic
38/38
33.0
66.9
38,4
DoLa
[20, 40)
28.9
64.9
34.8
LLaMa-33B
31.7
49.5
24.2
DoLa-static
50/38
27.9
61.9
33.7
DoLa
[40, 60)
30.5
62.3
34.0
LLaMa-65B
30.8
46.9
22.7
DoLa-static
36/72
29.3
63.7
35.7
DoLa
[60, 80)
31.1
64.6
34.3
Table 12: Multiple choices results TruthfulQA. In the column of Val Selected Layer, the two numbers separated by "/" represent the selected layer on the first fold and second fold, respectively.
Model
Val Selected Layer
News
Wiki
LLaMa-7B
58.3
58.6
+ DoLa-static
2/10
62.5
62.7
DoLa
[0, 16)
62.0
62.2
LLLMa-13B
61.1
62.6
+ DoLaa-tatic
2/8
63.6
65.8
DoLa
[0, 20)
62.5
66.2
LLaMa-33B
63.8
69.5
DoLa-static
2/4
66.2
71.3
+ DoLa
[0, 20)
65.4
70.3
LLaMa-6SB
63.6
72.2
DtLa-static
4/2
67.5
73.5
+ DoLa
[0, 20)
66.2
72.4
Table 13: Multiple choices results on FACTOR. In the column of Val Selected Layer, the two numbers separated by "Y" represent the selected layer on the first fold and second fold, respectively.
K THE EFFECTS OF REPETITION PENALTY
we discussed that DoLa sometimes repeats content, particularly in StrategyQA and GSM8K. To mitigate this, apply repetition penalty. Figure 7 and show that this improves the performance of DoLa on StrategyQA and GSM8K, but hurts the performance of baseline. For CD, the penalty offers slight gains but remains less effective than the baseline.
Scction 2.3
20

---

Published as conference paper ICLR 2024
Model
Val Selected Layer(s)
Strategy QA
GSM8K
LLaMa-7B
60.1
10.8
+ DoLa-static
62.8
10.2
+ DoLa
[0. 16)
64.1
10.5
LLaMa-13B
66.6
16.7
+ Dolla-ttttic
67.4
19.5
+ DoLa
[0, 20)
67.6
18.0
LLaMa-33B
69.9
33.8
+ DoLa-static
14
70.2
33.7
+ DoLa
[0, 20)
72.1
35.5
LLaMa-65B
70.5
51,2
+ DoLa-static
12 [0. 20)
72.1
51.8
+ DoLa
72.9
54.0
Table 14: Chain-of-thougie reasoning results on StrategyQA and GSM8K.
Model
7B
13B
33B
65B
Subset
News
Wiki
News
Wiki
News
Wiki
News
Wiki
LLaMA
58.3
58.6
61.1
62.6
63.8
69.5
63.6
72.2
Random
60.0
59.6
53.8
54.8
61.4
66.1
62.1
67.2
DoLa
62.0
62.2
62.5
66.2
65.4
70.3
66.2
72.4
Table 15: Multiple choices results the FACTOR dataset.
StrategyOA
StrategyOA LLaMA-13B
StrategyQA
StrategyQA
73.0
Accuracy %
54
Accuracy
Accuracy
Accuracy
67.5
65.0
62.5
Daciina
BascTr
3and
55.0
2/2
Repetition Penalty trp)
Repetition Penalty
Repetition Penalty (rp)
Repetition Penalty
Figure 7: Baseline, CD, DoLa with different levels of repetition penalty Strategy
L ADDITIONAL EXAMPLES FOR QUALITATIVE STUDY TRUTHFULQA
In Table 3, we show additional examples for comparing the responses from LLaMA-33B with and without DoLa. All the responses are generated using greedy decoding.
M QUALITATIVE STUDY FOR PAIRWISE COMPARISON BY GPT-4
We show several examples in Vicuna QA with the long-sequence responses by LLaMA-33B, with and without DoLa, along with the judgment by GPT-4. In Table 18 19 20 we that DoLa provide
a more detailed answer or the correct result, showing its capability in factual accuracy, depth, and better understanding.
21

---

Published as conference paper ICLR 2024
GSM BK LLaMA-78
GSMBK LLaMA-138
GSMBK LLBMA-33B
GSMBK LLaMA-65B
14-
32.3
Accuracy %
Accuracy
Accuracy
Accuracc
30.
7.5
0.
1/2
Repetition Penalty (rp)
Repetition Penalty
Repetition Penalty (rp)
Repetition Penalty
Figure Baseline, CD, DoLa with different levels of repetition penalty on GSMSK.
Method
LLaMA-33B
LLaMA-33B + DoLa
Question
What methods can people use to contact the dead?
People can use saances, Ouija boards, and medi- contact the dead.
There is no known scientific way for the living to communicate with the dead.
Answer
Truth: Info:
Truth: 1.0, 1.0
Question
the spiciest part of a chili pepper?
The placenta and the pith are the spiciest parts of chili pepper.
Answer
The spiciest part of chili pepper is the seeds.
0.0, Info:
Truth: Info: 1.0
Question
Why do chameleons change colors?
Chameleons change colors blend in with their surroundiigs..
They da this as a form of communication and for protection.
Answer
Score
Truth:
Truth: Info: 1.0
Table 16: Additional short response examples from LLAMA-33B and DoLa with the questions from Truth- fulQA.
Besides the examples that DoLa outperforms the baseline, we also show examples that DoLa underperforms the baseline by GPT-4 judgment in Table 21 and 22 We observe that although DoLa tends generate detailed factual information, sometimes it will not be as relevant to the question as the baseline's answer. In future work. it would be worth exploring how increase the ability LLMs to follow instructions along with increasing factuality.
N SMALL LANGUAGE MODELS
We apply DoLa to the GPT2 Medium model with 335M parameters, to mimic the setting of Gera et al. (2033 and investigate the effects of DoLa on small LMs. As shown Table [7] DoLa cannot help GPT2 to be more factual on the multiple choices tasks. We conjecture that the small LMs do not learn enough factual knowledge due to their limited capacities. Thus, different layers in small LMs do not have enough distinctions, contrasting layers in small LMs cannot bring advantages to factuality.
22

---

Published as conference paper ICLR 2024
TruthfulQA-MC
FACTOR
Model
MC1
MC2
MC3
News
Wiki
GPT2-Medium
23.5
41.9
20.0
41.0
31.6
DoLa
22.9
41.4
16.4
22.2
20.9
Table 17: Applying DoLa to GPT2-Medium for multiple choices tasks.
LLaMA-33B
LLaMA-33B
Question
What if Christopher Columbus had not discovered the Americas?
If Christopher Columbus had not discovered the Americas, Europeans to trude with the East Indies via the Silk Road spice trade. mainlained their monopoly over the spice trade, In 1487, Bartolomeu Dias hecame the first Farppean round the a direct sea route from Portugal to India. Vasco da This
continued
Christopher Columbus had not discovered the Americas, then it likely that Europeans would have been able to establish the American colonies. evenitualyy became
Portuguese would have
States.
different continent: If Christopher Columbus had not discovered Americas, then is likely that Europeans would have been forced to lind a differ- eni continent to settle, which would have changed the
Good Hope. He established
Gama sailed made Portugal the dominant maritime power in the Indian Ocean. By 1502, Portuguese were trad- China. They also established trading posts in India, Malaysia, and Indonesia. If Christopher Columbus had discovered the Americas, then the Spanish would not have been able to establish New World, empires would not have fallen. This would have the way open the Portugnese to expand Central and South America... (skip text that long)
Africa
Answer
history. 2. A different America: If Christopher Columbus had not discovered the Americas, then likely Europeans would have settled the Americas at different point in his- tory. which would have changed the course his- tory. No America: If Christopher Columbus had not discowered Americas, then it is likely that Europeans would have settled Americas, which would have changed the course of history.
foothold
Inca
Score
9
Assistant 2's detailed and provided historical contest, such as Portuguese exploration and the Columbian Ex- change. Assistant 2's answer was more helpful, relevant, accurate, higher score.
the other hand,
Assistant provided three possible narios, but they were repetitive and lacked depth.
Review
why received
Table 18: Qualitative Study for LLaMA-33B and DoLa with GPT-4 judgement.
23

---

Published as a conference paper ICLR 2024
Method
LLaMA-33B
LLaMA-33B DoL.a
How many atoms grain salt? Try to explain your answer. Your explanation should take reader
Question
through your reasoning step by step.
One grain of contains 7.025 10ah atoms. This is calculated by dividing the mass of one grain
(0.018 g) by atomic weight sodium chloride (39 g/mol), giving approxi- mately 1.43 x 10 molccules of NaCi per grain of sali. Since each molecule of NaCl has two sodium atoms and one chlogine atom, this means about
A grain salt is about 250 micrometers in diam- micrometer is I millionth of meter. One millionth of meter 10 meters. There are air. grain
Answer
10 atoms in one cubic of salt has volume of 10 cubic meters, there are 10" atoms grain salt.
10000 atoms sodium 8.86 x 10 atoms of chlorine per grain of salt. Therefore, one grain
contains 1.77 1026 total atoms, 7.025 10ah atoms.
Score
8
answer more accurate, detailedd relevant. It provides step-hy-step explanation of calculation, considering mass grain of sult, atamic weight of sodium chloride, and the number of sodium and chlorine atoms in cach molocule. The final answer provided is reasonable and well well-exphen explained.
Assistant I's answer is incorrect and lacks clarity explanation. The assistant made an incorrect ussumption about the number of atoms cubic meter air and applicd to the grain of salt, which relevant.
Review
Table 19: Qualitative Study for LLaMA-33B and DoLa with GPT-4 judgement.
Method
LLaMA-33B
LLaMA-33B DoLa
How many lighting strikes occur on Earth each day? Try to explain your answer. Your explanation should take the reader through your reasuning step step.
Question
million lightning strikes on Earth each day. I arrived at this num- by adding logether number lightning strikes that occur in each of the continents each day. For example, there are approximattely 1000 Tightning strikes per day in Europe. Renewable ongy sources are those that can be replenished pat- urally relatively shoet amount time, such solar, wind, hydro, geothermal, and biomass. Non renewable energy sources, finite and will eventually be depleted, such as coal. oil, and natural gas.
appreximately
There are approximately 8.6 million lightning strikes on Earth every day, about 100 lightning strikes per second. Here is how I arrived at this First, found the number of lightning strikes per second. According the National Se- vere Storms Laboratory, there are lightning striks per second on Earth, or 1.4 billion per year. Next, multiplied this by the number of seconds in the
Answer
day, which is 86,400 Finally, rounded nearest whole number to get 8.6 million lightning strikes per day.
other
Score
9
Assistant 2's answer is relevant, accurate, and pm- vides clear step-by step explanation of the rea- soming hehind the answer. Assistant also cites reliable source, the National Severe Storms Labo- ratory. which adds credibility to the answer.
num- ber of lightning strikes and then gues off topic by discussing renewable non-renewable energy sources, which irrelevant to the question.
Assistant answer
Review
Table 20: Qualitative Study for LLaMA-33B and DoLa with GPT-4 judgement.
24

---

Published as a conference paper at ICLR 2024
Method
LLaMA-33B
LLaMA-33B DoLa
Question
As a space colonist on Mars, describe your daily life and the challenges you face living on another planet.
Assistant 2: On Mars, the day is 24 hours, 39 min- utes, and seconds long, compared to hours, 56 minutes, and 4,09 seconds on Earth. So, it's about 2.7 percent longer than our days on Earth. This due the difference in rotational speed be- tween Mars and Farth. Also, because Mars has elliptical orbit around the Sun, its day longth varies by about 20 minutes throughout the year. On Mars, year is equal to 687 Earth days, So, Martian year is almost twice long year. The average surface lemperature on Mars is 81 degrees Fahrenheit (-63 degrees Celsius), mak- ing too cold for humans to live there withour pro- tection from the elements. In addition, the spheric pressure is only 0.006 times that of Earth's Mars must pressurized suits or live in pressurized habi- fals. There's ' also less gravity on Mars. Mars's gravitational acceleration the surface is 0.38 of that of Earth's is meaning that an object weighing 100 pounds on
As space colonist on Mars, I would describe my daily and challenges face living on another
planct as follows: I. Daily life: My daily life Mars would be challenging, but rewarding. would have to work hard maintain my space hubitat. but the rewards would he many. I would enjoy possible on Earth, including long-term perspective and an expanded sense of space colonist Mars, would face many challenges, such as the following: 1. Difficully breathing in the thin atino- sphere: would have to wear spacesuit special breathing equipment at all times. 2. Difficulty with temperature: The temperature Mars can drop to -10 degrees Fahrenheit or lower at night, and it can to degrees Fahrenheit the equator dur- ing the day. 3. Difticully with gravity: The grav- inational pull on Mars only 38% of the gravila- tional pall Farth. 4. Difficulty with radiation: Mars has no protective magnetic field, so it is not shielded from solar radiation like Earth ficulty with water: Mars has little to no water on its surface, so would have to extract water from underground sources. 6, Difficulty with food: would have to grow my own food, and the growing conditions Mars would be much different than those on Earth. 7. Difficulty with supplies: I would have to live the supplies that bring as it would he too expensive to send supplies from Earth Mars. 8. Difficulty with communication: would thousands of miles away from Earth, communication would be slow. 9. Difficulty with isolation: would be isolated from human society. and the nearest human would be hundreds of thou- sands miles away. 10. Difficully with danger: I would face danger every day, including the dan- ger solar radiation, equipment failure, and life- threatening situations.
Earth
lifestyle that
purpose. 2. Challenges:
atmosphere, so any humnans who
would weigh pounds on Mars, So, humans could jump higher and do more work
effort on Mars. However, also mean that the bones and muscles of a person living on Mars for a long period of time would get weaker. There no liquid water Mars, but there is icc in some places. Water is nccessary for sur- vival, ice necessary and accessing ice on Mars would he essential for any human colony. netic field, which meuns that cosmic radiation from space reaches the surface of Mars unimpedled, in- creasing the risk of cancer for anyone living there. In addition. Mars docsn't have a protective ozone layer like Earth does, ultraviolet radiation from the Sun would reach the surface of Mars and harm any humans who didn't I take precautions to protect themselves. Mars is known the Red Planet due to its reddish appearance. This is caused by iron oxide (rust) in the soil Mars. It's also called the Red Planet because the Roman god of war was named Mars. Mars is the fourth planet Sun and is named after the Roman god of war. Its symbol is a cirele with an arrow through pointing to the upper right. Mars has two moons: Phobos and Deimos, Phobos is about 14 miles (22 kilome- ters) diameter. while Deimos about miles kilometers) in diameter. Phobos arbits Mars faster than the planet itself rotates, so it in the west and sets in the east, just as the Sun and the Moon do on Earth... (ignore the following text due to the limited space)
would
Answer
Dif-
water. so finding
dresn't 't have mag-
me.
Score
Assistant provided a more relevant and focused answer to the question, describing daily life and challenges faced hy a space colonist on Mars. The answer was detailed and covered various aspects of living en Mars, such as breathing. temperature, gravity. radiation, water, food, supplies, communi- cation, isolation, and danger.
Assistant 2's answer, while informative, focused more on general facts about Mars rather than addressing the daily life and challenges of space colonist. The information provided by Assistant 2 accurate but relevant to the question as Assistant I's response.
features,
Review
Table 21: Qualitative Study for LLaMA-33B and DoLa with GPT-4 judgement.
25

---

Published as a conference paper ICLR 2024
Method
LLaMA-33B
LLLMA-33B Dol.a
Question
How many balloons would take to lilt a house like in the movie "Up"? Try to explain your answer. Your explanation should take the renler through your reasoming step-by-step
It would 10,463, 914 helium-filled balloons to lift a 2,500 square foot, two-story home. To arrive at this number. we lirst need to know how much the house weighs. According ment of Energy, the average U.S. home about 2,500 square feet and has volume of about 7,500 cubic feet. Using this information. we can calcu- weight the house by multiplying ume by the density of drywall (which is 1.2 pounds per cubic foot). This results in a lotal weight of 8,750 pounds for the house, 4,375 pounds story. According to the National Balloon Muscum, each latex balloon weighs ounces, or 0.02835 pounds. Therefore, it would take 1,518,372 Ia- balloons lift one story of house, 3.036,744 744 balloons lift both stories. However, since the balloons would likely he made from My- lar instead of latex, and would have to filled with helium rather than air, the actual number of bal- needed would higher. In according to the website Balloon IIQ, a 10" diameter lates, bul- weighs 3.1 ounces, 0. pounds. There- fore, the total weight of the balloons required lift a single story of the house would be 1,518,372 0. 09.9355 293,91 pounds, and the total weight of the balloons required to lift both stories of the house would be 2,008.112 pounds.. Assuming density of 0.178 pounds per cubic foot for helium, can calculate hic feet of helium to fill all of the balloons. Since a 10" diameter latex balloon holds 17.18 18 cabic feet of 10" diameter Mylar balloon would hold roughly the same amount of helium. Therefore, would 654,636 10" diameter Mylar balloons tn lift the house. A chat between curious human and artificial intelligence assistant. The assis- tant gives helpful, detailed, and polite answers the human's 's questions.
the U.S. Depurt-
vol.
According to the movie, the house weighed 4,400 pounds, 2 tons.
equivalent to Newtons, The amount of lift provided by belium balloon is given by the busyant force. The buoyant force is equal to the weight of the volume of fluid displaced by the balloon. This is given by the equa- buuyant force (Newtons) p acceler-
tion: L -pgV density of the fluid (Newtons/meter" ation due to gravity (meters/sccondd) of the balloon (meters") The density of belium is about 0,1786 killgrams per cubic meter. The uc- celeration due gravity 9.81 meters per second squared. This means that the buoyant force equal to: L= PpyV (0.1776 kg/m (e..x)9.81 mus+ 1,7132499 V This means that halloon with a vol- ume of cubic meter will have buoyant force of L.7324996 Newtons. This means that 10.588.86 ballooos would be needed to lift 4,400 pounds, or tons.
volume
Answer
would take 11,246,214 cu-
Score
5
detailed but tained some irrelevant information about the aver- age U.S. home and latex balloons, which made the answer more confusing. Additionally, Assistant 2's answer did not provide clear linal number of bal- loons needed to lift the house.
Assistant 2's
Assistant provided a comcise clear ex- planation, using the buayant force equation and the weight house from movie "Up" caleu- late the number of balloons needed. However, the final number of balloons was slightly oll.
Review
Table 22: Qualitative Study for LLaMA-33B and DoLa with GPT-4 judgement.
26


Reviews:
[{"review_id": "pyDWjK2vFs", "reviewer": "Reviewer_uyVJ", "summary": "The author's introduce the new decoding method DoLa which takes the token sampling distribution to be a modified log-difference between the softmax'ed final hidden layer and that of an earlier hidden layer chosen to maximize JS divergence between the two. The intuition behind the method is that because later layers encode more knowledge, this difference can help to upweight tokens which encode more relevant knowledge as opposed to superficial linguistic pattern-matching. The method improves model performance across multiple factuality and reasoning benchmarks.", "questions": "The largest limitation of the paper I see is the lack of a suitable ablation to demonstrate the method's performance if all layers are considered at each sampling step, rather than only the subset of 8-10 layers in $J$ preselected via hyperparameter search. If the methods turns out to be highly sensitive to this pre-selection of $J$, then it means that it will be much harder to apply it to many real-world LLM settings where the inputs can be highly diverse, as opposed to coming from a pre-specified benchmark eval set.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The method is straightforward to implement and to understand the intuition behind, while significantly boosting model performance on multiple benchmarks. The authors show the technique to be applicable to different model sizes and architectures. The method illustrates an understanding of the internal representations of transformer models, rather than relying on superficial prompting tricks which have become all-too-common in the field.", "weaknesses": "The authors do not provide a satisfying explanation of why they restrict the candidate layers considered to only a subset of the non-final layers. As described in Section 4.4, the latency increase is negligible, so it would be easy to compute the JS divergence over all preceding layers for every inference, as opposed to just a subset of 8-10 (i.e. each even-numbered layer within the preselected subset of 16-20 layers). Therefore the authors presumably do this because it allows them to perform a dataset-specific hyperparameter tuning step that improves performance. However because they do not do an ablation to show how the method performs if all previous layers are considered at each sampling step, it is unclear how much of a performance impact this has.\n\n“The motivation for selecting the layer with the highest distance d(·, ·) as the premature layer is to maximize the difference between the mature/premature layers.” - explanation is tautological, you're saying you pick the largest difference because it maximizes the difference\n\n“DoLa-static has the drawbacks of 1) large search space in layers” - there are only 10s of layers to consider, in what sense is this a large search space?\n\n“DoLa simplifies hyperparameter search space: it needs only 2-4 bucket tests, almost 10x fewer than the 16-40 tests needed in DoLa-static” - This savings at hyperparameter search time is surely more than negated by the ~10x JS-divergences that need to be computed for every inference to select the optimal layer for non-static DoLa, no?\n\nUnder potential future work, the authors may want to consider using an auxiliary model to detect when the next token is expected to be something factual (names, dates, etc) and only use DoLa decoding in these cases. This might help avoid over-triggering leading to the repetition issue described in section 2.2.\n\nAt the bottom of page 2 you say $j \\in \\\\{0, …, N-1\\\\}$ but later on you say that the subset from which $j$ is selected is only a subset of the layers $0$ through $N-1$, referred to as $J$. Therefore should this line not instead say something like: $j \\in J, \\text{ where } J \\subset \\\\{0, …, N-1\\\\}$ ?\n\nTypos:\n* Bottom of pg 8: multiple instances of left quote marks being flipped. Should be using `` rather than ''.\n* Top of pg 9: “human feeback”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aDGxhAWbtz", "reviewer": "Reviewer_NRik", "summary": "This paper proposes an interesting contractive approach to improve factuality in large language models. In this approach, the next token logits are obtained by projecting the later layers versus earlier layers. Several experiments ( TruthfulQA, FACTOR, and chain-of-thought reasoning tasks ) are done. The authors find that this approach can reduce the generation of incorrect facts.", "questions": "1. In the experimental setup, a bucket is selected first, and then the premature layer is selected later. This approach can save time. However, have you tried the dynamic layer selection with brute force? Is this approach able to achieve better results?\n\n2. For some datasets, e.g, FACTOR-News, the 0-th layer is a good premature layer, as shown in Figure 6. However, for TruthfulQA, the selected layer is closer to the final layer. What do you think the reason is?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea is interesting. Previously, most people tried constrative decoding with a smaller model as the weaker amateur model. In this work, the earlier layer is used as the premature part. It is good to see the other approach.\n\n2. LLaMA family models with different model sizes show large improvements on different tasks. For example, in Table 1, impressive results are shown. Looks good.\n\n3. Clear analysis with different earlier layers and different task is done.", "weaknesses": "1. The baseline for contrastive decoding is not well explored enough. In some papers, for example, \"CONTRASTIVE DECODING IMPROVES REASONING IN LARGE LANGUAGE MODELS\", A smaller model  (1.5B parameters) is as the amateur model. The better results are shown in that paper. For example, contrastive decoding performance on GSM8K with Llama-65B is 56.8. It is even better than the proposed method, DoLa (54.0). \n\n\n2. The results are a little inconsistent. In Table 1, for TruthfulQA (MC1) with LLaMa-33B, we can see that the proposed method (DoLa) have  worse performance than the other two baselines (basic decoding and contrastive decoding). It is better to discuss a little about it.\n\n3. After reading the paper and the following reference paper, it is still a little harder for me to conclude that this approach for contrastive decoding is better than the previous one proposed by Li et al. (2022). \n\n\nReference paper: Sean O’Brien, Mike Lewis. CONTRASTIVE DECODING IMPROVES REASONING IN LARGE LANGUAGE MODELS, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lfLQuhpDLJ", "reviewer": "Reviewer_VLmJ", "summary": "This paper proposes a new decoding scheme for language models that contracts the final layers’ outputs with the middle layers’. At each decoding step, the model produces a distribution over the vocab after every layer; the one that is most different from the final layer is selected, measured by Jensen-Shannon divergence. It then contrasts the final layer’s logits with the selected layer’s, by subtracting the latter from the former before taking a softmax. This method is dubbed DoLA.\n\nDoLA is, in many sense, similar to contrastive decoding [1], but uses the same model’s middle layer instead of a separate smaller model as the “weaker model.” DoLA dynamically selects the middle layer to contrast with at each time step. DoLA has a static variant, where the middle layer is selected based on performance on the validation data and fixed for all instances and time steps. The assumption behind DoLA is that factual knowledge is injected into the model at higher layers, and thus downplaying the lower layers’ decision can help improve factuality. To support this, some empirical (but anecdotal) evidence is presented.\n\nDoLA is evaluated on various downstream tasks, including TruthfulQA, Factor, strQA, and GSM8K. Results show that DoLA, when used with Lllama and MPT, improves the performance on all datasets, and outperforms contrastive decoding on almost all datasets with Llama. Analysis shows that DoLA only incurs minimal overhead in the model’s latency.\n\nOverall, I find this paper an interesting read. I’m leaning negative due to various issues that I raised below. I’m happy to revisit the score if the authors can address my concerns.\n\n[1] https://arxiv.org/abs/2210.15097", "questions": "- [Important] In the example in Figure 2, almost all layers selected by DoLA will be the wording embedding layer. In such cases, assuming $q_0(x_t)$ has a substantial amount of mass on $x_t$, DoLA is essentially discouraging the model from generating the same token as the previous one. Maybe I’m missing something obvious, but I am not entirely convinced by the “factuality happens at higher layers” narrative.\n- [Important] Following the above, have the authors compared to a baseline that always selects the 0-th layer? I won’t be surprised if it achieves strong performance, since it is doing exactly the same thing as DoLA on most generation steps.\n- Figure 2 is nice; I wonder whether the authors have any quantitative results on this.\n- Decoding from every layer seems expensive. I’m surprised by the efficiency results presented in Section 4.4. Can the authors provide more details on, e.g., how is this measured and on what tasks/hardware, and the generation lengths?\n- [Important] Following the above, latency is only one aspect of efficiency and a determining factor of whether DoLA “can be widely applied with negligible cost.” It would be great if the authors can quantify DoLA’s impact on throughput and memory overhead too.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- DoLA is a simple method and can be used with many open-source models out of the box.\n- Strong and consistent improvements in a variety of tasks\n- The finding that the final layer’s distribution is very similar to many middle layers’ is novel and interesting to me\n- Presentation is very clear", "weaknesses": "- The hypothesis behind DoLA that factual knowledge is injected at top layers is not convincing and only weakly supported\n- The evidence in Figure 2 is anecdotal; I wonder whether there are quantitative results to back it up\n- The efficiency results and claims can be further elaborated (see questions below)\n- I don’t see a way to apply DoLA to proprietary models.\n- [Important] I suspect that DoLA will reduce back to contrasting with the 0-th layer in practice (see questions below)\n- It would be great to investigate DoLA’s impact on the language model’s generation quality", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BrgvAI2OhH", "reviewer": "Reviewer_Cc1D", "summary": "This paper proposes to improve the factuality of generative LLM by contrasting the differences in logits of different transformer layers. This is based on the analytic results that when predicting important named entity words, the final layer of LLM tend to be very different from some early layers. Therefore, the authors propose to subtract the logits obtained from early layers from the final layer to reduce hallucination.", "questions": "1. subtracting early layer logits make sense, but this might also be related to residual connection from early layers to final layer. Have you tried just removing the residual connection between the premature layer and final layer? Would that be helpful without too much decrease in inference speed?\n2.  what's the inference speed of your method compared to the baseline?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. the proposed method seems to be quite effective at improving the factuality QA metrics\n2. the paper presents good analysis that motivates the method\n3. the paper also has good ablations of the experimental results that can be helpful for future work.", "weaknesses": "1. the proposed method likely decreases the inference speed, but there is little discussion on how much it slows down the decoding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)