Venue: NAACL 2025

# Entropy-Based Decoding for Retrieval-Augmented Large Language Models

 Zexuan Qiu\({}^{1}\)   Zijing Ou\({}^{2}\)  Bin Wu\({}^{3}\)  Jingjing Li\({}^{1}\)  Aiwei Liu\({}^{4}\) Irwin King\({}^{1}\)

\({}^{1}\)The Chinese University of Hong Kong Imperial College London

\({}^{3}\)University College London Tsinghua University

Correspondence to: zxqiu22@cse.cuhk.edu.hk.

###### Abstract

Augmenting Large Language Models (LLMs) with retrieved external knowledge has proven effective in improving the factual accuracy of generated responses. Despite their success, retrieval-augmented LLMs still face the distractibility issue, where the generated responses are negatively influenced by noise from both external and internal knowledge sources. In this paper, we introduce a novel, training-free decoding method guided by entropy considerations to mitigate this issue. Our approach utilizes entropy-based document-parallel ensemble decoding to prioritize low-entropy distributions from retrieved documents, thereby enhancing the extraction of relevant information of context. Additionally, it incorporates a contrastive decoding mechanism that contrasts the obtained low-entropy ensemble distribution with the high-entropy distribution derived from the model's internal knowledge across layers, which ensures a greater emphasis on reliable external information. Extensive experiments on open-domain question answering datasets demonstrate the superiority of our method.1

Footnote 1: Our code is available at [https://github.com/zekuanqiu/entropy-based-decoding](https://github.com/zekuanqiu/entropy-based-decoding).

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4616–4627

## 1 Introduction

In recent years, Large language models (LLMs) have revolutionized natural language processing, showcasing remarkable performance across various downstream tasks Brown et al. (2020); Ouyang et al. (2022); Touvron et al. (2023). However, they still struggle with hallucination due to the inaccuracy of parametric memory Bubeck et al. (2023) and inherently tend to produce outdated information Kasai et al. (2024). In contrast, explicitly augmenting LLMs with retrieved external knowledge from reliable datastores Lewis et al. (2020); Borgeaud et al. (2022) can enable LLMs to generate content that exhibits less deviation from the truth, and benefit downstream knowledge-intensive tasks Petroni et al. (2020).

Despite the success of retrieval-augmented LLMs, the augmented generation is still sub-optimal due to the _distractibility issue_, where the generated responses are easily negatively affected by noise from both external knowledge and intrinsic model knowledge. As for the input context, LLMs' understanding of context can be _explicitly distracted by irrelevant parts_ within the retrieved context Shi et al. (2023a). A typical illustrative case is the "lost in the middle" distraction phenomenon observed in the synthetic multi-document question-answering scenario Liu et al. (2024); Qiu et al. (2024), where the oracle document containing the correct answer is encircled by numerous retrieved distracting documents. In this scenario, LLMs frequently fail to deliver the correct answer unless the oracle document is strategically placed at the very beginning or end of the context. With regard to intrinsic knowledge, LLMs are easily _implicitly distracted by the parametric knowledge_ acquired during pre-training. This is in conflict with retrieval-augmented generation which is expected to generate responses based on reliable retrieved context. Particularly in the domain of question answering, previous works Longpre et al. (2021); Xie et al. (2023) show that LLMs stubbornly adhere to their built-in knowledge even when it conflicts with external knowledge.

How to eliminate the impact of the above-mentioned distractibility issue, so as to extract useful knowledge from the retrieved context for the input query, is our research focus. Although existing works strive to effectively leverage the retrieved context by directly fine-tuning retrieval-augmented LLMs Lin et al. (2023); Zhang et al. (2024); Li et al. (2024) or incorporating trainable encoder modules Izacard and Grave (2020); Yen et al. (2024), these approaches require additional training, rendering them potentially impractical in

resource-constrained environments. In this paper, we propose a novel decoding method guided by entropy considerations to simultaneously mitigate the impact of noisy information from both the external context and parametric knowledge. The proposed method can be seamlessly integrated into LLMs without requiring additional tuning.

Specifically, to enhance LLMs' ability to extract useful information from multiple retrieved documents, we let LLMs process each retrieved document in parallel and ensemble of the output distributions from each document to determine the next-token distribution, with the ensemble weights adaptively assigned based on the uncertainty of each document-conditioned distribution. At each generation step, documents with lower uncertainty (_i.e._, lower entropy) in the LLM output are given more attention during decoding. Ultimately, we obtain a low-entropy distribution aggregated among documents. Furthermore, to alleviate the potential distraction from parametric knowledge, we refine the next-token distribution by contrasting the obtained low-entropy distribution when feeding the retrieved documents, against the distribution without context. Here, we propose to use the distribution from the layer exhibiting the highest entropy without context for contrast, in order to highlight the proportional changes in token probabilities after introducing external knowledge.

The proposed decoding method shows an impressive performance in the synthetic challenging multi-document scenario Liu et al. (2024) where the negative impact of retrieved distractor documents is emphasized. We further conduct extensive experiments across four LLMs of varying sizes on four diverse open-domain question answering tasks including NQ Kwiatkowski et al. (2019), TriviaQA Joshi et al. (2017), WebQ Berant et al. (2013) and PopQA Mallen et al. (2023). Experimental results confirm the superiority of our methods and validate the effectiveness of each component.

## 2 Methodology

In this work, we will investigate a new decoding approach for retrieval-augmented generation (RAG). Given an input query \(x\), RAG first retrieves top-\(K\) relevant documents \(D:=\{d_{1},d_{2},\ldots,d_{K}\}\) from the knowledge base via a retriever as external evidence, which is then incorporated with the query as the input to large language model parametrized by \(\theta\) for generating a faithful response \(y\).

Despite the integration of external knowledge, the decoding manner does matter to achieve desirable performance in RAG. A common approach, termed as Naive RAG", involves concatenating the query \(x\), the previously generated response \(y_{<t}\), and the retrieved documents \(D\) as the input sequence, resulting in the following decoding method

\[p_{\theta}(y_{t}|D,x,y_{<t})=p(y_{t}|d_{0}\circ\cdots\circ d_{k}\circ x\circ y _{<t}), \tag{1}\]

where \(\circ\) denotes the concatenation operation. Even though this method has demonstrated superior performance, Liu et al. (2024) highlight that it suffers from the "lost in the middle" distraction phenomenon, where LLMs tend to overlook the oracle document due to the distraction from other documents in \(D\), unless the oracle document is placed at the very beginning or the end. Moreover, this simple approach does not consider the potential negative effects of the underlying parametric knowledge of LLMs. In the subsequent sections, we discuss how to mitigate these two issues simultaneously by entropy considerations.

### Entropy-Based Document Ensemble

Instead of naively concatenating the documents \(D\), we propose to alleviate the "loss in the middle" issue using the product-of-experts2 ensemble approach Hinton (2002). Specifically, we model the log probability of the next-token distribution as

Footnote 2: Empirically, we find that using product-of-experts and mixture-of-experts methods yield similar performance.

\[\log p_{\theta}(y_{t}|D)\propto\!\sum^{K}_{j=1}\!w_{j,t}\log p_{\theta}(y_{t} |d_{j}\circ x\circ y_{<t}), \tag{2}\]

where \(\sum_{j}w_{j,t}=1,\forall t\), \(p_{\theta}(y_{t}|D,x,y_{<t})\) is denoted as \(p_{\theta}(y_{t}|D)\) for short and \(w_{j,t}\) denotes the weight of the \(j\)-th document on generating the token at the \(t\)-th time step. In Eq. (2), each document in \(D\) is concatenated with the query and the previously generated response. This combined input is then individually fed into the LLM. The output logit scores are subsequently averaged using the weights \(w_{j,t}\). This ensemble approach, which leverages parallel decoding, helps mitigate position bias and provides a more effective means of utilizing the retrieved documents.

There are multiple choices to compute the weights. A straightforward option is to use uniform weighting, _i.e._, \(w_{j,t}=\frac{1}{|D|}\). However, this method may fail to effectively extract valuable information when irrelevant documents are included 

among the top-\(K\) retrieved documents. Another option is to utilize a similarity score \(s(d_{j},x)\) between the query and the retrieved document, _e.g._, BM25, as the time-independent ensemble weights \(w_{j,t}\propto s(d_{j},x)\)Lewis et al. (2020). However, this retriever-based scoring approach may hinder the LLM's ability to extract relevant information when the retrieved documents contain numerous distractors, drastically reducing the factual accuracy of the responses as illustrated in SS 4.1.

We posit that the uncertainty present in the next-token distribution inherently serves as a reliable indicator of the informativeness of the retrieved documents. Similar concepts have been employed in previous work to reduce hallucinations in LLMs Van der Poel et al. (2022); Varshney et al. (2023). Consequently, we propose using an entropy-based score \(w^{\text{H}}_{j,t}\) as the preference weight for each document at each decoding step:

\[\begin{split} w^{\text{H}}_{j,t}&=\frac{\exp^{(- H_{j,t})/\tau}}{\sum_{d_{k}\in D}\exp^{(-H_{j,t})/\tau}},\\ H_{j,t}&=-\sum_{y_{t}\in\mathcal{V}}p_{\theta}( y_{t}|d_{j})\log p_{\theta}(y_{t}|d_{j}),\end{split} \tag{3}\]

where \(p_{\theta}(y_{t}|d_{j})\) denotes \(p_{\theta}(y_{t}|d_{j}\circ x\circ y_{<t})\) for short, \(\mathcal{V}\) represents the vocabulary set and \(\tau\) is a hyper-parameter controlling the concentration level of distributions. The motivation behind Eq. (3) is that the LLM can autonomously evaluate the significance of each document during the generation process. Intuitively, it implies that those documentationed distributions with lower uncertainty will be assigned higher weights. Such a time-dependent approach can effectively capture useful information from the retrieved documents at each generation step, thereby influencing the generation process more significantly. We refer to this method as **LeEns** (**L**ow-**entropy **E**nsemble).

### Entropy-Based Contrastive Decoding

While **LeEns** can effectively help LLMs discern valuable evidence from external knowledge, the parametric knowledge of LLMs embedded during the pre-training phase might affect the answer generation, especially when these two types of knowledge conflict Longpre et al. (2021); Xie et al. (2023). In this section, we propose to address this issue via entropy-based contrastive decoding.

Contrastive Decoding with PMI.Inspired by the success in contrastive decoding to mitigate hallucination of LLMs Shi et al. (2023b); Li et al. (2022), we adjust the logit score \(z_{t}\) for the generated token \(y_{t}\in\mathcal{V}\) at the \(t\)-th time step by incorporating the pointwise mutual information (PMI) between \(y_{t}\) and the document set \(D\), given the query \(x\):

\[\begin{split} z_{t}=\log p^{\text{H}}_{\theta}(y_{t}|D)+\overbrace {\beta\log\frac{p^{\text{HI}}_{\theta}(y_{t}|D)}{p_{\theta}(y_{t}|x,y_{<t})}} ^{\text{PMI}}\\ =&(1+\beta)\log p^{\text{H}}_{\theta}(y_{t}|D)- \beta\log p_{\theta}(y_{t}|x,y_{<t}),\end{split} \tag{4}\]

where \(\beta\) is a positive coefficient to control the contrast intensity, and \(p^{\text{H}}_{\theta}(y_{t}|D)\) denotes the previously proposed entropy-based document ensemble distribution. Intuitively, PMI serves as a measurement of information gains. It is evident that the model tends to generate tokens with a high probability of \(p^{\text{H}}_{\theta}(y_{t}|D)\) and a low probability of \(p_{\theta}(y_{t}|x,y_{<t})\). These tokens provide greater information gain for the next token generation. Consequently, incorporating PMI can enhance the model's reliance on external knowledge.

Layer-wise Contrast with High Entropy.To perform contrastive decoding, it is necessary to compute \(p_{\theta}(y_{t}|x,y_{<t})\). This can be achieved by taking the hidden states from the last layer of LLMs and passing them through the classification head.

Figure 1: Overview of the decoding process of CLeHe.

 

However, the distribution derived from the last layer may exhibit overconfidence, characterized by extremely low probabilities for most words and disproportionately high probabilities for a few. Such overconfidence can erroneously amplify external knowledge when conducting contrasting, potentially leading to false positive failures, as illustrated in Figure 6 in the Appendix.

To address this issue, we propose selecting the layer that contains the most "ambiguous" parametric knowledge among the layers as a proper reference for contrast. This allows the model to more effectively leverage external knowledge, reducing overconfidence and improving the accuracy of the generated outputs. Specifically, for a LLM consisting of a total of \(L\) layers, we denote the probability for \(y_{t}\in\mathcal{V}\) in the \(l\)-th layer as:

\[p^{l}_{\theta}(y_{t}|x,y_{<t})=\mathrm{softmax}(\boldsymbol{W}_{\text{LM}}h^{ l}_{t-1}), \tag{5}\]

where \(h^{l}_{t-1}\) denotes the hidden state for layer \(l\) out of \(L\), and \(\boldsymbol{W}_{\text{LM}}\) denotes the linear classification head in the LLM. At each decoding step, we dynamically select the layer with the maximum uncertainty for contrast:

\[l^{*} =\arg\max_{l\in\mathcal{L}}H^{l}_{t}, \tag{6}\] \[H^{l}_{t} =-\sum_{y_{t}\in\mathcal{V}}p^{l}_{\theta}(y_{t}|x,y_{<t})\log p^{ l}_{\theta}(y_{t}|x,y_{<t}),\]

where \(\mathcal{L}\) is the set of candidate layers, which are set as the last few layers of LLMs practically to ensure that each of them contains certain plausible information. Combining Eq. (3) and Eq. (6), the adjusted next-token distribution is formulated as:

\[y_{t} \sim\mathrm{softmax}\Big{[}(1+\beta)\!\!\sum_{d_{j}\in D}w^{ \textsc{H}}_{j,t}\log p_{\theta}(y_{t}|d_{j},\!x,\!y_{<t})\] \[\quad-\beta\log p^{l^{*}}_{\theta}(y_{t}|x,y_{<t})\Big{]}. \tag{7}\]

Here, \(\beta\) represents the amplification intensity of external knowledge. When \(\beta=0\), Eq. (7) degenerates to the proposed LeEns in SS 2.1. The ultimate distribution in Eq. (7) can be interpreted as a two-stage ensemble process. Firstly, it ensembles the retrieved documents with uncertainty to generate a low-entropy distribution that more effectively captures the external knowledge within these documents. Secondly, it performs a contrastive ensemble by differentiating the logits of this low-entropy distribution from the high-entropy distribution of parametric knowledge selected across different layers, thereby prioritizing factual information from external sources. In this regard, we term the method in Eq. (7) as **CLeHe** (Contrasting Low-entropy distribution with High-entropy distribution). Figure 1 illustrates the overall pipeline of the proposed CLeHe.

## 3 Related Works

Retrieval-Augmented Language Models.Enhancing large language models (LLMs) with information retrieved from external knowledge bases has proven effective for various knowledge-intensive tasks. Initially, mainstream research in retrieval-augmented language models (RALM) focused on leveraging retrieved knowledge during the pre-training phase of LLMs Guu et al. (2020); Izacard et al. (2023); Borgeaud et al. (2022). To mitigate the computational costs, some studies have concentrated on lightweight fine-tuning methods to integrate retrieval capabilities into LLMs Lewis et al. (2020); Lin et al. (2023); Zhang et al. (2024). Notably, models like FiD Izacard and Grave (2020) and CEPE Yen et al. (2024) perform parallel encoding of multiple retrieved documents using a fine-tuned encoder, enabling decoder-only LLMs to more effectively capture and utilize external knowledge. Another approach leverages the in-context learning abilities of LLMs to incorporate external knowledge in a training-free manner Ram et al. (2023); Shi et al. (2023c). The work most closely related to ours is RePlugShi et al. (2023c), which utilizes the RAG-token model Lewis et al. (2020) to perform parallel retrieval augmentation based on retrieval scores. However, we empirically demonstrate that focusing on the inherent uncertainty within the LLM's output distribution, rather than relying solely on pre-existing retrieval scores, can significantly improve the factual accuracy of content generated from retrieved documents.

Contrastive Decoding.The idea of contrastive decoding (CD) has been previously applied in controllable text generation to produce non-toxic by DE Experts Liu et al. (2021). Later, Li et al. (2022) formalized CD as a method to enhance open-ended text generation without any additional training by maximizing the difference in log probabilities between an expert LLM and an amateur LLM. This approach has demonstrated strong performance in various domains, including reasoning O'Brien and Lewis (2023) and neural machine translation Waldendorf et al. (2024). CD 

can also be interpreted as maximizing pointwise mutual information (PMI), which has proven effective in other scenarios. For instance, Li et al. (2016) uses a training objective that maximizes PMI to generate more diverse conversational responses, while CAD Shi et al. (2023b) employs a PMI-adjusting distribution to resolve the knowledge conflict. Chuang et al. (2023) proposes a decoding strategy that contrasts different layers of the same LLM to more effectively highlight factual knowledge. Similar principles are also applied in visual LLMs, where Leng et al. (2023) mitigates object hallucination by contrasting distributions derived from original and distorted visual inputs. Alternatively, our proposed CLeHe leverages layer-wise entropy-based contrastive decoding to prioritize external knowledge over the parametric knowledge inherent in the LLM itself.

## 4 Experiments

Baselines.We mainly compare three training-free baselines, two of which are ensemble-based methods. Naive: this method concatenates all retrieved documents directly along with the question to form the prompt for LLMs; RePlugShi et al. (2023c): it utilizes a normalized retriever weight to ensemble during the decoding process; AvgEns : it follows the formulation in Eq.(2) and assigns the same weight to each document during each generation step. In SS 4.2.2, we also incorporate two contrastive decoding methods (_i.e._, CAD Shi et al., 2023b) and DoLaChuang et al. (2023)) into our setting and compare with them to validate the effectiveness of our method. Experiments are conducted on 4 LLMs: Llama-2-7B, Llama-2-13B, Mistral-7B-v0.1, and Llama3-8B. Among them, Llama-3-8B supports an 8K context length, while the other three LLMs support a 4K context length. Few evaluation samples exceed 4k in context length, and they are removed to ensure the validity of the assessment.

Implementation Details.Our method introduces two hyperparameters: \(\tau\) to control the relative importance of different documents during decoding; and \(\beta\) balancing contextual and parametric knowledge. We extract a subset from the WebQ training set for validation to determine the hyperparameter value for each LLM. Ultimately, \(\tau\) is set as 0.25 for Llama-3-8B and as 0.1 for the other three models. For \(\beta\), 5.0 is chosen for Llama-2-7B, while the other models are assigned a value of 0.25. During the inference, greedy decoding is utilized for reproducibility. When looking for the layer with the highest entropy, we focus our search exclusively on the candidate layers. In our preliminary experiments, this approach improves computational efficiency and slightly enhances model performance. For Llama-2-7B, Mistral-7B-v0.1, and Llama-3-8B with 32 hidden layers, the candidate layers are set to \(\{17,\ldots,32\}\), and only even-numbered layers will be searched. For Llama-2-13B with 40 hidden layers, the candidate layers are set to \(\{31,\ldots,40\}\). All experiments are conducted on a single A100 80GB GPU.

### Analyzing the Distraction Phenomenon in Retrieved Context

We are particularly interested in a challenging QA scenario proposed by Liu et al. (2024), in which the oracle document is surrounded by numerous semantically similar distractor documents. This configuration challenges the efficacy of the Naive LLM-RAG, preventing it from accurately deriving answers based on contextual information, thereby leading to the "Lost in the middle" phenomenon. Following Liu et al. (2024), given a query from NaturalQuestions-Open Kwiatkowski et al. (2019), we select a Wikipedia paragraph containing the answer from the NaturalQuestions annotations as the oracle document. Then, Contriever Gautier et al. (2022) is employed to extract K-1 additional paragraphs from the Wikipedia corpus that are highly relevant to the query yet do not include the ground truth answer, functioning as distractor documents. The query, the oracle document, and K-1 distractor documents are subsequently processed by the LLM to generate an answer.

From Figure 2, it is evident that the performance of the Naive method which concatenates all documents to compose the context, is highly sensitive to the placement of the oracle document within the retrieved documents. Among the four evaluated LLMs, the performance of the Naive method significantly deteriorates when the oracle document is neither at the very beginning nor at the end. In contrast, since the proposed LeEns processes each document in parallel during decoding, its performance is naturally independent of the position of the oracle document. In almost all positions, LeEns substantially surpasses the performance of the Naive method. Notably, in this challenging scenario, RePlug which ensemble documents' distributions based on retriever weights perform 

exceedingly poorly, achieving results merely on par with AvgEns . Based on this observation, we further conduct the weight analysis in Figure 3. As depicted in Figure 2(a), in only approximately 57% of instances, Contriever identifies the oracle document as more similar to the query than the distractor documents. Figure 2(b) shows the distribution of entropy differences for the first token when conditioned on oracle documents versus distractor documents. It indicates that the entropy is generally lower when the response to a query is based on the oracle document rather than the distractor documents. Moreover, the proposed CLEHe derived from LeEns further enhances the accuracy of responses in LLMs, particularly evident in the Llama2-7B and Llama2-13B. This indicates that contrasting low-entropy distributions of contextual knowledge with high-entropy distributions of parameterized knowledge can further strengthen LLMs' understanding of context.

Open-Domain Question Answering Datasets and Metrics.We evaluate our proposed method using four open-domain QA datasets. Natural Questions Kwiatkowski et al. (2019), TriviaQA Joshi et al. (2017), WebQ Berant et al. (2013) and PopQA Mallen et al. (2023). Natural Questions includes real anonymized queries from Google's search engine. We utilize a filtered test set Lee et al. (2019) of 3,610 samples with answers limited to no more than five tokens. TriviaQA comprises trivia question-answer pairs that were scraped from the web. We evaluate its development set containing 7,993 samples. WebQ consists of questions generated through the Google Suggest API, with answers that are entities in Freebase. We use its test set of 2,032 samples for evaluation. PopQA is a novel entity-centric open-domain QA dataset that spans a wide range of entity popularity, emphasizing long-tail knowledge. We utilize its test set which includes 14,267 samples for evaluation. For each dataset, we retain only the questions and their corresponding answers. DPR Karpukhin et al. (2020) is employed to retrieve the top-k passages from the Wikipedia corpus Dec. 20, 2018) via as evidence documents for each question. Specifically, we report the per

Figure 3: (a) The distribution of the similarity difference between the query and the oracle document versus the query and distractor documents. (b) The distribution of the difference in entropy of the first token generated by LLMs when given the oracle document versus distractor documents.

Figure 2: Impact of positioning the oracle document on multi-document question answering performance. A 10-document context typically uses less than 2K tokens; a 20-document context usually uses less than 4K tokens.

 

formance of different decoding methods when retrieving the top-5, top-10, and top-20 documents. Following Liu et al. (2024), exact match accuracy is utilized for performance evaluation.

#### 4.2.1 Overall Performance

Table 1 presents the overall performance comparison between our proposed method and existing baselines on public benchmark datasets. The results show that when compared to the Naive method, our entropy-ensemble-based LeEns demonstrates significant average performance improvements across various LLMs, indicating its superior ability to extract useful information from the context. Moreover, LeEns outperforms RePlug and AvgEns in almost all settings, indicating that using the uncertainty of LLM output distributions for document scoring more effectively facilitates generating answers than static retriever similarity and unweighted averaging. Comparing LeEns with CLeHe, we observe that further contrasting the ensemble-based low-entropy contextual distribution with the high-entropy distribution of the parametric knowledge leads to performance improvements, particularly noticeable in Llama2-7B and Llama2-13B. These observations substantiate that the proposed entropy-based decoding mechanism markedly augments the extraction and utilization of contextual information. Further, on Mistral-7b-v0.1 and Llama-3-8B, CLEeHe performs similarly to LeEns, indicating no significant enhancement from the contrastive ensemble. We speculate that these two models are less distracted by parametric knowledge when generating answers.

#### 4.2.2 Ablation Study

Within the contrastive decoding framework, we investigate the compositional effects on performances by combining different modeling techniques for external and parametric knowledge. To extract knowledge from retrieved external documents, we explore two modeling approaches.: Naive RAG and our entropy-based document ensemble modeling (LeEns). Additionally, we explore three layer-based strategies to derive parametric knowledge: (i) Last-Layer strategy. It defines parametric knowledge using the distribution from the last layer of LLMs when without retrieved

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline
**Dataset** & \multicolumn{2}{c}{**NQ**} & \multicolumn{2}{c}{**TQA**} & \multicolumn{2}{c}{**WebQ**} & \multicolumn{2}{c}{**PopQA**} & \multirow{2}{*}{**Gain**} \\ \cline{2-9} \# of Docs & 5 & 10 & 20 & 5 & 10 & 20 & 5 & 10 & 20 & 5 & 10 & 20 & \\ \hline \multicolumn{9}{l}{Llama2-2TB} \\ Naive & 19.56 & 26.76 & 26.37 & 52.53 & 59.05 & 66.00 & 16.04 & 18.06 & 18.35 & 25.64 & 25.50 & 26.10 & 0.00 \\  & 14.93 & 13.51 & 12.78 & 51.04 & 49.84 & 48.87 & 12.50 & 12.06 & 11.35 & 14.03 & 12.49 & 11.74 & -9.40 \\  & RePlug & 23.85 & 23.87 & 23.92 & 55.56 & 55.37 & 55.31 & 16.63 & 16.39 & 16.77 & 23.85 & 23.80 & 23.91 & -1.56 \\  & LeEns & 25.48 & 25.79 & 25.90 & 61.87 & 62.72 & 63.16 & 20.28 & 21.65 & 21.66 & 27.84 & 26.57 & 25.07 & +2.50 \\  & CleHe & **37.62** & **36.29** & **35.48** & **69.56** & **69.92** & **69.72** & **36.12** & **36.22** & **35.77** & **32.12** & **31.11** & **28.89** & **+11.74** \\ \hline \multicolumn{9}{l}{Llama2-13B} \\ Naive & **37.98** & **39.67** & 29.07 & 66.82 & 68.15 & 69.66 & 29.83 & 37.02 & 27.58 & **34.77** & **35.14** & 31.62 & 0.00 \\  & AvgEns & 23.52 & 20.19 & 18.71 & 66.92 & 65.48 & 63.86 & 25.89 & 24.16 & 22.64 & 24.28 & 22.58 & 22.02 & -8.92 \\  & RePlug & 34.12 & 33.82 & 34.01 & 67.83 & 67.77 & 67.60 & 31.25 & 31.30 & 30.88 & 30.79 & 30.95 & 31.00 & -1.33 \\  & LeEns & 36.54 & 34.65 & 33.63 & 71.87 & 72.07 & 72.16 & 36.07 & 35.63 & 35.42 & 34.63 & 33.31 & 31.72 & +1.70 \\  & CleHe & 37.31 & 36.01 & **34.95** & **72.24** & **72.82** & **72.55** & **38.19** & **37.45** & **36.66** & 34.22 & 33.16 & **31.79** & **+3.18** \\ \hline \multicolumn{9}{l}{Mistral-7B-v0.1} \\ Naive & 46.20 & 44.43 & 42.20 & 73.53 & 70.31 & 73.89 & 47.69 & 45.03 & 40.61 & 40.23 & 37.72 & 38.94 & 0.00 \\  & AvgEns & 40.91 & 39.36 & 38.22 & 76.28 & 75.73 & 74.91 & 47.98 & 48.13 & 47.83 & 37.11 & 34.61 & 33.44 & -0.52 \\  & RePlug & 44.35 & 44.44 & 44.58 & 74.62 & 74.80 & 74.60 & 47.59 & 47.49 & 47.21 & 37.76 & 37.75 & 37.78 & +1.02 \\ \hline \multicolumn{9}{l}{LleEns} & **46.40** & **46.45** & **44.65** & **78.26** & **78.97** & **79.25** & **49.21** & 49.70 & 50.32 & **42.12** & 43.31 & 43.87 & **+4.32** \\  & CleHe & 46.32 & 46.07 & 44.64 & 78.19 & 78.88 & 79.14 & 49.06 & **49.76** & **50.37** & 42.11 & **43.34** & **43.89** & +4.25 \\ \hline \multicolumn{9}{l}{Llama-3-8B} \\ Naive & **52.35** & **51.69** & **52.33** & 79.92 & 81.11 & 82.10 & 50.49 & 50.15 & 50.12 & 40.92 & 42.24 & 42.95 & 0.00 \\  & AvgEns & 47.12 & 45.70 & 44.51 & 81.31 & 80.73 & 79.78 & 51.82 & 51.24 & 51.01 & 38.95 & 36.49 & 35.07 & -2.72 \\  & RePlug & 50.39 & 50.33 & 50.50 & 79.07 & 79.16 & 78.76 & 50.20 & 50.79 & 50.27 & 39.08 & 39.24 & 39.04 & -1.63 \\  & LeEs & 51.74 & 50.53 & 49.47 & **81.80** & 82.68 & 83.02 & **52.17** & 50.98 & 51.80 & 43.63 & **44.92** & **45.62** & **+1.00** \\  & CleHe & 52.02 & 50.78 & 49.67 & 81.78 & **82.84** & **83.14** & 51.67 & **51.62** & **52.39** & **43.86** & 44.84 & 45.57 & **+1.17** \\ \hline \end{tabular}
\end{table}
Table 1: Performance (%) comparison of different ensemble-based methods on benchmark datasets. “Gain” refers to the average absolute improvement (%) across all datasets and different numbers of retrieved documents when compared to the naive baseline.

 

context. CAD [20] utilizes this strategy, _i.e._, contrasting the distribution derived from Naive RAG against the last-layer context-free distribution. (ii) JSD-based strategy. It first calculates the Jensen-Shannon Divergence (JSD) between the RAG-derived distribution and the distribution of each layer when without retrieved documents, then selects the layer with the highest JSD for contrast. (iii) Our proposed entropy-based strategy. It directly selects the layer with the highest entropy as the proxy of intrinsic knowledge. As shown in Table 2, compared to the other two layer selection strategies, the proposed entropy-based strategy consistently and significantly enhances model performance in both external knowledge modeling ways of Naive RAG and our LeEns. This improvement is particularly notable in Llama-2-7B. The last-layer strategy markedly impairs the performance of some LLMs such as Mistral-7B-v0.1. Moreover, in our setting, the JSD-based strategy for contrast is found unstable and results in severe performance degradation in Llama-2-13B. Appendix A.2 displays the average entropy of tokens from various layers, indicating that higher entropy layers often yield better performance. Additionally, Appendix A.1 details the hyper-parameter needed to replicate Table 2.

#### 4.2.3 Hyper-Parameter and Latency Analysis

We study the influence of the introduced hyperparameters: \(\tau\) and \(\beta\). As shown in Figure 4, a small value of \(\tau\) (_e.g._, 0.1 or 0.25) typically results in better performance; as \(\tau\) increases, the performance gradually declines. Ideally, when \(\tau\rightarrow\infty\), the performance of the proposed LeEns will match that of AvgEns. Regarding \(\beta\), it's observed that for Llama-2-7B, a high \(\beta\) (e.g., 5) enables it to effectively contrast the differences between external and parametric knowledge for improved performance. For other evaluated LLMs, we suggest setting it to a small value, saying [0.25, 0.5].

As for decoding latency, Table 3 shows that compared to the Naive method, our LeEns and CLEHE increase the decoding time by factors of less than 1.18, indicating that they can be applied at a reasonable cost. Appendix A.5 shows the latency of Llama-2-13B that exhibits a similar cost trend.

## 5 Conclusions

In this paper, we proposed a novel decoding method that is guided by entropy considerations to mitigate the distractibility issue from both external retrieved documents and parametric knowledge. First, we conducted parallel retrieval augmentation with entropy-based ensemble weight to obtain the low-entropy distribution of context. Furthermore, we contrasted this distribution against the highest-entropy distribution among layers when without context to amplify the external knowledge preserved in context. Extensive experiments showed

Figure 4: Hyper-parameter analysis using 1K evaluation samples of NQ under the top-5 document setting.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{Llama-2-7B} & \multicolumn{3}{c}{Llama-2-13B} & \multicolumn{3}{c}{Mistral-7B-v0.1} & \multicolumn{3}{c}{Llama-3-8B} \\ \cline{2-16}  & NQ & TQA & WebQ & NQ & TQA & WebQ & NQ & TQA & WebQ & NQ & TQA & WebQ \\ \hline Naive & 19.56 & 52.53 & 16.04 & 37.98 & 66.82 & 29.83 & 46.20 & 73.53 & **47.69** & **52.35** & 79.92 & **50.49** \\ w/ ISD (DoLa) & 38.72 & 68.15 & 35.38 & 27.34 & 45.15 & 21.90 & 46.20 & 74.42 & 46.80 & 52.30 & 81.36 & 49.54 \\ w/ Last\_Layer (CAD) & 38.92 & 65.92 & 30.77 & **41.52** & **68.96** & **33.76** & 44.32 & 70.27 & 41.49 & 51.80 & 79.01 & 46.26 \\
**w/ Entropy** & **41.36** & **70.32** & **37.30** & 39.91 & 67.38 & 32.73 & **46.30** & **74.59** & 47.05 & 52.28 & **81.60** & 49.61 \\ \hline
**LeEns** & 25.48 & 61.87 & 20.28 & 36.54 & 71.87 & 36.07 & **46.40** & 78.26 & 49.21 & 51.74 & **81.80** & **52.17** \\ w/ ISD (DaLo) & 35.84 & 67.11 & 33.75 & 17.22 & 35.17 & 11.02 & 46.29 & 78.19 & 49.16 & 52.04 & 81.73 & 51.82 \\ w/ Last\_Layer & 30.74 & 62.85 & 23.08 & **38.19** & 71.64 & **38.09** & 45.57 & 75.54 & 48.12 & **52.60** & 81.36 & 52.01 \\
**w/ Entropy (CLHe)** & **37.62** & **69.56** & **36.12** & 37.31 & **72.24** & **38.19** & 46.32 & **78.26** & **49.76** & 52.02 & 81.78 & 51.62 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance on combining different external and parametric knowledge modeling methods. Experiments are conducted under the top-5 document setting.

 

the proposed method's effectiveness in retrieval-augmented open-domain question answering.

## 6 Limitations

One limitation of our study is that we only validated the effectiveness of our method on question answering datasets, without testing it on other knowledge-intensive tasks such as fact verification. Extending the method proposed in this paper to other retrieval-augmented scenarios will be a future research direction. Additionally, due to computational power constraints, we only tested the effectiveness of the proposed method on models with fewer than 13B parameters. However, whether the method proposed in this paper is applicable to LLMs with more parameters (_e.g._, 70B or more) remains to be explored in future research.

## Acknowledgements

The work described in this paper was partially supported by Laboratory for AI-Powered Financial Technologies, InnoHK initiative and The Government of the HKSAR. The work described in this paper was also partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 2410072, RGC R1015-23).

## References

* J. Berant, A. Chou, R. Frostig, and P. Liang (2013)Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1533-1544. Cited by: SS1.
* S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J. Lespiau, B. Damoc, A. Clark, et al. (2022)Improving language models by retrieving from trilions of tokens. In International conference on machine learning, pp. 2206-2240. Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakanton, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 0037-947X, Link, Document Cited by: SS1.
* S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. (2023)Sparks of artificial general intelligence: early experiments with gpt-4. arXiv preprint arXiv:2303.12712. Cited by: SS1.
* Y. Chuang, Y. Xie, H. Luo, Y. Kim, J. R. Glass, and P. He (2023)Dola: decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, Cited by: SS1.
* I. Gautier, C. Mathilde, H. Lucas, R. Sebastian, W. Piotr, J. Armand, and G. Edouard (2022)Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research. Cited by: SS1.
* K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang (2020)Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. Cited by: SS1.
* G. E. Hinton (2002)Training products of experts by minimizing contrastive divergence. Neural computation14 (8), pp. 1771-1800. Cited by: SS1.
* G. Izacard and E. Grave (2020)Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282. Cited by: SS1.
* G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave (2023)Atlas: few-shot learning with retrieval augmented language models. Journal of Machine Learning Research24 (251), pp. 1-43. Cited by: SS1.
* M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017)Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. ACL (1), pp. 1601-1611. Note: Association for Computational Linguistics Cited by: SS1.
* V. Karpukhin, B. Oguz, S. Min, P. S. H. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020)Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6781. Note: Association for Computational Linguistics Cited by: SS1.
* J. Kasai, K. Sakaguchi, R. Le Bras, A. Asai, X. Yu, D. Radev, N. A. Smith, Y. Choi, K. Inui, et al. (2024)Realtime qa: what's the answer right now?. Advances in Neural Information Processing Systems36. Cited by: SS1.
* T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. (2019)Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics7, pp. 453-466. Cited by: SS1.
* K. Lee, M. Chang, and K. Toutanova (2019)Latent retrieval for weakly supervised open domain question answering. In ACL (1), pp. 6086-6096. Note: Association for Computational Linguistics Cited by: SS1.

 

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2023. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. arXiv preprint arXiv:2311.16922.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.
* Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In _HLT-NAACL_, pages 110-119. The Association for Computational Linguistics.
* Li et al. (2024) Muzhi Li, Cehao Yang, Chengjin Xu, Xuhui Jiang, Yiyan Qi, Jian Guo, Ho-fung Leung, and Irwin King. 2024. Retrieval, reasoning, re-ranking: A context-enriched framework for knowledge graph completion. _arXiv preprint arXiv:2411.08165_.
* Li et al. (2022) Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022. Contrastive decoding: Open-ended text generation as optimization. _arXiv preprint arXiv:2210.15097_.
* Lin et al. (2023) Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. 2023. Ra-dit: Retrieval-augmented dual instruction tuning. _arXiv preprint arXiv:2310.01352_.
* Liu et al. (2021) Alisa Liu, Maarten Sap, Kiming Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In _ACLI/ICNLP (1)_, pages 6691-6706. Association for Computational Linguistics.
* Liu et al. (2024) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Parangjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. _Transactions of the Association for Computational Linguistics_, 12:157-173.
* Longpre et al. (2021) Shayne Longpre, Kartik Perisella, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. _arXiv preprint arXiv:2109.05052_.
* Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In _ACL (1)_, pages 9802-9822. Association for Computational Linguistics.
* O'Brien and Lewis (2023) Sean O'Brien and Mike Lewis. 2023. Contrastive decoding improves reasoning in large language models. _arXiv preprint arXiv:2309.09117_.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhnii Agarwal, Katarina Alama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in neural information processing systems_, 35:27730-27744.
* Petroni et al. (2020) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Vazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2020. Kill: a benchmark for knowledge intensive language tasks. _arXiv preprint arXiv:2009.02252_.
* Qiu et al. (2024) Zexuan Qiu, Jingjing Li, Shijue Huang, Wanjun Zhong, and Irwin King. 2024. Clongeval: A chinese benchmark for evaluating long-context large language models. _arXiv preprint arXiv:2403.03514_.
* Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. _Transactions of the Association for Computational Linguistics_, 11:1316-1331.
* Shi et al. (2023a) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Scharli, and Denny Zhou. 2023a. Large language models can be easily distracted by irrelevant context. In _International Conference on Machine Learning_, pages 31210-31227. PMLR.
* Shi et al. (2023b) Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau Yih. 2023b. Trusting your evidence: Hallucinate less with context-aware decoding. _arXiv preprint arXiv:2305.14739_.
* Shi et al. (2023c) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023c. Replug: Retrieval-augmented black-box language models. _arXiv preprint arXiv:2301.12652_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Van der Poel et al. (2022) Liam Van der Poel, Ryan Cotterell, and Clara Meister. 2022. Mutual information alleviates hallucinations in abstractive summarization. _arXiv preprint arXiv:2210.13210_.
* Varshney et al. (2023) Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. _arXiv preprint arXiv:2307.03987_.
* Walodendorf et al. (2024) Jonas Walodendorf, Barry Hadow, and Alexandra Birch. 2024. Contrastive decoding reduces hallucinations in large multilingual machine translation models. In _Proceedings of the 18th Conference of the European 

Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2526-2539. Cited by: SS4.
* J. Xie, K. Zhang, J. Chen, R. Lou, and Y. Su (2023)Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: SS1, SS2.
* H. Yen, T. Gao, and D. Chen (2024)Long-context language modeling with parallel context encoding. arXiv preprint arXiv:2402.16617. Cited by: SS1, SS2.
* T. Zhang, S. G. Patil, N. Jain, S. Shen, M. Zaharia, I. Stoica, and J. E. Gonzalez (2024)Raft: adapting language model to domain specific rag. arXiv preprint arXiv:2403.10131. Cited by: SS1, SS2.

## Appendix A Appendix

### Additional Implementation Details

As shown in Table 4, for both the dynamic JSD-based selection strategy and the proposed entropy-based strategy, we set the \(\beta\) parameter to 5.0 for llama-2-7B and \(\beta\) to 0.25 for other LLMs. This configuration ensures that contrastive decoding achieves superior performance across different LLMs. For the last-layer strategy, we found that setting a higher beta significantly severely reduces the performance of LLMs. Therefore, for this strategy, we set \(\beta\) to 0.25 for all LLMs.

### Relations between Entropy and Performance

Figure 5 shows the average entropy values of tokens generated on the WebQ dataset by Llama2-7B and Llama2-13B when using different layers for contrast. For Llama2-7B, the entropy initially shows a decreasing trend and later an increasing trend among the tested layers, which correlates somewhat with the overall performance trend. For Llama2-13B, it was found that both the highest performance layer and the layer with the highest entropy are the last layer. This might explain why the performance of our entropy maximization-based layer selection is overall consistent with the performance of the last layer.

### Illustration of the False Positive Case

Figure 6 illustrates a false positive scenario. In this example, the retrieval-augmented model assigns high probabilities to the words "Washington", "New York", and "Columbia" as candidate positives. However, in the low-entropy output of a specific layer (typically the last layer) without context, the probability assigned to "Columbia" is notably low. If contrastive decoding is applied, it would mistakenly increase "Columbia's" probability, leading to an incorrect prediction.

### Prompt Format

```
Write ahigh-qualityanswerforthegivenquestionusingonlytheprovidedsearchresults. Document[1](Title:...)...Document[2](Title:...)...Document[3](Title:...)...... .. Question:{Question} Answer:
``` Write ahigh-qualityanswerforthegivenquestionusingonlytheprovidedsearchresults. Document(Title:...)... Question:{Question} Answer: ```
Write ahigh-qualityanswerforthegivenquestionusingonlytheprovidedsearchresults. Document(Title:...)... Question:{Question} Answer:
```

Table 5: The Prompt format for NaiveRAG (Top); and the prompt format for the proposed CLeHe (Bottom) results.

Footnote 8: [https://github.com/facebook/caffe/Caffe_Towards](https://github.com/facebook/caffe/Caffe_Towards) 

As shown in Table 6, when our methods are applied to the 13B model, the increase in decoding time is capped at factors of 1.22. This increase is considered acceptable given the superior performance our methods deliver.

Figure 5: Average entropy of generated tokens and performance for each layer.

Figure 6: Illustration of the false positive case.