Venue: NAACL 2025

# Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation

 Tianyu Liu   Jiru Qi   Paul He \({}^{\dagger}\)

&Arianna Bisazza     Mrinmaya Sachan    Ryan Cotterell

ETH Zurich   CLCG, University of Groningen   University of Toronto

{tianyu.liu, mrinmaya.sachan,ryan.cotterell}@inf.ethz.ch

###### Abstract

Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order in which the retrieved documents are presented to the model when solving tasks such as question answering (QA). However, there is no method to date that exploits this phenomenon to improve generation. To fill this gap, in this study, we show that the pointwise mutual information between a context and a question is an effective gauge for language model performance. Importantly, this gauge does not depend on knowing the answer to the question _a priori_. Through experiments on two question-answering datasets using a variety of large language models, we find evidence for an empirical correlation between answer accuracy and pointwise mutual information. Additionally, we propose two methods that use the pointwise mutual information between a document and a question as a gauge for selecting and constructing prompts that lead to better performance, whose effectiveness we demonstrate through experimentation.1

Footnote 1: Our code is available at [https://github.com/lyutyuh/poptimizer](https://github.com/lyutyuh/poptimizer).

+
Footnote †: leftmargin=*]Primary definitionDefinition

## 1 Introduction

Prompt design is an important factor when applying language models (LMs) to downstream tasks, including LMs that make use of retrieval-augmented generation (RAG; Lewis et al., 2020). Well-constructed prompts can improve LMs' answers to user-input questions and help generate responses that better align with user expectations Gao et al. (2021); Izacard et al. (2024); Liu et al. (2024); Schulhoff et al. (2024); Ma et al. (2024), _inter alia_).

Under the RAG framework, a prompt typically consists of three components. First, an instruction provides a textual description of the overall task and general guidance for the language model. Second, a specific question encodes the precise task or query the model should perform. Third, a context encodes a set of documents retrieved from an external source by a retriever Karpukhin et al. (2020); Ni et al. (2022). Then, an answer is sampled from the language model. Previous work has explored various empirical approaches to prompt engineering, including the manual design of prompts that mimic human reasoning Wei et al. (2023); Yao et al. (2023). Recently, Liu et al. (2024) demonstrated that language model performance is significantly influenced by the _order_ of the retrieved documents that comprise the context. Specifically, QA accuracy peaks when the gold document2 is positioned at the beginning or end of the context. Although extensive experimental evidence was adduced to validate this phenomenon Liu et al. (2024), the underlying mechanisms remain poorly understood. This gap in understanding limits the applicability of these findings in the design and optimization

Figure 1: For the _same_ question, a permutation of documents with a higher \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) tends to lead to a better answer.



of prompts for real-world applications.

While Liu et al.'s (2024) results are interesting, choosing the optimal permutation of the documents requires knowledge of the answer and, thus, cannot be directly used to improve RAG. In this article, we develop a proxy for the optimal permutation: We show that the pointwise mutual information between the question and the context under an LM acts as a useful proxy in determining the optimal permutation. To our knowledge, ours is the first to present in-depth analyses of the relation between question likelihood and model performance under the RAG framework.

Our findings in this paper are summarized in the following list:

* We show that the pointwise mutual information between the question and the context positively correlates with answer accuracy at the corpus level on NQ-Open Kwiatkowski et al. (2019); Lee et al. (2019) and ELI5 Fan et al. (2019).
* Given a question and a fixed set of documents, we demonstrate a strong correlation between the position of the gold document, the PMI between the question and the context, and QA accuracy.
* We validate the effectiveness of using question likelihood as a gauge for prompt optimization and demonstrate that likelihood-based prompt optimization is a promising direction for future study.

## 2 Setting the Stage

### Language Modeling and RAG

Language Modeling Background.Let \(\Sigma\) be an alphabet, i.e., a finite, non-empty set of **tokens**. A **language model**\(p\) is a distribution over \(\Sigma^{*}\), the set of all strings with tokens drawn from \(\Sigma\). With \(\mathbf{\sigma}\preceq\mathbf{\sigma}^{\prime}\), we denote that \(\mathbf{\sigma}\) is a proper prefix of \(\mathbf{\sigma}^{\prime}\). Let \(Y\) be a \(\Sigma^{*}\)-valued random variable distributed according to \(p\) and \(\mathbf{\sigma}\in\Sigma^{*}\). We define the **prefix probability3\(\overrightarrow{p}(\mathbf{\sigma})\)** as the probability that \(Y\) has \(\mathbf{\sigma}\) as a prefix:

Footnote 3: See Vieira et al. (2024) for a more in-depth discussion.

\[\overrightarrow{p}(\mathbf{\sigma}) \triangleq\mathop{\mathbb{P}}_{Y\sim p}\left(Y\succeq\mathbf{\sigma}\right) \tag{1a}\] \[=\sum_{\mathbf{\sigma}^{\prime\prime}\in\Sigma^{*}}\mathbbm{1}\! \left\{\mathbf{\sigma}^{\prime}\succeq\mathbf{\sigma}\right\}p(\mathbf{\sigma}^{\prime})\] (1b) The conditional prefix probability tells us how certain the model is that \[\mathbf{\sigma}^{\prime}\] naturally follows from its preceding string \[\mathbf{\sigma}\] . Finally, we define an **infix probability**, i.e., the probability of generating a string that contains \[\mathbf{\sigma}\Box\mathbf{\sigma}^{\prime}\] ; where as is a gap, as follows \[\overrightarrow{p}(\mathbf{\sigma}\Box\mathbf{\sigma}^{\prime\prime}) \triangleq\mathop{\mathbb{P}}_{Y\sim p}\bigl{(}Y\succeq\mathbf{\sigma}\Box \mathbf{\sigma}^{\prime\prime}\bigr{)}\] (2a) \[=\sum_{\mathbf{\sigma}^{\prime\prime\prime}\in\Sigma^{*}}\sum_{\mathbf{ \sigma}^{\prime}\in\Sigma^{*}}\mathbbm{1}\!\left\{\mathbf{\sigma}^{\prime\prime \prime}\succeq\mathbf{\sigma}\mathbf{\sigma}^{\prime}\mathbf{\sigma}^{\prime\prime}\right\} p(\mathbf{\sigma}^{\prime\prime\prime}) \tag{2b}\]

Retrieval-augmented Generation.Modern language models are often used to perform question-answering tasks. When solving such a task with a language model, string encoding the question question \(q\in\Sigma^{*}\) is given to the model. We assume each question \(q\) has a unique correct answer, which we will denote \(a\). This is, of course, a simplifying assumption, but it does jibe with how question-answering is typically evaluated. We will adorn symbols with \(\widetilde{\cdot}\), e.g., \(\widetilde{a}\), to indicate an answer generated from \(\overrightarrow{p}\left(\cdot~{}|~{}q\right)\) that may or may not be correct. Generating \(\widetilde{a}\) from \(\overrightarrow{p}\left(\cdot~{}|~{}q\right)\) may be done using either a deterministic method, e.g., beam search, or a stochastic method, e.g., ancestral sampling.4 In RAG, the model is additionally given a set of documents \(\mathcal{D}=\{d_{k}\}_{k=1}^{K}\), where \(d_{k}\in\Sigma^{*}\), and a permutation of the documents \(\pi\colon\{1,\cdots,K\}\to\{1,\cdots,K\}\). Given \(\mathcal{D}\) and \(\pi\), a context \(c\) is constructed by concatenating the documents in the order defined by \(\pi\), i.e., \(c_{\mathcal{D}}(\pi)\triangleq\mathbf{d}_{\pi(1)}\mathbf{\cdot}\dots\mathbf{d}_{\pi(K)}\). Then, we generate an answer from \(\overrightarrow{p}\left(\cdot~{}|~{}\mathbf{c}\boldsymbol{\cdot}~{}q\right)\). We provide an example below.

Footnote 4: In this study, we consider greedy decoding, i.e., beam search with a beam of size 1.

**Example 2.1**.: _Consider \(\mathcal{D}=\{``Llandudno Pier is a Grade II" listed pier...", "Garth Pier is a Grade II listed structure...", "Southend Pier is \(a\)..."\}\), and \(\pi(1)=2,\pi(2)=1,\pi(3)=3\). We have \(c_{\mathcal{D}}(\pi)=\) "Garth Pier is ... Llandudno Pier is a Grade II" listed pier...Southend Pier is ..."._

Let \(\widetilde{a}_{\pi}\) denote an answer generated from \(\overrightarrow{p}\left(\cdot~{}|~{}c_{\mathcal{D}}(\pi)\star q\right)\). To evaluate the quality of \(\widetilde{a}_{\pi}\), we define an evaluation metric \(g(\widetilde{a}_{\mp},a)\). In addition, we assume the ground truth answer \(a\) to be unique for a question-context pair \((\mathbf{q},\mathbf{c})\).

Pointwise Mutual Information.In RAG question answering, we consider the following **pointwise mutual information**

\[\text{PMI}(\mathbf{q},\mathbf{c})\triangleq\log\frac{\overrightarrow{p}(\mathbf{q}~{}|~{} c)}{\overrightarrow{p}(\mathbf{q})}\] (3) 

between \(q\) and \(c\), where \(c=d_{\pi(1)}\cdot\cdots\cdot d_{\pi(K)}\). In other words, Eq. (3) measures the degree of association of \(q\) with \(c\).

### A Concrete Hypothesis

Returning to the central goal of this paper, i.e., trying to find a proxy that helps determine the optimal permutation of the documents for RAG, we hypothesize that, given a question \(q\), a set of documents \(\mathcal{D}\), and the ground truth answer \(a\), the pointwise mutual information \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) correlates with \(\log\frac{\overrightarrow{p}(a|q\boldsymbol{\cdot c_{\mathcal{D}}(\pi)})}{1- \overrightarrow{p}(a|q\boldsymbol{\cdot c_{\mathcal{D}}(\pi)})}\), the log odds ratio, and can be deemed a gauge for the expected accuracy of the generated answer. In symbols, our hypothesis is as follows.

**Hypothesis 2.1**.: _In RAG question answering, for a fixed question \(q\), a set of documents \(\mathcal{D}\) permuted by \(\pi\), and the ground truth answer \(a\), we have the following relation between \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) and \(\overrightarrow{p}(a~{}|~{}q\boldsymbol{\cdot c_{\mathcal{D}}(\pi)})\)_

\[\begin{split}\text{PMI}&(q,c_{\mathcal{D}}(\pi))\\ &=a\log\frac{\overrightarrow{p}(a~{}|~{}q\boldsymbol{\cdot c_{ \mathcal{D}}(\pi)})}{1-\overrightarrow{p}(a~{}|~{}q\boldsymbol{\cdot c_{ \mathcal{D}}(\pi)})}+b\end{split} \tag{4}\]

_for constants \(a\in\mathbb{R}_{>0}\), \(b\in\mathbb{R}\)._

### A Bit of Analysis

In words, Hypothesis 2.1 says that when \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) is high, we expect the LM to answer the question \(q\) with high accuracy, and, moreover, we expect the relationship between \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) and accuracy to be affine. Although this is empirically true in many cases Gonen et al. (2023), we offer an assumption that enables a derivation of this property.

**Assumption 2.1**.: _For all question-context pairs \((q,c)\), let \(a\) be the correct answer, then we have_

\[\overrightarrow{p}(q~{}|~{}c_{\mathcal{D}}(\pi)\Box a) =\overrightarrow{p}(q~{}|~{}c_{\mathcal{D}}(\pi)) \tag{5a}\] \[\overrightarrow{p}(q~{}|~{}c_{\mathcal{D}}(\pi)\Box\bar{a}) =\overrightarrow{p}(q) \tag{5b}\]

_for any \(\bar{a}\in\Sigma^{*}\) such that \(\bar{a}\not\preceq a\).5_

Footnote 5: Eq. (5a) and Eq. (5b) can be seen as a form of context-specific conditional independence Boutilier et al. (1996).

We now give a brief qualitative justification of Assumption 2.1. Conditioned on the event that the model _incorrectly_ answers the question given the context, Eq. (5b) says that the question \(q\) is not dependent on the provided context. Because, in RAG, we assume the correct answer is given to the model in the context and the model's job is to retrieve it, our assumption corresponds to the notion that an incorrect response by the model should _not_ be influenced by the context. Eq. (5a) corresponds to the notion that since the correct answer is already contained in the context, conditioning on the correct answer does not provide any new information to generating \(q\).

**Proposition 2.1**.: _Under assumptions given in Assumption 2.1, we have_

\[\begin{split}\log&\frac{\overrightarrow{p}(a~{}|~{} q\boldsymbol{\cdot c_{\mathcal{D}}(\pi)})}{1-\overrightarrow{p}(a~{}|~{}q \boldsymbol{\cdot c_{\mathcal{D}}(\pi)})}\\ &=\text{PMI}(q,c_{\mathcal{D}}(\pi))+C(a,c_{\mathcal{D}}(\pi)) \end{split} \tag{6}\]

_for an answer-dependent constant \(C(a,c_{\mathcal{D}}(\pi))\)._

Proof.: See Appendix G. 

In other words, the pointwise mutual information \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) is equivalent up to an additive constant to the log odds ratio \(\log\frac{\overrightarrow{p}(a|q\boldsymbol{\cdot c_{\mathcal{D}}(\pi)})}{1- \overrightarrow{p}(a|q\boldsymbol{\cdot c_{\mathcal{D}}(\pi)})}\).

Foreshadowing the Results.In the empirical portion of this paper, we test Hypothesis 2.1 through experiments on two QA benchmarks--NQ-Open and ELIS--using a range of state-of-the-art open LMs, including LLaMA-2, LLaMA-3, LLaMA-3.1,Mistral-v0.3, and MPT. Our findings demonstrate that, as the position of relevant information within the input context \(c\) varies, the pointwise mutual information, \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) and expected answer accuracy \(\overrightarrow{p}(a~{}|~{}q\boldsymbol{\cdot c_{\mathcal{D}}(\pi)})\) vary in tandem, i.e., they are strongly correlated. This correlation is illustrated in Figure 2. Specifically, LMs tend to provide better responses to questions where the documents in the context are permuted so as to have higher \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\). These results

Figure 2: We observe that the PMI and QA accuracy trace a U-shaped curve—nearly in lockstep—as the gold document position within the context changes. The result is computed with LLaMA-3-8B.

 

suggest that PMI serves both as a _performance gauge_ and as a strong indicator of the position of task-relevant information within the input context. Building on this insight, we propose a direction for prompt optimization through two specific methods. The first selects a permutation \(\pi\) of the documents that maximizes \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) to construct the context \(c\). The second builds on the findings of Liu et al. (2024) that the curve traced by permuting the position of the gold document results in a U-shaped curve. We exploit this finding to develop an efficient prompt ordering algorithm. Further experimentation demonstrates that our methods enhance answer accuracy across both datasets for instruction-tuned and base models alike, with the second approach achieving even greater gains.

## 3 PMI Correlates with Performance

As discussed in SS2, our first goal is to determine how \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) changes as a function of the permutation \(\pi\) of the documents \(\mathcal{D}\). Due to Hypothesis 2.1, we expect a strong correlation between \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) and expected answer accuracy.

### Experimental Setup

Datasets.We run experiments on two question-answering datasets, namely NQ-Open and ELI5. Details of the datasets are given in Appendix C. Let \(\mathcal{C}\triangleq\{(q_{m},\mathcal{D}_{m},a_{m})\}^{M}_{m=1}\) be a dataset of triples, where each \(a_{m}\) represents the ground truth answer to \(q_{m}\).

Empirical Metrics for LM Evaluation.In practice, LM performance is often evaluated with rule-based empirical metrics, denoted \(g\), such as accuracy, instead of the conditional likelihood \(\overrightarrow{p}(a~{}|~{}q\cdot c_{\mathcal{D}}(\pi))\). Although mathematically quantifying the relation between \(g\) and PMI is difficult, we contend that they positively correlate due to recent progress on language model calibration Zhao et al. (2023), i.e., the alignment between \(\overrightarrow{p}(a~{}|~{}q\cdot c_{\mathcal{D}}(\pi))\) and \(g(\widetilde{a},a)\). On NQ-Open, the ground truth answer for each question is either a word or a short phrase. The accuracy is \(1\) when the LM response contains the correct answer as a substring; otherwise, the accuracy is \(0\). Following Liu et al. (2024), we compute the model's average accuracy over the entire dataset. On ELI5, the correct answer for each question comprises three sub-claims, and a correct answer is expected to include all of these sub-claims. Examples are illustrated in Appendix B. We follow Gao et al. (2023) and take the recall rate of sub-claims to be the evaluation metric, which takes values from \(\{0,\nicefrac{{1}}{{3}},\nicefrac{{2}}{{3}},1\}\). The TRUE model,6 a T5-XXL model fine-tuned on natural language inference (NLI) tasks, is used to automatically evaluate whether a response entails a sub-claim.

Footnote 6: [https://huggingface.co/google/t5_xxl_true_nl_mixture](https://huggingface.co/google/t5_xxl_true_nl_mixture)

Language Model Settings.Most state-of-the-art closed LMs, such as OpenAI's ChatGPT and Anthropic's Claude, do not provide direct access to the likelihood of either input or output tokens. Thus, we select leading open LMs for our experiments, focusing on three families: LLaMA-2, LLaMA-3Touvron et al. (2023), and Mistral-v0.3Jiang et al. (2023). We also evaluate MPT on NQ-Open.7 Following the settings of Liu et al. (2024), we adopt greedy decoding for all models when generating responses. We set the maximum number of decoded tokens to \(100\) on NQ-Open and \(300\) on ELI5.

Footnote 7: In our preliminary experiments, MPT fails to generate sufficiently long responses on ELI5, resulting in performance that is not directly comparable to other LMs.

Prompt Templates.We follow the suggested usage and prompt formatting instructions of each LM we use. For chat and instruction-tuned models, we present the context and query to the LM in the role of user, and elicit the response from LMs in the role of assistant. For base models, we elicit responses from LMs as sentence completion.

### Technical Interlude: Sets of Permutations

In many of our experiments, we would like to take a sum or a max over all permutations of \(K\) items, i.e., take a sum or max over the symmetric group \(\mathbb{S}_{K}\). However, \(|\mathbb{S}_{K}|=K!\), which grows too large to enumerate efficiently. To cope with the size of \(\mathbb{S}_{K}\), in this paper, we perform computations over a subset of \(\mathbb{S}_{K}\). Specifically, starting a user-specified permutation \(\pi\), we consider the cyclic group generated by \((\pi)\) where the group operation is functional composition, as is standard. Let \(\sigma=(1,2,\cdots,K)\) be a shifting permutation. It is easy to see that \(|(\pi)|=K\), and the \(k^{\text{th}}\) element of \((\pi)\) is given by \(\widetilde{\pi}_{k}\triangleq\sigma^{k-1}\circ\pi=\underbrace{\sigma\circ \cdots\circ\sigma}_{(k-1)~{}\text{times}}\circ\pi\),8 or, equivalently we have

Footnote 8: Here, \(\diamond\) denotes function composition, the standard product of permutations.

\[\widetilde{\pi}_{k}(i)=(i+k-1)\mod K,\] (7) 



**Example 3.1**.: _Given a permutation \(\pi=(1,2,3)\). The cyclic group \((\pi)\) generated by \(\pi\) is equal to \(\{\widetilde{\pi}_{1},\widetilde{\pi}_{2},\widetilde{\pi}_{3}\}\) where \(\widetilde{\pi}_{1}=(1,2,3)\), \(\widetilde{\pi}_{2}=(2,3,1)\), and \(\widetilde{\pi}_{3}=(3,1,2)\)._

### Results

We now discuss our empirical findings.

Corpus-Level Correlation.As our first evaluation metric, we consider a corpus-level correlation. For each \(q_{m},\mathcal{D}_{m}\) in a corpus \(\mathcal{C}\), we compute the average PMI for the \(m^{\text{th}}\) instance as follows

\[\rho_{m}\triangleq\frac{1}{K}\sum_{k=1}^{K}\text{PMI}(q_{m},c_{\mathcal{D}_{ m}}(\widetilde{\pi}_{k})) \tag{8}\]

We then bin the elements of \(\{\rho_{m}\}_{m=1}^{M}\) into three bins according to which tertile they fall it when \(\{\rho_{m}\}_{m=1}^{M}\) are arranged into a histogram. Then, we compute the average sub-claim recall rate (ELI5) and accuracy (NQ-Open) for each bin. Our results, shown in Figure 3, demonstrate that LMs tend to perform better on the prompts with a higher PMI\((q,c_{\mathcal{D}}(\pi))\) compared to those with lower PMI\((q,c_{\mathcal{D}}(\pi))\).

Instance-Level Correlation.We further analyze the instance-level correlation between PMI\((q,c_{\mathcal{D}}(\pi))\) and accuracy by varying context while keeping the question fixed. In symbols, we compute

\[\eta_{k}\triangleq\frac{1}{M}\sum_{m=1}^{M}\text{PMI}(q_{m},c_{\mathcal{D}_{ m}}(\widetilde{\pi}_{k})) \tag{9}\]

where \(\widetilde{\pi}_{k}\) is the permutation in which the \(k\)-th document \(d_{k}\) contains relevant information. We then plot the curve of \(\{\eta_{k}\}_{k=1}^{K}\), to see how PMI is affected by the position of a relevant document within a context.

Revisiting Liu et al. (2024).We now revisit the findings of Liu et al. (2024), who observed a drop in _QA accuracy_ when the gold document is positioned within the middle of \(c\). We first experiment on NQ-Open by varying the position of the gold document9 in \(c\). The set of retrieved documents and the order of non-gold documents remain the same. As the gold document is placed in different positions in \(c\), we find that both PMI\((q,c_{\mathcal{D}}(\pi))\) and QA accuracy fluctuate--nearly in lockstep. To further explore this correlation between PMI\((q,c)\) and QA accuracy, we calculate the expected accuracy with the prompt of the highest and lowest PMI\((q,c_{\mathcal{D}}(\pi))\). Results are given in Table 1, showing that LMs perform better when the document order in the prompt leads to the highest PMI\((q,c_{\mathcal{D}}(\pi))\), while

Figure 3: Corpus-level correlation between PMI\((q,c_{\mathcal{D}}(\pi))\) and answer accuracy on NQ-Open and ELI5.



the prompt with the lowest \({\text{PMI}}(q,c_{\mathcal{D}}(\pi))\) results in inferior performance.

Experiments on ELI5.Compared to NQ-Open, ELI5 is a more challenging long-form QA dataset where questions are mostly about _how/why/what_, and the answers are expected to be more comprehensive and cover multiple aspects. Due to the lack of gold document annotations on ELI5, we adopt permutations from the cyclic group \((\pi)\) and random shuffling. In random shuffling for \(K\) documents, we randomly shuffle the document set \(K\) (i.e., same as the number of documents) times and obtain \(K\) document sequences for consistency. Given multiple prompts for a question, among which only the document orders in the context are different, we calculate the average performance of the prompts with the highest and lowest \({\text{PMI}}(q,c_{\mathcal{D}}(\pi))\) for each question in the same fashion as described in SS3.3 for NQ-Open. Results in Table 2 show that LMs achieve higher answer accuracy on the prompts with the highest \({\text{PMI}}(q,c_{\mathcal{D}}(\pi))\), compared with the prompts with the lowest \({\text{PMI}}(q,c_{\mathcal{D}}(\pi))\). This indicates LMs can better answer questions with higher question likelihood through document shuffling, demonstrating the strong instance-level correlation between \({\text{PMI}}(q,c_{\mathcal{D}}(\pi))\) with answer accuracy.

## 4 Improving RAG via Reordering

In SS3.3, we offered evidence for Hypothesis 2.1, i.e., that \({\text{PMI}}(q,c_{\mathcal{D}}(\pi))\) correlates with model performance. In light of this finding, we propose two methods to permute the documents presented to the LM in RAG _without_ knowledge of the answer.

### Method 1: Search by PMI

Our empirical findings showed that the permutation of the documents in the context that leads to the highest value of \({\text{PMI}}(q,c_{\mathcal{D}}(\pi))\) leads to superior performance on QA tasks. This suggests a natural algorithm

\[\pi^{\star}=\operatorname*{arg\,max}_{\pi\in\mathbb{S}_{K}}{\text{PMI}}(q,c_{ \mathcal{D}}(\pi)) \tag{10}\]

However, as discussed in SS3.2, the set of all permutations (the symmetric group) \(\mathbb{S}_{K}\) is too large to enumerate. Thus, we fall back on a simple approximation. Given a user-provided permutation \(\pi\), we search over the cyclic group generated by \(\pi\), denoted as \((\pi)\). Using the notation introduced in SS3.2, we choose \(\widetilde{\pi}_{k^{\star}}\) where we select

\[k^{\star}=\operatorname*{arg\,max}_{k=1}^{K}{\text{PMI}}(q,c_{\mathcal{D}}( \widetilde{\pi}_{k})), \tag{11}\]

where \(\widetilde{\pi}_{k}\) is defined in SS3.2.

### Method 2: Search by Curvature

We now develop a second algorithm based on the observation in Figure 4 that accuracy and PMI change simultaneously and exhibit a U-shaped curve as the gold document position within the permutation of documents in \(c\). Our algorithm is based on a discrete notion of convexity and an assumption based on our findings in SS3.3, which we introduce in the abstract below.

Technical Interlude Discrete Convexity.A sequence of real values \(\{a_{n}\}_{n=1}^{N}\) is called **convex** if we have

\[\Delta_{n}^{2}\triangleq 2a_{n}-a_{n+1}-a_{n-1}\leq 0 \tag{12}\]

for all \(n\in\{2,\ldots,N-1\}\). In the abstract, the problem we wish to solve is this: Given an arbitrary finite sequence of reals \(\{b_{n}\}_{n=1}^{N}\), find a permutation \(\tau\colon[N]\to[N]\) that renders \(\{b_{n}\}_{n=1}^{N}\) convex, i.e., that Eq. (12) holds after applying the permutation to the sequence's indices. We call such a choice of \(\tau\) a **convex permutation**. Note that convex permutations may not always exist.10 To achieve a U-shape curve, we do not just want a convex permutation, but in addition the one that results in a convex sequence that has as much upwards curvature as possible. In other words, if \(\tau\) is a convex permutation, then, _in addition_ we want the following sum to be _minimized_

Footnote 10: E.g., the sequence \([0,1,1,1]\) has no convex permutation.

\[\sum_{n=2}^{N-1}\Delta_{n}^{2} =-(b_{1}+b_{N})+\sum_{n=2}^{N-1}b_{m} \tag{13a}\] \[=-2(b_{1}+b_{N})+B\] (13b) \[\leq 0, \tag{13c}\]

where \(B\triangleq\sum_{n=1}^{N}b_{n}\). However, because \(B\) is constant, the total curvature induced by a convex permutation \(\tau\)_only_ depends on \(b_{\tau(1)}\) and \(b_{\tau(N)}\). This implies that we simply need to choose the endpoints to be those elements of \(\{b_{\tau(n)}\}_{n=1}^{N}\) that are largest; we can always permute the remaining \((N-2)\) elements to ensure the permutation is convex afterward. Thus, relaxing the requirement th 

the permutation be convex, we choose a permutation \(\tau\) such that \(b_{\tau(1)}+b_{\tau(N)}\) is maximized. This definition motivates a new definition: We call a sequence \(\{b_{n}\}_{n=1}^{N}\)**U-shaped** iff \(b_{1}\geq b_{i}\) and \(b_{N}\geq b_{i}\) for \(i\in\{2,3,\cdots,N-1\}\).

A Simple Algorithm.The abstract discussion in the previous paragraph suggests a simple algorithm. First, we construct a real-valued sequence

\[b_{\tau(k)}\triangleq\text{PMI}(q,c_{\mathcal{D}}(\vec{\pi}_{k})) \tag{14}\]

of length \(K\) where \(\tau\colon[K]\to[K]\) is a permutation and \(\text{PMI}(q,c_{\mathcal{D}}(\vec{\pi}_{k}))\) is defined in SS3.2. Then, relaxing the requirement that the permutation be convex, we optimize

\[\tau^{\star}=\operatorname*{arg\,max}_{\tau\in(\pi)}b_{\tau(1)}+b_{\tau(K)} \tag{15}\]

While, in general, \(\tau^{\star}\) may not be convex, we do have a guarantee that \(\tau^{\star}\) will induce a U-shaped sequence. To compute the optimization problem given in Eq. (15), we sort \(\tau\in(\pi)\) according to \(b_{\tau(1)}+b_{\tau(K)}\) in descending order, obtaining the sequence \(\{\tau_{k}\}_{k=1}^{K}\). Then, we construct the resulting permutation \(\tau^{\prime}=(\tau_{1}(1),\tau_{2}(1),\cdots,\tau_{K}(1))\), among which \(d_{\tau(1)}\) is most likely to be the gold document.

### Results and Analysis

Shown in Table 3, both search by PMI and search by curvature can boost answer accuracy. On NQ-Open, where only one document in the sequence is relevant to the question, gold document reordering significantly improves the answer accuracy and narrows the gap to the upper bound. Furthermore, on the more challenging and practical QA benchmark ELI5, we also observe a modest improvement

Figure 4: QA accuracy, PMI, and log odds ratio of answer likelihood on 20 docs evaluated on LLaMA-3.1–8B and LLaMA-3.1–8B-Instruct.

Figure 5: When the position of the gold document changes, both \(\text{PMI}(q,c)\) and accuracy curves are U-shaped. In contrast, both curves are flat for randomly chosen documents.

 

in answer accuracy, indicating that improving question likelihoods via document reordering can effectively obtain better LM responses.

Regarding efficiency, our proposed methods are mildly time-dependent thanks to the parallelizable computation of question likelihoods, where only the LM encoding module is used, with no reliance on LM decoding.11 Shown in Table 4, in our experiments, the average runtime for decoding a response of an instance in ELI5 is 10 seconds, while it only takes an extra 0.8 seconds and 2 seconds, respectively, to encode the input prompts of naive likelihood-based selection and gold document reordering. The increment in timely cost is marginal compared with heuristic prompt engineering which requires whole decoding to judge the prompt quality (e.g., an extra 10 seconds for decoding another candidate prompt).

Footnote 11: LM decoding (i.e., generation) requires a runtime approximately proportional to the number of generated tokens. Empirically, the only extra computational time for our methods is on the encoding phase for calculating likelihoods, so the overall runtime is the vanilla Runtime\({}_{\mathrm{LM}}\) plus _one_ extra LM going through, which is equivalent to generating the response with _one_ additional token.

In summary, both proposed methods are effective and efficient. Although the improvement on ELI5 is relatively marginal compared to that on NQ-Open, given the more challenging nature of long answers and no specified gold document on ELI5, it still indicates that optimizing prompts with \(\overrightarrow{p}(q~{}|~{}c)\) as a gauge is a promising direction.

Experimental Setup.We experiment on the ELI5 dataset and a subset of 500 questions from NQ-Open, using Mistral-7B-Inst-v@.3, LLaMA-3.1-8B, and LLaMA-3.1-8B-Inst. Each question is associated with 10 (on ELI5) and 20 (NQ-Open) retrieved documents.

### Synthetic Experiment

Real-world datasets might have been used during the training of LLMs. Thus, their likelihoods might exhibit an _exposure bias_(Bengio et al., 2015; Ranzato et al., 2016; Cotterell et al., 2024). To avoid such potential bias, we follow Liu et al. (2024) and conduct a synthetic key-value retrieval experiment.

Key-Value Retrieval.To imitate question-answering tasks on random strings, we construct Python-style key-value pairs in which the keys and

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline \#Doc & \multicolumn{2}{c}{PMI(\(q,c_{D}(\tau)\)) Mistral-7B-Inst} & \multicolumn{2}{c}{LLaMA-3-8B} & \multicolumn{2}{c}{LLaMA-3.1–8B} & \multicolumn{2}{c}{LLaMA-3-8B-Inst} \\ \hline \hline  & \multicolumn{5}{c}{EL5 with Rotational permutation} \\ \cline{2-9}
5 & Highest & **13.97** (-3.72) & **11.37** (-2.23) & **12.60** (-2.28) & **14.23** (-2.21) & **13.97** (-2.26) \\  & Lowest & 13.50 (-4.06) & 11.10 (-2.39) & 12.50 (-2.43) & 13.17 (-2.54) & 13.93 (-2.48) \\ \hline  & Highest & **15.23** (-3.53) & 11.27 (-2.19) & 12.50 (-2.29) & **14.50** (-2.10) & **16.17** (-2.23) \\  & Lowest & 14.47 (-3.99) & **11.50** (-2.39) & **13.10** (-2.48) & 14.07 (-2.55) & 15.77 (-2.54) \\ \hline  & Highest & **16.20** (-2.13) & 11.13 (-2.19) & **12.77** (-2.28) & **16.20** (-2.13) & **17.17** (-2.18) \\  & Lowest & 15.80 (-2.73) & **11.20** (-2.42) & 12.13 (-2.48) & 15.80 (-2.73) & 15.67 (-2.54) \\ \hline  & \multicolumn{5}{c}{EL15 with Random Shuffling} \\ \hline \multirow{2}{*}{5} & Highest & **14.27** (-3.73) & 10.73 (-2.24) & **12.57** (-2.28) & **14.10** (-2.23) & **14.20** (-2.27) \\  & Lowest & 14.10 (-4.04) & **11.20** (-2.39) & 12.33 (-2.42) & 12.77 (-2.52) & 14.00 (-2.48) \\ \hline  & Highest & **15.63** (-3.54) & **11.47** (-2.19) & **12.73** (-2.29) & **15.70** (-2.11) & **16.90** (-2.23) \\  & Lowest & 15.07 (-3.97) & 11.23 (-2.39) & 12.20 (-2.48) & 14.57 (-2.52) & 16.70 (-2.53) \\ \hline  & Highest & 16.10 (-3.44) & 10.83 (-2.19) & **12.60** (-2.28) & **16.13** (-2.14) & **17.20** (-2.18) \\  & Lowest & **16.53** (-4.00) & **11.20** (-2.42) & 11.87 (-2.49) & 15.53 (-2.71) & 17.10 (-2.54) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Instance-level correlation between PMI\((q,c)\) and answer accuracy on ELI5. The average QA accuracy is computed over prompts that yield the highest and lowest PMI\((q,c)\) as the input documents are reordered with (1) rotational reordering and (2) random shuffling as introduced in §3.3. We report the QA accuracy and the average PMI\((q,c)\) in parentheses.

\begin{table}
\begin{tabular}{l r r r|r} \hline \hline Model & Baseline & PMI & Curvature & Upper Bound \\ \hline \multicolumn{5}{c}{NQ-Open (Answer Accuracy)} \\ \hline Mistral & 62.89 & 65.18 & **65.72** & 69.24 \\ LLaMA-3.1 & 47.74 & 51.29 & **51.36** & 66.88 \\ LLaMA-3.1–Inst & 61.49 & 63.34 & **63.56** & 66.35 \\ \hline \multicolumn{5}{c}{EL5 (Answer Accuracy)} \\ \hline Mistral & 15.35 & **15.63** & 15.40 & - \\ LLaMA-3.1 & 12.61 & 12.73 & **13.33** & - \\ LLaMA-3.1–Inst & 16.14 & **16.90** & 16.83 & - \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance of our methods on NQ-Open and ELI5, the number of documents \(K\) is set to 20 and 10. Mistral, LLaMA and LLaMA-Inst stands for Mistral-7B-Inst-v@.3, LLaMA-3.1–8B and LLaMA-3.1–8B-Inst respectively. Baseline refers to the mean performance over \(K\) random document shuffling on each instance. The upper bound on NQ-Open is calculated as the performance when positioning the gold document at the beginning of the document sequence, which is not applicable for ELI5 since no gold document is marked in this practical dataset.

 

values are UUID strings of 32 hexadecimal digits. An example is given in Figure 14. In Tables 5 to 7, we observe that both \(\text{PMI}(q,c)\) and \(\overrightarrow{p}(a~{}|~{}c\cdot q)\) show synchronous U-shaped patterns as the location of the key in context changes, consistent with the RAG-based QA experiments in SS3.3, indicating the generalizability of the findings on unseen data.

## 5 Discussion

### Instruction-tuned vs. Base Models

In our analysis, we find base LMs, e.g., LLaMA-3-8B, tend to be more sensitive to the permutation of the documents. Specifically, we observe that QA performance drops when the gold document is placed in the middle of the document sequence. On the other hand, the performance of instruction-tuned models is more robust to permutations of the documents in the context, as shown in Figure 4. However, we still do observe a U-shaped curve, but the drop in QA performance is less significant for the instruction-tuned model when the gold document is positioned at the middle.

The fact that PMI serves as a useful gauge for both the base and instruction-tuned models suggests that \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) is affected little by the instruction tuning.

### Placing the Context after the Question

In our experiments, we only explore the correlation between PMI and accuracy when the question follows the context. However, one could also use a prompt template in which the context follows the question. We remark that in this case, PMI can be computed according to the equation

\[\text{PMI}(q,c)=\log\frac{\overrightarrow{p}(c~{}|~{}q)}{\overrightarrow{ p}(c)}. \tag{16}\]

## 6 Conclusion

In our study, we analyzed the relationship between the PMI between question and context \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) and question-answering performance under the retrieval-augmented generation framework. Through experimentation, we demonstrated that \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) is affected by the order of documents in the input context. We find evidence for a positive correlation between question likelihood and answer accuracy at both the corpus level and instance level. Our findings show that it is possible to use \(\text{PMI}(q,c_{\mathcal{D}}(\pi))\) to gauge language model performance and improve the quality of input prompts. We propose two practical methods for prompt optimization based on these findings. Experimental results show that both effectively and efficiently improve LM's accuracy on QA tasks, demonstrating that using PMI as a gauge for optimizing prompts is a promising direction.

## Limitations

One major limitation of our work is that only open-source LMs are studied in this work since we need full access probabilities under the LM. Thus, closed language models such as GPT-4 cannot be used for selecting permutations In addition, our prompt modification is limited to document permutation in this work. Other prompt modifications may also contribute to obtaining a higher \(\text{PMI}(q,c)\) and improve QA performance.

## References

* Asai et al. (2021) Akari Asai, Xinyan Yu, Jungo Kasai, and Hanna Hajishirzi. 2021. One question answering model for many languages with cross-lingual dense passage retrieval. In Advances in Neural Information Processing Systems, Vol. 34, pp. 7547-7560. Curran Associates, Inc. Cited by: SS1.
* Volume 1, NIPS'15, Cambridge, MA, USA, MIT Press. External Links: ISBN 978-3-319-3861-1, Link, Document Cited by: SS1.
* S. Borgeaud, Arthur Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. Van Den Driessche, J. Lespiau, B. Damoc, A. Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. Cited by: SS1.
* C. Boutilier, N. Friedman, M. Goldszmidt, and D. Koller (1996)Context-specific independence in Bayesian networks. In Proceedings of the Twelfth International Conference on Uncertainty in Artificial Intelligence, pp. 115-123. Cited by: SS1.
* R. Cotterell, A. Sveute, C. Meister, T. Liu, and Li Du (2024)Formal aspects of language modeling. Preprint arXiv:2311.04329.

\begin{table}
\begin{tabular}{r|c c} \hline Decoding & Likelihood Based & Gold Document \\ \hline
10s & 0.8s & 2s \\ \hline \end{tabular}
\end{table}
Table 4: The average runtime for decoding an LLM response vs. the extra time for the two proposed methods.

 

* Ekin (2023) Spatt Ekin. 2023. Prompt engineering for chatgpt: a quick guide to techniques, tips, and best practices. _Authorea Preprints_.
* Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. EL15: Long form question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
* Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3816-3830, Online. Association for Computational Linguistics.
* Gao et al. (2023) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6465-6488, Singapore. Association for Computational Linguistics.
* Giray (2023) Louie Giray. 2023. Prompt engineering with chatgpt: a guide for academic writers. _Annals of biomedical engineering_, 51(12):2629-2633.
* Gonen et al. (2023) Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 10136-10148, Singapore. Association for Computational Linguistics.
* Izaccard et al. (2022) Gautier Izaccard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. _Preprint_, arXiv:2112.09118.
* Izcard and Grave (2021) Gautier Izcard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 874-880, Online. Association for Computational Linguistics.
* Tzecard et al. (2024) Gautier Izcard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2024. Atlas: Few-shot learning with retrieval augmented language models. _J. Mach. Learn. Res._, 24(1).
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lilio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral 7B. _Preprint_, arXiv:2310.06825.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, Online. Association for Computational Linguistics.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_, NIPS '20, Red Hook, NY, USA. Curran Associates Inc.
* Liu et al. (2023) Nelson Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 7001-7025, Singapore. Association for Computational Linguistics.
* Liu et al. (2024) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Parangjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. _Transactions of the Association for Computational Linguistics_, 12:157-173.
* Ma et al. (2024) Lijia Ma, Xingchen Xu, and Yong Tan. 2024. Crafting knowledge: Exploring the creative mechanisms of chat-based search engines. _arXiv preprint arXiv:2402.19421_.
* Marvin et al. (2023) Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakamura-Nabende. 2023. Prompt engineering in large language models. In _International conference on data intelligence and cognitive informatics_, pages 387-402. Springer.
* Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022. Teaching language models to support answers with verified quotes. _arXiv preprint arXiv:2203.11147_ 

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. _arXiv preprint arXiv:2112.09332_.
* Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9844-9855, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Petroni et al. (2020) Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktaschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models' factual predictions. In _Automated Knowledge Base Construction_.
* Piktus et al. (2021) Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Oguz, Edouard Grave, Wen-tau Yih, et al. 2021. The web is your oyster-knowledge-intensive nlp against a very large web corpus. _arXiv preprint arXiv:2112.09924_.
* Pryzant et al. (2023) Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Cheng Guang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with "gradient descent" and beam search. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 7957-7968, Singapore. Association for Computational Linguistics.
* Qi et al. (2024) Jirui Qi, Gabriele Sarti, Raquel Fernandez, and Arianna Bisazza. 2024. Model internals-based answer attribution for trustworthy retrieval-augmented generation. _arXiv preprint arXiv:2406.13663_.
* Manzato et al. (2016) Marc'Aurelio Manzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
* Schulhoff et al. (2024) Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, Hyolung Han, Sevien Schulhoff, et al. 2024. The prompt report: A systematic survey of prompting techniques. _arXiv preprint arXiv:2406.06608_.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. _Preprint_, arXiv:2302.13971.
* Vieira et al. (2024) Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J. O'Donnell, and Ryan Cotterell. 2024. From language models over tokens to language models over characters. _Preprint_, arXiv:2412.03719.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. _Preprint_, arXiv:2201.11903.
* Yao et al. (2023) Shunyau Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. _Preprint_, arXiv:2305.10601.
* Zhao et al. (2023) Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2023. Calibrating sequence likelihood improves conditional language generation. In _The Eleventh International Conference on Learning Representations_.
* Zhou et al. (2023) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Pasteer, Siliviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In _The Eleventh International Conference on Learning Representations_ 

Related Work

### Prompt Engineering

Prompt engineering is important for making the best use of LMs in real-world applications (Giray, 2023; Ekin, 2023; Gonen et al., 2023). The most straightforward prompt engineering method is to manually design prompts using heuristics, which requires human experts to design prompts based on domain-specific knowledge and select the prompts that lead to better performance on downstream tasks (Zhou et al., 2023; Marvin et al., 2023). Meanwhile, another line of work explores automatic approaches for prompt engineering (Gao et al., 2021; Pryzant et al., 2023). However, they both require decoding for outputs from LMs to evaluate the quality of prompts, thus incurring high computational costs.

### Retrieval-Augmented Generation

Retrieval-augmented generation is a technique for improving LMs' ability to solve knowledge-intensive tasks (Lewis et al., 2020; Asai et al., 2021; Borgeaud et al., 2022). In the RAG framework, a set of documents relevant to a user query is retrieved from an external source and inserted into prompts as a context, to provide additional information to the LM and improve response quality (Petroni et al., 2020; Lewis et al., 2020). RAG tasks can be divided into two types: short-form and long-form, depending on the topic of the questions and the format of the expected answers. Short-form QA (Izacard and Grave, 2021; Liu et al., 2024) usually concerns factual questions about real-world facts. The expected answers are often unambiguous and concrete words or short phrases. Long-form QA (Fan et al., 2019; Gao et al., 2023) involves _how_, _why_, and _what_ questions that seek more comprehensive responses.

### Effect of Document Order in RAG

Liu et al. (2024) finds that LMs perform better when the document with relevant information is positioned at the beginning or the end of the prompt.12 Specifically, when moving the task-relevant information from the beginning to the end of the document sequence, answer accuracy exhibits a U-shaped trend on a multi-document QA task and a synthetic key-value retrieval task, both using RAG pipelines. However, Liu et al. (2024) mainly focuses on an empirical study with less in-depth analysis, resulting in a gap between the phenomenon and its practical implications. In this work, we attempt to bridge this gap.

Footnote 12: In Liu et al.’s (2024) experimental settings, the gold document is unique in a prompt for each question.

## Appendix B Illustration of Evaluation Metrics

The evaluation metrics for NQ-Open and ELIS are illustrated with two examples in Figure 6.

## Appendix C Datasets

Nq-Open.We first experiment on the NQ-Open dataset following Liu et al. (2024). This dataset covers 2655 factual questions curated from the Natural Questions dataset (Kwiatkowski et al., 2019; Lee et al., 2019) under CC-BY-SA-3.0 license. Each question is accompanied by \(K\) documents retrieved from Wikipedia, among which _exactly one_ contains the answer to the question, namely the gold document. The remaining \(k-1\) documents are termed **distractors**, which are relevant to the topic of the question but do not contain any ground truth answers, retrieved using Contriever (Izacard et al., 2022). In our experiments, the total number of documents \(K\) is drawn from \(\{10,20,30\}\).13

Footnote 13: We remark that NQ-Open was specifically synthesized to examine how answer accuracy is affected by changing the position of relevant information. In real-world applications, the retrieved documents for one question may contain multiple gold documents or none. Nevertheless, it mimics the RAG setup underlying many commercial generative search and QA systems.

Eli5.To validate the generality of our findings, we also experiment on an open-ended non-factual QA dataset ELI5 (Fan et al., 2019) under the BSD license. ELIS consists of questions beginning with _how_, _why_ or _what_ curated from the Reddit forum "Explain Like I'm Five"14, where the answers are expected to be more comprehensive and diverse. Each question is accompanied by \(K\) documents retrieved from Sphere (Piktus et al., 2021)--a filtered version of Common Crawl15, where \(K\) is taken to be 



## Appendix A

Figure 6: Evaluation metrics used in our experiments. On NQ-Open, the evaluation metric is exact string match. One ELI5, a pretrained NLI model is used to evaluate whether the LM output entails the reference claims.



Weite a high-quality answer for the given question using only the provided search results (some of which might be irrelevant). Document[1]Title: Southend Pier is a major landmark in... Document[2]Title: Llandudno Pier Llandudno Pier Islandudno Pier is a Grade II* listed pier in the seaside resort of LIandudno.. Document[3]Title: Garth Pier) Garth Pier Garth Pier is a Grade II listed structure in Bangor.... Question: what is the longest pier in the UK According to the provided documents, the answer is Southend Pier.

to avoid truncation due to the long questions and LMs responses for the long-form QA task. In contrast to NQ-Open, ELI5 does not provide the annotations of gold documents, which aligns with real-world RAG application scenarios, making it a more practical and challenging dataset Nakano et al. (2021); Menick et al. (2022); Liu et al. (2023).

## Appendix D Prompt Templates

The prompt templates used for our experiments are given in Figures 7-9.

Figure 8: An example prompt and LM output on ELI5. The prompt comprises (1) an instruction that describes the task to be solved, consistent with previous works on ELI5 Gao et al. (2023); Qi et al. (2024), (2) a context that contains the information for solving the task, but _no gold document_ is marked, and (3) a question that describes the specific query. At the end of the prompt, we append an exemplar output that gives the ground truth answer to the question for evaluating the likelihood of the answer.

Figure 7: An example prompt and LM output on NQ-Open. The prompt comprises (1) an instruction that describes the task to be solved, (2) a context that contains the information for solving the task, in which the gold document contains the ground truth answer, and (3) a question that describes the specific query. At the end of the prompt, we append an exemplar output that gives the ground truth answer to the question for evaluating the likelihood of the answer.

 

* "74602160-8/04f4-4zb-5f7a-e2a313b649892",  "51f95eb81-f114-abd-xbc-d19e0581fcda+",  "6618b4a-abb4-aa-438-acdc12a463";  ... "16jdb6c1-1682-d6b1-54be-7ef2fe63fd6ce^.
* ) Key: "749d28bd-8/04f4-4zb-87fa-e2a3106#99592" Value:

## Appendix E Full Results on NQ-Open

We show the full results on NQ-Open in Figures 10-13.

 

Figure 10: QA accuracy, PMI, and log odds ratio of answer likelihood on 10 documents.

 

Figure 11: QA accuracy, PMI, and log odds ratio of answer likelihood on 10 documents.

 

Figure 12: QA accuracy, PMI, and log odds ratio of answer likelihood on 20 documents.

 


* [ENDFIGURE]

Figure 14: Example input for key–value retrieval task.

Figure 13: QA accuracy, PMI, and log odds ratio of answer likelihood on 30 documents.

 

Proof of Proposition 2.1

**Proposition 2.1**.: _Under assumptions given in Assumption 2.1, we have_

\[\begin{split}&\log\frac{\overrightarrow{p}(a\mid q\cdot c_{D}( \pi))}{1-\overrightarrow{p}(a\mid q\cdot c_{D}(\pi))}\\ &=\text{PMI}(q,c_{D}(\pi))+C(a,c_{D}(\pi))\end{split} \tag{6}\]

_for an answer-dependent constant \(C(a,c_{D}(\pi))\)._

Proof.: First note that, by Bayes' rule, we have

\[\overrightarrow{p}(a\mid c_{D}(\pi)\mathbin{\boldsymbol{\cdot}}q)=\frac{ \overrightarrow{p}(\Box a\mid c_{D}(\pi))\overrightarrow{p}(q\mid c_{D}(\pi) \Box a)}{\overrightarrow{p}(c_{D}(\pi)\mathbf{\cdot}q)}. \tag{17}\]

Then,

\[\log \frac{\overrightarrow{p}(a\mid q\cdot c_{D}(\pi))}{1- \overrightarrow{p}(a\mid q\cdot c_{D}(\pi))}=\log\frac{\overrightarrow{p}(a \mid q\mathbin{\boldsymbol{\cdot}}c_{D}(\pi))}{\sum_{\tilde{a}\in\Sigma^{*} }\mathbbm{1}\{\tilde{a}\not\preceq a\}\overrightarrow{p}(\bar{a}\mid q \mathbin{\boldsymbol{\cdot}}c_{D}(\pi))} \tag{18a}\] \[=\log\frac{\overrightarrow{p}(\Box a\mid c_{D}(\pi)) \overrightarrow{p}(q|c_{D}(\pi)\square a)}{\left(\sum_{\tilde{a}\in\Sigma^{*} }\mathbbm{1}\{\tilde{a}\not\preceq a\}\overrightarrow{p}(\Box\tilde{a}\mid c _{D}(\pi))\right)\overrightarrow{p}(q)}\] (Bayes' rule) (18b) \[=\log\frac{\overrightarrow{p}(\Box a\mid c_{D}(\pi)) \overrightarrow{p}(q\mid c_{D}(\pi)\Box a)}{\sum_{\tilde{a}\in\Sigma^{*}} \mathbbm{1}\{\tilde{a}\not\preceq a\}\overrightarrow{p}(\Box\tilde{a}\mid c _{D}(\pi))\overrightarrow{p}(q\mid c_{D}(\pi)\Box\tilde{a})}\] (18c) \[=\log\frac{\overrightarrow{p}(\Box a\mid c_{D}(\pi)) \overrightarrow{p}(q\mid c_{D}(\pi))}{\left(\sum_{\tilde{a}\in\Sigma^{*}} \mathbbm{1}\{\tilde{a}\not\preceq a\}\overrightarrow{p}(\Box\tilde{a}\mid c _{D}(\pi))\right)\overrightarrow{p}(q)}\] (Assumption 2.1) (18d) \[=\log\frac{\overrightarrow{p}(\Box a\mid c_{D}(\pi))}{\sum_{ \tilde{a}\in\Sigma^{*}}\mathbbm{1}\{\tilde{a}\not\preceq a\}\overrightarrow{ p}(\Box\tilde{a}\mid c_{D}(\pi))}\overrightarrow{p}(q\mid c_{D}(\pi))\] (18e) \[=\underbrace{\log\frac{\overrightarrow{p}(\Box a\mid c_{D}(\pi)) }{\sum_{\tilde{a}\in\Sigma^{*}}\mathbbm{1}\{\tilde{a}\not\preceq a\} \overrightarrow{p}(\Box\tilde{a}\mid c_{D}(\pi))}}_{\triangleq C(a,c_{D}(\pi))}+ \log\frac{\overrightarrow{p}(q\mid c_{D}(\pi))}{\overrightarrow{p}(q)}\] (18f) \[=\text{PMI}(q,c_{D}(\pi))+C(a,c_{D}(\pi)), \tag{18g}\]

where \(C(a,c_{D}(\pi))\) is constant with respect to \(q\)