SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence
Authors: Zhining Liu, Rana Ali Amjad, Ravinarayana Adkathimar, Tianxin Wei, Hanghang Tong
Accepted at: acl25

SELFELICIT: Your Language Model Secretly Knows Where is the Relevant Evidence

Zhining Liu1, Rana Ali Amjad2, Ravinarayana Adkathimari2, Tianxin Wei1, Hanghang Tong1
1University of Illinois Urbana-Champaign, 2Amazon Science
{liu326, twei10, htong}@illinois.edu {raamjad,adkathi}@amazon.com

Abstract

Providing Language Models (LMs) with relevant evidence in the context (either via retrieval or user-provided) can significantly improve their ability to provide better-grounded responses. However, recent studies have found that LMs often struggle to fully comprehend and utilize key evidence from the context, especially when it contains noise and irrelevant information—an issue common in real-world scenarios. To address this, we propose SELFELICIT, an inference-time approach that helps LMs focus on key contextual evidence through self-guided explicit highlighting. By leveraging the inherent evidence-finding capabilities of LMs using the attention scores of deeper layers, our method automatically identifies and emphasizes key evidence within the input context, facilitating more accurate and grounded responses without additional training or iterative prompting. We demonstrate that SELFELICIT brings consistent and significant improvement on multiple evidence-based QA tasks for various LM families while maintaining computational efficiency. Our code and documentation are available at https://github.com/ZhiningLiu1998/SelfElicit.

1 Introduction

Language models have demonstrated remarkable capabilities (OpenAI, 2022, 2023; Kaddour et al., 2023), yet hallucinations and factual inaccuracies remain a significant limitation (Ji et al., 2023; Wang et al., 2023a). Furnishing the input context with relevant evidence passage is a popular approach to enhance LM’s ability to generate correct and well-grounded responses (Ram et al., 2023; Jiang et al., 2023b; Wei et al., 2025b). However, recent studies find that LMs can fail to properly leverage supporting facts within context, leading to incorrect answers despite available evidence (Shi et al., 2024; Zhao et al., 2024). Such failure largely stems from the noise and irrelevant information in context (Wu et al., 2024; Cuconasau et al., 2024), which is usually inevitable in practice (Gao et al., 2023).

Recently, improved prompting (Zhou et al., 2023) or decoding (Shi et al., 2024) methods have been proposed to enhance the focus on context. However, they treat the whole context as a single entity, overlooking the fact that not all information provided in the context is important. Meanwhile, another line of work highlights the crucial role of attention mechanisms in identifying key contextual information, leveraging attention patterns for KV cache compression (Li et al., 2024a) and document reordering (Peysakhovich and Lerer, 2023). Motivated by these insights, we delve deeper into the capabilities of attention-based mechanisms for fine-grained, inference-time evidence highlighting,

![SelfElicit workflow on a real example with Llama3.1-8B. By locating and explicitly highlighting the initially overlooked 2nd-hop evidence ("SAS was founded in 1941 ...") within the context, SelfElicit guides the model to arrive at the correct answer "1941".](page_1012_370_388_388.png)

Figure 1: SelfElicit workflow on a real example with Llama3.1-8B. By locating and explicitly highlighting the initially overlooked 2nd-hop evidence ("SAS was founded in 1941 ...") within the context, SelfElicit guides the model to arrive at the correct answer "1941".
enhancing LMs’ ability to concentrate on key supporting information within the context. Our contributions in this work are as follows:

• By analyzing the attention scores during response generation, we demonstrate that the LMs have an inherent ability to identify the relevant evidence in the context, regardless of whether they respond correctly or not. This observation holds across various LM families.

• Leveraging this inherent ability, we propose an inference time context augmentation approach, SELFELICIT, that highlights the important evidence in the context and improves LM’s ability to provide correct and well-grounded responses. SELFELICIT is efficient for inference, training-free, and robust to noise and hyper-parameter choice.

• In a comprehensive study, we show the significant performance improvement that SELFELICIT brings across model families and tasks. We also offer extensive studies and analyses on various characteristics of SELFELICIT, including but not limited to its evidence-eliciting accuracy, effect of different design choices, robustness under noisy/informative contexts, runtime efficiency, etc.

Fig. 1 illustrates the SELFELICIT workflow. By identifying and highlighting key evidence within the context, SELFELICIT guides the LM towards important information that it might otherwise overlook, thus leading to the correct response.

2 Preliminaries

Problem description. Given an LM \( \Phi \), question q, context c and QA prompt template \( \tau_{QA} \), we obtain the generated answer g for the question by combining context and question as input: \( g \leftarrow \Phi(\tau_{QA}(c, q)) \). We use the following template as baseline to test different LMs’ base ability in leveraging contextual evidence for QA tasks:

A direct prompt template \( \tau_{QA} \) for context-based QA
Directly answer the question based on the context passage, no explanation is needed. If the context does not contain any evidence, output “I cannot answer based on the given context.”
Context: {context}
Question: {question}

Notation. We only outline the notation and details of Transformer models needed for this work, please see (Vaswani et al., 2017) for more detailed exposition on transformer architecture. Given an input sequence with n tokens, a decoder only transformer LM generates next token by adaptively attending to all previous n tokens. Denote the attention probability vector for attention head h and layer \( \ell \) by \( \mathbf{a}^{(\ell,h)} \in \mathbb{R}^n \). Then we define

\[
\mathbf{a}^{(\ell)} = \frac{1}{H} \sum_{h=1}^H \mathbf{a}^{(\ell,h)}
\]

where H is the total number of heads in a layer and L is the number of layers in the transformer model. Intuitively, \( \mathbf{a}^{(\ell)} \) provides a holistic view of layer \( \ell \)'s attention distribution over the input sequence, summarizing which tokens the model deems most informative at this stage of processing.

3 SELFELICIT

SELFELICIT is an inference-time context augmentation framework to enhance generative LM’s capability to rely on relevant information present in the context in order to reduce hallucinations and provide factually correct grounded response. SELFELICIT requires no additional training, is efficient at inference time and provides significant performance improvement across models and tasks.

3.1 Self-guided Contextual Evidence Eliciting

The first step in SELFELICIT is to leverage the LM’s internal representations to identify the relevant evidence sentences in the context. Consider the context-based QA setup described in Sec. 2. Denote the number of tokens in input \( \tau(c, q) \) by n and the number of sentences\( ^1 \) in the context c by m. Define \( S : \{ s_1, s_2, \cdots, s_m \} \) where \( s_i \) denotes i-th sentence. Let \( (t_{s_i}^{\text{start}}, t_{s_i}^{\text{end}}) \) denote the start and end token index of i-th sentence. Define sentence-level attention vector \( \hat{\mathbf{a}}^{(\ell)} \in \mathbb{R}^m \) from token level attention \( \mathbf{a}^{(\ell)} \) as follows

\[
\hat{\mathbf{a}}^{(\ell)} := [\hat{a}_1^{(\ell)}, \hat{a}_2^{(\ell)}, \cdots, \hat{a}_m^{(\ell)}] \in \mathbb{R}^m,
\]
where
\[
\hat{a}_i^{(\ell)} = \frac{1}{|t_{s_i}^{\text{end}} - t_{s_i}^{\text{start}} + 1|} \sum_{j=t_{s_i}^{\text{start}}}^{t_{s_i}^{\text{end}}} \mathbf{a}_j^{(\ell)}.
\]

Intuitively \( \hat{a}_i^{(\ell)} \) tells the relative importance of each context sentence \( s_i \) at layer \( \ell \). In Fig. 2 we plot

\( ^1 \)The sentences in a document can be extracted with many natural language processing packages, in this paper we use spaCy (Honnibal and Montani, 2017).
Figure 2: Relative attention3 to the evidence/non-evidence sections (y-axis) across the layers (x-axis) for different LM families on HotpotQA. Deeper layers pay much greater attention to crucial evidence (green lines) in the context, even when LM responds incorrectly (dashed lines). Best viewed in color.

how \( \bar{a}_i^{(\ell)} \) varies across layers for evidence vs non-evidence sentences in the context when generating the first response token. We can clearly see that, regardless of whether the model responds correctly or not, the deeper layers of the LM pay significantly higher attention to the relevant evidence2 in the context. This observation holds across models families and datasets, see Appendix B.2.

We leverage this insight to define the sentence evidence scores \( e_i \) for each sentence \( s_i \in S \), which we use at the inference time to predict the evidence containing sentences in the context, as follows: We use a pre-selected subset of layers, denoted by \( \mathcal{L}_{ER} \) and referred to as evidence-reading layers, and aggregate \( \bar{a}_i^{(\ell)} \) to get \( e_i \), i.e.,

\[
e := [e_1, e_2, \cdots, e_m] \in \mathbb{R}^m,
\]
where
\[
e_i = \frac{1}{|\mathcal{L}_{ER}|} \sum_{\ell \in \mathcal{L}_{ER}} \bar{a}_i^{(\ell)}
\]

In Section 4.3 (RQ4) we study different choices of \( \mathcal{L}_{ER} \) for their ability to identify the relevant evidence. We find that using the last 50% of the layers as \( \mathcal{L}_{ER} \) performs consistently well across models and datasets so we use it as default for experiments.

Next, we introduce an evidence thresholding parameter \( \alpha \in [0, 1] \) and use it in combination \( e_i \) to predict evidence sentences \( S_{SE} \)

\[
S_{SE} = \{s_i | s_i \in S; e_i \geq \alpha \cdot \max(e)\},
\]

A higher value of \( \alpha \) will result in fewer sentences being highlighted, and vice versa. It can be seen as a trade-off parameter between precision and recall in evidence elicitation, which can be tuned based on task requirements. SELFELICIT is generally robust to \( \alpha \), see Section 4.3 (RQ5), with \( \alpha = 0.5 \) yielding significant performance improvement over baseline across models and tasks. Therefore, We use \( \alpha = 0.5 \) as our default setting for experiments.

We note that similar analyses can be presented at a token level to identify evidence tokens instead of sentences. However, as discussed in Appendix B.4, augmenting the original context by highlighting complete sentences is semantically more meaningful and beneficial than individual tokens.

3.2 Contextual Evidence Highlighting

The second step of SELFELICIT is to modify the original context to highlight the evidence sentences identified in Sec. 3.1 and to modify the original instructions in Sec. 2 to guide the model to use the highlighted evidence. Prioritizing simplicity and efficiency, we introduce a prompt augmentation strategy to achieve this. Specifically for highlighting, in the raw context passage \( c \) we place text markers "<start_important>" and "<end_important>" before and after each sentence in \( S_{SE} \), resulting in the new context \( c^* \). Furthermore, we update the task instructions to guide the LM’s attention towards the highlighted sentences. The response is generated using the mark-highlighted \( c^* \) and the updated prompt template \( \tau_{SEQA} \).

SELFELICIT Prompt Template \( \tau_{SEQA} \)

{Original QA Instructions} Within the context, <start_important> and <end_important> are used to mark the important evidence sentences. read carefully. Do not include the markers in the output.
Context: {elicited_context}
Question: {question}

The overall procedure is summarized in Algorithm 1. We note that SELFELICIT is designed to prioritize efficiency and simplicity, generating only
one additional token during inference to identify \( S_{SE} \) (line 1 in Alg. 1). The highlighting method also outperforms more complex strategies, detailed further in Appendices B.5 and B.7.

Algorithm 1 SELFELICIT

Input Language model \( \Phi \), question \( q \), context \( c \), prompt templates \( \tau_{QA} \) and \( \tau_{SEQA} \), evidence-reading layers \( \mathcal{L}_{ER} \), eliciting threshold \( \alpha \).
1: Generate one token with \( \Phi(\tau_{QA}(c, q)) \);
2: Compute \( \hat{a}^{(l)} \) for all \( l \in \mathcal{L}_{ER} \); \( \triangleright \) Eq. (2)
3: Get evidence score \( e \) with \( \mathcal{L}_{ER} \); \( \triangleright \) Eq. (3)
4: Select evidence sentences \( S_{SE} \); \( \triangleright \) Eq. (4)
5: Derive new context \( c^* \) by highlighting \( S_{SE} \);
6: Generate answer \( g_{SE} \leftarrow \Phi(\tau_{SEQA}(c^*, q)) \)

Output: The final answer \( g_{SE} \)

4 Experiments

We conduct comprehensive experiments across six LMs from different families with varying sizes and on four single- and multi-hop reasoning open-book QA tasks from various domains to investigate:

• RQ1: How does SELFELICIT perform in terms of improving answer quality and accuracy?
• RQ2: How do the evidence scores and the context sentences highlighted by SELFELICIT correlate with relevant evidence?
• RQ3: Robustness to noise in the context?
• RQ4&5: How do different choices of evidence-reading layers & threshold \( \alpha \) affect SELFELICIT?

4.1 Experimental Setup

Datasets and Metrics. We test 4 datasets: HotpotQA (Yang et al., 2018) and the MRQA version (Fisch et al., 2019) of NewsQA (Trischler et al., 2017), TriviaQA (TQA) (Joshi et al., 2017), and Natural Questions (NQ) (Kwiatkowski et al., 2019). These datasets feature context passages from diverse sources (e.g., web/Wikipedia/news reports), requiring the model to reason over a single or multiple pieces of evidence within the context. This provides a comprehensive test of SELFELICIT in real-world applications. For all datasets, we use the official validation split on HuggingFace for testing. We apply greedy decoding to get deterministic answers. Exact Match (EM) and Token-level F1 scores are used for QA performance evaluation.

Models and Baselines. We test six open-source instruction fine-tuned models: Llama-3.1 (8B, 70B) (Dubey et al., 2024), Mistral (7B, 12B) (Jiang et al., 2023a), and Qwen2.5 (7B, 32B) (Yang et al., 2024). We compare SELFELICIT to the following prompting/evidence-elicitating approaches:

• COT (Wei et al., 2022a): Chain-of-thought prompting encourages the model to reason through intermediate steps before reaching a final answer. While CoT promotes step-by-step reasoning based on the evidence in the context, it does not explicitly highlight important information within the context. Therefore, we use it as a natural baseline for validating the advantage of explicit evidence elicitation in SELFELICIT. We implement it by adding the COT prompt "Think step by step to provide the answer." at the end of the instruction.

• FULLELICIT: A naive approach that highlights the entire context as important. Comparing with it shows the necessity of fine-grained, sentence-level evidence elicitation.

• PROMPTELICIT: This method leverages the LM itself for generative evidence extraction. It involves two steps: first, the LM is prompted to select the most relevant evidence sentences from the context passage that can help answer the question. Then, we highlight the extracted evidence and get new context for QA following Section 3.2. Note that this method involves generative evidence extraction in iterative prompting, which requires the model to generate a large amount of additional tokens. This serves as a strong baseline for evaluating whether the quality of evidence selected by SELFELICIT can match that of evidence extracted through generative approach. We use the following prompt for generative evidence extraction step:

PROMPTELICIT Prompt Template \( \tau_{PE} \)

Please find the supporting evidence sentences from the context for the question, then copy-paste the original text to output. Template for output: '- [sentence1] - [sentence2] ...'
Context: {context}
Question: {question}

Since our focus is on analyzing and improving the inherent ability of the generative LM to provide factually grounded responses based on relevant information provided in the context, we do not compare to any dataset specific and task specific fine-tuning methods. For a fair comparison, we do not tune the hyper-parameters for SELFELICIT. They are fixed as \( \mathcal{L}_{ER} \) being the last 50% layers and \( \alpha = 0.5 \).
Table 1: Results of applying different evidence-eliciting methods to 6 LMs across 4 context-based QA tasks. We report EM and Token F1 scores (in \( \times 10^{-2} \)) with the gains over direct QA. The "average" columns present the average QA performance and the inference time (per sample) across all datasets. The best results are bolded.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="4">Dataset</th>
    <th colspan="3">Average</th>
  </tr>
  <tr>
    <th>HotpotQA<br>EM<br>Token F1</th>
    <th>NewsQA<br>EM<br>Token F1</th>
    <th>TQA<br>EM<br>Token F1</th>
    <th>NQ<br>EM<br>Token F1</th>
    <th>Ranking<br>EM<br>Token F1</th>
    <th>Inference Time (ms)</th>
  </tr>
  <tr>
    <td rowspan="5">Llama-3.1</td>
    <td>BASE</td>
    <td>58.9</td>
    <td>57.7</td>
    <td>64.3</td>
    <td>56.2</td>
    <td>72.8</td>
    <td>66.1</td>
    <td>59.7</td>
    <td>61.6</td>
    <td>4.38</td>
    <td>4.75</td>
    <td>224.1</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>60.4</td>
    <td>58.6</td>
    <td>64.9</td>
    <td>55.8</td>
    <td>74.4</td>
    <td>67.4</td>
    <td>59.6</td>
    <td>62.1</td>
    <td>3.75</td>
    <td>4.00</td>
    <td>224.8</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>60.7</td>
    <td>59.0</td>
    <td>65.9</td>
    <td>56.6</td>
    <td>72.8</td>
    <td>66.3</td>
    <td>61.1</td>
    <td>62.5</td>
    <td>3.12</td>
    <td>3.25</td>
    <td>226.3</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>66.3</td>
    <td>66.3</td>
    <td>62.8</td>
    <td>56.7</td>
    <td>76.0</td>
    <td>69.6</td>
    <td>61.8</td>
    <td>65.6</td>
    <td>2.75</td>
    <td>2.00</td>
    <td>1672.0</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td><b>68.5</b></td>
    <td><b>69.5</b></td>
    <td><b>66.9</b></td>
    <td><b>60.8</b></td>
    <td><b>79.4</b></td>
    <td><b>72.7</b></td>
    <td><b>64.0</b></td>
    <td><b>67.8</b></td>
    <td><b>1.00</b></td>
    <td><b>1.00</b></td>
    <td><b>264.1</b></td>
  </tr>
  <tr>
    <td rowspan="5">70B</td>
    <td>BASE</td>
    <td>71.8</td>
    <td>74.2</td>
    <td>66.7</td>
    <td>57.4</td>
    <td>78.0</td>
    <td>71.2</td>
    <td>59.3</td>
    <td>63.2</td>
    <td>3.25</td>
    <td>3.00</td>
    <td>1389.8</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>72.4</td>
    <td>74.0</td>
    <td>67.2</td>
    <td>56.4</td>
    <td>77.0</td>
    <td>69.9</td>
    <td>60.3</td>
    <td>63.1</td>
    <td>3.12</td>
    <td>4.50</td>
    <td>1394.2</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>71.3</td>
    <td>73.8</td>
    <td>66.2</td>
    <td>56.8</td>
    <td>77.5</td>
    <td>70.7</td>
    <td>58.2</td>
    <td>61.8</td>
    <td>4.25</td>
    <td>4.50</td>
    <td>1408.1</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>73.4</td>
    <td>76.2</td>
    <td>63.1</td>
    <td>58.2</td>
    <td>77.0</td>
    <td>72.3</td>
    <td>64.0</td>
    <td>68.5</td>
    <td>3.38</td>
    <td>2.00</td>
    <td>8124.0</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td><b>75.9</b></td>
    <td><b>79.0</b></td>
    <td><b>69.2</b></td>
    <td><b>62.1</b></td>
    <td><b>80.0</b></td>
    <td><b>74.4</b></td>
    <td><b>65.4</b></td>
    <td><b>68.7</b></td>
    <td><b>1.00</b></td>
    <td><b>1.00</b></td>
    <td><b>1566.9</b></td>
  </tr>
  <tr>
    <td rowspan="5">7B</td>
    <td>BASE</td>
    <td>70.4</td>
    <td>45.6</td>
    <td>61.4</td>
    <td>32.6</td>
    <td>81.8</td>
    <td>47.9</td>
    <td>65.7</td>
    <td>29.9</td>
    <td>3.75</td>
    <td>4.25</td>
    <td>538.4</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>71.4</td>
    <td>44.0</td>
    <td>60.4</td>
    <td>31.4</td>
    <td>82.2</td>
    <td>48.2</td>
    <td>67.5</td>
    <td>29.3</td>
    <td>2.25</td>
    <td>4.75</td>
    <td>560.8</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>70.6</td>
    <td>47.6</td>
    <td>62.0</td>
    <td>34.0</td>
    <td>81.3</td>
    <td>51.5</td>
    <td>66.8</td>
    <td>30.5</td>
    <td>3.00</td>
    <td>3.00</td>
    <td>541.4</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>71.0</td>
    <td>64.1</td>
    <td>57.7</td>
    <td>42.2</td>
    <td>81.9</td>
    <td>62.3</td>
    <td>65.6</td>
    <td>42.8</td>
    <td>4.00</td>
    <td>1.25</td>
    <td>1877.8</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td><b>74.3</b></td>
    <td><b>61.6</b></td>
    <td><b>60.6</b></td>
    <td><b>41.7</b></td>
    <td><b>83.6</b></td>
    <td><b>61.3</b></td>
    <td><b>66.4</b></td>
    <td><b>43.4</b></td>
    <td><b>2.00</b></td>
    <td><b>1.75</b></td>
    <td><b>431.3</b></td>
  </tr>
  <tr>
    <td rowspan="5">12B</td>
    <td>BASE</td>
    <td>59.2</td>
    <td>65.9</td>
    <td>51.9</td>
    <td>51.6</td>
    <td>72.9</td>
    <td>68.7</td>
    <td>54.8</td>
    <td>61.6</td>
    <td>5.00</td>
    <td>4.75</td>
    <td>281.9</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>62.0</td>
    <td>69.0</td>
    <td>53.0</td>
    <td>52.6</td>
    <td>75.6</td>
    <td>71.1</td>
    <td>55.3</td>
    <td>62.0</td>
    <td>3.25</td>
    <td>3.25</td>
    <td>284.6</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>59.8</td>
    <td>65.8</td>
    <td>53.1</td>
    <td>51.9</td>
    <td>73.7</td>
    <td>68.8</td>
    <td>55.5</td>
    <td>62.6</td>
    <td>2.88</td>
    <td>4.00</td>
    <td>283.9</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>62.8</td>
    <td>73.1</td>
    <td>52.2</td>
    <td>56.6</td>
    <td>79.9</td>
    <td>77.6</td>
    <td>55.5</td>
    <td>65.7</td>
    <td>2.38</td>
    <td>1.75</td>
    <td>1455.0</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td><b>63.6</b></td>
    <td><b>72.9</b></td>
    <td><b>54.9</b></td>
    <td><b>58.6</b></td>
    <td><b>82.6</b></td>
    <td><b>79.9</b></td>
    <td><b>55.3</b></td>
    <td><b>66.0</b></td>
    <td><b>1.50</b></td>
    <td><b>1.25</b></td>
    <td><b>339.1</b></td>
  </tr>
  <tr>
    <td rowspan="5">7B</td>
    <td>BASE</td>
    <td>65.2</td>
    <td>65.8</td>
    <td>58.3</td>
    <td>45.4</td>
    <td>77.4</td>
    <td>66.1</td>
    <td>62.2</td>
    <td>59.9</td>
    <td>3.75</td>
    <td>3.75</td>
    <td>245.2</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>70.7</td>
    <td>37.9</td>
    <td>59.2</td>
    <td>31.9</td>
    <td>78.6</td>
    <td>41.6</td>
    <td>63.8</td>
    <td>32.3</td>
    <td>2.00</td>
    <td>5.00</td>
    <td>421.5</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>65.5</td>
    <td>65.7</td>
    <td>57.5</td>
    <td>48.2</td>
    <td>77.1</td>
    <td>67.3</td>
    <td>64.4</td>
    <td>60.6</td>
    <td>3.50</td>
    <td>2.75</td>
    <td>249.6</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>64.7</td>
    <td>67.7</td>
    <td>54.9</td>
    <td>46.3</td>
    <td>75.4</td>
    <td>66.8</td>
    <td>64.5</td>
    <td><b>65.0</b></td>
    <td>4.25</td>
    <td>2.25</td>
    <td>1165.1</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td><b>69.1</b></td>
    <td><b>71.4</b></td>
    <td><b>59.6</b></td>
    <td><b>50.8</b></td>
    <td><b>78.1</b></td>
    <td><b>67.8</b></td>
    <td><b>65.0</b></td>
    <td><b>64.7</b></td>
    <td><b>1.50</b></td>
    <td><b>1.25</b></td>
    <td><b>289.4</b></td>
  </tr>
  <tr>
    <td rowspan="5">32B</td>
    <td>BASE</td>
    <td>71.8</td>
    <td>68.4</td>
    <td>60.0</td>
    <td>44.7</td>
    <td>79.0</td>
    <td>69.3</td>
    <td>62.7</td>
    <td>59.3</td>
    <td>3.12</td>
    <td>3.25</td>
    <td>928.2</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>71.3</td>
    <td>67.1</td>
    <td>60.0</td>
    <td>43.5</td>
    <td>79.5</td>
    <td>66.8</td>
    <td>59.5</td>
    <td>55.3</td>
    <td>3.62</td>
    <td>5.00</td>
    <td>998.6</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>71.3</td>
    <td>68.2</td>
    <td>61.6</td>
    <td>45.7</td>
    <td>78.5</td>
    <td>68.8</td>
    <td>63.2</td>
    <td>58.1</td>
    <td>3.25</td>
    <td>3.75</td>
    <td>936.3</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>71.3</td>
    <td>74.5</td>
    <td>59.0</td>
    <td>51.5</td>
    <td>78.0</td>
    <td>69.9</td>
    <td>64.8</td>
    <td>68.1</td>
    <td>4.00</td>
    <td>2.00</td>
    <td>5109.8</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td><b>73.3</b></td>
    <td><b>75.0</b></td>
    <td><b>65.6</b></td>
    <td><b>57.3</b></td>
    <td><b>82.1</b></td>
    <td><b>74.8</b></td>
    <td><b>66.8</b></td>
    <td><b>69.8</b></td>
    <td><b>1.00</b></td>
    <td><b>1.00</b></td>
    <td><b>980.7</b></td>
  </tr>
</table>

4.2 Main Results

RQ1: SELFELICIT consistently improves grounded factuality. Table 1 compares SELFELICIT to the other methods. We observe that

• As expected FULLELICIT does not provide meaningful and consistent improvement since it doesn’t elicit fine-grained evidence.

• COT also does not provide a meaningful and consistent gain which highlights the fact that asking the model to reason carefully does not improve its ability to leverage the relevant evidence while generating the response. COT results also show variation across LM families where we don’t observe meaningful gains for Llama and Mistral models. When combined with Qwen models, COT generates significantly longer answers with intermediate steps. While this results in a slight increase in EM scores in a few cases, it also leads to a large drop (up to 27.9) in F1 scores due to the inclusion of redundant information, as well as longer inference times (e.g., 171.9% more inference time for Qwen2.5-7B).

• SELFELICIT significantly and consistently improves the performance across all datasets and models of different sizes (5.0%-11.7% gain over baseline). Even when compared to computationally expensive (average inference time increase of 878%/939% for Llama3.1-8B/70B) iterative prompting approach PROMPTELICIT, SELFELICIT outperforms for 40 out of 48 model-task metric pairs while incurring a fraction of the computational cost increment (only ~3-5% increase when compared to PROMPTELICIT).

Appendix B.1 provides a more detailed discussion of computational efficiency of the approaches.

RQ2: SELFELICIT highlights relevant evidence. We demonstrate both qualitatively and quantitatively how the context highlighted by SELFELICIT is the relevant evidence for the task which leads to the performance gains reported in Table 1.
Table 2: Examples of how SELFELICIT helps LM, with blue text highlighting the evidence selected by SELFELICIT. Due to space limitation we only display a small portion of the full context, please see Appendix B.8 for full context.

<table>
  <tr>
    <th>(Partial) Context Passage</th>
    <th>Question & Answers</th>
  </tr>
  <tr>
    <td>(Displaying 123 of 794 Words) Tiger Please is an Indie / Alternative five-piece band from Cardiff, Wales. The band formed in August 2008. The band's influences are U2, Sigur Rós, Kings of Leon, John Mayer and Counting Crows. They signed with Walnut Tree Records in 2009 and released their debut EP "They Don't Change Under Moonlight". "Kerrang!" magazine, "Rock Sound" magazine, and "Classic Rock" magazine praised the EP and featured the band on the "Rock Sound" and "Classic Rock" cover-mount albums. ... ... Black Rebel Motorcycle Club (often abbreviated as BRMC) is an American rock band from San Francisco, California. The group consists of Peter Hayes (vocal, guitar, harmonica), Robert Levon Been (vocal, bass, guitar), and Leah Shapiro (drums). Former drummer Nick Jago left the band in 2008 ... ...</td>
    <td>Question: Are the bands Tiger Please and Black Rebel Motorcycle Club from the same country?<br>True Answer: No.<br>Base: Yes. X<br><span style="color:blue;">*SELFELICIT*: No. ✓</span></td>
  </tr>
  <tr>
    <td>(Displaying 129 of 227 Words) Home Monthly was a monthly women's magazine published in Pittsburgh, Pennsylvania in the late 19th century. The Strategy of the Were-Wolf Dog is a short story by Willa Cather. It was first published in "Home Monthly" in December 1896. The Count of Crow's Nest is a short story by Willa Cather. It was first published in "Home Monthly" in October 1896. Mirabella was a women's magazine published from June 1899 to April 2000. It was created by and named for Grace Mirabella, a former "Vogue" editor in chief, in partnership with Rupert Murdoch. "Mirabella" and "Willa Cather" are short stories by Willa Cather. "The Count" on 5/1/21, 1897 and the month later in "Home Monthly". "The Prodigies" is a short story by Willa Cather ... ...</td>
    <td>Question: Which women's magazine was published first, Mirabella or Home Monthly?<br>True Answer: Home Monthly.<br>Base: Mirabella. X<br><span style="color:blue;">*SELFELICIT*: Home Monthly. ✓</span></td>
  </tr>
  <tr>
    <td>(Displaying 153 of 1014 Words) ... Lars Lunde (born 21 March 1964) is a Danish former professional football player, who played in the striker position. Lunde got his breakthrough with Brøndby IF in 1983, and he made his debut for the Denmark national football team in October 1984. He was loaned to Young Boys Berne in Switzerland before moving to German club Bayern Munich in 1986. He was a part of the Bayern Munich squad that won the German Bundesliga championship in 1987, and he came on as a late substitute when Bayern lost the 1987 European Cup Final to FC Porto. He played the last of his three matches for the Danish national team in April 1987, before leaving Bayern during the 1987–88 season. He went on to play for a number of smaller clubs, ending his career with FC Baden in Switzerland. ... ...</td>
    <td>Question: Which team did Lars Lunde play for when defeated for the 1987 European Cup Final?<br>True Answer: Bayern Munich<br>Base: FC Porto. X<br><span style="color:blue;">*SELFELICIT*: Bayern Munich. ✓</span></td>
  </tr>
  <tr>
    <td>(Displaying 146 of 532 Words) ... This sent Olympic down to play in the Premier League in 2007. Adelaide City won the title with games to spare after being runaway leaders, finishing the season unbeaten. Norwood is a suburb of Adelaide, about 4 km east of the Adelaide city centre. The suburb is in the City of Norwood Payneham & St Peters, the oldest South Australian local government municipality, with a city population over 34,000. Whyalla railway station was the terminus station of the Whyalla line serving the South Australian city of Whyalla. Walter Frank Giffen (20 September 1861 in Norwood – 28 June 1949 in Adelaide) was an Australian cricketer who played in 3 Tests between 1887 and 1892. He was the brother of the great all-rounder George Giffen. The City of Burnside is a local government area with an estimated population of 44,300 people in the South Australian city of Adelaide. ... ...</td>
    <td>Question: Walter Giffen is from a suburb of which South Australian city?<br>True Answer: Adelaide.<br>Base: Norwood. X<br><span style="color:blue;">*SELFELICIT*: Adelaide. ✓</span></td>
  </tr>
</table>

For qualitative illustration, in Table 2 we show various types of examples (e.g., true/false, comparison, fact retrieval, multi-hop reasoning) from the HotpotQA dataset. SELFELICIT identifies the most relevant supporting facts across different QA tasks. For instance, in the 2nd "comparison" example that asks which magazine was published first, SELFELICIT highlights the information stating the publication dates of the magazines and related articles. In the 4th "multi-hop reasoning" example, it emphasizes the overlooked 2nd-hop evidence “Norwood is a suburb of Adelaide”, which helps the model arrive at the correct answer “Adelaide”.

Table 3: SELFELICIT accurately identifies contextual evidence sentences across different datasets and models.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="3">Model</th>
  </tr>
  <tr>
    <th>Llama3.1</th>
    <th>Mistral</th>
    <th>Qwen2.5</th>
  </tr>
  <tr>
    <th colspan="4">Metric: Sentence-level AUROC</th>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>91.24</td>
    <td>85.35</td>
    <td>88.21</td>
  </tr>
  <tr>
    <td>NewsQA</td>
    <td>92.68</td>
    <td>88.68</td>
    <td>91.54</td>
  </tr>
  <tr>
    <td>TQA</td>
    <td>73.27</td>
    <td>68.89</td>
    <td>70.59</td>
  </tr>
  <tr>
    <td>NQ</td>
    <td>90.87</td>
    <td>85.51</td>
    <td>87.43</td>
  </tr>
  <tr>
    <th colspan="4">Metric: Sentence-level NDCG</th>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>91.36</td>
    <td>82.45</td>
    <td>87.05</td>
  </tr>
  <tr>
    <td>NewsQA</td>
    <td>82.79</td>
    <td>70.65</td>
    <td>82.37</td>
  </tr>
  <tr>
    <td>TQA</td>
    <td>66.41</td>
    <td>63.32</td>
    <td>67.19</td>
  </tr>
  <tr>
    <td>NQ</td>
    <td>91.45</td>
    <td>86.45</td>
    <td>87.65</td>
  </tr>
</table>

For quantitative evaluation, we assess the accuracy of evidence elicitation by checking whether SELFELICIT assigns higher evidence scores to ground-truth evidence sentences. Specifically, for HotpotQA, we use the "supporting_facts" annotations to derive ground-truth evidence labels, while for other datasets, a sentence is treated as ground truth evidence if it contains at least one of the correct answers. Since SELFELICIT computes continuous evidence scores for each sentence (see Eq. (3)), we utilize AUROC and NDCG@all to assess the score for accurately classifying/ranking contextual evidence. Table 3 shows that SE can accurately locate evidence sentences across models and datasets, with 80-95 AUROC or NDCG scores in most cases. We note that results on the TQA dataset are relatively lower because TQA often has multiple candidate correct answers and thus a larger set of evidence. During inference, the model typically focuses on evidence for only one of the answers, which reduces the evidence score of other (candidate) evidence sentences. However, this does not hinder SELFELICIT’s ability to help LM respond better, as demonstrated by the results in Table 1.

4.3 Additional Analysis

RQ3: SELFELICIT is effective in presence of context noise. To explore the impact of real world noise on SELFELICIT we study it’s performance for the "distractor" variant of HotpotQA dataset containing additional distracting information, retrieved
(a) Performance of base and SELFELICIT-augmented LM with/without distracting context information, evaluated by Exact Match and Token F1 score.

(b) Context elicitation ratio of SELFELICIT with/without distracting context information.

Figure 3: SELFELICIT demonstrates robust advantage even in the presence of noisy context (Fig. 3(a)). When the context passages contain more distracting information, SELFELICIT tends to select a significantly smaller portion of text as evidence (Fig. 3(b)) to prevent the LM from being distracted by irrelevant contexts.

from Wikipedia (Yang et al., 2018), in the context. The distractor setting increases the average length of the context by 1443% by introducing irrelevant information. Figure 3(a) shows that SELFELICIT maintains the advantage over the baseline even in the presence of substantial context noise.

However, similar to the baseline, the performance of SELFELICIT for "distractor" variant worsens meaningfully when compared to SELFELICIT performance for the gold context setup, also shown in Figure 3(a). To investigate this we do a deep dive into the evidence elicitation capability of SELFELICIT in the presence of substantial noise. Figure 3(b) contrasts the elicit ratio (i.e., the proportion of elicited evidence within the original context) under gold and distractor settings. With gold context, SELFELICIT tends to select a larger proportion as evidence since most contextual information provided is relevant. By contrast, in the distractor setting where most of the context is irrelevant, the proportion of evidence selected decreases substantially (typically <10%) for the same \( \alpha = 0.5 \). This shows that even with a static \( \alpha \), SELFELICIT naturally exhibits adaptiveness by selecting a lower evidence ratio in noisy contexts, focusing only on the relevant evidence without distraction from noise.

RQ4: Deeper Layers are better for \( \mathcal{L}_{ER} \). We already saw in Fig. 2 that the deeper layers of all the model families exhibit higher attention scores and a clear ability to distinguish relevant evidence within the context. In Table 4 we verify this further empirically by comparing the evidence elicit accuracy and QA performance of seven different choices of \( \mathcal{L}_{ER} \) for Llama3.1-8B evaluated on HotpotQA with \( \alpha = 0.5 \). We see that, consistent with the observation in Fig. 2, choosing last 50% of the layers as \( \mathcal{L}_{ER} \) leads to (close to) best metrics in terms of both evidence elicitation and task performance. Based on the qualitative observation in Fig. 2, this choice also looks robust across model families which is then further verified by the results in Table 1 where we see a consistent significant improvement across all model families and tasks for this choice of \( \mathcal{L}_{ER} \). While there may be other choices of \( \mathcal{L}_{ER} \) which lead to better performance gains for a specific model-dataset pair, finding them requires model and task specific hyper-parameter tuning unlike universally applicable default setting.

<table>
  <tr>
    <th>ER Layer Span</th>
    <th>Elicit Accuracy<br>AUROC</th>
    <th>Elicit Accuracy<br>NDCG</th>
    <th>QA Performance<br>EM</th>
    <th>QA Performance<br>Token F1</th>
  </tr>
  <tr>
    <td>0%-100%</td>
    <td>89.02</td>
    <td>75.50</td>
    <td>62.57</td>
    <td>63.33</td>
  </tr>
  <tr>
    <td>0%-50%</td>
    <td>70.38</td>
    <td>44.99</td>
    <td>62.14</td>
    <td>62.65</td>
  </tr>
  <tr>
    <td>50%-100%</td>
    <td><b>91.55</b></td>
    <td><b>80.11</b></td>
    <td><b>64.86</b></td>
    <td><b>65.23</b></td>
  </tr>
  <tr>
    <td>0%-25%</td>
    <td>59.01</td>
    <td>37.55</td>
    <td>61.86</td>
    <td>61.83</td>
  </tr>
  <tr>
    <td>25%-50%</td>
    <td>74.82</td>
    <td>48.80</td>
    <td>62.57</td>
    <td>62.73</td>
  </tr>
  <tr>
    <td>50%-75%</td>
    <td><b>91.66</b></td>
    <td>79.96</td>
    <td>63.57</td>
    <td>64.14</td>
  </tr>
  <tr>
    <td>75%-100%</td>
    <td>91.02</td>
    <td>78.72</td>
    <td>63.43</td>
    <td>64.10</td>
  </tr>
</table>

Table 4: Effect of different evidence-reading (ER) layer choices on elicit accuracy and QA performance.

RQ5: Balancing elicitation precision and comprehensiveness in choice of \( \alpha \). The choice of \( \alpha \) in SELFELICIT acts as a proxy for trade-off between evidence elicitation precision vs coverage. For example, for \( \alpha = 1 \) we only select the sentence with highest \( e_i \) whereas for \( \alpha = 0 \) we have the whole context selected, analogue to FULLELICIT. In Figure 4 we show the impact of \( \alpha \) on the Token F1 score and evidence elicitation ratio for Llama3.1-8B on four QA datasets. We observe

• For all datasets the performance quickly rises and the evidence elicitation ratio quickly drops with increasing \( \alpha \) for smaller values of
Figure 4: Impact of elicit threshold \( \alpha \) (x-axis) on the QA performance gain (blue bars, left y-axis) and evidence elicit ratio (orange lines, right y-axis) of SELFELICIT on four QA tasks. Best viewed in color.

\( \alpha \) indicating clearly that evidence elicitation upto a threshold helps all datasets.
• Beyond \( \alpha = 0.5 \) there is relatively minor variation in the performance with change in \( \alpha \), illustrating robustness of SELFELICIT to choice of \( \alpha \in [0.5, 1] \) across datasets. The minor variation is dependent on the nature of the dataset. For datasets that require multi-hop reasoning, relying on multiple pieces of relevant evidence, we achieve optimal performance for \( \alpha \) closer to 0.5 whereas for NQ requiring simpler reasoning we achieve the best performance for \( \alpha = 1 \).

To strike a balanced trade-off between evidence elicitation precision and coverage, we choose \( \alpha = 0.5 \) as default for experiments which leads to consistent gains across models and datasets (Table 1).

5 Related Works

Context-based question answering. Furnishing LMs with relevant context is an effective way of providing up-to-date external and/or private knowledge (Ji et al., 2023; Asai et al., 2023) to help mitigate hallucination and improve response accuracy (Huang et al., 2023). Retrieval-augmented Generation (RAG) is a widely adopted paradigm for this purpose (Gao et al., 2023; Fan et al., 2024). Despite its popularity, recent studies have pointed out that context retrieved from external sources often contains noise and irrelevant information, leading to confusion for LM (Cuconasu et al., 2024; Wu et al., 2024; Yu et al., 2024). Motivated by this, we explore how to leverage contextual information more effectively at a finer granularity by highlighting critical information within the context. To the best of our knowledge, we are the first to investigate automated contextual evidence highlighting based on LM internal representations.

Factuality and internal representation. Recent studies have explored ways to understand QA factuality by analyzing the internal representations to identify important attention layers (Yukselgonul et al., 2024; Chen et al., 2024b) or heads (Halawi et al., 2024) that are crucial for generation correctness or hallucination. However, they primarily focus on how LMs utilize their parametric knowledge in controlled generation (Yukselgonul et al., 2024; Halawi et al., 2024) or closed-book QA (Chen et al., 2024b), with little discussion on context-based QA that rely on using external non-parametric knowledge. For context-based QA, Peyshakhovich and Lerer (2023) propose combating positional bias by iteratively ranking retrieved context documents based on average attention scores. Wang et al. (2023b) employs a fine-tuned Llama-2 evidence extractor to filter context information. Nevertheless, these works either employ document-level average attention over all layers and heads or necessitate training an auxiliary model for generative filtering. In contrast, our work delves into a detailed analysis of the behavior of each attention layer at the sentence level, enabling fine-grained and efficient evidence highlighting at inference time.

6 Conclusion

This paper proposes a novel method SELFELICIT that boosts LM’s ability to utilize context with automatic contextual evidence highlighting. It harnesses the inherent ability of specific attention layers within the LM to differentiate evidence within the context, enabling test-time auto-elicitating that is general, efficient and requires no additional training. Comprehensive experiments on context based QA validate the effectiveness of SELFELICIT, showing a performance improvement of 5.0% to 11.7% across multiple datasets and LMs. Compared to more costly generative evidence extraction, our method achieves better performance with far less additional computational overhead.
Limitations

We evaluated the effectiveness of SELFELICIT across several open-source LMs. However, we cannot assess SELFELICIT on proprietary LMs due to the need to access attention scores. Prioritizing efficiency and simplicity, we used a static eliciting threshold in the paper. We discussed the effect of different threshold \( \alpha \) choices in Figure 4. Our method is generally robust to different threshold \( \alpha \), with a broad range of values yielding consistently strong improvements, but one can still fine-tune it on specific tasks for better performance. Moreover, an interesting future direction would be to automatically select the optimal elicit thresholds based on the model and input characteristics. We provide a more detailed case analysis and discussion on the potential benefit of dynamic threshold adjustment in Appendix C.1.

Acknowledgments

We thank Danai Koutra for her insights and discussions throughout the project. This work is partially supported by NSF (2134079 ) and AFOSR (FA9550-24-1-0002). The content of the information in this document does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.

References

Mengting Ai, Tianxin Wei, Yifan Chen, Zeming Guo, and Jingrui He. 2023. Mlp fusion: Towards efficient fine-tuning of dense and mixture-of-experts language models. arXiv e-prints, pages arXiv–2307.

Mengting Ai, Tianxin Wei, Yifan Chen, Zhichen Zeng, Ritchie Zhao, Girish Varatkar, Bita Darvish Routhani, Xianfeng Tang, Hanghang Tong, and Jingrui He. 2025. Resmoe: Space-efficient compression of mixture of experts llms via residual restoration. arXiv preprint arXiv:2503.06881.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.

Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. 2021. Ee-net: Exploitation-exploration neural networks in contextual bandits. arXiv preprint arXiv:2110.03177.

Wenxuan Bao, Tianxin Wei, Haohan Wang, and Jingrui He. 2023. Adaptive test-time personalization for federated learning. Advances in Neural Information Processing Systems, 36:77882–77914.

Eunice Chan, Zhining Liu, Ruizhong Qiu, Yuheng Zhang, Ross Maciejewski, and Hanghang Tong. 2024. Group fairness via group consensus. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1788–1808.

Lingjie Chen, Ruizhong Qiu, Siyu Yuan, Zhining Liu, Tianxin Wei, Hyunsik Yoo, Zhichen Zeng, Deqing Yang, and Hanghang Tong. 2024a. Wapiti: A watermark for finetuned open-source llms. arXiv preprint arXiv:2410.06467.

Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. 2024b. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. In Forty-first International Conference on Machine Learning.

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2024. Dola: Decoding by contrasting layers improves factuality in large language models. In International Conference on Learning Representations (ICLR).

Florin Cuconasu, Giovanni Trapolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 719–729.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501.

Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. Mrqa 2019 shared task: Evaluating generalization in reading comprehension. In EMNLP 2019 MRQA Workshop, page 1.

Dongqi Fu, Liri Fang, Zihao Li, Hanghang Tong, Vettle I. Torvik, and Jingrui He. 2024. Parametric graph representations in the era of foundation models: A survey and position. CoRR, abs/2410.12126.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
Kai Guo, Xiaofeng Cao, Zhining Liu, and Yi Chang. 2023. Taming over-smoothing representation on heterophilic graphs. Information Sciences, 647:119463.

Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. 2024. Overthinking the truth: Understanding how language models process false demonstrations. In The Twelfth International Conference on Learning Representations.

Xinrui He, Tianxin Wei, and Jingrui He. 2023. Robust basket recommendation via noise-tolerated graph contrastive learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 709–719.

Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear, 7(1):411–420.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. In ACM Computing Surveys.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825.

Zhengbao Jiang, Frank F Xu, Luoy Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. Active retrieval augmented generation. In Empirical Methods in Natural Language Processing (EMNLP).

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL).

Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and applications of large language models. In arXiv preprint arXiv:2307.10169.

Tom Kwiatkowski, Jennimaria Polamaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenneth Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. In Transactions of the Association of Computational Linguistics (TACL).

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024a. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469.

Yunzhe Li, Junting Wang, Hari Sundaram, and Zhining Liu. 2025. A zero-shot generalization framework for llm-driven cross-domain sequential recommendation. arXiv preprint arXiv:2501.19232.

Zihao Li, Lecheng Zheng, Bowen Jin, Dongqi Fu, Baoyu Jing, Yikun Ban, Jingrui He, and Jiawei Han. 2024b. Can graph neural networks learn language with extremely weak text supervision? CoRR, abs/2412.08174.

Xiao Lin, Jian Kang, Weilin Cong, and Hanghang Tong. 2024a. Bemap: Balanced message passing for fair graph neural network. In Learning on Graphs Conference, pages 37–1. PMLR.

Xiao Lin, Zhiming Liu, Dongqi Fu, Ruizhong Qiu, and Hanghang Tong. 2024b. Backtime: Backdoor attacks on multivariate time series forecasting. arXiv preprint arXiv:2410.02195.

Lihui Liu, Boxin Du, Yi Ren Fung, Heng Ji, Jiejun Xu, and Hanghang Tong. 2021a. Kompare: A knowledge graph comparative reasoning system. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’21, page 3308–3318, New York, NY, USA. Association for Computing Machinery.

Lihui Liu, Boxin Du, Jiejun Xu, Yinglong Xia, and Hanghang Tong. 2022. Joint knowledge graph completion and question answering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, page 1098–1108, New York, NY, USA. Association for Computing Machinery.

Lihui Liu, Blaine Hill, Boxin Du, Fei Wang, and Hanghang Tong. 2024a. Conversational question answering with language models generated reformulations over knowledge graph. In Findings of the Association for Computational Linguistics: ACL 2024, pages 839–850, Bangkok, Thailand. Association for Computational Linguistics.

Zhining Liu, Wei Cao, Zhifeng Gao, Jiang Bian, Hechang Chen, Yi Chang, and Tie-Yan Liu. 2020a. Self-paced ensemble for highly imbalanced massive data classification. In 2020 IEEE 36th international conference on data engineering (ICDE), pages 841–852. IEEE.

Zhining Liu, Jian Kang, Hanghang Tong, and Yi Chang. 2021b. Imbens: Ensemble class-imbalanced learning in python. arXiv preprint arXiv:2111.12776.

Zhining Liu, Ruizhong Qiu, Zhichen Zeng, Hyunsik Yoo, David Zhou, Zhe Xu, Yada Zhu, Kommy Welde-mariam, Jingrui He, and Hanghang Tong. 2024b.
Class-imbalanced graph learning without class rebalancing. In Forty-first International Conference on Machine Learning.

Zhining Liu, Ruizhong Qiu, Zhichen Zeng, Yada Zhu, Hendrik Hamann, and Hanghang Tong. 2024c. Aim: Attributing, interpreting, mitigating data unfairness. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2014–2025.

Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao, Jiang Bian, and Yi Chang. 2020b. Mesa: boost ensemble imbalanced learning with meta-sampler. Advances in neural information processing systems, 33:14463–14474.

OpenAI. 2022. Introducing chatgpt. URL https://openai.com/blog/chatgpt.

OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.

Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427.

Ruizhong Qiu, Jun-Gi Jang, Xiao Lin, Lihui Liu, and Hanghang Tong. 2024a. TUCKET: A tensor time series data structure for efficient and accurate factor analysis over time ranges. Proceedings of the VLDB Endowment, 17(13).

Ruizhong Qiu, Zhiqing Sun, and Yiming Yang. 2022. DIMES: A differentiable meta solver for combinatorial optimization problems. In Advances in Neural Information Processing Systems, volume 35, pages 25531–25546.

Ruizhong Qiu and Hanghang Tong. 2024. Gradient compressed sensing: A query-efficient gradient estimator for high-dimensional zeroth-order optimization. In Proceedings of the 41st International Conference on Machine Learning.

Ruizhong Qiu, Dingsu Wang, Lei Ying, H Vincent Poor, Yifang Zhang, and Hanghang Tong. 2023. Reconstructing graph diffusion history from a single snapshot. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1978–1988.

Ruizhong Qiu, Zhe Xu, Wenxuan Bao, and Hanghang Tong. 2024b. Ask, and it shall be given: On the Turing completeness of prompting. arXiv, 2411.01992.

Ruizhong Qiu, Weiliang Will Zeng, Hanghang Tong, James Ezick, and Christopher Lott. 2024c. How efficient is LLM-generated code? A rigorous & high-standard benchmark. arXiv, 2406.06647.

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. In Association for Computational Linguistics (ACL).

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 783–791.

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. ACL 2017, page 191.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS).

Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. 2023a. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. In arXiv preprint arXiv:2310.07521.

Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. 2023b. Learning to filter context for retrieval-augmented generation. arXiv preprint arXiv:2311.08377.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022a. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Tianxin Wei, Yifan Chen, Xinrui He, and Jingrui He. 2025a. Connecting domains and contrasting samples: A ladder for domain generalization.

Tianxin Wei, Fuli Feng, Jiawei Chen, Ziwei Wu, Jinfeng Yi, and Xiangnan He. 2021. Model-agnostic counterfactual reasoning for eliminating popularity bias in recommender system. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 1791–1800.

Tianxin Wei, Zeming Guo, Yifan Chen, and Jingrui He. 2023. Ntk-approximating mlp fusion for efficient language model fine-tuning. In International Conference on Machine Learning, pages 36821–36838. PMLR.

Tianxin Wei and Jingrui He. 2022. Comprehensive fair meta-learned recommender system. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1989–1999.
Tianxin Wei, Bowen Jin, Ruirui Li, Hansi Zeng, Zhengyang Wang, Jianhui Sun, Qingyu Yin, Hanqing Lu, Suhang Wang, Jingrui He, et al. 2024a. Towards unified multi-modal personalization: Large vision-language models for generative recommendation and beyond. arXiv preprint arXiv:2403.10667.

Tianxin Wei, Ruizhong Qiu, Yifan Chen, Yunzhe Qi, Jiacheng Lin, Wenju Xu, Sreyashi Nag, Ruirui Li, Hanqing Lu, Zhengyang Wang, Chen Luo, Hui Liu, Suhang Wang, Jingrui He, Qi He, and Xianfeng Tang. 2024b. Robust watermarking for diffusion models: A unified multi-dimensional recipe.

Tianxin Wei, Ziwei Wu, Ruirui Li, Ziniu Hu, Fuli Feng, Xiangnan He, Yizhou Sun, and Wei Wang. 2020. Fast adaptation for cold-start collaborative filtering with meta-learning. In 2020 IEEE International Conference on Data Mining (ICDM), pages 661–670. IEEE.

Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen, Jingrui He, and Zhangyang Wang. 2022b. Augmentations in hypergraph contrastive learning: Fabricated and generative. Advances in neural information processing systems, 35:1909–1922.

Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2025b. InstructRAG: Instructing retrieval-augmented generation via self-synthesized rationales. In The Thirteenth International Conference on Learning Representations.

Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li. 2025c. Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning. Preprint, arXiv:2505.16421.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45.

Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024. How easily do irrelevant inputs skew the responses of large language models? arXiv preprint arXiv:2404.03302.

Haobo Xu, Yuchen Yan, Dingsu Wang, Zhe Xu, Zhichen Zeng, Tarek Abdelzaher, Jiawei Han, and Hanghang Tong. Slog: An inductive spectral graph neural network beyond polynomial filter. In Forty-first International Conference on Machine Learning.

Zhe Xu, Ruizhong Qiu, Yuzhong Chen, Huiyuan Chen, Xiran Fan, Menghai Pan, Zhichen Zeng, Mahashweta Das, and Hanghang Tong. 2024. Discrete-state continuous-time diffusion for graph generation. In Advances in Neural Information Processing Systems, volume 37.

Yuchen Yan, Yuzhong Chen, Huiyuan Chen, Xiaoting Li, Zhe Xu, Zhichen Zeng, Lihui Liu, Zhining Liu, and Hanghang Tong. 2024a. Thegcn: Temporal heterophilic graph convolutional network. arXiv preprint arXiv:2412.16435.

Yuchen Yan, Yuzhong Chen, Huiyuan Chen, Minghua Xu, Mahashweta Das, Hao Yang, and Hanghang Tong. 2023a. From trainable negative depth to edge heterophily in graphs. Advances in Neural Information Processing Systems, 36:70162–70178.

Yuchen Yan, Yongyi Hu, Qinghai Zhou, Lihui Liu, Zhichen Zeng, Yuzhong Chen, Menghai Pan, Huiyuan Chen, Mahashweta Das, and Hanghang Tong. 2024b. Pacer: Network embedding from positional to structural. In Proceedings of the ACM on Web Conference 2024, pages 2485–2496.

Yuchen Yan, Yongyi Hu, Qinghai Zhou, Shurang Wu, Dingsu Wang, and Hanghang Tong. 2024c. Topological anonymous walk embedding: A new structural node embedding approach. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 2796–2806.

Yuchen Yan, Baoyu Jing, Lihui Liu, Ruijie Wang, Jinning Li, Tarek Abdelzaher, and Hanghang Tong. 2023b. Reconciling competing sampling strategies of network embedding. Advances in Neural Information Processing Systems, 36:6844–6861.

Yuchen Yan, Lihui Liu, Yikun Ban, Baoyu Jing, and Hanghang Tong. 2021. Dynamic knowledge graph alignment. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 4564–4572.

Yuchen Yan, Qinghai Zhou, Jinning Li, Tarek Abdelzaher, and Hanghang Tong. 2022. Dissecting cross-layer dependency inference on multi-layered interdependent networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 2341–2351.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Owen2 technical report. arXiv preprint arXiv:2407.10671.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Empirical Methods in Natural Language Processing (EMNLP).

Hangting Ye, Zhining Liu, Wei Cao, Amir M Amiri, Jiang Bian, Yi Chang, Jon D Lurie, Jim Weinstein, and Tie-Yan Liu. 2023. Web-based long-term spine treatment outcome forecasting. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3082–3092.

Hyunsik Yoo, SeongKu Kang, Ruizhong Qiu, Charlie Xu, Fei Wang, and Hanghang Tong. 2025a. Embracing plasticity: Balancing stability and plasticity in
continual recommender systems. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval.

Hyunsik Yoo, Ruizhong Qiu, Charlie Xu, Fei Wang, and Hanghang Tong. 2025b. Generalizable recommender system during temporal popularity distribution shifts. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining.

Hyunsik Yoo, Zhichen Zeng, Jian Kang, Ruizhong Qiu, David Zhou, Zhining Liu, Fei Wang, Charlie Xu, Eunice Chan, and Hanghang Tong. 2024. Ensuring user-side fairness in dynamic recommender systems. In Proceedings of the ACM on Web Conference 2024, pages 3667–3678.

Erxin Yu, Jing Li, and Chunpu Xu. 2024. PopALM: Popularity-aligned language models for social media trendy response prediction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12867–12878.

Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. 2024. Attention satisfies: A constraint-satisfaction lens on factual errors of language models. In International Conference on Learning Representations (ICLR).

Zhichen Zeng, Boxin Du, Si Zhang, Yinglong Xia, Zhining Liu, and Hanghang Tong. 2024a. Hierarchical multi-marginal optimal transport for network alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16660–16668.

Zhichen Zeng, Ruizhong Qiu, Zhe Xu, Zhining Liu, Yuchen Yan, Tianxin Wei, Lei Ying, Jingrui He, and Hanghang Tong. 2024b. Graph mixup on approximate gromov–wasserstein geodesics. In Forty-first International Conference on Machine Learning.

Zhichen Zeng, Si Zhang, Yinglong Xia, and Hanghang Tong. 2023a. Parrot: Position-aware regularized optimal transport for network alignment. In Proceedings of the ACM Web Conference 2023, pages 372–382.

Zhichen Zeng, Ruike Zhu, Yinglong Xia, Hanqing Zeng, and Hanghang Tong. 2023b. Generative graph dictionary learning. In International Conference on Machine Learning, pages 40749–40769. PMLR.

Yuheng Zhang, Dian Yu, Tao Ge, Linfeng Song, Zhichen Zeng, Haitao Mi, Nan Jiang, and Dong Yu. 2025. Improving llm general preference alignment via optimistic online mirror descent. arXiv preprint arXiv:2502.16852.

Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. 2024. Enhancing contextual understanding in large language models through contrastive decoding. arXiv preprint arXiv:2405.02750.

Lecheng Zheng, Baoyu Jing, Zihao Li, Hanghang Tong, and Jingrui He. 2024a. Heterogeneous contrastive learning for foundation models and beyond. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, pages 6666–6676. ACM.

Lecheng Zheng, Baoyu Jing, Zihao Li, Zhichen Zeng, Tianxin Wei, Mengting Ai, Xinrui He, Lihui Liu, Dongqi Fu, Jiaxuan You, Hanghang Tong, and Jingrui He. 2024b. Pyg-ssl: A graph self-supervised learning toolkit. CorR, abs/2412.21151.

Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhaq Chen. 2023. Context-faithful prompting for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14544–14556.

A Reproducibility Details

QA pipeline details. We use PyTorch (Paszke et al., 2019) and HuggingFace Transformers (Wolf et al., 2020) to implement all involved language models (LMs). Specifically, the LMs used in the paper are the instruct fine-tuned version of Llama3.1 8B\(^3\) and 70B\(^4\). Mistral 7B (instruct-v0.3)\(^5\) and 12B (Nemo)\(^6\), as well as Qwen2.5 7B\(^7\) and 32B\(^8\). We use the official apply_chat_template function to get the input tokens from text input (\( \tau(c, q) \)) and let the model generate the answer based on it. We use greedy decoding with a temperature of 0 to get deterministic answers.

Dataset details. For all the datasets used, we test the LM on the official public validation split, with 7,405/4,212/7,785/12,836 context-question pairs for HotpotQA/NewsQA/TQA/NQ. The context passages in those datasets are retrieved from diverse sources including Wikipedia (Yang et al., 2018; Kwiatkowski et al., 2019), news articles (Trischler et al., 2017), and web search queries (Joshi et al., 2017). The detailed statistics and descriptions are reported in Table 5. We use Llama-3.1 tokenizer to compute the number of tokens in the context passage.

\footnotetext{3https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct}
\footnotetext{4https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct}
\footnotetext{5https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3}
\footnotetext{6https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407}
\footnotetext{7https://huggingface.co/Qwen/Qwen2.5-7B-Instruct}
\footnotetext{8https://huggingface.co/Qwen/Qwen2.5-32B-Instruct}
Table 5: Dataset description and statistics.

<table>
  <tr>
    <th>Dataset</th>
    <th>#Context Tokens Avg.</th>
    <th>#Context Tokens Max.</th>
    <th>#Samples</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>1251.6</td>
    <td>3346</td>
    <td>7405</td>
    <td>Wikipedia</td>
  </tr>
  <tr>
    <td>NewsQA</td>
    <td>625.24</td>
    <td>940</td>
    <td>4212</td>
    <td>CNN News Article</td>
  </tr>
  <tr>
    <td>TQA</td>
    <td>892.89</td>
    <td>1974</td>
    <td>7785</td>
    <td>Bing Search Query</td>
  </tr>
  <tr>
    <td>NQ</td>
    <td>249.38</td>
    <td>2679</td>
    <td>12836</td>
    <td>Wikipedia</td>
  </tr>
</table>

**SELFELICIT implementation details.** Unless otherwise specified, all SELFELICIT results in this paper use the default settings stated in the paper: the last 50% of layers are designated as evidence-reading layers, with an elicitation threshold \( \alpha = 0.5 \). We use SpaCy (Honnibal and Montani, 2017) to segment the context passages into sentences, and SELFELICIT performs the sentence-level evidence score computation and elicitation based on this segmentation. Note that this is only for SELFELICIT, for PROMPTELICIT, we directly search for exact text-level matches of the generatively extracted evidence to ensure that all extracted information is utilized.

B Additional Results and Analysis

B.1 Computational Efficiency

Due to space constraints, we report only the average running time per sample across all datasets in Table 1. Table 6 provides a detailed breakdown of efficiency results for each dataset, including the average inference time per sample (i.e., context-question pair) and the total number of tokens generated during inference. Note that Mistral-12B generates shorter answers (7.34 tokens on average) for each sample compared to Mistral-7B (24.06 tokens), thus showing a shorter inference time for each sample as it generates fewer tokens. We observe that: (i) While FULLELICIT is highly efficient as a naive method, it barely helps with the QA performance, as shown in Table 1. (ii) PROMPTELICIT, based on generative evidence extraction, requires the model to generate a large volume of extra tokens during inference, up to 852% more than direct QA, resulting in considerably slower inference times. (iii) As discussed before, the response to COT is not consistent for different LMs. Generally, for Llama and Mistral models, COT have little impact on the QA performance, and inference time as well. For Qwen-series models, COT obtains slightly better EM by generating the intermediate reasoning steps before answering, but this also leads to a great drop in F1 score and significantly higher inference time (e.g., +72.0% for Qwen2.5-7B) due to the generation of intermediate step tokens. (iv) SELFELICIT, by helping the model focus on contextual evidence, generates shorter but more accurate answers. Additionally, as SELFELICIT does not involve generative evidence extraction, it remains computationally efficient, with only 3-5% of the inference time overhead compared to PROMPTELICIT.

B.2 More Layer-wise Attention Visualization

In Figure 2, we present the differences in attention paid by different layers of the model to evidence versus non-evidence context information on the HotpotQA (Yang et al., 2018) dataset. We chose HotpotQA because it provides human-annotated sentence-level "supporting_facts" annotations. In this section, we extend our analysis to other datasets. Since human annotations are unavailable for these datasets, we use a simple rule to distinguish evidence from non-evidence sentences: any sentence containing at least one correct answer is considered evidence. While this is not a rigorous approach for labeling evidence sentences, it still allows us to visualize patterns and roughly validate whether our observations hold across different datasets. As shown in Figure 5, deeper layers of various LMs consistently demonstrate a strong ability to differentiate evidence across datasets.

B.3 Impact of Evidence Highlighting on Model Attention

To assess the effectiveness of SELFELICIT in directing model attention toward relevant context, we analyzed the average attention score per token for the selected evidence sentences before and after highlighting. We conducted this analysis across four QA datasets using Llama-3.1-8B, averaging attention scores over all layers and heads. As shown in Table 7, we observe a consistent increase in attention scores for the highlighted evidence across all datasets. This demonstrates that the highlighting strategy effectively shifts model focus toward critical contextual information, reinforcing its ability to retrieve and utilize evidence more effectively.

B.4 Sentence-level versus Token-level Eliciting

We now show that sentence-level evidence eliciting holds a significant advantage over token-level eliciting due to its ability to highlight evidence with better semantic coherence. Table 8 presents the experimental results using the llama3.1-8B model across four datasets. We compare the normal version SELFELICIT (sentence) and a new token-level
Table 6: Detailed runtime and efficiency results. we report the average inference time and number of generated tokens for each sample (i.e., context-question pair) for each dataset.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="4">Dataset</th>
    <th colspan="3">Average</th>
  </tr>
  <tr>
    <th>HotpotQA<br>Time (ms)</th>
    <th>HotpotQA<br>#Tokens</th>
    <th>NewsQA<br>Time (ms)</th>
    <th>NewsQA<br>#Tokens</th>
    <th>TQA<br>Time (ms)</th>
    <th>TQA<br>#Tokens</th>
    <th>Inference<br>Time/sample (ms)</th>
    <th>#Generated<br>Tokens</th>
  </tr>
  <tr>
    <td rowspan="5">Llama-7B</td>
    <td>BASE</td>
    <td>161.46</td>
    <td>6.68</td>
    <td>290.76</td>
    <td>11.79</td>
    <td>230.18</td>
    <td>6.72</td>
    <td>213.18</td>
    <td>10.42</td>
    <td>223.90 (+0.8%)</td>
    <td>8.90 (+0.0%)</td>
  </tr>
  <tr>
    <td>Cot</td>
    <td>163.14</td>
    <td>6.52</td>
    <td>290.86</td>
    <td>11.77</td>
    <td>230.21</td>
    <td>6.76</td>
    <td>213.85</td>
    <td>10.36</td>
    <td>224.51 (+0.3%)</td>
    <td>8.85 (-0.6%)</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>163.55</td>
    <td>6.49</td>
    <td>290.79</td>
    <td>11.83</td>
    <td>236.30</td>
    <td>6.97</td>
    <td>213.64</td>
    <td>10.47</td>
    <td>226.07 (+1.0%)</td>
    <td>8.94 (+0.4%)</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>1419.08</td>
    <td>75.78</td>
    <td>1718.02</td>
    <td>85.18</td>
    <td>2412.91</td>
    <td>116.81</td>
    <td>1117.64</td>
    <td>61.40</td>
    <td>1666.91 (+64.5%)</td>
    <td>84.79 (+85.2%)</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td>204.65</td>
    <td>6.98</td>
    <td>318.74</td>
    <td>10.54</td>
    <td>297.71</td>
    <td>7.44</td>
    <td>233.78</td>
    <td>9.83</td>
    <td>263.72 (+17.8%)</td>
    <td>8.70 (-2.3%)</td>
  </tr>
  <tr>
    <td rowspan="5">7B</td>
    <td>BASE</td>
    <td>910.68</td>
    <td>5.17</td>
    <td>1850.04</td>
    <td>11.32</td>
    <td>1397.99</td>
    <td>5.95</td>
    <td>1421.32</td>
    <td>11.18</td>
    <td>1395.01 (+0.0%)</td>
    <td>8.40 (+0.0%)</td>
  </tr>
  <tr>
    <td>Cot</td>
    <td>917.37</td>
    <td>5.15</td>
    <td>1856.47</td>
    <td>10.61</td>
    <td>1398.79</td>
    <td>5.88</td>
    <td>1423.61</td>
    <td>10.88</td>
    <td>1399.06 (+0.3%)</td>
    <td>8.13 (-3.2%)</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>915.09</td>
    <td>5.41</td>
    <td>1851.00</td>
    <td>11.13</td>
    <td>1448.56</td>
    <td>6.32</td>
    <td>1434.97</td>
    <td>10.31</td>
    <td>1412.41 (+1.2%)</td>
    <td>8.29 (-1.3%)</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>8546.64</td>
    <td>69.39</td>
    <td>7712.69</td>
    <td>55.77</td>
    <td>8283.76</td>
    <td>56.14</td>
    <td>7139.59</td>
    <td>60.06</td>
    <td>7920.42 (+467.8%)</td>
    <td>60.34 (+618.1%)</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td>1198.53</td>
    <td>6.05</td>
    <td>1854.49</td>
    <td>9.48</td>
    <td>1672.19</td>
    <td>6.15</td>
    <td>1398.81</td>
    <td>9.88</td>
    <td>1531.00 (+9.7%)</td>
    <td>7.89 (-6.1%)</td>
  </tr>
  <tr>
    <td rowspan="5">Mistral</td>
    <td>BASE</td>
    <td>468.26</td>
    <td>21.60</td>
    <td>564.08</td>
    <td>24.60</td>
    <td>479.97</td>
    <td>18.67</td>
    <td>647.02</td>
    <td>31.38</td>
    <td>539.83 (+0.0%)</td>
    <td>24.06 (+0.0%)</td>
  </tr>
  <tr>
    <td>Cot</td>
    <td>489.55</td>
    <td>23.31</td>
    <td>568.91</td>
    <td>25.27</td>
    <td>502.71</td>
    <td>19.59</td>
    <td>688.56</td>
    <td>34.02</td>
    <td>562.43 (+4.2%)</td>
    <td>25.55 (+6.2%)</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>477.01</td>
    <td>19.74</td>
    <td>567.08</td>
    <td>23.57</td>
    <td>479.98</td>
    <td>19.61</td>
    <td>647.23</td>
    <td>31.78</td>
    <td>542.83 (+3.6%)</td>
    <td>23.42 (-2.7%)</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>1444.44</td>
    <td>70.58</td>
    <td>2043.11</td>
    <td>97.09</td>
    <td>2230.07</td>
    <td>102.03</td>
    <td>1788.89</td>
    <td>89.59</td>
    <td>1876.62 (+467.8%)</td>
    <td>89.82 (+273.3%)</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td>330.42</td>
    <td>12.74</td>
    <td>493.56</td>
    <td>18.33</td>
    <td>440.59</td>
    <td>13.32</td>
    <td>462.43</td>
    <td>21.00</td>
    <td>431.75 (+20.0%)</td>
    <td>16.34 (-32.1%)</td>
  </tr>
  <tr>
    <td rowspan="5">Qwen-7B</td>
    <td>BASE</td>
    <td>205.88</td>
    <td>5.62</td>
    <td>380.22</td>
    <td>10.46</td>
    <td>287.35</td>
    <td>5.32</td>
    <td>251.74</td>
    <td>7.97</td>
    <td>281.30 (+0.0%)</td>
    <td>7.34 (+0.0%)</td>
  </tr>
  <tr>
    <td>Cot</td>
    <td>208.52</td>
    <td>5.57</td>
    <td>381.90</td>
    <td>10.09</td>
    <td>287.36</td>
    <td>5.35</td>
    <td>258.02</td>
    <td>7.41</td>
    <td>283.95 (+0.9%)</td>
    <td>7.11 (-3.2%)</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>208.39</td>
    <td>5.65</td>
    <td>380.59</td>
    <td>10.54</td>
    <td>289.87</td>
    <td>5.41</td>
    <td>284.23</td>
    <td>7.90</td>
    <td>283.27 (+0.7%)</td>
    <td>7.37 (-0.4%)</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>1273.57</td>
    <td>46.77</td>
    <td>1560.04</td>
    <td>52.11</td>
    <td>1749.14</td>
    <td>54.64</td>
    <td>1226.14</td>
    <td>46.18</td>
    <td>1452.22 (+416.9%)</td>
    <td>49.93 (+580.0%)</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td>278.07</td>
    <td>6.29</td>
    <td>428.84</td>
    <td>9.02</td>
    <td>385.85</td>
    <td>5.45</td>
    <td>260.08</td>
    <td>7.47</td>
    <td>338.14 (+20.2%)</td>
    <td>7.06 (-3.9%)</td>
  </tr>
  <tr>
    <td rowspan="5">Qwen-13B</td>
    <td>BASE</td>
    <td>143.78</td>
    <td>6.36</td>
    <td>303.72</td>
    <td>13.53</td>
    <td>247.26</td>
    <td>8.26</td>
    <td>286.31</td>
    <td>15.49</td>
    <td>245.27 (+0.0%)</td>
    <td>10.91 (+0.0%)</td>
  </tr>
  <tr>
    <td>Cot</td>
    <td>149.74</td>
    <td>25.10</td>
    <td>407.30</td>
    <td>20.06</td>
    <td>370.10</td>
    <td>15.93</td>
    <td>460.24</td>
    <td>26.51</td>
    <td>421.85 (+72.0%)</td>
    <td>21.90 (+100.7%)</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>145.24</td>
    <td>6.18</td>
    <td>306.38</td>
    <td>12.89</td>
    <td>255.94</td>
    <td>6.85</td>
    <td>291.09</td>
    <td>16.35</td>
    <td>249.66 (+1.8%)</td>
    <td>10.57 (-3.1%)</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>1109.49</td>
    <td>62.77</td>
    <td>1227.45</td>
    <td>64.27</td>
    <td>1312.63</td>
    <td>64.84</td>
    <td>1005.11</td>
    <td>58.32</td>
    <td>1163.67 (+473.9%)</td>
    <td>62.55 (+573.9%)</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td>194.21</td>
    <td>6.90</td>
    <td>357.74</td>
    <td>12.92</td>
    <td>292.41</td>
    <td>7.59</td>
    <td>313.18</td>
    <td>16.05</td>
    <td>285.38 (+18.0%)</td>
    <td>10.87 (-0.4%)</td>
  </tr>
  <tr>
    <td rowspan="5">Qwen-33B</td>
    <td>BASE</td>
    <td>624.64</td>
    <td>6.55</td>
    <td>1271.39</td>
    <td>15.35</td>
    <td>773.30</td>
    <td>6.51</td>
    <td>1044.09</td>
    <td>14.50</td>
    <td>928.35 (+0.0%)</td>
    <td>10.73 (+0.0%)</td>
  </tr>
  <tr>
    <td>Cot</td>
    <td>625.01</td>
    <td>7.46</td>
    <td>1447.84</td>
    <td>18.24</td>
    <td>871.12</td>
    <td>8.13</td>
    <td>1048.22</td>
    <td>14.59</td>
    <td>998.05 (+7.5%)</td>
    <td>12.10 (+12.8%)</td>
  </tr>
  <tr>
    <td>FULLELICIT</td>
    <td>636.89</td>
    <td>6.63</td>
    <td>1275.19</td>
    <td>15.06</td>
    <td>780.95</td>
    <td>6.80</td>
    <td>1052.68</td>
    <td>14.21</td>
    <td>936.43 (+9.9%)</td>
    <td>10.67 (-0.5%)</td>
  </tr>
  <tr>
    <td>PROMPTELICIT</td>
    <td>5556.76</td>
    <td>82.86</td>
    <td>5468.46</td>
    <td>76.23</td>
    <td>5290.17</td>
    <td>69.83</td>
    <td>4096.52</td>
    <td>62.62</td>
    <td>5102.97 (+497.7%)</td>
    <td>72.89 (+579.5%)</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td>738.98</td>
    <td>6.32</td>
    <td>1220.44</td>
    <td>12.29</td>
    <td>1011.58</td>
    <td>7.02</td>
    <td>948.09</td>
    <td>11.47</td>
    <td>979.77 (+5.5%)</td>
    <td>9.28 (-13.5%)</td>
  </tr>
</table>

Table 7: Effect of highlighting on model attention, measured as average attention per token (APT) for selected evidence sentences.

<table>
  <tr>
    <th>APT (\( \times 10^{-3} \))</th>
    <th>HotpotQA</th>
    <th>NewsQA</th>
    <th>TQA</th>
    <th>NQ</th>
  </tr>
  <tr>
    <td>Before Highlighting</td>
    <td>0.4362</td>
    <td>0.3465</td>
    <td>0.2090</td>
    <td>0.9631</td>
  </tr>
  <tr>
    <td>After Highlighting</td>
    <td>0.6351</td>
    <td>0.5783</td>
    <td>0.3451</td>
    <td>1.1917</td>
  </tr>
  <tr>
    <td>After/Before Ratio</td>
    <td>145.61%</td>
    <td>166.91%</td>
    <td>165.10%</td>
    <td>123.73%</td>
  </tr>
</table>

Table 8: Sentence-level versus token-level eliciting.

<table>
  <tr>
    <th>Elicit Level</th>
    <th colspan="4">Dataset</th>
  </tr>
  <tr>
    <th></th>
    <th>HotpotQA</th>
    <th>NewsQA</th>
    <th>TQA</th>
    <th>NQ</th>
  </tr>
  <tr>
    <td>Metric: EM</td>
    <td>Token</td>
    <td>66.5</td>
    <td>63.5</td>
    <td>74.0</td>
    <td>57.5</td>
  </tr>
  <tr>
    <td></td>
    <td>Sentence</td>
    <td><b>68.5</b></td>
    <td><b>66.9</b></td>
    <td><b>79.4</b></td>
    <td><b>64.0</b></td>
  </tr>
  <tr>
    <td>Metric: Token F1</td>
    <td>Token</td>
    <td>68.6</td>
    <td>58.4</td>
    <td>70.9</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td></td>
    <td>Sentence</td>
    <td><b>69.5</b></td>
    <td><b>60.8</b></td>
    <td><b>72.7</b></td>
    <td><b>67.8</b></td>
  </tr>
</table>

variant SELFELICIT (token) by treating each token as a separate "sentence". It is evident that token-level eliciting performs worse than sentence-level eliciting on all datasets. The underlying reason is that tokenization does not ensure each token represents a complete entity or concept, resulting in highlighted evidence that lacks full semantic meaning. For example, we notice that a year within the context, say "2002", can be tokenized into two tokens "200" and "2" and token-level eliciting could end up highlighting only the first token "200.". Similar "partial highlighting" issues also occur with names, locations, and other uncommon phrases. In contrast, sentence-level eliciting highlights the entire sentence, maintaining semantic continuity in the processed context, which better aids the model in utilizing context information for QA.

B.5 Alternative Highlighting Strategies

In our primary experiments, we utilized in-context highlighting, where relevant evidence is marked directly within the original passage. This approach aligns with the core insight of SELFELICIT —leveraging the LM's inherent ability to identify relevant evidence—while maintaining the semantic continuity of the input. However, alternative methods of evidence integration could potentially influence the model's ability to utilize highlighted information effectively. To explore this, we conducted additional experiments using non-in-context highlighting, where all extracted evidence sentences were aggregated and appended either to the beginning or end of the original context. We evaluated these
Figure 5: Across different LM families and datasets, the deep attention layers highlight the crucial evidence sentences within the context, even when the LM gives incorrect answers (dashed lines). The X-axis is the depth of attention layers, and the Y-axis is the ratio of the average attention per token (APT) in the evidence/non-evidence sections to the APT across the entire context. For the HotpotQA (Yang et al., 2018) dataset, We leverage the “supporting_facts” annotations to differentiate evidence and non-evidence sentences within the context. For other datasets, we treat a context sentence as evidence if it contains at least one of the correct answers.

variations using Llama-3.1-8B across four datasets and reported F1 scores in Table 9. The results indicate that in-context highlighting generally yields the best performance, likely due to its preservation of the original passage’s semantic flow, allowing the model to process highlighted evidence within its natural context. Appending evidence to the beginning or end also improves performance over the base model but is slightly less effective, particularly for multi-hop tasks like HotpotQA, where contextual relationships between facts are crucial.

We emphasize that SELFELICIT adopts in-context highlighting primarily for its simplicity and efficiency. However, our framework is flexible, and alternative highlighting strategies can be seamlessly incorporated. Exploring more effective context augmentation techniques remains an intriguing direction for future work.

<table>
  <tr>
    <th>Highlighting</th>
    <th>HotpotQA</th>
    <th>NewsQA</th>
    <th>TQA</th>
    <th>NQ</th>
  </tr>
  <tr>
    <td>-</td>
    <td>57.7</td>
    <td>56.2</td>
    <td>66.1</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td>In-context</td>
    <td>69.5</td>
    <td><b>60.8</b></td>
    <td><b>72.7</b></td>
    <td><b>67.8</b></td>
  </tr>
  <tr>
    <td>Attach at Begin</td>
    <td><b>69.8</b></td>
    <td>59.8</td>
    <td>69.3</td>
    <td><b>67.8</b></td>
  </tr>
  <tr>
    <td>Attach at End</td>
    <td>69.0</td>
    <td>58.8</td>
    <td>67.0</td>
    <td>65.6</td>
  </tr>
</table>

Table 9: Comparison of different highlighting strategies.

B.6 Potentially Uninformative Context

To further analyze SELFELICIT’s robustness in handling cases where the provided context may lack relevant evidence, we constructed an additional QA task based on HotpotQA. This experiment assesses whether SELFELICIT can correctly reject answers when the context does not support a response. Specifically, we randomly selected 50% of the samples as unanswerable, replacing their original context with unrelated passages from other samples. The correct response in these cases was set to: "I cannot answer based on the given context." We explicitly instructed the LLM to generate this response whenever relevant information was absent. In addition to standard EM and F1 scores, we introduced Rejection Accuracy, which measures the model’s ability to correctly identify and reject uninformative contexts. The results on Llama-3.1-8B are shown in Table 10.

These findings, along with Table 1 in the main paper, demonstrate that SELFELICIT does not degrade—rather, it slightly enhances—the model’s
Table 10: Results on potentially unanswerable QA task.

<table>
  <tr>
    <th>Method</th>
    <th>F1</th>
    <th>EM</th>
    <th>RejectionAccuracy</th>
  </tr>
  <tr>
    <td>Base</td>
    <td>78.48</td>
    <td>79.58</td>
    <td>91.25</td>
  </tr>
  <tr>
    <td><b>SELFELICIT</b></td>
    <td><b>82.65</b></td>
    <td><b>81.93</b></td>
    <td><b>92.86</b></td>
  </tr>
</table>

ability to recognize when the provided context lacks sufficient information for answering. At the same time, SELFELICIT significantly improves the model’s ability to utilize relevant information when it is available by highlighting the most crucial evidence in relation to the user query.

B.7 Highlighting versus Filtering Strategies

In this section, we compare SELFELICIT to the Filco method which filters context using a fine-tuned Llama-2 model (Wang et al., 2023b). Additionally, we explore the impact of highlighting versus filtering evidence. To analyze the impact of different evidence extraction and highlighting strategies, we conduct experiments on four SELFELICIT/Filco variants: (i) SELFELICIT-highlight: The standard SELFELICIT approach, where identified evidence sentences within the context are explicitly highlighted. (ii) SELFELICIT-filter: Instead of highlighting, this method extracts the identified evidence sentences and concatenates them to form a new, filtered context passage. (iii) Filco-filter: The standard Filco approach, which employs a fine-tuned Llama-2 model to generate a filtered version of the context for answering questions. (iv) Filco-highlight: This variant highlights the evidence sentences identified by Filco within the original context, rather than filtering them out.

This experimental setup enables a direct comparison between highlighting vs. filtering, as well as SELFELICIT vs. Filco, in their ability to enhance LMs’ utilization of contextual information. We use Llama-3.1-8B-Instruct as the base LM and evaluate performance across four QA datasets. The F1 scores are presented in Table 11.

Table 11: Comparison of SELFELICIT and Filco variants on four QA datasets, evaluated using F1 scores. SELFELICIT-highlight achieves the highest performance across all datasets.

<table>
  <tr>
    <th>Method</th>
    <th>HotpotQA</th>
    <th>NewsQA</th>
    <th>TQA</th>
    <th>NQ</th>
  </tr>
  <tr>
    <td>Base</td>
    <td>57.7</td>
    <td>56.2</td>
    <td>66.1</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td>SELFELICIT-highlight</td>
    <td><b>69.5</b></td>
    <td><b>60.8</b></td>
    <td><b>72.7</b></td>
    <td><b>67.8</b></td>
  </tr>
  <tr>
    <td>SELFELICIT-filter</td>
    <td>62.4</td>
    <td>55.6</td>
    <td>66.5</td>
    <td>65.1</td>
  </tr>
  <tr>
    <td>Filco-highlight</td>
    <td>67.8</td>
    <td>57.2</td>
    <td>68.9</td>
    <td>65.3</td>
  </tr>
  <tr>
    <td>Filco-filter</td>
    <td>62.6</td>
    <td>54.7</td>
    <td>66.3</td>
    <td>64.5</td>
  </tr>
</table>

The key findings are: (i) SELFELICIT consistently achieves the highest performance across all datasets, outperforming not only its filtering-based counterpart but also Filco, a more complex, fine-tuned approach. This highlights the effectiveness of inference-time evidence elicitation over training-based filtering. (ii) Highlighting-based methods outperform their filtering-based counterparts for both SELFELICIT and Filco. Filtering removes context that may contain auxiliary or indirect supporting information, such as intermediate reasoning steps, leading to information loss. (iii) Filtering-based methods struggle particularly on NewsQA, which consists of news articles as context. Unlike structured knowledge sources such as Wikipedia, news articles tend to be semantically cohesive—making it harder to extract self-contained and sufficiently informative evidence sentences. This may explain why Filco was originally evaluated on Wikipedia-based datasets, where evidence extraction is more straightforward.

These findings reinforce that highlighting key evidence within the original context is more effective than filtering, as it preserves both primary evidence and surrounding contextual cues that may contribute to reasoning. Moreover, SELFELICIT, despite being a lightweight inference-time method, surpasses Filco, which relies on additional model fine-tuning—demonstrating the advantage of leveraging pretrained LMs’ inherent ability to locate and utilize relevant evidence.

B.8 Examples with full context

Finally, we present the full context of examples from Table 2 to illustrate SELFELICIT’s ability to identify supporting facts/evidence within noisy long contexts. The results are shown in Tables 12 and 13. These examples are from HotpotQA’s distractor setting, where the context passage contains distracting information retrieved from Wikipedia using the question as a query (Yang et al., 2018). Note that these distractors are not completely irrelevant random noise that can be easily filtered out by retrieval systems. Instead, they appear related to the question but do not actually support answering it, serving as "hard negatives." This scenario is quite common in practice and significantly impacts the performance of retrieval-augmented generation (Wu et al., 2024; Cuconasu et al., 2024). As shown in Tables 12 and 13, the LM struggles to effectively use contextual evidence to provide correct answers under such a situation. Despite this, SELFELICIT accurately highlights the critical
evidence within the noisy long context, helping the model focus on the most relevant facts and thus arrive at the correct answer.

C Additional Discussions

C.1 Failure Case Analysis and Potential Improvements

While SELFELICIT significantly enhances LM’s ability to identify and utilize relevant evidence, there are cases where it fails to aid the model in generating correct answers. Through manual analysis, we found that incomplete evidence highlighting is a primary cause of failure. Consider the following example where Llama-3.1-8B, both with and without SELFELICIT, gives a negative response.

An example case with \( \alpha = 0.5 \)

Context: Alexander S. Wadsworth Commodore Alexander Scammel Wadsworth (1790–April 5, 1851) was an officer of the United States Navy. His more than 40 years of active duty included service in the War of 1812. War of 1812 The War of 1812 (1812–1815) was a conflict fought between the United States, the United Kingdom, and their respective allies. Historians in Britain often see it as a minor theater of the Napoleonic Wars; in the United States and Canada, it is seen as a war in its own right ...
Question: Alexander S. Wadsworth served during the war that was fought between which two countries?
Base Model: I cannot answer based on the given context. ✗
SELFELICIT: I cannot answer based on the given context. ✗

We can see that although SELFELICIT successfully highlighted the sentence: "The War of 1812 (1812–1815) was a conflict fought between the United States, the United Kingdom, and their respective allies." as important evidence, this alone was insufficient for the model to infer that Alexander S. Wadsworth served in this war. We observed that lowering \( \alpha \) can mitigate such failures by allowing SELFELICIT to highlight more comprehensive evidence. Reducing \( \alpha \) to 0.1 enabled the method to additionally highlight the missing link—"Commodore Alexander Scammel Wadsworth (1790–April 5, 1851) was an officer of the United States Navy. His more than 40 years of active duty included service in the War of 1812.", and lead to the correct answer.

An example case with \( \alpha = 0.1 \)

Context: Alexander S. Wadsworth Commodore Alexander Scammel Wadsworth (1790–April 5, 1851) was an officer of the United States Navy. His more than 40 years of active duty included service in the War of 1812. War of 1812 The War of 1812 (1812–1815) was a conflict fought between the United States, the United Kingdom, and their respective allies. Historians in Britain often see it as a minor theater of the Napoleonic Wars; in the United States and Canada, it is seen as a war in its own right ...
Question: Alexander S. Wadsworth served during the war that was fought between which two countries?
Base Model: I cannot answer based on the given context. ✗
SELFELICIT: United States, the United Kingdom, and their respective allies. ✓

This suggests that a fixed threshold may not be optimal across different questions and contexts, and SELFELICIT could potentially benefit from a more adaptive thresholding mechanism: Instead of using a fixed \( \alpha \), a dynamic threshold that adjusts based on the complexity of the input context or question could provide more flexible evidence selection. Nevertheless, as shown and discussed in Figure 4, SELFELICIT remains robust to the choice of \( \alpha \), with a broad range of values yielding consistently strong improvements. Even with static thresholding, SELFELICIT provides stable and significant gains in LM performance across QA tasks.

C.2 Understanding Layer-Wise Evidence Utilization in Language Models

Our analysis reveals that while intermediate layers of language models effectively emphasize relevant evidence within the context, this emphasis does not necessarily translate into correct final responses. As shown in Figure 2 and 5, attention to key evidence tends to diminish from mid-layers to the final layers, indicating that the final layers do not prioritize contextual evidence as strongly as mid-layers do. This suggests that although models inherently recognize relevant information at intermediate layers, their final token selection during decoding is influenced by additional factors that may override this evidence, leading to incorrect responses.

This phenomenon aligns with recent findings in closed-book QA and controlled generation settings. Prior work has shown that factually correct tokens often receive higher decoding probabilities in intermediate layers, while incorrect tokens can maintain consistently high probabilities across all layers and still be selected as the final output (Chuang et al., 2024). Similarly, studies on the "overthinking" behavior of LMs indicate that removing the last few layers can sometimes improve factual accuracy, implying that deeper layers may introduce unnecessary complexity or spurious reasoning (Halawi et al., 2024). Our findings provide further empiri-
cal support for these observations, explaining why models can correctly identify relevant evidence yet still produce incorrect answers.

C.3 Potential Risks
Generative AI tools such as language models have an increasing impact on our daily lives in the era of big data and AI (Yan et al., 2021, 2022, 2023a,b, 2024a,b,c; Xu et al., 2024; Ban et al., 2021; Lin et al., 2024a,b; Guo et al., 2023; Wei et al., 2025c, 2024b; Qiu et al., 2024c,b,a, 2023, 2022; Zeng et al., 2023a,b, 2024a,b; Qiu and Tong, 2024; Wei et al., 2025a; Ai et al., 2023; Yoo et al., 2025a,b, 2024; He et al., 2023; Bao et al., 2023; Wei et al., 2023, 2024a; Wei and He, 2022; Wei et al., 2022b, 2020, 2021; Li et al., 2025), such as finance (Chan et al., 2024; Liu et al., 2024c) and healthcare (Ye et al., 2023; Liu et al., 2024b), especially with the recent trends of foundation models (Zheng et al., 2024a; Fu et al., 2024; Zheng et al., 2024b; Li et al., 2024b; Zhang et al., 2025; Ai et al., 2025). This study focuses on enhancing language models’ ability to effectively utilize information from contextual documents. However, retrieved documents from the internet may contain unethical or discriminatory content, which the model might read and incorporate into its outputs. While addressing ethical concerns in Retrieval-Augmented Generation (RAG) or fact-based question-answering tasks (Liu et al., 2024a, 2022, 2021a) is beyond the scope of this work, such issues can typically be mitigated by using detectors to filter harmful information from the context documents provided to the language model, or other general techniques such as ensemble multi-model answers (Liu et al., 2021b, 2020a,b) or watermarking (Chen et al., 2024a) AI-generated contents.

C.4 Usage of Artifacts and AI Assistants
All models and datasets used in this study are publicly available on HuggingFace, and we adhered to their respective licenses and terms of use, limiting our work to non-commercial academic research. These models and datasets have been reviewed by their developers/creators to minimize the inclusion of personally identifiable information or offensive content and are widely adopted by the research community. The datasets primarily consist of English-language content and focus on fact-based question-answering tasks. We used AI tools to assist with language refinement during the writing process, but the paper contains no AI-generated paragraphs. All material has been carefully reviewed to ensure accuracy and adherence to ethical standards.
Table 12: Full context of example #1 & #2 from Table 2 demonstrating how our method assists the LM in context-based QA, with blue text highlighting the evidence selected by SELFELICIT. The base LM used is Llama3.1-8B-Instruct.

<table>
  <tr>
    <th>Full Context Passage</th>
    <th>Question & Answers</th>
  </tr>
  <tr>
    <td><b>Tiger Please is an Indie / Alternative five-piece band from Cardiff, Wales. The band formed in August 2008. The band’s influences are U2, Sigur Rós, Kings of Leon, John Mayer and Counting Crows. They signed with Walnut Tree Records in 2009 and released their debut EP “They Don’t Change Under Moonlight”, “Kerrang!” magazine, “Rock Sound” magazine, and “Classic Rock” magazine praised the band on the “Rock Sound” and “Classic Rock” cover-mount albums. The band toured with Kids In Glass Houses, InMe, Twin Atlantic and Funeral For A Friend. Taxi Victoria is a six-piece alternative rock band from Cardiff, Wales, guitarists include David Spy (vocal), Adam King (bass), Louis Nel (drums), Rian Zietsman (guitar) and Locu V Reuter (guitar/bass). They have released five studio albums “Unite Yourself” (The Turn) (2009), “*” (2011), “Soul Shake” (2013), and “Tenfold” (2014). They are influenced by bands such as Black Rebel Motorcycle Club, Queens of the Stone Age, Led Zeppelin, Jimi Hendrix, Nirvana, and Pearl Jam. This is discography of the American rock band Black Rebel Motorcycle Club Black Rebel Motorcycle Club (often abbreviated as BRMC) is an American rock band from San Francisco, California. The group consists of Peter Hayes (vocal, guitar), Nick Jago (vocal, rhythm guitar), Sam Roberts (bass, guitar), and Sam Shain (drums). Nick Jago left the band in 2008 to focus on his solo project. Skybombers is a rock band from Melbourne. They were formed as Collusion by Scotch College students Hugh Gurney, Ravi Sharma, Scott McMurrle and Sam Bethune. They later changed to Skybombers, a name inspired by an icy-pole. Their placing a demo song “It Goes Off!” on MySpace brought them their first TV appearances. They had early international attention when “It Goes Off!” of their EP “Sirens” made the most-played list on L.A.’s Indie FM and played a showcase gig at The Viper Room. They have toured Australia, Japan and USA, and their third album “Take Me To” was released in 2007. The Wipers are a Red Rebel Motorcycle Club. The band made their way on video game media in 2007 when “It Goes Off!” appeared on the soundtrack for “Burnout Dominator”, the song later reappeared on “Burnout Paradise” in early 2008. Black Carousel was recorded in L.A. again with Rick Parker at the helm. Masaki Liu, sometimes referred to as “Saki”, is the engineer and producer operating One Way Studio, a digital recording studio in Benicia, California. Masaki has recorded and produced music for many artists including Five Iron Frenzy, Blue Cheer, Black Rebel Motorcycle Club, Echo and the Bunnymen, The Wipers, Avalon Seven, Dan Russell is a musician and songwriter in addition to an artist manager and advocate, musician, songwriter, concert promoter, record producer and music supervisor for television and film. A graduate of Walpole High School in Massachusetts and later Barrington College, Russell is known for managing both the American rock band The Call and songwriter Michael Beent and has worked in various capacities with such artists as Black Rebel Motorcycle Club, Sam Phillips, Mark Heard, U2 and Robin Lane, Ramona Silver, Vigilantes of Love and others. Black Rebel Motorcycle Club: Live is a DVDs of Black Rebel Motorcycle Club concerts and was released from their 2006 tour. The DVD “Public Glitter” was released in 2007 as part of the band’s 2007 tour in support of “Baby 81”. Additionally, it includes intimate, behind-the-scenes footage, glimpses into the making of 2005’s Howl and is rounded out with a bonus live album featuring 14 songs. KAV (Kav Sandhu) is a British musician from Leicester UK, based in Los Angeles. KAV played guitar with British band Happy Mondays for 4 years after helping reform the band with frontman Shaun Ryder in 2004. He launched his solo project the month of “KAV” in 2008 with long time friend and drummer Jamie Forte. His hits include “I’m Not Going” which was featured by the media to myocardium Iggy & The Stooges, Black Rebel Motorcycle Club, Prince, Screamin, Kasabian, The Rolling Stones & Bob Dylan. He plays live with a full live band, which sometimes features guest musicians from various bands. Vagrant Records is an American record label based in California. It was founded in 1995 by Rich Egan and Jon Cohen. The label focuses on rock but features artists in a variety of other genres including folk, soul, electronic, and pop. It is home to artists such as The 1975, Death Spells, Eel, The Suns, Edwyn Collins and the Magnetic Zeros, CRUISR, Action Child, PJ Harvey, School of Seven Bells, Black Rebel Motorcycle Club, The Vincent McMorrow, Black Joe Lewis, Wake Owl, Blitzan Trapper, and Bombay Bicycle Club. Originally, Vagrant Records was mostly focused on emo bands such as Dashboard Confessional, Saves the Day, The Get Up Kids, and Alkaline Trio.</b></td>
    <td>
      <b>Question:</b> Are the bands Tiger Please and Black Rebel Motorcycle Club from the same country?<br>
      <b>True Answer:</b> No.<br>
      <b>Base:</b> Yes. ✗<br>
      <b>+selfelicit:</b> No. ✓
    </td>
  </tr>
  <tr>
    <td><b>Home Monthly was a monthly women's magazine published in Pittsburgh, Pennsylvania in the late 19th century. "The Strategy of the Were-Wolf Dog" is a short story by Willa Cather. It was first published in "Home Monthly" in December 1896. The Count of Crow's Nest is a short story by Willa Cather. It was first published in "Home Monthly" in October 1896. Mirabella was a women's magazine published from June 1899 to April 2000. It was created by and named for Grace Mirabella, a former 'Vogue' editor-in-chief, in partnership with Rupert Murdoch. 'Nanette: An Aside' is a short story by Willa Cather. It was first published in "Home Monthly" on July 1897. "A Bar Mitzvah" is a short story by Willa Cather. It was first published in "Home Monthly" in July 1897. "A Bar Mitzvah" is a short story by American writer Willa Cather. It was first published in "Home Monthly" in April 1897. Tommy, The Unsentimental is a short story by Willa Cather. It was first published in "Home Monthly" in August 1896. The Princess Baladina is a short story by Willa Cather. It was first published in "Home Monthly" in 1896 under the pseudonym of Charles Douglass. "The Way of the World" is a short story by Willa Cather. It was first published in "Home Monthly" in April 1898.</b></td>
    <td>
      <b>Question:</b> Which women's magazine was published first, Mirabella or Home Monthly?<br>
      <b>True Answer:</b> Home Monthly.<br>
      <b>Base:</b> Mirabella. ✗<br>
      <b>+selfelicit:</b> Home Monthly. ✓
    </td>
  </tr>
</table>
Table 13: Full context of example #3 & #4 from Table 2 demonstrating how our method assists the LM in context-based QA, with blue text highlighting the evidence selected by SELFELICIT. The base LM used is Llama3.1-8B-Instruct.

<table>
  <tr>
    <th>Full Context Passage</th>
    <th>Question & Answers</th>
  </tr>
  <tr>
    <td>The 1987 European Cup Final was a football match held at the Prater Stadium, Vienna, on 27 May 1987, that saw Porto of Portugal defeat Bayern Munich of West Germany 2–1. Both sides were missing key players: the Portuguese were without their injured striker Fernando Gomes, while the Germans were missing their sweeper, and captain, Klaus Augenthaler, who was suspended, along with striker Roland Wohlfarth and midfielder Hans Dorfler, who were both injured. The Portuguese scored back from 1–0 to win their first European Cup, with the goals coming in a 5–4 final heel by Rabah Madjer and a volley from Józéf, after a Larsen Kogler goal had put the Germans in the lead half. The final was the first European Cup final that Bayern, and their captain Lothar Matthäus would lose to successive late goals, repeated 12 years later in the 1999 UEFA Champions League Final against Manchester United. The 1993 European Cup Winners' Cup Final was a football match contested between Parma of Italy and Royal Antwerp of Belgium. The final was held at Wembley Stadium in London, England on 1 May 1993. It was the final match of the 1992–93 European Cup Winners' Cup and the 33rd European Cup Winners' Cup. Parma beat Antwerp 3–1 to become the first different Italian team to win a European trophy. The 1970 European Cup Winners' Cup Final was a football match between Manchester City of England and Górnik Zabrze of Poland on 29 April 1970 at Prater Stadium in Vienna, Austria. It was the final match of the 1969–70 European Cup Winners' Cup and the tenth European Cup Winners' Cup final. Both sides made their first appearance in a European final. Manchester City won the match 2–1 thanks to goals by Neil Young and Francis Lee. The victory was City's first European trophy. The 1976 European Cup Winners' Cup Final was a football match between Liverpool of England and Rapid Wien of Austria on 25 May 1977 at the Stadio Olimpico, Rome, Italy (the venue was decided in Bern by the UEFA Executive Committee on 17 September 1976). The showpiece event was the final match of the 1976–77 season of Europe's premier cup competition, the European Cup. Both teams were appearing in their first European cup final, although the two sides had previously met in the 1973 UEFA Cup Final, which Liverpool won 3–2 on aggregate over two legs. The 1988 European Cup Winners' Cup Final was a football match contested between Mechelen of Belgium and Fiorentina of Italy, played on 24 May 1988 at the Stade de la Meinau in Strasbourg, France. Mechelen won the match 1–0 thanks to a goal by Piet den Boer. The 1961 European Cup Winners' Cup Final was a football match contested between Fiorentina of Italy and Rangers of Scotland. It was the final of the 1960–61 European Cup Winners' Cup the first UEFA Cup Winners' Cup final. It was the only time that the final was played over two legs. The first leg was played at Ibrox Park in Glasgow, and the second leg at the Stadio Artemio Franchi. It was Rangers first European final in doing so became the first British team to reach the final of a European football competition. It was Fiorentina's second European final having previously reached the 1957 European Cup final. The 1987 European Cup Winners' Cup Final was a football match contested between Ajax of Netherlands and Lokomotive Leipzig of East Germany. It was the final match of the 1986–87 European Cup Winners' Cup and the 27th European Cup Winners' Cup Final. The final was held at Olympic Stadium in Athens, Greece. Ajax won the match 1–0 with a 20th-minute header from Marco van Basten. The Lunde (1 March 1964) is a Danish former professional footballer who played as a player in the forward position. Lunde got his breakthrough with Brondby IF in 1983, and he made his debut for the Denmark national football team in October 1983. He was sold to Young Boys Bern in Switzerland, before moving to German club Bayern Munich in 1986. He was a part of the Bayern team which won the German Bundesliga championship in 1987, and he came on as a late substitute when Bayern lost the 1987 European Cup Final to FC Porto. He played the last of his three matches for the Danish national team in April 1988. He retired from football in 1990 due to injury problems. He went on to play for a number of smaller clubs, ending his career with FC Baden in Switzerland. The 1978 European Cup Final was a football match between Liverpool of England and Club Brugge of Belgium on 10 May 1978 at Wembley Stadium, London, England (the venue was decided in Bern by the UEFA Executive Committee on 20 September 1977). It was the final match of the 1977–78 season of Europe's premier cup competition, the European Cup. Liverpool were the reigning champions and were appearing in their second European Cup final. Club Brugge were appearing in their first European Cup final. The two sides had met once before in Europe, in the 1977–78 UEFA Cup, where Liverpool won 3–0, but 5–3 on aggregate. The 1985 European Cup Winners' Cup Final was a football match contested between Everton of England and Rapid Wien of Austria. It was the final match of the 1984–85 European Cup Winners' Cup and the 25th European Cup Winners' Cup Final. The final was held at Feyenoord Stadion in Rotterdam, Netherlands, on 15 May 1985. Everton, which dominated throughout, won the match 3–1 thanks to goals by Andy Gray, Trevor Steven and Kevin Sheedy. Everton were unable to defend the trophy, as league champions they would have entered the 1985–86 European Cup, but they were not permitted to play in Europe for the following two seasons as a result of rival Liverpool fans at the Heysel Stadium, which saw all English clubs banned from European competitions.</td>
    <td>
      <b>Question:</b> Which team did Lars Lunde play for when defeated for the 1987 European Cup Final?
      <br>
      <b>True Answer:</b> Bayern Munich
      <br>
      <b>BASE:</b> FC Porto.
      <br>
      <b>*SELFELICIT:</b> Bayern Munich.
    </td>
  </tr>
  <tr>
    <td>Mount Barker Junction railway station is a disused station on the Adelaide to Wolseley line serving the South Australian city of Mount Barker. Mount Gambier railway station was the junction station for the Naracoorte–Milllicent and Mount Gambier–Heywood lines in the South Australian city of Mount Gambier. Frewville is a small suburb in the South Australian city of Adelaide. It is three kilometres south-east of Adelaide's central business district (CBD). The 2006 South Australian Super League was the first season of the South Australian Super League, the new top division of association football in South Australia, replacing the South Australian Premier League, which became the second division. It was also the first year that football in South Australia was run by the Football Federation of South Australia, which replaced the South Australian Soccer Federation. The season came down to the final round relegation battle between White City Woodville and Adelaide Olympic. Olympic lost 3–1 at Modbury while White City went down 1–0 away to Cumberland. This sent Olympic down to play in the Premier League in 2007. Adelaide City won the title with games to spare after being runaway leaders, finishing the season unbeaten. Norwood is a suburb of Adelaide, about 4 km east of the Adelaide city centre. The suburb is in the City of Norwood Payneham & St Peters the oldest South Australian local government municipality, with a city population of 34,000. Willunga is a small town west of Adelaide, on the main railway line between Adelaide and the Australian city of Whyalla. Walter Frank Giffen (20 September 1861 in Norwood – 28 June 1949 in Adelaide) was an Australian cricketer who played in 3 Tests between 1887 and 1892. He was the brother of the great all-rounder George Giffen. The City of Burnside is a local government area with an estimated population of 44,300 people in the South Australian city of Adelaide. Burnside was founded in August 1856 as the District Council of Burnside, and was classed as a city in 1943. It is named after the name of an old settler, William Henry Burnside. The city is bounded by the Adelaide River. It is bounded by Adelaide's Adelaide Hills Council, Campbelltown, Mitcham, Norwood Payneham and St Peters and Urrbrae. The city has an area of 27.53 km². Glenunga is a small southern suburb of 2,539 people in the South Australian city of Adelaide. It is located five kilometres southeast of the Adelaide city centre. The name Glenunga is taken from an Aboriginal language "unga" meaning near and "glen" because of its proximity to Glen Osmond (see Manning's places of South Australia by Geoffrey H. Manning published in 1990). Bounded by the north by Windsor Road, the east by Portrush Road and southwest by Glen Osmond Road and the south by Kingsbridge Street, the suburb is bounded by the city of Adelaide. It is close to other Burnside council suburbs of Torok Gardens and Glenelde. Collina is a suburb of the Australian city of Griffith in the Riverina region of New South Wales. The suburb is in the City of Griffith local government area. Collina is 4 km northwest of the Griffith city centre and reflects the city's rapid growth in the early 2000s.</td>
    <td>
      <b>Question:</b> Walter Giffen is from a suburb of which South Australian city?
      <br>
      <b>True Answer:</b> Adelaide
      <br>
      <b>BASE:</b> Norwood.
      <br>
      <b>*SELFELICIT:</b> Adelaide.
    </td>
  </tr>
</table>

Reviewer paper summaries: This paper proposes a simple but effective method to address the challenge that LMs can not comprehend and utilize key evidence from the context, especially when there are noises. The paper first identifies that regardless of whether the model responds correctly or not, the deeper layers of the LM pay attention to relevant evidence in the context. They then utilize this evidence to propose an evidence score which will select relevant sentences and then reformat the context to highlight the evidence sentences. The results show consistent gains across various QA datasets and models.

This paper proposes
Summary of strengths: * The idea of utilizing attention scores as evidence selection criteria is valid and provides efficiency compared to those that require iterative prompting.
* The paper is written clearly
* The paper conducts significant amount of experiments and analysis.

1. The proposed context highlighting method boosts performances on the selected QA tasks. 
2. The paper is clear and easy to read. 
3. The proposed method identified relevant context, as shown in Table 3.
Summary of weaknesses: * The idea of utilizing attention scores for regularizing contexts is not novel in the literature. For example, SnapKV [1] utilizes attention scores to filter KV cache. Attention Sorting [2] utilizes attention scores to sort the context and highlight those important information. Given that all these cited works are highly relevant and impactful, but not cited by this paper. It is doubtful that the authors did not do enough literature survey.
* Despite its effectiveness and efficiency, the method requires some hyper-parameter tuning that might be hard at some applications. For example, when both global (asking about a summary) and local (asking about a fact) questions are asked by the user, then it would require different $\alpha$ to control the number of evidence to be selected.
* Context filtering algorithms are previously proposed to handle noise contexts [3]. How does the method compare to that? And how does the method performs when applying attention as a criterion of filtering.

[1] SnapKV: LLM Knows What You are Looking for Before Generation
[2] Attention Sorting Combats Recency Bias In Long Context Language Models
[3] Learning to Filter Context for Retrieval-Augmented Generation

1. The setting is relatively easy as only gold evidences are provided. Experiments on harder tasks where evidence may or may not be useful would be of more interest to the community. 
2. The results may not generalize out-of-domain. The observation is made on factoid QA tasks, and the evidence-reading layers are selected based on HotpotQA (which is also one of the evaluated datasets). The proposed method is then only tested on factoid QA tasks, where three out of four datasets use evidences from Wikipedia. I wonder if the results would hold on tasks where outputs on longer (e.g. long-form QA tasks such as ASQA). 
3. The claims about factuality should be toned down. There are no experiments on factuality. Framing answer correctness on factoid QA is not accurate. 
4. The AUROC / nDCG numbers in Table 3 are lower for TQA. No explanations are given. 


Reference: 
Stelmakh, Ivan, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. "ASQA: Factoid questions meet long-form answers." arXiv preprint arXiv:2204.06092 (2022).
Comments, suggestions and typos: No additional comments

L375: TK%
Overall assessment scores: 2

3