Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion
Authors: Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, Kunwoo Park
Accepted at: acl25

Hypothetical Documents or Knowledge Leakage?
Rethinking LLM-based Query Expansion

Yejun Yoon♡   Jaeyoon Jung♣◊   Seunghyun Yoon♣   Kunwoo Park☆♡
♡Department of Intelligent Semiconductors, Soongsil University
♣School of AI Convergence, Soongsil University
◊MAUM AI Inc.
☆Adobe Research, USA
{yejun0382, jaeyoonskr}@soongsil.ac.kr, syoon@adobe.com, kunwoo.park@ssu.ac.kr

Abstract

Query expansion methods powered by large language models (LLMs) have demonstrated effectiveness in zero-shot retrieval tasks. These methods assume that LLMs can generate hypothetical documents that, when incorporated into a query vector, enhance the retrieval of real evidence. However, we challenge this assumption by investigating whether knowledge leakage in benchmarks contributes to the observed performance gains. Using fact verification as a testbed, we analyze whether the generated documents contain information entailed by ground-truth evidence and assess their impact on performance. Our findings indicate that, on average, performance improvements consistently occurred for claims whose generated documents included sentences entailed by gold evidence. This suggests that knowledge leakage may be present in fact-verification benchmarks, potentially inflating the perceived performance of LLM-based query expansion methods.

1 Introduction

Zero-shot retrieval aims to identify relevant documents without requiring any relevance supervision for training a retriever (Zhao et al., 2024). Because obtaining query-document pairs, such as MS-MARCO (Bajaj et al., 2016), for supervised training is challenging, developing zero-shot retrieval methods is both difficult and highly desirable for effectively addressing knowledge-intensive applications (Lewis et al., 2020), including question answering (Zhu et al., 2021) and fact verification (Guo et al., 2022).

Recent studies have leveraged the natural language generation capabilities of large language models (LLMs) to enhance the performance of zero-shot retrieval (Thakur et al., 2021). LLM-based query expansion (QE) uses LLMs to generate documents that extend a query (Jagerman et al., 2023; Lei et al., 2024; Mackie et al., 2023a).

<table>
  <tr>
    <th>Claim</th>
    <th>Gold Evidence</th>
  </tr>
  <tr>
    <td>Trump said 'there were fine people on both sides' in far-right protests.</td>
    <td>Trump made the remarks in the wake of a <b>Unite the Right</b> rally in Charlottesville, Virginia, in 2017 when he said "You all had some very fine people."</td>
  </tr>
  <tr>
    <td>LLM-based Query Expansion</td>
    <td></td>
  </tr>
  <tr>
    <td>Here is a fact-checking article to verify the claim.</td>
    <td>This claim refers to comments made by former President Donald Trump in the aftermath of the <b>2017 "Unite the Right" rally in Charlottesville, Virginia.</b></td>
  </tr>
</table>

Figure 1: Illustration of potential knowledge leakage in LLM-based query expansion.

Approaches such as HyDE (Gao et al., 2023) and Query2doc (Wang et al., 2023), which have been widely adopted in recent research (Wang et al., 2024a; Chen et al., 2024b; Yoon et al., 2024), have achieved notable performance gains across various benchmarks without retriever parameter updates. These approaches prompt LLMs to generate documents that answer a question or verify a claim. Although these generated documents, referred to as <i>hypothetical</i> documents, may contain factual errors or hallucinations, it is assumed that incorporating them into a query can enhance retrieval of relevant <i>real</i> documents (Gao et al., 2023).

In this paper, we challenge the underlying assumption, as illustrated by a counterexample in Figure 1: <i>Do LLMs truly generate hypothetical documents, or are they merely reproducing what they already know?</i> LLMs are extensively pretrained on vast corpora, primarily collected from the web. As common retrieval targets, such as Wikipedia and web documents, are often included in these pretraining corpora (Groeneveld et al., 2024; Touvron et al., 2023; Du et al., 2022; Brown et al., 2020), many available LLMs may already contain knowledge relevant to a given query and retrieval targets in benchmark datasets, a phenomenon we refer to as <i>knowledge leakage</i>. If knowledge leakage occurs and an LLM reproduces what it already knows
about a given query, its generation cannot be seen as hypothetical documents. Furthermore, assessing QE performance on a benchmark where leakage occurs could lead to overestimation, as retrieving verbatim or semantically equivalent sentences is a much more trivial task than predicting the relevance between a query and document when the relevance signal is not explicit.

To investigate whether knowledge leakage exists in popular benchmarks and how it affects downstream performance, this study uses fact verification as a testbed. We analyze whether LLM-generated documents contain sentences entailed by gold evidence and assess their impact on performance. Across experiments spanning three benchmarks and seven LLMs, we observed a consistent trend: QE methods were effective, on average, only when the generated documents included sentences entailed by gold evidence. These findings suggest that knowledge leakage may be present in these benchmarks, potentially inflating the perceived effectiveness of LLM-based QE methods.

2 Related Works

LLM-based Query Expansion  QE has been explored as a means to enhance retrieval performance by enriching the initial query representation (Azad and Deepak, 2019). One widely studied approach is relevance feedback (Rocchio, 1971; Lavrenko and Croft, 2001; Amati and Van Rijsbergen, 2002), which leverages feedback signals to expand the query. Recent work has explored LLM’s generative capabilities for QE (Zhu et al., 2023; Jagerman et al., 2023; Lei et al., 2024). Mackie et al. (2023b), for instance, introduced a method that utilizes LLM-generated documents as relevance feedback. Meanwhile, other researchers proposed HyDE (Gao et al., 2023) and Query2doc (Wang et al., 2023), which employ LLMs to generate hypothetical documents based on an initial query. Despite their simplicity, these methods have demonstrated substantial effectiveness across benchmarks for zero-shot retrieval and knowledge-intensive tasks (Wang et al., 2024a), including fact verification (Yoon et al., 2024).

Data Leakage and LLM Memorization  Previous research has investigated various forms of data leakage in LLMs (Kandpal et al., 2023; Samuel et al., 2025; Deng et al., 2024; Xu et al., 2024a). One study used perplexity to detect potential data leakage, uncovering substantial instances of training or even test set misuse (Xu et al., 2024b). Deng et al. (2023) further examined data contamination by predicting masked tokens in test sets and found that GPT 3.5 could reconstruct missing portions of MMLU (Hendrycks et al., 2020) test instances with 57% accuracy. Other research has explored the boundaries of LLM knowledge (Yin et al., 2023; Dong et al., 2024; Burns et al., 2022; Kadavath et al., 2022; Mallen et al., 2023), including a refusal-aware instruction tuning method that trains LLMs to reject uncertain questions (Zhang et al., 2024)—where an LLM is deemed uncertain if its generated response does not match the ground truth. Another study leveraged response consistency to estimate an LLM’s confidence in its knowledge (Cheng et al., 2024). In this work, we apply NLI (MacCartney, 2009) to LLM-generated documents, referencing gold evidence, to examine what an LLM knows.

3 Methodology

3.1 Task and Dataset

Fact verification aims to predict the veracity label of a textual claim \( c \). Depending on the dataset, the veracity label can fall into one of three or four categories\footnote{AVeriTeC provides four categories, whereas FEVER and SciFact use three, excluding *conflicting evidence*. In SciFact, *CONTRADICT* is treated equivalently to *refuted*.}: *supported*, *refuted*, *not enough evidence*, or *conflicting evidence*.

The fact-verification task consists of two subtasks: evidence retrieval and verdict prediction. In evidence retrieval, a retrieval method \( R(\cdot) \) identifies an evidence set \( \tilde{E} = \{e_1, \cdots, e_k\} \) from a knowledge store \( K \) (e.g., Wikipedia), used to verify \( c \). The performance of \( R \) is evaluated by comparing \( \tilde{E} \) with \( E = \{e_1, \cdots, e_l\} \), the gold evidence set. Verdict prediction then determines the veracity label of \( c \) based on \( \tilde{E} \).

We choose fact verification as the target task to test our hypothesis for two main reasons. First, in real-world fact-checking scenarios (Miranda et al., 2019; Nakov et al., 2021), retrieving evidence about niche or novel knowledge is crucial. If QE is effective only when the relevant knowledge has been seen during language model pretraining, its practical usefulness may be limited. Second, verdict prediction is a classification task, which enables clearer evaluation of how QE influences end performance compared to generation-based tasks, such as factual QA (Joshi et al., 2017; Kwiatkowski
et al., 2019).

We employ three datasets that provide annotated evidence and corresponding veracity labels, along with an external knowledge store: FEVER (Thorne et al., 2018), SciFact (Wadden et al., 2020), and AVeriTeC (Schlichtkrull et al., 2023). While FEVER and SciFact contain verbatim extractions from \( K \) as gold evidence (i.e., \( E \)), AVeriTeC uses human-written evidence referencing \( K \) to verify claims. These three datasets cover diverse knowledge domains, as summarized in Table 1.

<table>
  <tr>
    <th>Dataset</th>
    <th># claim</th>
    <th># gold evidence per claim</th>
    <th># documents (knowledge source)</th>
  </tr>
  <tr>
    <td>FEVER</td>
    <td>6,666</td>
    <td>1.66</td>
    <td>5,416,536 (Wikipedia)</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>5,183 (Paper abstracts)</td>
  </tr>
  <tr>
    <td>SciFact</td>
    <td>693</td>
    <td>1.8</td>
    <td>2,623,538 (Paper abstracts)</td>
  </tr>
  <tr>
    <td>AVeriTeC</td>
    <td>3,563</td>
    <td>2.77</td>
    <td>2,623,538 (Web documents)</td>
  </tr>
</table>

Table 1: Dataset statistics

3.2 LLM-based Query Expansion

We consider two LLM-based QE methods.

Query2doc (Wang et al., 2023) generates a pseudo-document \( d \) based on a query \( q \). It then forms an expanded query \( q^+ \) by concatenating \( d \) with multiple copies of \( q \) (Equation 1).

\[
q^+ = concat(q \times n, d)
\] (1)

The expanded query \( q^+ \) is then used to retrieve documents via BM25 (Lin et al., 2021). Following Jagerman et al. (2023), we set \( n \) as 5.

HyDE (Gao et al., 2023) employs an LLM to generate hypothetical documents \([d_1, ..., d_N]\) in response to a query \( q \). A dense retriever \( g(\cdot) \) encodes \( q \) and each \( d_k \) separately, and their encoded embeddings are averaged to form the query vector \( v_{q^+} \) (Equation 2).

\[
v_{q^+} = \frac{1}{N+1} \sum_{k=1}^{N} [g(d_k) + g(q)]
\] (2)

Here, we set \( N \) to 1 in this study. We use Contriever (Izacard et al., 2021) as \( g(\cdot) \) with prompts provided in Appendix E.

3.3 Matching Algorithm

We present an algorithm to determine whether an LLM-generated document \( d \) for a claim \( c \) contains a sentence that is verbatim or semantically equivalent to the gold evidence \( E = \{e_1, \cdots, e_m\} \). If such a sentence exists, it may indicate that the backbone LLM was exposed to knowledge related to the corresponding evidence during training. To automatically identify such claims, we rely on natural language inference (NLI) assigning each claim \( c \) to one of two conditions: matched (\( M \)) or unmatched (\( \neg M \)). The algorithm consists of three steps.

(1) Sentence Segmentation: Segment \( d \) into sentences and remove reproductions of \( c \) to construct \( S = \{s_1, \cdots, s_n\} \).

(2) NLI Labeling: Annotate a label \( l_{(i,j)} \) for each pair \( (e_i, s_j) \in E \times S \), where \( l_{(i,j)} \in \{entailment, contradiction, neutral\} \).

(3) Label Aggregation: Aggregate all labels \( \{l_{(1,1)}, \cdots, l_{(i,j)}, \cdots, l_{(m,n)}\} \) into a single label \( l \). If there exists \( l_{(i,j)} \) labeled as entailment, assign matched, otherwise assign unmatched.

We use the sentence segmentation module provided by spaCy\footnote{https://huggingface.co/spacy/en_core_web\_lg}, and employ GPT-4o-mini for NLI (Figure A3). To filter out sentences of reproduced claims in LLM responses, we apply ROUGE-2 (Lin, 2004) with a threshold of 0.95, based on manual inspection.

4 Experimental Results

We conducted evaluation experiments on three fact-verification benchmarks. Based on eight repetitions of LLM generation, we report the average performance along with the standard error.

Are LLM-based query expansion methods effective for fact verification? To assess the effectiveness of LLM-based QE methods, we compared their performance against BM25 and Contriever that use \( c \) as query, respectively, as baseline retrievers. For evidence retrieval, we used Recall@k and NDCG@k (\( k = 5 \)) as evaluation metrics (Manning, 2009) on the FEVER and SciFact datasets, where both the ground-truth evidence \( E \) and retrieved evidence \( \hat{E} \) come from the knowledge store \( K \). In contrast, \( \hat{E} \) in AVeriTeC consists of human-written evidence rather than extracts from \( K \). Therefore, following previous studies (Schlichtkrull et al., 2023; Chen et al., 2022), we applied the Hungarian algorithm (Kuhn, 1955) with METEOR (Banerjee and Lavie, 2005) and BERTScore (Zhang et al.,
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td>BM25</td>
    <td>31</td>
    <td>25</td>
    <td>54</td>
    <td>51.2</td>
    <td>45.5</td>
    <td>48.6</td>
    <td>17.8</td>
    <td>11.6</td>
    <td>32</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="10">Performance by varying LLMs</th>
  </tr>
  <tr>
    <th>GPT-4o-mini</th>
    <td>36.4±0.1</td>
    <td>29.3±0.1</td>
    <td>55.6±0.1</td>
    <td>55.1±0.2</td>
    <td>47.9±0.1</td>
    <td>52.5±0.5</td>
    <td>19.1±0.0</td>
    <td>12.4±0.0</td>
    <td>32.6±0.1</td>
  </tr>
  <tr>
    <th>Claude-3-haiku</th>
    <td>35.2±0.1</td>
    <td>28.3±0.1</td>
    <td>55.4±0.1</td>
    <td>56.0±1.1</td>
    <td>48.2±0.1</td>
    <td>52.0±0.5</td>
    <td>19.3±0.0</td>
    <td>12.5±0.0</td>
    <td>33.1±0.1</td>
  </tr>
  <tr>
    <th>Gemini-1.5-flash</th>
    <td>36.2±0.1</td>
    <td>29.2±0.1</td>
    <td>55.8±0.1</td>
    <td><b>56.2±0.2</b></td>
    <td><b>49.4±0.1</b></td>
    <td><b>52.2±0.5</b></td>
    <td>18.9±0.0</td>
    <td>12.5±0.0</td>
    <td><b>33.3±0.2</b></td>
  </tr>
  <tr>
    <th>Llama-3.1-8b-it</th>
    <td>35.7±0.1</td>
    <td>28.6±0.2</td>
    <td>55.6±0.2</td>
    <td>54.9±0.2</td>
    <td>47.8±0.2</td>
    <td>51.9±0.3</td>
    <td>19.0±0.0</td>
    <td>12.4±0.0</td>
    <td>32.2±0.2</td>
  </tr>
  <tr>
    <th>Llama-3.1-70b-it</th>
    <td><b>38.3±0.1</b></td>
    <td><b>31±0.1</b></td>
    <td><b>56.1±0.1</b></td>
    <td><b>56.4±0.3</b></td>
    <td>49.2±0.1</td>
    <td>52.4±0.7</td>
    <td><b>19.3±0.1</b></td>
    <td><b>12.7±0.0</b></td>
    <td><b>33.4±0.2</b></td>
  </tr>
  <tr>
    <th>Mistral-7b-it</th>
    <td>35.1±0.3</td>
    <td>28±0.2</td>
    <td>55.4±0.2</td>
    <td>55.1±0.1</td>
    <td>47.9±0.1</td>
    <td>51.9±0.6</td>
    <td>19.2±0.0</td>
    <td>12.6±0.0</td>
    <td>32.8±0.1</td>
  </tr>
  <tr>
    <th>Mixtral-8x7b-it</th>
    <td>35.1±0.2</td>
    <td>27.9±0.2</td>
    <td>55.3±0.2</td>
    <td>54.6±0.2</td>
    <td>47.7±0.1</td>
    <td>51.9±0.4</td>
    <td><b>19.4±0.0</b></td>
    <td><b>12.7±0.0</b></td>
    <td>33.2±0.1</td>
  </tr>
</table>

(a) Query2doc

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td>Contriever</td>
    <td>26.8</td>
    <td>20.2</td>
    <td>53.1</td>
    <td>55.1</td>
    <td>47.3</td>
    <td>51.2</td>
    <td>17.6</td>
    <td>12.6</td>
    <td>33.9</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="10">Performance by varying LLMs</th>
  </tr>
  <tr>
    <th>GPT-4o-mini</th>
    <td>37.3±0.1</td>
    <td>28.8±0.0</td>
    <td>55.6±0.1</td>
    <td>61.2±0.2</td>
    <td>53.1±0.1</td>
    <td>54±0.5</td>
    <td>18.7±0.0</td>
    <td>13.2±0.0</td>
    <td><b>35.7±0.6</b></td>
  </tr>
  <tr>
    <th>Claude-3-haiku</th>
    <td>36.7±0.1</td>
    <td>28.1±0.0</td>
    <td>55.6±0.1</td>
    <td><b>62.8±0.1</b></td>
    <td><b>54.7±0.1</b></td>
    <td><b>53.7±0.4</b></td>
    <td><b>19.3±0.0</b></td>
    <td><b>14±0.0</b></td>
    <td><b>36.2±0.6</b></td>
  </tr>
  <tr>
    <th>Gemini-1.5-flash</th>
    <td>35±0.1</td>
    <td>26.7±0.1</td>
    <td>55.2±0.2</td>
    <td>61±0.2</td>
    <td>52.9±0.2</td>
    <td>53.5±0.7</td>
    <td>18±0.0</td>
    <td>12.4±0.0</td>
    <td><b>35.7±0.5</b></td>
  </tr>
  <tr>
    <th>Llama-3.1-8b-it</th>
    <td>36.7±0.1</td>
    <td>28.4±0.1</td>
    <td>55.4±0.1</td>
    <td>61.2±0.2</td>
    <td>53.4±0.2</td>
    <td><b>53.6±0.7</b></td>
    <td>18.9±0.0</td>
    <td>13.6±0.0</td>
    <td>35.5±0.4</td>
  </tr>
  <tr>
    <th>Llama-3.1-70b-it</th>
    <td><b>40.4±0.2</b></td>
    <td><b>31.7±0.2</b></td>
    <td><b>55.9±0.1</b></td>
    <td>61.9±0.3</td>
    <td>54.1±0.2</td>
    <td><b>53.6±0.5</b></td>
    <td>19±0.2</td>
    <td>13.7±0.1</td>
    <td>35.4±0.7</td>
  </tr>
  <tr>
    <th>Mistral-7b-it</th>
    <td>36.3±0.1</td>
    <td>27.8±0.0</td>
    <td>55.3±0.1</td>
    <td>60.7±0.2</td>
    <td>52.7±0.2</td>
    <td>53.4±0.4</td>
    <td>19±0.0</td>
    <td>13.6±0.0</td>
    <td><b>35.8±0.7</b></td>
  </tr>
  <tr>
    <th>Mixtral-8x7b-it</th>
    <td>37.6±0.1</td>
    <td>29.1±0.1</td>
    <td>55.7±0.1</td>
    <td>61.3±0.2</td>
    <td>53.1±0.1</td>
    <td>53.3±0.3</td>
    <td>19.2±0.0</td>
    <td>13.7±0.0</td>
    <td><b>35.8±0.7</b></td>
  </tr>
</table>

(b) HyDE

Table 2: Fact verification performance using baseline retrievers and LLM-based query expansion methods, with the number of retrieved evidence set to five (\( k = 5 \)).

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">FEVER</th>
    <th colspan="2">SciFact</th>
    <th colspan="2">AVeriTeC</th>
  </tr>
  <tr>
    <th>Query2doc</th>
    <th>HyDE</th>
    <th>Query2doc</th>
    <th>HyDE</th>
    <th>Query2doc</th>
    <th>HyDE</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>75.8±0.1</td>
    <td><b>83.5±0.1</b></td>
    <td>52.8±0.4</td>
    <td>59.1±0.7</td>
    <td>40.4±0.2</td>
    <td>68±0.3</td>
  </tr>
  <tr>
    <td>Claude-3-haiku</td>
    <td>76.6±0.1</td>
    <td>77.8±0.1</td>
    <td>56.1±0.4</td>
    <td>53.8±0.2</td>
    <td>40.8±0.1</td>
    <td>62.3±0.2</td>
  </tr>
  <tr>
    <td>Gemini-1.5-flash</td>
    <td>69.9±0.3</td>
    <td>70.2±0.3</td>
    <td>50.8±0.6</td>
    <td><b>27.6±0.7</b></td>
    <td>44.1±0.1</td>
    <td>59.6±0.3</td>
  </tr>
  <tr>
    <td>Llama-3.1-8b-it</td>
    <td>68.5±1.0</td>
    <td>73±0.9</td>
    <td>53.9±0.5</td>
    <td>48.2±0.6</td>
    <td>37.4±1.1</td>
    <td>53.8±1.0</td>
  </tr>
  <tr>
    <td>Llama-3.1-70b-it</td>
    <td>78.3±0.7</td>
    <td>71.7±0.7</td>
    <td>57.5±0.3</td>
    <td>55±0.8</td>
    <td>48.1±0.8</td>
    <td>47±4.8</td>
  </tr>
  <tr>
    <td>Mistral-7b-it</td>
    <td>72.5±0.2</td>
    <td>75±0.2</td>
    <td>51.1±0.5</td>
    <td>49.4±0.8</td>
    <td>44.7±0.2</td>
    <td>55.6±0.3</td>
  </tr>
  <tr>
    <td>Mixtral-8x7b-it</td>
    <td>78.7±0.1</td>
    <td>81±0.1</td>
    <td>55.9±0.6</td>
    <td>54.7±0.7</td>
    <td>49.4±0.3</td>
    <td>56.7±1.3</td>
  </tr>
</table>

Table 3: Proportion of expanded queries containing sentences entailed by ground truth evidence.

2020) on the top five retrieved sentences, computing token-level and embedding-level similarity, respectively, based on a binary assignment between generated and reference sequences. For verdict prediction, we used GPT-4o-mini with the five retrieved evidence and evaluated performance using macro F1. Evaluation details are provided in Appendix B.

As shown in Table 2, both Query2doc and HyDE consistently outperformed BM25 and Contriever across all three datasets and for seven different backbone LLMs (three proprietary and four open models). The performance gap between each baseline and its respective expansion method was statistically significant (\( p<0.001 \)), demonstrating the effectiveness of the QE methods for evidence retrieval and, consequently, verdict prediction in these benchmarks. Results for \( k = 10 \) are available in Appendix F, showing a consistent trend.

Do LLM-generated documents include ground truth evidence? Table 3 presents the proportion of matched claims observed for the three datasets by using seven different LLMs. In most cases, more than 40% of the claims were matched, with a few exceptions. The lowest proportion (27.6%) was observed for SciFact when claims were expanded using HyDE with Gemini-1.5-flash—still a notable fraction. The highest proportion (83.5%) was observed for FEVER when using HyDE with GPT-4o-mini. Several examples of LLM-generated documents and gold evidence are provided in Table A5.

How does performance vary with the matching condition? Table 4 presents fact-verification performance based on whether LLM-generated documents contained sentences entailed by gold evidence, focusing on GPT-4o-mini as the QE model. We observed a consistent trend across the
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td>BM25</td>
    <td>ALL</td>
    <td>31</td>
    <td>25</td>
    <td>54</td>
    <td>51.2</td>
    <td>45.5</td>
    <td>48.6</td>
    <td>17.8</td>
    <td>11.6</td>
    <td>32</td>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>36.4±0.1</td>
    <td>29.3±0.1</td>
    <td>55.6±0.1</td>
    <td>55.1±0.2</td>
    <td>47.9±0.1</td>
    <td>52.5±0.5</td>
    <td>19.1±0.0</td>
    <td>12.4±0.0</td>
    <td>32.6±0.1</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>40.5±0.1</b></td>
    <td><b>32.8±0.1</b></td>
    <td><b>58.4±0.0</b></td>
    <td><b>63.3±0.4</b></td>
    <td><b>57.1±0.3</b></td>
    <td><b>53.7±0.3</b></td>
    <td><b>21.6±0.1</b></td>
    <td><b>17.6±0.1</b></td>
    <td><b>38.3±0.3</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>23.8±0.3</td>
    <td>18.5±0.2</td>
    <td>44.9±0.1</td>
    <td>45.9±0.4</td>
    <td>37.6±0.3</td>
    <td>49.0±0.4</td>
    <td>17.4±0.0</td>
    <td>9.4±0.0</td>
    <td>27.6±0.1</td>
  </tr>
  <tr>
    <td>Contriever</td>
    <td>ALL</td>
    <td>26.8</td>
    <td>20.2</td>
    <td>53.1</td>
    <td>55.1</td>
    <td>47.3</td>
    <td>51.2</td>
    <td>17.6</td>
    <td>12.6</td>
    <td>33.9</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>37.3±0.1</td>
    <td>28.8±0.0</td>
    <td>55.6±0.1</td>
    <td>61.2±0.2</td>
    <td>53.1±0.1</td>
    <td>54±0.5</td>
    <td>18.7±0.0</td>
    <td>13.2±0.0</td>
    <td>35.7±0.6</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>40±0.1</b></td>
    <td><b>30.9±0.1</b></td>
    <td><b>57.8±0.0</b></td>
    <td><b>68.4±0.3</b></td>
    <td><b>61.4±0.3</b></td>
    <td><b>57.1±0.2</b></td>
    <td><b>19.8±0.0</b></td>
    <td><b>15.5±0.0</b></td>
    <td><b>37±0.2</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>23.4±0.4</td>
    <td>17.9±0.3</td>
    <td>41.9±0.2</td>
    <td>50.8±0.5</td>
    <td>41.2±0.4</td>
    <td>48.9±0.4</td>
    <td>16.4±0.0</td>
    <td>8.3±0.1</td>
    <td>30.3±0.4</td>
  </tr>
</table>

Table 4: Fact verification performance based on whether documents generated by query expansion methods with GPT-4o-mini contain sentences entailed by gold evidence. Results for other LLMs are presented in Table A2.

three datasets and two expansion methods: performance measured on matched claims (where LLM-generated documents contain sentences entailed by gold evidence) was significantly higher than that on both all and unmatched claims, with statistical significance at \( p < 0.001 \). Moreover, with a few exceptions, performance on unmatched claims was lower than that of the corresponding baseline methods without query expansion—BM25 for Query2doc and Contriever for HyDE. These trends were consistent across seven different LLMs and for \( k = 10 \), as shown in Table A2 and Table A3.

5 Discussion

Query2doc and HyDE boosted fact-verification performance across three benchmarks. Our findings support the effectiveness of LLM-based query expansion in zero-shot retrieval tasks (Gao et al., 2023; Jagerman et al., 2023; Lei et al., 2024; Wang et al., 2024a), particularly on fact-verification benchmarks. The increased verification performance through query expansion further highlights the crucial role of retrieving suitable evidence for claim verification (Chen et al., 2024a), suggesting a need for further research into evidence retrieval methods, such as claim decomposition (Chen et al., 2022; Fan et al., 2020).

Documents generated by LLM-based query expansion methods frequently included sentences that were entailed by ground-truth evidence, indicating potential knowledge leakage. By applying an NLI-based matching algorithm, we examined whether LLMs reproduced gold evidence in response to query expansion prompts. Our results suggest that the seven LLMs studied in this paper were likely exposed to knowledge sources from the three benchmarks during training. This observation aligns with prior research on data leakage (Kandpal et al., 2023; Samuel et al., 2025) and memorization in LLMs (Cheng et al., 2024; Burns et al., 2022), representing the first empirical demonstration in the context of fact verification and query expansion.

Performance improvements from query expansion were consistent only when LLM-generated documents contained sentences entailed by gold evidence. This finding suggests that the success of hypothetical document generation may be largely attributable to the LLM’s internal knowledge encompassing benchmark knowledge sources. Furthermore, these results indicate that LLM-based query expansion may be limited in real-world scenarios that require retrieving niche or novel knowledge. Future research could address these limitation by incorporating external knowledge sources into the query expansion process, as demonstrated in a recent study (Lei et al., 2024).

6 Conclusion

This study examined the leakage hypothesis for query expansion, questioning whether an LLM generates truly hypothetical documents or merely reproduces what was exposed during the LLM’s training phase. Through experiments on three widely used fact-verification benchmarks, we observed a consistent trend: on average, retrieval and verification performance increased only when the expanded query was entailed by gold evidence. This finding suggests that knowledge leakage may be present in these benchmarks, artificially inflating the perceived effectiveness of LLM-based query expansion. Since verifying claims involving niche or novel knowledge is often essential in real-world scenarios, such leakage can hinder accurate evaluation and, consequently, impede method development. These results highlight the need for future research on LLM-based query expansion methods capable of handling unknown queries, as well as the creation of evaluation benchmarks that more accurately reflect real-world conditions involving previously unseen knowledge.
Limitations

While our analysis provides the evidence that certain LLM behaviors can signal memorization and that these behaviors correlate with downstream performance gains, we acknowledge three areas where further validation is needed. (1) Causal mechanism: Our experiments focus on LLM behaviors rather than tracing whether specific training data directly results in the generation of associated knowledge. Consequently, we do not claim a causal link between data leakage and generation in response to expansion prompts. Establishing such a connection remains an open challenge and would benefit from controlled studies that expose models to synthetic or novel knowledge (Liu et al., 2024; Kasai et al., 2024). (2) NLI-based matching: To corroborate the NLI-based automatic assessment, we manually annotated a representative sample and observed a consistent trend (Table A4) with the automatic results (Table 4). Adopting a stronger NLI model would further strengthen our conclusions. (3) Generalizability: Our analysis focused on three datasets within the fact-verification domain, and hence the findings are limited in scope. Further experiments are necessary to determine whether these findings generalize to other tasks, such as factual QA (Joshi et al., 2017; Kwiatkowski et al., 2019). The methodology presented here could be adapted to those settings.

Ethics and Impact Statement

Our findings suggest that knowledge leakage may exist in three popular fact-verification benchmarks; namely, LLMs may have been exposed to information related to the benchmarks’ gold evidence during pretraining and subsequently reproduced it in response to query-expansion prompts. This observation has important implications for both fact verification and broader benchmark-oriented NLP research, as the performance of LLM-based methods may be artificially inflated due to the opaqueness of the pretraining stage. Consequently, more trustworthy evaluation frameworks and transparent LLM development practices, such as those exemplified by Groeneveld et al. (2024) and Gao et al. (2020), are encouraged to better reflect real-world performance. We used ChatGPT to proofread portions of this manuscript.

Acknowledgments

This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the Innovative Human Resource Development for Local Intellectualization (IITP-2025-RS-2022-00156360) and Graduate School of Metaverse Convergence support (IITP-2025-RS-2024-00430997) programs, supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation). Kunwoo Park is the corresponding author.

References

Gianni Amati and Cornelis Joost Van Rijsbergen. 2002. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Transactions on Information Systems (TOIS), 20(4):357–389.

Hiteshwar Kumar Azad and Akshay Deepak. 2019. Query expansion techniques for information retrieval: a survey. Information Processing & Management, 56(5):1698–1735.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.

Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Média, Inc.".

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2022. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827.

Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. 2024a. Complex claim verification with evidence retrieved in the wild. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies (Volume 1: Long Papers), pages 3569–3587, Mexico City, Mexico. Association for Computational Linguistics.

Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. 2022. Generating literal and implied sub-questions to fact-check complex claims. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3495–3516, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Xinran Chen, Xuanang Chen, Ben He, Tengfei Wen, and Le Sun. 2024b. Analyze, generate and refine: Query expansion with LLMs for zero-shot open-domain QA. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11908–11922, Bangkok, Thailand. Association for Computational Linguistics.

Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. 2024. Can AI assistants know what they don’t know? In Forty-first International Conference on Machine Learning.

Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2023. Benchmark probing: Investigating data leakage in large language models. In NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly.

Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8706–8719, Mexico City, Mexico. Association for Computational Linguistics.

Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Zhifang Sui, and Lei Li. 2024. Statistical knowledge assessment for large language models. Advances in Neural Information Processing Systems, 36.

Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547–5569. PMLR.

Angela Fan, Aleksandra Piktus, Fabio Petroni, Guillaume Wenzek, Marzieh Saeidi, Andreas Vlachos, Antoine Bordes, and Sebastian Riedel. 2020. Generating fact checking briefs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7147–7161, Online. Association for Computational Linguistics.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.

Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, Toronto, Canada. Association for Computational Linguistics.

Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muenninghoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. 2024. OLMo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789–15809, Bangkok, Thailand. Association for Computational Linguistics.

Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178–206.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Gautier Izard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.

Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expansion by prompting large language models. arXiv preprint arXiv:2305.03653.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language
models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696–15707. PMLR.

Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. 2024. Realtime qa: what’s the answer right now? Advances in Neural Information Processing Systems, 36.

Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenneth Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.

Victor Lavrenko and W. Bruce Croft. 2001. Relevance based language models. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’01, page 120–127, New York, NY, USA. Association for Computing Machinery.

Yibin Lei, Yu Cao, Tianyi Zhou, Tao Shen, and Andrew Yates. 2024. Corpus-steered query expansion with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 393–401.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356–2362.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Bill MacCartney. 2009. Natural language inference. Stanford University.

Iain Mackie, Shubham Chatterjee, and Jeffrey Dalton. 2023a. Generative relevance feedback with large language models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 2026–2031, New York, NY, USA. Association for Computing Machinery.

Iain Mackie, Shubham Chatterjee, and Jeffrey Dalton. 2023b. Generative relevance feedback with large language models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2026–2031.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.

Christopher D Manning. 2009. An introduction to information retrieval.

Sebastião Miranda, David Nogueira, Afonso Mendes, Andreas Vlachos, Andrew Secker, Rebecca Garrett, Jeff Mitchel, and Zita Marinho. 2019. Automated fact checking in the news room. In The world wide web conference, pages 3579–3583.

Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, Giovanni Da San Martino, et al. 2021. Automated fact-checking for assisting human fact-checkers. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 4551–4558. International Joint Conferences on Artificial Intelligence Organization.

JJ Rocchio. 1971. Relevance feedback in information retrieval. The SMART Retrieval System-Experiments in Automatic Document Processing/Prentice Hall.

Vinay Samuel, Yue Zhou, and Henry Peng Zou. 2025. Towards data contamination detection for modern large language models: Limitations, inconsistencies, and oracle challenges. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5058–5070, Abu Dhabi, UAE. Association for Computational Linguistics.

Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. Averitec: a dataset for real-world claim verification with evidence from the web. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 65128–65167.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_eval: An extremely fast python interface to trec_eval. In SIGIR. ACM.

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association for Computational Linguistics.

Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9414–9423, Singapore. Association for Computational Linguistics.

Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. 2024a. Searching for best practices in retrieval-augmented generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17716–17736, Miami, Florida, USA. Association for Computational Linguistics.

Yuxia Wang, Minghan Wang, and Preslav Nakov. 2024b. Rethinking STS and NLI in large language models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 965–982, St. Julian’s, Malta. Association for Computational Linguistics.

Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. 2024a. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244.

Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024b. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824.

Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665, Toronto, Canada. Association for Computational Linguistics.

Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, and Kunwoo Park. 2024. HerO at AVeriTeC: The herd of open large language models for verifying real-world claims. In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 130–136, Miami, Florida, USA. Association for Computational Linguistics.

Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024. R-tuning: Instructing large language models to say ‘I don’t know’. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7113–7139, Mexico City, Mexico. Association for Computational Linguistics.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. Dense text retrieval based on pretrained language models: A survey. ACM Trans. Inf. Syst., 42(4).

Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774.

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107.

Appendix

A Target Dataset

This study used three fact verification benchmarks. From the BEIR benchmark (Thakur et al., 2021), we used the test set for FEVER and the train (505) and test (188) sets for SciFact. For AVeriTeC, we used the train (3,063) and dev (500) sets, as its test set is not publicly available. We excluded claims for which gold evidence was unavailable. Table 1 presents the descriptive statistics of the three datasets used in our experiments.

B Evaluation Metrics

Below, we describe the details of evaluation metric for evidence retrieval. Recall@K and NDCG@K are widely used evaluation metrics for information
retrieval (Manning, 2009), adopted in this study for evaluating evidence retrieval performance for FEVER and SciFact. Recall@K assesses the proportion of relevant items retrieved within the top K results. NDCG@K measures the quality of ranked results by considering both the relevance of retrieved documents and their positions within the ranking. We used pyrec_eval (Van Gysel and de Rijke, 2018) to measure Recall@K and NDCG@K.

\[
S(\hat{Y}, Y) = \frac{1}{|Y|} \max_{\hat{y} \in \hat{Y}} \sum_{y \in Y} f(\hat{y}, y) X(\hat{y}, y)
\]

(3)

For AVeriTeC, where gold evidence is not selected from a knowledge store but written by human annotators, we used METEOR and BERTScore with the Hungarian algorithm. Equation 3 presents the algorithm, where \( f \) is a pairwise scoring function, and \( X \) is a boolean function representing the assignment between the generated sequences \( \hat{Y} \) and the reference sequences \( Y \). To measure token-level and embedding-level similarity, we used METEOR and BERTScore for \( f \), respectively. METEOR was computed using NLTK (Bird et al., 2009), and BERTScore was calculated with DeBERTa-xlarge-MNLI\textsuperscript{3}. We reported observations for varying \( k \): Table 2 and Table 4 for \( k = 5 \), and Table A1 and Table A3 for \( k = 10 \).

C Experimental Setups

For HyDE, we encoded queries and documents using Contriever\textsuperscript{4}. For Query2doc, we used BM25 provided in PySerini (Lin et al., 2021). Following the same settings in previous study (Gao et al., 2023), we set LLM hyperparameters as follows: temperature as 0.7, top_p as 1.0, and max_tokens as 512. We used the Mann–Whitney U test for statistical testing on performance differences.

D Computing Environment

We ran experiments using two machines. The first machine has four Nvidia RTX A6000 GPUs (48GB per GPU) and 256GB RAM. The second machine has two Nvidia H100 GPUs (80GB per GPU) and 480GB RAM. The experiments were conducted in a computing environment with the following configuration: Python 3.11.10, PyTorch 2.3.1, Transformers 4.43.4, vLLM 0.5.3, pyrec-eval 0.5, Faiss 1.8, Pyserini 0.40.0, NLTK 3.9.1, bert-score 0.3.13, rouge-score 0.1.2.

We used GPT-4o-mini, Claude 3 Haiku, Gemini 1.5 Flash via API, while Llama 3.1 (8B and 70B) and Mistral 7B, and Mixtral 8x7B were accessed through pretrained checkpoints. The model ids and parameter sizes are provided below.

• GPT-4o-mini: gpt-4o-mini-2024-07-18 (Parameter size: unknown)
• Claude-3-haiku: claude-3-haiku-20240307 (Parameter size: unknown)
• Gemini-1.5-flash: gemini-1.5-flash (Parameter size: unknown)
• Llama-3.1-8b-it: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct (Parameter size: 8B)
• Llama-3.1-70b-it: https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct (Parameter size: 70B)
• Mistral-7b-it: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 (Parameter size: 7B)
• Mixtral-8x7b-it: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 (Parameter size: 46.7B)

E Prompts

Query Expansion Figure A1a, A1b, and A1c illustrate the HyDE prompts used in this study where the original prompt is adapted to each dataset. For Query2doc, we used the same prompt by following the suggestion in Wang et al. (2023), as shown in Figure A1d.

Verdict Prediction Figure A2 presents the prompt used for verdict prediction with GPT-4o-mini.

Natural Language Inference We used GPT-4o-mini for natural language inference, employing a prompt proposed in a previous study (Wang et al., 2024b), as illustrated in Figure A3. To support its validity, two authors manually annotated the labels
Please write a wikipedia passage to verify the claim.
Claim: [CLAIM]
Passage: [OUTPUT]

(a) The prompt used for HyDE in the FEVER dataset.

Please write a scientific paper passage to support/refute the claim.
Claim: [CLAIM]
Passage: [OUTPUT]

(b) The prompt used for HyDE in the SciFact dataset.

Please write a fact-checking article to verify the claim.
Claim: [CLAIM]
Passage: [OUTPUT]

(c) The prompt used for HyDE in the AVeriTeC dataset.

Write a passage that answers the following query:
[CLAIM]
[OUTPUT]

(d) The prompt used for Query2doc.

Figure A1: Prompts used for query expansion.

Your task is to predict the verdict of a claim based on the provided evidence. Select one of the following labels: [LABEL]. Generate only the label without additional explanation or content.

Claim: [CLAIM]

Evidence 1: [EVIDENCE 1]
...
Evidence N: [EVIDENCE N]

Label: [OUTPUT]

Figure A2: The prompt used for fact verification with GPT-4o-mini. \( N \) denotes the total number of retrieved evidence.

Given the premise sentence S1, determine if the hypothesis sentence S2 is entailed or contradicted or neutral, by three labels: entailment, contradiction, neutral.
Respond only with one of the labels.
S1: [GOLD EVIDENCE]
S2: [LLM-GENERATED SENTENCE]
Label: [OUTPUT]

Figure A3: The prompt used for NLI.

for a randomly selected set of 200 pairs following the guidelines presented in Figure A4. The GPT-based NLI model achieved an F1 score of 0.8 on the sampled data.

F Supplementary Results

LLM comparison for fact verification Table 2 present the results for evidence retrieval and verdict prediction by varying backbone LLMs for query expansion. We observed that Llama-3.1-70b-it generally performed well when used with Query2doc. For HyDE, while Llama-3.1-70b-it achieved the best performance on FEVER, Claude-3-haiku obtained higher evaluation scores in SciFact and AVeriTeC.

Proportion of matched claims across different benchmarks Table 3 presents the distribution of matched claims across three datasets, varying the LLMs used for query expansion. On average, the estimated proportion was higher for FEVER than for the other two datasets. While FEVER was constructed using public Wikipedia documents, SciFact is based on scientific literature, covering niche knowledge, and AVeriTeC is the most recent dataset based on web documents collected by human annotators, covering recent knowledge. Given these characteristics, the highest proportion observed in FEVER partially supports the validity of the NLI-based estimation.

Effects of retrieving more evidence Table A1 presents the fact verification performance with an increased number of retrieved evidence (\( k = 10 \)). Performance improves in every case across different LLMs and datasets compared to the results with \( k = 5 \). Table A3 shows performance depending on the matching condition, showing a consistent trend with Table A2 when retrieving more evidence.

Manual annotation To support the validity of the NLI-based matching algorithm, we conducted manual annotations on a sampled dataset. Two authors independently reviewed documents generated by Query2doc and HyDE for all 500 samples in the AVeriTeC development set, following the guideline presented in Figure A5. A claim was labeled as *matched* if the LLM-generated document contained all or part of any gold evidence. For the backbone LLMs, we used Llama-3.1-70b-it for Query2doc and Claude-3-haiku for HyDE, as these models achieved the best performance for their respective methods. The two annotators achieved
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>FI</th>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td>BM25</td>
    <td>37.1</td>
    <td>27.1</td>
    <td>55.3</td>
    <td>58.4</td>
    <td>48.3</td>
    <td>50.6</td>
    <td>21</td>
    <td>15</td>
    <td>33.1</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="10">Performance by varying LLMs</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>44.2±0.1</td>
    <td>32.0±0.1</td>
    <td>57.0±0.1</td>
    <td>64.0±0.0</td>
    <td>51.4±0.1</td>
    <td>53.2±0.6</td>
    <td>22.3±0.0</td>
    <td>15.9±0.0</td>
    <td>34.6±0.1</td>
  </tr>
  <tr>
    <td>Claude-3-haiku</td>
    <td>43.4±0.1</td>
    <td>31.0±0.1</td>
    <td>56.9±0.2</td>
    <td>64.3±0.2</td>
    <td>51.5±0.1</td>
    <td>53.0±0.5</td>
    <td>22.5±0.0</td>
    <td>16.0±0.0</td>
    <td>34.7±0.1</td>
  </tr>
  <tr>
    <td>Gemini-1.5-flash</td>
    <td>43.4±0.1</td>
    <td>31.7±0.0</td>
    <td>56.9±0.1</td>
    <td><b>64.7±0.2</b></td>
    <td><b>52.8±0.1</b></td>
    <td><b>53.1±0.6</b></td>
    <td>22.3±0.0</td>
    <td>16.2±0.0</td>
    <td><b>34.7±0.2</b></td>
  </tr>
  <tr>
    <td>Llama-3-1.8b-it</td>
    <td>43.6±0.1</td>
    <td>31.3±0.1</td>
    <td>56.8±0.1</td>
    <td>63.2±0.3</td>
    <td>51.2±0.1</td>
    <td>53.1±0.5</td>
    <td>22.3±0.0</td>
    <td>15.9±0.0</td>
    <td>34.5±0.1</td>
  </tr>
  <tr>
    <td>Llama-3.1-70b-it</td>
    <td><b>46.1±0.2</b></td>
    <td><b>33.7±0.1</b></td>
    <td><b>57.2±0.2</b></td>
    <td><b>64.6±0.2</b></td>
    <td><b>52.5±0.1</b></td>
    <td><b>53.1±0.3</b></td>
    <td><b>22.7±0.1</b></td>
    <td><b>16.3±0.0</b></td>
    <td><b>34.8±0.1</b></td>
  </tr>
  <tr>
    <td>Mistral-7b-it</td>
    <td>43.2±0.2</td>
    <td>30.8±0.2</td>
    <td>56.8±0.1</td>
    <td>63.1±0.2</td>
    <td>51±0.1</td>
    <td>52.6±0.4</td>
    <td>22.5±0.0</td>
    <td>16.2±0.0</td>
    <td>34.6±0.1</td>
  </tr>
  <tr>
    <td>Mistral-8x7b-it</td>
    <td>43.4±0.2</td>
    <td>30.8±0.2</td>
    <td>56.8±0.2</td>
    <td>63.6±0.2</td>
    <td>51.3±0.1</td>
    <td><b>52.8±0.4</b></td>
    <td>22.6±0.0</td>
    <td>16.1±0.0</td>
    <td><b>34.9±0.1</b></td>
  </tr>
</table>

(a) Query2doc

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>FI</th>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td>Contriever</td>
    <td>34.4</td>
    <td>22.8</td>
    <td>54.8</td>
    <td>65</td>
    <td>51.2</td>
    <td>54</td>
    <td>20.8</td>
    <td>16.1</td>
    <td>34.8</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="10">Performance by varying LLMs</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>46.7±0.1</td>
    <td>32.1±0.0</td>
    <td>56.7±0.1</td>
    <td>70.1±0.1</td>
    <td>56.7±0.1</td>
    <td>54.6±0.3</td>
    <td>22.3±0.0</td>
    <td>17.0±0.0</td>
    <td>36.6±0.4</td>
  </tr>
  <tr>
    <td>Claude-3-haiku</td>
    <td>46.2±0.0</td>
    <td>31.4±0.0</td>
    <td>56.7±0.1</td>
    <td><b>71.6±0.2</b></td>
    <td><b>58.3±0.1</b></td>
    <td><b>55±0.3</b></td>
    <td><b>22.8±0.0</b></td>
    <td><b>17.8±0.0</b></td>
    <td><b>37.6±0.3</b></td>
  </tr>
  <tr>
    <td>Gemini-1.5-flash</td>
    <td>44.2±0.1</td>
    <td>29.9±0.1</td>
    <td>56.5±0.1</td>
    <td>69.8±0.1</td>
    <td>56.6±0.2</td>
    <td><b>54.8±0.3</b></td>
    <td>21.6±0.0</td>
    <td>16.3±0.0</td>
    <td>37.1±0.8</td>
  </tr>
  <tr>
    <td>Llama-3.1-8b-it</td>
    <td>46.4±0.1</td>
    <td>31.8±0.1</td>
    <td>56.6±0.1</td>
    <td>70.1±0.2</td>
    <td>57±0.1</td>
    <td>54.3±0.5</td>
    <td>22.4±0.0</td>
    <td>17.4±0.0</td>
    <td>36.8±0.3</td>
  </tr>
  <tr>
    <td>Llama-3.1-70b-it</td>
    <td><b>49.7±0.2</b></td>
    <td><b>35±0.2</b></td>
    <td><b>57±0.1</b></td>
    <td>70.8±0.2</td>
    <td>57.8±0.2</td>
    <td><b>55±0.5</b></td>
    <td>22.5±0.2</td>
    <td>17.5±0.1</td>
    <td><b>36.7±0.9</b></td>
  </tr>
  <tr>
    <td>Mistral-7b-it</td>
    <td>46.1±0.1</td>
    <td>31.2±0.0</td>
    <td>56.5±0.1</td>
    <td>69.8±0.2</td>
    <td>56.4±0.2</td>
    <td><b>54.8±0.3</b></td>
    <td>22.7±0.0</td>
    <td>17.5±0.0</td>
    <td><b>37.3±0.5</b></td>
  </tr>
  <tr>
    <td>Mistral-8x7b-it</td>
    <td>47.6±0.0</td>
    <td>32.6±0.0</td>
    <td>56.9±0.2</td>
    <td>70.2±0.2</td>
    <td>56.8±0.1</td>
    <td><b>54.8±0.5</b></td>
    <td><b>22.8±0.0</b></td>
    <td>17.6±0.0</td>
    <td><b>37.2±0.5</b></td>
  </tr>
</table>

(b) HyDE

Table A1: Fact verification performance using baseline retrievers and LLM-based query expansion methods, with the number of retrieved evidence set to ten (\( k = 10 \)). We report the average performance of query expansion methods along with standard errors, obtained by repeating the generations eight times.

Premise: [GOLD EVIDENCE]
Hypothesis:
[LLM-GENERATED SENTENCE]

Given the premise, determine whether the hypothesis is entailed.

☐ Entailment
☐ Non-Entailment

Figure A4: Manual labeling guidelines for natural language inference.

Claim: [CLAIM]
Gold Evidence: [GOLD EVIDENCE]
LLM-generated Document:
[LLM-GENERATED DOCUMENT]

Determine whether the LLM-generated document contains the whole or part of any gold evidence.

☐ Included
☐ Not Included

Figure A5: Manual labeling guidelines for determining whether LLM-generated documents contain gold evidence.

a high-level of inter-annotator agreement, with a Cohen’s kappa of 0.837 across 1,000 generations. The estimated proportion of matched claims were 50.8% and 59.6%, respectively, closely aligning with those from the NLI-based method, with differences falling within the error margin.

Table A4 presents performance depending on manually annotated matching conditions, showing a consistent trend with the results from the NLI-based method (Table 4 and Table A2).

G Qualitative Analysis

Table A5 presents examples of LLM-generated documents along with gold evidence. In example (a), the claim concerns Nigeria’s history, and the gold evidence specifies the period under military rule, which was reproduced in the generated document. Example (b) pertains to U.S. Supreme Court Justice Ruth Bader Ginsberg, where the gold evidence provides her bibliography and medical history. The LLM-generated text includes this information along with specific years. Notably, it also introduces an additional fact about lung cancer, which is not covered by the gold evidence. Examples (c) and (d) illustrate unmatched cases where the generated text contains factual errors.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>35.2±0.1</td>
    <td>28.3±0.1</td>
    <td>55.4±0.1</td>
    <td>36.0±1.1</td>
    <td>48.2±0.1</td>
    <td>52.0±0.5</td>
    <td>19.3±0.0</td>
    <td>12.5±0.0</td>
    <td>33.1±0.1</td>
  </tr>
  <tr>
    <td>M</td>
    <td>38.7±0.1</td>
    <td>31.2±0.1</td>
    <td>58.0±0.0</td>
    <td>63.7±0.3</td>
    <td>56.7±0.3</td>
    <td>54.1±0.2</td>
    <td>22.0±0.0</td>
    <td>17.7±0.0</td>
    <td>38.8±0.3</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>23.8±0.1</td>
    <td>18.5±0.1</td>
    <td>44.7±0.1</td>
    <td>46.2±0.6</td>
    <td>37.2±0.4</td>
    <td>48.0±0.5</td>
    <td>17.4±0.0</td>
    <td>8.9±0.0</td>
    <td>27.6±0.1</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>36.7±0.0</td>
    <td>28.1±0.0</td>
    <td>55.6±0.0</td>
    <td>62.8±0.1</td>
    <td>54.7±0.1</td>
    <td>53.7±0.4</td>
    <td>19.3±0.0</td>
    <td>14.0±0.0</td>
    <td>36.2±0.6</td>
  </tr>
  <tr>
    <td>M</td>
    <td>39.9±0.2</td>
    <td>30.7±0.1</td>
    <td>57.5±0.1</td>
    <td>71.1±0.5</td>
    <td>63.8±0.4</td>
    <td>57.5±0.3</td>
    <td>20.4±0.1</td>
    <td>16.5±0.1</td>
    <td>37.8±0.3</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>25.5±0.2</td>
    <td>18.9±0.2</td>
    <td>45.0±2</td>
    <td>53.0±0.5</td>
    <td>44.1±0.5</td>
    <td>48.8±0.4</td>
    <td>17.3±0.1</td>
    <td>9.8±0.1</td>
    <td>31.9±0.3</td>
  </tr>
</table>

(a) Claude-3-haiku

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>36.2±0.1</td>
    <td>29.0±0.1</td>
    <td>55.8±0.1</td>
    <td>56.2±0.2</td>
    <td>49.7±0.1</td>
    <td>52.2±0.5</td>
    <td>18.9±0.0</td>
    <td>12.2±0.0</td>
    <td>33.3±0.2</td>
  </tr>
  <tr>
    <td>M</td>
    <td>38.5±0.1</td>
    <td>31.4±0.0</td>
    <td>58.8±0.0</td>
    <td>63.4±0.5</td>
    <td>56.9±0.4</td>
    <td>55.1±0.5</td>
    <td>20.5±0.0</td>
    <td>16.5±0.1</td>
    <td>38.1±0.4</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>30.7±0.1</td>
    <td>24.1±0.1</td>
    <td>48.0±2</td>
    <td>48.9±0.4</td>
    <td>41.6±0.3</td>
    <td>48.7±0.3</td>
    <td>17.6±0.0</td>
    <td>9.4±0.1</td>
    <td>28.5±0.2</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>35.0±0.1</td>
    <td>26.7±0.1</td>
    <td>55.2±0.2</td>
    <td>61.0±0.2</td>
    <td>52.9±0.2</td>
    <td>53.3±0.7</td>
    <td>18.4±0.0</td>
    <td>12.4±0.0</td>
    <td>35.7±0.5</td>
  </tr>
  <tr>
    <td>M</td>
    <td>37.1±0.1</td>
    <td>28.2±0.1</td>
    <td>59.0±1</td>
    <td>67.1±0.6</td>
    <td>60.1±0.4</td>
    <td>57.3±0.5</td>
    <td>18.8±0.0</td>
    <td>14.7±0.1</td>
    <td>37.3±0.2</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>30.1±0.1</td>
    <td>23.1±0.1</td>
    <td>45.0±2</td>
    <td>58.7±0.4</td>
    <td>50.2±0.2</td>
    <td>52.0±3</td>
    <td>16.8±0.1</td>
    <td>9.2±0.1</td>
    <td>31.6±0.3</td>
  </tr>
</table>

(b) Gemini-1.5-flash

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>35.7±0.1</td>
    <td>28.6±0.2</td>
    <td>55.6±0.2</td>
    <td>54.9±0.2</td>
    <td>47.8±0.2</td>
    <td>51.9±0.3</td>
    <td>19.0±0.0</td>
    <td>12.4±0.0</td>
    <td>32.2±0.2</td>
  </tr>
  <tr>
    <td>M</td>
    <td>39.0±0.2</td>
    <td>31.4±0.2</td>
    <td>58.6±0.1</td>
    <td>63.9±0.4</td>
    <td>57.5±0.4</td>
    <td>55.1±0.4</td>
    <td>21.7±0.1</td>
    <td>17.3±0.1</td>
    <td>36.6±0.3</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>28.5±0.3</td>
    <td>22.0±0.3</td>
    <td>48.5±0.3</td>
    <td>44.2±0.6</td>
    <td>36.5±0.4</td>
    <td>46.0±0.3</td>
    <td>17.4±0.0</td>
    <td>9.4±0.1</td>
    <td>28.5±0.1</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>36.7±0.1</td>
    <td>28.4±0.1</td>
    <td>55.4±0.1</td>
    <td>61.0±0.2</td>
    <td>53.7±0.2</td>
    <td>53.0±0.7</td>
    <td>18.0±0.0</td>
    <td>13.6±0.0</td>
    <td>35.7±0.4</td>
  </tr>
  <tr>
    <td>M</td>
    <td>39.2±0.1</td>
    <td>30.6±0.1</td>
    <td>58.2±0.1</td>
    <td>68.7±0.7</td>
    <td>62.3±0.6</td>
    <td>56.2±0.4</td>
    <td>20.9±0.1</td>
    <td>16.1±0.1</td>
    <td>37.7±0.2</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>29.2±0.2</td>
    <td>22.4±0.2</td>
    <td>46.8±0.2</td>
    <td>54.1±0.8</td>
    <td>45.1±0.6</td>
    <td>50.6±0.4</td>
    <td>17.7±0.1</td>
    <td>10.6±0.1</td>
    <td>31.8±0.3</td>
  </tr>
</table>

(c) Llama-3.1-8b-it

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>38.3±0.1</td>
    <td>31.0±1</td>
    <td>56.1±0.1</td>
    <td>56.4±0.3</td>
    <td>49.2±0.1</td>
    <td>52.4±0.7</td>
    <td>19.3±0.1</td>
    <td>12.7±0.0</td>
    <td>33.4±0.2</td>
  </tr>
  <tr>
    <td>M</td>
    <td>41.3±0.1</td>
    <td>33.6±0.1</td>
    <td>58.6±0.1</td>
    <td>65.8±0.3</td>
    <td>58.2±0.3</td>
    <td>54.9±0.5</td>
    <td>21.6±0.1</td>
    <td>17.2±0.1</td>
    <td>38.1±0.3</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>27.6±0.4</td>
    <td>21.7±0.4</td>
    <td>45.9±0.3</td>
    <td>44.9±0.5</td>
    <td>37.0±0.3</td>
    <td>47.9±0.2</td>
    <td>17.3±0.1</td>
    <td>8.6±0.1</td>
    <td>27.6±0.2</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>40.4±0.2</td>
    <td>31.7±0.2</td>
    <td>55.9±0.1</td>
    <td>61.9±0.3</td>
    <td>54.1±0.2</td>
    <td>53.8±0.5</td>
    <td>19.0±0.2</td>
    <td>13.7±0.1</td>
    <td>35.4±0.7</td>
  </tr>
  <tr>
    <td>M</td>
    <td>44.3±0.5</td>
    <td>35.0±0.5</td>
    <td>58.4±0.1</td>
    <td>69.2±0.4</td>
    <td>62.1±0.4</td>
    <td>56.7±0.3</td>
    <td>20.9±0.2</td>
    <td>17.3±0.3</td>
    <td>38.3±0.2</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>30.4±0.3</td>
    <td>23.3±0.3</td>
    <td>48.9±0.2</td>
    <td>52.9±0.7</td>
    <td>44.3±0.6</td>
    <td>48.9±0.5</td>
    <td>17.4±0.1</td>
    <td>10.6±0.2</td>
    <td>31.5±0.4</td>
  </tr>
</table>

(d) Llama-3.1-70b-it

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>35.1±0.3</td>
    <td>28.0±0.2</td>
    <td>55.4±0.2</td>
    <td>55.1±0.3</td>
    <td>47.9±0.1</td>
    <td>51.9±0.6</td>
    <td>19.2±0.0</td>
    <td>12.6±0.0</td>
    <td>32.8±0.1</td>
  </tr>
  <tr>
    <td>M</td>
    <td>39.0±0.2</td>
    <td>31.3±0.2</td>
    <td>58.2±0.1</td>
    <td>64.8±0.3</td>
    <td>58.2±0.2</td>
    <td>54.8±0.5</td>
    <td>21.5±0.1</td>
    <td>17.3±0.1</td>
    <td>37.5±0.5</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>24.8±0.4</td>
    <td>19.1±0.3</td>
    <td>46.7±0.2</td>
    <td>43.4±0.5</td>
    <td>37.2±0.3</td>
    <td>48.3±0.3</td>
    <td>17.4±0.0</td>
    <td>8.8±0.1</td>
    <td>27.7±0.2</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>36.3±0.1</td>
    <td>27.8±0.0</td>
    <td>55.3±0.1</td>
    <td>60.7±0.2</td>
    <td>52.7±0.2</td>
    <td>53.4±0.4</td>
    <td>19.0±0.0</td>
    <td>13.6±0.0</td>
    <td>35.8±0.7</td>
  </tr>
  <tr>
    <td>M</td>
    <td>39.2±0.2</td>
    <td>30.1±0.1</td>
    <td>57.7±0.1</td>
    <td>67.8±0.5</td>
    <td>60.9±0.4</td>
    <td>55.5±0.7</td>
    <td>20.1±0.1</td>
    <td>16.1±0.1</td>
    <td>36.8±0.4</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>27.5±0.2</td>
    <td>20.9±0.1</td>
    <td>46.4±0.2</td>
    <td>53.8±0.3</td>
    <td>44.8±0.3</td>
    <td>50.4±0.3</td>
    <td>17.7±0.1</td>
    <td>10.6±0.1</td>
    <td>31.1±0.4</td>
  </tr>
</table>

(e) Mistral-7b-it

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AVeriTeC</th>
  </tr>
  <tr>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>Recall@5</th>
    <th>NDCG@5</th>
    <th>FI</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>FI</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>35.1±0.2</td>
    <td>27.9±0.2</td>
    <td>55.3±0.2</td>
    <td>54.6±0.2</td>
    <td>47.7±0.1</td>
    <td>51.9±0.4</td>
    <td>19.4±0.0</td>
    <td>12.7±0.0</td>
    <td>33.2±0.1</td>
  </tr>
  <tr>
    <td>M</td>
    <td>38.6±0.2</td>
    <td>30.9±0.2</td>
    <td>57.8±0.1</td>
    <td>63.5±0.5</td>
    <td>57.3±0.4</td>
    <td>54.1±0.3</td>
    <td>21.5±0.1</td>
    <td>17.0±1</td>
    <td>37.3±0.3</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>22.3±0.3</td>
    <td>17.1±0.2</td>
    <td>44.3±0.2</td>
    <td>43.4±0.4</td>
    <td>35.6±0.3</td>
    <td>47.8±0.3</td>
    <td>17.3±0.1</td>
    <td>8.4±0.1</td>
    <td>27.7±0.3</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>37.6±0.1</td>
    <td>29.1±0.1</td>
    <td>55.7±0.1</td>
    <td>61.3±0.2</td>
    <td>53.1±0.1</td>
    <td>53.3±0.3</td>
    <td>19.2±0.0</td>
    <td>13.7±0.0</td>
    <td>35.8±0.7</td>
  </tr>
  <tr>
    <td>M</td>
    <td>40.3±0.1</td>
    <td>31.2±0.1</td>
    <td>57.7±0.0</td>
    <td>68.9±0.4</td>
    <td>61.4±0.3</td>
    <td>55.4±0.2</td>
    <td>20.3±0.1</td>
    <td>16.1±0.1</td>
    <td>37.1±0.2</td>
  </tr>
  <tr>
    <td>~M</td>
    <td>26.4±0.2</td>
    <td>20.4±0.2</td>
    <td>45.2±0.2</td>
    <td>52.2±0.3</td>
    <td>43.1±0.2</td>
    <td>49.7±0.4</td>
    <td>17.7±0.1</td>
    <td>10.6±0.1</td>
    <td>32.6±0.4</td>
  </tr>
</table>

(f) Mistral-8x7b-it

Table A2: Fact verification performance depending on whether the document generated by query expansion methods contains sentences entailed by gold evidence, with the number of retrieved evidence set to five (\( k = 5 \)). We report performance using different backbone LLMs for query expansion.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AvenTec</th>
  </tr>
  <tr>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>44.2±0.1</td>
    <td>32.0±0.1</td>
    <td>57.0±1.0</td>
    <td>64.0±0.0</td>
    <td>51.4±0.1</td>
    <td>53.2±0.6</td>
    <td>22.3±0.0</td>
    <td>15.9±0.0</td>
    <td>34.6±0.1</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>48.8±0.1</b></td>
    <td><b>35.7±0.1</b></td>
    <td><b>59.6±0.1</b></td>
    <td><b>71.7±0.3</b></td>
    <td><b>60.6±0.3</b></td>
    <td><b>54.1±0.6</b></td>
    <td><b>25.3±0.1</b></td>
    <td><b>21.4±0.1</b></td>
    <td><b>40.6±0.3</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>29.9±0.2</td>
    <td>20.5±0.1</td>
    <td>47.3±0.2</td>
    <td>55.4±0.1</td>
    <td>41.2±0.3</td>
    <td>50.1±0.4</td>
    <td>20.3±0.0</td>
    <td>12.2±0.0</td>
    <td>29.2±0.2</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>46.7±0.1</td>
    <td>32.1±0.0</td>
    <td>56.7±0.1</td>
    <td>70.0±1.1</td>
    <td>56.7±0.1</td>
    <td>54.6±0.3</td>
    <td>22.3±0.0</td>
    <td>17.0±0.0</td>
    <td>36.6±0.4</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>50.2±0.1</b></td>
    <td><b>34.5±0.0</b></td>
    <td><b>58.8±0.0</b></td>
    <td><b>76.5±0.3</b></td>
    <td><b>64.8±0.2</b></td>
    <td><b>57.7±0.2</b></td>
    <td><b>23.6±0.0</b></td>
    <td><b>19.5±0.0</b></td>
    <td><b>38.1±0.2</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>29.4±0.4</td>
    <td>20±0.3</td>
    <td>44.2±0.2</td>
    <td>60.5±0.4</td>
    <td>44.9±0.4</td>
    <td>49.5±0.3</td>
    <td>19.4±0.1</td>
    <td>11.8±0.1</td>
    <td>31±0.1</td>
  </tr>
</table>

(a) GPT-4o-mini

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AvenTec</th>
  </tr>
  <tr>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>43.4±0.1</td>
    <td>31±0.1</td>
    <td>56.9±0.2</td>
    <td>64.3±0.2</td>
    <td>51.3±0.1</td>
    <td>53±0.5</td>
    <td>22.8±0.0</td>
    <td>16±0.0</td>
    <td>34.7±0.1</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>47.3±0.1</b></td>
    <td><b>34.2±0.1</b></td>
    <td><b>59.2±0.1</b></td>
    <td><b>70.9±0.3</b></td>
    <td><b>59.7±0.3</b></td>
    <td><b>55.3±0.3</b></td>
    <td><b>25.7±0.1</b></td>
    <td><b>21.5±0.1</b></td>
    <td><b>40.4±0.4</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>30.5±0.2</td>
    <td>20.7±0.1</td>
    <td>47.3±0.2</td>
    <td>55.8±0.4</td>
    <td>40.9±0.4</td>
    <td>49.3±0.2</td>
    <td>20.4±0.1</td>
    <td>12.2±0.1</td>
    <td>29.3±0.2</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>46.2±0.0</td>
    <td>31.4±0.0</td>
    <td>56.7±0.1</td>
    <td>71.6±0.2</td>
    <td>58.3±0.1</td>
    <td>55±0.3</td>
    <td>22.8±0.0</td>
    <td>17.8±0.0</td>
    <td>37.6±0.3</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>50±0.1</b></td>
    <td><b>34.3±0.1</b></td>
    <td><b>58.5±0.1</b></td>
    <td><b>78.9±0.3</b></td>
    <td><b>67.1±0.4</b></td>
    <td><b>58.7±0.2</b></td>
    <td><b>24.2±0.1</b></td>
    <td><b>20.4±0.1</b></td>
    <td><b>38.9±0.2</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>32.8±0.3</td>
    <td>21.4±0.2</td>
    <td>46.6±0.2</td>
    <td>63±0.6</td>
    <td>48.1±0.6</td>
    <td>50.5±0.3</td>
    <td>20.5±0.1</td>
    <td>13.5±0.1</td>
    <td>33.6±0.3</td>
  </tr>
</table>

(b) Claude-3-haiku

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AvenTec</th>
  </tr>
  <tr>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>43.3±0.1</td>
    <td>31.7±0.0</td>
    <td>56.9±0.1</td>
    <td>64.7±0.2</td>
    <td>52.8±0.1</td>
    <td>53.1±0.6</td>
    <td>22.3±0.0</td>
    <td>16.2±0.0</td>
    <td>34.7±0.2</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>46.3±0.0</b></td>
    <td><b>34.1±0.0</b></td>
    <td><b>59.8±0.1</b></td>
    <td><b>71.4±0.3</b></td>
    <td><b>60.2±0.3</b></td>
    <td><b>55.3±0.3</b></td>
    <td><b>24.4±0.0</b></td>
    <td><b>20.4±0.1</b></td>
    <td><b>39.5±0.3</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>36.7±0.1</td>
    <td>26.1±0.0</td>
    <td>49.7±0.1</td>
    <td>57.9±0.4</td>
    <td>45.1±0.3</td>
    <td>49.9±0.3</td>
    <td>20.7±0.0</td>
    <td>12.8±0.0</td>
    <td>29.9±0.3</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>44.2±0.1</td>
    <td>29.9±0.1</td>
    <td>56.5±0.1</td>
    <td>69.8±0.1</td>
    <td>56.6±0.2</td>
    <td>54.8±0.3</td>
    <td>21.6±0.0</td>
    <td>16.3±0.0</td>
    <td>37.1±0.8</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>47.1±0.1</b></td>
    <td><b>31.8±0.1</b></td>
    <td><b>59.9±0.2</b></td>
    <td><b>75.5±0.5</b></td>
    <td><b>63.8±0.4</b></td>
    <td><b>58.3±0.3</b></td>
    <td><b>22.6±0.0</b></td>
    <td><b>18.6±0.1</b></td>
    <td><b>38.5±0.3</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>37.5±0.2</td>
    <td>25.6±0.1</td>
    <td>47.6±0.2</td>
    <td>67.7±0.3</td>
    <td>53.8±0.2</td>
    <td>53.4±0.2</td>
    <td>20.1±0.1</td>
    <td>12.9±0.1</td>
    <td>33.1±0.4</td>
  </tr>
</table>

(c) Gemini-1.5-flash

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AvenTec</th>
  </tr>
  <tr>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>43.6±0.1</td>
    <td>31.3±0.1</td>
    <td>56.8±0.1</td>
    <td>63.2±0.3</td>
    <td>51.2±0.1</td>
    <td>53±0.5</td>
    <td>22.3±0.0</td>
    <td>15.9±0.0</td>
    <td>34.5±0.1</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>47.2±0.1</b></td>
    <td><b>34.3±0.2</b></td>
    <td><b>59.5±0.0</b></td>
    <td><b>71.2±0.3</b></td>
    <td><b>60.6±0.4</b></td>
    <td><b>55.7±0.4</b></td>
    <td><b>25.4±0.1</b></td>
    <td><b>21.1±0.1</b></td>
    <td><b>38.5±0.1</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>35.8±0.3</td>
    <td>24.8±0.3</td>
    <td>50.4±0.2</td>
    <td>53.8±0.7</td>
    <td>40.2±0.5</td>
    <td>49.3±0.2</td>
    <td>20.4±0.0</td>
    <td>12.8±0.1</td>
    <td>30.7±0.2</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>46.1±0.2</td>
    <td>31.8±0.2</td>
    <td>56.5±0.2</td>
    <td>70.2±0.7</td>
    <td>57.8±0.2</td>
    <td>55±0.5</td>
    <td>22.3±0.2</td>
    <td>17.5±0.1</td>
    <td>36.7±0.9</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>54.4±0.4</b></td>
    <td><b>38.6±0.4</b></td>
    <td><b>59.3±0.0</b></td>
    <td><b>77.5±0.5</b></td>
    <td><b>65.6±0.5</b></td>
    <td><b>57.8±0.3</b></td>
    <td><b>24.7±0.2</b></td>
    <td><b>21.3±0.1</b></td>
    <td><b>39.6±0.3</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>37.9±0.2</td>
    <td>25.9±0.2</td>
    <td>50.7±0.2</td>
    <td>62.7±0.6</td>
    <td>48.2±0.6</td>
    <td>50.8±0.3</td>
    <td>20.7±0.1</td>
    <td>14.2±0.2</td>
    <td>32.9±0.4</td>
  </tr>
</table>

(d) Llama-3.1-8b-it

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AvenTec</th>
  </tr>
  <tr>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>46.1±0.2</td>
    <td>35.2±0.1</td>
    <td>57.2±0.2</td>
    <td>65.4±0.2</td>
    <td>52.5±0.1</td>
    <td>53.1±0.3</td>
    <td>22.7±0.0</td>
    <td>16.3±0.0</td>
    <td>34.8±0.1</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>49.4±0.2</b></td>
    <td><b>36.4±0.2</b></td>
    <td><b>59.5±0.2</b></td>
    <td><b>72.5±0.4</b></td>
    <td><b>60.6±0.2</b></td>
    <td><b>55.4±0.4</b></td>
    <td><b>25.4±0.1</b></td>
    <td><b>21.1±0.1</b></td>
    <td><b>39.8±0.3</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>28.7±0.2</td>
    <td>19.2±0.2</td>
    <td>46.7±0.2</td>
    <td>53.8±0.5</td>
    <td>40.8±0.2</td>
    <td>49.4±0.3</td>
    <td>20.4±0.0</td>
    <td>12.2±0.1</td>
    <td>29.3±0.2</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>49.7±0.2</td>
    <td>35±0.2</td>
    <td>57±0.1</td>
    <td>70.8±0.2</td>
    <td>57.8±0.2</td>
    <td>55±0.5</td>
    <td>22.3±0.2</td>
    <td>17.5±0.1</td>
    <td>36.7±0.9</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>50.9±0.1</b></td>
    <td><b>33.9±0.1</b></td>
    <td><b>58.9±0.1</b></td>
    <td><b>76.7±0.4</b></td>
    <td><b>64.7±0.4</b></td>
    <td><b>57.4±0.7</b></td>
    <td><b>23.9±0.1</b></td>
    <td><b>20.1±0.1</b></td>
    <td><b>38.5±0.4</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>34.3±0.2</td>
    <td>23.3±0.1</td>
    <td>48.4±0.2</td>
    <td>63±0.2</td>
    <td>48.5±0.2</td>
    <td>51.5±0.3</td>
    <td>21.1±0.1</td>
    <td>14.3±0.1</td>
    <td>34.4±0.3</td>
  </tr>
</table>

(e) Llama-3.1-70b-it

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data</th>
    <th colspan="3">FEVER</th>
    <th colspan="3">SciFact</th>
    <th colspan="3">AvenTec</th>
  </tr>
  <tr>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>Recall@10</th>
    <th>NDCG@10</th>
    <th>F1</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>43.4±0.2</td>
    <td>30.8±0.2</td>
    <td>56.8±0.2</td>
    <td>63.6±0.2</td>
    <td>51.3±0.1</td>
    <td>52.8±0.4</td>
    <td>22.6±0.0</td>
    <td>16.1±0.0</td>
    <td>34.9±0.1</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>47.3±0.2</b></td>
    <td><b>33.9±0.2</b></td>
    <td><b>59±0.1</b></td>
    <td><b>71.3±0.5</b></td>
    <td><b>60.6±0.4</b></td>
    <td><b>54.6±0.6</b></td>
    <td><b>25.1±0.1</b></td>
    <td><b>20.7±0.1</b></td>
    <td><b>39±0.2</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>28.7±0.3</td>
    <td>19.2±0.2</td>
    <td>47.2±0.1</td>
    <td>53.8±0.5</td>
    <td>39.5±0.3</td>
    <td>49.2±0.4</td>
    <td>20.2±0.1</td>
    <td>11.7±0.1</td>
    <td>29.5±0.2</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>47.6±0.0</td>
    <td>32.6±0.0</td>
    <td>56.9±0.2</td>
    <td>70.2±0.2</td>
    <td>56.8±0.1</td>
    <td>54.8±0.5</td>
    <td>22.8±0.0</td>
    <td>17.6±0.0</td>
    <td>37.2±0.5</td>
  </tr>
  <tr>
    <td><i>M</i></td>
    <td><b>50.9±0.1</b></td>
    <td><b>34.9±0.0</b></td>
    <td><b>58.8±0.0</b></td>
    <td><b>77.7±0.1</b></td>
    <td><b>65.2±0.2</b></td>
    <td><b>56.8±0.3</b></td>
    <td><b>24.2±0.0</b></td>
    <td><b>20.1±0.1</b></td>
    <td><b>38.7±0.3</b></td>
  </tr>
  <tr>
    <td>−<i>M</i></td>
    <td>33.5±0.3</td>
    <td>22.8±0.2</td>
    <td>47.2±0.1</td>
    <td>61.0±1.4</td>
    <td>46.6±0.3</td>
    <td>51.7±0.4</td>
    <td>21±0.1</td>
    <td>14.3±0.1</td>
    <td>33.8±0.4</td>
  </tr>
</table>

(f) Mistral-7b-it

Table A3: Fact verification performance depending on whether the document generated by query expansion methods contains sentences entailed by gold evidence, with the number of retrieved evidence set to ten (\( k = 10 \)). We report performance using different backbone LLMs for query expansion.
<table>
  <tr>
    <th>Method</th>
    <th>Data</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>20.5</td>
    <td>13.8</td>
    <td>34.7</td>
  </tr>
  <tr>
    <td>\( M \)</td>
    <td><b>21.6</b></td>
    <td><b>15.4</b></td>
    <td><b>40.9</b></td>
  </tr>
  <tr>
    <td>\( \neg M \)</td>
    <td>19.4</td>
    <td>12.2</td>
    <td>27.3</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>19.9</td>
    <td>14.7</td>
    <td>38.7</td>
  </tr>
  <tr>
    <td>\( M \)</td>
    <td><b>20.7</b></td>
    <td><b>16.1</b></td>
    <td><b>40.9</b></td>
  </tr>
  <tr>
    <td>\( \neg M \)</td>
    <td>18.8</td>
    <td>12.5</td>
    <td>33.6</td>
  </tr>
</table>

(a) \( k = 5 \)

<table>
  <tr>
    <th>Method</th>
    <th>Data</th>
    <th>METEOR</th>
    <th>BERTScore</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Query2doc</td>
    <td>ALL</td>
    <td>23.6</td>
    <td>17.3</td>
    <td>36.7</td>
  </tr>
  <tr>
    <td>\( M \)</td>
    <td><b>24.9</b></td>
    <td><b>19</b></td>
    <td><b>45.2</b></td>
  </tr>
  <tr>
    <td>\( \neg M \)</td>
    <td>22.2</td>
    <td>15.5</td>
    <td>26.6</td>
  </tr>
  <tr>
    <td rowspan="3">HyDE</td>
    <td>ALL</td>
    <td>23.9</td>
    <td>18.2</td>
    <td>38.3</td>
  </tr>
  <tr>
    <td>\( M \)</td>
    <td><b>24.6</b></td>
    <td><b>19.7</b></td>
    <td><b>39.4</b></td>
  </tr>
  <tr>
    <td>\( \neg M \)</td>
    <td>22.7</td>
    <td>16.1</td>
    <td>34.9</td>
  </tr>
</table>

(b) \( k = 10 \)

Table A4: Fact verification performance on 500 samples from the AVeriTeC development set using manually annotated NLI labels. For query expansion, we used Llama-3.1-70b-it for Query2doc and Claude-3-haiku for HyDE, as each model achieved the best performance for its respective expansion method. GPT-4o-mini was used for verdict prediction.
<table>
  <tr>
    <th>Claim</th>
    <td>U.S. President Donald Trump and his wife, First Lady Melania Trump, have tested positive for COVID-19</td>
  </tr>
  <tr>
    <th>Gold Evidence</th>
    <td>
      <ul>
        <li>Donald trump tweeted that he had tested positive for Covid 19 on 1st October 2020</li>
        <li><b>Donald trump tweeted that he and first lady Melania had tested positive for Covid 19 on 1st October 2020</b></li>
        <li><b>The President’s physician, Navy Cmdr. Dr. Sean Conley, wrote that he received confirmation of the positive tests on Thursday evening.</b></li>
      </ul>
    </td>
  </tr>
  <tr>
    <th>Generation</th>
    <td>
      On October 1, 2020, it was announced that U.S. President Donald Trump and his wife, First Lady Melania Trump, had tested positive for COVID-19. This news came after one of the President’s closest advisors, Hope Hicks, had also tested positive for the virus earlier that day. The President and the First Lady began experiencing symptoms shortly after Hicks’ diagnosis and were subsequently tested. Both Trumps received their positive test results in the early hours of October 2, 2020.<br>
      Following their diagnosis, the President and the First Lady immediately began self-isolating in the White House. President Trump’s physician, Dr. Sean Conley, issued a statement saying that the President and the First Lady would remain in quarantine for the next 14 days, as per Centers for Disease Control and Prevention (CDC) guidelines.<br>
      The news of the President’s positive test result sparked widespread concern and reaction, with leaders from around the world offering their well wishes for a speedy recovery. The diagnosis also led to increased scrutiny of the White House’s handling of the pandemic, particularly with regards to the safety protocols in place at the time. Fortunately, both President Trump and the First Lady made a full recovery from the virus. President Trump was hospitalized for a few days at Walter Reed National Military Medical Center, where he received treatment and was later discharged on October 5, 2020.
    </td>
  </tr>
</table>

(a) Matched example

<table>
  <tr>
    <th>Claim</th>
    <td>US Supreme Court Justice, Ruth Bader Ginsburg, fought cancer for decades before she passed away in September 2020.</td>
  </tr>
  <tr>
    <th>Gold Evidence</th>
    <td>
      <ul>
        <li>Ruth Bader Ginsburg was a U.S. Supreme Court justice, the second woman to be appointed to the position.</li>
        <li><b>On September 18, 2020, Justice Ruth Bader Ginsburg, the second woman to serve on the Supreme Court of the United States, passed away at the age of eighty-seven</b></li>
        <li><b>She died of complications from metastatic pancreatic cancer.</b></li>
        <li><b>colon cancer (first diagnosed in 1999), pancreatic cancer (2009)</b></li>
      </ul>
    </td>
  </tr>
  <tr>
    <th>Generation</th>
    <td>
      Ruth Bader Ginsburg, a trailblazing U.S. Supreme Court Justice, demonstrated unwavering resilience in the face of adversity, including her prolonged battle with cancer. Throughout her life, she faced numerous health challenges, including colorectal cancer in 1999, <b>pancreatic cancer in 2009</b>, and lung cancer in 2018. Furthermore, in 2019, she underwent radiation therapy for a tumor on her pancreas. Despite these setbacks, Ginsburg continued to serve on the Supreme Court, inspiring countless individuals with her strength, determination, and unwavering commitment to her work. Tragically, her courageous fight against cancer came to an end <b>on September 18, 2020</b>, when she passed away at the age of 87, leaving behind an enduring legacy as a champion of gender equality, social justice, and the rule of law.
    </td>
  </tr>
</table>

(b) Matched example
<table>
  <tr>
    <th>Claim</th>
    <td>While serving as Town Supervisor on Grand Island, Nebraska, US Nate McMurray voted to raise taxes on homeowners.</td>
  </tr>
  <tr>
    <th rowspan="2">Gold Evidence</th>
    <td>• Grand Island raised its tax levy on November 8th 2018.<br>
      • A Public Hearing was held on Thursday, November 8, 2018 at 7:00p.m. for the purpose of hearing anyone who wants to comment on the proposed 2019 Town of Grand Island budget. Supervisor Nate McMurray closed the Public Hearing.</td>
  </tr>
  <tr>
    <th>Generation</th>
    <td>After researching Nate McMurray’s political career, there is no evidence that he has ever served as Town Supervisor on Grand Island, Nebraska. Grand Island is a city in central Nebraska, not a town, and it does not have a position of Town Supervisor. Nate McMurray is a former town councilman in the town of Grand Island, New York, which is located in Erie County, New York, not Nebraska. However, there are no reports or records indicating that McMurray voted to raise taxes on homeowners while serving on the Grand Island, New York town council.<br>
      In conclusion, the claim that Nate McMurray voted to raise taxes on homeowners while serving as Town Supervisor on Grand Island, Nebraska is false. Nate McMurray has never held a political position in the state of Nebraska, and the town of Grand Island, Nebraska does not exist. This appears to be a fabricated claim with no basis in fact.</td>
  </tr>
</table>

(c) Unmatched example
<table>
  <tr>
    <th>Claim</th>
    <td>U.S. mortalities this year are not materially different from the previous 5 years</td>
  </tr>
  <tr>
    <th rowspan="2">Gold Evidence</th>
    <td>• The cumulative number of deaths in 2020 is far greater than the cumulative number of deaths of any year from 2015 to 2019, as shown in the figures in link below.<br>
    • The Historical data on death rate shows that there were more deaths recorded in the singular year 2020 compared to the death rate rate in each year from 2015-2019</td>
  </tr>
  <tr>
    <td>**U.S. Mortality Rates Remain Steady, Showing No Significant Increase from Previous Years**<br>
    According to recent data, mortality rates in the United States for this year are not showing any significant deviations from the past five years. Despite concerns about various health crises and trends, the overall number of deaths in the country has remained relatively stable.<br>
    When analyzing the data, it becomes clear that the mortality rates for this year are largely in line with the averages seen in the previous five years. This suggests that, despite fluctuations in specific causes of death or regional variations, the overall trend in mortality rates has not undergone a significant shift.<br>
    Several factors could be contributing to this stability, including advances in medical technology, improvements in healthcare access and quality, and public health initiatives aimed at reducing mortality rates. Additionally, the country’s demographics, such as population size and age distribution, have not changed dramatically enough to impact mortality rates substantially.<br>
    While specific causes of death, such as deaths from accidents, diseases, or other factors, may have seen some variation, the overall mortality rate has not been significantly affected. This steady trend is a positive sign, indicating that the country’s healthcare system and public health efforts are effectively managing and mitigating mortality risks.<br>
    It is essential to note, however, that mortality rates can fluctuate over time due to various factors, including emerging health crises, changes in population demographics, or shifts in healthcare policies. Therefore, continuous monitoring and analysis of mortality data are crucial to identifying potential trends and areas for improvement.<br>
    In conclusion, the current mortality rates in the United States are not materially different from the previous five years, indicating a stable trend. This stability is a testament to the country’s healthcare system and public health efforts, but it also highlights the need for continued vigilance and monitoring to address potential future challenges.</td>
  </tr>
</table>

(d) Unmatched example

Table A5: Examples of generated documents and gold evidence for the target claims. Colored highlights indicate information in the generated documents that overlaps with the gold evidence.

Reviewer paper summaries: This paper investigates the effectiveness of query expansion methods based on LLMs in zero-shot retrieval tasks and raises a question: whether the observed performance improvements are partly attributable to knowledge leakage?
This paper introduces the fact verification task as an experimental test platform. The study finds that query expansion methods consistently improve performance only when the generated documents include sentences entailed by the gold evidence. This suggests that knowledge leakage may exist in these benchmarks, potentially inflating the perceived effectiveness of LLM-based query expansion methods, particularly in real-world scenarios requiring the retrieval of niche or novel knowledge.
Summary of strengths: 1. Clear writing, the content is easy to follow.
2. Very interesting research topic. I also think there should be some leakages in LLM-based query expansion. This work tries to explore this topic more.
3. The authors did experiments on both sparse and dense query expansion methods, i.e., query2doc and HyDe.
Summary of weaknesses: 1. As acknowledged in the "Limitations" section of the paper, the study only identifies a pattern of association between the generated documents and the gold evidence but fails to demonstrate whether these patterns are directly caused by knowledge leakage from the model's training data. 
2. Additionally, I believe that leakage certainly exists; it is merely a matter of degree. Therefore, as the authors suggest, whether the effectiveness of LLM-based query expansion methods is overestimated in real-world scenarios requiring the retrieval of niche or novel knowledge should be supported by more experimental data. For instance, constructing a new dataset (e.g., one that the LLM has not encountered during its pre-training phase) could be used to test whether these LLM-based query expansion methods remain effective and to compare their effectiveness against traditional QE methods, determining whether their effectiveness is enhanced or diminished.
3. Why the datasets are picked? More discussions on the dataset will be better.
Comments, suggestions and typos: The research question of this work is quite interesting, but the conclusions drawn so far are relatively superficial. It would be more compelling if the study could demonstrate, using a newly constructed dataset, that the current LLM-based query expansion methods are indeed overestimated. Additionally, further explanation is needed regarding the selection of datasets.
Overall assessment scores: 2.5
Metareview score: 2.5
Metareview: This paper presents a novel and timely analysis questioning the true source of performance gains in LLM-based query expansion, highlighting the potential role of knowledge leakage. It offers comprehensive empirical evidence across multiple datasets and models, using fact verification and NLI to support its findings. However, reviewers point out the following weakness:

1. Limitations: 
- Lack of Causal Proof: All reviewers note that the paper only shows correlation, not causation, between knowledge leakage and performance. Further experiments with controlled variables can be carried out. Current conclusions are preliminary.

- Concerns about NLI: The detection of leakage is based on GPT-4o-mini. The accuracy may affect the validity of conclusions; although mitigated through manual verification, some concerns remain.

2. Experimental Shortcomings: including appendix overload issue pointed out by reviewer TXM4, reviewer Dr4a requests more discussion on dataset choice and its implications.

An interesting experimental direction would be to investigate the impact on performance when partial phrases from the matched sentences are inserted into the generated documents, rather than removing the matched sentences entirely. This can potentially test whether the performance increase is due to augmentation on lexical level or knowledge leakage. Furthermore, I would suggest reporting performances of using**the negation of the matched sentence** for retrieval. This experiment would be important to evaluate authors' hypothesis.

And here is a relevant paper the AC recommends authors to cite: Mallen A, Asai A, Zhong V, et al. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories[J]. arXiv preprint arXiv:2212.10511, 2022.
Summary of reasons to publish: The topic is novel