Enhancing Retrieval-Augmented Generation via Evidence Tree Search
Authors: Hao Sun, Hengyi Cai, Yuchen Li, Xuanbo Fan, Xiaochi Wei, Shuaiqiang Wang, Yan Zhang, Dawei Yin
Accepted at: acl25

Enhancing Retrieval-Augmented Generation via Evidence Tree Search

Hao Sun1,2*, Hengyi Cai3†, Yuchen Li3, Xuanbo Fan1,2
Xiaochi Wei3, Shuaiqiang Wang3, Yan Zhang1,2*, Dawei Yin3
1State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China
2School of Intelligence Science and Technology, Peking University, 3Baidu Inc
sunhao@stu.pku.edu.cn

Abstract

Retrieval-Augmented Generation (RAG) is widely used to enhance Large Language Models (LLMs) by grounding responses in external knowledge. However, in real-world applications, retrievers often return lengthy documents with redundant or irrelevant content, confusing downstream readers. While evidence retrieval aims to address this by extracting key information, it faces critical challenges: (1) inability to model synergistic inter-dependencies among evidence sentences, (2) lack of supervision for evaluating multi-sentence evidence quality, and (3) computational inefficiency in navigating exponentially growing search spaces of candidate evidence sets. To tackle these challenges, we propose ETS (Evidence Tree Search), a novel framework that reformulates evidence retrieval as a dynamic tree expansion process. Our approach first constructs an evidence tree where each path represents a candidate evidence set, explicitly modeling inter-sentence dependencies through context-aware node selection. We then leverage Monte Carlo Tree Search (MCTS) to efficiently assess evidence quality and introduce an Early-Terminating Beam Search strategy to efficiently accelerate the model inference. Extensive experiments on five datasets demonstrate that ETS significantly outperforms existing methods across different readers. Our code and datasets will be released to facilitate future research.

1 Introduction

Large Language Models (LLMs) (Taylor et al., 2022; Chowdhery et al., 2022; Zhao et al., 2023a) have demonstrated remarkable performance across a wide range of downstream tasks (Xia et al., 2024; Yamauchi et al., 2023; Imani et al., 2023; Lewkowycz et al., 2022). Despite these advancements, LLMs are still prone to generating responses that include hallucinated facts or inaccurate information (Ji et al., 2023; Shuster et al., 2021; Zhang et al., 2023a), which undermines their reliability. To mitigate this issue, Retrieval-Augmented Generation (RAG) has been introduced, which integrates external knowledge into the generation process (Ram et al., 2023; Shi et al., 2023; Rashkin et al., 2021; Gao et al., 2022; Bohnet et al., 2022; Menick et al., 2022).

In typical RAG systems, a user’s query is utilized to retrieve relevant documents from external sources, which are then fed into the reader LLM to produce more accurate responses. However, in real-world applications, imperfect retrievers often return long documents, such as web pages, that may contain substantial irrelevant content. Only specific portions of these documents are relevant for answering the query. Directly using these lengthy documents in RAG systems poses several challenges: readers may struggle with processing long texts, and irrelevant content could distract the reader from

![Illustration of Evidence Retrieval for RAG. Given an input query, a long article is retrieved using a search engine and segmented into sentences. The evidence finder then performs a reward-guided tree search to identify the optimal set of evidence, which is subsequently passed to the reader LLM for answer generation.](page_784_654_627_312.png)

Figure 1: Illustration of Evidence Retrieval for RAG. Given an input query, a long article is retrieved using a search engine and segmented into sentences. The evidence finder then performs a reward-guided tree search to identify the optimal set of evidence, which is subsequently passed to the reader LLM for answer generation.
generating the correct response. Therefore, evidence retrieval plays a critical role in addressing these issues by identifying and extracting the most relevant evidence sentences from the retrieved documents before passing it to the reader.

However, evidence retrieval is a non-trivial task due to several inherent challenges. **First**, relevant evidence often comprises multiple interrelated pieces of information. Traditional retrieval methods focus on modeling the relationship between the query and each individual sentence, inherently failing to capture the complex inter-dependencies among evidence sentences. **Second**, assessing the quality of evidence sets presents significant difficulties due to the lack of explicit supervision signals. Most RAG corpora only provide binary document-level relevance labels, offering no direct feedback on which specific sentence combinations optimally support answer generation. **Finally**, the combinatorial explosion of candidate evidence sets in lengthy documents creates an enormous search space, making it computationally intensive to efficiently identify the optimal set of evidence.

To address these challenges, as depicted in Figure 1, we propose ETS, a novel evidence retrieval framework with Evidence Tree Search. **To capture the complex inter-dependencies among evidence sentences**, ETS models evidence retrieval as a dynamic tree expansion process, where each node represents a sentence and every root-to-leaf path constitutes a candidate evidence set. This hierarchical structure inherently captures inter-sentence dependencies — during tree growth, new sentences are selected based on accumulated contextual evidence along the current path, enabling holistic evaluation of sentence combinations’ collective relevance to the query. **In order to assess the quality of evidence sets**, we introduce Monte Carlo Tree Search (MCTS) (Silver et al., 2017) to guide the exploration of potential evidence combinations. Through iterative MCTS simulations, rewards derived from reader correctness given the candidate evidence set are back-propagated to update node values, providing dense training signals for optimizing evidence selection. **To combat the exponential search space**, we propose pruning the search space using a value model trained on reward signals derived from the evidence tree constructed through MCTS annotation. By leveraging this model, the framework efficiently narrows down the search space, retaining only the most promising evidence combinations. Additionally, to further reduce inference latency, we introduce Early Terminating during the expansion phase of beam search. This technique terminates the generation of candidate sentences early when a sentence prefix uniquely identifies its position in the original context, significantly improving computational efficiency without compromising accuracy.

Our contributions are summarized as follows:

• We propose ETS, a tree-based framework that enhances LLMs’ evidence-grounding through a tree expansion process.

• We introduce MCTS to assess evidence quality and Early Terminating Beam Search to reduce search space and improve selection efficiency.

• Extensive experiments on five datasets show our method outperforms existing approaches across different readers.

2 Background

In Retrieval-Augmented Generation (RAG), a retriever first retrieves relevant text evidence \( \mathcal{D} \) from a corpus \( \mathcal{C} \) given a query \( q \). The retrieved evidence is then used by a reader LLM to generate an answer:

\[
\mathcal{D} = \text{Retriever}(q, \mathcal{C}), \quad \mathcal{A} = \text{Reader}(q, \mathcal{D}),
\]

The retriever can be implemented using open-source models (e.g., DPR (Karpukhin et al., 2020)) or commercial search engines (e.g., Google), while the reader is typically a LLM.

In practice, retrievers often return lengthy documents (e.g., web pages or articles) instead of concise evidence, making it difficult for the reader to generate accurate answers. To address this, evidence retrieval aims to identify and extract a subset of key text spans, referred to as supporting evidence \( \mathcal{T} = \{t_1, \ldots, t_k\} \subset \mathcal{D} \), that are most relevant to answering the query \( q \). This selection process can be formalized as a mapping function \( f(\cdot) \):

\[
\mathcal{T} = \{t_1, \ldots, t_k\} = f(\mathcal{D}).
\]

In this work, we define \( f(\cdot) \) as a tree expansion process, where each node represents a sentence, and the path from the root node to a leaf node represents the selected evidence.
Figure 2: Single iteration of MCTS Annotation. The iteration is repeated until the maximum number of iterations is reached or no further nodes in the tree can be expanded.

3 Approach

3.1 Overview
Figure 3 presents an overview of our framework. The process begins with the application of the Monte Carlo Tree Search (MCTS) algorithm to construct evidence trees for the queries in the training dataset. From these trees, we extract both correct and incorrect paths, which are subsequently utilized to train the policy model and the value model, respectively. During the inference phase, we introduce an early terminating beam search mechanism. This approach efficiently minimizes the search space and terminates the generation process once the prefix uniquely identifies the evidence, thereby optimizing computational efficiency and accuracy. Next, we will first illustrate the process of the MCTS annotation.

3.2 MCTS Annotation
Given an input query, a long article is retrieved using a search engine and segmented into individual sentences, which serve as the initial candidate set. However, due to the excessive length of the article, the candidate set can become prohibitively large, with many sentences being irrelevant to the query. This not only increases the computational complexity of MCTS but also introduces noise into the labeling process. To address this issue, we introduce a filtering mechanism that leverages the BGE model (Xiao et al., 2023) to compute the similarity between the query and each candidate sentence. By retaining only the most semantically similar sentences, we significantly reduce the candidate set, thereby improving the overall efficiency and accuracy of the annotation process.

During the search process, MCTS runs for multiple simulations. This process terminates when the maximum iteration number is reached or no paths can be expanded. For the \( i \)-th simulation, it conducts four operations to expand the tree:

Selection  During the \( i \)-th simulation of the MCTS, the process begins with \( s_0 \), which represents the input query. The algorithm then proceeds to explore the tree by selecting nodes according to the Upper Confidence Bound for Trees (UCT) criterion (Rosin, 2011). This selection process is mathematically represented as:

\[
UCT(s_t) = V(s_t) + w \sqrt{\frac{\ln N_{parent}(s_t)}{N(s_t)}}
\]

where \( w \) controls the balance between exploration and exploitation. The node value \( V(s_t) \) and its visiting count \( N(s_t) \) will be updated as the search progresses. \( N_{parent}(s_t) \) represents the visiting count of the parent node of \( s_t \).

Expansion  After selecting the node to be expanded, the next step is to choose the most suitable sentence from the candidate set. A straightforward way to evaluate the quality of the evidence set is to measure the likelihood of the reader generating the correct answer when the evidence set is used as the model input. To accelerate the labeling process, we employ parallel expansion, where we concurrently compute the likelihood of outputting the correct answer after incorporating each candidate sentence into the evidence set. This is expressed as:

\[
R(s_t) = P(y|q, d_t)
\]

where \( y \) represents the correct answer, and \( d_t \) denotes the evidence set after adding the candidate sentence \( s_t \). From these calculations, we select the top \( M \) sentences that are most likely to lead to the final correct answer as the expansion nodes.

Simulation  The simulation phase aims to determine whether the current path can lead the reader to generate the final correct answer. To achieve
Figure 3: Overview of the proposed method: Given an input query, a long article is retrieved using a search engine and segmented into individual sentences, which serve as the initial candidate set. Then the MCTS algorithm constructs evidence trees for the input query from which correct and incorrect paths are extracted to train the policy and value models. During inference, the early terminating beam search minimizes the search space and terminates generation once the generated prefix uniquely identifies a sentence in the candidate set.

this, for each node, we construct the evidence set by concatenating the sentences from the root node to the current node. The reader then answers the question based on this evidence set. If the reader provides the correct answer, indicating that sufficient evidence has been identified, the reward is set to 1. Otherwise, the reward is defined as the likelihood of the reader outputting the correct answer based on the evidence set.

Backpropagation At the end of the \( i \)-th simulation, each edge along the path from the leaf node \( s_t \) to the root undergoes a backward pass update. The updates to their values and visiting counts are executed according to the following rule:

\[
N(s_t) \leftarrow N(s_t) + 1 \\
V(s_t) \leftarrow V(s_t) + \frac{1}{N(s_t)} (R(s_t) - V(s_t))
\]

where \( N(s_t) \) represents the visiting count of node \( s_t \), and \( V(s_t) \) denotes its value.

3.3 Model Training

In our framework, the policy model \( \pi_\theta \) is initialized with a pre-trained LLM. We extend this model to derive the value model \( V_\phi \) by adding an auxiliary linear layer with a Sigmoid activation function. This layer operates alongside the traditional softmax layer responsible for token prediction, as illustrated in the rightmost panel of Figure 3. This design ensures that the policy model and the value model share the majority of their parameters, promoting parameter efficiency and joint optimization.

To construct the training signals for the policy model and the value model, we sample solution paths from the tree constructed through multiple rounds of MCTS. These paths are denoted as \( x^+ \) (correct solutions) and \( x^- \) (incorrect solutions). We then apply a multi-task loss function to jointly update both models:

\[
\mathcal{L} = -\log \pi_\theta(x^+|q) + \beta \cdot \sum_{t=1}^{T(x)} \|V_\phi(s_t) - V(s_t)\|^2
\]

Here, the first term represents the negative log-likelihood loss for next-token prediction in correct solutions, which guides the policy model to generate accurate predictions. The second term captures the loss in value prediction for both correct and incorrect solutions, ensuring the value model provides reliable estimates of expected rewards at each node. \( T(x) \) denotes the number of steps in the solution path \( x \), and \( \beta \) is a tunable hyperparameter that controls the weight of the value loss term.

3.4 Model Inference

After obtaining the trained policy model, it can be directly used to generate grounding evidence. However, this iterative generation process has two limitations: it fails to fully explore the evidence space, and the generation can be slow due to the large input length. To address these issues, we propose an early terminating beam search mechanism, which efficiently explores the evidence space while bypassing unnecessary decoding steps.

Early Terminating Beam Search Specifically, given the input query and the long article, the policy model is used to generate multiple candidate sentences. Since the generation target originates directly from the source text, once the generation prefix is determined, the generated prefix can be used to locate the corresponding text in the source article. This allows us to skip decoding intermediate tokens, significantly speeding up the process. The value model is then employed to evaluate the
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">2WikiMulti</th>
    <th colspan="2">HotpotQA</th>
    <th colspan="2">MusiQue</th>
    <th colspan="2">MultiField</th>
    <th colspan="2">Qasper</th>
    <th colspan="2">AVG</th>
  </tr>
  <tr>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
  </tr>
  <tr>
    <td colspan="13"><b>Qwen2.5-14B-Instruct-1M</b></td>
  </tr>
  <tr>
    <td>BM25</td><td>32.00</td><td>30.08</td><td>39.50</td><td>38.32</td><td>13.50</td><td>12.09</td><td>25.33</td><td>39.90</td><td>16.00</td><td>16.30</td><td>25.27</td><td>27.34</td>
  </tr>
  <tr>
    <td>BGE-base-en</td><td>37.50</td><td>37.01</td><td>40.50</td><td>38.45</td><td>19.50</td><td>19.98</td><td>27.33</td><td>44.07</td><td>21.50</td><td>23.74</td><td>29.27</td><td>32.65</td>
  </tr>
  <tr>
    <td>LLM-Embedder</td><td>36.00</td><td>36.91</td><td>39.50</td><td>36.56</td><td>20.50</td><td>20.21</td><td>26.67</td><td>45.92</td><td>25.50</td><td>26.97</td><td>29.63</td><td>33.31</td>
  </tr>
  <tr>
    <td>ChatGLM3-6B</td><td>32.50</td><td>30.90</td><td>30.00</td><td>31.59</td><td>14.00</td><td>15.03</td><td>16.00</td><td>33.61</td><td>21.50</td><td>23.14</td><td>22.80</td><td>26.85</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td><td>40.50</td><td>37.20</td><td>46.00</td><td>42.59</td><td>18.50</td><td>18.67</td><td>24.00</td><td>41.50</td><td>22.50</td><td>23.74</td><td>30.30</td><td>32.74</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td><td>37.50</td><td>37.25</td><td>43.50</td><td>44.28</td><td>25.00</td><td>24.85</td><td>28.67</td><td>42.05</td><td>19.00</td><td>18.94</td><td>30.73</td><td>33.47</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td><td>35.00</td><td>32.11</td><td>42.00</td><td>40.42</td><td>22.00</td><td>22.06</td><td>22.67</td><td>41.07</td><td>25.00</td><td>27.06</td><td>29.33</td><td>32.54</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td><td>42.50</td><td>36.39</td><td>43.00</td><td>41.81</td><td>18.50</td><td>19.01</td><td>19.33</td><td>32.66</td><td>15.00</td><td>15.38</td><td>27.67</td><td>29.05</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td><td>41.00</td><td>38.09</td><td>43.00</td><td>43.68</td><td>19.50</td><td>21.96</td><td>24.67</td><td>37.07</td><td>13.50</td><td>14.60</td><td>28.33</td><td>31.08</td>
  </tr>
  <tr>
    <td>CFIC-7B</td><td>45.00</td><td>42.52</td><td>45.50</td><td>39.06</td><td>27.50</td><td>26.06</td><td>28.67</td><td>45.94</td><td>26.00</td><td>27.92</td><td>34.53</td><td>36.30</td>
  </tr>
  <tr>
    <td>ETS-7B</td><td><b>58.00</b></td><td><b>56.23</b></td><td><b>51.00</b></td><td><b>51.26</b></td><td><b>39.50</b></td><td><b>39.85</b></td><td><b>29.33</b></td><td><b>48.38</b></td><td><b>27.00</b></td><td><b>29.02</b></td><td><b>40.97</b></td><td><b>44.95</b></td>
  </tr>
  <tr>
    <td colspan="13"><b>Qwen2.5-72B-Instruct</b></td>
  </tr>
  <tr>
    <td>BM25</td><td>31.00</td><td>28.59</td><td>42.00</td><td>40.45</td><td>12.00</td><td>14.62</td><td>23.33</td><td>39.96</td><td>16.50</td><td>20.52</td><td>24.97</td><td>28.83</td>
  </tr>
  <tr>
    <td>BGE-base-en</td><td>41.00</td><td>40.83</td><td>43.50</td><td>42.98</td><td>19.50</td><td>19.79</td><td>28.67</td><td>46.02</td><td>24.50</td><td>27.23</td><td>31.43</td><td>35.37</td>
  </tr>
  <tr>
    <td>LLM-Embedder</td><td>40.00</td><td>40.36</td><td>41.50</td><td>41.56</td><td>21.00</td><td>21.65</td><td>27.33</td><td>45.11</td><td>27.50</td><td>30.10</td><td>31.47</td><td>35.75</td>
  </tr>
  <tr>
    <td>ChatGLM3-6B</td><td>41.50</td><td>33.19</td><td>39.50</td><td>35.39</td><td>15.00</td><td>15.32</td><td>13.33</td><td>32.13</td><td>18.50</td><td>22.08</td><td>25.57</td><td>27.62</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td><td>44.00</td><td>43.25</td><td>46.50</td><td>44.50</td><td>20.50</td><td>22.09</td><td>24.00</td><td>41.14</td><td>20.50</td><td>27.26</td><td>31.10</td><td>35.65</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td><td>43.50</td><td>38.55</td><td>48.00</td><td>44.10</td><td>25.50</td><td>27.13</td><td>28.67</td><td>45.95</td><td>21.00</td><td>24.93</td><td>33.33</td><td>36.13</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td><td>43.00</td><td>41.21</td><td>43.00</td><td>42.42</td><td>23.00</td><td>25.42</td><td>21.33</td><td>40.19</td><td>22.00</td><td>27.10</td><td>30.47</td><td>35.27</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td><td>52.00</td><td>41.37</td><td>47.00</td><td>44.23</td><td>21.50</td><td>22.90</td><td>18.00</td><td>35.68</td><td>17.50</td><td>23.31</td><td>31.20</td><td>33.50</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td><td>53.00</td><td>38.79</td><td>47.50</td><td>42.77</td><td>23.50</td><td>25.50</td><td>22.67</td><td>39.90</td><td>15.50</td><td>21.58</td><td>32.43</td><td>33.71</td>
  </tr>
  <tr>
    <td>CFIC-7B</td><td>54.00</td><td>43.67</td><td>53.00</td><td>48.46</td><td>30.50</td><td>31.26</td><td>29.33</td><td>46.83</td><td>29.00</td><td>30.81</td><td>39.17</td><td>40.21</td>
  </tr>
  <tr>
    <td>ETS-7B</td><td><b>60.50</b></td><td><b>54.82</b></td><td><b>60.50</b></td><td><b>57.72</b></td><td><b>40.00</b></td><td><b>40.84</b></td><td><b>30.00</b></td><td><b>47.87</b></td><td><b>31.00</b></td><td><b>34.91</b></td><td><b>44.40</b></td><td><b>47.23</b></td>
  </tr>
</table>

Table 1: Main experiment results on five QA datasets across two reader LLMs. The best results are in bold. The AVG column shows the average EM and F1 scores across all datasets.

quality of the expanded sentences, retaining only the most valuable nodes. This process is iterated until the maximum depth is reached or no further paths can be expanded. Finally, the answer with the highest value is selected as the output.

4 Experiments

4.1 Datasets and Metrics

We conduct experiments on five datasets from LongBench (Bai et al., 2023), including 2Wiki-MultihopQA (Ho et al., 2020), HotpotQA (Yang et al., 2018), MuSiQue (Trivedi et al., 2022), MultiFieldQA (Bai et al., 2023), and Qasper (Dasigi et al., 2021). Following the LongBench benchmark, we use the EM score and F1-score as evaluation metrics. For further details on LongBench, please refer to Bai et al. (2023). The statistical information of all datasets is provided in Table 4.

4.2 Baselines

In this study, we benchmark our method against the following three categories of baselines:

Retrieval-Based Methods Retrieval-based methods typically segment lengthy documents into smaller passages and use retrieval techniques to prioritize relevant passages. For a fair comparison, we use sentences as the retrieval unit and employ BM25 (Yang et al., 2017), BGE-base-en-v1.5 (Xiao et al., 2023), and LLM-Embedder (Zhang et al., 2023b) as retrieval models. The highest-ranking sentences are selected as input context for the reader to support QA tasks.

Extraction-Based Methods For extraction-based methods, we evaluate state-of-the-art long-context models, including ChatGLM3-6B-128K (GLM et al., 2024), Qwen2.5-7B-Instruct-1M (Team, 2025), GLM-4-9B-Chat-1M (GLM et al., 2024), Qwen2.5-14B-Instruct-1M (Team, 2025), Qwen2.5-32B-Instruct (Team, 2024), and Qwen2.5-72B-Instruct (Team, 2024). These models refine lengthy documents into concise text evidence, which serves as context for the reader LLM to support QA tasks.
Learning-Based Methods  Unlike the above methods, which use off-the-shelf models, learning-based methods fine-tune models specifically for the task of evidence retrieval to improve performance. We implement the current state-of-the-art model, CIFC (Qian et al., 2024), following its open-sourced code and data.

To ensure fairness, all models provide a comparable volume of textual evidence for the reader.

4.3 Implementation Details

During the MCTS annotation process, the Qwen2.5-7B-Instruct-1M model is utilized as the reader. The maximum number of iterations is configured to 20. Following Chen et al. (2024), the parameter \( w, \beta \) is set to 1.4 and 0.1, respectively. For the training dataset, we sample 6,000 correct paths and 6,000 incorrect paths. The policy model is initialized using the Qwen2.5-7B-Instruct-1M. Training involves fine-tuning the model for two epochs with a batch size of 1 and a learning rate of \( 1 \times 10^{-6} \), utilizing 8 NVIDIA A100 80GB GPUs. To evaluate the generalizability of our approach, we employ two additional reader models: Qwen2.5-14B-Instruct-1M and Qwen2.5-72B-Instruct. We intend to open-source the code to facilitate the reproducibility of our methodology. For more implementation details, please refer to Appendix B.

4.4 Main Results

In this section, we present the experimental results on five QA datasets using Qwen2.5-14B-Instruct-1M and Qwen2.5-72B-Instruct as the readers. Based on the results shown in Table 1, we can have the following observations:

First, our method consistently outperforms baselines across all datasets and readers, demonstrating its effectiveness. Notably, with Qwen2.5-14B-Instruct-1M as the reader, our approach achieves a 22% relative average improvement over the best-performing baseline. By modeling evidence retrieval as a tree expansion process, we enable the policy model to thoroughly explore the sentence space, significantly enhancing performance.

Second, among the baselines, smaller retrieval models such as BGE-base-en exhibit strong performance compared to extraction-based methods, aligning with findings from Qian et al. (2024). This is attributed to the retrieval method’s ability to precisely compare query-sentence relevance, whereas extraction-based methods may overlook specific sentences, leading to information loss.

![Bar charts showing EM and latency results for removing beam search and early termination](page_1012_120_377_295.png)

Figure 4: We conduct an ablation study by removing beam search and early termination. The latency is calculated using the latency of our method as the basic unit.

Third, the performance gap between Qwen2.5-14B-Instruct-1M and Qwen2.5-72B-Instruct is relatively small despite their substantial difference in parameter sizes. This underscores the effectiveness of our policy model, which alleviates the reader’s burden by precisely identifying relevant evidence. As a result, users can choose smaller reader models without sacrificing model performance. This finding is particularly impactful, as it highlights the potential to substantially reduce inference costs while maintaining high-quality results.

4.5 Discussion

Ablation Study  During inference, we propose early terminating beam search to accelerate the process while ensuring comprehensive exploration of the sentence space. To evaluate its impact, we systematically remove beam search and early termination, conducting experiments on the 2Wiki-MultihopQA, HotpotQA, and MusiQue datasets. The results are shown in Figure 4.

Removing beam search significantly degrades performance, as the model fails to fully explore the sentence space, leading to suboptimal evidence retrieval. In contrast, removing early termination substantially improves latency, as the model can uniquely identify the location of the original sentence early in the decoding process, eliminating the need to decode subsequent tokens. Additionally, as the input length increases across the datasets, the inference latency grows significantly. However, with early termination, the additional latency caused by longer inputs is minimal, demonstrating its effectiveness in long-input scenarios. These results highlight the importance of both components in balancing performance and efficiency.

Hyperparameter Study  In our method, beam search is employed to efficiently explore the sentence space, with two key hyperparameters—Expansion Size \( B_1 \) and Beam Size
<table>
  <tr>
    <th rowspan="2">Size</th>
    <th colspan="2">HotpotQA</th>
    <th colspan="2">2WikiMulti</th>
    <th colspan="2">MusiQue</th>
  </tr>
  <tr>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <th colspan="7">Expansion Size \( B_1 \)</th>
  </tr>
  <tr>
    <td>1</td>
    <td>49.00</td>
    <td>50.72</td>
    <td>55.00</td>
    <td>53.98</td>
    <td>29.00</td>
    <td>29.63</td>
  </tr>
  <tr>
    <td>2</td>
    <td>50.50</td>
    <td>50.41</td>
    <td>60.50</td>
    <td>56.80</td>
    <td>35.50</td>
    <td>34.80</td>
  </tr>
  <tr>
    <td>3</td>
    <td>51.00</td>
    <td>51.26</td>
    <td>58.00</td>
    <td>56.23</td>
    <td>39.50</td>
    <td>39.85</td>
  </tr>
  <tr>
    <td>4</td>
    <td>55.00</td>
    <td>55.30</td>
    <td>59.50</td>
    <td>58.21</td>
    <td>39.00</td>
    <td>38.70</td>
  </tr>
  <tr>
    <td>5</td>
    <td>53.00</td>
    <td>55.28</td>
    <td>61.00</td>
    <td>60.52</td>
    <td>37.00</td>
    <td>36.74</td>
  </tr>
  <tr>
    <th colspan="7">Beam Size \( B_2 \)</th>
  </tr>
  <tr>
    <td>1</td>
    <td>49.00</td>
    <td>51.76</td>
    <td>56.00</td>
    <td>54.21</td>
    <td>32.50</td>
    <td>33.33</td>
  </tr>
  <tr>
    <td>2</td>
    <td>50.50</td>
    <td>51.68</td>
    <td>61.00</td>
    <td>58.88</td>
    <td>35.50</td>
    <td>36.20</td>
  </tr>
  <tr>
    <td>3</td>
    <td>51.00</td>
    <td>51.26</td>
    <td>58.00</td>
    <td>56.23</td>
    <td>39.50</td>
    <td>39.85</td>
  </tr>
  <tr>
    <td>4</td>
    <td>51.50</td>
    <td>52.86</td>
    <td>61.50</td>
    <td>59.65</td>
    <td>32.50</td>
    <td>33.37</td>
  </tr>
  <tr>
    <td>5</td>
    <td>51.00</td>
    <td>51.76</td>
    <td>60.50</td>
    <td>57.91</td>
    <td>31.00</td>
    <td>32.37</td>
  </tr>
</table>

Table 2: Hyperparameter study. We tune the parameters using Qwen2.5-14B-Instruct-1M as the reader.

\( B_2 \)—playing a crucial role. To investigate their impact on model performance, we conduct experiments on the 2WikiMultihopQA, HotpotQA, and MusiQue datasets, tuning these parameters within the range of 1 to 5.

Expansion size indicates the number of sentences expanded at each layer while beam size refers to the number of sentences retained at each layer during the search process. The results reveal a consistent pattern: performance peaks when the beam size or expansion size is set to approximately 4. Values smaller or larger than this threshold lead to a decline in performance. This is primarily because a larger beam size or expansion size offers the model a broader scope to explore potential sentences, increasing the likelihood of identifying the most relevant and optimal candidates. However, when these values exceed a certain threshold, the model encounters challenges in effectively ranking and selecting the best sentences from the expanded pool. This inefficiency arises because the increased search space introduces more noise and less relevant candidates, making it harder for the model to discern the best options. Additionally, increasing these values incurs higher computational costs, highlighting the need for a balanced configuration.

Effect of Input Length In this section, we analyze the effect of input length on model performance. Specifically, we vary the input length across {5000, 10000, 30000, 50000, 70000} and observe the resulting performance changes, comparing our method with using the full article as grounding evidence.

As shown in 5, both models exhibit a performance decline as input length increases. However, our method consistently outperforms the full-article input approach. While the performance of the full-article method drops significantly, our method shows only a slight decrease, demonstrating its robustness. This is because long-context models struggle to effectively capture the interaction between the query and the entire sentence set as input length grows. Our method addresses this challenge by modeling the process as node expansion, where the policy model is guided to thoroughly explore the sentence space at each step, ensuring robust performance even with longer inputs.

![Two line graphs showing F1 score vs. input length for Ours and Full methods on two datasets](page_1012_120_410_246.png)

Figure 5: Effect of input length. We vary the input length and observe the performance change.

4.6 Case Study

In this section, we compare the evidence retrieval results of different methods using the MusiQue dataset. Based on the examples shown in Table 3, several observations can be made:

First, both the retrieval-based method BGE-base-en and the extraction-based method Qwen2.5-72B fail to accurately locate the relevant grounding text, leading to the generation of irrelevant information for downstream tasks. Although CFIC-7B can identify the location of the grounding text, it lacks the necessary background information, making it difficult for the reader to deduce the correct answer. In contrast, both GPT-4 and our method successfully identify two critical entities: Adam Smith and the University of Cambridge. However, our method distinguishes itself by offering a more detailed and comprehensive grounding text. Specifically, our approach provides a clearer explanation of the relationship between Adam Smith and the Lectures on Jurisprudence, which is crucial for understanding the context. This enhanced clarity and depth of information make it significantly easier for readers to deduce the correct answer, as the grounding text not only identifies the relevant entities but also elaborates on their connections in a way that is both informative and accessible. Given the large param-
Query: Who gives out the prize named after the author of Lectures on Jurisprudence?
Answer: University of Cambridge

BGE-base-en: Lectures on Jurisprudence, also called Lectures on Justice, Police, Revenue and Arms (1763) is a collection of Adam Smith’s lectures... The prize winner receives a monetary award and will complete a lecture tour within the UK.

Qwen2.5-72B: The prize - named after Scottish philosopher and economist Adam Smith - was originally established in 1891 and awarded triennially for the best submitted essay on a subject of the writer’s choice..

CFIC-7B: "A library of his scholarly works, notes, and correspondences is held by the University of Cambridge. The Nobel Prize was not awarded between 1940 and 1942 due to the outbreak of World War II.

GPT-4: The Adam Smith Prizes are prizes currently awarded for the best overall examination performance and best dissertation in Part IIB of the Economics Tripos ... at the University of Cambridge.

ETS-7B: Lectures on Jurisprudence, also called Lectures on Justice ... is a collection of Adam Smith’s lectures... The Adam Smith Prizes are prizes currently awarded for the best overall examination ... at the University of Cambridge.

Table 3: Examples of evidence retrieval on dataset MusiQue. The text colored with teal refers to the key information related to the user query.

eter size gap between GPT-4 and our method, we believe our approach demonstrates strong performance at a minimal computational cost.

5 Related Work

5.1 Retrieval-Augmented Generation

Despite significant progress, Large Language Models (LLMs) still produce responses that contain hallucinated facts and inaccuracies (Ji et al., 2023; Shuster et al., 2021; Zhang et al., 2023a), undermining their overall reliability. To address this issue, Retrieval-Augmented Generation (RAG) has been introduced as a method to integrate external knowledge and enhance the accuracy of model responses (Ram et al., 2023; Shi et al., 2023; Rashkin et al., 2021; Gao et al., 2022; Bohnet et al., 2022; Menick et al., 2022).

Among existing approaches, some studies propose retrieving information only once at the beginning of the generation process (Shi et al., 2023; Wang et al., 2023b; Zhang et al., 2023c; Yu et al., 2023a,c). Other works (Qian et al., 2023; Yu et al., 2023b) suggest retrieving information multiple times throughout the generation process, offering greater flexibility in determining when and what to retrieve. For example, Jiang et al. (2023) advocate for retrieval only when the generation model produces low-confidence tokens, while Ram et al. (2023) recommend refreshing retrieved documents every n tokens, a method shown to outperform single retrieval approaches. Additionally, Wang et al. (2023a); Asai et al. (2023); Zhao et al. (2023b) propose retrieving information only when the LLM determines it is necessary, further improving retrieval efficiency.

5.2 Evidence Retrieval in RAG

While RAG has demonstrated strong performance, it faces challenges when dealing with lengthy and complex retrieved documents. To address this limitation, techniques like chunking and retrieval have been developed to improve passage relevance. For instance, Mao et al. (2021) introduced a chunking and retrieval method to enhance evidence selection, while Guu et al. (2020); Lin et al. (2023) proposed jointly training the retriever and generator to improve knowledge utilization and contextual understanding.

However, chunking can often be suboptimal, as determining the appropriate granularity for chunking is a challenging task. Improper chunking can disrupt the semantic coherence of the document, leading to less accurate evidence selection. Other approaches (Ratner et al., 2022; Xu et al., 2023) focus on adapting large language models to process longer contexts by training them on extended text lengths. For example, Chen et al. (2023) extended the context size of LLMs via parameter-efficient fine-tuning. Qian et al. (2024) proposed a chunking-free method to identify relevant evidence for user queries. Similarly, FILCO (Wang et al., 2023b) filters the input context with a context-filtering model. Despite these advancements, these methods often struggle to model the complex interactions between sentences in long contexts and may fail to accurately pinpoint the grounding evidence.

6 Conclusion

In this paper, we propose ETS, a novel framework that enhances evidence retrieval in RAG systems. By modeling evidence retrieval as a tree expansion process and leveraging MCTS and Early Terminat-
ing Beam Search, ETS effectively addresses the challenges of complex evidence interdependencies, lack of supervision signals, and large search spaces. Our extensive experiments demonstrate that ETS outperforms existing methods.

Limitations

In this paper, we propose a tree expansion method for evidence retrieval. We acknowledge two limitations of our method. First, The MCTS annotation requires multiple simulations, which can result in additional labeling costs. Second, due to resource constraints, we conduct experiments using only Qwen2.5-7B-Instruct-1M as the model backbone. We leave experiments with larger model backbones for future work.

Acknowledgement

This work is supported in part by Ucap Cloud and the State Key Laboratory of General Artificial Intelligence.

Ethics Statement

This work complies with the ACL Ethics Policy. All datasets and LLMs used are publicly available. Our research focuses on evidence retrieval, and we do not anticipate any negative ethical impacts.

References

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. Preprint, arXiv:2308.14508.

Bernd Bohnet, Vinh Q Tran, Pat Verga, Roei Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037.

Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024. Alphamath almost zero: process supervision without process. arXiv preprint arXiv:2405.03553.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023. Longora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. Preprint, arXiv:2105.03011.

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2022. Rarr: Researching and revising what language models say, using language models. arXiv preprint arXiv:2210.08726.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. Preprint, arXiv:2406.12793.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.

Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983.
Vladimir Karpukhin, Barlas Ouzg, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. Online. Association for Computational Linguistics.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857.

Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. 2023. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint arXiv:2310.01352.

Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Rider: Reader-guided passage reranking for open-domain question answering. arXiv preprint arXiv:2101.00294.

Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147.

Hongjin Qian, Zhicheng Dou, Jiejun Tan, Haonan Chen, Haoqi Gu, Ruofei Lai, Xinyu Zhang, Zhao Cao, and Ji-Rong Wen. 2023. Optimizing factual accuracy in text generation through dynamic knowledge selection. arXiv preprint arXiv:2308.15711.

Hongjin Qian, Zheng Liu, Kelong Mao, Yujia Zhou, and Zhicheng Dou. 2024. Grounding language model with chunking-free in-context retrieval. arXiv preprint arXiv:2402.09760.

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083.

Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021. Measuring attribution in natural language generation models. arXiv preprint arXiv:2112.12870.

Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2022. Parallel context windows improve in-context learning of large language models. arXiv preprint arXiv:2212.10947.

Christopher D Rosin. 2011. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence, 61(3):203–230.

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652.

Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2017. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815.

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. CoRR, abs/2211.09085.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Qwen Team. 2025. Qwen2.5-1m: Deploy your own qwen with context length up to 1m tokens.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multi-hop questions via single-hop question composition. Preprint, arXiv:2108.00573.

Yile Wang, Peng Li, Maosong Sun, and Yang Liu. 2023a. Self-knowledge guided retrieval augmentation for large language models. arXiv preprint arXiv:2310.05002.

Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. 2023b. Learning to filter context for retrieval-augmented generation. arXiv preprint arXiv:2311.08377.

Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2024. Evaluating mathematical reasoning beyond accuracy. arXiv preprint arXiv:2404.05692.

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.07597.

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025.

Ryutaro Yamauchi, Sho Sonoda, Akiyoshi Sannai, and Wataru Kumagai. 2023. Lpml: Ilm-prompting markup language for mathematical reasoning. arXiv preprint arXiv:2309.13078.
Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pages 1253–1256.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Preprint, arXiv:1809.09600.

Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023a. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210.

Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. 2023b. Improving language models via plug-and-play retrieval feedback. arXiv preprint arXiv:2305.14002.

Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. 2023c. Augmentation-adapted retriever improves generalization of language models as generic plug-in. arXiv preprint arXiv:2305.17331.

Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Sricharan Kumar. 2023a. Sac3: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15445–15458.

Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. 2023b. Retrieve anything to augment large language models. Preprint, arXiv:2310.07554.

Yunxiang Zhang, Muhammad Khalifa, Lajantugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023c. Merging generated and retrieved knowledge for open-domain qa. arXiv preprint arXiv:2310.14393.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023a. A survey of large language models. arXiv preprint arXiv:2303.18223.

Xinran Zhao, Hongming Zhang, Xiaoman Pan, Wenlin Yao, Dong Yu, and Jianshu Chen. 2023b. Thrust: Adaptively propels large language models with external knowledge. arXiv preprint arXiv:2307.10442.
A  Dataset Statistics

<table>
  <tr>
    <th>Settings</th>
    <th>2WikiMultihopQA<br>(Ho et al., 2020)</th>
    <th>HotpotQA<br>(Yang et al., 2018)</th>
    <th>MuSiQue<br>(Trivedi et al., 2022)</th>
    <th>MultiFieldQA<br>(Bai et al., 2023)</th>
    <th>Qasper<br>(Dasigi et al., 2021)</th>
  </tr>
  <tr>
    <td>Task</td>
    <td>Multi-doc QA</td>
    <td>Multi-doc QA</td>
    <td>Multi-doc QA</td>
    <td>Single-doc QA</td>
    <td>Single-doc QA</td>
  </tr>
  <tr>
    <td>Train Data</td>
    <td>5,000</td>
    <td>5,000</td>
    <td>5,000</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Test Data</td>
    <td>200</td>
    <td>200</td>
    <td>200</td>
    <td>200</td>
    <td>200</td>
  </tr>
  <tr>
    <td>Average Length</td>
    <td>29,495</td>
    <td>56,446</td>
    <td>69,269</td>
    <td>28,947</td>
    <td>23,640</td>
  </tr>
  <tr>
    <td>Metrics</td>
    <td>EM, F1</td>
    <td>EM, F1</td>
    <td>EM, F1</td>
    <td>EM, F1</td>
    <td>EM, F1</td>
  </tr>
</table>

Table 4: Statistics and experimental settings of different tasks/datasets.

B  Training Details

Dataset Construction  We sample 5000 queries from the training data of 2WikiMultihopQA, HotpotQA, and MuSiQue datasets and conduct the MCTS annotations. We then sample 6,000 correct paths and 6,000 incorrect paths, and the correct paths are used to train the policy model, while both paths are used to train the value model.

Training Process  The policy model is initialized using the Qwen2.5-7B-Instruct-1M. Training involves fine-tuning the model for two epochs with a batch size of 1 and a learning rate of \( 1 \times 10^{-6} \), utilizing 8 NVIDIA A100 80GB GPUs. During inference, the expansion size and the beam search size are set to 3.

C  Model Sources

• BGE-base-en-v1.5: https://huggingface.co/BAAI/bge-base-en-v1.5
• LLM-Embedder: https://huggingface.co/BAAI/llm-embedder
• ChatGLM3-6B-128K: https://huggingface.co/THUDM/chatglm3-6b-128k
• Qwen2.5-7B-Instruct-1M: https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-1M
• GLM-4-9B-Chat-1M: https://huggingface.co/THUDM/glm-4-9b-chat-1m
• Qwen2.5-14B-Instruct-1M: https://huggingface.co/Qwen/Qwen2.5-14B-Instruct-1M
• Qwen2.5-32B-Instruct: https://huggingface.co/Qwen/Qwen2.5-32B-Instruct
• Qwen2.5-72B-Instruct: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct

D  Prompts

Extraction Prompt
Extract sentences from the following documents to answer the question. Do not alter the content of the sentences.
Documents: {background}
Question: {query}
Relevant Sentences:

Answering Prompt
Use only the information from the following documents to answer the question with one short phrase.
Documents: {background}
Question: {query}
Output:

Reviewer paper summaries: This paper proposes a new RAG framework that aims at addressing evidence retrieval issues. It formulates evidence retrieval as a dynamic tree expansion process. It treats the evidence sentences as tree nodes and uses MCTS to guide the exploration of potential evidence combinations. Values derived from MCTS are used to train a evidence quality evaluation model which is later used in inference time to reduce the exploration cost. Experiments are conducted with multiple RAG datasets that proves the effectiveness of ETS.

This paper introduces Evidence Tree Search (ETS), a novel framework for enhancing Retrieval-Augmented Generation (RAG) by improving evidence retrieval from lengthy documents. The authors identify three key challenges in evidence retrieval: (1) the inability to model interdependencies among evidence sentences, (2) lack of supervision for evaluating multi-sentence evidence quality, and (3) computational inefficiency in searching large spaces of candidate evidence sets.

The proposed ETS framework addresses these challenges by reformulating evidence retrieval as a dynamic tree expansion process where each path represents a candidate evidence set. The approach has three main components: (1) an evidence tree construction that models inter-sentence dependencies through context-aware node selection, (2) Monte Carlo Tree Search (MCTS) to efficiently assess evidence quality, and (3) an Early-Terminating Beam Search strategy to accelerate model inference.

Experiments conducted across five diverse datasets (2WikiMultihopQA, HotpotQA, MusiQue, MultiFieldQA, and Qasper) demonstrate that ETS significantly outperforms existing methods across different reader models.
Summary of strengths: * Combinatorial search space is a notorious problem in RAG and the paper correctly identifies it.
* The proposed combined training of policy and value model is novel and has potential in reducing inference time. Although the model might consider to include more ablations to study the effectiveness of value network.
* The paper conducts thorough experiments with improved performances.

(1) Novel framework: The paper introduces ETS (Evidence Tree Search), which reformulates evidence retrieval as a dynamic tree expansion process, addressing important limitations in current RAG systems.

(2) Well-designed methodology: The paper presents a coherent approach using Monte Carlo Tree Search (MCTS) to guide evidence exploration and an Early-Terminating Beam Search strategy to improve efficiency.

(3) Comprehensive evaluation: The authors conduct extensive experiments across five datasets, demonstrating significant performance improvements over existing methods with different reader models.
Summary of weaknesses: * The paper might include a study on the efficiency of ETS compared with other RAG models.
* The paper heuristically choose train additional value model. What if the model only trains on the correct part from the MCTS search. This might provide a better understanding of the effectiveness of ETS.

Resource requirements not clearly detailed: The computational cost and resource requirements for training and inference are not thoroughly documented, making it difficult to assess practicality.

Need More comparisons. The retrieval-based methods in the experiments are Sac3 and BGE-base-en-v1.5, which are not the most advanced text embedding models currently available.
Comments, suggestions and typos: N/A

(1) Add specific inference time measurements for ETS-7B vs. baselines.

(2) Add gte-large embedding as an additional baseline in experiments.
Overall assessment scores: 4

3.5