RISE: Reasoning Enhancement via Iterative Self-Exploration in Multi-hop Question Answering
Authors: Bolei He, Xinran He, Mengke Chen, xianwei xue, Ying Zhu, Zhen-Hua Ling
Accepted at: acl25

RISE: Reasoning Enhancement via Iterative Self-Exploration in Multi-hop Question Answering

Bolei He1,2* Xinran He2* Mengke Chen2 Xianwei Xue2
Ying Zhu2 Zhen-Hua Ling1†
1University of Science and Technology of China, Hefei, China
2Baidu Inc., Beijing, China
hebl@mail.ustc.edu.cn, zhling@ustc.edu.cn,
{hexinran, xuexianwei, zhuying11}@baidu.com, anthreebody@gmail.com

Abstract

Large Language Models (LLMs) excel in many areas but continue to face challenges with complex reasoning tasks, such as Multi-Hop Question Answering (MHQA). MHQA requires integrating evidence from diverse sources while managing intricate logical dependencies, often leads to errors in reasoning. Retrieval-Augmented Generation (RAG), widely employed in MHQA tasks, faces challenges in effectively filtering noisy data and retrieving all necessary evidence, thereby limiting its effectiveness in addressing MHQA challenges. To address these challenges, we propose RISE: Reasoning Enhancement via Iterative Self-Exploration, a novel framework designed to enhance models’ reasoning capability through iterative self-exploration. Specifically, RISE involves three key steps in addressing MHQA tasks: question decomposition, retrieve-then-read, and self-critique. By leveraging continuous self-exploration, RISE identifies accurate reasoning paths, iteratively self-improving the model’s capability to integrate evidence, maintain logical consistency, and enhance performance in MHQA tasks. Extensive experiments on multiple MHQA benchmarks demonstrate that RISE significantly improves reasoning accuracy and task performance.

1 Introduction

Large language models (LLMs) demonstrate outstanding capabilities in natural language understanding and generation (Brown et al., 2020; Zhang et al., 2022; Zeng et al., 2022; Chowdhery et al., 2023; Touvron et al., 2023). However, LLMs still face challenges with complex Multi-Hop Question Answering (MHQA) tasks. MHQA requires models to integrate evidence from multiple sources and manage intricate logical relationships. This involves both retrieving and combining various pieces of evidence and constructing coherent reasoning chains. Prompt-based methods, such as Chain-of-Thought (CoT) (Wei et al., 2022b; Wang et al., 2023a; Yu et al., 2023), are employed to address MHQA by split complex problems into smaller, thereby harnessing the reasoning potential of LLMs. However, these methods often lack external knowledge, resulting in key evidence being overlooked and generate hallucinations (Rawte et al., 2023; Ji et al., 2023; Ye et al., 2023).

Retrieval-Augmented Generation (RAG) methods (Guu et al., 2020; Lewis et al., 2020; Izacard et al., 2022; Nakano et al., 2021; Asai et al.,

![The upper part of the figure (blue) illustrates an Evidence Aggregation Error, where the Blu-ray release year of Fire Birds (2015) is mistaken for its theatrical release year. The lower part (green and red) shows a Reasoning Decomposition Error. The incorrect path formulates the sub-question as the production year of The Book of Eli (2009) instead of its release year (2010).](page_1016_370_420_384.png)

Figure 1: The upper part of the figure (blue) illustrates an Evidence Aggregation Error, where the Blu-ray release year of Fire Birds (2015) is mistaken for its theatrical release year. The lower part (green and red) shows a Reasoning Decomposition Error. The incorrect path formulates the sub-question as the production year of The Book of Eli (2009) instead of its release year (2010).
2023; Ma et al., 2023; Yu et al., 2024; Shi et al., 2024a) have been proposed to address the aforementioned challenges. By incorporating external knowledge, RAG effectively mitigates hallucination phenomena and achieves significant results in MHQA tasks through multiple retrievals. However, RAG is constrained by the performance of the retrievers, inevitably introducing noise. Additionally, the multi-round retrieval process may lead to error propagation, resulting in two main types of errors: Evidence Aggregation Errors and Reasoning Decomposition Errors. As illustrated in Figure 1, Evidence Aggregation Errors occur when the model fails to accurately integrate evidence from multiple evidences, leading to hallucinations. Reasoning Decomposition Errors arise when problem decomposition phase generates sub-questions that do not align with original question’s intent. These issues are particularly pronounced in smaller models with weaker reasoning capabilities.

Distillation and fine-tuning (Uesato et al., 2022; Luo et al., 2023; Shridhar et al., 2023) effectively enhance the reasoning capabilities of LLMs by leveraging large-scale models or high-quality, manually annotated data to improve performance. However, biases brought by human subjective annotations may undermine the performance of fine-tuning (Casper et al., 2023; Lightman et al., 2023), and these methods are costly, requiring substantial human or computational resources. Meanwhile, self-iteration methods (Yuan et al., 2024; Wang et al., 2024; Madaan et al., 2024) demonstrate tremendous potential in complex reasoning tasks. Unlike approaches that depend on large-scale models and manual annotations, self-iteration methods enable models to generate and learn from their own data, achieving outstanding results in complex tasks such as code generation and intelligent agents (Jiang et al., 2023; Ni et al., 2024; Qiao et al., 2024). Nevertheless, research on combination self-iteration methods with RAG remains limited. The integration of these two approaches has the potential to improve performance in complex reasoning tasks and leads to cost reduction.

In this paper, we introduce an innovative framework, RISE (Reasoning Enhancement via Iterative Self-Exploration), which combines the paradigms of RAG and self-iteration to address key challenges in MHQA tasks. Specifically, RISE defines three core actions: question decomposition, retrieve-then-read, and self-critique. By repeatedly executing these actions, the model autonomously explores accurate reasoning paths for problems. During this process, RISE accumulates experience datasets for the three actions and updates the model based on this experience. Through multiple iterations, RISE significantly enhances the model’s reasoning capabilities in MHQA tasks. Experimental results demonstrate that RISE outperforms baseline methods on several MHQA benchmark datasets, strongly validating its effectiveness in solving MHQA tasks while offering lower usage costs. Our main contributions are as follows:

• We propose RISE, which combines RAG and self-iteration to address two key challenges in MHQA tasks: Evidence Aggregation Errors and Reasoning Decomposition Errors.

• We design self-exploration mechanism, converts MHQA in RAG into multi-objective optimization problem, thus improving model’s reasoning capability and reducing costs.

• We integrate self-iteration paradigm with RAG, bridging gap in applying self-iteration strategies within MHQA RAG framework.

2 Methods

2.1 Overview

In this section, we provide a concise description of RISE, focusing on its algorithmic process. As shown in algorithm 1, RISE begins with a seed question set \( Q_0 \) and an initial model \( M_0 \). The model iteratively performs self-exploration for each question \( q \in Q_i \), with details presented in Section 2.2. The exploration results are stored as historical experience \( \mathcal{H} \). After completing the exploration for all questions, the accumulated experiences optimize the model through multi-objective training, yielding an enhanced model \( M_{i+1} \). Subsequently, \( M_{i+1} \) expands the question set based on the previous seed questions \( Q_i \), generating \( Q_{i+1} \) to initiate the next round of exploration. This self-iterative process enables RISE to continuously improve capabilities without external supervision, leveraging the model’s intrinsic potential.

2.2 Self-Exploration Mechanism

The self-exploration mechanism enables the model to address complex problems through iterative reasoning, comprising three core actions: question decomposition, retrieve-then-read, and self-critique.
Figure 2: A complete iteration cycle in RISE. a) Self-Exploration: Model \( M^i \) decomposes complex questions \( q_0 \) into simpler sub-questions, generates sub-answers via retrieve-then-read, and evaluates their validity, leading to a final answer \( a_0 \). Interactions are stored as historical data \( \mathcal{D} \). b) Iterative Optimization: RISE optimizes model \( M^i \) using historical data \( \mathcal{D} \) to create an enhanced model \( M^{i+1} \), which generates new questions \( Q^{i+1} \) for the next cycle.

These actions collectively form a structured exploration pathway, with the resulting information collected as historical data to support the model’s self-improvement in complex problem-solving. The related prompts are provided in Appendix A.1.1.

**Question Decomposition.** In this task, the model incrementally decomposes the initial complex question into fine-grained sub-questions. At the \( t \)-th exploration node, the model uses previously explored sub-questions and answers as historical information, denoted as \( \mathcal{H} = (subq_1, suba_1), \cdots, (subq_{t-1}, suba_{t-1}) \). The original question \( q_0 \) is combined with \( \mathcal{H} \) and input into model \( M \) to generate the next sub-question. The model ends exploration by generating the final answer if the historical information suffices for \( q_0 \). Formally, this process is represented as Formula 1:

\[
subq_t = \mathcal{F}_d(M, \mathcal{H}, q_0) \tag{1}
\]
\[
a_0 = M(q_0, \mathcal{H}), \quad \text{if } \mathcal{H} \text{ is sufficient.} \tag{2}
\]

Additionally, all decomposition steps, including the original question and generated sub-questions, are recorded to form the dataset \( \mathcal{D}_d = \{ \{q_0, \mathcal{H}, subq\}_{i=1}^{n_p}\}_{j=1}^{N_q} \). By leveraging this fine-grained and structured dataset, the model learns the logical dependencies and relationships between questions and sub-questions, thereby improving its ability to decompose complex problems.

**Retrieve-then-Read.** This task follows the standard RAG paradigm to provide evidence-based answers for sub-questions. At the \( t \)-th exploration node, a retriever obtains relevant fragments \( r_t \) based on the sub-question, and model \( M \) generates answer using the retrieved evidence:

\[
suba_t = \mathcal{F}_g(M, subq_t, r_t) \tag{3}
\]

Each sub-question and its answer form an exploration node \( (subq_i, suba_i) \), added to the historical information \( \mathcal{H}_{t+1} = \mathcal{H}_t \cup \{ (subq_i, suba_i) \} \). All nodes are recorded to construct the dataset \( \mathcal{D}_r = \{ (subq, r, suba)_{i=1}^{n_p}\}_{j=1}^{N_q} \). Training on this dataset helps the model integrate evidence into reasoning, improving answer accuracy and reliability.

**Self-Critique.** In this task, the model’s critique capability is incorporated into the exploration process. Specifically, after completing the question decomposition and retrieve-then-read tasks at the \( t \)-th exploration node, the model \( M \) critiques the relevance and utility of the node for solving the original question and outputs a binary decision. If critiqued as True, it is retained, and exploration proceeds to the next step. If critiqued as False, the node is temporarily stored, and the process reverts
Algorithm 1 RISE
Input: Seed question set \( Q_0 \), Initial model \( M_0 \), Retriever \( R \), Maximum nodes \( N_{max} = 20 \)
1: Initialize: History \( \mathcal{H} \leftarrow \emptyset \), Model index \( i \leftarrow 0 \)
2: while True do
3:   for each question \( q \in Q_i \) do
4:     \( n \leftarrow 0 \) ▷ Start self-exploration.
5:     while \( M_i(q, \mathcal{H}) = \) More information needed and \( n < N_{max} \) do
6:       subq \( \leftarrow M_i(\mathcal{H}) \)
7:       \( r \leftarrow R(subq) \)
8:       suba \( \leftarrow M_i(subq, r) \)
9:       \( \sigma \leftarrow M_i(subq, suba) \)
10:      if \( \sigma = 1 \) then
11:        Add (\( subq, suba \)) to \( \mathcal{H} \)
12:      end if
13:      \( n \leftarrow n + 1 \)
14:    end while
15:    \( a \leftarrow M_i(\mathcal{H}) \)
16:    \( \sigma_{final} \leftarrow M_i(a, q, \mathcal{H}) \)
17:  end for ▷ End self-exploration.
18:  \( M_{i+1} \leftarrow \) Multi-Objective Train\((M_i, \mathcal{H})\)
19:  \( Q_{i+1} \leftarrow \) Question Expansion\((M_{i+1}, Q_i)\)
20:  \( i \leftarrow i + 1 \)
21: end while
Output: Final model \( M_i \)

<table>
  <tr>
    <th></th>
    <th colspan="5">weights ratios</th>
  </tr>
  <tr>
    <th>\( \alpha \)</th>
    <td>1</td>
    <td>1</td>
    <td>4</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <th>\( \beta \)</th>
    <td>1</td>
    <td>4</td>
    <td>1</td>
    <td>2</td>
    <td>3</td>
  </tr>
  <tr>
    <th>\( \gamma \)</th>
    <td>4</td>
    <td>1</td>
    <td>1</td>
    <td>3</td>
    <td>2</td>
  </tr>
  <tr>
    <th>Accuracy</th>
    <td>41.55</td>
    <td>40.32</td>
    <td>41.94</td>
    <td>38.71</td>
    <td>37.63</td>
  </tr>
  <tr>
    <th>\( \alpha \)</th>
    <td>2</td>
    <td>2</td>
    <td>3</td>
    <td>3</td>
    <td>2</td>
  </tr>
  <tr>
    <th>\( \beta \)</th>
    <td>1</td>
    <td>3</td>
    <td>1</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <th>\( \gamma \)</th>
    <td>3</td>
    <td>1</td>
    <td>2</td>
    <td>1</td>
    <td>2</td>
  </tr>
  <tr>
    <th>Accuracy</th>
    <td>39.78</td>
    <td>40.32</td>
    <td>43.01</td>
    <td>41.13</td>
    <td>44.27</td>
  </tr>
</table>

Table 1: Performance on the 2WikiMultiHopQA dataset under varying weight ratios of \( \alpha \), \( \beta \), and \( \gamma \).

to the preceding valid node to generate a new node. This process is formalized in Formula 4:

\[
\sigma_t = \mathcal{F}_c(M, subq_t, suba_t), \quad \sigma_t \in \{0, 1\}
\] (4)

Similarly, we record critique historical information and then construct the dataset \( \mathcal{D}_c = \{ \{ \langle subq, suba \rangle, \sigma \}_{t=1}^{n_p} \}_{i=1}^{N_q} \), to enhance the self-critique capabilities of the model, ensuring logical consistency and relevance within the exploration.

2.3 Self-Iterative Optimization

RISE is a self-iterative fine-tuning framework that optimizes the model in each training round based on data generated by the model itself, gradually enhancing its generalization ability and reasoning performance. Through a closed-loop iteration of data generation and model training, RISE effectively uncovers the model’s reasoning potential in complex tasks, driving continuous self-improvement.

Initialization. We initially use randomly sampled question set \( Q_0 \) from the training sets of three tasks—2Wiki, HotpotQA, and Musique, with 800 examples from each. Subsequently, we employ self-exploration mechanism to automatically expand and collect the three types of dataset. Subsequently, we employ self-exploration mechanism to automatically expand and collect \( \mathcal{D}_d, \mathcal{D}_r \), and \( \mathcal{D}_c \) datasets for subsequent model training.

Multi-Objective Optimization. These three datasets, \( \mathcal{D}_d, \mathcal{D}_r \), and \( \mathcal{D}_c \), are interconnected, with sample sizes ranging from 2k to 8k (detailed statistics are provided in Appendix 5). We believe that joint training facilitates complementary learning and enhances model capabilities. Therefore, we adopt a multi-objective optimization approach to integrate the objectives of different tasks into a unified optimization goal. The effectiveness of this approach is validated through ablation study at Section 4.3. The overall loss function is defined as follows Formula 5:

\[
\mathcal{L} = \alpha \mathcal{L}_d + \beta \mathcal{L}_r + \gamma \mathcal{L}_c
\] (5)

Here, \( \alpha \), \( \beta \), and \( \gamma \) represent task weights, determined by the proportion of each capability in the training data. Specifically, \( \mathcal{L}_d \) (Formula 6) models the autoregressive loss for sub-question generation, \( \mathcal{L}_r \) (Formula 7) models the loss for sub-answer generation conditioned on retrieved context, and \( \mathcal{L}_c \) (Formula 8) denotes a binary classification loss for self-critique judgments, where True and False represent the predicted probabilities of the respective tokens.

\[
\mathcal{L}_d = - \sum_i \log P(\text{subq}_i \mid q_0, \text{subq}_{<i})
\] (6)

\[
\mathcal{L}_r = - \sum_i \log P(\text{suba}_i \mid \text{subq}_i, r_i)
\] (7)

\[
\mathcal{L}_c = - \sigma \log P(\text{True} \mid \text{subq}_i, \text{suba}_i) \\
- (1 - \sigma) \log P(\text{False} \mid \text{subq}_i, \text{suba}_i)
\] (8)

\footnotetext{0} \( P(\text{True}) = \frac{P(\text{Token(True)})}{P(\text{Token(True)})+P(\text{Token(False)})} \) and \( P(\text{False}) \) accordingly.
Meanwhile, Experimental results (Table 1) indicate that the weights assigned to different tasks have an impact on model performance, and appropriate weight adjustments facilitate fine-grained performance optimization. Notably, to avoid potential overfitting caused by manual weight tuning which may affect the final evaluation we do not perform any fine-tuning of the task weights in our experiments. Instead, we adopt a uniform weighting strategy, assigning equal weights to all tasks.

Question Expansion After completing multi-objective optimization, we use the questions generated in the previous iteration as seed data for \( M^{t+1} \) to perform question expansion, thereby acquiring training data for the next iteration. This method is inspired by (Wang et al., 2023c), leveraging multi-round in-context learning to ensure the diversity and richness of the newly generated questions. Detailed information about the question expansion prompts is provided in the appendix Figure 10.

3 Experiments Setup

Datasets: For the main experiments, we use three QA datasets: 2WikiMultiHopQA (2WIKI) (Ho et al., 2020), HotpotQA (Hotpot) (Yang et al., 2018), and MuSiQue (MSQ) (Trivedi et al., 2022), which provide diverse reasoning challenges to evaluate the robustness of our framework. Additionally, for the analysis experiments, we include Natural Questions (NQ) (Kwiatkowski et al., 2019), Web Questions (WebQ) (Berant et al., 2013) and TriviaQA (Joshi et al., 2017) to assess the model’s performance on open-domain Question Answering tasks, further extending the evaluation scope.

Models and Methods: In our experiments, we use LLaMA-3.1-8B (Dubej et al., 2024) as the base model for our method in main experiments. Similarly, most of the reproduced methods are also implemented using LLaMA-3.1-8B. Additionally, based on the characteristics of MHQA tasks, we select and reproduce a variety of methods, categorized into non-retrieval-based methods and retrieval-based methods. Non-retrieval-based methods include Naive LLM (LLaMA-3.1-8B, GPT-3.5-turbo), CoT (Wei et al., 2022b), CoT-SC (Wang et al., 2023a) and GenRead (Yu et al., 2023), while the retrieval-based methods consist of Naive RAG, Self-Ask (Press et al., 2023), WebGLM (Liu et al., 2023), Self-RAG (Asai et al., 2023), RRR (Ma et al., 2023), and GenGround (Shi et al., 2024a). In the analysis experiments, we employ GPT-4o\(^1\) as the evaluation model, combining subjective analysis with specific metrics to comprehensively assess model performance.

Retrieval: We adopt a two-stage retrieval framework (Liu et al., 2023), consisting of coarse-grained web search (via Chrome) followed by fine-grained LLM-enhanced retrieval. We consistently use the same retrieval method to reproduce results for other approaches that incorporate retrievers.

Evaluation Metrics: We assess model performance primarily using standard metrics for question answering tasks: Accuracy (Acc), F1 score (F1), and Exact Match (EM), which together provide a comprehensive measure of answer correctness and completeness. In addition to answer quality, we evaluate the generated reasoning chains by examining their length as an objective measure of complexity. Furthermore, we conduct a qualitative assessment of the reasoning chains from four subjective perspectives: conciseness, rationality, sequencing, and goal orientation.

We provide comprehensive experimental details in Appendix A.2, including implementation details, datasets, and other relevant information.

Figure 3: Changes in model accuracy (a) and reasoning length (b) across datasets. Accuracy consistently improves across datasets, while reasoning length, despite some fluctuations, shows an overall decreasing trend.

\footnotetext{1We use GPT models accessed via the OpenAI API: https://openai.com/api/.}
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Model</th>
    <th colspan="3">MHQA</th>
    <th colspan="2">SHQA</th>
  </tr>
  <tr>
    <th>2Wiki</th>
    <th>Hotpot</th>
    <th>MSQ</th>
    <th>NQ</th>
    <th>WebQ</th>
    <th>Trival</th>
  </tr>
  <tr>
    <td colspan="8"><i>w/o retrieval</i></td>
  </tr>
  <tr>
    <td>Naive LLM</td>
    <td>LLaMA-3.1-8B</td>
    <td>35.90</td>
    <td>27.30</td>
    <td>11.30</td>
    <td>57.50</td>
    <td>61.25</td>
    <td>71.50</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-3.5-turbo</td>
    <td>47.10</td>
    <td>41.50</td>
    <td>19.10</td>
    <td>57.25</td>
    <td>58.30</td>
    <td>80.25</td>
  </tr>
  <tr>
    <td>CoT (Wei et al., 2022b)</td>
    <td>LLaMA-3.1-8B</td>
    <td>43.00</td>
    <td>34.60</td>
    <td>16.20</td>
    <td>56.75</td>
    <td>62.00</td>
    <td>71.75</td>
  </tr>
  <tr>
    <td>CoT-SC* (Wang et al., 2023a)</td>
    <td>GPT-3.5-turbo</td>
    <td>21.00</td>
    <td>30.60</td>
    <td>8.90</td>
    <td>39.75</td>
    <td>38.50</td>
    <td>79.25</td>
  </tr>
  <tr>
    <td>GenRead (Yu et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>20.00</td>
    <td>28.40</td>
    <td>10.30</td>
    <td>46.25</td>
    <td>43.00</td>
    <td>67.00</td>
  </tr>
  <tr>
    <td colspan="8"><i>w retrieval</i></td>
  </tr>
  <tr>
    <td>Naive RAG</td>
    <td>LLaMA-3.1-8B</td>
    <td>43.50</td>
    <td>37.50</td>
    <td>15.00</td>
    <td>51.75</td>
    <td>57.75</td>
    <td>73.75</td>
  </tr>
  <tr>
    <td>Self-Ask (Press et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>22.90</td>
    <td>29.70</td>
    <td>12.70</td>
    <td>44.50</td>
    <td>46.50</td>
    <td>66.00</td>
  </tr>
  <tr>
    <td>WebGLM (Liu et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>37.60</td>
    <td>36.50</td>
    <td>13.30</td>
    <td>55.75</td>
    <td>62.25</td>
    <td>73.75</td>
  </tr>
  <tr>
    <td>Self-RAG* (Asai et al., 2023)</td>
    <td>LLaMA-2.7B</td>
    <td>32.00</td>
    <td>30.20</td>
    <td>8.00</td>
    <td>54.50</td>
    <td>60.25</td>
    <td>74.25</td>
  </tr>
  <tr>
    <td>RRR (Ma et al., 2023)</td>
    <td>LLaMA-2.13B</td>
    <td>30.80</td>
    <td>29.40</td>
    <td>8.80</td>
    <td>55.25</td>
    <td>58.75</td>
    <td>75.00</td>
  </tr>
  <tr>
    <td></td>
    <td>LLaMA-3.1-8B</td>
    <td>23.70</td>
    <td>11.80</td>
    <td>5.40</td>
    <td>22.50</td>
    <td>30.50</td>
    <td>36.75</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-3.5-turbo</td>
    <td>28.20</td>
    <td>29.70</td>
    <td>8.70</td>
    <td>57.75</td>
    <td>56.50</td>
    <td>80.50</td>
  </tr>
  <tr>
    <td>GenGround (Shi et al., 2024a)</td>
    <td>LLaMA-3.1-8B</td>
    <td>37.90</td>
    <td>36.10</td>
    <td>17.80</td>
    <td>48.50</td>
    <td>44.50</td>
    <td>75.25</td>
  </tr>
  <tr>
    <td><b>RISE(Ours)</b></td>
    <td>LLaMA-3.1-8B</td>
    <td>49.40</td>
    <td>40.50</td>
    <td>21.70</td>
    <td>59.50</td>
    <td>62.50</td>
    <td>80.25</td>
  </tr>
</table>

Table 2: Comparison of RISE’s accuracy with other methods on 2WikiMultiHopQA, HotpotQA, MuSiQue, Natural Questions, Web Questions, and TriviaQA. Methods marked with an asterisk (*) involve specific considerations: CoT-SC uses GPT-3.5 due to LLaMA-3.1’s instruction-following limitations, and Self-RAG employs public model weights as its dataset is unavailable. Other methods use LLaMA-3.1-8B. <span style="background-color:#e0f2d4;">green</span> denote Prompting-based Methods, while <span style="background-color:#ffffcc;">yellow</span> denote Finetuning-based Methods. Due to space constraints, <b>F1</b> and <b>EM</b> metrics are in Appendix A.4.

4 Results and Analysis

In this section, we evaluate RISE from three aspects. First, we validate effectiveness of multi-round self-iterative and compare RISE with mainstream MHQA methods. Second, we conduct an in-depth analysis of the performance of question decomposition, retrieve-then-read, and self-critique using objective metrics and AI-based evaluations. Finally, we conduct ablation studies to verify the importance of different tasks in enhancing performance.

4.1 Overall Performance

<b>RISE Outperforms Other Methods:</b> Table 2 presents the experimental results across three MHQA datasets and three SHQA datasets. We observe that retrieval-based enhancement is crucial for MHQA tasks. While CoT achieves relatively good performance, other non-retrieval methods generally perform worse than most RAG approaches with the same model. For the relatively simpler SHQA tasks, retrieval-based enhancement does not seem to offer significant advantages. Notably, RISE achieves outstanding results in both task types over all datasets, even surpassing GPT-3.5. Furthermore, our method excels in F1 and EM metrics, demonstrating its efficiency (additional metrics are provided in Appendix A.4).

<b>Steady Performance Improvement:</b> Meanwhile, as shown in Figure 3 (a) <i>Accuracy per Iteration</i>, we illustrate how the model’s accuracy evolves over four iterations on multiple datasets. The results demonstrate a consistent upward trend in accuracy with each iteration, further validating the effectiveness of our proposed self-training method in improving the model’s overall performance.

4.2 Analysis Experiments

4.2.1 Question Decomposition Capability

To evaluate improvement in the model’s decomposition capability for MHQA tasks, we first analyze the changes in reasoning length. As shown in Figure 3 (b) <i>Reasoning Length per Iteration</i>, accuracy steadily improves, while reasoning length initially increases and then decreases, ultimately showing downward trend. This trend reflects model’s decomposition ability progressively improves over iterations.

To further analyze changes in decomposition ability, we using GPT-4o as a judge to evaluate the model’s query decomposition across four dimensions (including conciseness, rationality, sequencing and goal orientation, see Appendix A.1.2 for more details.). As illustrated in Figure 4, we compare the performance of the model across iterations and observe newer model consistently outperforms the previous iteration. These findings demonstrate that self-training not only improves reasoning paths but also enhances the rationality of decomposition.
<table>
  <tr>
    <th></th>
    <th>Win</th>
    <th>Tie</th>
    <th>Loss</th>
  </tr>
  <tr>
    <th>Round 1 vs. Round 0</th>
    <td>48.4%</td>
    <td>20.3%</td>
    <td>31.3%</td>
  </tr>
  <tr>
    <th>Round 2 vs. Round 1</th>
    <td>49.8%</td>
    <td>16.2%</td>
    <td>34%</td>
  </tr>
  <tr>
    <th>Round 3 vs. Round 2</th>
    <td>47.6%</td>
    <td>23.0%</td>
    <td>29.4%</td>
  </tr>
  <tr>
    <th>Round 4 vs. Round 3</th>
    <td>57.3%</td>
    <td>19.7%</td>
    <td>23.0%</td>
  </tr>
</table>

Figure 4: Evaluating the win rates between the current and previous iterations using GPT-4o to assess model’s question decomposition capability. Results indicate that each new iteration consistently outperforms the previous one in subjective effectiveness, demonstrating RISE’s continuously enhance the model’s decomposition capability.

![Bar chart showing win, tie, and loss rates for different rounds](page_184_120_1080_246.png)

Figure 5: Changes in the model’s retrieve-then-read capability. (a) Results on simpler datasets (NQ, TriviaQA, WebQ). (b) Results on more complex datasets (2Wiki, HotpotQA, MSQ), where accuracy shows consistent growth with each iteration, even in challenging scenarios.

![Bar charts comparing accuracy across rounds for simple and complex questions](page_184_420_1080_246.png)

4.2.2 Retrieve-then-Read Capability
In MHQA tasks, models often struggle to integrate logical information from extensive evidence, especially in filtering irrelevant content. To evaluate the changes in the model’s summarization capability over iterations, we disable the decomposition functionality and instead allow model to perform single-round retrieval and direct question-answering. To ensure robustness in the experiments, we introduce simpler datasets such as NQ, WebQ, and TriviaQA (Figure 5 (a) *Simple Questions*) while retaining the complex datasets from main experiments (Figure 5 (b) *Complex Questions*). The experimental results show that, as iterations progress, RISE consistently improves its performance across six datasets. This demonstrates the advantage of RISE in MHQA tasks and its effectiveness in conventional QA tasks, further validating its generalizability.

4.2.3 Self-Critique Capability
To evaluate the changes in the model’s self-critique capability, we designed a third set of experiments. In this experiment, both our model and GPT-4o assess the same set of decomposition results, with GPT-4o serving as a reference. By analyzing the consistency between our model and GPT-4o evaluations, we measure the improvement in the model’s self-critique capability. As shown in Table 3, the consistency between our model and GPT-4o steadily increases with each iteration. This indicates that the iterative process in RISE effectively enhances the model’s self-criticism capability. (For more experiment details see Appendix A.2.3.)

4.3 Ablation Study
To evaluate the impact of each synthesized training dataset on the model’s performance, we conduct an ablation study. As shown in Table 4, the experiment uses the same three MHQA datasets as before and the three training datasets generated in the round1, with accuracy as the primary evaluation metric.

Removing the question decomposition dataset leads to accuracy drop of 3.5% on 2Wiki, highlighting its importance in enabling effective multi-hop
<table>
  <tr>
    <th></th>
    <th colspan="3">Consistency with GPT-4o (%)</th>
  </tr>
  <tr>
    <th></th>
    <th>2WIKI</th>
    <th>HotpotQA</th>
    <th>MSQ</th>
  </tr>
  <tr>
    <td>Round 1</td>
    <td>74.30</td>
    <td>64.70</td>
    <td>60.00</td>
  </tr>
  <tr>
    <td>Round 2</td>
    <td>72.67</td>
    <td>66.30</td>
    <td>76.00</td>
  </tr>
  <tr>
    <td>Round 3</td>
    <td>79.67</td>
    <td>77.33</td>
    <td><b>79.33</b></td>
  </tr>
  <tr>
    <td>Round 4</td>
    <td><b>80.67</b></td>
    <td><b>79.33</b></td>
    <td>78.00</td>
  </tr>
</table>

Table 3: Consistency analysis with GPT-4o on each datasets. The results show progressive improvements in consistency with GPT-4o, highlighting the model’s enhanced self-critique ability through iterative training.

<table>
  <tr>
    <th></th>
    <th>2WIKI</th>
    <th>Hotpot</th>
    <th>MSQ</th>
  </tr>
  <tr>
    <th></th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
  </tr>
  <tr>
    <td>w/o Decomp</td>
    <td>37.63</td>
    <td>33.89</td>
    <td>11.08</td>
  </tr>
  <tr>
    <td>w/o R-t-R</td>
    <td>40.59</td>
    <td>33.06</td>
    <td>9.46</td>
  </tr>
  <tr>
    <td>w/o Critique</td>
    <td>38.98</td>
    <td>33.89</td>
    <td>10.27</td>
  </tr>
  <tr>
    <td>Separate</td>
    <td>40.86</td>
    <td>34.72</td>
    <td>10.54</td>
  </tr>
  <tr>
    <td>RISE</td>
    <td><b>41.13</b></td>
    <td><b>35.83</b></td>
    <td><b>11.89</b></td>
  </tr>
</table>

Table 4: Ablation study on 2WIKI, HotpotQA, and MSQ, showing the impact of removing individual tasks (<i>Question Decomposition</i>, <i>Retrieve-then-Read</i>, and <i>Self-Critique</i>) and comparing joint training (<i>RISE</i>) with separate training (<i>Separate</i>) of individual tasks.

reasoning. Excluding the retrieve-then-read dataset causes accuracy declines on HotpotQA (2.77%) and Musique (2.43%), highlighting the importance of this dataset in synthesizing evidence from diverse sources to mitigate the impact of noise. The removal of the self-critique dataset results in consistent accuracy reductions across all three datasets, emphasizing its pivotal function in refining reasoning paths processes. These results demonstrate the complementary and indispensable contributions of the question decomposition, retrieve-then-read, and self-critique datasets to the model’s performance.

Furthermore, we conduct separate training for the three tasks (Separate), where three LLMs are individually trained for decomposition, retrieve-then-read, and self-critique tasks. Compared to joint training (RISE), the accuracy of separate training is consistently lower across all datasets.

5 Related Works

Multi-hop Question Answering: MHQA tasks address questions that require integrating information from multiple sources and performing multi-step reasoning to produce a complete answer (Zhang et al., 2024; Li and Du, 2023). Question decomposition has been a pivotal approach for understanding and solving multi-hop questions, some works (Wei et al., 2022a; Wang et al., 2023b; Zhou et al., 2023; Shi et al., 2024b) leverage LLMs to divide complex questions into simpler single-hop sub-questions that are solved sequentially. Self-Ask (Press et al., 2023) uses LLMs to generate and resolve follow-up sub-questions with an external search engine. However, the effectiveness of these approaches depends on LLM’s inherent question decomposition capability, which constrained by hallucinations.

Retrieval-Augmented Generation for MHQA: RAG (Guu et al., 2020; Lewis et al., 2020; Izacard et al., 2022; Nakano et al., 2021; Asai et al., 2023; Ma et al., 2023; Yu et al., 2024; Shi et al., 2024a) integrates retrieval with generation to solve knowledge-intensive tasks (Zhu et al., 2024; Feng et al., 2024). The original RAG framework excels at single-hop QA but faces significant challenges in handling multi-hop QA and complex reasoning tasks (Lewis et al., 2020; Xu et al., 2024).

To address these challenges, various methods have been proposed. Chain of Thought (CoT) (Wei et al., 2022b) and Tree of Thought (ToT) (Yao et al., 2024) are integrated with RAG to enable multi-step reasoning and iterative retrieval (Press et al., 2023; Yao et al., 2023; Zhou et al., 2023; Khattab et al., 2023), allowing the model to incorporate a broader range of external knowledge and improve its reasoning capabilities. However, existing retrieval-augmented systems are inevitably affected by the limitations of retrievers, often introducing irrelevant or noisy information (Yin et al., 2023; Xu et al., 2024; Ma et al., 2023). Enhancing the model’s reasoning capabilities to filter noise and focus on critical evidence is essential for accurate summaries, which our method achieves through reasoning decomposition, improving both logical reasoning and QA performance.

Self-Improvement in Large Language Models: Self-improvement refers to the process by which models generate and utilize their own output data to enhance performance (Zelikman et al., 2024; Singh et al., 2024; Gülçehre et al., 2023). Existing approaches, such as self-training (Du et al., 2021) and self-play (Yuan et al., 2024; Chen et al., 2024), leverage pseudo-label generation and iterative policy optimization to improve the utilization of unlabeled data and enhance decision-making capabilities. Self-Rewarding (Yuan et al., 2024) employs the LLM-as-Judge paradigm to strengthen reasoning abilities, while Self-Refine (Madaan et al., 2024) iteratively optimizes generated out-
puts through self-feedback mechanisms.

In complex tasks like code generation and agent-based learning, self-improvement proves effective. Methods such as Self-Evolve (Jiang et al., 2023), NErT (Ni et al., 2024), and AutoAct (Qiao et al., 2024) leverage self-feedback, self-guided tracking, and self-planning to enhance performance. However, the application of self-iterative techniques in RAG scenarios remains underexplored. Our method addresses this gap by integrating self-exploration into RAG to generate diverse training data, enabling continuous model evolution and enhancing performance in complex tasks.

6 Conclusion

We propose RISE, a framework that addresses two key errors in MHQA tasks: Evidence Aggregation and Reasoning Decomposition. Through self-exploration, RISE continuously enhances reasoning capabilities. Additionally, RISE integrates self-iterative paradigm with RAG framework, bridging the gap in applying self-iterative strategies to MHQA scenarios without requiring manual intervention or reliance on large models, thereby offering a cost-effective solution. Experimental results on MHQA benchmarks demonstrate significant improvements in reasoning accuracy and task performance, highlighting RISE’s robustness and adaptability in tackling complex reasoning challenges.

Limitation

While RISE achieves strong performance in complex reasoning tasks, there remain opportunities for further enhancement. The current framework relies on external retrieval mechanisms without explicit optimization, which may limit the quality of evidence for downstream reasoning. Future work could explore self-improvement across the entire pipeline—spanning question decomposition, retrieval, generation, and reflection—to achieve more seamless integration and efficiency.

References

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research.

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. In Forty-first International Conference on Machine Learning.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.

Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. 2021. Self-training improves pre-training for natural language understanding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5408–5418, Online. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.

Zhangyin Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, and Bing Qin. 2024. Retrieval-generation synergy augmented large language models. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11661–11665.

Çağlar Gülçehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen
Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. CoRR.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625. Barcelona, Spain (Online). International Committee on Computational Linguistics.

Gautier Izacard, Patrick Lewis, Maria Lomeli, Luqas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.

Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. Self-evolve: A code evolution framework via large language models. Preprint, arXiv:2306.02907.

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551.

Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2023. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. Preprint, arXiv:2212.14024.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Ilia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Ruosen Li and Xinya Du. 2023. Leveraging structured information for explainable multi-hop question answering and reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6779–6789. Singapore. Association for Computational Linguistics.

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.

Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. Webglm: Towards an efficient web-enhanced question answering system with human preferences. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4549–4560.

Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. Preprint, arXiv:2308.09583.

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303–5315, Singapore. Association for Computational Linguistics.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.

Ansong Ni, Militiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. Next: Teaching large language models to reason about code execution. In Forty-first International Conference on Machine Learning.

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711.

Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Huajun Chen, et al. 2024. Autoact: Automatic agent learning from scratch for qa via self-planning. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.

Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922.

Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren.
2024a. Generate-then-ground in retrieval-augmented generation for multi-hop question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7339–7353.

Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. 2024b. Generate-then-ground in retrieval-augmented generation for multi-hop question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7339–7353, Bangkok, Thailand. Association for Computational Linguistics.

Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073.

Avi Singh, John D Co-Reyes, and Rishabh Agarwal. 2024. Beyond human data: Scaling self-training for problem-solving with language models. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashykhov, Soumya Batra, Prajwali Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Harsh Trivedi, Niranjana Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome-based feedback. Preprint, arXiv:2211.14275.

Tianduo Wang, Shichen Li, and Wei Lu. 2024. Self-training with direct preference optimization improves chain-of-thought reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11917–11928.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023a. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023c. Self-instruct: Aligning language models with self-generated instructions. In The 61st Annual Meeting Of The Association For Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022a. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2024. Search-in-the-chain: Interactively enhancing large language models with search for knowledge-intensive tasks. In Proceedings of the ACM Web Conference 2024, WWW ’24, page 1362–1373, New York, NY, USA. Association for Computing Machinery.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.

Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. 2023. Cognitive mirage: A review of hallucinations in large language models. arXiv preprint arXiv:2309.06794.

Xunjian Yin, Baizhou Huang, and Xiaojun Wan. 2023. ALCUNA: Large language models meet new knowledge. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1397–1414, Singapore. Association for Computational Linguistics.

Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, S Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context
generators. In International Conference on Learning Representations.

Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Rankrag: Unifying context ranking with retrieval-augmented generation in llms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In Forty-first International Conference on Machine Learning.

Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. 2024. Self-taught optimizer (stop): Recursively self-improving code generation. In OPT 2023: Optimization for Machine Learning.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glim-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations.

Jiahao Zhang, Haiyang Zhang, Dongmei Zhang, Liu Yong, and Shen Huang. 2024. End-to-end beam retrieval for multi-hop question answering. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1718–1731, Mexico City, Mexico. Association for Computational Linguistics.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher Deenan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations.

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. 2024. Large language models for information retrieval: A survey. Preprint, arXiv:2308.07107.
A Appendix

A.1 Prompts

A.1.1 Self-Exploration Prompts
We designed detailed prompts for the three tasks in the self-exploration phase: question decomposition (Figure 6), retrieve-then-read(Figure 7), and self-critique(Figure 8). The examples used in the decomposition prompt are inspired by self-ask (Press et al., 2023).

A.1.2 Self-Decomposition Evaluation Prompt
In this paper, the evaluation of the question decomposition capability is conducted using GPT-4o with prompt as shown in Figure 9. The analysis involves assessing and scoring the decomposition results of different iterations across multiple dimensions, ultimately leading to a comparative analysis of the two models. The dimensions of the analysis include:

• Conciseness: Whether the decomposition avoids redundancy while ensuring comprehensiveness.

• Rationality: Whether the decomposed sub-problems are closely related to the original problem.

• Sequencing: Whether the decomposition of sub-problems follows a logical order and facilitates the problem-solving process.

• Goal Orientation: Whether the decomposition is clearly centered around addressing the main problem’s objective. Are the sub-problems closely aligned with the core goal of the main problem? Does it avoid redundant issues that deviate from the primary objective?

A.2 Experiment detail

A.2.1 Implementation Details
We conduct all experiments on a server equipped with four NVIDIA A800 80G GPUs. For the experimental setup, we use the following hyperparameters: learning rate of \( 1 \times 10^{-4} \), batch size of 64,and cut-off length of 8192. Furthermore, for the weighting parameters \( \alpha, \beta, \) and \( \gamma \) in the overall loss function, values of 1 are uniformly adopted in this research.

<table>
  <tr>
    <th>Datasets</th>
    <th>\( D_d \)</th>
    <th>\( D_r \)</th>
    <th>\( D_c \)</th>
  </tr>
  <tr>
    <td>Round 1</td>
    <td>3276</td>
    <td>2501</td>
    <td>3925</td>
  </tr>
  <tr>
    <td>Round 2</td>
    <td>8309</td>
    <td>6311</td>
    <td>8074</td>
  </tr>
  <tr>
    <td>Round 3</td>
    <td>4858</td>
    <td>2106</td>
    <td>2312</td>
  </tr>
  <tr>
    <td>Round 4</td>
    <td>6913</td>
    <td>4759</td>
    <td>5307</td>
  </tr>
</table>

Table 5: Number of samples accumulated in datasets \( D_d, D_r, \) and \( D_c \) after each round of self-exploration.

<table>
  <tr>
    <th></th>
    <th>2WIKI</th>
    <th>HotpotQA</th>
    <th>MSQ</th>
  </tr>
  <tr>
    <td>Round 1</td>
    <td>223</td>
    <td>194</td>
    <td>180</td>
  </tr>
  <tr>
    <td>Round 2</td>
    <td>218</td>
    <td>199</td>
    <td>228</td>
  </tr>
  <tr>
    <td>Round 3</td>
    <td>239</td>
    <td>232</td>
    <td>238</td>
  </tr>
  <tr>
    <td>Round 4</td>
    <td>242</td>
    <td>238</td>
    <td>234</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>300</td>
    <td>300</td>
    <td>300</td>
  </tr>
</table>

Table 6: Number of instances in each round’s self-critique capability evaluation that aligned with GPT-4o.

A.2.2 Datasets
The cold-start dataset \( Q^0 \) consists of 800 randomly sampled instances from the training sets of 2Wiki-MultiHopQA, HotpotQA, and MuSiQue, totaling 2,400 cold-start samples. Table 5 provides detailed information on the training datasets constructed during each round of self-exploration. The evaluation datasets we used 2WikiMultiHopQA, HotpotQA, and MuSiQue each contain 1,000 samples, Nature Questions, Web Questions, and TriviaQA each contain 400 samples.

A.2.3 Self-Critique Capability Experiments Details
To demonstrate the improvement in the self-critique capability of the model across iterations, we sampled 300 instances from the generated \( D_c \) at each round and compared them with GPT-4o. The responses from GPT-4o were used as ground truth to calculate the self-critique accuracy of our model. In Table 6, we present the number of instances in each round’s self-critique capability evaluation that aligned with GPT-4o.

A.3 Additional Experiments

A.3.1 RISE Robustness
To further verify the robustness of our experimental conclusions, we conducted additional experiments using the Qwen2.5-7B model. Specifically, we performed four rounds of self-exploration following the same experimental setup as in our original RISE framework. The results consistently demonstrate the effectiveness of RISE, with performance
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">2WIKI</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">MSQ</th>
  </tr>
  <tr>
    <th>Acc</th><th>F1</th><th>EM</th>
    <th>Acc</th><th>F1</th><th>EM</th>
    <th>Acc</th><th>F1</th><th>EM</th>
  </tr>
  <tr>
    <td>Round 0</td><td>38.98</td><td>23.28</td><td>12.10</td><td>31.94</td><td>21.72</td><td>10.28</td><td>7.30</td><td>8.98</td><td>2.70</td>
  </tr>
  <tr>
    <td>Round 1</td><td>45.97</td><td>46.49</td><td>32.50</td><td>35.83</td><td>44.55</td><td>32.50</td><td>10.54</td><td>18.65</td><td>8.92</td>
  </tr>
  <tr>
    <td>Round 2</td><td>46.24</td><td>46.33</td><td>30.83</td><td>36.11</td><td>41.58</td><td>30.00</td><td>11.89</td><td>16.19</td><td>8.92</td>
  </tr>
  <tr>
    <td>Round 3</td><td><b>47.31</b></td><td><b>47.10</b></td><td><b>34.41</b></td><td><b>38.33</b></td><td>42.82</td><td>31.39</td><td><b>13.24</b></td><td><b>20.70</b></td><td><b>10.81</b></td>
  </tr>
</table>

Table 7: Performance of the Qwen2.5-7B model after four rounds of self-exploration on different datasets, showing improvements in accuracy, F1, and EM scores across 2WIKI, HotpotQA, and MSQ.

<table>
  <tr>
    <th>Methods</th>
    <th>Average Input Token</th>
    <th>Average Output Tokens</th>
    <th>Number of LLMs Calls</th>
  </tr>
  <tr>
    <td>Naive LLM</td>
    <td>18</td>
    <td>244</td>
    <td>1</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>24</td>
    <td>284</td>
    <td>1</td>
  </tr>
  <tr>
    <td>CoT-SC</td>
    <td>240</td>
    <td>2840</td>
    <td>10</td>
  </tr>
  <tr>
    <td>GenRead</td>
    <td>307</td>
    <td>233</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Naive RAG</td>
    <td>440</td>
    <td>104</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Self-Ask</td>
    <td>782</td>
    <td>172</td>
    <td>2</td>
  </tr>
  <tr>
    <td>WebGLM</td>
    <td>436</td>
    <td>103</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Self-RAG</td>
    <td>866</td>
    <td>294</td>
    <td>2</td>
  </tr>
  <tr>
    <td>RRR</td>
    <td>496</td>
    <td>86</td>
    <td>2</td>
  </tr>
  <tr>
    <td>GenGround</td>
    <td>2449</td>
    <td>167</td>
    <td>5</td>
  </tr>
  <tr>
    <td>RISE</td>
    <td>2881</td>
    <td>192</td>
    <td>5</td>
  </tr>
</table>

Table 8: Token consumption comparison between RISE and other baseline methods, showing average input tokens, average output tokens, and the number of LLMs calls required for each approach. RISE demonstrates a higher input token consumption due to its multi-step reasoning, but maintains efficient reasoning performance.

improvements observed across multiple datasets after each iteration. This confirms that RISE maintains strong generalization capabilities and stable performance even when applied to different large language models.

A.3.2 Token Consumption Details
In addition to performance evaluation, we analyzed the token consumption of RISE compared to other baseline methods. We measured both the average input token consumption and the average output token length, as well as the number of model calls required in each approach. The results reveal that while RISE consumes more input tokens due to its multi-step reasoning process, it achieves higher efficiency in output generation and overall reasoning effectiveness. This analysis highlights the trade-off between token usage and model performance, demonstrating that RISE achieves a balanced optimization in complex reasoning tasks.

A.3.3 The necessity of multiple iterations
To further validate the necessity and effectiveness of our multi-round training strategy, we conduct additional experiments comparing single-round and multi-round training setups on three benchmark datasets: 2Wiki, HotpotQA, and MSQ. The results are summarized in Table ??.

These results demonstrate that multi-round training significantly improves accuracy (Acc) and exact match (EM) metrics across all datasets, highlighting the advantage of iterative self-exploration over static single-round training. Additionally, we compare joint training with alternating training strategies to clarify their differences. As shown in Table ??, joint training better preserves learned capabilities and achieves performance gains through synergistic task-chain interactions.

A.3.4 Reliability Analysis of Self-Evaluation Mechanism
We further investigate the reliability of the model’s self-critique mechanism by evaluating the alignment between self-assessment and actual correctness. Table ?? presents the distribution of cases where the model’s self-judgment matches or mismatches the ground truth for both the baseline GPT-4o and our method.

These results confirm a positive correlation between the model’s self-assessment and actual answer correctness, demonstrating the effective-
<table>
  <tr>
    <th rowspan="2">Setup</th>
    <th colspan="3">2Wiki</th>
    <th colspan="3">Hotpot</th>
    <th colspan="3">MSQ</th>
  </tr>
  <tr>
    <th>Acc</th><th>F1</th><th>EM</th>
    <th>Acc</th><th>F1</th><th>EM</th>
    <th>Acc</th><th>F1</th><th>EM</th>
  </tr>
  <tr>
    <td>Single-round</td>
    <td>41.10</td><td>43.32</td><td>30.60</td>
    <td>34.10</td><td>43.28</td><td>29.50</td>
    <td>11.10</td><td>17.78</td><td>8.60</td>
  </tr>
  <tr>
    <td>Multi-round</td>
    <td><b>49.40</b></td><td>43.01</td><td><b>32.70</b></td>
    <td><b>40.50</b></td><td>42.00</td><td><b>30.50</b></td>
    <td><b>21.70</b></td><td><b>22.87</b></td><td><b>11.70</b></td>
  </tr>
</table>

Table 9: Comparison between single-round and multi-round training on three datasets.

ness and reliability of the proposed self-evaluation mechanism.

A.4 Supply Metrics of Main Results
This section provides additional details to supplement the main results, including comprehensive Exact Match (EM) and F1 scores across six QA datasets: 2WikiMultiHopQA, HotpotQA, MuSiQue, Natural Questions, Web Questions, and TriviaQA. We compare RISE (Ours) with both prompting-based and fine-tuning-based methods, under settings with and without retrieval. The results offer a deeper understanding of RISE’s performance, highlighting its consistent improvements over baseline models.

A.5 Case Study
To further illustrate the design of self-exploration process, we present representative case study. Each task is marked with a task-specific prefix, and follows a carefully curated instruction template.(Shown in Figure 11, 12, 13, 14, 15, 16)
<table>
  <tr>
    <th>Model</th>
    <th>Correct and judged Correct</th>
    <th>Correct but judged Wrong</th>
    <th>Wrong but judged Correct</th>
    <th>Wrong and judged Wrong</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>32.58%</td>
    <td>9.71%</td>
    <td>11.21%</td>
    <td>46.50%</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>28.65%</td>
    <td>7.58%</td>
    <td>23.43%</td>
    <td>40.33%</td>
  </tr>
</table>

Table 10: Alignment between model self-judgment and ground truth correctness on initial question set \( Q_0 \).

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Model</th>
    <th colspan="3">MHQA</th>
    <th colspan="2">SHQA</th>
    <th rowspan="2">Trival</th>
  </tr>
  <tr>
    <th>2WIKI Hotpot</th>
    <th>MSQ</th>
    <th>NQ</th>
    <th>WebQ</th>
  </tr>
  <tr>
    <td colspan="8"><i>w/o retrieval</i></td>
  </tr>
  <tr>
    <td>Naive LLM</td>
    <td>LLaMA-3.1-8B</td>
    <td>0.00</td>
    <td>0.30</td>
    <td>0.00</td>
    <td>0.25</td>
    <td>0.00</td>
    <td>2.25</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-3.5-turbo</td>
    <td>0.50</td>
    <td>5.50</td>
    <td>0.20</td>
    <td>2.50</td>
    <td>0.75</td>
    <td>26.75</td>
  </tr>
  <tr>
    <td>CoT (Wei et al., 2022b)</td>
    <td>LLaMA-3.1-8B</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>CoT-SC* (Wang et al., 2023a)</td>
    <td>GPT-3.5-turbo</td>
    <td>18.40</td>
    <td>31.90</td>
    <td>9.60</td>
    <td>30.75</td>
    <td>22.75</td>
    <td>71.75</td>
  </tr>
  <tr>
    <td>GenRead (Yu et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>17.00</td>
    <td>25.60</td>
    <td>7.60</td>
    <td>35.75</td>
    <td>25.50</td>
    <td>61.75</td>
  </tr>
  <tr>
    <td colspan="8"><i>w retrieval</i></td>
  </tr>
  <tr>
    <td>Naive RAG</td>
    <td>LLaMA-3.1-8B</td>
    <td>0.30</td>
    <td>1.70</td>
    <td>0.20</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>2.25</td>
  </tr>
  <tr>
    <td>Self-Ask (Press et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>17.00</td>
    <td>25.60</td>
    <td>9.60</td>
    <td>28.50</td>
    <td>21.50</td>
    <td>58.75</td>
  </tr>
  <tr>
    <td>WebGLM (Liu et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>0.00</td>
    <td>0.30</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Self-RAG* (Asai et al., 2023)</td>
    <td>LLaMA2-7B</td>
    <td>2.90</td>
    <td>3.90</td>
    <td>0.80</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>2.00</td>
  </tr>
  <tr>
    <td></td>
    <td>LLaMA2-13B</td>
    <td>3.40</td>
    <td>2.50</td>
    <td>0.40</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>4.50</td>
  </tr>
  <tr>
    <td>RRR (Ma et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>0.00</td>
    <td>0.40</td>
    <td>0.00</td>
    <td>0.75</td>
    <td>0.75</td>
    <td>2.75</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-3.5-turbo</td>
    <td>3.20</td>
    <td>2.20</td>
    <td>0.30</td>
    <td>2.50</td>
    <td>2.00</td>
    <td>20.00</td>
  </tr>
  <tr>
    <td>GenGround (Shi et al., 2024a)</td>
    <td>LLaMA-3.1-8B</td>
    <td>23.50</td>
    <td>24.30</td>
    <td>10.20</td>
    <td>20.50</td>
    <td>18.50</td>
    <td>60.25</td>
  </tr>
  <tr>
    <td><b>RISE(Ours)</b></td>
    <td>LLaMA-3.1-8B</td>
    <td>32.70</td>
    <td>30.50</td>
    <td>11.70</td>
    <td>28.50</td>
    <td>19.25</td>
    <td>59.50</td>
  </tr>
</table>

Table 11: <b>EM</b> metrics of RISE with other methods on the 2WikiMultiHopQA, HotpotQA, MuSiQue, Natural Questions, Web Questions and TriviaQA. Methods marked with asterisk (*) involve specific considerations: CoT-SC uses GPT-3.5 due to LLaMA-3.1’s limitations in adhering to instructions, and Self-RAG employs publicly released model weights because its dataset is unavailable. All other methods are reproduced with LLaMA-3.1-8B. <span style="background-color:#e0f2d4;">green</span> represent Prompting-based Methods, while <span style="background-color:#ffffcc;">yellow</span> represent Finetuning-based Methods.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Model</th>
    <th colspan="3">MHQA</th>
    <th colspan="2">SHQA</th>
    <th rowspan="2">Trival</th>
  </tr>
  <tr>
    <th>2WIKI Hotpot</th>
    <th>MSQ</th>
    <th>NQ</th>
    <th>WebQ</th>
  </tr>
  <tr>
    <td colspan="8"><i>w/o retrieval</i></td>
  </tr>
  <tr>
    <td>Naive LLM</td>
    <td>LLaMA-3.1-8B</td>
    <td>7.99</td>
    <td>5.49</td>
    <td>2.67</td>
    <td>4.09</td>
    <td>5.46</td>
    <td>17.55</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-3.5-turbo</td>
    <td>13.38</td>
    <td>17.66</td>
    <td>8.10</td>
    <td>16.53</td>
    <td>19.14</td>
    <td>44.11</td>
  </tr>
  <tr>
    <td>CoT (Wei et al., 2022b)</td>
    <td>LLaMA-3.1-8B</td>
    <td>2.95</td>
    <td>2.31</td>
    <td>1.53</td>
    <td>1.77</td>
    <td>2.19</td>
    <td>4.93</td>
  </tr>
  <tr>
    <td>CoT-SC* (Wang et al., 2023a)</td>
    <td>GPT-3.5-turbo</td>
    <td>24.31</td>
    <td>39.59</td>
    <td>15.36</td>
    <td>40.98</td>
    <td>36.54</td>
    <td>80.65</td>
  </tr>
  <tr>
    <td>GenRead (Yu et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>22.39</td>
    <td>34.51</td>
    <td>15.26</td>
    <td>47.90</td>
    <td>42.55</td>
    <td>69.59</td>
  </tr>
  <tr>
    <td colspan="8"><i>w retrieval</i></td>
  </tr>
  <tr>
    <td>Naive RAG</td>
    <td>LLaMA-3.1-8B</td>
    <td>6.39</td>
    <td>9.60</td>
    <td>4.02</td>
    <td>9.02</td>
    <td>9.72</td>
    <td>19.37</td>
  </tr>
  <tr>
    <td>Self-Ask (Press et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>23.42</td>
    <td>36.14</td>
    <td>19.67</td>
    <td>40.35</td>
    <td>39.61</td>
    <td>67.02</td>
  </tr>
  <tr>
    <td>WebGLM (Liu et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>8.27</td>
    <td>6.03</td>
    <td>3.76</td>
    <td>5.57</td>
    <td>6.68</td>
    <td>9.30</td>
  </tr>
  <tr>
    <td>Self-RAG* (Asai et al., 2023)</td>
    <td>LLaMA2-7B</td>
    <td>17.38</td>
    <td>15.44</td>
    <td>6.02</td>
    <td>25.56</td>
    <td>21.45</td>
    <td>11.34</td>
  </tr>
  <tr>
    <td></td>
    <td>LLaMA2-13B</td>
    <td>14.82</td>
    <td>13.41</td>
    <td>6.70</td>
    <td>15.65</td>
    <td>9.34</td>
    <td>15.34</td>
  </tr>
  <tr>
    <td>RRR (Ma et al., 2023)</td>
    <td>LLaMA-3.1-8B</td>
    <td>5.35</td>
    <td>3.21</td>
    <td>1.48</td>
    <td>4.46</td>
    <td>6.68</td>
    <td>9.56</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-3.5-turbo</td>
    <td>13.70</td>
    <td>16.76</td>
    <td>6.08</td>
    <td>22.80</td>
    <td>23.16</td>
    <td>43.68</td>
  </tr>
  <tr>
    <td>GenGround (Shi et al., 2024a)</td>
    <td>LLaMA-3.1-8B</td>
    <td>34.33</td>
    <td>34.53</td>
    <td>19.81</td>
    <td>36.29</td>
    <td>28.93</td>
    <td>67.20</td>
  </tr>
  <tr>
    <td><b>RISE(Ours)</b></td>
    <td>LLaMA-3.1-8B</td>
    <td>43.01</td>
    <td>42.00</td>
    <td>22.87</td>
    <td>42.97</td>
    <td>36.22</td>
    <td>72.70</td>
  </tr>
</table>

Table 12: <b>F1</b> metrics of RISE with other methods on the 2WikiMultiHopQA, HotpotQA, MuSiQue, Natural Questions, Web Questions and TriviaQA. Methods marked with asterisk (*) involve specific considerations: CoT-SC uses GPT-3.5 due to LLaMA-3.1’s limitations in adhering to instructions, and Self-RAG employs publicly released model weights because its dataset is unavailable. All other methods are reproduced with LLaMA-3.1-8B. <span style="background-color:#e0f2d4;">green</span> represent Prompting-based Methods, while <span style="background-color:#ffffcc;">yellow</span> represent Finetuning-based Methods.
Decomposition Prompt

Instruction: Please answer the following questions according to the given format. Strictly follow each format specification, as this will ensure consistency and clarity in your response.

- Only add follow-up questions if additional details are needed to arrive at the final answer.
- For each follow-up question, use exactly this format: 'Follow up: question'
- Ensure each follow-up question is direct and structured to be easily searchable, focusing on key information for efficient search engine retrieval.
- For each answer to a follow-up question, use exactly this format: 'Intermediate answer: answer'
- Do not repeat or alter any previously generated follow-up questions or intermediate answers.
- Conclude with the final answer using this exact format: 'So the final answer is: final answer' if no further questions are needed.

Use the examples below to understand the expected structure, and follow this format without deviating from these instructions.

Question: Who lived longer, Muhammad Ali or Alan Turing?
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali.

Question: When was the founder of craigslist born?
Are follow up questions needed here: Yes.
Follow up: Who was the founder of craigslist?
Intermediate answer: Craigslist was founded by Craig Newmark.
Follow up: When was Craig Newmark born?
Intermediate answer: Craig Newmark was born on December 6, 1952.
So the final answer is: December 6, 1952.
...
—
Now, **continue the response** using the following question and information provided below.
Only add follow-up questions if necessary to reach the final answer.
**Ensure all follow-up questions are optimized for search engine queries, making each question concise, direct, and easily searchable. Avoid modifying or repeating any existing content.**
—
Question (ORIGINAL): {question}
Are follow up questions needed here: Yes.
Retrieve-then-Read Prompt

## Question-Answering-in-Reference-Task ##

Instruction:
- Use the references provided to answer the question as specifically and completely as possible.
- If the references do not directly answer the question, combine relevant information from multiple references to create a well-supported answer.
- When references are Null or insufficient, use your own knowledge to provide a clear and relevant answer.
- When a direct answer cannot be determined, list any information in the references that could be relevant or provide partial insights related to the question. Avoid responses such as 'I don’t know' or 'more information is needed.'
- Always prioritize specificity and relevance in your answer, providing helpful context or details that approach a complete answer.

Reference [1]
Reference [2]
...
Question: {question}

Figure 7: Retrieve-then-Read prompt template.
Self-Critique Prompt

## Critique-Task ##
Main Question: {question}
Below is a list of previously generated subquestions and their intermediate answers, created as part of a multi-step reasoning process to answer the main question.
Your task is to evaluate whether the information in the current subquestion is necessary and contributes incrementally towards solving the main question.

Previously generated subquestions and answers:
{previous subquestions}

Current subquestion and answer candidate:
{subquestion and intermediate answer }

Instruction:
- Step 1: Check for Redundancy. Check if the current subquestion or answer repeats information already provided in previous subquestions. If it does, return 'flag = False' as this information is redundant.
- Step 2: Assess Relevance. If the information is not a duplicate, analyze its relevance to the main question. Determine whether it provides new, relevant information that helps move closer to solving the main question, even if it only provides indirect context or background.
Note that information does not need to directly answer the main question to be considered relevant; it can also support understanding or provide necessary context. Mark it as 'flag = True'.
- Step 3: Based on your analysis, provide a final judgment in the following format:

**Final Judgment**: [flag = True or flag = False]

Examples:

Main Question: “Who lived longer, Muhammad Ali or Alan Turing?”
• Follow up: “How old was Muhammad Ali when he died?” (Flag = True, relevant for lifespan comparison.)
• Follow up: “How old was Alan Turing when he died?” (Flag = True, completes lifespan comparison.)
• Redundant Example: “How old was Muhammad Ali when he passed?” (Flag = False, redundant with earlier subquestion.)

Main Question: “Are both the directors of Jaws and Casino Royale from the same country?”
• Follow up: “Who directed Jaws?” (Flag = True, needed for director identification.)
• Follow up: “Where is Steven Spielberg from?” (Flag = True, relevant to nationality check.)
• Irrelevant Example: “What is Steven Spielberg’s favorite genre?” (Flag = False, not relevant to nationality.)

Reminder: Use “flag = True” for any subquestion that provides useful information or context toward solving the main question, even if indirectly. Set “flag = False” only if it is redundant or entirely irrelevant.

Figure 8: Self-Critique prompt template.
Self-Decomposition Evaluation Prompt

You are given two problem decomposition results for the same complex problem. Your task is to compare these results from Conciseness, Rationality, Sequencing and Goal Orientation. Analyze the two decomposition results using the criteria above. Clearly explain which approach is more effective for solving the problem and why, while highlighting the strengths and weaknesses of each approach in detail.

# Scoring Criteria:
- Score each dimension on a scale of 1-5, where:
  - 1: Poor
  - 2: Needs Improvement
  - 3: Average
  - 4: Good
  - 5: Excellent
# The output follows the format below. Do not add any additional text: {
"Conciseness": {
"Result 1 Score": X,
"Result 2 Score": Y,
"Explanation": "How effectively does each decomposition avoid unnecessary complexity while still addressing all relevant aspects of the problem? Is the explanation clear and straightforward?"
},
"Rationality": {
"Result 1 Score": X,
"Result 2 Score": Y,
"Explanation": "Are the identified components logical and directly related to the problem? Do the solutions align well with the identified components?"
},
"Sequencing": {
"Result 1 Score": X,
"Result 2 Score": Y,
"Explanation": "Is the order of steps or components logical and easy to follow? Does the sequence facilitate efficient problem-solving?"
},
"Goal Orientation": {
"Result 1 Score": X,
"Result 2 Score": Y,
"Explanation": "Do the sub-questions stay aligned with the core goal of the main problem? Are there any redundant sub-questions that deviate from the primary objective?"
},
"Result": "Decomposition Results 1 Decomposition Results 2 Tie"
}
# Problem:
{problem} # Decomposition Results to Compare:
- Decomposition Results 1:
{result1}
- Decomposition Results 2:
{result2} # Output:

Figure 9: GPT-4o decomposition prompt template.
Multi-Hop Question Generation Prompt

#Multi-Hop-Question-Generation-in-{Task}#

Instruction:
- You are an AI assistant tasked with generating multi-hop questions similar to those in the {task} dataset.
- These questions require combining multiple pieces of information to reach the answer.
- Typically, these questions involve indirect references and nested relationships.

Examples from the {task} dataset:
Example 1: Question: [Insert Example Here]
Example 2: Question: [Insert Example Here]
Example 3: Question: [Insert Example Here]
Example 4: Question: [Insert Example Here]
Example 5: Question: [Insert Example Here]
Example 6: Question: [Insert Example Here]
Example 7: Question: [Insert Example Here]
Example 8: Question: [Insert Example Here]

Please generate a **new and unique multi-hop question** that meets the following criteria:
- **Requires reasoning across multiple facts or entities**.
- **Asks for only one piece of information or answer** without multiple sub-questions.
- **Is a single, coherent question** that requires multi-step reasoning to answer.
- **Includes indirect references or nested relationships**.
- **Matches the complexity and structure** of questions in the {task} dataset.
- **Is concise** (one sentence) and clearly worded.
- **Covers new topics** or involves different entities from the examples above.
- **Avoids being a duplicate** of the examples above.
- **Avoids using conjunctions like 'and', 'or', or commas** to ask more than one thing.

Your response should **only contain the generated question** without any extra text and follow the format:
Question: {question}

Figure 10: Multi-hop question generation prompt template.
Case Study: Question Decomposition Input

## Question-Decomposition-Task ##

Please answer the following questions according to the given format. Strictly follow each format specification, as this will ensure consistency and clarity in your response.
- Only add follow-up questions if additional details are needed to arrive at the final answer.
- For each follow-up question, use exactly this format: 'Follow up: question'
- Ensure each follow-up question is direct and structured to be easily searchable, focusing on key information for efficient search engine retrieval.
- For each answer to a follow-up question, use exactly this format: 'Intermediate answer: answer'
- Do not repeat or alter any previously generated follow-up questions or intermediate answers.
- Conclude with the final answer using this exact format: 'So the final answer is: final answer' if no further questions are needed.

Use the examples below to understand the expected structure, and follow this format without deviating from these instructions.

Question: Who lived longer, Muhammad Ali or Alan Turing?
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali. ...

–
Now, continue the response using the following question and information provided below.
Only add follow-up questions if necessary to reach the final answer.
Ensure all follow-up questions are optimized for search engine queries, making each question concise, direct, and easily searchable. Avoid modifying or repeating any existing content.

–
Question: Are the directors of both films Inter Nos and La Bandera (Film) from the same country?
Are follow up questions needed here:

Figure 11: Example of Question Decomposition task input.

Case Study: Question Decomposition Output

Yes.
Follow up: Who is the director of Inter Nos?

Figure 12: Example of Question Decomposition task output.
Case Study: Retrieve-then-Read Input

## Question-Answering-in-Reference-Task ##

Instruction: - Use the references provided to answer the question as specifically and completely as possible. - If the references do not directly answer the question, combine relevant information from multiple references to create a well-supported answer. - When references are Null or insufficient, use your own knowledge to provide a clear and relevant answer. - When a direct answer cannot be determined, list any information in the references that could be relevant or provide partial insights related to the question. Avoid responses such as 'I don't know' or 'more information is needed.' - Always prioritize specificity and relevance in your answer, providing helpful context or details that approach a complete answer.

Reference [1]: Congratulations to the four Directors of Inter Nos...
Reference [2]: ...
...(More retrieve contents)

-
Question: Who is the director of Inter Nos?

Figure 13: Example of Retrieve-then-Read task input.

Case Study: Retrieve-then-Read Output

According to the references provided, the directors of Inter Nos include President Juan Carlos Martinez. Additionally, Cynthia Vaughn has been named Associate Editor of Inter Nos, and she will edit the publication's "Independent Voices" section. It is not specified in the references provided who the other directors of Inter Nos are.

Figure 14: Example of Retrieve-then-Read task output.
Case Study: Self-Critique Input

## Critique-Task ##
Main Question: Are the directors of both films Inter Nos and La Bandera (Film) from the same country?

Below is a list of previously generated subquestions and their intermediate answers, created as part of a multi-step reasoning process to answer the main question. Your task is to evaluate whether the information in the current subquestion is necessary and contributes incrementally towards solving the main question.

Previously generated subquestions and answers: Null

Current subquestion and answer candidate:
Follow up: Who is the director of Inter Nos?
Intermediate answer: According to the references provided, the directors of Inter Nos include President Juan Carlos Martinez. Additionally, Cynthia Vaughn has been named Associate Editor of Inter Nos, and she will edit the publication’s “Independent Voices” section. It is not specified in the references provided who the other directors of Inter Nos are.

Instruction:
- Step 1: Check for Redundancy. Check if the current subquestion or answer repeats information already provided in previous subquestions. If it does, return ‘flag = False’ as this information is redundant.
- Step 2: Assess Relevance. If the information is not a duplicate, analyze its relevance to the main question. Determine whether it provides new, relevant information that helps move closer to solving the main question, even if it only provides indirect context or background. Note that information does not need to directly answer the main question to be considered relevant; it can also support understanding or provide necessary context. Mark it as ‘flag = True.
- Step 3: Based on your analysis, provide a final judgment in the following format:
**Final Judgment**: [flag = True or flag = False]

Examples:
Main Question: “Who lived longer, Muhammad Ali or Alan Turing?”
• Follow up: “How old was Muhammad Ali when he died?” (Flag = True, relevant for lifespan comparison.)
• Follow up: “How old was Alan Turing when he died?” (Flag = True, completes lifespan comparison.)
• Redundant Example: “How old was Muhammad Ali when he passed?” (Flag = False, redundant with earlier subquestion.)

Main Question: “Are both the directors of Jaws and Casino Royale from the same country?”
• Follow up: “Who directed Jaws?” (Flag = True, needed for director identification.)
• Follow up: “Where is Steven Spielberg from?” (Flag = True, relevant to nationality check.)
• Irrelevant Example: “What is Steven Spielberg’s favorite genre?” (Flag = False, not relevant to nationality.)

Reminder:
Use “flag = True” for any subquestion that provides useful information or context toward solving the main question, even if indirectly. Set “flag = False” only if it is redundant or entirely irrelevant.

Figure 15: Example of Self-Critique task input.

Case Study: Self-Critique Output

flag = True.

Figure 16: Example of Self-Critique task output.

Reviewer paper summaries: This paper introduces RISE, a framework designed to enhance reasoning capabilities in Multi-Hop Question Answering tasks through iterative self-exploration. RISE addresses two critical challenges in MHQA: Evidence Aggregation Errors and Reasoning Decomposition Errors. The framework combines RAG with a self-iterative process involving three core steps: question decomposition, retrieve-then-read, and self-critique. By continuously refining reasoning paths and integrating evidence, RISE autonomously improves model performance without relying on large-scale models or extensive manual annotations. Extensive experiments on multiple MHQA benchmarks demonstrate that RISE significantly outperforms existing methods, establishing its effectiveness in tackling complex reasoning tasks.

The paper introduces a framework for improving MHQA tasks by combining question decomposition, multi-step RAG, and self-critique that aim to identify errors. The authors describe the Self-Exploration mechanism which is the iterative process that in inference time aims to refine its reasoning by revisiting intermediate steps until the final answer. The authors also describe their iterative optimization method which is a multi-objective optimization step using all the outputs of the exploration part (decomposition, rag, and critique). Overall the authors show improvements in MHQA benchmarks compared to some other baselines
Summary of strengths: 1. The proposed RISE is clearly motivated.
2. Extensive experimental results demonstrate that RISE can consistently outperform baseline models.

- The Self-Critique with the iterative optimization introduces an interesting method for training the  "critique mechanism" for MHQA.  I am not aware of other works that combine it together with other multi-step reasoning steps that include rag and question decomposition.


- The paper employs diverse evaluation metrics, including both common objective  (accuracy, F1, EM) and subjective (conciseness, rationality, sequencing, and goal orientation) measures that employ llm-as-a-judge,  offering a (somewhat) comprehensive assessment of the model.
Summary of weaknesses: 1. The QA datasets used in the experiments are relatively small. Although the RISE model demonstrates promising results on these datasets, its scalability and performance on larger or more diverse QA datasets remain unclear.
2. The Self-Critique module may encounter situations where the critique process becomes stuck in loop nodes, continuously processing without converging to a solution.
3. While the overall loss function is mentioned in Line 252, the specific definitions and roles of each component loss, such as L_d , L_r , and L_c , which are not clearly explained.
4. The Rerank mechanism and the proposed Self-Critique module appear conceptually similar and what is the difference between them?

-- While this framework shows an improvement in performance over other baselines, ablation studies suggest that each step contributes very little, meaning that the overall increase in performance comes from the iterative optimization. It needs to be clear that iterative RAG or question decomposition are not novel, and critique on its own is also not novel. If their combination is not very helpful, then a question arises regarding its necessity.

-- Maybe I did not get it right, but is the iterative optimization iterating on the training set? Again, maybe I did not get it right, but could it be that the improvement in performance arises from overfitting the data? It also suggests that your system is somewhat dependent on the data, meaning that it might not behave as expected for a new dataset or out-of-domain data. Please explain this issue?

--
Comments, suggestions and typos: 1. The paper could clarify the differences between RISE and GraphReader [1], as GraphReader also employs a similar mechanism involving node exploration and reverting to preceding valid nodes in QA reasoning tasks.
2. It would be helpful to provide a comparison of token consumption between RISE and other RAG-based methods.
3. In Line 375, should “Figure 5” actually refer to “Figure 4”? If this is not a typo, it would be beneficial to specify where in the main text Figure 4 is discussed to avoid confusion.

 

[1] Li, S., He, Y., Guo, H., Bu, X., Bai, G., Liu, J., ... & Zheng, B. (2024). GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models. arXiv preprint arXiv:2406.14550.

None
Overall assessment scores: 2.5

2.5
Metareview score: 3
Metareview: This paper introduces RISE, a framework designed to enhance reasoning capabilities in Multi-Hop Question Answering tasks through iterative self-exploration.
Summary of reasons to publish: Extensive experimental results demonstrate that RISE can consistently outperform baseline models.  The experimental results are strong.

The proposed Self-Critique with the iterative optimization is quite novel.