Self-Critique Guided Iterative Reasoning for Multi-hop Question Answering
Authors: Zheng Chu, huiming fan, Jingchang Chen, Qianyu Wang, Mingda Yang, Jiafeng Liang, Zhongjie Wang, Hao Li, Guo Tang, Ming Liu, Bing Qin
Accepted at: acl25

Self-Critique Guided Iterative Reasoning for Multi-hop Question Answering

Zheng Chu1, Huiming Fan1, Jingchang Chen1, Qianyu Wang1, Mingda Yang1, Jiafeng Liang1, Zhongjie Wang1, Hao Li1, Guo Tang1, Ming Liu1,2*, Bing Qin1,2
1Harbin Institute of Technology, Harbin, China
2Peng Cheng Laboratory, Shenzhen, China
{zchu,mliu}@ir.hit.edu.cn

Abstract
Although large language models (LLMs) have demonstrated remarkable reasoning capabilities, they still face challenges in knowledge-intensive multi-hop reasoning. Recent work explores iterative retrieval to address complex problems. However, the lack of intermediate guidance often results in inaccurate retrieval and flawed intermediate reasoning, leading to incorrect reasoning. To address these, we propose Self-Critique Guided Iterative Reasoning (SiGIR), which uses self-critique feedback to guide the iterative reasoning process. Specifically, through end-to-end training, we enable the model to iteratively address complex problems via question decomposition. Additionally, the model is able to self-evaluate its intermediate reasoning steps. During iterative reasoning, the model engages in branching exploration and employs self-evaluation to guide the selection of promising reasoning trajectories. Extensive experiments on three multi-hop reasoning datasets demonstrate the effectiveness of our proposed method, surpassing the previous SOTA by 8.6%. Furthermore, our thorough analysis offers insights for future research. Our code, data, and models are available at Github1.

1 Introduction
With the advancement of large-scale pre-training techniques, LLMs have achieved remarkable performance on reasoning tasks with chain-of-thought (CoT) prompting (Wei et al., 2022; Chu et al., 2024b; OpenAI, 2023; DeepSeek-AI et al., 2024a). Moreover, when augmented with Slow Thinking mechanisms, several advanced models have demonstrated reasoning capabilities that either surpass or are comparable to those of human experts (DeepSeek-AI et al., 2024b; OpenAI, 2024).

Despite their remarkable capabilities, LLMs are prone to generating factually inaccurate responses in knowledge-intensive tasks, primarily attributable to their limited access to domain-specific knowledge. This limitation becomes particularly pronounced in multi-hop question answering (MHQA) (Yang et al., 2018; Ho et al., 2020; Trivedi et al., 2022), which substantially compromises both the reasoning accuracy and factual reliability of the model’s outputs. Furthermore, the complexity of multi-hop questions presents additional challenges for global planning during the reasoning process.

Retrieval-augmented generation is a promising strategy for mitigating the knowledge limitations of LLMs (Jiang et al., 2024). However, single-round retrieval is insufficient to encompass all relevant evidences to multi-hop questions (Lewis et al., 2020). To address this, iterative retrieval has been proposed (Trivedi et al., 2023; Asai et al., 2024). By alternating between retrieval and generation, it can acquire more comprehensive relevant knowledge , improving retrieval performance. Besides, some research focuses on the problems from the viewpoint of decomposition (Cao et al., 2023; Chu et al., 2024a; Su et al., 2024). It breaks down the complex multi-hop question into simpler sub-questions, thereby reducing the difficulty of reasoning and

![A comparison of iterative retrieval and self-critique guided reasoning (ours) in MHQA. Our method incorporates self-critique during iterative reasoning and conduct searches based on the received feedback.](page_1012_370_393_246.png)

Figure 1: A comparison of iterative retrieval and self-critique guided reasoning (ours) in MHQA. Our method incorporates self-critique during iterative reasoning and conduct searches based on the received feedback.
planning, and enabling more accurate retrieval.

Although the aforementioned methods have partially addressed some aspects of challenges, they still suffer from certain limitations. First, decomposing a complex question into sub-questions at the beginning of reasoning is highly challenging, and any mistakes can cause subsequent reasoning to deviate from the correct path. Second, the iterative retrieval approach struggles to handle complex problem planning and articulate retrieval intents clearly, resulting in inaccurate retrieval and reasoning. Finally, the absence of guidance for intermediate steps makes these methods susceptible to errors during reasoning, resulting in cascading errors.

To address these, our research focuses on following key aspects: (i) iterative question decomposition facilitates complex question planning and accurate retrieval. (ii) intermediate feedback guided reasoning expands the reasoning space and reduces cascading errors. Based on this motivation, we introduce Self-Critique Guided Iterative Reasoning (SiGIR). SiGIR iteratively solves complex questions through sub-question decomposition and selects promising reasoning paths guided by the self-critique feedback of intermediate reasoning steps.

Specifically, during training phase, we synthesize iterative reasoning rationales with intermediate supervision from advanced LLMs. Thereafter, the model is equipped with the ability to perform question decomposition, trigger retrieval, knowledge reasoning, self-evaluation, and question reduction by means of end-to-end training. During inference phase, the model conducts iteration-level (step-level) beam search guided by self-feedback, as illustrated in Figure 1. At each reasoning step, the model performs branch expansion for sub-question decomposition, retrieval, and reasoning. It then self-evaluates the current reasoning status and retain the top-k promising candidates based on self-critique rewards. The model continues this iterative reasoning process until the final answer is derived. Finally, the reasoning trajectory with the highest cumulative reward is selected as the final answer.

We conduct extensive experiments on three knowledge-intensive MHQA datasets: HotpotQA, 2WikiMQA, and MuSiQue (Yang et al., 2018; Ho et al., 2020; Trivedi et al., 2022). DeepSeek-V2.5, Mistral, LLaMA2, and Qwen2.5 are used during experiments (DeepSeek-AI et al., 2024a; Jiang et al., 2023a; Touvron et al., 2023; Yang et al., 2024). Our method demonstrates significant advantages, with an average improvement of 8.6% compared to the SOTA method. Furthermore, the thorough analysis highlights the effectiveness of our method in retrieval, self-critique, search, and efficiency.

2 Related Work

2.1 Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) is an effective approach for mitigating hallucinations in LLMs (Gao et al., 2023; Huang et al., 2023). Retrieval-then-read, a widely-used approach in RAG, consists of query-based retrieval followed by conditional generation (Lewis et al., 2020; Borgeaud et al., 2022). However, its effectiveness is critically reliant on the quality of retrieval. To address this, various methods have been proposed. Query rewriting improves retrieval quality by reformulating retrieval queries (Ma et al., 2023; Chan et al., 2024; Xin et al., 2024). Active retrieval dynamically adjusts retrieval timing and intent using adaptive mechanisms (Jiang et al., 2023b; Asai et al., 2024). Multi-round retrieval enables the acquisition of comprehensive information relevant to the problem (Trivedi et al., 2023; Shao et al., 2023).

Our proposed method employs multi-round active retrieval to acquire more precise and comprehensive content for complex multi-hop questions.

2.2 Multi-hop Question Answering

MHQA requires gathering multiple pieces of evidence and systematic multi-step reasoning to arrive at the final answer. Generally, single-round retrieval is inadequate to acquire comprehensive information (Lewis et al., 2020; Borgeaud et al., 2022). Some methods adopt question decomposition to enhance accurate retrieval and reasoning (Cao et al., 2023; Chu et al., 2024a; Su et al., 2024). However, decomposing complex questions from the very beginning is particularly challenging. To this end, iterative reasoning approach is introduced, which employs iterative multi-round reasoning to gradually tackle complex problems (Trivedi et al., 2023; Yu et al., 2024; Li et al., 2024).

In this work, we utilize iterative question decomposition and reduction to simplify the decomposition process and improve retrieval effectiveness.

2.3 Inference-Time Scaling

Recent studies have found that increasing computational resources at inference can improve model performance, termed inference-time scaling (Snell
Figure 2: Overview of SiGIR. (I) Training recipe to get the iterative reasoner with self-critique, \( R_{sc} \). (II) Reasoning characteristic of SC-Reasoner, which includes question decomposition, trigger retrieval, retrieval-augmented reasoning, self-evaluation, and question reduction. (III) SC-Reasoner engages in exploration during the iterative reasoning process and conducts reward-guided searches to select promising reasoning trajectories.

et al., 2024; Brown et al., 2024). Wang et al. (2023) employs a sampling ensemble technique to scale up, but the improvement is limited. Other methods adopt search strategies, such as Monte Carlo Tree Search, which yield significant improvements, but involve higher computational overhead (Zhang et al., 2024; Qi et al., 2024; Wang et al., 2024).

In contrast, we introduce reward-guided search that circumvents the overhead that simulations entail. It improves performance while ensuring reasonable costs, leading to efficient reasoning.

3 Self-Critique Guided Iterative Reasoning for Multi-hop QA

In this paper, we introduce Self-Critique Guided Iterative Reasoning (SiGIR) framework for multi-hop question answering, as illustrated in Figure 2. First, we synthesize iterative reasoning trajectories with intermediate supervision signals to train the model, which enables it to give self-critique of intermediate steps during reasoning. During reasoning, the model iteratively explores and selects the optimal trajectory guided by self-critique feedback.

In this section, we will outline the data synthesis pipeline and training recipe in Section 3.1, introduce the inference pattern of self-critique iterative reasoner in Section 3.2, and elaborate on the self-critique guided reasoning process in Section 3.3.

3.1 Training Recipe

Our goal is to enable the model to perform iterative reasoning and to conduct self-critique of the intermediate reasoning steps. To achieve this, we first train an iterative reasoner \( R \), followed by a critic model \( C \). Subsequently, we use \( R \) and \( C \) to generate rationales and train a model \( R_{sc} \) capable of iterative reasoning and self-critique. Figure 2(I) and Algorithm 2 illustrate the training procedure.2

Iterative Reasoner. We use advanced LLMs to synthesize iterative reasoning rationales. Specifically, given a corpus \( X \), where each instance \( x_i \) includes a question \( q_i \), an answer \( a_i \), and corresponding documents \( \{d_1, \cdots, d_n\} \). For each instance, we use few-shot prompts with the question and supporting documents as input to the LLM, which yields question decompositions and sub-question reasoning. Subsequently, we organize the sub-question decomposition, retrieval, and reasoning in an interleaved manner, and incorporate retrieval triggers before retrieved documents. We train the smaller model with the parsed training instances to get the vanilla iterative reasoner \( R \).3

Critic. Due to the high cost of invoking LLMs for reward evaluation, we attempt to use a small model to clone the reward evaluation of LLMs. To

2Notations of training and inference are shown in Table 14.
3Prompts for data synthesis can be found in Appendix C.
this end, we use iterative reasoning trajectories to construct the data for critic training. Notably, to prevent the critic from overfitting to the generator’s training data, we select a small amount of non-overlapping corpus with \( X \) for critic data synthesis, denoted as \( X_{critic} \). Additionally, we employ \( R \) instead of the LLM for reasoning synthesis, as \( R \) tends to make mistakes during reasoning. This helps ensure a balance between positive and negative samples. Subsequently, we employ the LLM to evaluate rewards on retrieval, reasoning, and overall quality, obtaining training data for critic model. We use it to train a separate critic model \( C \).

Self-Critique Reasoner. We aim to incorporate the critique signals into the reasoning process. Besides, recent studies have found that the generator and critic can enhance each other, and merging them into a single model can reduce inference cost. To this end, we use the critic \( C \) to annotate the sub-processes of the reasoning trajectories. For retrieval evaluation, we take the query and retrieved documents as input, with retrieval relevance as outputs. For sub-question reasoning, the inputs include sub-questions, documents, and reasoning. For overall reward, we take the entire reasoning as input, and the output is the overall quality. Subsequently, we add special tokens following the sub-processes, enabling the model to generate self-critique signals throughout the reasoning process. We train the model with this corpus to obtain the reasoner with self-critique, denoted as SC-Reasoner, or \( R_{sc} \).

SC-Reasoner is capable of reasoning and self-evaluation, facilitating data augmentation. Specifically, it can conduct reasoning on unlabeled questions and evaluate the quality of synthetic rationales, retaining high-quality rationales for subsequent training. This self-improvement pipeline can improve the performance in data scarcity scenarios. We use the standard causal language modeling objective for optimization during training.

3.2 Iterative Reasoner with Self-Critique

This section introduces the reasoning patterns of SC-Reasoner. Overall, SC-Reasoner is capable of question decomposition and planning, triggering retrieval, reasoning, and self-evaluating the reasoning process. As illustrated in Figure 2(II), given a question, the model first determines whether the question needs decomposition. If needed, the model decomposes one atomic sub-question at a time. Then, the model triggers external retrieval and conducts knowledge reasoning, along with self-evaluating the quality of both retrieval and reasoning.

Whenever the model resolves a sub-question, it reduces the original question to identify the remaining unresolved part, thereby simplifying the overall question. The model iteratively repeats this process until the reduced atomic question is resolved, leading to the final answer. At this point, it conducts self-evaluation of the entire reasoning trajectory.

3.3 Self-Critique Guided Iterative Reasoning

After training, SC-Reasoner can conduct iterative reasoning and self-critique. Accordingly, we introduce self-critic guided iterative reasoning, an iteration-level beam search algorithm. This algorithm searches within the iterative reasoning process, employing self-rewards to guide the selection of promising reasoning candidates, as illustrated in Figure 2(III) and Algorithm 1. Generally, in each iteration, the model first expands the exploration space by branching based on the current reasoning prefix, and then selects top-k candidate trajectories guided by self-rewards. At the end of the reasoning, we use self-rewards to choose the final answer.

Branch Exploration Each iteration consists of the following steps: question decomposition, retrieval, reasoning, remaining question reduction, and self-critique. At timestamp \( t \), the model branches out and expands downward through temperature sampling in question decomposition, retrieval, and reasoning, as shown in equations below.

\[
q_{t+1} \sim p(y \mid q_t^{(r)}, x_{<t}) \tag{1}
\]
\[
s_{t+1} \sim p(y \mid q_{t+1}, d_t, x_{<t}) \tag{2}
\]

where \( \sim \) is sampling, \( q_t^{(r)} \) is remaining question, \( q \) is sub-question, \( s \) is sub-question reasoning, \( p \) is probabilistic LM, and \( d \) is retrieved documents.

During the aforementioned process, the model also conducts self-evaluation and assigns rewards, as shown in Equ. (2,3). At the end of an iteration, the model will reduce the question for the next iteration based on current reasoning status \( x_{t+1} \).

\[
r_{t+1}^{ret} = p(y \mid q_{t+1}, d_t) \tag{3}
\]
\[
r_{t+1}^{reas} = p(y \mid q_{t+1}, d_t, s_{t+1}) \tag{4}
\]
\[
x_{t+1} = \oplus[x_t, q_{t+1}, d, r_{t+1}^{ret}, s_{t+1}, r_{t+1}^{reas}] \tag{5}
\]
\[
q_{t+1}^{(r)} = p(y \mid x_{t+1}) \tag{6}
\]

Candidate Selection Through branch exploration, the model’s reasoning investigates a wide space. Subsequently, we aim to select the most
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="5">2WikiMQA</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="4">MuSiQue</th>
  </tr>
  <tr>
    <th>Overall</th>
    <th>Brg.</th>
    <th>Infer.</th>
    <th>Comp.</th>
    <th>B.C.</th>
    <th>Overall</th>
    <th>Brg.</th>
    <th>Comp.</th>
    <th>Overall</th>
    <th>2hop</th>
    <th>3hop</th>
    <th>4hop</th>
  </tr>
  <tr>
    <th colspan="13"><i>Inference-based Methods</i></th>
  </tr>
  <tr>
    <td>Chain-of-thought*</td>
    <td>31.45</td>
    <td>11.54</td>
    <td>31.56</td>
    <td>51.90</td>
    <td>49.67</td>
    <td>35.31</td>
    <td>29.97</td>
    <td>54.94</td>
    <td>18.93</td>
    <td>22.57</td>
    <td>16.34</td>
    <td>12.49</td>
  </tr>
  <tr>
    <td>Self-Ask*</td>
    <td>28.20</td>
    <td>11.39</td>
    <td>26.16</td>
    <td>48.04</td>
    <td>41.97</td>
    <td>30.49</td>
    <td>24.12</td>
    <td>53.90</td>
    <td>15.91</td>
    <td>20.81</td>
    <td>12.27</td>
    <td>7.56</td>
  </tr>
  <tr>
    <td>RAG</td>
    <td>40.21</td>
    <td>19.22</td>
    <td>26.02</td>
    <td>69.25</td>
    <td>55.25</td>
    <td>53.42</td>
    <td>50.33</td>
    <td>65.49</td>
    <td>22.06</td>
    <td>30.24</td>
    <td>16.64</td>
    <td>8.32</td>
  </tr>
  <tr>
    <td>IRCoT</td>
    <td>46.80</td>
    <td>29.89</td>
    <td>25.55</td>
    <td>73.13</td>
    <td>61.44</td>
    <td>49.03</td>
    <td>45.33</td>
    <td>63.45</td>
    <td>21.43</td>
    <td>27.15</td>
    <td><u>18.73</u></td>
    <td>9.86</td>
  </tr>
  <tr>
    <td>ProbTree†</td>
    <td>59.90</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>50.40</td>
    <td>-</td>
    <td>-</td>
    <td>27.00</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>BeamAggR†</td>
    <td>63.20</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>55.20</td>
    <td>-</td>
    <td>-</td>
    <td>32.30</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <th colspan="13"><i>Training-based Methods</i></th>
  </tr>
  <tr>
    <td>Self-RAG</td>
    <td>31.60</td>
    <td>16.91</td>
    <td>14.06</td>
    <td>50.39</td>
    <td>49.00</td>
    <td>54.60</td>
    <td>52.51</td>
    <td>62.75</td>
    <td>22.00</td>
    <td>30.04</td>
    <td>13.21</td>
    <td>14.77</td>
  </tr>
  <tr>
    <td>RAG-Star†</td>
    <td>46.80</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>60.00</td>
    <td>-</td>
    <td>-</td>
    <td>30.70</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>GenGround†</td>
    <td>50.21</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>52.26</td>
    <td>-</td>
    <td>-</td>
    <td>27.36</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Auto-RAG</td>
    <td>56.06</td>
    <td>44.12</td>
    <td>40.94</td>
    <td>79.44</td>
    <td>59.48</td>
    <td>50.03</td>
    <td>46.96</td>
    <td>62.02</td>
    <td>22.08</td>
    <td>29.32</td>
    <td>15.90</td>
    <td>11.54</td>
  </tr>
  <tr>
    <td>DR-Distillation</td>
    <td>70.51</td>
    <td>46.13</td>
    <td>56.34</td>
    <td><b>96.90</b></td>
    <td><b>96.00</b></td>
    <td>58.06</td>
    <td>54.08</td>
    <td><b>73.62</b></td>
    <td>22.74</td>
    <td>35.08</td>
    <td>12.06</td>
    <td>6.57</td>
  </tr>
  <tr>
    <th>Ours</th>
    <td><b>74.47</b></td>
    <td><b>56.39</b></td>
    <td><b>58.69</b></td>
    <td>95.47</td>
    <td>94.94</td>
    <td><b>63.09</b></td>
    <td><b>59.45</b></td>
    <td><b>77.37</b></td>
    <td><b>37.15</b></td>
    <td><b>49.51</b></td>
    <td><b>27.55</b></td>
    <td><b>18.37</b></td>
  </tr>
  <tr>
    <td>- Qwen2.5</td>
    <td>70.51</td>
    <td>54.98</td>
    <td>40.39</td>
    <td>94.90</td>
    <td>90.72</td>
    <td>60.93</td>
    <td>56.59</td>
    <td>77.87</td>
    <td>33.29</td>
    <td>40.11</td>
    <td>28.18</td>
    <td>22.90</td>
  </tr>
  <tr>
    <td>- LLaMA2</td>
    <td>65.82</td>
    <td>50.93</td>
    <td>43.36</td>
    <td>90.44</td>
    <td>79.25</td>
    <td>52.92</td>
    <td>47.92</td>
    <td>72.46</td>
    <td>28.56</td>
    <td>39.00</td>
    <td>20.60</td>
    <td>12.98</td>
  </tr>
</table>

Table 1: Experimental results on three multi-hop reasoning datasets: 2WikiMQA, HotpotQA, and MuSiQue. Mistral-7B is used as the default backbone. Best and second results are highlighted by bold and underline. The evaluation metric is F1. * indicates closed-book reasoning, and † denotes results are derived from the original paper.

promising candidates among them based on rewards. Selected candidates will serve as the starting point for the next iteration of reasoning. Specifically, we use the cumulative process reward as the criterion for candidate selection. The cumulative reward consists of reward from previous iteration and current iteration. We retain the candidates with top-k largest rewards, as shown in Equ. (7, 8). Once the reasoning is finished, we need to select one candidate as the final answer, using either cumulative reward of entire trajectory or outcome reward.

\[
r_{t+1}^c = \underbrace{r_t^c}_{\text{previous reward}} + \underbrace{r_{t+1}^{\text{retr}} + r_{t+1}^{\text{reas}}}_{\text{current reward}}
\] (7)

\[
x_{t+1}^* = \operatorname{top-k}(x_{t+1}, r_{t+1}^c)
\] (8)

\[
r^c = \sum_{i=1}^n (r_i^{\text{retr}} + r_i^{\text{reas}})
\] (9)

\[
r^o = p(y \mid q^{(r)}, x_{<n}, a)
\] (10)

where \( r^c \) and \( r^o \) denote cumulative process reward and outcome reward, respectively.

4 Experimental Setup

4.1 Benchmarks

We evaluate SiGIR on three knowledge-intensive multi-hop reasoning datasets: HotpotQA (Yang et al., 2018), 2WikiMQA (Ho et al., 2020), and MuSiQue (Trivedi et al., 2022). Following previous work (Trivedi et al., 2023; Cao et al., 2023), we use the development set as the test data. The experiments are conducted in the open-domain setting. We use token-level F1 as the evaluation metric.

4.2 Baselines

We select extensive baselines for comparison, including the following methods: Fewshot CoT (Wei et al., 2022), Self-Ask (Press et al., 2023), IRCot (Trivedi et al., 2023), ProbTree (Cao et al., 2023), BeamAggR (Chu et al., 2024a), RAG (Asai et al., 2024), RAG-Star (Jiang et al., 2024), Gen-Ground (Shi et al., 2024), Auto-RAG (Yu et al., 2024), and DR-Distillation (Li et al., 2024). 4

4.3 Implementation Details

We use DeepSeek-v2.5 (DeepSeek-AI et al., 2024a) as the advanced LLM in our experiments. By default, we use BM25 (Robertson and Zaragoza, 2009) for sparse retrieval. In analysis, we additionally use Contriever (Izacard et al., 2022) for dense retrieval. During training, we use LoRA (Hu et al., 2022) for parameter-efficient fine-tuning, with Huggingface Accelerate for mixed-precision training. The entire training process takes approximately 3 hours on 4 × A100-SXM4-80GB GPUs.

4Details about benchmarks, retrieval, baselines, implementation, and task definition can be found in Appendix A.
<table>
  <tr>
    <th>Settings</th>
    <th>2WikiMQA</th>
    <th>HotpotQA</th>
    <th>MuSiQue</th>
  </tr>
  <tr>
    <th>SGIR</th>
    <td>74.47</td>
    <td>63.09</td>
    <td>37.15</td>
  </tr>
  <tr>
    <th colspan="4">Fine-grained Reward</th>
  </tr>
  <tr>
    <td>(a) w/o. guided search</td>
    <td>72.37</td>
    <td>60.86</td>
    <td>35.47</td>
  </tr>
  <tr>
    <td>(b) w/o. reward</td>
    <td>67.23</td>
    <td>54.94</td>
    <td>29.45</td>
  </tr>
  <tr>
    <td>(c) w/o. self-critique</td>
    <td>66.96</td>
    <td>56.86</td>
    <td>28.62</td>
  </tr>
  <tr>
    <th colspan="4">Coarse-grained Reward</th>
  </tr>
  <tr>
    <td>(d) w. coarse reward</td>
    <td>73.40</td>
    <td>59.85</td>
    <td>34.32</td>
  </tr>
  <tr>
    <td>(e) w/o. guided search</td>
    <td>72.32</td>
    <td>60.89</td>
    <td>35.40</td>
  </tr>
</table>

Table 2: Ablation experiments on rewards and search algorithm in fine-grained and coarse-grained scenarios. The experiments are conducted with Mistral-7B.

<table>
  <tr>
    <th>Stage</th>
    <th>#Labeled</th>
    <th>#Unlabeled</th>
    <th>#Total</th>
    <th>Quantile</th>
  </tr>
  <tr>
    <td>Training Set</td>
    <td>49000</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Full SFT</td>
    <td>38625</td>
    <td>0</td>
    <td>38625</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>SP Start Point</td>
    <td>15000</td>
    <td>0</td>
    <td>15000</td>
    <td>95.3%</td>
  </tr>
  <tr>
    <td>First Iteration</td>
    <td>15000</td>
    <td>21669</td>
    <td>36669</td>
    <td>97.1%</td>
  </tr>
  <tr>
    <td>Second Iteration</td>
    <td>15000</td>
    <td>29659</td>
    <td>44659</td>
    <td>97.8%</td>
  </tr>
</table>

Table 3: Comparison of the amount of data used for training and performance quantiles. #Labeled refers to data annotated by the teacher model with reasoning trajectories, and #Unlabeled refers to data synthesized by the smaller model itself. Quantile indicates the percentage of performance compared to full SFT.

5 Experimental Results

We conduct extensive experiments on three multi-hop reasoning datasets, as shown in Table 1. We notice that although CoT is considered proficient in multi-step reasoning, but its performance is poor due to a lack of question-relevant knowledge within the LLM. When single-round retrieval is employed, an average performance improvement of 35.0% is observed. Iterative multi-round retrieval results in improved performance, achieving up to a 48.8% increase on 2WikiMQA. Additionally, methods based on question decomposition further improve reasoning performance. These results suggest that precise retrieval and reasoning planning are key factors affecting multi-hop reasoning.

Some methods equip the model with iterative reasoning capabilities through fine-tuning, introducing strategies such as active retrieval, question decomposition, and search during the iterative reasoning process. These methods can better address complex problems and achieve more accurate retrieval, further enhancing the performance.

As shown in Table 1, our method demonstrates remarkable superiority over baselines. It surpasses the previous state-of-the-art methods on all three datasets, achieving improvement of 2WikiMQA (+3.96), HotpotQA (+3.09), and MuSiQue (+4.85). Furthermore, our method excels particularly on the challenging MuSiQue dataset, which involves 3 to 4 hop hard questions. It achieves a significant improvement of 15.0% on the MuSiQue dataset compared to previous methods, with enhancements of 47.1% and 24.37% on 3/4-hop challenging questions, respectively. Additionally, we also conduct experiments on the Qwen2.5 and LLaMA2 models, achieving competitive performance that demonstrates the generalizability of our method.

We identify three key factors that contribute to our improvement: (a) Iterative multi-round reasoning and sub-question planning reduce the complexity of questions while improving retrieval accuracy. (b) Self-critique provides valuable feedback signals for intermediate steps during the reasoning process. (c) Reward-guided search selects superior reasoning trajectories among the exploration space.

6 Analysis

6.1 Ablation Study

We conduct ablation experiments to investigate the effects of different modules, as shown in Table 2. Our method employs fine-grained rewards by default. First, removing reward-guided search (a) and substituting it with greedy inference leads to considerable performance declines on all three datasets, highlighting the importance of search algorithms. Next, we remove the reward signal from greedy inference (b), which means the model’s inference process would not receive any guidance. This change results in a substantial performance decrease of 10%, demonstrating the essential role of reward signals in the inference process. Additionally, the exclusion of self-critique data from the training data produces a minor performance decline.

To further investigate the role of reward signals, particularly the impact of reward granularity on model performance, we replace the rewards with coarse-grained binary rewards, as shown in Table 2. In reward-guided greedy inference (e), fine-grained and coarse-grained rewards have similar effects. However, when a broader search is required (d), coarse-grained rewards exhibit certain drawbacks, resulting in a 2.7% performance decrease compared to fine-grained rewards. This indicates that fine-grained rewards are more effective in search-form reasoning, particularly when there are a substantial number of intermediate reasoning steps.
Figure 3: The impact of increasing training data and self-improvement on performance. As data scales, the performance consistently improves. Starting with 40% of the data, two self-improvement iterations boost performance to nearly 80%, with only 2% gap with full sft.

<table>
  <tr>
    <th>Strategy</th>
    <th>2WikiMQA</th>
    <th>HotpotQA</th>
    <th>MuSiQue</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="5">Search Strategy</th>
  </tr>
  <tr>
    <td>Guided Search</td>
    <td>74.47</td>
    <td>63.09</td>
    <td>37.15</td>
    <td>58.23</td>
  </tr>
  <tr>
    <td>(a) one candidate</td>
    <td>72.97</td>
    <td>61.27</td>
    <td>36.21</td>
    <td>56.81</td>
  </tr>
  <tr>
    <td>(b) no branching</td>
    <td>68.87</td>
    <td>52.88</td>
    <td>28.34</td>
    <td>50.03</td>
  </tr>
  <tr>
    <td>Greedy Inference</td>
    <td>72.37</td>
    <td>60.86</td>
    <td>35.47</td>
    <td>56.23</td>
  </tr>
  <tr>
    <th colspan="5">Selection Strategy</th>
  </tr>
  <tr>
    <td>(c) cumulative reward</td>
    <td>74.47</td>
    <td>63.09</td>
    <td>37.15</td>
    <td>58.23</td>
  </tr>
  <tr>
    <td>(d) outcome reward</td>
    <td>71.79</td>
    <td>60.96</td>
    <td>35.23</td>
    <td>55.99</td>
  </tr>
  <tr>
    <td>(e) random select</td>
    <td>71.17</td>
    <td>60.10</td>
    <td>35.17</td>
    <td>55.81</td>
  </tr>
</table>

Table 4: Analysis on various search and voting strategies on our method. Retaining more candidate reasoning trajectories and exploring different sub-questions help improve performance. Utilizing cumulative process rewards to select the final reasoning trajectory is effective.

6.2 Training Scaling and Self-Improvement

We investigate the performance under different amounts of training data and the effect of self-improvement data synthesis, as shown in Figure 3. Initially, as the amount of labeled data increases, the performance improves consistently. The most significant improvement occurs when 40% of the training data is utilized; beyond this threshold, the rate of performance enhancement decreases.

Starting with the model trained on 40% of the data, we perform iterative self-improvement. As shown in Figure 3 and Table 3, in the first iteration of self-improvement, the model achieves performance corresponding to the 60% training data quantile. In the second iteration, the model’s performance approaches the 80% quantile, with only a 2.2% gap from the performance achieved with the full labeled data fine-tuning. This suggests that our approach can effectively synthesize data through self-critique, yielding competitive performance in situations where labeled data is scarce.

<table>
  <tr>
    <th>Reward / Task</th>
    <th>Retrieval</th>
    <th>Reasoning</th>
    <th>Overall</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Direct Reward</td>
    <td>93.2</td>
    <td><b>95.8</b></td>
    <td>81.2</td>
    <td>90.1</td>
  </tr>
  <tr>
    <td>Post Explanation</td>
    <td>95.4</td>
    <td>95.4</td>
    <td>80.6</td>
    <td>90.5</td>
  </tr>
  <tr>
    <td>Chain-of-thought</td>
    <td><b>97.4</b></td>
    <td>88.6</td>
    <td><b>87.4</b></td>
    <td><b>91.1</b></td>
  </tr>
  <tr>
    <td>- Dedicated RM</td>
    <td>95.8</td>
    <td>93.6</td>
    <td>78.8</td>
    <td>89.4</td>
  </tr>
</table>

Table 5: The performance of different reward forms on reward modeling tasks. Generative rewards are more effective than discriminative rewards, and mixed reward modeling has advantages over dedicated reward models in complex outcome reward modeling task.

<table>
  <tr>
    <th>Strategy</th>
    <th>2WikiMQA</th>
    <th>HotpotQA</th>
    <th>MuSiQue</th>
    <th>Average</th>
    <th>Throughput</th>
  </tr>
  <tr>
    <td>GenCritic</td>
    <td>72.37</td>
    <td><b>60.86</b></td>
    <td>35.47</td>
    <td>56.23</td>
    <td>1.92 QPS</td>
  </tr>
  <tr>
    <td>Separated</td>
    <td>72.23</td>
    <td>58.45</td>
    <td><b>36.46</b></td>
    <td>55.71</td>
    <td>0.43 QPS</td>
  </tr>
</table>

Table 6: Comparison of performance and efficiency between generative and separated critic model. The generative critic model brings slightly better performance, but has significant higher inference throughput.

6.3 Search and Trajectory Selection

We conduct experiments to investigate the factors influencing the guided reasoning process, including search strategies and reasoning selection, as shown in Table 4. As indicated in the preceding discussion, reward-based search is more effective than greedy inference. During the search process, we retain only one candidate trajectory in each iteration (a), which results in a performance decrease of 2.4%. This indicates that maintaining more candidate sequences preserves a greater number of potentially correct reasoning trajectories. Next, we set the question decomposition exploration width to 1 (b), which means only retrieval will be expanded during reasoning. This leads to a substantial drop of 13.57%, indicating the significance of exploring various sub-questions in the reasoning process.

At the end of the search, we need to select one trajectory as the final answer, and we investigate various answer selection strategies. Using cumulative process rewards for trajectory selection yields the best performance, followed by using outcome rewards. Both strategies outperform random selection. This suggests that in complex reasoning processes, intermediate rewards can serve as a criterion for assessing the reasoning quality, and that process rewards are more effective at distinguishing between different levels of reasoning quality.

6.4 Reward Modeling

The forms of reward (critic) models are diverse, such as generative and discriminative forms. We
<table>
  <tr>
    <th>Retriever</th>
    <th>2WikiMQA</th>
    <th>HotpotQA</th>
    <th>MuSiQue</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="5">Reward-guided Greedy Inference</th>
  </tr>
  <tr>
    <td>Sparse</td>
    <td>72.37</td>
    <td>60.86</td>
    <td>35.47</td>
    <td>56.23</td>
  </tr>
  <tr>
    <td>Dense</td>
    <td>75.44</td>
    <td>59.53</td>
    <td>38.48</td>
    <td>57.81</td>
  </tr>
  <tr>
    <td>Hybrid</td>
    <td><b>76.04</b></td>
    <td><b>62.89</b></td>
    <td><b>39.38</b></td>
    <td><b>59.43</b></td>
  </tr>
  <tr>
    <th colspan="5">Reward-guided Search</th>
  </tr>
  <tr>
    <td>Sparse</td>
    <td>74.47</td>
    <td>63.09</td>
    <td>37.15</td>
    <td>58.23</td>
  </tr>
  <tr>
    <td>Dense</td>
    <td>75.72</td>
    <td>61.72</td>
    <td><b>40.36</b></td>
    <td><b>59.26</b></td>
  </tr>
  <tr>
    <td>Hybrid</td>
    <td><b>76.05</b></td>
    <td><b>63.47</b></td>
    <td>37.05</td>
    <td>58.85</td>
  </tr>
</table>

Table 7: The impact of retrieval systems on reasoning. Accurate retrieval leads to better reasoning performance. Our method naturally supports hybrid retrieval (sparse + dense) retrieval, which facilitates better reasoning.

<table>
  <tr>
    <th>Retriever</th>
    <th>2WikiMQA</th>
    <th>HotpotQA</th>
    <th>MuSiQue</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>BM25</td>
    <td>72.37</td>
    <td>60.86</td>
    <td>35.47</td>
    <td>56.23</td>
  </tr>
  <tr>
    <td>Contriever</td>
    <td>75.44</td>
    <td>59.53</td>
    <td>38.48</td>
    <td>57.81</td>
  </tr>
  <tr>
    <td>+ Hybrid</td>
    <td>76.04</td>
    <td>62.89</td>
    <td>39.38</td>
    <td>59.43</td>
  </tr>
  <tr>
    <td>BGE-large</td>
    <td>74.36</td>
    <td>64.27</td>
    <td>40.58</td>
    <td>59.74</td>
  </tr>
  <tr>
    <td>+ Hybrid</td>
    <td>75.85</td>
    <td>64.25</td>
    <td>41.03</td>
    <td>60.38</td>
  </tr>
  <tr>
    <td>ColBERT-v2</td>
    <td>75.37</td>
    <td>65.44</td>
    <td>42.24</td>
    <td>61.02</td>
  </tr>
  <tr>
    <td>+ Hybrid</td>
    <td>75.37</td>
    <td>65.69</td>
    <td>43.05</td>
    <td>61.37</td>
  </tr>
</table>

Table 8: Experiments on different dense retrievers and their hybrid retrieval variants. Hybrid retrieval strategies improve performance across various dense retrievers.

![Scatter plot showing F1 score vs. average inference iterations for different methods and datasets](page_1012_1012_388_388.png)

Figure 4: Relationship between performance and the average number of iterations to solve a multi-hop question. Upper-left represents efficient reasoning. Our method requires fewer iterations to solve one question with higher F1, demonstrating higher reasoning efficiency.

examine the performance of various types of critic models on our reward modeling task, as shown in Table 5. Overall, generative reward modeling outperforms discriminative modeling. Moreover, among generative forms, chain-of-thought (CoT) is preferred over post-hoc explanations. Notably, CoT excels at modeling complex outcome rewards because it systematically decomposes the entire reasoning trajectory for step-by-step analysis.

Additionally, we investigate dedicated RM (each model assigned to a specific task) in comparison to mixed RM (one model responsible for all tasks). The dedicated RM exhibits performance variations of -1.6% / +5.6% for retrieval and reasoning tasks, respectively. However, there is a significant performance decrease of 9.8% in modeling outcome rewards. Since outcome rewards assess the entire reasoning process by integrating various reward modeling tasks, this suggests that multi-task reward modeling of sub-tasks enhances the model’s ability to evaluate complex outcome rewards.

Furthermore, we highlight that generative critic models offer greater benefits than separated generators and critics. As shown in Table 6, the GenCritic outperforms the separated models in two of the three datasets, achieving an average performance improvement of 0.9%. Without sacrificing reasoning performance, the GenCritic achieves a throughput that is 4.46 times higher, demonstrating significantly better reasoning efficiency5.

6.5 Retrieval System and Hybrid Retrieval

Retrieval plays a critical role in knowledge reasoning. To investigate the effect of the retrieval system, we employ various retrieval systems in the experiments, as illustrated in Table 7. Dense retrieval has better performance than sparse retrieval. When employing dense retrieval, a notable improvement of 2.8% is observed, suggesting that retrieval quality impacts reasoning performance. Furthermore, since our method treats each retrieved document independently, it naturally facilitates the integration of different retrieval systems. The hybrid retrieval system achieves a significant improvement of 5.7%, demonstrating the superiority of our method. However, in a guided search, excellent reasoning performance can be achieved using only sparse retrieval. This suggests that when retrieval quality is poor, the reward-guided search can help the model in exploring and filtering irrelevant retrieved documents, thereby improving reasoning performance.

In addition, we also apply hybrid retrieval on different dense retrievers (BGE-large6 and ColBERT-v27), as shown in Table 8. The hybrid retrieval strategy consistently achieves significant performance improvements across different dense retrievers, demonstrating its effectiveness.

5Please refer to Appendix B.1 for details of analysis.

6http://huggingface.co/BAAI/bge-large-en-v1.5
7http://huggingface.co/colbert-ir/colbertv2.0
6.6 Iterative Reasoning Efficiency

We assess reasoning efficiency by measuring the iterations needed to solve a multi-hop problem alongside its reasoning performance, as illustrated in Figure 4. Among the mentioned methods, Auto-RAG / DR-Distillation shows commendable reasoning capabilities, but it demands more iterations of reasoning, which incurs substantial costs. Conversely, our method not only surpasses their performance but also reduces the average number of reasoning iterations by 26.7%, demonstrating superior reasoning efficiency. Additionally, we also measure the average inference latency of single samples and token consumption in Appendix B.2.

7 Conclusion

In this work, we introduce Self-Critique Guided Iterative Reasoning for knowledge-intensive multi-hop question answering. Initially, we enable the model to conduct iterative reasoning through iterative question decomposition and to self-evaluate the intermediate reasoning steps, through a novel data synthesis and training pipeline. During inference, the model performs self-critique guided search: it first explores multiple reasoning paths via branching exploration and then utilizes self-critique feedback to refine the search process and identify the most promising reasoning trajectory. Extensive experiments and thorough analysis demonstrate the effectiveness of our proposed method.

Limitations

Although we have conducted comprehensive experiments to demonstrate the effectiveness of our proposed methods, and provided detailed analysis on the components of our system, this paper still has some limitations.

For sub-question exploration, we employ a sampling-based approach rather than training a dedicated question decomposition module. Although this design improves the simplicity and efficiency of the overall system, it can limit the exploratory breadth of the model with respect to sub-questions. We plan to address this in future research, such as incorporating a query rewriting module.

Our method adopts a route of synthetic data followed by supervised fine-tuning (SFT), without incorporating reinforcement learning in the model’s sub-question reasoning process. In the future, we consider using strategies such as Monte Carlo tree search to generate richer reasoning trajectories and introducing reinforcement learning to optimize these trajectories, thereby further enhancing the model’s reasoning performance.

Acknowledgements

The research in this article is supported by the National Key Research and Development Project (2022YFF0903301), the National Science Foundation of China (U22B2059, 62276083). We also appreciate the support from China Mobile Group Heilongjiang Co., Ltd. @ on our research, the research is jointly completed by both parties. Ming Liu is the corresponding author.

References

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206–2240. PMLR.

Bradley C. A. Brown, Jordan Juravsky, Ryan Saul Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. CoRR, abs/2407.21787.

Shulin Cao, Jiajie Zhang, Jiaxin Shi, Xin Lv, Zijun Yao, Qi Tian, Lei Hou, and Juanzi Li. 2023. Probabilistic tree-of-thought reasoning for answering knowledge-intensive complex questions. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 12541–12560. Association for Computational Linguistics.

Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. RQ-RAG: learning to refine queries for retrieval augmented generation. CoRR, abs/2404.00610.

Zheng Chu, Jingchang Chen, Qianglong Chen, Haitian Wang, Kun Zhu, Xiyuan Du, Weijiang Yu, Ming
Liu, and Bing Qin. 2024a. Beamaggr: Beam aggregation reasoning over multi-source knowledge for multi-hop question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1229–1248. Association for Computational Linguistics.

Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2024b. Navigate through enigmatic labyrinth A survey of chain of thought reasoning: Advances, frontiers and future. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1173–1203. Association for Computational Linguistics.

DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiachi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxia Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushu Du, R. J. Chen, R. Jin, Ruqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuaing Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, Tao Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, and Xiaowen Sun. 2024a. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405.04434.

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiachi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxia Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushu Du, R. J. Chen, R. L. Jin, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuaing Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. 2024b. Deepseek-v3 technical report. CoRR, abs/2412.19437.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. CoRR, abs/2312.10997.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6609–6625. International Committee on Computational Linguistics.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. CoRR, abs/2311.05232.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023a. Mistral 7b. CoRR, abs/2310.06825.

Jinhao Jiang, Jiayi Chen, Junyi Li, Ruiyang Ren, Shijie Wang, Wayne Xin Zhao, Yang Song, and Tao Zhang. 2024. Rag-star: Enhancing deliberative reasoning with retrieval augmented verification and refinement. CoRR, abs/2412.12881.

Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie
Callan, and Graham Neubig. 2023b. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 7969–7992. Association for Computational Linguistics.

Patrick S. H. Lewis, Ethan Perez, Aleksandra Pikutus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Xiang Li, Shizhu He, Fangyu Lei, JunYang JunYang, Tianhuang Su, Kang Liu, and Jun Zhao. 2024. Teaching small language models to reason for knowledge-intensive multi-hop question answering. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 7804–7816. Association for Computational Linguistics.

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 5303–5315. Association for Computational Linguistics.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

OpenAI. 2024. Introducing openai o1.

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 5687–5711. Association for Computational Linguistics.

Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. Mutual reasoning makes smaller llms stronger problem-solvers. CoRR, abs/2408.06195.

Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.

Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 9248–9274. Association for Computational Linguistics.

Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. 2024. Generate-then-ground in retrieval-augmented generation for multi-hop question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 7339–7353. Association for Computational Linguistics.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314.

Xin Su, Tiep Le, Steven Bethard, and Phillip Howard. 2024. Semi-structured chain-of-thought: Integrating multiple sources of knowledge for improved language model reasoning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 8597–8613. Association for Computational Linguistics.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashyakov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Bleicher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harthorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashu Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multi-hop questions via single-hop question composition. Trans. Assoc. Comput. Linguistics, 10:539–554.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 10014–10037. Association for Computational Linguistics.
Chaojie Wang, Yanchen Deng, Zhiyi Lv, Zeng Liang, Jujie He, Shuicheng Yan, and Bo An. 2024. Q*: Improving multi-step reasoning for llms with deliberative planning. CoRR, abs/2406.14283.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Chunlei Xin, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, and Le Sun. 2024. Chain-of-rewrite: Aligning question and documents for open-domain question answering. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 1884–1896. Association for Computational Linguistics.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2369–2380. Association for Computational Linguistics.

Tian Yu, Shaolei Zhang, and Yang Feng. 2024. Autotag: Autonomous retrieval-augmented generation for large language models. CoRR, abs/2411.19443.

Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. Rest-mcts*: LLM self-training via process reward guided tree search. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

A Supplemental Materials on Setup

A.1 Datasets

HotpotQA (Yang et al., 2018)8 is a multi-hop reasoning dataset that includes two reasoning types: bridge and comparison. Bridge: sub-questions are linked by bridging entities. Comparison: compare two statements, including yes/no and partial order.

2WikiMQA (Ho et al., 2020)9 is a multi-hop reasoning dataset that contains four reasoning types. The bridge (compositional) and comparison are the same as those in HotpotQA. Besides, it includes two additional reasoning types: bridge-comparison and inference. Bridge-comparison: compare two bridge sub-questions. Inference: sub-questions are connected through reasoning relationships rather than bridging entities. For example, the predicate grandfather is equal to a father’s father.

MuSiQue (Trivedi et al., 2022)10 is a multi-hop reasoning dataset including 2-4 hops questions. All questions in MuSiQue are compositional, with the number of hops extended to a maximum of 4 hops. Besides, it includes multiple compositional questions, where a question contains more than one bridge entity. Due to the complexity of the questions, MuSiQue has the highest level of difficulty.

A.2 Models

We use DeepSeek-V2.5 (DeepSeek-AI et al., 2024a)11 as the advanced LLM to synthesis reasoning and critic rationales. In the experiments, we use Mistral-7B (Jiang et al., 2023a)12 as the student backbone model by default. Besides, we also conduct experiments on LLaMA2-7B (Touvron et al., 2023)13 and Qwen2.5-7B (Yang et al., 2024)14 to demonstrate the generalization of our method.

A.3 Training Corpus

We randomly sample subsets of the training data from 2WikiMQA, HotpotQA, and MuSiQue to construct the training corpus. Specially, we sample 20,000, 10,000, 19,000 instances from the respective datasets to construct the initial generator train-

8https://hotpotqa.github.io/
9https://github.com/Alab-NII/2wikimultihop
10https://github.com/StonyBrookNLP/musique
11https://huggingface.co/deepseek-ai/
DeepSeek-V2.5
12https://huggingface.co/mistral-community/
Mistral-7B-v0.2
13https://huggingface.co/meta-llama/Llama-2-7b
14https://huggingface.co/Qwen/Qwen2.5-7B
ing corpus \( X \), while an additional 3,000, 2,000, and 1,000 non-overlapping instances are sampled as the critic model training corpus \( X_{critic} \).

It is noteworthy that, to demonstrate the generality of our approach, we train a unified model on a mixture of all datasets, enabling it to perform reasoning across diverse multi-hop QA datasets instead of dataset-specific training.

A.4 Retrieval

Following prior work (Trivedi et al., 2023), we construct a retrieval corpus using supporting facts and irrelevant documents from the training instances, and conduct retrieval within this scope.

By default, we employ BM25 (Robertson and Zaragoza, 2009) provided by Elastic Search\footnote{https://www.elastic.co/} for sparse retrieval in our experiments. Additionally, we incorporate dense retrieval and hybrid retrieval in our analysis. For dense retrieval, we use Contriever (Izacard et al., 2022)\footnote{https://huggingface.co/facebook/contriever-msmarco}. For hybrid retrieval, we combine the results from both sparse and dense retrieval by taking their union. Since our method independently processes each retrieved document and selects relevant ones based on self-evaluation, it inherently supports hybrid retrieval, thereby delivering better retrieval performance.

A.5 Baselines

Chain-of-Thought (Wei et al., 2022) generates step-by-step reasoning before the final answer. We use 20-shot demonstrations.

RAG (Lewis et al., 2020) retrieves k documents and conducts chain-of-thought reasoning conditional on the retrieval contents. We use BM25 to retrieve 5 documents, and 3-shot demonstrations.

IRCoT (Trivedi et al., 2023) interleaves retrieval and reasoning until the model arrives at the final answer. We set the number of documents per iteration to 4, with a maximum of 4 iterations.

Self-Ask (Press et al., 2023) iteratively breaks down complex questions and answers the resulting sub-questions one by one. We use 4-shot prompts.

ProbTree (Cao et al., 2023) decomposes the question into a tree and uses bottom-up aggregation reasoning based on log-prob to get the final answer.

BeamAggR (Chu et al., 2024a) parses the complex question into a structured decomposition tree. It then performs bottom-up beam aggregation reasoning to select the promising reasoning trajectory.

Self-RAG (Asai et al., 2024) enables dynamic retrieval and generation through self-reflective tokens. We set the max retrieval documents per round to 5, and the max reasoning depth to 7.

RAG-Star (Jiang et al., 2024) uses Monte Carlo Tree Search to guide LLM-based tree-like reasoning and employs retrieval-augmented verification to consolidate internal and external knowledge.

GenGround (Shi et al., 2024) alternates between generating single-hop question-answer pairs using LLMs and grounding them in retrieved documents to correct potential knowledge errors.

Auto-RAG (Yu et al., 2024) leverages LLMs’ reasoning and decision-making capabilities to plan retrievals, query rewriting, and gather external retrieval knowledge through multi-turn dialogues.

DR-Distillation (Li et al., 2024) distills two small models (Decomposer and Responser) to interactively solve knowledge-intensive multi-hop questions by breaking them into simpler sub-questions and conducting retrieval-augmented reasoning.

A.6 Task Definition

Given a multi-hop QA pair (\( Q, a \)) that requires multiple reasoning steps \( r_i \) to solve, each reasoning step \( r_i \) is based on a specific piece of supporting evidence \( d_i \). We use the LLM to iteratively decompose the question \( q_i \), employ an external retriever Retr to retrieve relevant documents based on the sub-question, and conduct retrieval-augmented reasoning conditional on the retrieved documents. Once the reasoning is finished, the answer in the last iteration is regarded as the final answer \( a^* \).

\[
Q \rightarrow r_1 \rightarrow r_2 \cdots \rightarrow a^*
\]

A.7 Implementation Details

The training and inference are conducted on 4 and 1 NVIDIA Tesla A100-80G GPUs, respectively. The training process is initiated from the base model, and a chat template structured as '
 {Instruction} 
 {Output}' is manually added. During the training process, we exclusively train the Output while masking the retrieved document portions to prevent them from being included in the
<table>
  <tr>
    <th>Hyperparameters</th>
    <th>Values</th>
  </tr>
  <tr>
    <td colspan="2"><i>Data Synthesis Hyperparameters</i></td>
  </tr>
  <tr>
    <td>Temperature</td>
    <td>1.15</td>
  </tr>
  <tr>
    <td>N</td>
    <td>3</td>
  </tr>
  <tr>
    <td># Raw Generator Corpus</td>
    <td>49000</td>
  </tr>
  <tr>
    <td># Raw Critic Corpus</td>
    <td>6000</td>
  </tr>
  <tr>
    <td colspan="2"><i>Training Hyperparameters</i></td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>5e-5</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>128</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>LR Decay</td>
    <td>Cosine Annealing</td>
  </tr>
  <tr>
    <td>LoRA Rank</td>
    <td>128</td>
  </tr>
  <tr>
    <td>LoRA Alpha</td>
    <td>256</td>
  </tr>
  <tr>
    <td>LoRA Targets</td>
    <td>All Linear Layers</td>
  </tr>
  <tr>
    <td colspan="2"><i>Inference Hyperparameters</i></td>
  </tr>
  <tr>
    <td>Temperature</td>
    <td>1.15</td>
  </tr>
  <tr>
    <td>Repetition Penalty</td>
    <td>1.2</td>
  </tr>
  <tr>
    <td># Docs</td>
    <td>5 or 6 (hybrid retrieval)</td>
  </tr>
  <tr>
    <td># Decomposition Branches</td>
    <td>2</td>
  </tr>
  <tr>
    <td># Candidates</td>
    <td>2</td>
  </tr>
  <tr>
    <td># Retrieval Reward</td>
    <td>1.0 / 0.5 / -1.0</td>
  </tr>
  <tr>
    <td># Reasoning Reward</td>
    <td>0.5 / 0.25 / -0.5</td>
  </tr>
</table>

Table 9: Hyperparameters in experiments.

optimization. We use vLLM\footnote{https://docs.vllm.ai/en/latest/} in the inference. Hyperparameters are provided in Table 9.
B Supplemental Materials on Analysis

<table>
  <tr>
    <th>Methods</th>
    <th>2WikiMQA</th>
    <th>HotpotQA</th>
    <th>MuSiQue</th>
  </tr>
  <tr>
    <td>Self-Ask</td>
    <td>2.72</td>
    <td>2.28</td>
    <td>2.48</td>
  </tr>
  <tr>
    <td>IRCoT</td>
    <td>3.78</td>
    <td>2.86</td>
    <td>2.44</td>
  </tr>
  <tr>
    <td>Self-RAG</td>
    <td>2.05</td>
    <td>2.12</td>
    <td>2.13</td>
  </tr>
  <tr>
    <td>Auto-RAG</td>
    <td>2.67</td>
    <td>2.39</td>
    <td>3.74</td>
  </tr>
  <tr>
    <td>DR-Distillation</td>
    <td>3.33</td>
    <td>2.85</td>
    <td>3.07</td>
  </tr>
  <tr>
    <td>SiGIR</td>
    <td><b>2.40</b></td>
    <td><b>2.01</b></td>
    <td><b>2.37</b></td>
  </tr>
</table>

Table 10: The average number of iterations of iterative reasoning methods to solve a multi-hop question. Our method requires fewer iterations to solve a multi-hop question, resulting in higher reasoning efficiency.

<table>
  <tr>
    <th>Question Type</th>
    <th>Bdg.</th>
    <th>Infer.</th>
    <th>Cmp.</th>
    <th>Bdg-Cmp.</th>
  </tr>
  <tr>
    <td>Latency(s)</td>
    <td>2.41</td>
    <td>2.56</td>
    <td>2.91</td>
    <td>5.56</td>
  </tr>
  <tr>
    <td>Prompt</td>
    <td>2989.4</td>
    <td>2775.0</td>
    <td>1899.1</td>
    <td>4653.9</td>
  </tr>
  <tr>
    <td>Completion</td>
    <td>171.5</td>
    <td>202.4</td>
    <td>209.3</td>
    <td>375.2</td>
  </tr>
</table>

Table 11: Per-sample inference latency and token consumption of greedy inference on 2WikiMQA dataset.

<table>
  <tr>
    <th>Question Type</th>
    <th>Bridge</th>
    <th>Comparison</th>
  </tr>
  <tr>
    <td>Latency(s)</td>
    <td>3.71</td>
    <td>2.81</td>
  </tr>
  <tr>
    <td>Prompt</td>
    <td>2156.6</td>
    <td>1786.6</td>
  </tr>
  <tr>
    <td>Completion</td>
    <td>182.7</td>
    <td>220.3</td>
  </tr>
</table>

Table 12: Per-sample inference latency and token consumption of greedy inference on HotpotQA dataset.

<table>
  <tr>
    <th>Question Type</th>
    <th>2hop</th>
    <th>3hop</th>
    <th>4hop</th>
  </tr>
  <tr>
    <td>Latency(s)</td>
    <td>2.43</td>
    <td>3.41</td>
    <td>4.67</td>
  </tr>
  <tr>
    <td>Prompt</td>
    <td>2388.9</td>
    <td>4162.6</td>
    <td>4294.4</td>
  </tr>
  <tr>
    <td>Completion</td>
    <td>182.1</td>
    <td>258.2</td>
    <td>333.8</td>
  </tr>
</table>

Table 13: Per-sample inference latency and token consumption of greedy inference on MuSiQue dataset.

noteworthy that multiple retrieval calls can occur within a single reasoning iteration. The specific number of reasoning iterations corresponding to each method can be found in Table 10. Additionally, we present the inference latency of individual samples with greedy inference, along with the average prompt and completion token costs, as shown in Table 11, Table 12, and Table 13.

B.1 Reasoning Efficiency between Separated Critic and Generative Critic

As mentioned in §6.4, GenCritic achieves a higher throughput compared to Separated Critic. We will describe how to calculate the throughput in this section. We use the same computing platform, equipped with one A100-SXM4-80GB GPU, a Xeon(R) 6348 CPU with 28 cores and 56 threads, and 480GB of memory. We sample 500 instances from each of the three datasets, totaling 1,500 instances. We conduct parallel inference using the API server provided by vLLM, employing reward-guided greedy inference during benchmarking. The throughput is calculated based on the time consumed for different methods.

B.2 Iterative Reasoning Efficiency

We discuss the iterative reasoning efficiency in §6.6. Specifically, we consider a complete round of retrieval-generation as one reasoning iteration. It is
<table>
  <tr>
    <th>Notation</th>
    <th>Description</th>
  </tr>
  <tr>
    <th colspan="2">(a) Modules</th>
  </tr>
  <tr>
    <td><i>LLM</i></td>
    <td>The advanced large language model to synthesis iterative reasoning rationales and process supervision.</td>
  </tr>
  <tr>
    <td><i>R</i></td>
    <td>The iterative reasoner model.</td>
  </tr>
  <tr>
    <td><i>C</i></td>
    <td>The separated critic model.</td>
  </tr>
  <tr>
    <td><i>R_{sc}</i></td>
    <td>The iterative reasoner model with self-critique capabilities.</td>
  </tr>
  <tr>
    <td>Retr</td>
    <td>External retriever (sparse, dense, hybrid)</td>
  </tr>
  <tr>
    <th colspan="2">(b) Training Corpus</th>
  </tr>
  <tr>
    <td><i>X</i></td>
    <td>Raw training corpus for generator.</td>
  </tr>
  <tr>
    <td><i>X_{critic}</i></td>
    <td>Raw training corpus for critic, which is disjoint with <i>X</i>.</td>
  </tr>
  <tr>
    <th colspan="2">(c) Symbols</th>
  </tr>
  <tr>
    <td>\( q_i^{(r)} \)</td>
    <td>Multi-hop question after reduction.</td>
  </tr>
  <tr>
    <td>\( q_i \)</td>
    <td>Decomposed atomic sub-question.</td>
  </tr>
  <tr>
    <td>\( d_i \)</td>
    <td>The retrieved document.</td>
  </tr>
  <tr>
    <td>\( x \)</td>
    <td>Reasoning trajectory.</td>
  </tr>
  <tr>
    <td>\( s_i \)</td>
    <td>Sub-question reasoning.</td>
  </tr>
  <tr>
    <td>\( r_i^{\text{retr}} \)</td>
    <td>Reward for retrieval relevance.</td>
  </tr>
  <tr>
    <td>\( r_i^{\text{reas}} \)</td>
    <td>Reward for reasoning utility.</td>
  </tr>
  <tr>
    <td>\( r_i^c \)</td>
    <td>Cumulative process reward.</td>
  </tr>
  <tr>
    <td>\( r_i^o \)</td>
    <td>Outcome reward.</td>
  </tr>
</table>

Table 14: The formal notations mentioned in Section 3.
C  Prompts

We provide the manually annotated prompts for data synthesis. For iterative reasoning rationales, we use 1-shot prompts for each question type (Figure 5). For critic on retrieval and reasoning, we use 3-shot prompts (Figure 6, 7). For critic on outcome reward, we use zero-shot prompts (Figure 8).

Additionally, we provide an example of iterative reasoning with self-critique, shown in Figure 15.
Algorithm 1 Self-Critique Guided Iterative Reasoning
Require: Initial question \( q_0 \), retrieval model Retr, probabilistic language model \( p \), candidate size \( k \)
Ensure: Final answer \( a^* \)

Initialize:
1: \( x_0 \leftarrow [q_0] \) \( \triangleright \) Initial reasoning prefix
2: \( r_0^c \leftarrow 0 \) \( \triangleright \) Initial c reward
3: \( x_0 \leftarrow \{ (x_0, r_0^c) \} \) \( \triangleright \) Set of candidate trajectories

Main Loop:
4: for \( t = 0 \) to \( T - 1 \) do
5: \( x_{t+1} \leftarrow \varnothing \) \( \triangleright \) \( T \) is the maximum number of iterations
\( \triangleright \) Initialize set for next iteration

Branch Exploration:
6:   | for \( (x_t, r_t^c) \) in \( x_t \) do
7:     Decomposition:
8:       \( q_{t+1} \sim p(y \mid q_t^{(r)}, x_t) \) \( \triangleright \) Sample sub-question
9:     Retrieval:R
10:    \( d_i \leftarrow \operatorname{Retr}(q_{t+1}) \) \( \triangleright \) Retrieve relevant documents
11:   \( r_{t+1}^{\text{retr}} \leftarrow p(y \mid q_{t+1}, d_i) \) \( \triangleright \) Compute retrieval reward
12:  Reasoning:
13: \( s_{t+1} \sim p(y \mid q_{t+1}, d_i, x_t) \) \( \triangleright \) Sample reasoning step
14: \( r_{t+1}^{\text{reas}} \leftarrow p(y \mid q_{t+1}, d_i, s_{t+1}) \) \( \triangleright \) Compute reasoning reward
15: Update Reasoning Prefix:
16: \( x_{t+1} \leftarrow x_t \oplus [q_{t+1}, d_i, r_{t+1}^{\text{retr}}, s_{t+1}, r_{t+1}^{\text{reas}}] \) \( \triangleright \) Concatenate current status
17: Update Cumulative Reward:
18: \( r_{t+1}^c \leftarrow r_t^c + r_{t+1}^{\text{retr}} + r_{t+1}^{\text{reas}} \)
19: Question Reduction:
20: \( q_t^{(r)} \leftarrow \operatorname{ReduceQuestion}(x_t) \) \( \triangleright \) Reduce remaining question
21: \( x_{t+1} \leftarrow x_{t+1} \cup \{ (x_{t+1}, r_{t+1}^c) \} \) \( \triangleright \) Add new candidate

Candidate Selection:
23: \( x_{t+1} \leftarrow \operatorname{Top-k}(x_{t+1}, r_{t+1}^c) \) \( \triangleright \) Select top-k candidates

Final Answer Selection:
25: \( a^* \leftarrow \arg\max_a \left( \sum_{i=1}^n (r_i^{\text{retr}} + r_i^{\text{reas}}) \right) \) \( \triangleright \) Select answer with highest c reward
26: return \( a^* \)
Algorithm 2 Training SC-Reasoner

Require: Generator corpus \( X = \{x_i\} \), where \( x_i = (q_i, a_i, \{d_1, \ldots, d_n\}) \)
Require: Non-overlapping critic corpus \( X_{critic} \)
Require: Large Language Model (LLM)
Ensure: Self-Critique Reasoner \( R_{sc} \)

Phase 1: Train Iterative Reasoner \( R \)
1: \( X_{synthetic} \leftarrow \varnothing \) \( \triangleright \) Initialize synthetic reasoning corpus
2: for each instance \( x_i \in X \) do
3:   Step 1.1: Synthesize Reasoning Rationales
4:     decomp_i, reasoning_i \( \leftarrow \) LLM(\( q_i, \{d_1, \ldots, d_n\} \)) \( \triangleright \) Few-shot prompting for decomposition and reasoning
5:     trajectory_i \( \leftarrow \) Organize(decomp_i, reasoning_i) \( \triangleright \) Organize rationales
6:     \( X_{synthetic} \leftarrow X_{synthetic} \cup \{\text{trajectory}_i\} \)
7: end for
8: Step 1.2: Train Iterative Reasoner
9: \( R \leftarrow \) TrainModel(\( X_{synthetic} \)) \( \triangleright \) Train \( R \) on synthetic corpus

Phase 2: Train Critic Model \( C \)
11: \( X_{critic-data} \leftarrow \varnothing \) \( \triangleright \) Initialize critic training data
12: for each instance \( x_i \in X_{critic} \) do
13:   Step 2.1: Generate Reasoning Trajectories
14:     trajectory_i \( \leftarrow R(q_i, \{d_1, \ldots, d_n\}) \) \( \triangleright \) Use \( R \) to generate reasoning trajectories
15:   Step 2.2: Evaluate Rewards with LLM
16:     \( r_i^{\text{retr}}, r_i^{\text{reas}}, r_i^{\text{overall}} \leftarrow \) LLM(trajectory_i) \( \triangleright \) Evaluate retrieval, reasoning, and overall quality
17:     \( X_{critic-data} \leftarrow X_{critic-data} \cup \{(trajectory_i, r_i^{\text{retr}}, r_i^{\text{reas}}, r_i^{\text{overall}})\} \)
18: end for
19: Step 2.3: Train Critic Model
20: \( C \leftarrow \) TrainModel(\( X_{critic-data} \)) \( \triangleright \) Train \( C \) on critic data

Phase 3: Train Self-Critique Reasoner \( R_{sc} \)
22: \( X_{sc-data} \leftarrow \varnothing \) \( \triangleright \) Initialize self-critique training data
23: for each instance \( x_i \in X \) do
24:   Step 3.1: Annotate Reasoning Trajectories with Critic
25:     trajectory_i \( \leftarrow R(q_i, \{d_1, \ldots, d_n\}) \) \( \triangleright \) Generate reasoning trajectories with \( R \)
26:     \( r_i^{\text{retr}}, r_i^{\text{reas}}, r_i^{\text{overall}} \leftarrow C(\text{trajectory}_i) \) \( \triangleright \) Annotate with critic \( C \)
27:     trajectory_i^{annotated} \( \leftarrow \) AddSpecialTokens(trajectory_i, \( r_i^{\text{retr}}, r_i^{\text{reas}}, r_i^{\text{overall}} \))
28:     \( X_{sc-data} \leftarrow X_{sc-data} \cup \{\text{trajectory}_i^{annotated}\} \)
29: end for
30: Step 3.2: Train Self-Critique Reasoner
31: \( R_{sc} \leftarrow \) TrainModel(\( X_{sc-data} \)) \( \triangleright \) Train \( R_{sc} \) on annotated corpus

Self-Improvement Pipeline
33: while not converged do
34:   Step 4.1: Generate Synthetic Rationales
35:     \( X_{new} \leftarrow R_{sc}(\text{unlabeled questions}) \) \( \triangleright \) Generate rationales for unlabeled questions
36:   Step 4.2: Evaluate Rationale Quality
37:     \( X_{high-quality} \leftarrow \{x \in X_{new} \mid C(x) > \text{threshold}\} \) \( \triangleright \) Filter high-quality rationales
38:   Step 4.3: Retrain \( R_{sc} \)
39:     \( R_{sc} \leftarrow \) TrainModel(\( R_{sc}, X_{high-quality} \)) \( \triangleright \) Retrain with high-quality data
40: end while
41: return \( R_{sc} \)
<table>
  <tr>
    <th colspan="2">Iterative Reasoning Examples of Self-Critique Reasoner</th>
  </tr>
  <tr>
    <th>Q</th>
    <td>How many counties are there in the largest state in the region where the film The Family Stone was set? (Answer: sixteen)</td>
  </tr>
  <tr>
    <td>[Non-Atomic Question]<br><sub-question>In which region is the film "The Family Stone" set?<br><paragraph>The Family Stone ...</paragraph><br>Retrieval Quality: [Partially Relevant]<br>Reasoning: From Document, we know that "The Family Stone" is set in a small New England town. Therefore, the region is New England.<br></sub-question> Reasoning Quality: [Fully supported]<br>[Remaining Question] How many counties are there in the largest state in New England?</td>
  </tr>
  <tr>
    <td>[Non-Atomic Question]<br><sub-question>Which state is the largest in New England?<br><paragraph>New England ...</paragraph><br>Retrieval Quality: [Partially Relevant]<br>Reasoning: From Document, we know that Maine is the largest state in New England, constituting nearly one-half of the total area of New England.<br></sub-question> Reasoning Quality: [Fully supported]<br>[Remaining Question] How many counties are there in Maine?</td>
  </tr>
  <tr>
    <td>[Atomic Question]<br><paragraph>List of counties in Maine ...</paragraph><br>Retrieval Quality: [Relevant]<br>Reasoning: From Document, we know that there are sixteen counties in Maine.<br>Reasoning Quality: [Fully supported]<br>[Final Answer] @ @<b>sixteen</b>@ @<br>Overall Reasoning Quality: [4]</td>
  </tr>
</table>

Table 15: A reasoning example of self-critique reasoner. The data instance is from MuSiQue dataset.
Prompts for Iterative Reasoning Rationales

Please answer the following multi-hop questions with explanation based on the given documents. Wrap the final answer with @@ @@. Here is an example:

### Document #1: Cuidado con las imitaciones
Cuidado con las imitaciones is a 1948 Argentine film directed by Luis Bayón Herrera.

### Document #2: Luis Bayón Herrera
Luis Bayón Herrera (23 September 1889 – 30 March 1956) was a Spanish film director and screenwriter who worked in Argentine film of the 1940s and 1950s. He was one of the most important directors of the golden age of Argentine cinema. Herrera was born in Bilbao, Spain. He directed some 40 different Argentine feature films and in the last few years of his career in the early 1950s he worked on Cuban production with films such as À La Habana me voyín 1951. He died in Buenos Aires in 1956, aged 66."

### Question: What nationality is the director of film Cuidado Con Las Imitaciones?

To answer this multi-hop question, we need to solve the following sub-questions:

### Decompose the multi-hop question into sub-questions
1. Who is the director of the film Cuidado Con Las Imitaciones?
2. What is the nationality of that director?

Next, let’s solve the sub-questions one by one.

### Sub-question: Who is the director of the film Cuidado Con Las Imitaciones?
From Document #1, we know that "Cuidado Con Las Imitaciones" is a Spanish film, and its director is José María Forqué.

### Remaining Question: What is the nationality of José María Forqué?
From Document #2, we know that José María Forqué was a Spanish director, which means his nationality is Spanish.

### Final Answer
@@Spanish@@

{SUPPORTING DOCUMENTS}

### Question: {MULTI-HOP QUESTION}
Prompts for Retrieval Relevance Evaluation

You will be given a question and an evidence document. Your task is to evaluate whether the evidence is relevant to the question and provide a clear explanation for your decision. Use the following scale to rate the relevance of the evidence:
[Relevant]: The evidence directly addresses or supports the question.
[Partially Relevant]: The evidence is somewhat related to the question but does not fully address it. For example, it may mention the core entity or topic of the question.
[Irrelevant]: The evidence does not relate to the question in any meaningful way.

### Question: In what country was the first Pan-African conference held?

### Evidence: First Pan-African Conference
The First Pan-African Conference was held in London from 23 to 25 July 1900 (just prior to the Paris Exhibition of 1900 in order to allow tourists of African descent to attend both events ). Organized primarily by the Trinidadian barrister Henry Sylvester Williams, it took place in Westminster Town Hall (now Caxton Hall) and was attended by 37 delegates and about 10 other participants and observers from Africa, the West Indies, the US and the UK, including Samuel Coleridge Taylor (the youngest delegate), John Alcindor, Dadabhai Naoroji, John Archer, Henry Francis Downing, and W.E.B. Du Bois, with Bishop Alexander Walters of the AME Zion Church taking the chair.

Rating: [Relevant]
Explanation: The evidence directly answers the question by stating that the first Pan-African conference was held in London, which is in the United Kingdom. The details provided, such as the dates and location of the conference, further confirm the relevance of the evidence to the question.

### Question: What is the administrative territorial entity for Malta Township?

### Evidence: Minsk Region
Minsk Region or Minsk Voblasć or Minsk Oblast (, "Minskaja voblasć" ; , "Minskaja oblastj") is one of the regions of Belarus. Its administrative center is Minsk, although it is a separate administrative territorial entity of Belarus. As of 2011, the region's population is 1,411,500.

Rating: [Irrelevant]
Explanation: The evidence provided discusses the Minsk Region, which is a region in Belarus. The question, however, asks about the administrative territorial entity for Malta Township. Since the evidence pertains to Belarus and not Malta, it is not relevant to the question.

### Question: Did Bob Rosa produce any work for Patrick Love?

### Evidence: Bob Rosa
Bob Rosa is an American record producer that has worked with artists such as Madonna, Janet Jackson, Mariah Carey and Espen Lind. One if his first recording projects was working on the track "Planet Rock" by Afrika Bambaataa & the Soulsonic Force in 1982.

Rating: [Partially Relevant]
Explanation: The evidence mentions Bob Rosa's work with various artists, including Madonna, Janet Jackson, Mariah Carey, and Espen Lind. However, it does not specifically mention Patrick Love or any work Bob Rosa may have done for him. Therefore, the evidence is partially relevant to the question.

### Question: {}

### Evidence: {}
<table>
  <tr>
    <th>Prompts for Reasoning Utility Evaluation</th>
  </tr>
  <tr>
    <td>You will be provided with a question, an evidence document, along with a response. Your job is to determine whether the response is supported by the evidence, and provide explanation for your decision. Use the following scale to rate the response:<br>[Fully supported] Most of the information in the response is supported by the evidence.<br>[Partially supported] Some of the information in the response is supported by the evidence, but there are some parts that are speculative.<br>[Not supported] The response is not supported by the evidence / The response does not provide useful reasoning.</td>
  </tr>
</table>

### Question: In which country is the Financial Conduct Authority located?

### Evidence: Martin Wheatley
Martin Wheatley is a British financier, formerly managing director of the Consumer and Markets Business Unit of the Financial Services Authority in the UK, and is the former CEO of the Financial Conduct.

### Response: From Document, we know that Martin Wheatley is a British financier who formerly managed the Consumer and Markets Business Unit of the Financial Services Authority in the UK. This indicates that the Financial Conduct Authority is located in the United Kingdom.

Rating: [Fully supported]
Explanation: The response is fully supported by the evidence. The evidence clearly states that Martin Wheatley is a British financier who formerly managed the Consumer and Markets Business Unit of the Financial Services Authority in the UK and was the former CEO of the Financial Conduct Authority. The response accurately extracts this information and concludes that the Financial Conduct Authority is located in the United Kingdom, which is directly supported by the evidence provided.

### Question: What is the birthplace of Jane Siberry?

### Evidence: Gavin Bradley
Gavin Bradley is an award-winning Canadian singer-songwriter and producer based in Toronto who has worked with artists like Nelly Furtado, Tori Amos and Jane Siberry. Fusing acoustic and electronic elements, his work is identifiable for its signature "warm" piano sound and live strings mixed with filtered synthesizers and other electronic manipulations. Besides production, Bradley is a solo recording artist. His debut album 'Deep Freeze' was released on UMI Records in 2006.

### Response: The response is partially supported because the evidence mentions Gavin Bradley’s Toronto connection but does not directly state Jane Siberry’s birthplace. The inference about her possible connection to Toronto is speculative.

Rating: [Partially supported]
Explanation: The response correctly identifies that the evidence mentions Gavin Bradley’s connection to Toronto but does not directly state Jane Siberry’s birthplace. The inference that Jane Siberry might be connected to Toronto based on her collaboration with Gavin Bradley is speculative, as the evidence does not provide explicit information about her birthplace. Therefore, the response is partially supported.

### Question: {}

### Evidence: {}

### Response: {}
<table>
  <tr>
    <th>Prompts for Outcome Reward Evaluation</th>
  </tr>
  <tr>
    <td>You will be provided with a question, along with a reasoning trajectory.<br>Your job is to determine whether the reasoning trajectory is useful for answering the question, and provide explanation for your decision.<br>Use the following scale to rate the reasoning:<br>[5]: The reasoning process is clear, logically structured, and well-supported by the evidence.<br>[4]: The reasoning process is mostly clear, partially logically structured, and supported by evidence, but may contains minor logical flaws.<br>[3]: The reasoning is somewhat unclear with noticeable flaws and uses limited or weak evidence.<br>[2]: The reasoning is flawed, lacking supporting evidence, which results in an incorrect conclusion.<br>[1]: The reasoning fails to provide a definitive answer.<br><br>### Question: {}<br><br>### Reasoning: {}<br><br>Rating:<br>Explanation:</td>
  </tr>
</table>

Figure 8: The prompts for reasoning trajectories outcome reward evaluation.

Reviewer paper summaries: This paper proposes SiGIR, a Self-Critique Guided Iterative Reasoning approach to alleviate the flawed intermediate reasoning problem. SiGIR first uses LLMs to synthesize reasoning and evaluation data, then trains a model supporting reasoning and evaluation (SC-Reasoner), and finally performs Self-Critique Guided Iterative Reasoning based on SC-Reasoner. Extensive experiments on three multi-hop reasoning datasets demonstrate the effectiveness of SiGIR.

The SiGIR (Self-Critique Guided Iterative Reasoning) framework proposed in this paper enhances the accuracy and efficiency of multi-hop question-answering tasks through the synergistic optimization of hybrid retrieval and a self-critical mechanism. 

Its core contributions include: iterative path optimization achieved by dynamically evaluating retrieval quality (r_retr) and reasoning rationality (r_reas) (e.g., requiring only 2.4 iterations on HotpotQA), augmenting training with synthetic data generated by large language models (achieving 95% of full-data performance using only 40% of the data), and demonstrating broad applicability across multiple model architectures, such as Mistral-7B. Experimental results show that SiGIR achieves state-of-the-art performance on benchmarks like HotpotQA and MuSiQue (e.g., a 15% improvement on 4-hop questions in MuSiQue). This method provides an efficient and scalable solution for knowledge-intensive tasks, particularly showcasing robustness in complex multi-hop reasoning scenarios.
Summary of strengths: 1. The figures and tables in this paper are well-designed, and the authors put significant effort into constructing a comprehensive pipeline for LLM-based reasoning. 
2. The experimental results in this paper are comprehensive.

1. SiGIR achieves efficient reasoning in multi-hop question answering (e.g., reaching 81.2 F1 on HotpotQA) through a self-critical iterative mechanism, while also demonstrating cross-architecture generalizability across different models such as Mistral-7B and LLaMA2-7B.
2. The method is still able to achieve performance close to that of using the full dataset (Table 3) in low-resource scenarios (with only 40% of the training data), demonstrating its data efficiency and generalization capability.
3. This paper is well-written and demonstrates a good level of overall quality.
Summary of weaknesses: 1. The method proposed in this paper is not exhaustive. For example, in Iterative Reasoner with Self-Critique, it is unclear how atomic question decomposition is performed or how self-evaluation for retrieval and reasoning is conducted.
2. This paper appears to draw on the concepts of process-supervised reward models (PRMs) and outcome-supervised reward models (ORMs). However, the authors do not clearly describe how rewards are used for model optimization, which is a crucial aspect of PRMs and ORMs.
3. Performing atomic question decomposition and evaluation based on the Self-Critique method itself raises concerns about cascading errors caused by hallucinations during this process.
4. Resource consumption needs to further analyze.

1. Although SC-Reasoner achieves performance improvements through training, its essence still relies on the 'knowledge transfer' from external large language models, rather than a fully autonomous evolution of reasoning capabilities. This design may lead to the model's failure in real-world scenarios due to data distribution shifts or limitations of the LLMs, and it also disconnect with the "self" aspect claimed in the paper.
2. The hybrid retrieval setup of SC-Reasoner (BM25+Contriever) differs from baseline methods like GenGround's ColBERTv2 or RAG-Star's BGE-large. This difference might blur the line between the retriever's impact and the method's own contribution, so direct comparisons with reported baseline results unfair. It would be better if the authors included experiments under the same configuration to more clearly demonstrate SC-Reasoner's effectiveness.
3. In Table 9 mentions that SC-Reasoner reduces the number of iterations compared to the baselines and claims "higher reasoning efficiency." However, the experimental section (e.g., Appendix A.7) only mentions the GPU models used for training and inference without providing critical data on computational resource consumption, such as training time, peak memory usage, single-sample inference latency, input/output token counts, or the number of LLM calls. These metrics are essential to comprehensively verify the actual cost of the claimed efficiency improvements.
Comments, suggestions and typos: 1. PRMs and ORMs have been successfully applied to mitigate cascading errors in intermediate reasoning processes [1,2]. Further analysis of the differences between the proposed method and these studies would help readers gain a deeper understanding.
2. Algorithm 2 leverages an LLM to train the Self-Critique Reasoner, and the hallucination issues arising in this process are concerning. Further analysis of the proposed solution would be helpful.
3. I notice that the results of ProbTree and BeamAggR are derived from the original paper, but why are these results inconsistent with those reported in the original paper?
4. Some concepts require further explanation: "fine-grained rewards" and "coarse-grained binary rewards" (Table 2); Equ. (2,3) (Line 304). Additionally, in Line 309, what operation is performed in Equation (5), i.e., $\oplus$? What does $d$ represent, which specific document or documents?
5. The $r^c$ in Equation (9) and the $r^c_t$ in Equation (7) are logically disconnected.
6. I have listed some of the questions in the Weaknesses. 
7. Some other issues:
- A punctuation mark is missing (Line 048). 
- Lines 232-234 are missing citations.
- In Table 2, "SGIR" may be intended as "SiGIR"??

Reference:
[1] Let's Verify Step by Step. ICLR2024. 

[2] Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. ACL2024.

This paper does not provide the software and data necessary for reproduction. It is recommended to include such content in future versions to enhance the verifiability and credibility of the research.
Overall assessment scores: 3

2.5