Optimizing Inference-Time Reasoning in LLMs via Retrieval-Augmented Reflection

Authors: Zihao Wang, Haowei Lin, Ruilin Yan, Xiangyu Wang, Jiaqi Li, Weiye Shi, weipeng chen, Xiaojian Ma, Anji Liu, Yitao Liang
Primary area: foundation or frontier models, including LLMs
Keywords: Retrieval-augmented Generation, Reasoning, Large Language Models

Abstract:
Empowering LLMs to improve their performance through increased inference-time computation is a crucial step in developing self-improving agents capable of operating in open-ended natural language contexts. In this paper, we explore how iteratively revising a chain of thoughts guided by information retrieval significantly improves large language models' reasoning ability in challenging tasks, while hugely mitigating hallucination. In particular, the proposed method --- \emph{retrieval-augmented reflection} (RaR) --- revises the generation tokens step by step, leveraging multiple pieces of retrieved information relevant to the intermediate reasoning steps and the instruction. Applying RaR during inference-time to a various set of language models substantially improves their performances on various reasoning tasks; on relatively increasing scores by up to +16.4\% on code generation, +11.6\% on mathematical reasoning, and 29.1\% on embodied task planning. Moreover, we find that with more inference-time computation given to the LLM for multi-times retrieval-augmented reflection, the LLM can continuously improve on various reasoning benchmarks. A small LM can surpass the performance of the LM with more than 10 times the parameters, when giving more computation cost.

Decision: reject
Reviewer scores: 3, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1729068405229

thread_type:
official_review

<thread_object>
creator: reviewer_ux3s

<content>
summary:
This paper proposes RaR (Retrieval-augmented Reflection), which introduces retrieval-augmented reflection in multi-step reasoning, combining relevant information from external retrieval to correct the intermediate reasoning steps, thereby improving the model's reasoning performance.

Specifically, the proposed method first uses Zeroshot CoT to generate a step-by-step reasoning trajectory, in which may contain erroneous parts.
Subsequently, the paper attempts to improve the reasoning by retrieving relevant information from an external knowledge base through a retrieval-augmented approach, and using this information to correct any potential errors in the reasoning.
Chain-of-thought reasoning is step-by-step, and to achieve more fine-grained and precise corrections, this paper proposes to correct the reasoning steps one by one from the beginning to the end.
At each step of reasoning verification, relevant external information needs to be retrieved. Assuming that the current step i needs to be corrected, the question and reasoning steps 1 through i are used as the retrieval query to retrieve relevant materials, and the retrieved content is used to correct the reasoning.
The paper conducted experiments on tasks such as code generation, mathematical reasoning, and knowledge-intensive question answering.

strengths:
1. The paper proposes a reasoning framework that combines retrieval-augmented and reflection, called RaR (Retrieval-augmented Reflection), which can correct potential errors in the reasoning process based on external knowledge.
2. The paper proposes a multi-round reflection method, where each round only retrieves and reflects on the current step, thereby more accurately identifying the location of errors. On another dimension, this approach can increase computational FLOPs during reasoning, allowing for inference-time scaling based on the inference scaling laws.
3. The paper conducted experiments on several datasets, including tasks such as code generation, mathematical reasoning, and question answering, demonstrating the effectiveness of the method.

weaknesses:
1. The writing of method part in the paper (Line 216~Line 239) is somewhat vague. While I can understand the general idea of the proposed method, I cannot accurately grasp the specific details through the formulas and descriptions, and clarification from the authors is needed.
2. The author claims that by expanding n (the number of reasoning steps) to achieve Inference Scaling. However, this dimension seems not easy to expand. On one hand, the number of reasoning steps generated by Zeroshot CoT is not entirely controllable.  On the other hand, for a specific problem, the number of reasoning steps is clearly limited. These two aspects make it less feasible and scalable to increase the number of reasoning steps in order to increase the number of verification rounds.
3. The QA dataset used in this paper is TriviaQA, which only requires single-hop retrieval and reasoning to complete. Given the author's motivation to use retrieval to improve multi-step reasoning, using multi-hop reasoning datasets such as MuSiQue, 2WikiMHQA, etc., seems to be a more appropriate choice.

questions:
1. In the paragraph from Line 220 to Line 224, the description and the formula on Line 224 seem to be incorrect or unclear. It is difficult to understand what the formula is intended to express.
2. What are the differences between the two sets of formulas on Line 219, 224, and Line 233~235?
3. The mathematical reasoning in the article uses simple arithmetic reasoning, with the GSM8K and GSM-Hard datasets. Unlike MATH, which involves some advanced mathematical knowledge, these two datasets consist of simple multi-step arithmetic operations and basic linear equations, without involving external knowledge. Secondly, the retrieval for mathematical reasoning uses Jupyter code corpus, which does not seem to be helpful for solving mathematical reasoning problems.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730086309319

thread_type:
official_review

<thread_object>
creator: reviewer_pozp

<content>
summary:
This paper uses LLMs to generate chains-of-thought (CoT) for coding, math, and planning tasks, then uses RAG to iteratively improve each individual step in the original CoT multiple times to improve performance at the cost of extra compute and time during inference.  Their method outperforms many other baselines (including RAG, CoT, Self-Refine, and others) and also shows performance increases as models scale in parameters and more iterations of refinements are done. Finally, they show that refining a CoT with a single prompt in one generation is not as powerful as their method, where you fix each individual step of the CoT one by one.

strengths:
1. Strong results across many commonly used baselines showing their method (RaR) outperforms all of them.
2. Timely and well-motivated, I think scaling during inference is probably the easiest way to get better outputs from closed-source LLMs right now.

weaknesses:
1. Lack of an important baseline, RaG + CoT. There already are a ton of baselines here, but I do feel that RAG + CoT is a very close baseline to the method proposed and is a very common one that I think people would want to see before using RaR.  I think this baseline would also highlight (similarly to CoT+RAG in Table 3) that even with all the documents/passages required to answer a question, the LMs still benefit from iterative refinement.  This would be a big win for the paper and make it extremely clear to other researchers why RaR should be used.  

2. More thorough analysis of why the baselines are failing.  Tables 3 and 4 are good ablations showing the need for iteratively refining each individual step without showing the full CoT, but I think the paper would also benefit from detailing where traditional methods are failing.  For example, if we used CoT+RAG or RAG+CoT with all the same documents retrieved in RAR, would we get similar performances?  I think more discussion on where RaR is outperforming the other baselines would help researchers understand why RaR is an effective method and where it can be improved for future work.

questions:
- Have you tried matching the amount of compute for RAR with other baselines like Self-refine?
- Have you tried a baseline similar to RAR but without RAG? (Just asking the LLM to verify/fix steps with no additional context to establish how important the retrieved documents are)
- "With lower inference-time computation (FLOPs), a small LM can surpass the performance of the LM with more than 10 times parameters." I wasn't sure what this meant in the abstract.  Are you saying a small LM can outperform a larger one when there is a large compute budget and RAR is used?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730285105950

thread_type:
official_review

<thread_object>
creator: reviewer_yr3i

<content>
summary:
Many research efforts focus on correcting misinformation generated by language models through retrieval augmentation. This paper explores how iteratively revising a chain of thoughts with the help of information retrieval significantly improves large language models' reasoning abilities across various tasks, such as code generation, mathematical reasoning, and embodied planning. Experimental results show that, compared to methods like language model self-consistency and simple retrieval augmentation, the proposed RaR framework is more effective in mitigating hallucinations in the content generated by models.

strengths:
This paper explores iteratively revising a chain of thoughts with the help of information retrieval to improve large language models' reasoning abilities. Its strengths include:

1. The writing in the paper is clear, and the figures are intuitive, effectively conveying the main ideas and supporting the overall arguments.

2. The method proposed by this paper incorporates a recursive correction mechanism by revising each thought step based on previously refined steps, allowing for continual consultation of relevant information. This significantly improves the accuracy and reliability of generated outputs compared to traditional RAG methods.

3. RaR is flexible in handling various tasks, such as code generation, mathematical reasoning, and embodied planning.

weaknesses:
1. The baseline methods compared in Table1 are weak, and the proposed approach has not been compared with the latest related RAG research, which does not demonstrate the significant differences between RaR and existing work (Active-RAG, IRCoT, ...).

questions:
1. Is the number of modification iterations in RaR related to the number of sentences in the answers? In that case, when comparing with RAG, is the number of retrievable instances in RAG consistent with that of RaR?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665113364

thread_type:
official_review

<thread_object>
creator: reviewer_xhx9

<content>
summary:
This paper studies the problem of leveraging RaG to improve LLMs’ reasoning capabilities, and proposes a new approach called Retrieval-augmented Reflection (RaR). RaR first generates an initial solution, and then alternates the process retrieval and revision to revise each step in the initial solutions. The paper evaluates the proposed approach on three domains and compares against a series of zero-shot baselines. The results suggest that the method can improve LLM performance on code generation and embodied planning tasks.

strengths:
The results presented in the paper suggest good empirical performance.

The paper evaluates the method on multiple datasets spanning several domains.

weaknesses:
I have several concerns, mainly regarding the evaluation. The comparison is probably unfair, which overstates the performance gains. Additionally, insufficient detail about the evaluation setup makes it difficult to validate the results. Specifically:

First, the comparison is unfair. The paper compares RaR against direct/CoT/self-consistency/RAG approaches. Among these, Direct/Cot and RAG are approaches that produce answers with one LLM call. In contrast, RaR, involves revising each reasoning step. IIUC, RaR would involve T LLM calls for producing an answer, where T is the number of steps, plus the additional costs of processing retrieved documents. To ensure fair comparison and to fully understand the differences of different approaches in terms of computation cost, I think it is necessary to 1) list out the computation cost needed for each approach (in terms of number of tokens) 2) compare methods under the same computation constraints.

Second, the missing details on the experimental setup makes it harder to interpret these results. Below are some concrete points:
* Table 1: Are all methods using the same computation budget? For self-consistency, how many samples are considered? For self-refine, how many rounds of evaluation are made?
* Table 2 (math reasoning and task planning sections): Which models are employed?
* Table 2 (QA section): The table structure is unclear. Different rows appear to use different base models (e.g., Reasoning row vs. RAG row), and the base models for the fourth RAG row are unspecified.

In particular, given the lack of description of experimental details. I also find some numbers seemingly need further verification.
In Table 1, GPT-4's reported direct performance on HumanEval and MBPP (57.3% and 60.0%) differs from the GPT-4 technical report (67.6% and 68.3% for zero-shot, pass@1).
* For instance, in Table 1, it is reported GPT-4 gets direct performance of 57.3% and 60.0 on human-eval and MBPP, respectively. But some other work, including GPT-4 tech report, and AgentCoder (Huang et al., 24), reports that GPT-4 gets a performance of 67.6 and 68.3 (zero-shot, pass@one) on HumanEval and MBPP respectively. It is unclear what leads to such mismatch.
* Table 2's math reasoning results show CoT significantly underperforming DIRECT, which seems counterintuitive.

The paper needs to provide comprehensive experimental details and restructure Tables 1 and 2 for clarity and accuracy.

Additionally, the paper lacks sufficient analysis of its results, especially for the qualitative outcomes on mathematical tasks. According to the results, the approach shows significant performance improvements in code generation and mathematical reasoning. But the underlying reasons for these improvements are not sufficiently explained. For code-related tasks, the improvements are somewhat intuitive as there often exist multiple code implementations with similar semantics.  For mathematical reasoning tasks, which are typically more problem-specific, it would be valuable to understand what drives such improvements. The paper does not provide analysis in the main body nor in the appendices to explain these performance gains. It would be good to provide some analysis around this.

questions:
See weakness
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Retrieval-Augmented Reflection (RaR), a framework for improving reasoning in LLMs by iteratively revising reasoning steps using external retrieval. While the method demonstrates potential in enhancing code generation, mathematical reasoning, and task planning, the submission contains substantial weaknesses. The evaluation lacks strong baselines and comprehensive comparisons, raising concerns about fairness. Details about experimental settings and results are insufficient, with discrepancies in reported performances. Methodological clarity is also lacking, particularly in the description of key components like stepwise reflection and scaling with inference-time computation.

Strengths include the focus on improving inference-time reasoning and promising empirical results. However, these are undermined by poor experimental rigor, weak baselines, and limited novelty.