Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models

Authors: Tian Yu, Shaolei Zhang, Yang Feng
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Retrieval-Augmented Generation, Large Language Models, Autonomous Decision-Making

Abstract:
Iterative retrieval refers to the process in which the model continuously queries the retriever during generation to enhance the relevance of the retrieved knowledge, thereby improving the performance of Retrieval-Augmented Generation (RAG). Existing work typically employs few-shot prompting or manually constructed rules to implement iterative retrieval. This introduces additional inference overhead and overlooks the remarkable reasoning capabilities of Large Language Models (LLMs). In this paper, we introduce Auto-RAG, an autonomous iterative retrieval model centered on the LLM's powerful decision-making capabilities.  Auto-RAG engages in multi-turn dialogues with the retriever, systematically planning retrievals and refining queries to acquire valuable knowledge. This process continues until sufficient external information is gathered, at which point the results are presented to the user. To this end, we develop a method for autonomously synthesizing reasoning-based decision-making instructions in iterative retrieval and fine-tuned the latest open-source LLMs.
The experimental results indicate that Auto-RAG is capable of autonomous iterative interaction with the retriever, effectively leveraging the remarkable reasoning and decision-making abilities of LLMs, which lead to outstanding performance across six benchmarks. Further analysis reveals that Auto-RAG can autonomously adjust the number of iterations based on the difficulty of the questions and the utility of the retrieved knowledge, without requiring any human intervention. Moreover, Auto-RAG expresses the iterative retrieval process in natural language, enhancing interpretability while providing users with a more intuitive experience.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730300778154

thread_type:
official_review

<thread_object>
creator: reviewer_kjp1

<content>
summary:
This paper introduces an autonomous iterative retrieval framework for retrieval-augmented generation. By designing appropriate prompts, the authors automatically synthesize instruction-finetuning data, which is used for the model finetuning. Experimental results validate the viability and effectiveness of the proposed framework, demonstrating performance improvements across six benchmarks.

strengths:
1. This paper is well-written and well-organized.
2. The authors propose an autonomous iterative retrieval framework that, in my opinion, not only demonstrates performance improvements but also offers an efficient approach to addressing the long-context limitations of current LLMs' context windows.
3. Extensive experiments demonstrate its effectiveness, and the provided prompts used to synthesize instruction-finetuning data are valuable to this community.

weaknesses:
1. Although the authors provide a detailed description of the prompts used to synthesize the dataset, the paper lacks essential information on the training process, which weakens its overall impact (see my questions for specifics).

2. The authors used Llama-3-Instruct-8B to synthesize the data and subsequently fine-tuned the same model with the generated dataset. While synthesizing data with larger models like GPT-4 or Claude would likely yield better results, the choice of an open-source model is understandable due to cost constraints. However, it raises the question of why larger Llama models, such as 70B or even 405B, weren’t used. Larger models could produce higher-quality data, potentially enhancing the training outcomes.

3. Some experimental results using GPT-4 or Claude should be provided, even if only on one dataset. While it's not expected for the finetuned Llama-3-Instruct-8B to outperform these models, such comparisons would give a clearer sense of the performance gap, better illustrating the value of fine-tuning smaller-scale language models.

questions:
1. In Section 3.2, please clarify the specifics of the input $x_t$ and the output $y_t$ in Equation 3. Although $x_t$ and the output $y_t$
may seem intuitive, the synthesized datasets described in Line 17 of Algorithm 1 do not explicitly define these variables. Providing a clear description of $x_t$ and the output $y_t$ would improve understanding.

2. Could you explain what is meant by "d contains a sub-answer of X"? I wasn’t able to find the criteria used to determine this. Some examples, especially related to Natural Questions (NQ) and multi-hop question answering (HQA), would be helpful for understanding your approach.

3. In Lines 5 through 10, if the condition in Line 9 is not met after iterating over all queries in 𝑄, then $Q_k$ and $D_k$ remain undefined, which could lead to issues in Line 11. Could you clarify how this situation is handled?

4. How do you determine the value of K? Is there a risk that the context length might exceed a manageable size? If so, do you have a method for truncating context to ensure efficiency?

5. What does it mean "if no information need in R then" in Line 12 of Algorithm 1, please provide an accurate definition for that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730369861777

thread_type:
official_review

<thread_object>
creator: reviewer_b21a

<content>
summary:
This paper presents Auto-RAG, an autonomous iterative retrieval model that engages in multi-turn dialogues with the retriever. It systematically plans its retrievals and refines queries to gather valuable knowledge. In addition, they develop a method for autonomously synthesizing reasoning-based decision-making instructions in iterative retrieval and fine-tuned the latest open-source LLMs. The experimental results show that Auto-RAG can engage in autonomous iterative interactions with the retriever, effectively utilizing the impressive reasoning and decision-making capabilities of large language models, resulting in excellent performance across six benchmarks.

strengths:
(1)They have conducted thorough research and organization of the work in the area of iterative retrieval within the RAG field.
(2)They proposed an effective data synthesis method utilizing large language models specifically for this direction, enabling fine-tuning based on this approach.
(3)The entire Auto-RAG process is efficient, with clear and coherent language expression throughout.

weaknesses:
(1)The contribution of “parametric knowledge” to the overall reasoning process is not clearly articulated. Could you provide concrete examples of how parametric knowledge influences the model's reasoning or decision-making?
(2)The overall approach does not seem to present significant innovations compared to previous work, such as self-rag; it primarily enriches the prompt content.
(3)The whole pipeline relies on the presence of the word "answer" in the generated content to determine when to conclude the retrieval process. This method may be susceptible to the model's "hallucinations."

questions:
(1)In the inference phase of Auto-RAG, why is the utilization of parametric knowledge placed later in the process? Would it not be more beneficial for overall efficiency to incorporate it before retrieval? Additionally, further analysis is needed to quantify the contribution of parametric knowledge to the final generation. Could you provide empirical evidence or reasoning for why they chose this particular ordering of parametric knowledge utilization?
(2)In the results presented in Table 1, why does Naive Gen perform better on 2Wiki than nearly all the baselines? Is it possible that the baselines were not provided with appropriately configured prompts to elicit suitable answers?
(3)Why are the performance metrics for Iter-RetGen not displayed in Figure 7?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730375919307

thread_type:
official_review

<thread_object>
creator: reviewer_esb7

<content>
summary:
This paper introduces Auto-RAG, an autonomous iterative retrieval model leveraging large language models’ (LLM) decision-making capabilities. Auto-RAG engages in multi-turn dialogues with the retriever, methodically planning and refining queries to acquire necessary external knowledge. This iterative process continues until sufficient information is gathered, at which point the results are provided to the user. The authors propose a method to autonomously generate reasoning-based decision-making instructions in iterative retrieval and fine-tune the latest open-source LLMs to enhance performance.

strengths:
- The model effectively adapts iteration count for optimized performance.
- This work opens a promising new direction in automating retrieval-augmented generation (RAG).

weaknesses:
- Novelty seems to  limited, as automated iteration in prompt-based systems has been explored in other works, including ReACT, Tool-LLM, and Chameleon (https://arxiv.org/abs/2304.09842). The approach to fine-tuning also appears similar to Alpaca.

- Critical baselines are absent, such as BeamAGGR (https://arxiv.org/pdf/2406.19820), Self-CRAG, and Open-RAG (https://arxiv.org/abs/2410.01782)—though the latter is concurrent and could be considered optional.

- Some reported results raise questions. For instance, on the TQA dataset, Self-RAG reports a 66+ accuracy using Llama 2, yet Auto-RAG with Llama 3.1 only achieves 38, while the baseline is 55.7. Similar discrepancies are noted for other datasets, like PQA.

- Section 5.5’s evaluation method of Self-RAG is unclear. Although these models say they process top-k retrievals in parallel but for arguments even if we consider the processing is individually to generate multiple answers, speed should theoretically scale with the number of retrievals. Figure 7 is therefore confusing: why does Self-RAG demonstrate slower speeds with fewer retrievals (e.g., in 2 Wiki)?

- Algorithms 1 and 2 are insufficiently clear. For example, in Algorithm 1, line 9’s method for determining if d contains a sub-answer of X is ambiguous. Additionally, the writing from line 216 onward does not consistently align with the notations in Algorithms 1 and 2 (e.g., maximum iterations are denoted as T vs. K). Improving and synchronizing these algorithmic and training details would benefit clarity.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671029758

thread_type:
official_review

<thread_object>
creator: reviewer_zzb2

<content>
summary:
This paper proposed a new method to automatically refine RAG retrieval query with reasoning. It showed that it can induce multi-step retrieval queries for multi-hop questions, and refine queries for simple questions, based on cases in Figure 1 and Figure 8. The authors fine-tuned Llama-3-Instruct-8B for the reasoning process, and used Qwen1.5-32B-Chat for crafting the query re-writing process. The performances of the new methods are better compared with other naive methods, single-time retrieval methods an iterative retrieval methods based on the results in the paper.

strengths:
- The paper clearly described the methods with examples
- The paper employed recent open-source models like llama3-8b

weaknesses:
- lack of comparison with GPT-4, GPT-4o or other closed AI models
- lack of case comparison with other RAG algorithms on multi-hop questions, in particular, for questions requiring a lot of reasoning process

questions:
- Although the author provided reasoning examples in Figure 1, it did not compare with other RAG algorithms in case study section. Based on the case study part in Figure 8, the Auto-RAG refined the input query from "When was the first Ford F-150 produced" to "When was the first Ford F-150 model introduced". This is a relatively minor refinement on the query wording, instead of reasoning. If the authors can show more examples about reasoning in particular for multi-hop questions, comparing with other RAG methodologies, that would be great. The question in Figure 1 is an example multi-hop question. Some other similar questions can be found in relevant papers like "The Argentine PGA Championship record holder has won how many tournaments worldwide?" in the paper "Multi-hop Question Answering" https://arxiv.org/abs/2204.09140
- Can you also compare the performances with closed AI models, like GPT-4, GPT-4o etc.? For example, in the paper of "From Local to Global: A Graph RAG Approach to Query-Focused Summarization", it uses GPT-4 for the RAG solution.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732243122370

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kind Reminder for Submission13930 Reviewers

comment:
Dear Submission13930 Reviewers:

We sincerely thank all reviewers for their constructive feedback during the review process! We have responded to your comments and made the corresponding revisions to the paper. If you have any further questions, please feel free to contact us, and we will respond promptly.

Best regards,

Submission13930 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732521396468

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you to all the reviewers for your valuable feedback and time!

comment:
Dear reviewers:

We sincerely thank all the reviewers for their diligent efforts during the review process. Your feedback has been instrumental in further enhancing the quality of our paper. Based on your valuable suggestions, we have revised the manuscript, and the key updates can be summarized as follows:
- Added experimental results comparing our approach with closed-source models (Appendix A2, Page 18).
- Investigated the impact of the order in which parametric knowledge and external knowledge are provided to Auto-RAG (Appendix A3, Pages 18–19).
- Included additional experiments comparing our approach with Self-RAG (Appendix A4, Page 19).
- Conducted additional case studies to compare the strengths and weaknesses of Auto-RAG with other iterative retrieval methods (Appendix C8, Pages 29–32).
- Updated Algorithm 1 and revised some detailed descriptions (Pages 3–4, highlighted in red).

We hope that our responses and revisions have addressed the your concerns. We would greatly appreciate it if you could take the time to review our replies in detail and consider raising your scores where appropriate. As the rebuttal period is nearing its end, please feel free to reach out if you have any further questions, and we will be more than happy to continue the discussion.

Best regards,

Submission 13930 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533882603

thread_type:
official_comment

<thread_object>
creator: area_chair_r9le

<content>
title:
Action Required: Respond to Author Rebuttals - Nov 27

comment:
Dear ICLR Reviewers,

The author discussion phase is ending soon. Please promptly review and respond to author rebuttals for your assigned papers. Your engagement is critical for the decision-making process.

Deadlines:
- November 26: Last day for reviewers to ask questions to authors.
- November 27: Last day for authors to respond to reviewers.
- November 28 - December 10: Reviewer and area chair discussion phase.

Thank you for your timely attention to this matter.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732697632191

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kind Reminder for Submission13930 Reviewers

comment:
Dear Submission13930 Reviewers:

We sincerely thank all reviewers for their constructive feedback during the review process! 

**We responded to your reviews and revised the paper based on your feedback. We are unsure if our response has addressed all of your concerns.** If you still have any questions, we would be happy to continue the discussion with you.

Best regards,

Submission13930 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents Auto-RAG, an autonomous iterative retrieval model that leverages LLMs' decision-making capabilities to enhance retrieval-augmented generation through multi-turn dialogues with the retriever. The approach involves synthesizing reasoning-based decision-making instructions and fine-tuning open-source LLMs. Reviewers acknowledge several strengths of the work, including comprehensive experimentation across six benchmarks, clear presentation, and practical value in addressing LLMs' context window limitations. The experimental results demonstrate performance improvements and the model's ability to autonomously adjust iteration counts based on question difficulty. However, reviewers raise some concerns about the work's contributions. First, the technical novelty appears limited, as automated iteration in prompt-based systems has been explored in previous works like ReACT, and the fine-tuning approach resembles existing methods. Second, there are questions about missing critical baselines (e.g., BeamAGGR, Self-CRAG) and some inconsistencies in reported results compared to existing work. Third, several technical details remain unclear, including the role of parametric knowledge, the inference pipeline's reliance on specific keywords, and training process specifics. While the authors have addressed some concerns through additional analysis and clarifications, particularly around baseline comparisons and experimental results, the fundamental questions about novelty and technical advancement remain. Overall, the work would benefit from substantial improvements before meeting the publication threshold.