Boosting the Potential of Large Language Models with an Intelligent Information Assistant
Authors: Yujia Zhou
Zheng Liu
Zhicheng Dou
Venue: NeurIPS 2024 poster (2024)
Keywords: information retrieval
large language model
information assistant
Primary area: natural_language_processing

Abstract: The emergence of Large Language Models (LLMs) has significantly advanced natural language processing, but these models often generate factually incorrect information, known as "hallucination." Initial retrieval-augmented generation (RAG) methods like the "Retrieve-Read" framework was inadequate for complex reasoning tasks. Subsequent prompt-based RAG strategies and Supervised Fine-Tuning (SFT) methods improved performance but required frequent retraining and risked altering foundational LLM capabilities. To cope with these challenges, we propose Assistant-based Retrieval-Augmented Generation (AssistRAG), integrating an intelligent information assistant within LLMs. This assistant manages memory and knowledge through tool usage, action execution, memory building, and plan specification. Using a two-phase training approach—Curriculum Assistant Learning and Reinforced Preference Optimization—AssistRAG enhances information retrieval and decision-making. Experiments show AssistRAG significantly outperforms benchmarks, especially benefiting less advanced LLMs, by providing superior reasoning capabilities and accurate responses.

Boosting the Potential of Large Language Models with an Intelligent Information Assistant
Yujia Zhou"
Zheng Liu" The Hong Kong Polytechnic University zhengliu102 gmail. com
Tsinghua University
zhouyujia@mail tsinghua. edwaaaddducoom en
Zhicheng Dou
Renmin University of China douêruc. edu. on
Abstract
The emergence of Large Language Models (LLMs) has significantly advanced natural language processing, but these models often generate factually incorrect in- formation, known "hallucination". Initial rerrieval-augmented generation (RAG) methods like the "Retrieve-Redt framework was inadequate for complex reason- ing tasks. Subsequent prompt-base based RAG strategies and Supervised Fine-Tuning (SFT) methods improved performance but required frequent retraining and risked altering foundational LLM capabilities. To cope with these challenges, we propose Assistant-bases Setrieval-Augmente Generation (AsSISTRAGE AG), integrating an intelligent information assistant within LLMs. This assistant manages memory and knowledge through tool usage, action execution, memory building, and plan speci- fication. Using a two-phase training approach Curriculum Assistant Learning and Reinforced Preference and pecisiin-mmkingg Experiments show ASSISTR AG significantly outperforms benchmarks, especially benefiting less advanced LLMs, by providing superior reasoning capabilities and accurate responses.
iitiiaattionAssssTI
enhances information retrieval
Introduction
The emergence of Large Language Models (LLMs) has significantly advanced the field of natural language processing, demonstrating an impressive ability to mimic human-like language patterns [ However, despite their extensive knowledge acquired during training, L.MMs occasionally generate factually incorrect information, phenomenon referred to as "hallucination" [. To address this, the integration of retrieval systems with LLMs has been suggested, allowing these models tap into external databases to generate more reliable responses
Initially, ettrieval-augmentr generation (RAG) relied on simple "Retrieve-Read framework [3). which was adequate for basic
but insufficient for complex, multi-step reasoning tasks. As language models advanced, various prompt-basee based RAG strategies emerged 66 77 incorpo- rating pre-retrieval and post-retrieval prompts to refine the process. However, these strategies heavily relied the foundational capabilities of the language models. Consequently, the focus shifted to Supervised Fine-Tune Tuning (SFT)-based RAG methods [8], which involve fine-tuning language models specifically for RAG tasks enhance their performance.
While SFT-based methods have improved the quality of generated responses, they face two limitations that hinder their practical application. Firstly, these fine-tuned models are not easily adaptable to
Equal Contributions; Correspondence to Zheng Liu and Zhicheng Dou.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Q Question
Q Question
Question
Pre-netrieval Franpting Query beuriting
Retzieve
Q Question
Nemory Panaprmant
Peanty Construct cesstruction rewnry Retelieuol
Decide
Reteieve
Froeen
L
Proepting Recument fe-ranking Becuesnt Suatantzation
Trainab
Trainable
Andistario
Frazen
LLM
Generate
Criticice
knouledge Menagement Pntraction
P
änswer
Frozen LIM
Useful
Answer
P
Answer
Art Antsarn
Naive RAG
Prompt-based RAG
SFT-based RAG
Assistant based RAG
Figure 1: Comparisons of Naive, Prompt-based, SFT-based and our Assistant-based RAG frameworks.
emerging LLMs, requiring retraining for each new foundational LLM. Secondly, directly fine-tuning foundational LLM in the RAG scenario may change its innate abilities, potentially leading to negative impacts on the model's performance on other tasks. To address these challenges, we propose Assistant-base based Retrieval-Augnee Augmenttd Generation (AsSISTRAG) which integrates intelligent information assistant as plugin within LLMs. This approach comprises trainable assistant for information management and static main LLM dedicated task execution, depicted in Figure
As an intelligent information assistant, ASSISTRAG operates in two primary categories to handle complex tasks: memory management and knowledge management. Memory management involves integrating and analyzing content from internal memory, while knowledge management focuses on leveraging external knowledge. These two main functions are supported by four core capabilities of ASSISTRAG: (1) Tool usage, which involves recalling relevant information from both internal memory and external knowledge bases through retriever; (2) Action execution, which involves processing, analyzing, and extracting information; (3) Memory building, which involves recording essential knowledge and reasoning patterns from historical interactions; (4) Plan specification, which involves determining the necessity of each step in the process. These four capabilities work together ensure that ASSISTRAG can provide accurate and comprehensive support the main LLM.
To implement ASSISTRAG, we adopt a two-phase training approach. The first phase, Curriculum Assistant Learning, enhances the assistant's capabilities in note-taking, question decomposition, and knowledge extraction through progressively complex tasks. The second phase, Reinforced Preference Optimization, uses reinforcement learning tailor the assistant's feedback the main LLM's specific needs, optimizing knowledge extraction based on feedback from the main LLM.
During the inference stage, ASSISTRAC AG operates through a three-step process: (1) Information Retrieval and Integration: The assistant understands the main LLM's needs, retrieves relevant knowledge from internal and external sources, and extracts valuable information. (2) Decision Making: The assistant evaluates and decides whether to provide the retrieved memories and knowledge to the main LLM based on their relevance. (3) Answer Generation and Memory Updating: The main LLM generates an answer using its internal knowledge and the assistant's information, while the assistant updates its memory with crucial reasoning steps.
Results from experiments across three complex question-answerrin datasets reveal that ASSISTRAC exhibits superior reasoning capabilities and markedly outperforms existing benchmarks. Notably, when applied different foundational LLMss, AsSISTRAG appears confer more pronounced benefits on less advanced LLMs.
2 Related Work
2.1 Retrieval-Augmented Generation
RAG represents significant advancement in the domain of LLMss, particularly for tasks demanding extensive knowledge. This paradigm begins with retrieval step, where the L.M accesses an external database to gather relevant information before addressing queries. Traditionally, RAG follows

---

with efforts focused on refining either the retriever or the generator through pre-training approaches augment RAG's accuracy. Building on this foundation, new RAG strategies have emerged, including the use of prompt-based methods like Chain-of- Thoouhh
Seerieve-Read" framework (9 5 10
[6], and leveraging L..M-generaeed content for dynamic retrieval [ [7]]8] These strategies underscore the L.LMs' ability to select relevant information adaptively in response to specific contexts. Concurrently, research on fine- tuning LLMs for RAG applications is rapidly expanding [19] 20] focusing on enhancing skills such as query reformulation [2I] and knowledge integration 22 23 24,, as well as developing critical functions like determining the necessity of retrieval and appraising the value of retrieved data 18 [2]_ Departing from these approaches, our paper introduces Assistant- based RAG, integrating an intelligent information assistant with the main LLM to boost its potential.
(CoT) reasoning 12,, iterative retrieval processes 13 14
15
2.2 LLM-based Autonomous Agents
Recent advancements in LLMs have facilitated the development LLM-based autonomous agents such as AutoGPT [26]. Toolformer (27]. and MetaGPT (28). which utilize LLMs for effective decision-making. Notably, ReAct (29] combines LLMs with external tools to manage knowledge- intensive tasks, allowing for dynamic responses to environmental changes. Additionally, models like WebGPT [0 integrate reinforcement learning with GPT-3, enabling the autonomous operation of search engines during text generation. Innovative methods used by Flare [17] and Self-Ask (6] deter- mine optimal times for information retrieval, while Reflexion 31 endows LLMs with introspective mechanisms that continually refine their outputs. Our proposed Assistant-based RAG model further enhances LLM capabilities by combining memory management and knowledge management, thus providing robust support to the main LLM in tackling complex tasks.
3 Methodology
In this section, we first define the task of RAG and then introduce our proposed framework, As- SISTRAG. ASSISTRAG enhances the capabilities of LLMs through the support of an intelligent information assistant. With abilities to use tools, execute actions, build memory, and plan, ASSIS- TRAG can provide precise memory and knowledge management services for LLMs.
3.1 Task Definition
Given a question and collection of documents D = [d:)][) the main LLM aims to generate an answer y based on both the question and the relevant documents. This can be formalized as LLMmain (( 4I), where D, represents the set of documents retrieved for the query and F, denotes the concatenation of the retrieved documents with the query. Expanding this concept, ASSISTRAG employs an intelligent information assistant, LLMAssist to enhance the main LLM's responses by providing relevant information, formalized as y LLM (LLLMsss 9l). In the following sections, we will detail the capabilities of the ASSISTRAG framework, along with its training and inference procedures.
3.2 ASSISTRAG Overview
By incorporating an intelligent information assistant, ASSISTRAG aims to boost the potential of LLMs in handling complex reasoning tasks. As illustrated in Figure 2 this framework consists of two main components: a frozen main LLM tasked with generating answers based on the information provided, and trainable assistant LLM responsible for information management. This assistant LLM is designed with two tasks: Memory Management involves storing interactions with the main LLM and retrieving relevant past memories to assist in addressing similar questions. Knowledge Management encompasses retrieving relevant information from external databases and processing it to support the main LLM in formulating responses to new questions.
To effectively accomplish these tasks, we have endowed the assistant with four key capabilities:
Retrieving relevant information from internal memory and external knowledge bases. Action Execution: Reasoning, analyzing information need, and extracting knowledge.
Tool Usage:
Memory Building: Recording essential knowledge and reasoning patterns from past interactions.

---

LLMS
ChatGPT
Flan-Ts
LLaMA
Bloom
Q Question
Answer
Memory Management
Knowledge Management
AssistRAG
Tools
Action
Memory
Planning
Memory Retriever Search in the memory
Take Notes Record key-facts from historical interactions Decompose Query
Memory Construction
Whether provide retrieved
Memory
Index recorded keyfacts into retrieval
memory
22
Knowledge Useful?
Decompose the question into
Whether to provide extracted
Knowledge Retriever Search the Web
multiple sub-queries
knowledge to main
Memory Updating
> Extract Knowledge
Replace the old memary with new memory
Extract relevant snippets
retrieved Results
Figure 2: Overview of ASSISTRAG. ASSISTRAG enhances LLMs by providing an intelligent information assistant. Endowed with the ability of tool usage, action execution, memory building and plan specification, can achieve effective memory and knowledge management.
Plan Specification: Determining the necessity of assistance during answer generation.
These four capabilities synergize to ensure that ASSISTRAG offers precise and comprehensive support to the main LLM. the following sections, we will provide a detailed examination of the role and implementation of each capability.
3.2.1 Memory Management
Effective memory management is crucial for enhancing the main LLM's performance by storing and retrieving historical interactions. This functionality comprises two key processes: capturing new insights and retrieving previously stored information. This stage activates the following three capabilities of AssistRAG:
Action Note Taking. This action JNT records critical information and the reasoning patterns behind each historical interaction. Given the historical interactions of the main LLM,, which include question g, reference r, and answer y, the assistant is tasked with memorizing the key reasoning process behind the answer into the memory slot my: my &- FNT(y, y). The accumulation of memory slots for all prior questions forms the assistant's 's memory M, which utilized for subsequent memory retrieval.
Tool Memory Retriever. Given the question and the assistant's memory M, the memory retriever retrieves historically relevant memories, represented as: M, -- Rmemory M).
Plan I: Assessing the Usefulness of Retrieved Memory, If the question is entirely new, the retrieved memories may not only be unhelpful but also negatively impact the main LLM's response. Therefore, we implement this plan to determine whether the retrieved memory slots should be provided to the main LLM. Using prompt, the assistant evaluates whether the retrieved memories are beneficial for answering the current question. Only if the answer is affirmative will the retrieved memories be supplied the main LLM.

---

Q
Query Decomposition
Note Taking
Sampled
Difficully
Difficllyy
Curriculum Assistant Learning
Reinforced Preference Optimization
AssistRAG
Diffiultty
DPO Training
Knowledge Extraction
Main LLM
Figure 3: Training framework of ASSISTRAG, It undergoes two-stage training pipeline through curriculum assistant learning and reinforced preference optimization.
3.2.2 Knowledge Management
Effective knowledge management is essential for an intelligent information assistant, involving the efficient gathering of necessary knowledge support the main LLM. This process includes analyzing the information needs of the main LLM, retrieving relevant knowledge, and integrating it. This process involves the following four capabilities of AssistRAG:
Action Question Decomposition. This action FOD aims to break down the current question into multiple sub-quee sub-queries to facilitate the retrieval of knowledge across various aspects: Q' (Q) where Q' represents a series sub-queries derived from the question q-
Tool II: Knowledge Retriever. Utilizing batch of sub-queries Q', the knowledge retriever sources relevant documents from external knowledge bases D, denoted as: Dør f- Rknowledge((Q D).
Action III: Knowledge Extraction. This action FKR involves extracting essential knowledge from a large number retrieved documents. Given the question and the retrieved documents Dor, the assistant is responsible for extracting the relevant knowledge Kg from the search results: Kgg FRu(q, )
Plan II: Evaluating the Relevance of Extracted Knowledge. To ensure the accuracy and relevance of the information provided to the main LLM, this plan determines whether the extracted knowledge should be included in the response generation process. Similarly, we prompt the assistant to assess whether the extracted knowledge is relevant the current question.
To summarize, we have endowed the assistant with memory capabilities and designed three actions, two retrieval tools, and two planning strategies integral to AsSISTRAG. Next, will introduce the training strategies developed for ASSISTRAC AGG focusing on enhancing the accuracy of these actions and ensuring their compatibility with the main LLM.
3.3 ASSISTRAG Training
The training objectives of ASSISTRAG focus two main goals: (1) enhancing the effectiveness of each action within the RAG process, and (2) ensuring that its outputs align with the main LLM's requirements. To achieve these two goals, depicted in Figure B implement curriculum-bai based assistant learning and reinforced preference optimization to optimize the training of ASSISTRAG:
Several studies have demonstrated that GPT-4 can achieve human-like annotation accuracy [32 Based on this consideration, we leverage collect training data for the three actions. The supervised training samples for each specific action are cataloged as COD. CRE, and CNT. preparing these for the assistant's subsequent training phase.
3.3.1 Curriculum Assistant Learning
Motivation. The tasks of question decomposition, knowledge extraction, and note-taking are interconnected, each contributing towards navigating the reasoning path from question to its answer.
5

---

To equip the assistant with a comprchensive understanding of the RAG process, we devise a step-wise curriculum assistant learning strategy designed to evolve from simpler to more complex tasks to foster deepened mastery over time.
Training Objective. The curriculum learning strategy integrates training samples across three sequential phases COD + CKE + CNT. Each phase dedicates 60% of its focus to the task at hand, with the remaining 40% evenly divided between the other tasks. The assistant's training employs the standard next token prediction target based on the training set D
(e..))DDgen Po
Po (ylx),
where á symbolizes the generator's adjustable parameters, and (I, y) is the pair of input and expected output. This methodical training strategy is designed to progressively refine the assistant's proficiency in each component of the RAG process, thereby boosting its effectiveness.
3.3.2 Reinforced Preference Optimization
Motivation. Although ASSISTRAG effectively handles RAG tasks after assistant learning, its output may sometimes not fully meet the downstream LLM's specific needs. To enhance integration, we implement reinforced preference optimization, a technique that adjusts the assistant's 's output based on feedback from the main LLM, ensuring tailored assistance that better meets its requirements.
Training Objective. To optimize the assistant for better alignment with the main LLM, we adopt Direct Preference Optimization (DPO) [33. This approach involves generating two sets of references, one from externally retrieved knowledge and the other generated by the assistant itself. The main I.MM evaluates these sets, with preference determined by comparing the F1 scores of its responses against correct answers. For reinforced preference optimization, we leverage the DPO algorithm's optimization objective, utilizing paired preference data Ddpo-
E Ma) [log (log ra (2, v1) logro(Zz, y2))]
where (1, v1) / [x(l)))) is the reward implicitly defined by the language model лy and the
reference model Tee-- This reinforced training stage enhances the assistant's capability to deliver assistance that aligns more closely with the main LLM's preferences, enhancing overall efficacy.
3.4 ASSISTRAG Inference
Upon completing its training phase, ASSISTRAG initiates its inference process through three steps:
Information Retrieval and Integration. At this initial stage, ASSISTRAG first activates Action II understand the main LLM's information needs. then uses Tool I and Tool II retrieve relevant information from internal memory and external knowledge bases, respectively. Subsequently, it invokes Action III to extract essential knowledge from the retrieved documents.
Decision Making. In this stage, ASSISTRAG decides whether to provide the retrieved memories and extracted knowledge the main LLM. It activates Plan and Plan II to evaluate the relevance and usefulness of the retrieved memories and knowledge for the current question. If the assistant deems them helpful, they supplied the main LLM aid in answer generation.
Answer Generation and Memory Updating. In the final phase, we prompt the main LLM to generate answer based on the question, its internal knowledge, and the information provided by the assistant. Following this, A ASSISTRAC activates Action I to utilize its note note-taking taking feature, capturing crucial reasoning steps from the interaction and incorporating them into its memory. This ensures the assistant's 's knowledge base remains up-to- uptto-date.
Experimental Setup
4.1 Datasets and Evaluation Metrics
In this study, we evaluate the effectiveness of our proposed method through experiments on three intricate question-answering datasets: HotpotQA 34 2WikiMultiHopAA [35), and Bamboogle ( These datascts, all derived from Wikipedia documents, provide a uniform corpus and retrieval

---

Table I: The evaluation results for three datasets. The "Main LLM" indicates the LLM employed for question answering. The best results are shown in bold, while the second-best results are underlined.
HotpotQA
2Wiki
Bamboogle
Method
Main LLM
EM
Prec.
Prec.
EM
Fl
Prec.
Baselines without retrieval
CloseBook
LLaMa2-chat 78
13.2
18.4
17.8
14.4
18.2
17.8
10.4
16.3
16.7
CloseBook
ChatGLM 6n
15.6
20.4
19.9
15.8
19.5
20.0
12.6
17.6
16.9
CloseBook
shasGPTs.5
20.0
25.8
26.4
21.6
25.7
24.5
14.4
22.0
22.3
Baselines with retrieval
Naive RAG
LLaMA2-chat
18.2
23.0
22.5
17.4
23.7
22.8
13.2
20.4
20.3
Naive RAG
ChatGLM 60
21.8
27.2
25.8
17.8
25.2
15.8
21.1
20.8
Naive RAG
3ha1GPT3.5
24.6
33.0
34.5
23.8
30.2
31.1
18.4
24.4
24.7
ReAct
ChatGPT3.5
26.8
41.7
42.6
25.0
33.0
31.6
28.8
37.7
38.2
IRCoT
ChatGPT3..
31.4
40.3
41.6
30.8
42.6
42.3
30.2
38.8
37.9
Self-Ask
shatGPTs.5
28.2
43.1
44.8
28.6
37.5
42.8
23.2
32.8
30.8
SELF-RAG
SELF-RAG 78
31.0
42.4
42.3
35.0
40.7
41.0
29.8
35.5
37.8
LLMLingua
ChatGPTa.s
28.2
40.2
40.0
29.4
38.6
37.8
25.2
31.3
30.8
ASSISTRAG
LLaMA2-chen
32.4
41.5
42.6
36.2
41.0
40.5
33.0
39.6
38.7
ASSISTRAG
ChatGLM 6R
33.0
42.4
43.5
38.0
43.2
42.8
32.8
39.8
39.0
ASSISTRAG
ChatGPT3.5
34.4
44.8
46.5
45.6
45.7
34.6
41.4
mechanisms to supply external references for LLMs. To manage costs, follow similar approach as previous studies [17] by selecting a maximum 500 questions from each dataset's validation set for our experiments. To assess the performance, we employ Exact Match (EM), F1 score, and Precision (Prec.).
4.2 Baselines
We benchmark model against three foundational models: LLaMA2-chat [36]] ChatGLM3 68 [37]. and ChatGPT, assessing their performance in Closebook, Naive RAG, and AsSISTRAG settings. To compare our RAG framework with other RAG models, include ad- vanced prompt-bused methods ReAct (29). IRCoT 77, Self-Ast 6). an SFT-based RAG model Self-RAG g and know ledge extraction model LLMLingua 38
We ensure fair comparison by
standardizing evaluation conditions across all models.
4.3 Implementation Details
Training Settings: In the assistant learning phase, create dataset comprising 50k training samples based on instruction-following input-output pairs across three distinct task types. The assistant LLM, which is based ChatGLM3-6i [37], is fully fine-tuned across all parameters. The training is conducted over epochs with a batch size of 32 and a peak learning rate of 2e-5. For the preference optimization phase, employ DPO trainer and uses LoRA for fine-tuning, with learning rate set to le-5 and the training duration extended to epochs. The training code and data be accessed at httts:: whtbb............ com. sma
Inference Settings: For employing ChatGPT, we opt the gpt 35 35-turbo-- 6k model, accessed through its API at temperature setting of O. We use a Wikipedia dump as our document corpus, breaking down articles into 100-token passages. For both memory and knowledge retrieval, we deploy the off-the-shelf LLM Embedder [39] to fetch up to 5 documents per input.
5
Results and Analysis
5.1
Main Results
The main results presented in Table Several key findings be observed follows:
Comparison among Different Reasoning Types. Applying our ASSISTRAG framework Chat- GPT demonstrates significant performance advantage over other models across all datasets. Specifi-

---

cally, ASSISTRAG showcases its adaptability with different base LLMs, consistently outperforming them in both Closebook and Naive RAG settings. This result highlights the advantage of ASSISTRAG in effectively assisting a variety of downstream LLMSs. Additionally, our method surpasses contempo- rary approaches employing prompt engineering or supervised fine-tuning, validating the efficacy of our curriculum assistant learning and reinforced preference optimization training strategies.
Comparison among Different Base LLMs. By comparing the performance of ASSISTRAG across various base LLMs. it is observed that stronger base LLMs yield higher quality responses across all reasoning types. Notably, compared to Naive RAG settings, ASSISTRAG achieves performance improvements of 78%, 51%, and 40% for LLaMA, ChatGLM, and ChatGPT, respectively. This indicates that ASSISTRAG brings more substantial benefits to weaker base LLMs. A likely reason is that weaker models inherently have less robust noise resistance. Benefiting from the assistant's inowledge extraction capability, the main L.M only receives relevant knowledge generate answers, leading to improved responses.
5.2 Analysis
Ablation Studies. ASSISTRAG integrates memory and knowledge management to sup- port the main LLM, encompassing three actions: note-taking, question decomposition, and knowl- edge extraction. To evaluate their contribution, we conduct ablation studies by removing each action or freezing the parameters of the assis- tant. Additionally, we assess the effects of not implementing planning (w/o. Planning), cur- riculum learning (w/o. Curriculum), and rein- forced preference optimization (w/o. DPO) to explore there contribution to the Fl score. Ta- ble iilustrrats that removing or freezing any of the ASSISTRAG's s actions results in decreased performance, underscoring the value of the as- sistant learning in the RAG context. Notably, maintaining these actions in a frozen state still outperforms completely removing them, highlighting their critical role in the RAG process. Concern- ing training strategies, the absence of planning, curriculum learning, and preference optimization slightly diminishes performance, indicating that structured progression from simple to complex tasks and aligning with downstream LLM preferences contribute to the assistant providing more accurate information to the downstream LLM, thereby enhancing the accuracy of LLM responses.
Table 2: Ablation Studies of ASSISTRAG.
Method
Hotpot.
2Wiki
Bamb.
Memory Management
Remove Fst
40.2
42,0
39.0
Freeze FNT
41.3
43.1
39.9
Knowledge Management
Remove FoD
39.5
37.8
37.0
Freeze FoD
41.3
40.3
37.8
Remove FRE
39.2
38.5
38.7
Freeze FRE
40.9
39.7
39.4
ASSISTRAG
44.8
45.6
41.4
w/o. Planning
43.0
44.5
40.7
w/o. Curriculum
43.2
44.3
40.0
w/o. DPO
42.5
43.2
40.5
Token Usage of Different Methods. A notable benefit of our ASSISTRAG framework is its ef- ficiency in preprocessing extensive information prior to engaging the main LLM. This not only enhances the inference speed of the main LLM but also minimizes token usage, which is particu- larly valuable when utilizing online API services like ChatGPT. We select representative model for each reasoning type to compare their token consumption in terms of online API and SFT model, alongside their performance metrics Fl on the 2Wiki dataset. The results, as outlined in Table 3, reveal significant differences in token usage among the methods. Prompt-basse based RAG methods tend to consume a large number of tokens due to their dependency on multiple API calls. On the other hand, SFT-based methods are more economical terms of API calls but require retraining for new LLM adaptations. In contrast, our ASSISTR AG demonstrates a balanced approach by reducing API token costs while maintaining adaptability across different L.LMs without the need for retraining. This method not only lowers the overall costs associated with API usage but also achieves superior performance.
Table 3: Token usage comparison. "tok." is the average token input length preceding the answer.
Method
API tok.
SFT tok.
Fl
CloseBook
18
0
25.7
Naive RAG
782
30.2
IR-CoT
1890
42.6
SELF-RAG
0
1456
40.7
LLML.ngua
176
780
38.6
ASSISTRAG
90
1528
45.6
8

---

ClosaBook
Naive RAG
IR-CoT
Seff-RAG
LiMLingua
AssistRAG
02.5
62.5
0.230
40.0
Accuraci
40.0
7
Cost Cost (cents)
0:125
0.100
15.0
35.0
0.075
100
30.0
27.5
21.5
0.025
0.00
0.10
Inference Time
Cost (cents)
Inference Time
Figure 4: The relationships between inference time, cost, and Fl accuracy for different methods.
0.46
0.44
0.40
0.42
0.38
0.40
0.40
L
0.36
0.58
0.34
AssiRRAG
AssisRAA
AssidRAAG
Curfculuu
w/o, Curriclum
0.37
wbb, Curriculum
0.58
wro. DPO Training
0.34
w/o DPO Training
wio. DPO Training
0.30
10
10
20
50
20
50
Training data size
Training data size
Training data size
(a) HotpoiQA
(b) 2WikiMuliHHoQQA
(c) Bamboogle
Figure 5: Performance with different training data sizes.
Accuracy, Efficiency, and Cost Analysis. When evaluating an algorithm's 's value, we consider three dimensions: accuracy, efficiency, and cost. To compare different RAG methods, we calculate each method's Fl accuracy, inference speed, and cost. We then illustrate the relationships between these variables using three separate plots. From Figure we observe that AsSISTRAG stands out as the most balanced method, achieving the highest F1 accuracy of 45.6, while maintaining a comparable inference time of 5.73 seconds and a low cost of 0.009 cents per question. Although methods like IR-CoT show higher costs and longer inference times, they do not surpass ASSISTRAG in accuracy. These results demonstrate that ASSISTRAG is advantageous for applications requiring high accuracy without incurring significant costs.
Impact of Dataset Size and Training Strategy. We examine the effect of training dataset size on model performance by creating subsets of 5k, 10k, and 20k instances from our original 50k training samples. These subsets fine-tune three separate model versions. evaluated on three datasets, and compared to the model trained on the full 50k dataset. We also compare the impact of curriculum learning and DPO training by evaluating performance with each strategy omitted. Figure 3 shows a clear performance improvement for ASSISTRAG as the training dataset size increases from 5k to 50k across both datasets, indicating potential further gains with larger datasets. The curriculum learning strategy performs better than random mixed training, especially with smaller datasets, showing its advantage when data limited. contrast, DPO training benefits more from larger datasets, likely because more data enables better training for high-quality data generation.
Case Study. Table Iis is case study that highlights the capabilities of AssistRAG in processing and answering complex comparative questions. In this case study, the main question "Who is older, Danny Green or James Worthy?" is systematically broken down by ASSISTRAG into simpler sub-questions regarding the birth dates of both individuals. This decomposition enables targeted information retrieval, allowing the system to accurately locate and extract relevant birth date information from the corpus. ASSISTRAG effectively retrieves multiple pieces of information, including relevant and irrelevant entries, and filters through them to extract the birth dates of both Danny Green and James Worthy and ignores unrelated entries, such those concerning another individual named Danny Green who a boxer. The memory retrieval capability is then utilized to access previous similar questions and their answers, which aids in reinforcing
necessary facts. For instance, identifies

---

Table 4: Case Study of AssistRAG.
Question: Who is older, Danny Green or James Worthy?
Question Decomposition: AssistRAG initially breaks down the main question into sub cuub--ueetoons to facilitate targeted information retrieval:
When was Danny Green born?
When was James Worthy born?
Knowledge Retrieval: AssistRAG retrieves relevant information from the corpus:
- [1] James Worthy...
- [2] Danny Green (basketball)...
[3] Danny Green (boxer)
Knowledge Extraction: From the retrieved information, AssistRAG extracts key facts:
Danny Green (born June 22, 1987) is an American professional basketball player who last played for the Philadelphia 76crs of the National Basketball Association (NBA). In his NBA career, Green has played for six teams. As of 2020, Green is one of just four players in history to have won NBA championships with three different teams.
American former professional basketball player who currently a commentator, television host, and analyst. standout at the University of North Carolina, the ft 9 in (2.06 m) small forward shared College Player of the Year honors en route to leading the Tar Heels to the 1982 NCAA Championship.
-James Worthy James Ager Worthy (born Fehruary 27, 1961)
Memory Retrieval: AssistRAG retrieves similar questions and answers from the memory:
1. question: Who is older, Danny Jones David Coverdale? answer: David Coverdale was born 22 September 1951, while Danny Jones was born 12 March 1986. Since David Coverdale was born earlier, he older than Danny Jones. So the answer is David Coverdale.
2. question: Who is older, Danny Shirley or Kevin Parker? answer: Danny Shirley was born on August 12, 1956, while Kevin Parker was born on January 20, 1986. Since Danny Shirley was born earlier than Kevin Parker, he older. So the answer is Danny Shirley.
Planning: Both retrieved memory and extracted knowledge useful for the question.
Final Output: Based the analysis, Main LLM outputs: James Worthy was born February 1961, while Danny Green was born on June 27, 1987. Since James Worthy was born earlier, he is older. So the answer is James Worthy.
Final Answer: James Worthy
Ground-truth: James Worthy
the current decision Decision-making process. This step provides context and supports consistency in the reasoning pattern applied by the system. Combining the extracted knowledge and retrieved memory, AssISTRAG plans the final response by confirming the usefulness of both sources of information. The Main LLM then generates comprehensive answer, stating that James Worthy, born on February 27, 1961, is older than Danny Green, born on June 22, 1987. This case study showcases ASSISTRAG's ability to manage complex tasks by leveraging its multi-step reasoning process, ensuring the delivery of accurate and reliable answers. The superior performance in accurately answering comparative questions is testament to the system's 's robust architecture and its effective integration of question decomposition, information retrieval, knowledge extraction, planning, and memory capabilities.
Conclusion
In this study, we introduce ASSISTRAC AG to augment LLMs with an intelligent information assistant, significantly improving their ability to tackle tasks requiring complex reasoning. By implement- ing two-stage training methodology that integrates curriculum assistant learning with reinforced preference optimization, we enhance the assistant's skills in memory and knowledge management. Experiments demonstrate that ASSISTRAG surpasses existing baselines with notable margin. Looking ahead, we plan to further expand the assistant's 's skills to include long-text processing [40] and personalized support [41, thereby providing more effective assistance the main LLM.
10

---

References
[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[2] Chunting Zhou, Graham Neubig. Jiatao Gu, Mona T. Diab, Francisco Guzmán, Luke Zettle- moyer, and Marjan Ghazvininejad. Detecting hallucinated content in conditional neural sequence generation. In ACIJIJNNIP COENNPP(Fidieie (Findings), volume ACL/IICNLP 2021 of Findings of ACI., pages 1393-1404. Association for Computational Linguistics, 2021.
[3] Weihang Su, Changyue Wang. Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. arXiv preprint arXiv:2403. wr33022400.00644, 2024.
[4] Zheng Liu, Yujia Zhou, Yutao Zhu, Jianxun Lian, Chaozhuo Li, Zhicheng Dou, Defu Lian, and Jian- Yun Nie. Information retrieval meets large language models. In Companion Proceedings of the ACM Web Conference 2024, WWW '24, page 1589, 2024.
[5] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In EACL, pages 874-880. Association for Computational Linguistics, 2021.
[6] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. CoRR, abs/2210.03350, 2022.
[7] Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with offiin-of-though reasoning for knowledge-intensin multi-step questions. arXiv preprint arXiv:2212. 105.0 2022.
[8] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.1 15511, 2023.
[9] Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yuu Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. CoRR, abs/2208 03299, 2022.
[10] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmenta generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459- 9474, 2020.
[11] Yujia Zhou, Yan Liu, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Zheng Liu, Chaozhuo Li, Zhicheng Dou, Tsung-Yi Ho, and Philip S Yu. Trustworthiness in retrieval-augmeentd generation systems: A survey, arXiv preprint arXiv:2409. 10102, 2024.
[12] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. pemonstrate-search preecct Composing retrieval and language models for knowledge-intensivee NLP. CoRR, abs/2212.11024,, 2022.
[13] Urvashi Khandelwal. Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. General- ization through memorization: Nearest neighbor language models. In ICIRR. OpenReview.net, 2020.
[14] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean- -aantttttstt Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In ICMI. volume 162 of Proceedings of Machine Learning Research, pages
PMLR, 2022.
11

---

[15] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Leyton-Brown, and Yoav Shoham. In-context retrieval- -uggeeted language models. CoRR, abs/2302.00083, 2023.
[16] Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. CoRR, abs/2305.15594 2023.
[17] Zhengbao Jiang. Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. CoRR, abs/2305.06983, 2023.
[18] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson. Peter Clark, and Ashish Sabharwal. Decomposed prompting: modular approach for solving complex tasks. In ICLR. OpenReview.net, 2023.
[19] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: retrieval-augmente black-box language models. CoRR, abs/2301.1266 2023.
[20] Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning, arXiv preprint arXiv:2305.15225,, 2023.
[21] Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting for reugenedl-uugmented large language models. arXiv preprint arXiv:2305. 2023.
[22] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving aegneeeal Ims with compression and selective augmentation. arXiv preprint arXiv:23/0.04408, 2023.
[23] Jiajie Jin, Yutao Zhu, Yujia Zhou, and Zhicheng Dou. Bider: Bridging knowledge incon- sistency for efficient retrieval-augmentt augmented Iims via key supporting evidence. arXiv preprint arXiv:2402. 12174, 2024.
[24] Xiaoxi Li, Yujia Zhou, and Zhicheng Dou. Unigen: A unified generative framework for retrieval and question answering with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8688-8696, 2024.
[25] Yujia Zhou, Zheng Liu, Jiajie Jin, Jian-Yur Nie, and Zhicheng Dou. Metacognitive retrieval- augmented large language models. In Proceedings of the ACM on Web Conference 2024, pages 1453 1463, 2024.
[26] Significant Gravitas. AutoGPT.
[27] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Seialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
[28] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.0033 2023.
[29] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[30] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:21 2tti:://22.09300 12.0933, 2021.
[31] Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint 1rXiv:3333.11366 2023.
12

---

[32] Jaromir Savelka, Kevin D Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu. Can gpt-4 support analysis of textual data in tasks requiring highly specialized domain expertise? arXiv preprini arXiv:2306. 1 13906, 2023.
[33] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprini arXiv:2305. /8290, 2023.
[34] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 2018.
[35] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In COLING, pages 6625. International Committee on Computational Linguistics, 2020.
[36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi .... and Thomas Scialom. Llama 2: Open foundation and tine-tuned chat models. CoRR, abs/2307 abs/2307.09288,, .02288, 2023.
[37] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 320-335, 2022.
[38] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023.
[39] Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian- Yun Nie. Retrieve anything to augment large language models. arXiv preprint arXiv:2310.07554, 2023.
[40] Hongjin Qian, Zheng Liu, Kelong Mao, Yujia Zhou, and Zhicheng Dou. Grounding language model with chunking- in-context retrieval. arXiv preprint arXiv:2402.09766. 2024.
[41] Yujia Zhou, Qiannan Zhu, Jiajie Jin, and Zhicheng Dou. Cognitive personalized search integrat- ing large language models with efficient memory mechanism. In Proceedings of the ACM Web Conference 2024, pages 1464-1473, 2024.
A Appendix/ supplemental material
A.1
Implementation Details
To achieve the effective functioning of ASSISTRAG, we meticulously designed and executed the training and inference phases, ensuring optimal use of computational resources and robust fine-tuning processes.
A.1.1 Training Settings
The training of the assistant LLM involved a two-phase approach: Assistant Learning and Preference Optimization.
1. Assistant Learning Phase:
We created a dataset comprising 50,000 training samples based on instruction- following input-output pairs. These pairs were categorized across three distinct task types: question decomposition, note-taking, and knowledge extraction.
Dataset Creation:
Model Architecture: The assistant LLM is based on ChatGLM3-6B [37] a otate-of-the-art language model known for its robust performance in various NLP tasks.
Training Procedure: The assistant L.M was fully fine-tuned across all parameters over 2 epochs, with a batch size of 32 and a peak learning rate of 2e-5. This phase focused on enhancing the model's is ability to decompose complex queries, take notes, and extract relevant knowledge.
13

---

Table 5: Training samples note-taking.
Input
You are given 1) the guestion, the answer, 3) the supporting facts where the are supposed figure out the reasoning process towards the answer step-by-step without other content. Be concise and direct.
derived. You
Question: American politician Joe Heck ran unsuccessfully against Democrat Catherine Cortez Masto, a woman who previously served the 32nd Attomne General of where?
Answer: Nevada
Supporting Facts:
Joseph John Heck (horn October 30, 1961) an American politician, physician, and U.S. Army Brigadier General who had served as the U.S. Representative for Nevada's 3rd congressional district from 201 to 2017. He ran unsuccessfully against Democrat Catherine Cortez Masto in the general election for the open Nevada United States Senate Nevada from 2007 2015.
2016. She previously served the 32nd Attomey General of
Reasoning:
Output
Catherine Cortez Masto previously served as the 32nd Attorney General of Nevada. Joe Heck ran unsuccess- fully against her in the general election for the open Nevada United States Senate seat in 2016. Therefore, the answer is Nevada." "
Preference Optimization Phase:
Optimization Technique: We employed DPO (Distributed Preference Optimization) trainer to
refine the assistant's feedback mechanisms.
Fine-tuning with LoRA: Low-Rank Adaptation (LoRA) utilized for fine-tuning, which helps adjusting a subset model parameters efficiently, reducing the computational load.
Learning Rate and Duration: The learning rate was set to 1e-5, with the training duration extending to 2 epochs.
The entire training process conducted using 8 A800 GPUs, providing substantial computational power to handle the intensive training tasks efficiently.
Training Resources:
A.1.2 Inference Settings
During the inference phase, the ASSISTRAG system was fine-tuned to operate seamlessly with the main LLM, ensuring effective retrieval and processing of information.
For the main LLM, we opted for the gpt- 35-turbo-- model, accessed through its API. This model was chosen for its extended context window and advanced capabilities in handling complex queries.
Model Selection:
The temperature was set to O, ensuring deterministic outputs which are crucial for consistency in inference tasks.
Temperature Setting:
Document Corpus: A Wikipedia dump used the primary document corpus. Articles were segmented into 100-token passages to facilitate efficient retrieval.
Retrieval tool: Both memory and knowledge retrieval processes utilized the off-the-shelf LLM Embedder 39 capable of fetching up to documents per input, ensuring comprehensive coverage of relevant information.
Inference Resources:
The inference tasks were efficiently handled using a single A800 GPU, highlighting the system's 's capability to deliver high performance with minimal resource usage.
A.1.3 Data Annotation and Samples
The training data annotation process was critical ensuring the quality and relevance the dataset. We utilized GPT4-turbo for annotating the dataset, ensuring high accuracy and consistency across the training samples. The dataset included approximately 50k training samples, labeled to cover a wide range of instractioonfollloiing tasks.
Instructions and Training Data Samples: The following tables (Table and provide detailed examples of the instruction types and training data samples used in the training phase.
14

---

Table Training samples of query decomposition.
Input
Please generate series of search queries that be used to find information relevant the given question. Question: The 1968 Baltimore Orioles season included replacement of Hank Bauer by the manager who spent how many seasons with the Orioles?. Search queries:
Output Who replaced Hank Bauer as the manager of the 1968 Baltimore Orioles? How many seasons did Earl Weaver spend with the Baltimore Orioles?
Table 7: Training samples of knowledge extraction.
Input
Please extract relevant snippets from search results that would helpful answering this question. Question: Musician and satirist Allie Goertz wrote a song about the The Simpsons character Milhouse, who Matt Groening named after who?.
Search results:
Allic Goertz Allie Goertz Allison Beth Allie Goertz (born March 1991) is an American musician. Goertz is known for her satirical songs based on various pop culture topics. Her vidcos are posted on YouTube under the name of Cossbysweater. Subjects of her songs have included the film The Room
[2] Allie Goertz - and logo of the series. The album was made possible through Kickstarter. She is co-host of Everything's s Coming Up Simpsons podcast, formerly known Everything's Coming Up Podcast, Simpsons focused podcast along with Julia Prescott. Goertz also served social media producer
[3] Snippets:
Output Allie Goertz Allie Goertz Allison Beth Allie Goertz (born March 2, 1991) is an American musician. Goertz is known for her satirical songs hased on various pop culture topics. Milhouse Van Houten Milhouse Mussolini Van Houten is fictional character in the animated television series The Simpsons, voiced by Pamela Hayden, and created by Matt Groening who named character after ...
A.2 Error Analysis
We have conducted a comprehensive error analysis on the performance of AssistRAG. To facilitate this analysis, selected 50 erroneous examples from the HotpotQA dataset and calculated the proportion of each error type:
1. Insufficient Knowledge Retrieval: Instances where the retrieved knowledge does not contain the answer.
2. Knowledge Extraction Errors:
Cases where the answer is present in the retrieved knowl- edge, but the assistant fails to extract this information.
3. Answer Reasoning Mistakes: Situations where the assistant extracts the correct information but the main LLM produces incorrect answer.
4. Other: Including errors such as non-exact match answers.
From Table our findings indicate that more than half of the errors stem from insufficient knowl- edge retrieval, which is likely linked to the performance of the retriever and the manner in which questions are reformulated. Additionally, a significant portion of errors are due to reasoning mistakes, highlighting the importance of the main LLM'' reasoning capabilities. Given that HotpotQA involves multi-hop question-answering tasks, these findings underscore the high demands placed on reasoning abilities.
A.3 Limitations
Despite the advancements offered by ASSISTRAG, several limitations warrant consideration. Firstly, relying on an intelligent information assistant introduces additional computational complexity and latency. The two-phase training approach and its operation during inference require substantial computational resources, which may limit the practical application of ASSISTRAG in environments with restricted processing capabilities or where real-time responses critical.
15

---

Table 8: Proportion of each error type in the analysis.
Error Type
Proportion
Insufficient Knowledge Retrieval 58%
Knowledge Extraction Errors
12%
Answer Reasoning Mistakes
20%
Other
10%
Secondly, the effectiveness of ASSISTRAG depends on the quality and comprehensiveness of the external knowledge bases and internal memory it accesses. In scenarios where the available data is sparse, outdated, biased, the assistant's 's ability retrieve and integrate relevant information may be compromised, leading to suboptimal erroncous outputs from the main LLM. This dependence data quality underscores the need for continuous updates and maintenance of the knowledge sources.
Lastly, the decision-making process during inference, which involves the assistant evaluating the rele- vance of retrieved information, is inherently difficult. The assistant's ability accurately determine the necessity and applicability of specific knowledge is crucial for effective support. However, this process susceptible errors, particularly in scenarios involving ambiguous or multifaceted queries. Enhancing the precision and reliability of this decision-making mechanism is a key area for further research.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips... cc/
publ / guides Codeeubmme ss ssionns cisionnolico icy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. cc/ wwwpiicggiiiipoooooo / //odiddee/Cdeeeeeeee Cooeeeooeeeeeeeeee SudeSuumiies Swwmissiiiii wwwssnoooiiiyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: the paper provides evaluation results and error analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:::////// sthhiss/ttts:///eeeeee /ttttts://eeurpss.... httpa//ppp...c.ppooo...cccctaaaa
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[Yes
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssw hopersswtccoee com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
21

---

Answer:[Yee Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "aI0wvF0MJp", "reviewer": "Reviewer_xG33", "summary": "This paper proposes AssistRAG, an architecture for augmenting LLMs with a separate, trainable agent that helps with information retrieval and memory/knowledge management. The authors motivate such an architecture (as opposed to, say, fine-tuning the main LLM for RAG) and describe how to build and train it. The conduct experiments with 3 information heavy benchmarks and show consistent performance gains compared to a number of recent baselines.", "questions": "Please see the weakness section.", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The motivation behind AssistRAG is strong -- directly fine-tuning the main LLM for retrieval augmented generation is costly and can/does negatively impact its other abilities. Thus, freezing the main LLM and instead training a smaller auxiliary LLM to decide what to retrieve (tailored to the needs of the main LLM) and how to manage retrieved information is a highly desirable approach.\n\n* The paper is very clearly written. The structure is easy to follow and balanced. The motivation, as noted above, is well articulated. The baselines and method are clearly described and the results well-summarized.\n\n* Ablations show the value of individual components, and additional experiments confirm the benefits of the proposed approach even as the amount of training data is varied.", "weaknesses": "I don't see any notable weaknesses in this paper. It's possible that I am not aware of the latest in this line of work and my assessment of novelty may not be accurate. I will defer to the other reviewers on the novelty aspect.\n\nSome of the datasets used in this paper are rather old -- they are from the era of the train-test paradigm with 10s of thousands of training instances (e.g., HotpotQA was published in 2018!). I realize they are being used here in the model context of prompted LLMs. Nevertheless, I wonder if there are newer datasets that might be more timely.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our paper. We sincerely appreciate all the feedback from the reviewers and will make revisions to enhance the paper's impact and applicability based on your valuable suggestions.\n\nRegarding the datasets, we have compiled the publication years of all commonly used QA datasets, as shown in the table below:\n\n| Year | Datasets                                                |\n| ---- | ------------------------------------------------------- |\n| 2023 | Bamboogle                                               |\n| 2022 | PopQA, ASQA, Musique                                    |\n| 2020 | 2WikiMultiHopQA                                         |\n| 2019 | SIQA, NQ, CommenseQA, BoolQ, PIQA, Fermi, ELI5, AmbigQA |\n| 2018 | HotpotQA, NarrativeQA, WikiQA                           |\n| 2017 | TriviaQA, MSMARCO-QA                                    |\n| 2016 | SQuAD                                                   |\n\nAs our paper focuses on complex QA tasks, we selected Multihop QA datasets for our study. These include Bamboogle (2023), 2WikiMultiHopQA (2020), and HotpotQA (2018). Among these, Bamboogle is the most recent Multihop QA dataset available. \n\nThank you once again for your constructive feedback. We greatly appreciate the time and effort you have invested in reviewing our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tDEn3C8hQ0", "reviewer": "Reviewer_Y6YL", "summary": "To address the limitation of LLMs generating factually incorrect information, the authors have introduced AssistRAG, an intelligent information assistant with LLMs, building upon existing retrieval-augmented generation (RAG) strategies. The system operates in two main categories: memory management and knowledge management. AssistRAG employs a two-phase training approach: Curriculum Assistant Learning and Reinforced Preference Optimization. During inference, AssistRAG follows a three-step process: information retrieval and integration, decision-making, and answer generation with memory updating.", "questions": "1. How does AssistRAG perform in terms of computational efficiency and scalability when applied to very large datasets or in real-time applications? Can the authors provide any benchmarks or comparisons?\n2. The ablation studies are informative, but could the authors include additional analysis on the sensitivity of AssistRAG to different hyperparameters or training configurations?\n3. Incorporate user studies or qualitative evaluations to assess the practical usability and effectiveness of the assistant in aiding human users.", "limitations": "1. Llama 2 Chat (7B parameters) and ChatGLM3 (6B parameters) have substantially fewer parameters than GPT-3.5, which is reported to have a much higher parameter count. Moreover, GPT4-turbo is used for annotating the dataset (as mentioned in Appendix section A.1.3), however, the model isn’t used, or mentioned, in the experiments. This disparity in model size should be taken into account when comparing the performance of these models, particularly in relation to GPT-3.5.\n2. The analysis done on token usage (section 5.2) is only conducted on the 2WikiMultiHopQA dataset, however other experiments in the study are done on the HotpotQA, 2WikiMultiHopQA, and Bamboogle datasets. The paper could benefit from a consistent dataset used for all evaluations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel integration of curriculum assistant learning and reinforced preference optimization, distinguishing it from traditional approaches.\n2. AssistRAG demonstrates consistent outperformance across multiple datasets and base models, highlighting its potential for broad applications in improving LLM capabilities.", "weaknesses": "1. The paper might benefit from addressing potential scalability issues and providing more extensive comparisons with a broader range of state-of-the-art models. \n2. Including real-world application scenarios and discussing potential limitations or challenges in deploying AssistRAG in practical settings would also enhance the paper's impact and applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback and recognition of our paper. We hope that the following responses will address your concerns:\n\n***\n**R to W1:**\n\nTo further demonstrate the effectiveness of our model, we have included additional experiments with two agent solutions on the 2wiki dataset. The results are as follows:\n\n|            | EM       | F1       | Recall   |\n| ---------- | -------- | -------- | -------- |\n| Toolformer | 27.2     | 38.6     | 43.2     |\n| Reflexion  | 31.8     | 41.7     | 44.2     |\n| AssistRAG  | **39.6** | **45.6** | **45.7** |\n\nThese results show that AssistRAG outperforms both Toolformer and Reflexion, demonstrating its superior performance in RAG tasks.\n\n***\n**R to W2:**\n\nThank you for your insightful suggestion. In Appendix A.4, we have discussed some potential limitations and challenges in deploying AssistRAG in practical settings. We will provide a more detailed discussion in future versions to further enhance the paper's impact and applicability following your advice.\n\n***\n**R to Q1:**\n\nComputational efficiency in real-time applications is indeed crucial. To address the concerns, we first analyze the composition of inference time when encountering a new question:\n\n1. The time taken by the Assistant LLM to perform actions, including Question Decomposition and Knowledge Extraction.\n2. The time taken by the Assistant LLM to call retrievers, including memory retrieval and knowledge retrieval.\n3. The time taken by the Main LLM to generate the answer.\n\nAmong these, steps 1 and 3 are independent of dataset size. Dataset size only affects step 2. In step 2, to achieve low-latency retrieval, we have utilized the FAISS library to index the documents and employed an IVF index structure for acceleration. This allows for millisecond-level retrieval speed even with datasets in the millions of documents range. Compared to the seconds-level time taken by steps 1 and 3, the retrieval time in step 2 is negligible. Therefore, very large datasets (below the billion-level) would not significantly impact computational efficiency.\n\n***\n**R to Q2:**\n\nThank you for your insightful suggestion. We have conducted additional experiments to analyze the sensitivity of AssistRAG (ChatGLM 6B) to different hyperparameters, specifically the learning rate during the Assistant Learning phase and the number of documents retrieved (K) during the knowledge retrieval phase. The results are as follows:\n\n| Top K     | K=1  | K=2  | K=3  | K=4  | K=5  |\n| --------- | ---- | ---- | ---- | ---- | ---- |\n| F1| 20.4 | 36.6 | 39.8 | 42.4 | 43.2 |\n\n| Learning Rate | lr=2e-4 | lr=1e-4 | lr=5e-5 | lr=2e-5 | lr=1e-5 |\n| ------------- | ------- | ------- | ------- | ------- | ------- |\n| F1     | 40.6    | 41.4    | 42.0    | 43.2    | 42.6    |\n\nWe hope these additional analyses provide a clearer understanding of how AssistRAG performs under different hyperparameter settings and training configurations.\n\n***\n**R to Q3:**\n\nThis is indeed a valuable suggestion. Following your recommendations, we conducted a user study involving three participants. Each participant was asked to write 20 factual questions. Subsequently, answers were generated using ChatGPT in three configurations: CloseBook, Naive RAG, and AssistRAG. The participants then evaluated their satisfaction with the answers provided. The results are as follows:\n\n|           | User A   | User B   | User C   |\n| --------- | -------- | -------- | -------- |\n| CloseBook | 0.25     | 0.20     | 0.30     |\n| Naive RAG | 0.45     | 0.55     | 0.45     |\n| AssistRAG | **0.70** | **0.75** | **0.65** |\n\nThese results demonstrate that users were more satisfied with the answers generated by AssistRAG compared to the other configurations. We believe this user study highlights the practical usability and effectiveness of AssistRAG in aiding human users.\n\n***\n**R to L1:**\n\nWe apologize for the confusion. Our intention in selecting models with varying parameter counts was to demonstrate that AssistRAG significantly outperforms CloseBook and Naive RAG inference methods, regardless of the model size. We appreciate your suggestion and agree that presenting the results by grouping models of similar sizes will make the comparisons clearer. We will revise Table 1 to reflect this improvement, ensuring that models with similar parameter counts are compared directly.\n\n***\n**R to L2:**\n\nThank you for your suggestion. In fact, we have conducted token usage analysis across all three datasets. Since these datasets share the same retrieval corpus, the lengths of the retrieved passages are similar, resulting in no significant differences in token usage across different datasets. Therefore, we chose to present the representative 2Wiki results. For consistency, we will include token usage results for all three datasets in future versions of the paper.\n\n***\nWe greatly appreciate the time and effort you have invested in reviewing our manuscript. Your insightful comments have been invaluable in helping us improve the quality and clarity of our paper. Thank you once again for your constructive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e2D0ESVm0p", "reviewer": "Reviewer_zFfj", "summary": "The paper introduces AssistRAG, a framework that integrates an intelligent information assistant within LLMs. AssistRAG employs a two-phase training approach involving Curriculum Assistant Learning and Reinforced Preference Optimization, focusing on memory management and knowledge management. Experiments on three QA datasets demonstrate the effectiveness.", "questions": "1. Please include more details about the (ChatGPT) model versions employed in this paper.\n2. The articles were segmented into 100-token passages - this is a relatively short length for LLM, and A.2 also shows that insufficient knowledge retrieval caused more than half of the errors, why don't consider increasing the chunk size?\n3. Experiment data and questions are focused on Wikipedia, how about other domains?", "limitations": "1. Given the current development of the Agent framework, it is necessary to compare the proposed method with the Agent solution.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Experiments show the proposed method is effective.", "weaknesses": "1. Questionable innovation, the key ideas of the proposed method (memory and knowledge management) become a standard practice in terms of Agent design.\n2. The proposed method involves model training and serving, which could be costly, and also make it harder to adapt to different foundation LLMs.\n3. A.2 shows that 58% of the errors stem from insufficient knowledge retrieval, but no further details are provided. Given such a scenario, the retrieval strategy may need to be improved. \n4. Using GPT-4 to generate fine-tuned data potentially limits the proposed method.\n5. Notations are unclear, e.g., line 97, what is d_i? Eq. 2 is not well explained.\n6. Table 1 shows that AssistRAG is combined with LLaMA2, ChatGLM, and ChatGPT, but the manuscript only mentioned using ChatGLM3 for training and ChatGPT in inference, which is inconsistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your valuable feedback. We have responded to each of the weaknesses (W), questions (Q), and limitations (L) you raised. We hope the following responses clarify the contributions of our work and address your concerns.\n\n***\n**R to W1:**\n\nWhile memory and knowledge management are common elements in agent design, our approach introduces significant innovations that set it apart:\n\n1. AssistRAG is the first to integrate a complete agent solution within RAG scenarios, encompassing tool usage, action execution, memory building, and plan specification. This sets a foundational approach for future intelligent assistants for human.\n\n2. By decoupling the RAG task into an Assistant LLM and a Main LLM, AssistRAG enhances the assistant's adaptability to RAG scenarios without compromising the main LLM's inherent capabilities through a two-phase training process.\n\n3. AssistRAG adapts efficiently to new LLMs without the need for retraining from scratch, achieving significant improvements with preference alignment.\n\nThe following table highlights the distinctions between AssistRAG and other representative works:\n\n|Model|Tool|Action|Memory|Plan|Training for RAG|No Impact on Main LLM|Adaptation to new LLMs|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ReAct|√|√|×|√|×|√|No training|\n|Self-RAG|√|√|×|√|√|×|Training from scratch|\n|Selfmem|√|×|√|×|√|×|Training from scratch|\n|AssistRAG|√|√|√|√|√|√|Preference alignment|\n\n***\n**R to W2:**\n\nWe understand your concerns regarding the cost and adaptability of the proposed method. In fact, our Assistant-based RAG framework is specifically designed to improve adaptability to different foundation LLMs.\n\n1. Decoupling the RAG task into an Assistant LLM and a Main LLM is intended to separate the training of the Assistant LLM from the Main LLM. The Assistant Learning phase remains unchanged regardless of the Main LLM. This allows a well-trained Assistant LLM to be adapted to various Main LLMs without the need for retraining from scratch.\n2. The DPO training phase is designed to enhance compatibility between the Assistant and Main LLMs. Even without this phase, the Assistant LLM can still effectively support the Main LLM, as demonstrated in Table 2 (No DPO Training).\n\n***\n**R to W3:**\n\nThere may have been a misunderstanding regarding our presentation. We hope the following points clarify this:\n\n1. The 58% refers to the proportion of errors caused by insufficient knowledge retrieval among all incorrect answers, not all questions.\n2. We employ a highly advanced and widely used model LLM-Embedder for retrieval, which is proposed in 2023 and has been downloaded over 56k times on Huggingface.\n3. Our focus is on complex multi-hop QA tasks, which require retrieving at least two correct documents to ensure knowledge completeness. This increases the difficulty for the retriever.\n\n***\n**R to W4:**\n\nWe would like to clarify the following points to address your concerns regarding the use of GPT-4:\n\n1. Using GPT-4 to generate fine-tuned data has been widely adopted in various scenarios, such as WizardLM, WizardCoder, WizardMath.\n2. Our proposed method does not rely exclusively on GPT-4. Recently released powerful open-source models, such as Llama-3.1 and DeepSeek-V2, can serve as alternatives.\n3. We will open-source our training data and models. Community researchers will be able to adapt our method to their own LLMs without needing to use GPT-4 for generating training data.\n\n***\n**R to W5:**\n\nWe apologize for the unclear notation. We will revise the manuscript to provide clearer definitions and explanations for the notations.\n\n***\n**R to W6:**\n\nWe acknowledge the inconsistency you pointed out between Table 1 and section 4.3 (Inference Settings). To clarify, the base model used for training the Assistant LLM is ChatGLM3. In order to verify that the Assistant LLM can adapt to various main LLMs, we conducted inference using LLaMA2, ChatGLM, and ChatGPT, which also involved providing preference data. We will improve our wording in future versions to prevent any misunderstandings.\n\n***\n**R to Q1:**\n\nWe appreciate your request for additional details regarding the model versions used. The specific ChatGPT model employed in this paper is gpt-35-turbo-16k, provided by Azure, with a release date of 2023-05-15.\n\n***\n**R to Q2:**\n\nThere are several reasons for choosing 100-token wikipedia passages as the retrieval corpus:\n\n1. Most baselines in its original papers used this setting, allowing for easier comparison and alignment with these baselines.\n2. Wikipedia's official passage-level retrieval corpus segments articles into 100-token passages.\n3. We do not input just one passage into the LLM but combine multiple retrieved passages. This length ensures that the combined input does not exceed the length limits of certain LLMs.\n\n***\n**R to Q3:**\n\nThis is indeed a valuable suggestion. We have expanded our study to include the ALCE-ELI5 dataset, which originates from the Reddit forum. This dataset is built upon the Sphere corpus, encompassing 899 million passages. The results are as follows:\n\n|Main LLM|Method|Fluency|Correctness|\n|-|-|-|-|\n|LLaMA-2-chat|CloseBook|50.3|20.3|\n|LLaMA-2-chat|Naive RAG|66.8|27.1|\n|LLaMA-2-chat|AssistRAG|65.9|33.5|\n|ChatGPT|LLMLingua|67.2|41.4|\n|ChatGPT|CloseBook|53.3|37.2|\n|ChatGPT|Naive RAG|67.4|40.0|\n|ChatGPT|AssistRAG|**67.8**|**45.4**|\n\n***\n**R to L1:**\n\nWe have conducted additional experiments comparing our proposed method with two Agent frameworks capable of performing RAG tasks: Toolformer and Reflexion. These experiments were conducted on the 2wiki dataset, and we will supplement results from other datasets in later version.\n\n| |EM|F1|Recall|\n|-|-|-|-|\n|Toolformer|27.2|38.6|43.2|\n|Reflexion|31.8|41.7|44.2|\n|AssistRAG|**39.6**|**45.6**|**45.7**|\n\n***\nWe hope these clarifications address your concerns and provide a better understanding of our work. If you have any further concerns, we would be delighted to continue the discussion with you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this paper addresses an important problem and makes a significant contribution to the existing literature on RAG-based systems. No major weaknesses were identified.

If ultimately accepted, I expect the authors to integrate the reviewer feedback into the final version.
Decision: Accept (poster)