Venue: NAACL 2024

# Adaptive-RAG: Learning to Adapt Retrieval-Augmented

Large Language Models through Question Complexity

Soyeong Jeong\({}^{1}\)  Jinheon Baek\({}^{2}\)   Sukmin Cho\({}^{1}\)   Sung Ju Hwang\({}^{1,2}\)    Jong C. Park\({}^{1}\)

School of Computing\({}^{1}\) Graduate School of AI\({}^{2}\)

Korea Advanced Institute of Science and Technology\({}^{1,2}\)

{starsuzi,jinheon.baek,nellpic,sjhwang82,jongpark}@kaist.ac.kr

Corresponding author

###### Abstract

Retrieval-Augmented Large Language Models (LLMs), which incorporate the non-parametric knowledge from external knowledge bases into LLMs, have emerged as a promising approach to enhancing response accuracy in several tasks, such as Question-Answering (QA). However, even though there are various approaches dealing with queries of different complexities, they either handle simple queries with unnecessary computational overhead or fail to adequately address complex multi-step queries; yet, not all user requests fall into only one of the simple or complex categories. In this work, we propose a novel adaptive QA framework that can dynamically select the most suitable strategy for (retrieval-augmented) LLMs from the simplest to the most sophisticated ones based on the query complexity. Also, this selection process is operationalized with a classifier, which is a smaller LM trained to predict the complexity level of incoming queries with automatically collected labels, obtained from actual predicted outcomes of models and inherent inductive biases in datasets. This approach offers a balanced strategy, seamlessly adapting between the iterative and single-step retrieval-augmented LLMs, as well as the no-retrieval methods, in response to a range of query complexities. We validate our model on a set of open-domain QA datasets, covering multiple query complexities, and show that ours enhances the overall efficiency and accuracy of QA systems, compared to relevant baselines including the adaptive retrieval approaches. Code is available at: [https://github.com/starsuzzi/Adaptive-RAG](https://github.com/starsuzzi/Adaptive-RAG).

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), ppas 7036–7050 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Recent Large Language Models (LLMs) Brown et al. (2020); OpenAI (2023); Touvron et al. (2023); Anil et al. (2023) have shown overwhelming performances across diverse tasks, including question-answering (QA) Yang et al. (2018); Kwiatkowski et al. (2019). However, they still generate factually incorrect answers since their knowledge solely relies on their parametric memory Kasai et al. (2022); Mallen et al. (2023). Meanwhile, memorizing all the (ever-changing) world knowledge may not be possible. To address this problem, retrieval-augmented LLMs Borgeaud et al. (2022); Izacard et al. (2023); Shi et al. (2023), which incorporate non-parametric knowledge into LLMs with additional retrieval modules, have gained much increasing attention. Specifically, these models access a knowledge base, which serves as an extensive repository of information across various subjects and disciplines, to retrieve information relevant to the given input, and then incorporate the retrieved information into LLMs, which enables them to stay accurate and current with the world knowledge.

A particularly salient application of retrieval-augmented LLMs is to handling QA tasks, whose goal is to provide correct answers in response to user queries, especially those of high complexity. Early work on retrieval-augmented LLMs focuses primarily on single-hop queries Lazaridou et al. (2022); Ram et al. (2023), whose answers are typically found within a single document; therefore, this approach involves retrieving a relevant document based on the query and subsequently integrating this information into QA models to formulate a response. However, unlike this single-hop QA, some queries require connecting and aggregating multiple documents, which are, furthermore,

Figure 1: QA performance (F1) and efficiency (Time/Query) for different retrieval-augmented generation approaches. We use the GPT-3.5-Turbo-Instruct as the base LLM.



often not answerable through a single-step process of retrieval-and-response. An example query is 'When did the people who captured Malakoff come to the region where Philipsburg is located?', which requires four reasoning steps to solve. Therefore, to effectively handle such complex queries, recent studies have concentrated largely on multi-step and multi-reasoning QA, which requires iterative accesses to both LLMs and retrievers multiple times Press et al. (2023); Trivedi et al. (2023), at the cost of heavy computational overheads.

Yet, we should rethink: In a real-world scenario, are all the requests from users complex? Instead, users might often ask simple and straightforward questions, while only occasionally asking complex ones. Specifically, a query such as 'Paris is the capital of what?' is likely to be asked more frequently, compared to the aforementioned multi-step query, and this simpler query might also be easily answered by the LLMs themselves, without accessing external knowledge. In other words, a multi-step QA approach could give rise to unnecessary computational overhead for simple queries, even though it would be vital for complex queries (see Figure 2 (A)). On the other hand, handling complex queries with single-step-retrieval or even non-retrieval strategies would be largely insufficient (Figure 2 (B)). This suggests the need for an adaptive QA system, which can dynamically adjust the operational strategies of retrieval-augmented LLMs based on the query complexity. While some recent approaches are capable of doing this based on the frequency of entities in queries Mallen et al. (2023); Zhao et al. (2023) or on the generated outputs from models for multi-step QA Trivedi et al. (2023), they are still suboptimal: the former methods are overly simplistic, failing to consider multi-hop queries; meanwhile, the latter are excessively complex, terminating answer solving steps after several rounds of module access.

In this work, considering diverse complexity levels of real-world queries, we argue that previous one-size-fits-all approaches might be inadequate to cover all of them. Instead, we propose to select the most suitable strategy from a range of (retrieval-augmented) LLMs, each of which is tailored to the specific complexity of the input query. Notably, a critical step in this process is pre-defining the query complexity, which is instrumental in determining the most fitting model to it. In this work, we operationalize this process with a novel classifier, which is a smaller model trained to predict the complexity level of incoming queries (see Figure 2 (c)). Moreover, we automatically collect its training datasets without human labeling, by leveraging the predicted outcomes (i.e., which models accurately respond to which queries) as well as by capitalizing on the inherent biases in existing datasets (i.e., samples in the datasets are designed either for single-step or for multi-step QA scenarios). This proposed method can offer a robust middle ground among the iterative LLM augmentation methods for complex queries, single-step methods for simpler queries, and even no-retrieval-augmented methods for the most straightforward queries (answerable by LLMs themselves), thus significantly enhancing the overall efficiency and accuracy, as shown in Figure 1. We refer to our framework as Adaptive Retrieval-Augmented Generation (Adaptive-RAG).

We validate Adaptive-RAG using benchmark open-domain QA datasets, covering a wide range of query complexity from single-hop Rajpurkar et al. (2016); Joshi et al. (2017); Kwiatkowski et al. (2019) to multi-hop Yang et al. (2018); Ho et al. (2020); Trivedi et al. (2022b) queries. The experimental results show that ours significantly improves the overall accuracy and efficiency, compared to the prior adaptive strategies, on multiple LLMs, such as GPT-3.5 Brown et al. (2020) and FLAN-T5 series Chung et al. (2022).

Figure 2: A conceptual comparison of different retrieval-augmented LLM approaches to question answering. (A) In response to a query, this single-step approach retrieves relevant documents and then generates an answer. However, it may not be sufficient for complex queries that require multi-step reasoning. (B) This multi-step approach iteratively retrieves documents and generates intermediate answers, which is powerful yet largely inefficient for the simple query since it requires multiple accesses to both LLMs and retrievers. (C) Our adaptive approach can select the most suitable strategy for retrieval-augmented LLMs, ranging from iterative, to single, to even no retrieval approaches, based on the complexity of given queries determined by our classifier.

 

Our contributions and findings are threefold:

* We point out the realistic scenario of queries of varying complexities, and find out that existing retrieval-augmented generation approaches tend to be overly simple or complex.
* We adapt retrieval-augmented LLMs to the query complexity assessed by the classifier, which enables the utilization of the most suitable approach tailored to each query.
* We show that our Adaptive-RAG is highly effective and efficient, balancing between the complexity and the simplicity for diverse queries.

## 2 Related Work

Open-domain QAOpen-domain QA is the task of accurately answering a query by sourcing for query-relevant documents, and then interpreting them to provide answers Chen et al. (2017); Zhu et al. (2021), which, thus, generally involves two modules: a retriever Karpukhin et al. (2020); Xiong et al. (2021) and a reader Yang et al. (2019); Izacard and Grave (2021); Jeong et al. (2023). Along with the emergence of LLMs with superior reasoning capabilities thanks to their billion-sized parameters Wei et al. (2022), a synergy between LLMs and retrieving has led to significant advancements Lazaridou et al. (2022); Ram et al. (2023). Specifically, this integration has been shown to enhance Open-domain QA by mitigating the hallucination problem from LLMs through strengthened reasoning abilities of the reader, as well as utilizing the retrieved, external documents Cho et al. (2023). Despite these advancements for single-hop retrieval-augmented LLMs, however, the complexity of some queries needs a more complex strategy.

Multi-hop QAMulti-hop QA is an extension of conventional Open-domain QA, which additionally requires the system to comprehensively gather and contextualize information from multiple documents (often iteratively), to answer more complex queries Trivedi et al. (2022); Yang et al. (2018). In the realm of multi-hop QA, the approach to iteratively access both LLMs and the retrieval module is generally employed. Specifically, Khattab et al. (2022), Press et al. (2023), Pereira et al. (2023) and Khot et al. (2023) proposed to first decompose the multi-hop queries into simpler single-hop queries, repeatedly access the LLMs and retriever to solve these sub-queries, and merge their solutions to formulate a complete answer. In contrast to this decomposition-based approach, other recent studies, such as Yao et al. (2023) and Trivedi et al. (2023), explored the interleaving of Chain-of-Thought reasoning Wei et al. (2022) -- a method where a logical sequence of thoughts is generated -- with document retrieval, repeatedly applying this process until the reasoning chain generates the answer. In addition, Jiang et al. (2023) introduced an approach to repeatedly retrieving new documents if the tokens within generated sentences have low confidence. However, the aforementioned methods overlooked the fact that, in real-world scenarios, queries are of a wide variety of complexities. Therefore, it would be largely inefficient to iteratively access LLMs and retrievers for every query, which might be simple enough with a single retrieval step or even only with an LLM itself.

Adaptive RetrievalTo handle queries of varying complexities, the adaptive retrieval strategy aims to dynamically decide whether to retrieve documents or not, based on each query's complexity. In this vein, Mallen et al. (2023) proposed to decide the query's complexity level based on the frequency of its entities and suggested using the retrieval modules only when the frequency falls below a certain threshold. However, this approach, focusing solely on the binary decision of whether to retrieve or not, may not be sufficient for more complex queries that require multiple reasoning steps. Additionally, Qi et al. (2021) proposed an approach that performs a fixed set of operations (retrieving, reading, and reranking) multiple times until the answer is derived for the given query, which is built upon traditional BERT-like LMs. However, unlike our Adaptive-RAG which pre-determines the query complexity and adapts the operational behavior of any off-the-shelf LLMs accordingly, this approach applies the same fixed operations to every query regardless of its complexity but also necessitates additional specific training to LMs. Concurrent to our work, Asai et al. (2024) suggested training a sophisticated model to dynamically retrieve, critique, and generate the text. Nevertheless, we argue that all the aforementioned adaptive retrieval methods that rely on a single model might be suboptimal in handling a variety of queries of a range of different complexities since they tend to be either overly simple or complex for all the input queries, which demands a new approach that can select the most suitable strategy of retrieval-augmented LLMs tailored to the query complexity 

Method

In this section, we describe our approach to adapting retrieval-augmented LLMs, by pre-determining the query complexity and then selecting the most fitting strategies for retrieval-augmented LLMs.

### Preliminaries

We begin with preliminaries, formally introducing different strategies of retrieval-augmented LLMs.

Non Retrieval for QALet us first define an LLM as a model \({}_{\mathsf{LLM}}\), which takes a sequence of tokens \(\mathbf{x}=[x_{1},x_{2},...,x_{n}]\) as an input and then generates a sequence of tokens \(\mathbf{y}=[y_{1},y_{2},...,y_{n}]\) as an output, which is formalized as follows: \(\mathbf{y}={}_{\mathsf{LIM}}(\mathbf{x})\). Then, in our problem setup for QA, \(\mathbf{x}\) and \(\mathbf{y}\) become the input query (\(\mathbf{q}\)) from the user and the generated answer (\(\bar{\mathbf{a}}\)) from the LLM, respectively: \(\mathbf{q}=\mathbf{x}\) and \(\bar{\mathbf{a}}=\mathbf{y}\). Also, subsequently, the most naive LLM-powered QA model can be represented as follows: \(\bar{\mathbf{a}}={}_{\mathsf{LLM}}(\mathbf{q})\). Ideally, \(\bar{\mathbf{a}}\) should match the actual correct answer \(\mathbf{a}\). This non-retrieval-based QA method is highly efficient and could be a somewhat promising approach to handling easy queries, as the size of LLMs becomes extremely large with its effect on storing a large amount of knowledge. However, this approach is largely problematic on queries that require precise or concurrent knowledge of specific people, events, or any subjects beyond the LLMs' internal knowledge.

Single-step Approach for QAtTo address the aforementioned scenarios where LLM may struggle with queries that are not answerable by LHM itself, we can utilize the external knowledge \(\mathbf{d}\), which includes useful information for queries, retrieved from the external knowledge source \(\mathcal{D}\) that could be an encyclopedia (e.g., Wikipedia) consisting of millions of documents. Specifically, to obtain such \(\mathbf{d}\) from \(\mathcal{D}\), a specific retrieval model is necessary, which returns documents based on their relevance with the given query. This process can be formulated as follows: \(\mathbf{d}=\texttt{Retriever}(\mathbf{q};D)\), where Retriver is the retrieval model, with \(\mathbf{d}\in\mathcal{D}\). Here, we can use any off-the-shelf retriever Robertson et al. (1994); Karpukhin et al. (2020).

After the retrieval step is done, we now have a pair of query \(\mathbf{q}\) and its relevant documents \(\mathbf{d}\). Then, in order to augment LLMs with this retrieved external knowledge, we can incorporate it into the input of LLMs, represented as follows: \(\bar{\mathbf{a}}={}_{\mathsf{LLM}}(\mathbf{q},\mathbf{d})\). This process allows LLMs to gain access to external information contained in \(\mathbf{d}\), which can provide the supplementary context that the internal knowledge of LLM lacks, which can subsequently improve the accuracy and concurrency of LLMs for QA.

Multi-step Approach for QAEven though the aforementioned single-step approach offers significant improvements over non-retrieval for \(\mathbf{q}\) that requires external knowledge, it encounters notable limitations, particularly when dealing with complex queries that necessitate synthesizing information from multiple source documents and reasoning over them. This is where a multi-step approach and reasoning for QA become essential.

In this multi-step approach, LLM interacts with Retriever in several rounds, progressively refining its understanding of \(\mathbf{q}\), until it formulates the final answer from findings accumulated across these multiple steps. Specifically, the process begins with the initial query \(\mathbf{q}\), and at every retrieval step \(i\), new documents \(\mathbf{d}_{i}\) are retrieved from \(\mathcal{D}\) and then incorporated into the input of LLMs, as follows: \(\bar{\mathbf{a}}_{i}={}_{\mathsf{LLM}}(\mathbf{q},\mathbf{d}_{i},\mathbf{c}_{i})\), where the additional context \(\mathbf{c}_{i}\) can be composed of previous documents and outcomes \((\mathbf{d}_{1},\mathbf{d}_{2},...,\mathbf{d}_{i-1},\bar{\mathbf{a}}_{1},\bar{\mathbf{a}}_{2},.. ,\bar{\mathbf{a}}_{i-1})\), and \(\mathbf{d}_{i}=\texttt{Retriever}(\mathbf{q},\mathbf{c}_{i};D)\)1. We would like to note that this iterative, multi-step process enables LLM to construct a more comprehensive and extensive foundation to solve queries effectively, specifically adept at complex multi-hop queries where answers depend on interconnected pieces of information. However, it is important to recognize that this multi-step approach can be resource-intensive due to the repeated accesses to Retriever and LLM, which entail substantial computational costs.

Footnote 1: It is worth noting that implementations of the LLM and retriever vary across different multi-step retrieval-augmented LLM approaches Trivedi et al. (2023); Press et al. (2023); Yao et al. (2023); therefore, the context \(\mathbf{c}_{i}\) may incorporate none, some, or all of the previous documents and answers.

### Adaptive-RAG: Adaptive Retrieval-Augmented Generation

We now introduce our adaptive retrieval-augmented LLMs, which are built upon three different strategies described in the previous section, and which are designed to select the most suitable strategy according to the complexity of queries.

Adapting Retrieval-Augmented LLMsNote that in real-world scenarios, not all \(\mathbf{q}\) from users have the same level of complexity, necessitating 

tailored strategies for handling each query. In other words, employing the most basic, non-retrieval-based approach \({\tt LLM}(\mathbf{q})\) to respond to the complex query \(\mathbf{q}\) would be also ineffective (Figure 2, A); conversely, using a more elaborate multi-step approach \({\tt LLM}(\mathbf{q},\mathbf{d},\mathbf{c})\) for simple \(\mathbf{q}\) would be inefficient (Figure 2, B). Therefore, our adaptive framework is designed to dynamically adjust the query-handling strategy of retrieval-augmented LLMs, which is achieved by determining the complexity of each query before attempting a solution. Notably, this framework can offer a robust middle ground with a range of solutions, from the simplest approach for the most straightforward queries, to the one-step approach for moderate queries, and up to the most comprehensive and rigorous approach for complex queries. In addition, since the operations of LLM and Retriever remain consistent regardless of inputs to them, our method can seemingly go back and forth across queries of different complexities, without changing the internal model architecture or parameters during adaption.

Query Complexity AssessmentTo operationalize our adaptive retrieval-augmented LLM framework, we should determine the query complexity, and to achieve this, we propose to model a complexity classifier, whose goal is to return the appropriate complexity level of the given query. Specifically, given the query \(\mathbf{q}\), our classifier can be formulated as follows: \(o={\tt Classifier}(\mathbf{q})\), where Classifier is a smaller Language Model that is trained to classify one of three different complexity levels and \(o\) is its corresponding class label. In our classifier design, there are three class labels: 'A', 'B', and 'C', where 'A' indicates that \(\mathbf{q}\) is straightforward and answerable by \({\tt LLM}(\mathbf{q})\) itself, 'B' indicates that \(\mathbf{q}\) has the moderate complexity where at least a single-step approach \({\tt LLM}(\mathbf{q},\mathbf{d})\) is needed, and 'C' indicates that \(\mathbf{q}\) is complex, requiring the most extensive solution \({\tt LLM}(\mathbf{q},\mathbf{d},\mathbf{c})^{2}\).

Training StrategyThe remaining step is to train the smaller Language Model for Classifier, to accurately predict its complexity \(o\) in response to the given query \(\mathbf{q}\). Yet, there is no annotated dataset available for query-complexity pairs. Hence, we propose to automatically construct the training dataset with two particular strategies.

To be specific, we first aim at labeling the query complexity based on the results from three different retrieval-augmented LLM strategies, in order to determine the label by its needs. For example, if the simplest non-retrieval-based approach correctly generates the answer, the label for its corresponding query is assigned 'A'. Also, to break the tie between different models in providing the label to the query, we provide a higher priority to a simpler model. In other words, if both single-step and multi-step approaches produce the same correct answer while the non-retrieval-based approach fails, we assign label 'B' to its corresponding query.

However, this labeling strategy has a limitation in that not all the queries are assigned labels, since the three retrieval-augmented approaches may all fail to generate the correct answer. On the other hand, the benchmark datasets may already have meaningful inductive biases about the most appropriate retrieval-augmented LLM strategies for their queries, considering the ways they are created (e.g., QA datasets that require sequential reasoning usually necessitate a multi-step approach; while queries of those with labeled single documents can be ideally answerable with the single-step approach). Therefore, for those queries that remain unlabeled after the first labeling step, we assign 'B' to queries in single-hop datasets and 'C' to queries in multi-hop datasets. Finally, we train Classifier with these automatically-collected query-complexity pairs3, by using a cross-entropy loss. Then, at inference, we can determine the complexity of the query, which is one of {'A', 'B', 'C'}, by forwarding it to Classifier: \(o={\tt Classifier}(\mathbf{q})\).

Footnote 3: As we automatically assign classifier labels, there might be errors in labeling and might be more advanced strategies to automatically assign labels, which we leave as future work.

## 4 Experimental Setups

In this section, we explain datasets, models, metrics, and implementation details. We provide additional details in Appendix A.

### Datasets

In order to simulate a realistic scenario, where different queries have varying complexities, we use both the single-hop and multi-hop QA datasets simultaneously, in the unified experimental setting.

Single-hop QAFor simpler queries, we use three benchmark single-hop QA datasets, which consist 

of queries and their associated documents containing answers, namely **1) SQuAD v1.1**Rajpurkar et al. (2016), **2) Natural Questions**Kwiatkowski et al. (2019), and **3) TriviaQA**Joshi et al. (2017).

Multi-hop QATo consider more complex query scenarios, we use three benchmark multi-hop QA datasets, which require sequential reasoning over multiple documents, namely **1) MuSiQue**Trivedi et al. (2022a), **2) HotpotQA**Yang et al. (2018), and **3) WikliMultiHopQA**Ho et al. (2020).

### Models

We compare our Adaptive-RAG against relevant models, including three retrieval-augmented LLM strategies (in Section 3.1) and the adaptive retrieval approaches Mallen et al. (2023); Asai et al. (2024), which can be grouped into one of three categories: Simple, Adaptive, and Complex. Specifically, Simple approaches include the **1) No Retrieval** and **2) Single-step Approach**-based methods. Adaptive approaches include the **3) Adaptive Retrieval**Mallen et al. (2023), **4) Self-RAF**Asai et al. (2024), and our **5) Adaptive-MAG**, which can adaptively perform retrieval based on the question complexity. For the **6) Multi-step Approach**, we use the most sophisticated state-of-the-art method Trivedi et al. (2023), iteratively accessing both the retriever and LLM with Chain-of-Thought reasoning Wei et al. (2022b), for every query. Note that models across different categories are not directly comparable. Yet, in the ideal setting, Adaptive approaches should be more effective than those in the Simple category while simultaneously being more efficient than the Complex one. Therefore, we also report the performance in an ideal scenario, **7) Adaptive-RAG w/ Oracle**, using the oracle classifier with our Adaptive-RAG.

### Evaluation Metrics

When it comes to evaluating adaptive models, it is essential to simultaneously consider both the task performance and efficiency along with their trade-offs. Thus, we report results with five metrics, where three of them measure the effectiveness and the other two measure the efficiency. In particular, for effectiveness, we use F1, EM, and Accuracy (Acc), following the standard evaluation protocol Mallen et al. (2023); Baek et al. (2023); Asai et al. (2024), where F1 measures the number of overlapping words between the predicted answer and the ground truth, EM measures whether they are the same, and Acc measures whether the predicted answer contains the ground-truth answer. For efficiency, we measure the number of retrieval-and-generate steps and the average time for answering each query relative to the one-step approach.

### Implementation Details

For a fair comparison and following Mallen et al. (2023) and Trivedi et al. (2023), we use the same retriever, a term-based sparse retrieval model known as BM25 Robertson et al. (1994), across all different models. For the external document corpus, we use different sources depending on the dataset type: the Wikipedia corpus preprocessed by Karpukhin et al. (2020) for single-hop datasets, and the pre-processed corpus by Trivedi et al. (2023) for multi-hop datasets. Regarding the LLMs that are used to generate answers, we use the FLAN-T5 series models Chung et al. (2022) of XL with 3B parameters and XXL with 11B parameters, and the GPT-3.5 model (gpt-3.5-turbo-instruct). For the retrieval-augmented LLM design, we follow the implementation details from Trivedi et al. (2023), which include input prompts, instructions, and the number of test samples for evaluation (e.g., 500 samples per dataset). In our Adaptive-RAG, for the query-complexity classifier, we use and train the T5-Large model Raffel et al. (2020). Specifically, the classifier is trained using the epoch that shows the best performance until 100 training iterations from the validation set, with the learning rate of 3e

\begin{table}
\begin{tabular}{l l c c c c c c c c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{**FLAN-TS-XL (1B)**} & \multicolumn{4}{c}{**FIPAN-TS-XXL (1B)**} \\ \cline{3-14}
**Types** & **Methods** & EM & F1 & Acc & Strip & Time & EM & F1 & Acc & Step & Time & EM & F1 & Acc & Step & Time \\ \hline \multirow{2}{*}{**Simple**} & **No Retrieved** & 14.87 & 21.12 & 15.97 & 0.00 & 0.11 & 17.83 & 25.14 & 19.33 & 0.00 & 0.08 & 35.77 & 48.56 & 44.27 & 0.00 & 0.71 \\  & **Singlegre Approach** & 34.83 & 44.31 & 38.87 & 1.00 & 1.00 & 37.87 & 47.63 & 41.90 & 1.00 & 1.00 & 34.73 & 46.99 & 45.27 & 1.00 & 1.00 \\  & **Adaptive Retrieval** & 23.87 & 32.24 & 26.73 & 0.50 & 0.56 & 26.93 & 36.57 & 26.73 & 0.50 & 0.54 & 35.90 & 48.26 & 45.30 & 0.50 & 0.86 \\
**Adaptive** & **Self-RAG** & 9.90 & 20.79 & 31.57 & 0.72 & 0.43 & 10.87 & 22.98 & 34.13 & 0.74 & 0.23 & 10.87 & 22.98 & 34.13 & 0.74 & 1.50 \\  & **Adaptive-RAG (Ours)** & **37.17** & **46.94** & **42.10** & **2.17** & **3.60** & **3.890** & **45.62** & **43.77** & **1.35** & **2.00** & **37.97** & **50.91** & **48.97** & **1.03** & **1.46** \\ \hline \multirow{2}{*}{**Complex**} & **Multi-active Approach** & 39.00 & 48.85 & 43.70 & 4.69 & 8.81 & 40.13 & 50.09 & 45.20 & 2.13 & 3.80 & 38.13 & 50.87 & 49.70 & 2.81 & 3.33 \\
**Oracle** & **Adaptive-RAG w/ Oracle** & 45.00 & 56.28 & 49.90 & 1.28 & 2.11 & 47.17 & 58.60 & 52.20 & 0.84 & 1.10 & 47.70 & 62.80 & 58.70 & 0.50 & 1.03 \\ \hline \end{tabular}
\end{table}
Table 1: Averaged results on a collection of benchmark datasets for open-domain question answering including the single-hop and multi-hop queries, with different LLMs. Self-RAG\({}^{*}\) is trained with a different base LLM, namely LLAMA2 Tuovron et al. (2023); therefore, we compare the results of FLAN-T5-XL (3B) with the results form Self-RAG with LLMaMA (7B) and the results of others with the results from Self-Radog with LLaMA2 (13B). We emphasize our results in bold, for easy comparisons.

 

5 and the AdamW (Loshchilov and Hutter, 2019) as an optimizer. Regarding its training data, we sample and annotate 400 queries from 6 datasets based on its inductive bias (single-hop for one-step approach and multi-hop for multi-step). In addition, we use predicted outcomes of three different strategies over 400 queries sampled from each dataset. Note that those queries used for classifier training do not overlap with the testing queries for QA.

## 5 Experimental Results and Analyses

In this section, we show the overall experimental results and offer in-depth analyses of our method.

Main ResultsFirst of all, Table 1 shows our main results averaged over all considered datasets, which corroborate our hypothesis that simple retrieval-augmented strategies are less effective than the complex strategy, while the complex one is significantly more expensive than the simple ones. In addition, we report the more granular results with FLAN-T5-XL on each of the single-hop and multi-hop datasets in Table 2 (and more with different LLMs in Table 7 and Table 8 of Appendix), which are consistent with the results observed in Table 1.

However, in a real-world scenario, not all users ask queries with the same level of complexity, which emphasizes the importance of the need for adaptive strategies. Note that among the adaptive strategies, our Adaptive-RAG shows remarkable effectiveness over the competitors (Table 1). This indicates that merely focusing on the decision of whether to retrieve or not is suboptimal. Also, as shown in Table 2, such simple adaptive strategies are particularly inadequate for handling complex queries in multi-hop datasets, which require aggregated information and reasoning over multiple documents. Meanwhile, our approach can consider a more fine-grained query handling strategy by further incorporating an iterative module for complex queries. Furthermore, in a realistic setting, we should take into account not only effectiveness but also efficiency. As shown in Table 1, compared to the complex multi-step strategy, our proposed adaptive strategy is significantly more efficient across all model sizes. This is meaningful in this era of LLMs, where the cost of accessing them is a critical factor for practical applications and scalability. Finally, to see the upper bound of our Adaptive-RAG, we report its performances with the oracle classifier where the classification performance is perfect. As shown in Table 1 and Table 2, we observe that it achieves the best performance while being much more efficient than our Adaptive-RAG without the oracle classifier. These results support the validity and significance of our proposal for adapting retrieval-augmented LLM strategies based on query complexity, and further suggest the direction to develop more improved classifiers to achieve optimal performance.



Classifier PerformanceTo understand how the proposed classifier works, we analyze its performance across different complexity labels. As Figure 3 (Left and Center) shows, the classification accuracy of our Adaptive-RAG is better than those of the other adaptive retrieval baselines, which leads to overall QA performance improvements. In other words, this result indicates that our Adaptive-RAG is capable of more accurately classifying the complexity levels with various granularities, which include not performing retrieval, performing retrieval only once, and performing retrieval multiple times. In addition to the true positive performance of our classifier averaged over all those three labels in Figure 3 (Left and Center), we further report its confusion matrix in Figure 3 (Right). We note that the confusion matrix reveals some notable trends: 'C (Multi)' is sometimes misclassified as 'B (One)' (about 31%) and 'B (One)' as 'C (Multi)' (about 23%); 'A (No)' is misclassified often as 'B (One)' (about 47%) and less frequently as 'C (Multi)' (about 22%). While the overall results in Figure 3 show that our classifier effectively categorizes the three labels, further refining it based on such misclassification would be a meaningful direction for future work.

Analyses on Efficiency for ClassifierWhile Table 1 shows the relative elapsed time for each of the three different RAG strategies, we further provide the exact elapsed time per query for our Adaptive-RAG and the distribution for predicted labels from our query-complexity classifier in Table 3. Similar to the results of the elapsed time in Table 1 (relative time), Table 3 (exact time) shows that efficiency can be substantially improved by identifying simple or straightforward queries.

Analyses on Training Data for ClassifierWe have shown that the classifier plays an important role in adaptive retrieval. Here, we further analyze the different strategies for training the classifier by ablating our full training strategy, which includes two approaches: generating silver data from predicted outcomes of models and utilizing inductive bias in datasets (see Section 3.2). As Table 4 shows, compared to the training strategy relying solely on the data derived from inductive bias, ours is significantly more efficient. This efficiency is partly because ours also takes into account the case that does not consider any documents at all, as also implied by the classification accuracy; meanwhile, queries in the existing datasets do not capture the information on whether the retrieval is required or not. On the other hand, in the case of only using the silver data annotated from the correct predictions, while its overall classification accuracy is high, the overall QA performance implies that relying on the silver data may not be optimal. This may be because this silver data does not cover complexity labels over incorrectly predicted queries, which leads to lower generalization effect on queries relevant to them. Meanwhile, by also incorporating complexity labels from dataset bias (single-hop vs multi-hop), the classifier becomes more accurate in predicting multi-hop queries, leading to the better performance. It is worth noting that our automatic labeling strategies are two particular instantiations for training the classifier, and that there could be other instantiations, which we leave as future work.

Analyses on Classifier SizeTo investigate the sensitivity of our classifier according to its varying sizes, we conducted further experiments. As shown in Table 6, we observe no significant performance differences among classifiers of various sizes, even with reduced complexity and fewer parameters in smaller classifiers. This indicates that our proposed classifier can contribute to resource-efficient settings in real-use cases with smaller sizes without compromising the performance.

Case StudyWe conduct a case study to qualitatively compare our Adaptive-RAG against Adaptive Retrieval. Table 5 shows the classified complexity and the query handling patterns for both simple and complex questions. First, for the simple single-hop question, our Adaptive-RAG identifies that it is answerable by only using the LLM's parametric knowledge about 'Google'. By contrast,

\begin{table}
\begin{tabular}{l c c c c c c} \hline  & \multicolumn{3}{c}{**QA**} & \multicolumn{3}{c}{**Classifier (Accuracy)**} \\ \cline{2-8}
**Training Strategies** & **F1** & **Step** & **All** & **No** & **One** & **Multi** \\ \hline \hline
**Adaptive-RAG (Ours)** & **46.94** & **1084** & **54.52** & **30.52** & **66.28** & **65.45** \\
**w/o Binary** & 43.43 & 640 & 60.30 & 62.19 & 65.70 & 39.55 \\
**w/o Silver** & 48.79 & 1464 & 40.00 & 0.00 & 53.98 & 75.91 \\ \hline \end{tabular}
\end{table}
Table 4: Results on QA and complexity classification with varying the data annotation strategies for training the classifier.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Labels** & **Time/Query (Sec.)** & **Percentage (\%)** \\ \hline \hline
**No (A)** & 0.35 & 8.60 \\
**One (B)** & 3.08 & 53.33 \\
**Multi (C)** & 27.18 & 38.07 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The exact elapsed time per query and the percentage of the predicted labels from the classifier over all samples.

 

Adaptive Retrieval fetches additional documents, leading to longer processing times and occasionally producing incorrect responses due to the inclusion of partially irrelevant information about 'Microsoft'. Meanwhile, faced with a complex question, Adaptive-RAG seeks out relevant information, including details like 'a son of John Cabot', which may not have been stored in LLMs, while Adaptive Retrieval fails to request such information from external sources, resulting in inaccurate answers.

## 6 Conclusion

In this work, we proposed the Adaptive Retrieval-Augmented Generation framework, referred to as Adaptive-RAG, to handle queries of various complexities. Specifically, Adaptive-RAG is designed to dynamically adjust its query handling strategies in the unified retrieval-augmented LLM based on the complexity of queries that they encounter, which spans across a spectrum of the nonretrieval-based approach for the most straightforward queries, to the single-step approach for the queries of moderate complexity, and finally to the multi-step approach for the complex queries. The core step of our Adaptive-RAG lies in determining the complexity of the given query, which is instrumental in selecting the most suitable strategy for its answer. To operationalize this process, we trained a smaller Language Model with query-complexity pairs, which are automatically annotated from the predicted outcomes and the inductive biases in datasets. We validated our Adaptive-RAG on a collection of open-domain QA datasets, covering the multiple query complexities including both the single- and multi-hop questions. The results demonstrate that our Adaptive-RAG enhances the overall accuracy and efficiency of QA systems, allocating more resources to handle complex queries while efficiently handling simpler queries, compared to the existing one-size-fits-all approaches that tend to be either minimalist or maximalist over varying query complexities.

### Limitations

While our Adaptive-RAG shows clear advantages in effectiveness and efficiency by determining the query complexity and then leveraging the most suitable approach for tackling it, it is important to recognize that there still exist potential avenues for improving the classifier from the perspectives of its training datasets and architecture. Specifically, as there are no available datasets for training the query-complexity classifier, we automatically create new data based on the model prediction outcomes and the inductive dataset biases. However, our labeling process is one specific instantiation of labeling the query complexity, and it may have the potential to label queries incorrectly despite its effectiveness. Therefore, future work may create new datasets that are annotated with a diverse range of query complexities, in addition to the labels of question-answer pairs. Also, as the performance gap between the ideal classifier in Table 1 and the current classifier in Figure 3 indicates, there is still room to improve the effectiveness of the classifier. In other words, our classifier design based on the smaller LM is the initial, simplest instantiation for classifying the query complexity, and based upon it, future work may improve the classifier architecture and its performance, which will positively contribute to the overall QA performance.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{**QA**} & \multicolumn{4}{c}{**Classifier (Accuracy)**} \\
**Sizes** & **F1** & **Step** & **All** & **No** & **One** & **Multi** \\ \hline \hline
**Small (60M)** & 45.83 & 964 & 53.48 & 26.65 & 70.62 & 53.18 \\
**Base (223M)** & 45.97 & 983 & 53.41 & 26.42 & 69.46 & 56.82 \\
**Large (770M)** & **46.94** & **1084** & **54.52** & 30.52 & 66.28 & 65.45 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results with varying model sizes for classifiers.

\begin{table}
\begin{tabular}{p{42.7pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline
**Dataset** & **Question** & **Adaptive Retrieval** & **Adaptive-RAG (Ours)** \\ \hline
**NQ** & Which famous corpora & **Query Type: B (Single-step Approach)** & **Query Type: A (Non Retrieval)** \\ (Single-hop) & rate logo changed to a flat colourful song sans & **Reasoning: The famous corporate log that changed to a flat color sans-saffent font in its first major change since 1999 is Microsoft. Microsoft changed its logo in 2012, replacing the italicized, multi-colored font with a simple, flat, grey font. So the answer is: Microsoft.** \\ \hline \hline
**MaSiQue** & Who is the child of the Italian navigator who explored the eastern coast of the continent & **Query Type: A (Non Retrieval)** & **Query Type: C (Multi-step Approach)** & **Reasoning: César Gaytan was born in Guadalajara, Jaísco. Mexico. The Italian navigator who explored the eastern coast of the continent for English was Giovanni Caboto, also known as John Cabot. So the answer is: Giovanni Caboto, Jahn Cabot. & **Query Type: C (Multi-step Approach)** & **Coast Resolution: César Gaytan was born in Guadalajara, Jalios. Mexico. The Italian navigator who explored the eastern coast of North America for the English was Giovanni Caboto, also known as John Cabot. John Cabot. John Cabot’s son is Sebastian Cabot. So the answer is: Sebastian Cabot.** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Case study with GPT-3.5 (Turbo), where we represent the factual error in red and the accurate information in blue.

 



## Ethics Statement

The experimental results on Adaptive-RAG validate its applicability in realistic scenarios, where a wide range of diverse user queries exist. Nonetheless, given the potential diversity of real-world user inputs, it is crucial to also consider scenarios where these inputs might be offensive or harmful. We should be aware that such inputs could lead to the retrieval of offensive documents and the generation of inappropriate responses by the retrieval-augmented LLMs. To address this challenge, developing methods to detect and manage offensive or inappropriate content in both user inputs and retrieved documents within the retrieval-augmented framework is essential. We believe that this is a critical area for future work.

## Acknowledgements

This work was supported by Institute for Information and communications Technology Promotion (IITP) grant funded by the Korea government (No. 2018-0-00582, Prediction and augmentation of the credibility distribution via linguistic analysis and automated evidence document collection), Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2023-00275747), and the Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City.

## References

* Anil et al. [2020] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choao, Aakanksha Chowdhery, Clement Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastiam Gehrmann, Lucas Gonzalez, and et al. 2023. Palm 2 technical report. _arXiv preprint arXiv:2305.10403_.
* Asai et al. [2024] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In _The Twelfth International Conference on Learning Representations_.
* Baek et al. [2023] Jinheon Baek, Soyeong Jeong, Minki Kang, Jong Park, and Sung Ju Hwang. 2023. Knowledge-augmented language model verification. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 1720-1736. Association for Computational Linguistics.
* Borgeaud et al. [2022] Sebastian Borgoeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Essen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_, volume 162 of _Proceedings of Machine Learning Research_, pages 2206-2240. PMLR.
* Brown et al. [2020]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prrafulla Dhariwal, Arvind Neelakantan, Prauxhy Chernaym, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* August 4, Volume 1: Long Papers_, pages 1870-1879. Association for Computational Linguistics.
* Cho et al. [2023] Sukmin Cho, Jeongyeon Seo, Soyeong Jeong, and Jong C. Park. 2023. Improving zero-shot reader by reducing distractions from irrelevant documents in open-domain question answering. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 3145-3157. Association for Computational Linguistics.
* Chung et al. [2020] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyu Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan

Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Ho et al. (2020) Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_, pages 6609-6625. International Committee on Computational Linguistics.
* 23, 2021_, pages 874-880. Association for Computational Linguistics.
* Izcard et al. (2023) Gautier Izcard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. _J. Mach. Learn. Res._, 24:251:1-251:43.
* Jeong et al. (2023) Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. 2023. Test-time self-adaptive small language models for question answering. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 15459-15469. Association for Computational Linguistics.
* Jiang et al. (2023) Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In _EMNLP 2023_.
* August 4, Volume 1: Long Papers_, pages 1601-1611. Association for Computational Linguistics.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, November 16-20, 2020_. Association for Computational Linguistics.
* Kasai et al. (2022) Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir R. Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime QA: what's the answer right now? _arXiv preprint arXiv:2207.13332_.
* Khattab et al. (2022) Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP. _arXiv preprint arXiv:2212.14024_, abs/2212.14024.
* Khot et al. (2023) Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Ilila Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Lazaridou et al. (2022) Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. 2022. Internet-augmented language models through few-shot prompting for open-domain question answering. _arXiv preprint arXiv:2203.05115_.
* Li et al. (2020) Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020. Efficient one-pass end-to-end entity linking for questions. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 6433-6441. Association for Computational Linguistics.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishrizi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 9802-9822. Association for Computational Linguistics.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lecrer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Z.

 

Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019_, pages 8024-8035.
* 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part II_, volume 13981 of _Lecture Notes in Computer Science_, pages 534-543. Springer.
* Press et al. (2023) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In _Findings of the Association for Computational Linguistics: EMNLP 2023_.
* Qi et al. (2021) Peng Qi, Haejun Lee, Tg Sido, and Christopher D. Manning. 2021. Answering open-domain questions of varying reasoning steps from text. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, Y-11 November, 2021_, pages 3599-3614. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21:140:1-140:67.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Loyprev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_, pages 2383-2392. The Association for Computational Linguistics.
* Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. _Transactions of the Association for Computational Linguistics_.
* Robertson et al. (1994) Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3. In _Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994_, volume 500-225 of _NIST Special Publication_, pages 109-126. National Institute of Standards and Technology (NIST).
* Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Mingoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yuh. 2023. REPLUG: retrieval-augmented black-box language models. _arXiv preprint arXiv:2301.12652_.
* Touvron et al. (2021) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyn Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Innan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloundann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu,uning Mao, Xavier Martinet, Todor Mihalyo, Pushkar Mishra, Igor Molbybo, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Scheiten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambout, Sharan Narang, Aurelein Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Trivedi et al. (2022a) Harsh Trivedi, Niranjan Balasubramanian, Tschar Khot, and Ashish Sabharwal. 2022a. Musique: Multihop questions via single-hop question composition. _Trans. Assoc. Comput. Linguistics_, 10:539-554.
* Trivedi et al. (2022b) Harsh Trivedi, Niranjan Balasubramanian, Tschar Khot, and Ashish Sabharwal. 2022b. \(\blacktriangle\) MuSiQue: Multihop questions via single-hop question composition. _Transications of the Association for Computational Linguistics_, 10:539-554.
* Trivedi et al. (2022c) Harsh Trivedi, Niranjan Balasubramanian, Tschar Khot, and Ashish Sabharwal. 2022d. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 10014-10037. Association for Computational Linguistics.
* Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. _Trans. Mach. Learn. Res._, 2022.
* Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. In _NeurIPS_.
* Wolf et al. (2022) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funowicz, 

Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos_, pages 38-45. Association for Computational Linguistics.
* Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Yang et al. (2019) Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Demonstrations_, pages 72-77. Association for Computational Linguistics.
* Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
* Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* 16, 2023_.
* Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. _arXiv preprint arXiv:2101.00774_.

 



## Appendix A Additional Experimental Setups

### Datasets

We use publicly open datasets for both single-hop and multi-hop QA datasets, referring to as Karpukhin et al. (2020) and Trivedi et al. (2023), respectively. We describe the characteristics of each dataset:

**1) SQuAD v1.1**Rajpurkar et al. (2016) is created through a process where annotators write questions based on the documents they read.

**2) Natural Questions**Kwiatkowski et al. (2019) is constructed by real user queries on Google Search.

**3) TriviaQA**Joshi et al. (2017) comprises trivia questions sourced from various quiz websites.

**4) MuSiQue**Trivedi et al. (2022a) is collected by compositing multiple single-hop queries, to form queries spanning 2-4 hops.

**5) HotpotQA**Yang et al. (2018) is constructed by having annotators create questions that link multiple Wikipedia articles.

**6) 2 WikiMultiHopQA**Ho et al. (2020) is derived from Wikipedia and its associated knowledge graph path, needing 2-hops.

### Models

We describe the details of models as follows:

**1) No Retrieval.** This approach uses only the LLM itself, to generate the answer to the given query.

**2) Single-step Approach.** This approach first retrieves the relevant knowledge with the given query from the external knowledge sources and then augments the LLM with this retrieved knowledge to generate the answer, which iterates only once.

**3) Adaptive Retrieval.** This baseline Mallen et al. (2023) adaptively augments the LLM with the retrieval module, only when the entities appearing in queries are less popular. To extract entities, we use the available entity-linking method Li et al. (2020), namely BLINK, for questions.

**4) Self-RAG.** This baseline Asai et al. (2024) trains the LLM to adaptively perform retrieval and generation, where the retrieval is conducted once it predicts the special retrieval token above a certain threshold, and the answer generation follows.

**5) Adaptive-RAG.** This is our model that adaptively selects the retrieval-augmented generation strategy, smoothly oscillating between the nonretrieval, single-step approach, and multi-step approaches4 without architectural changes, based on the query complexity assessed by the classifier.

Footnote 4: For the multi-step approach, we use the state-of-the-art question answering strategy from IRCoT Trivedi et al. (2023).

**6) Multi-step Approach.** This approach Trivedi et al. (2023) is the multi-step retrieval-augmented LLM, which iteratively accesses both the retriever and LLM with interleaved Chain-of-Thought reasoning Wei et al. (2022b) repeatedly until it derives the solution or reaches the maximum step number.

**7) Adaptive-RAG w/ Oracle** This is an ideal scenario of our Adaptive-RAG equipped with an oracle classifier that perfectly categorizes the query complexity.

### Implementation Details

For computing resources, we use A100 GPUs with 80GB memory. In addition, due to the significant costs associated with evaluating retrieval-augmented generation models, we perform experiments with a single run. Finally, we implemented models using PyTorch Paszke et al. (2019) and Transformers library Wolf et al. (2020).

## Appendix B Additional Experimental Results

Performance vs TimeWe further provide a comparison of different retrieval-augmented generation approaches with FLAN-T5-XL and FLAN-T5-XXL models in Figure 4 and Figure 5, respectively, in the context of performance and efficiency trade-offs. Similar to the observation made from the GPT-3.5 model in Figure 1, our proposed Adaptive-RAG is significantly more effective as well as efficient.

Figure 4: QA performance (F1) and efficiency (Time/Query) for different retrieval-augmented generation approaches. We use the FLAN-T5-XL (3B) as the base LLM.

Figure 5: QA performance (F1) and efficiency (Time/Query) for different retrieval-augmented generation approaches. We use the FLAN-T5-XXL (11B) as the base LLM.



Performance per DatasetIn addition to detailing the performance of each dataset with the FLAN-T5-XL model, as shown in Table 2, we also present the results for each dataset with the FLAN-T5-XXL and GPT-3.5 models in Table 2 and Table 8, respectively. The experimental results show that our Adaptive-RAG consistently balances between efficiency and accuracy. It is worth noting that while the GPT-3.5 model performs effectively in addressing straightforward queries even without document retrieval, it benefits significantly from our Adaptive-RAG in terms of effectiveness when solving complex multi-hop queries.

\begin{table} width=1 

\end{table}
Table 7: Results on each of a collection of datasets with FLAN-T5-XXL (11B) as the LLM. We emphasize our results in bold.

\begin{table} width=1 

\end{table}
Table 8: Results on each of a collection of datasets with GPT-3.5 (Yurbo) as the LLM. We emphasize our results in bold.