MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
Authors: Yixuan Tang
Yi Yang
Venue: COLM (2024)
Keywords: Retrieval-Augmented Generation; Benchmark; Multi-Hop Reasoning
TL;DR: Evaluate retrieval and reasoning across documents in the RAG pipelines.

Abstract: Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge, showing promising potential in mitigating LLM hallucinations and enhancing response quality, thereby facilitating the great adoption of LLMs in practice. However, we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. Furthermore, to our knowledge, no existing RAG benchmarking dataset focuses on multi-hop queries. In this paper, we develop a novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence. We detail the procedure of building the dataset, utilizing an English news article dataset as the underlying RAG knowledge base. We demonstrate the benchmarking utility of MultiHop-RAG in two experiments. The first experiment compares different embedding models for retrieving evidence for multi-hop queries. In the second experiment, we examine the capabilities of various state-of-the-art LLMs, including GPT-4, PaLM, and Llama2-70B, in reasoning and answering multi-hop queries given the evidence. Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries. We hope MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice. We make the dataset and benchmarking code publicly available via GitHub.

Published as a conference paper at COLM 2024
MultiHop-RAG: Benchmarking Retrieval-Augmentte Gener- ation for Multi-Hop Queries
Yixuan Tang, Yi Yang
The Hong Kong University of Science and Technology [yixuantang, i hk
Abstract
Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge, showing promising potential in mitigating LLM hallucinations and enhancing response quality, thereby facilitating the great adoption of LLMs in practice. However, we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. Furthermore, to our knowledge, no existing RAG benchmarking dataset focuses on multi-hop queries. In this paper, we develop a novel dataset, MultiHop-RAG, which consists of a knowledge base, large col- lection of multi-hop queries, their ground-truth answers, and the associated supporting evidence. We detail the procedure of building the dataset, uti- lizing an English news article dataset as the underlying RAG knowledge base. We demonstrate the benchmarking utility of MultiHop-RAC in two experiments. The first experiment compares different embedding models for retrieving evidence for multi-hop queries. In the second experiment, we examine the capabilities of various state-of-the-art LLMs, including GPT-4, PaLM, and Llama2-70B, in reasoning and answering multi-hop queries given the evidence. Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries. We hope MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice. We make the dataset and benchmarking code publicly available via GitHub?.
Introduction
The emergence of large language models (LLMs), such as ChatGPT, has fostered wide range of innovations, powering intelligent chatbots and other natural language processing (NLP) applications (OpenAI, 2023). One promising use case is Retrieval-Augmented Gener- ation (RAG) (Asai et al., 2023), which optimizes the output of a large language model by referencing an external knowledge base outside of the LLM training data sources before generating response. RAG improves LLM's response (Borgeaud et al., 2022) and also mitigates the occurrence of hallucinations, thereby enhancing the models' credibility (Gao et al., 2023). LLM-based frameworks, such as LlamaIndex (Liu, 2022) and LangChain (Chase, 2022), specialize in supporting RAG pipelines.
In real-world Retrieval-Augmeente Generation (RAG) applications, user's 's query often necessitates retrieving and reasoning over evidence from multiple documents, a process known as multi-hop query. For instance, consider financial analysis using a database of financial reports. A financial analyst might query, Which company among Google, Apple, and Nvidia reported the largest profit margins in their third-quarter reports for 2023? or inquire about specific company's performance over time, such as How does Apple's sales trend look over the past three years? These queries require evidence from multiple documents to formulate an answer. Due to the multifaceted nature of such queries, involving information from various

---

Published as a conference paper at COLM 2024
Multi Documents
Which company among Google, Apple, and Nvidia reported largest profit margins in their third- quarter reports for 2023?
Google
Chunk
Embedding
Apple
Chunk
Vector Database
Nvidia
Chunk
Prompt
LLM
Query
Response
Context
Figure 1: RAG with multi-hop query.
sources, traditional similarity matching methods like cosine similarity between query and financial report chunk embeddings might not yield optimal results. We demonstrate this multi-hop retrieval process in Figure
However, existing RAG benchmarks, such RGB (Chen et al., 2023) and RECALL (Liu et al., 2023), mainly evaluate simple case where the answer of query can be retrieved and solved using one single piece of evidence. None of these benchmarks assess the retrieval and reasoning capability of LLMs for complex multi-hop queries. To address this gap and make RAG benchmarking more closely resemble real-world scenarios, in this paper, we introduce MultiHop-RACe To our knowledge, MultiHop-RAG is one of the first RAG datasets focusing specifically on multi-hop queries.
Based on the RAG queries commonly encountered in real-world scenarios, we first cate- gorize multi-hop queries into four types: Inference query, Comparison query, Temporal query, and Null query. The first three types Inference, Comparison, and Temporal - require the retrieval and analysis of evidence from multiple sources, encompassing tasks like inferring relationships, comparing data points, and sequencing events over time. The Null query represents a scenario where the query cannot be derived from the knowledge base. This category is crucial for assessing whether LLM might hallucinate an answer to multi-hop query when the retrieved text lacks relevance.
We construct our RAG knowledge base using collection of news articles. Using GPT-4 data generator, we then take an extensive procedure to construct a diverse set of multi-hop queries, each requiring the retrieval and reasoning of query construction is shown Table First, we begin by extracting factual sentences from each news article evidence. For example, an extracted piece of evidence from an article may state: "Back then, just like today, home prices had boomed for years before Fed officials were ultimately forced to hike interest rates aggressively in an attempt to fight inflation." Second, we input each evidence piece into GPT-4, prompting it to rephrase the evidence into claim. This claim is clarified with a disambiguated topic and entity. For instance, GPT-4 might rephrase the aforementioned evidence into: "Federal Reserve officials were forced to aggressively hike interest rates to combat inflation after years of booming home prices", identifying "Interest rate hikes to combat inflation" as the topic and "Federal Reserve" as the entity. These topics and entities act as bridges for constructing multi-hop queries, known multi-hop queries related to the same bridge-topic or bridge-entity, accompanied by the correct answers. Lastly, undertake validation step to ensure the data quality.
multiple documents. An example
wridge-topic bridge-entity. Next, we GPT-4 to generate specific
We demonstrate the benchmarking capabilities of MultiHop-RAse using two experiments, utilizing RAG system implemented with LlamaIndex (Liu, 2022). The first experiment
2

---

Published as a conference paper at COLM 2024
News source
Fortune Magazine
The Sydney Morning Herald
Evidence
Back then, just like today, home prices had boomed for years before Fed officials were ultimately forced to hike interest rates aggres- sively in an attempt to fight inflation.
Postponements of such reports could compli- cate things for the Fed, which has insisted it will make upcoming decisions on interest rates based on what incoming data say about the economy.
Claim
Federal Reserve officials were forced to aggres sively hike interest rates to combat inflation after years of booming home prices. Interest rate hikes to combat inflation Federal Reserve
The Federal Reserve has insisted that will base upcoming decisions on interest rates on the incoming economic data. Interest rate decisions based on economic data Federal Reserve
Bridge-Topic
Bridge-
Entity
Query
Does the article from Fortune suggest that the Federal Reserve's is interest rate hikes are a response to past conditions, such as booming home prices, while The Sydney Morning Herald article indicates the Federal Reserve's 's future interest rate decisions will be based incoming economic data??
Answer
Yes
Table 1: An example of a multi-hop query, including supporting evidence from two news articles, the paraphrased claim, the bridge-topic and bridge-entity, and the corresponding answer.
involves comparison of different embedding models for retrieving relevant evidence for multi-hop queries. In the second experiment, we assess the reasoning and answering abilities of various state-of-the-art LLMs, including GPT-4, GPT-3.5, PaLM, Claude-2, Llama2-70B, and Mixtral-8x7B, for multi-hop queries when retrieved text is provided. The results from both experiments indicate that the current RAG implementations are inadequate for effectively retrieving and answering multi-hop queries, We publicly release this challenging MultiHop-RACC dataset and hope will be a valuable resource for the community in developing and benchmarking RAG systems, thereby unleashing the great potential of generative AI in practice.
2 RAG with multi-Hop queries
2.1 Retrieval-augmented Generation (RAG)
In an RAG application, we utilize an external corpus, denoted as D, which comprises multiple documents and serves as the knowledge base. Each document within this corpus, represented as di E 'D, is segmented into set of chunks. These chunks are then transformed into vector representations using an embedding model and stored in embedding database. Given a user query the system typically retrieves the top-K chunks that best match the query. These chunks constitute the retrieval set for query 4, represented as Ry {ri, "x. The retrieved chunks, combined with the query and optional prompt, are then fed into an LLM to generate a final answer, following the format: LLM(q, Rg. prompt) + answer.
2.2 Multi-Hop Query
We define a multi-hop query as one that requires retrieving and reasoning over multiple pieces of supporting evidence to provide an answer. In other words, for a multi-hop query 9, the chunks in the retrieval set R, collectively provide an answer to q. For example, the query "Which company among Google, Apple, and Nvidia reported the largest profit margins in their third-quarter reports for 2023?" requires 1) retrieving relevant pieces of evidence related to profit margins from the reports of the three companies; 2) generating an answer by comparing and reasoning from the multiple pieces of retrieved evidence. This differs from a single-hop query such as "What is Google's profit margin in the third reports for 2023," " where the answer be directly derived from single piece of evidence.
3

---

Published as a conference paper at COLM 2024
Based the queries commonly used in real-world RAG systems, we identify four types of multi-hop queries. For each type, we present hypothetical query within the context of financial RAG system, where the knowledge base consists of collection of annual reports.
Inference query: For such a query 4, the answer is deduced through reasoning from the retrieval set Rq. An example of an inference query might be: Which report discusses the supply chain risk of Apple, the 2019 anmual report or the 2020 anmual report?
Comparison query: For such query 9, the answer requires comparison of evidence within the retrieval set Rq. For instance, a comparison query might ask: Did Netflix or Google report higher revenue for the year 2023?"
Temporal query: For such a query 9, the answer requires an analysis of the temporal information of the retrieved chunks. For example, a temporal query may ask: Did Apple introduce the AirTag tracking device before or after the launch of the 5th generation iPad Pro?
Null query: For such query 9, the answer cannot be derived from the retrieved set Ry.. We include the null query to assess the generation quality, especially regarding the issue of hallucination. For a null query, even though retrieved set provided, an LLM should produce null response instead of hallucinating an answer. For example, assuming ABCD non-existent company, null query might ask: What are the sales of company ABCD as reported in its 2022 and 2023 annual reports?
2.3 Evaluation Metrics
An RAG system handling multi-hop queries can be assessed from two key aspects: retrieval evaluation and generation evaluation.
Evidently, the quality of the retrieval set R. determines the final generation quality. We compare the retrieved set with the ground truth evidence associated with each query, except for the null queries, as they have no evidence to derive from. Assuming the top-K chunks are retrieved, i.e, R2 K, we use retrieval evaluation metrics including Mean Average Precision at K (MAP@K), Mean Reciprocal Rank at K (MRR@K), and Hit Rate at K (Hit@K). MAP@K measures the average top-K retrieval precision across all queries. MRR@K calculates the average of the reciprocal ranks of the first relevant chunk for each query, considering the top-K retrieved set. Hil@K metric measures the fraction of evidence that appears in the top-K retrieved set.
Retrieval Evaluation:
Response Evaluation: Since the multi-hop query requires reasoning over multiple pieces of retrieved chunks, can also evaluate the reasoning capability of the LLM by comparing the LLM response with the ground truth answer the query.
3 A Benchmarking Dataset: MultiHop-RAG
In this section, we provide detailed information on the construction of the MultiHop-RAG dataset. Specifically, we describe the process of creating set of multi-hop queries, along with the corresponding ground truth evidence sets and answers derived from a collection of news articles.
3.1 MultiHop-RAG Construction
Step 1: Dataset Collection. We download news dataset using the mediastack API 2, REST API interface delivering worldwide news data. The news data source comprises various English-language websites covering a range of news categories: entertainment, business, sports, technology, health, and science. To mimic real-world RAG scenarios, where the knowledge base data, such as an enterprise's internal data, may differ from the L.MMs' training data, select news articles published from September 26, 2023, to December 26, 2023. This timeframe extends beyond the knowledge cutoff of some widely-used LLMs, including ChatGPT and LLaMA, as of the time of writing. This selection also helps in teasing
Phttps://mediastack.com
4

---

Published as a conference paper at COLM 2024
Dataset Collection
Evidence Extraction
Claim Generation
Claim Generation
Download Dataset
Extract Factual Sentences
Bridge-Entity Generation
Select
Sentences
Bridge-Topic Generation
Quality Assurance
Query
Generation
Manually Review
Inference
Comparison
MultiHop-RAG RAG
GPT-4 Review
Null
Temporal
Figure 2: MultiHop-RAG Construction Pipeline.
out the possibility of the underlying LLM having been exposed to these news articles. We only keep articles with a token length greater than or equal to 1,024. Every news article is paired with metadata, including the title, publish date, author, category, URL, and news source.
Step 2: Evidence Extraction. For each article, extract factual opinion sentences using trained language model 3. These factual sentences are later used as evidence for answering multi-hop queries. We retain only those news articles containing evidence that may have overlapping keywords with other news articles. This allows to later create multi-hop queries where the answer's 's evidences are drawn from multiple sources.
Step Claim, Bridge-Entity, Bridge-Topic Generation. Our goal to use GPT-4 to automatically generate high-quality multi-hop queries using the evidence set. However, the raw evidence obtained from Step is not ideal for query generation due inconsistency in linguistic structure. For example, some pieces of evidence use pronouns to refer to subjects and lack the actual entity in the text. To address this, we employ GPT-4 to paraphrase the evidence, which we refer to claims, given the original evidence and its context. To ensure consistency between the generated claim and the evidence, we further perform fact-checking using the UniEval (Zhong et al., 2022) framework to verify the alignment between the evidence and claim. Appendix A presents the prompt used for GPT-4 for claim generation.
Gridge-Entity and Bridge-Topic: The shared entity or topic across pieces of evidence is referred to the bridge-entity bridge-topic. These bridge-entities or bridge-topics can be used to link different pieces of evidence from which a multi-hop query's 's answer is derived. For example, in claim such as "Google reports its third-quarter results for 2023, showcasing a detailed overview of its financial performance, including revenue growth, profit margins", the term profit margin can be viewed as bridge-topic and the term Google can be viewed bridge-entity that links the different pieces of evidence. We prompt GPT-4 to identify the bridge-entity and bridge-topic for each claim. Appendix A also presents the prompt used for GPT-4 for bridge generation.
Step Query and Answer Generation. In this step, we leverage the bridge-entity or bridge-topic to generate multi-hop queries. Specifically, we first group the claims having the same bridge-entity or bridge-topic into claim set. We restrict the claim to have at least two claims but no more than four claims. For each type of query, we feed the claim set to GPT-4 and prompt it with an instruction to generate a query with information from each claim. Below, we explain the specifications for different multi-hop query types. In the construction each query, we also include the source the news article where the

---

Published as a conference paper at COLM 2024
Num. Evidence Needed
Count
Percentage
Query Category
Entry Count
Percentage
(Null Query)
301
11.78%
Inference Query
816
31.92%
2
1078
42.18%
Comparison Query
856
33.49%
3
779
30.48%
Temporal Query
583
22.81%
398
15.56%
Null Query
301
11.78%
Total
2,356
100.00 %
Total
2,556
100.00 %
Table 2: The distribution of the number of ev- idence required to answer multi-hop queries in MultiHop-RAG.
Table 3: The distribution of query types in MultiHop-RAG.
supporting evidence is associated with mimicking real-world RAG scenarios. Appendix A presents the prompts used for GPT-4 for query generation.
Inference Query: These queries formulated by synthesizing the various characteriza- tions of the bridge-entity across multiple claims, with the final answer being the identifica- tion of the entity itself.
Comparison Query: These queries are structured to compare the similarities and differences related to the bridge entity or topic. The resultant answer to such queries is typically definitive "yes" or "no", based on the comparison.
Temporal Query:
These queries explore the temporal ordering of events across different points in time. The answer to such queries is typically a "yes" or "no" or single temporal indicator word like "before" or "after".
Null Query: Null query is a query whose answer cannot be derived from the retrieved set. To create null queries, we generate multi-hop queries using entities that do not exist in the existing bridge-entities. To add complexity, we also include fictional news source metadata when formulating these questions, ensuring that the questions do not reference any contextually relevant content from the knowledge base. The answer to the null query should be "insufficient information" " or similar.
Step 5: Quality Assurance. Finally, we use two approaches to reassure the dataset quality, First, we manually review a subset sample of the generated multi-hop queries, their cor- responding evidence sets, and the final answers. The results of the manual examination indicate a high degree of accuracy and data quality. Second, we utilize GPT-4 to assess each example in the dataset against the following criteria: 1) The generated query must utilize all provided evidence in formulating the response; 2) The query should be answerable solely based on the provided evidence; 3) The response the generated query should be either single word or a specific entity; 4) The query must conform to its designated query type.
3.2 Descriptive Statistics
The MultiHop-RAG dataset contains six different types of news articles, covering 609 distinct news, with average of 2,046 tokens. The distribution of the news categories is shown in Table 4. MultiHop-RAC, contains four types of multi-hop queries and the distribution of these queries is shown Table In total, about 88% of queries the dataset are non-null queries where answers can be retrieved and reasoned from the knowledge base. In addition, the form of queries exhibits considerable diversity. Approximately 27% of interroga tive queries start with "does," around 15% initiate with "what," similar proportion start "which," and 14% begin with "who," " with the remainder incorporating a small percentage of other interrogative words such as "when." Moreover, the number of evidence required to answer a multi-hop query varies. Table 2 shows the distribution of evidence numbers for each query in the dataset. Around 42% of queries can be answered using two pieces of evidence, approximately 30% and 15% of queries can be answered using three or four pieces of evidence, respectively.

---

Published as a conference paper at COLM 2024
Category
Avg. Tokens
Entry Count
technology
2262.3
172
entertainment
2084.3
114
sports
2030.6
211
science
1745.5
21
business
1723.8
health
1481.1
10
total
2046.5
609
Table 4: Descriptive statistics of the news article knowledge base MultiHop-RAG.
4
Benchmarking RAG system using MultiHop-RAG
MultiHop-RAG can be used benchmark for various RAG-related tasks. Broadly speak- ing, RAG-related tasks can be categorized as retrieval-related tasks and generation-related tasks. retrieval-related task focuses on retrieving relevant text from the knowledge base, while generation-related task focuses on generating high-quality responses given the retrieved text. In this section, we showcase two use cases for each task where MultiHop-RAG can be employed.
4.1 Retrieval-related Task
An important design choice in RAG system is the selection of the embedding model. An embedding model converts data into numerical vectors and subsequently stores these vectors in embedding databases. In this experiment, we evaluate different embedding models by examining their retrieval quality.
Experiment Setup: We implement an RAG system using the LlamaIndex framework (Liu, 2022). We partition the documents in the MultiHop-RAG knowledge base into chunks, each consisting of 256 tokens. We then convert the chunks using an embedding model and save the embeddings into a vector database. Similarly, in the retrieval step, we convert a query using the same embedding model and retrieve the top-K most relevant chunks that have the highest cosine similarity with the query embedding. In this experiment, we test a variety set of embedding models, including the ada-embeddings by OpenAl (fext-ebbedding-ddd--)
et al., 2023), bge-large-en-vi.5 (Xiao et al., 2023), voyage-02 llm-embedderr (Zhang (Günther et al., 2023), e5-base-v2 (Wang et al., 2022), and instructor-large (Su et al., 2023). NULL queries excluded in this experiment because there is matching evidence to the query. Additionally, we also include a Reranker module to examine the retrieval performance, using bge-reranker-large (Xiao al., 2023). After retrieving 20 related chunks using the embedding model, we further select the top-K chunks using the Reranker.
coxtssearch-add--ueey----
Experiment Result: Table 5 shows the retrieval result of using different embedding models. It shows that there is still significant gap in retrieving relevant evidence for the multi-hop queries. While Rerank can effectively improve retrieval relevance, the highest Hits@10 is only 0.7467 when the Reranker technique is used. Moreover, the drop in the highest Hits@4 to 0.6625 is worrisome. In practical RAG systems, the underlying LLM often has a context window limit. As result, the number of retrieved chunks is usually restricted to small number. The low values of the retrieval metrics highlight the challenges in retrieving relevant pieces of evidence for multi-hop queries when using direct similarity matching between the multi-hop query and text chunks.
4.2
Generation-related Task
The underlying LLMs play a crucial role in generating responses in an RAG system. In this experiment, we evaluate the quality of generated responses under two different settings. In
hhttss//wwwwxxyyagee.com

---

Published as a conference paper at COLM 2024
Without Reranker
With bge-
Embedding
MRR@10
MAP@10
Hits@10
Hits@4
MRR@10
MAP@10
Hits@10
Hits@4
rext-embeedding-ad--0222
0,4203
0.3431
0.5040
0.5477
0.4625
0.7059
0.6169
text-searc--ade-uuryyy
0.4203
0.3431
0.6399
0.5031
0.5483
0.4625
0.7064
0.6174
lIm-embedder
0.2558
0.1725
0.4499
0.3189
0,4250
0.3059
0,5478
0.4756
bge-large-en-v1.5
0.4298
0.3423
0.5221
0.5630
0.4759
0.7183
0.6364
jina wew.eembbinnngg--bbaeeeeee
0,0621
0.0310
0.1479
0.0802
0,1412
0.0772
0.1909
0.1639
intfloat/ e5-base-v22
0.1843
0.1161
0.3556
0.2334
0.3237
0.2165
0.4176
0.3716
voyage-02
0.3934
0.3143
0.6506
0.4619
0.5860
0.4795
0.7467
0.6625
paanip/iinttucttrllare
0,3458
0.2650
0.5717
0.4229
0.5115
0.4118
0.6590
0.5775
Table 5: Retrieval performance of different embedding models.
Accuracy
Models
Retrieved Chunk Ground-truth Chunk
GPT-4
0.89
ChatGPT
0.44
0.57
(lama-2-7Ub-cha-hh
0.28
0,32
Mixtral-&x<U--Isttrree
0.32
0.36
Claude-2.1
0.52
0.56
Google-PaI.M
0.47
0174
Table 6: Generation accuracy of LLMs.
the first setting, we employ the best-performing retrieval model, namely voyage-02 with bge-reranker-large,
indicated in Table 5, to retrieve the top-K texts and then feed them into the LLM. In the second setting, we use the ground-truth evidence associated with each query as the retrieved text for the L.M. This setting represents ceiling performance for testing the LLM's response capabilities, as it utilizes the actual evidences.
Experiment Setup: In the first experiment, retrieve top-6 chunks that the total length of the retrieved text does not exceed 2,048. All queries in MultiHop-RAG are tested in the experiment. In the second experiment, since the null queries do not have associated evidence, we exclude this type of query in the experiment. For the LLMs used in the experiment, we consider state-of-the-art commercial models, including GPT-4 (OpenAI, 2023), GPT-3.5, Claude-2 (Anthropic, 2023), and Google-Pal MaagM (Google, 2023). We obtain answers using the provided APT of the respective models. We also assess some open- source models, including Mixtral-8&77-instruud (Jiang et al., 2024) and -lama-2-70b-chat-ht (Touvron al., 2023).
Table 6 shows the response accuracy of different LLMs. First, we can see that the response accuracy rate using the retrieved chunks not satisfactory, with the state-of-the-art GPT-4 model achieving only 0.56 accuracy. This is expected because the retrieval component falls short in retrieving relevant evidence from the knowledge base. Second, even when provide the LLM with ground-trute evidence, the response accuracy is far from being perfect. Open source LLM, such as Llama02-70B and Mixtral-8x7B, only achieve an accuracy of 0.32 and 0.36, respectively. GPT-4 achieves strong reasoning capability with an accuracy of 0.89, followed by the second-based LLM Google-PaLM with an accuracy of 0.74.
Experiment Results:
see that
Figure shows the detailed results of different query types for GPT-4 and Mixtral-8x7B- instruct. Both models show relatively high robustness on null queries, meaning they are generally good at determining when a query cannot be answered based on the retrieved text. This encouraging because one benefit of RAG to mitigate the LLM hallucination issue by augmenting LLM with retrieval knowledge. However, Mixtral-8x7B model performs significantly worse than the GPT-4 in comparison and temporal queries. Upon reviewing the incorrect responses, we find that Mixtral-8x77 fails to accurately handle logical negation, leading to misinterpretation of statements and thus low performance in the comparison queries. In addition, Mixtral-8x7B often fails to correctly identify the chronological order of events, which is crucial for answering temporal queries where timing is a key factor. Taken together, this experiment demonstrates that there is still room for improvement in
8

---

Published as a conference paper at COLM 2024
Retrieved Chunk
Ground-truth Chank
Mixtni-8x788
mill
comparison
CPT4
comparison
temporal
temporal
inference
inference
20
40
60
100
Accuracy
Accunssy (h4)
Figure 3: Generation accuracy for different query types.
the reasoning capabilities of LLMs, particularly those that are open-source, for multi-hop queries.
4.3 Other Use Cases
Beyond embedding models and LLM generation, there are other areas worth exploring. For example, query decomposition is a widely utilized technique in RAG frameworks, such as LLamalndex. This process involves breaking down the query into sub-questions/ it targets single document for retrieval and integrates the information subsequently, thereby poten- tially enhancing retrieval accuracy. Another advanced and promising approach involves building LLM-based agents that can automatically plan and execute multi-hop queries, such as AutoGPT (Gravitas, 2023). AutoGPT breaks down complex instructions into smaller steps. It analyzes feedback to continuously refine its response until the final decision. An- other promising direction is the exploration of hybrid retrieval approaches. These methods combine the precision of keyword-basee search with the contextual understanding afforded by embedding. By leveraging the strengths of both approaches, hybrid models aim to deliver more accurate and relevant search results. Overall, we believe that there are many potential areas for enhancing RAG's performance on multi-hop queries, and the curated dataset MultiHop-RAG can be valuable resource the community.
Related Work
RAG Evaluation: As RAG systems gain increasing popularity, a variety of RAG benchmark- ing datasets and evaluation tools have been developed. For instance, RGB (Chen et al., 2023) and RECALL (Liu et al., 2023) evaluate the performance of LLMs in generating responses for RAG systems under conditions involving noisy, integrative, and counterfactual queries. However, both datasets primarily focus on evaluating the generation aspect RAG systems without specifically addressing their retrieval accuracy. In addition, recent advancements have been made in automated RAG evaluation tools, such as ARES (Saad-Falcon et al., 2023) and RAGAS (Es et al., 2023). These tools utilize LLMs to automatically assess the quality of RAG generation, yet they do not introduce benchmarking datasets. Our work introduces one of the first RAG benchmarking datasets, consisting of knowledge base, large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence, thereby complementing existing RAG evaluations.
Retrieval datasets: Apart from the context of RAG, several benchmarking datasets exist for information retrieval evaluation. The FEVER (Fact Extraction and VERification) dataset, for instance, contains claims classified as Supported, Refuted, or NotEnoughInfo by the given Wikipedia article (Thorne et al., 2018). Similarly, the SciFact dataset comprises scientific claims paired with evidence-containing abstracts (Wadden et al., 2020). However, the claims in both datasets are single-hop statements, and the supporting evidence is from one single article, in contrast to the multi-hop queries discussed in this paper. Another dataset, Ho Ver, involves claims that require extracting and reasoning from multiple Wikipedia articles (Jiang et al., 2020). However, unlike our dataset, HoVer focuses solely on classifying claims as

---

Published as a conference paper at COLM 2024
either supported or not supported by the articles without evaluating an LLM generation step. Moreover, in HoVer, the Wikipedia articles from which evidence is drawn are given for claim verification, which is significantly different from our setting, where relevant pieces of evidence need to be extracted from a large knowledge base. Separately, (Kamalloo et al., 2023) evaluates a range of commercial embedding APIs for information retrieval, but this evaluation is not contextualized within the framework of RAG systems either.
Multi-document QA datasets: Question-answering (QA) is a fundamental task in NLP, and several popular benchmarks, such as HotpotQA (Yang et al., 2018), MultiRC (Khashabi et al., 2018), and Hop (Ho al., 2020), aim answer the question with multiple sources. This task is similar to our multi-hop query RAG task, as both involve reasoning from multiple sources of information. However, existing datasets are not tailored for the RAG task as they lack a retrieval step. Even modifying multi-hop query dataset to assess retrieval performance by integrating the input evidence into corpus does not adequately reflect the demands of real-world RAG tasks. A primary concern is that these datasets, often sourced from Wikipedia, overlap with training data used for large language models (LLMs). In practical RAG scenarios, datasets typically comprise private сorpora with metadata like titles or timestamps. This overlap can compromise the evaluation process, raising concerns that LLM responses may depend more on memorized training data rather than on reasoning from the retrieved knowledge base. The proposed dataset, in contrast, is specifically crafted for RAG tasks, including those designed to simulate real-life scenarios. It addresses these issues by introducing a very recent corpus paired with relevant metadata. This dataset can serve as benchmark to evaluate multi-evidence retrieval and multi-hop evidence generation capabilities.
6 Conclusion
In this work, we introduce MultiHop-RAG, novel and unique dataset designed for queries that require retrieval and reasoning from multiple pieces of supporting evidence. These types of multi-hop queries represent user queries commonly encountered in real-world scenarios. MultiHop-RAG consists of a knowledge base, a large collection of multi-hop queries, their ground-trute answers, and the associated supporting evidence. This paper details the creation process of MultiHop-RAG, employing hybrid approach that integrates human effort with GPT-4. Additionally, we explore two use cases of MultiHop-RAC in the benchmarking of RAG systems, thereby highlighting the potential applications of this dataset. By publicly releasing MultiHop-RAG, we aim to provide valuable resource to the community, contributing to the advancement and benchmarking of RAG systems.
Limitations
This work has several limitations that can be improved in future research. First, our ground truth answers are restricted to simple responses such as "yes", "no", entity names, or temporal indicators like "before" or "after" to facilitate the use of a straightforward accuracy metric for evaluating generation performance. Future work could consider allowing free text as answers and employing more sophisticated metrics to assess generation quality. Second, the current dataset limits supporting evidence for query to maximum of four pieces. Future work can extend the dataset by including queries that require retrieving and reasoning from even more evidence. Lastly, while our experiments utilize a basic RAG framework using LlamaIndex, future work could involve evaluating the answering of multi-hop queries using more advanced RAG frameworks or LLM-agent frameworks.
References
Anthropic. Claude 2.1 (May claude 2.1.
http:///ppp.aattppppccccc/////// 2023.
10

---

Published as a conference paper at COLM 2024
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications. In Proceedings of the 61st Anmual Meeting of the Association for Computa- fional Linguistics (Volume 6: Tutorial Abstracts), pp. 41-46, 2023.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saf- fron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2206-2240, PMLR, 17-23 Jul 2022. URL html.
https:/ https://proceeingg. mlr. prres//162/bbrgeeeee
Harrison Chase. LangChain, October 2022. URL https://github.... wowphuubacoommaaacaciiiiiaa-aaaa langchain.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation, 2023.
Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation, 2023.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations, 2023.
Google. PaLM 2 (May version). https: http:://geerrtiiiibnuuugggggool http://wwww....oogogooeeeooo/o/a models/, 2023. chat-bison-002.
Significant Gravitas. Autogpt. 2023. URL https://agpt.co
5
Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mo- hammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. Jina embeddings 2: 8192-token general- purpose text embeddings for long documents, 2023.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609-666 6625, Barcelona, Spain (Online), December 2020. Inter- national Committee on Computational Linguistics. doi: 10. 580. URL https: htttp::/aallntholline www.aannoooogyorg//ee..... col mooling-miin... 580.
main.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. HoVer: A dataset for many-hop fact extraction and claim verification. In Findings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso- Hermelo, Mehdi Rezagholizadeh, and Jimmy Lin. Evaluating embedding apis for infor- mation retrieval. Xri preprint Xrx::22305.666 00030000060,,,,, 2023.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple
11

---

Published as a conference paper at COLM 2024
Sentences. In Proc. of the Anmual Conference of the North American Chapter of the Asso- ciation for Computational Linguistics (NAACL), 2018. URL http::/cogcomg org/ hatg/ppprs//////// MM1-MultiRC-NNALL. pdf.
Jerry Liu. Llamalndex, 11 2022. URL https:
Yi Liu, Lianzhe Huang, Shicheng Li, Sishuo Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Recall: A benchmark for lims robustness against external counterfactual knowledge, 2023.
OpenAl. GPT4 (Nov 7 version).
com/chat, 2023, gpt-t-1106-preieew
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. Ares: An automated evaluation framework for retrieval-augmened generation systems, 2023,
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction- finetuned text embeddings,, 2023.
James Thorne, Andreas Vlachos, Christos Christodoullpoulls, and Arpit Mittal. Fever: large-scale dataset for fact extraction and verification, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7534-7550, Online, November 2020, Association for Computational Linguistics. doi: 10.18653/ vv/ w11/020.emnllmmmaatee.e 600. URL https: //tpa//aaaaaanooy....// wrw.org/////// orgg/220.000 emnlp-main. 609.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXio:2212.03533, 2022.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged re- sources to advance general chinese embedding, 2023.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (ÉMNLP), 2018
Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. Retrieve anything to augment large language models, 2023.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified multi-dimennional evaluator for text generation, 2022.
12

---

Published as a conference paper at COLM 2024
A Appendix A: GPT-4 Prompts Used for Data Generation
We present the prompts used for guiding GPT-4 for data generation. Table 7 shows the prompt used for claim generation, along with the corresponding topics and entities within these claims. Table 8, Table 9, and Table 10 respectively show the prompts used for generat- ing multi-hop queries of the inference, comparison, and temporal types.
A "claim" is statement assertion made within text expressing belief, opinion, or fact. Given evidence from the original context, please extract one claim and its associated topics.
Note: The claim should not contain ambiguous references, such as 'he', she,' and' it', and should use complete names. If there are multiple topics, give the most dominant one. The target of the claim (one entity)is the specific individual, group, or organization that the statement or assertion within a text is directed towards or about which it is making case. The topic of the claim should be a simple phrase representing the claim's central argument concept. If there is claim, please leave it blank. Please generate claim based on the given evidence. Don't generate the evidence yourself.
Please give the response following this format:
Evidence: [original context]
Claims: [extract claim]
Claim Target: [target]
Claim Topic: [topic]
Here are examples: (examples)
Now, it's your turn. (Nws))
(evidence)
Table 7: Claim Generation Prompting
A multi-hop question is query requiring multiple inferential leaps or accessing several pieces of information from different locations or sources to arrive at an answer. The following news articles' metadata and claims come from the articles. All the claims from the article are related to a similar target. Your task is to generate one multi-hop inference question based on the claims. Here are some instructions:
1. Find the Connection: The connection between claims is (target), which is how these key pieces of information are related or how they can be combined to form more complex idea.
Formulate the Question: Create a question that cannot be answered by relying on just one of the sentences but instead requires understanding and linking the information from all of the sources. The answer (target).
3. Ensure Coherence: Make sure the question flows logically from the combined informa- tion and is clear and unambiguous. 4. Use the keywords: (key set)
(examples)
Context:
(Context)
Table 8: Inference Query Generation Prompting
13

---

Published as a conference paper at COLM 2024
(Context)
The above are news articles' metadata and claims come from the articles. All the claims from the articles are related to a similar target. Your task is to generate one comparison question based all the claims from different sources. This question needs to compare some factual elements of the claims that are explicitly stated to find where they agree or differ. The correct answer this question is expressed statement of alignment, simple yes or no. To generate a comparative question from claims, you need use the following keywords: (key set)
comparative adjective,
The Good Comparison Questions:
(examples)
Your Comparison Question:
Table 9: Comparison Query Generation Prompting
(Confext)
Please create time-sensitive comparison question using metadata and excerpts from multiple news articles. That is to compare the consistency or sequence of reports on similar topics multiple different time points. If it is compare the consistency, please clearly mention the news source and time in the question using (time frame). If it is to compare sequences of reports, just clearly mention the news source and do not mention the timeline. Utilize the following keywords provided in the (key set) to construct the question. The correct answer should based on the factual excerpts and is only one word.
(examples)
Your time-sensitive comparison question:
Table 10: Temporal Query Generation Prompting
multi-hop question is query requiring multiple inferential leaps or accessing several pieces of information from different locations or sources to arrive at an answer. Consider- ing you have read at least two news articles on (entity), construct a multi-hop question that incorporates all the news sources. The source of the news should be stated in the question. Also, ensure that the answer to the question is single word/ /entity. Do not answer this question directly. Just give me the question:
Table 11: Null Query Generation Prompting
14

---

Published as a conference paper at COLM 2024
B Appendix B: Dataset Examples
In this appendix, we present an example of each type of multi-hop query included in the MultiHop-RAG dataset. These examples are illustrated in the respective tables: Table 12 for Inference Queries, Table 13 for Comparison Queries, Table 14 for Temporal Queries, and Table 15 for Null Queries. Each query paired with ground-truth answer for the evaluation of generation accuracy, while multiple pieces of supporting evidence are included for assessing retrieval performance. Additionally, metadata such as the title, source, and publication time of the news articles are provided as references.
Query: Which platform is the center discussions articles from Music Business Worldwide, Polygon, and FOX News Health, concerning the policing of AI-driven voice replication, the debate over "reaction" content, and being the most used app overnight by young people?
Answer: YouTube
Evidence List:
Title: Sony Music's artists aren't 't involved in YouTube's new voice-cloning Al experiment.
Source: Music Business Worldwide
Published Time:
Fact: During this period of discussion, YouTube has made a number of positive an- nouncements regarding the biggest issue for any rightsholder regarding Al-driven voice replication of artists: their ability to police it.
Title: YouTube demonetizes popular content creator SSSniperwolf after doxxing accusa- tions
Source: Polygon
Published Time: 1023-10-57718:1110--0000
The debate over "reaction" " content on YouTube has been brewing for years, but recent incident between two creators has refueled the urgency of the conversation.
Fact:
Title: Cell phone shocker as 97% of kids use their device during school hours and beyond, says study Source: FOX News Health
Published Time: 2023-10-01100:0552--0000
Overnight phone use was primarily spent engaging with the same media, although YouTube appeared to be the longest-running app because videos were often left playing during the night.
Fact:
Table 12: The example of inference questions
15

---

Published as a conference paper at COLM 2024
Query: Did the Cnbc - World Business News Leader report on Nike's net income and the article from The Age on the 10-year Treasury yield both report a decrease in their respective financial metrics?
Answer: Yes
Evidence List:
Title: Nike misses revenue expectations for the first time in two years, beats on earnings and gross margin
Source: Cnbc World Business News Leader
Published Time: 0023-09-88120:310000000
Fact: The company's s reported net income for the three-month period that ended August 31 was $1.45 billion, or 94 cents per share, compared with $1.47 billion, or 93 cents per share, year earlier.
Title: ASX set to open higher Wall Street rebounds; $A rises
Source: The Age
Published Time: 2023-10-04121::01000-00000
Fact: The yield on the 10-year Treasury, which is the centrepiece of the bond market, pulled back from its highest level since 2007, down 4.73 per cent from 4.80 per cent late on Tuesday.
Table 13: The example of comparison questions
Query: Was the performance the Chicago Bears' defense reported as improved by Yardbarker after Sporting News highlighted sack by the Bears' defense on Joshua Dobbs during the NFL 'Monday Night Football' game?
Answer: Yes
Evidence List:
Title: Bears vs. Vikings live score, updates, highlights from NFL 'Monday Night Football'
game
Source: Sporting News
Published Time: 2023-11-27723332:44--1
Fact: The Bears answer right back and sack Dobbs, with Sweat and Brisker in there to take him down.
Title: Hottest seat on each NFC team: Buns burning for these four head coaches
Source: Yardbarker
Published Time: 2023111-007222::333000000
Fact: In his second season as HC, the defense has improved, but positive results are hard come by behind lackluster offense ranked 19th in yards (323.2) and 21st in points per game (20.2).
Table 14: The example of time-sensitive questions
Query:
What is the first letter of the CEO's last name the news article from Bloomberg Tom and what is the first letter of the city where the company's headquarters is located in the news article from Reuters?
Answer: Insufficient information.
Table 15: The example of negative rejection questions
16


Reviews:
[{"review_id": "Eihs6jCLYd", "reviewer": "Reviewer_AfhS", "summary": "The paper introduces a novel dataset named MultiHop-RAG, specifically designed to address the inadequacies of existing Retrieval-Augmented Generation (RAG) systems in managing multi-hop queries. The authors establish that current RAG setups fall short in accurately handling these complex queries and highlight the absence of a dedicated benchmarking framework to assess RAG performance on multi-hop inquiries. The developed dataset, MultiHop-RAG, comprises a knowledge base built from English news articles, numerous multi-hop queries, their correct answers, and identifiable supporting evidence.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. In the following sections, we will provide detailed responses to each of your comments, aiming to address your concerns satisfactorily.\n\n**Q1. Lack of comparison to existing datasets**\n\nWe completely agree with you that a comparative analysis demonstrating the unique challenges and benefits of MultiHop-RAG over previous datasets could strengthen the case for its necessity and utility. We have discussed related datasets in the introduction and related work sections of our paper. To clarify and emphasize our dataset's unique position, we list the limitations of related datasets as follows:\n\n* Related Datasets: \n\t1. RAG Evaluation Datasets: RGB (Chen et al., 2023) ,RECALL (Liu et al., 2023) \n\t\t*  These datasets focus on single-hop questions and primarily measure the accuracy of final answers, not retrieval robustness.\n\n\t2. Fact Extraction datasets: FEVER (Thorne et al., 2018), SciFact (Wadden et al., 2020), HoVer (Jiang et al., 2020), HoVer (Jiang et al., 2020)\n\t\t*  These datasets neglecting broader QA performance and RAG framework robustness.\n\t\t\n\t3.  Multi-document QA datasets: HotpotQA (Yang et al., 2018), MultiRC (Khashabi et al., 2018), 2WikiMultiHopQA  (Ho et al., 2020).\n\t\t* Existing datasets predominantly utilize Wikipedia as their main data source, lack emphasis on the robustness of the RAG framework, and do not focus adequately on evidence retrieval capabilities.\n\t\t\n* MultiHop-RAG (this work): This work uses multi-hop questions from multiple documents to test the robustness of the RAG framework, ideal for evaluating its performance in large-scale corpus retrieval and complex question-answering scenarios. Moreover, the dataset includes news articles published from September 26, 2023, to December 26, 2023, beyond the knowledge cutoff of some widely-used LLMs, including ChatGPT and LLaMA, as of the time of writing. This selection also helps in teasing out the possibility of the underlying LLM having been exposed to these news articles, thereby simulating a common scenario of RAG where it is used to retrieve from non-public or proprietary datasets.\n\n**Q2. English news articles as the sole source.**\n\nWe acknowledge that relying exclusively on English news articles as a data source can introduce bias and potentially limit the scope of our queries. To address this limitation, we will discuss this in the limitations section of our paper and consider diversifying our sources in future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The development of MultiHop-RAG addresses a glaring gap in the research on RAG systems which is their inadequacy in handling multi-hop queries. This dataset provides a foundational step towards enhancing RAG performance and extending LLM applicability. By making the dataset and benchmarking code publicly available, the paper offers valuable resources for the wider research community. This openness facilitates replication and further research, potentially accelerating progress in the area.", "reasons_to_reject": "While the paper introduces a new dataset and conducts experiments, it may lack a clear comparison to existing datasets. A comparative analysis demonstrating the unique challenges and benefits of MultiHop-RAG over previous datasets could strengthen the case for its necessity and utility.\nIn addition, the use of an English news article dataset as the sole source for the knowledge base might introduce bias or limit the scope of queries the dataset can effectively support. The paper could benefit from a discussion on these limitations and potential mitigation strategies.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6qEfRLwEob", "reviewer": "Reviewer_w5Gn", "summary": "In this paper, the authors propose a multi-hop queries datasets named MultiHop-RAG for evaluating the LLMs' retrieving and reasoning capabilities. The dataset contains various well-designed types including inference, comparison, temporal and null queries. The detailed data statistics also present the integrity and adequacy of the dataset. From the Table 5 and Table 6, we can observe that the various LLMs can not perform the retrieve and reason well in this difficulty scenario, indicating that the dataset is valuable for the future research.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comment! Below, we respond to each of your comments in detail and hope to address your concerns satisfactorily.\n\n**Q1. The number of queries seems a bit small**\n\nThank you for your observation. Indeed, expanding the dataset could enhance the robustness of our findings. Our current evaluation dataset comprises 2,556 queries, a size determined by two key factors: first, we excluded a significant number of low-quality queries to maintain data integrity; second, we think the current dataset size might be sufficient for preliminary evaluations of retrieval and QA capabilities. For comparison, the widely referenced SciFact dataset (Wadden, D. et al., 2020) contains 1,409 entries. However, we acknowledge the validity of your suggestion and will consider incorporating a larger and more diverse set of queries in future iterations of our work.\n\n**Q2. Further explanation for table 6**\n\nAs you rightly point out, the QA performance highlighted in Table 6 is unsatisfactory and warrants further investigation. Our analysis points to two possible reasons:\n\n1. **Inadequate Retrieval**: The retrieval component in RAG struggles to retrieve relevant evidence effectively based on multi-hop queries from the knowledge base, leading to lower overall response accuracy. This might indicate that existing embedding models fall short in retrieving the most relevant context, suggesting a need for improvements in embedding models. \n\n2. **Limited Reasoning**: Even when provided with accurate ground-truth evidence, the response accuracy of LLMs, such as Mixtral-8x7B, remains unsatisfactory. This model particularly struggles with comparison and temporal queries (as shown in Figure 3), indicating that the comparison and temporal reasoning capabilities of LLMs are still weak, highlighting areas where further advancements are essential.\n\nPer your suggestion, we will include these explanations in the main text. Thanks!\n\nWe appreciate your thorough review and remain available for further discussion if you have any additional questions or require more clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author propose a novelty and effective problem for RAG scenario with LLMs to further investigate.\n\n2. The dataset MultiHop-RAG is constructed comprehensive and rationality, including various query types and categories.\n\n3. The experiments show the dataset is very challenge for various typical LLMs indicating the potential value of the dataset for future research work.", "reasons_to_reject": "1. The number of query seems a little bit small, the author can collect more in the future.\n\n2. For the Table 6, we can observe that the LLMs can not perform well in this dataset. However, the detailed reasons are not analyzed in the experiment results description.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y1NL9f7bPF", "reviewer": "Reviewer_zFdY", "summary": "This paper presents a new dataset to assess multihop RAG, with preliminary performance results by a number of available embeddings and models, with items to test inference, comparison, temporal relations, and a null option to assess hallucination. The dataset is recent (Sept-Dec 2023) so includes data that the tested models were not trained on. The abstract indicates that existing systems perform \"unsatisfactorily\", and the paper clarifies that conclusion by showing that best performance is 56% or less (table 6). \n\nThe paper is well-situated and fairly clear. I think it will be useful. \n\nOne shortcoming is in the quality control (step 5). The paper says \"we manually review a subset sample\", but we are not told how large that sample is, how it was selected, or whether reviewers agreed with each other. A related problem is the complete lack of individual item analysis in the paper. Accuracy per query type is reported, but across those types, were the same items difficult for all systems? And *those* difficult items are the ones that should be manually reviewed, and discussed as examples. A perhaps easier alternative sanity check could be provided with a quick check of human performance on the task. Another shortcoming is the lack of analysis of which features of existing systems best explain the range of performance results. None of these things are listed among the limitations that could be improved with future research.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! Below, we will respond to each of your comments in detail, and we hope can address your concerns satisfactorily.\n\n**Q1. Details of quality control**\n\nYour comment is well taken. Regarding the details of the review process, for each different question type, we randomly sampled 20 samples, which amounted to a total of 60 samples (excluding null queries). We then hired three native English-speaking crowd workers. Provide multiple full documents corresponding to each query and highlight the relevant evidence. We asked the annotators to answer the questions and rate their difficulty from 1 to 5. After keeping the accuracy of generated answers. We also summarized the difficult score distributions in the following table:\n\n| Query Category      | Below 3  |  Above or equal to 3  |\n| ----------- | -----------  | -----------  |\n| Inference Query     | 70%        |     30%      | \n| Comparison Query | 65%        |     35%      | \n| Temporal Query     | 20%         |     80%      | \n\nWe can observe that the human assessment of question difficulty aligns with the results shown in Figure 3 of the paper. Temporal and comparison questions are more challenging compared to inference questions.\n\n **Q2. Lack of analysis of existing poor performance**\n \nWe appreciate the insightful feedback. As you rightly point out, the QA performance highlighted in Table 6 is unsatisfactory and warrants further investigation. Our analysis points to two possible reasons:\n\n1. **Inadequate Retrieval**: The retrieval component in RAG struggles to retrieve relevant evidence effectively based on multi-hop queries from the knowledge base, leading to lower overall response accuracy. This might indicate that existing embedding models fall short in retrieving the most relevant context, suggesting a need for improvements in embedding models. \n\n2. **Limited Reasoning**: Even when provided with accurate ground-truth evidence, the response accuracy of LLMs, such as Mixtral-8x7B, remains unsatisfactory. This model particularly struggles with comparison and temporal queries (as shown in Figure 3), indicating that the comparison and temporal reasoning capabilities of LLMs are still weak, highlighting areas where further advancements are essential.\n\nBy showing the unsatisfactory performance of the Multi-hop RAG and making the dataset publicly available, we hope to spur further research and development in this important field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset and presentation address an important aspect of language models that needs  improvement.", "reasons_to_reject": "I think some additional attention should be given to quality control and item analysis. If the dataset is widely used, these will be of interest.", "questions_to_authors": "Some informal item analysis must have been done, and could be reported in the quality section (step 5).", "extra_scores": {}}, {"review_id": "J3D6lVLD50", "reviewer": "Reviewer_rLY3", "summary": "The paper introduces \"MultiHop-RAG,\" a dataset designed to evaluate retrieval-augmented generation (RAG) systems by focusing on multi-hop queries. The author conducts experiments on the various embedding models, testing their efficacy in evidence retrieval for multi-hop queries and, furthermore, evaluates the ability of off-the-shelf LLMs such as GPT-4 and Llama2-70B,  indicating that direct RAG methods struggle with the multi-hop query demands.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We believe there is some misunderstanding of our paper, and we hope to take this opportunity to clarify some important points.\n\n1. Regarding your comments on “.. overlooks existing datasets such as HotpotQA and 2WikiMultiHop...” we indeed discuss HotpotQA and 2WikiMultiHop in Section 5 “Related Work”. These datasets primarily focus on assessing a model’s reasoning skills, and they do not emphasize the retrieval of evidence from a knowledge base, thereby not tailored for RAG context. Additionally, their primary data sources, Wikipedia, significantly overlap with the training data of most existing LLMs. If we use these sources for benchmarking RAG systems, there is a potential concern that LLM responses might rely on training knowledge rather than reasoning from the retrieved knowledge base.\n\n2. Regarding your comments on “The evaluation methodology appears to inadequately address the core concept of multi-hop reasoning,” we acknowledge that there are more advanced methods to handle multi-hop queries, yet those methods are benchmarked on widely used public QA corpora. For example, [1] is evaluated on open-domain question-answering benchmarks, which is different from real-world RAG scenarios. Moreover, another outstanding concern is that those benchmark datasets are exposed to LLM pretraining. Multihop-RAG, instead, includes news articles published after September 26, 2023, beyond the knowledge cutoff of some widely-used LLMs, including ChatGPT and LLaMA, as of the time of writing. This selection also helps in teasing out the possibility of the underlying LLM having been exposed to these news articles, thereby simulating a common scenario of RAG where it is used to retrieve from non-public or proprietary datasets. Therefore, we believe Multihop-RAG can serve as a useful resource for benchmarking advanced RAG techniques.\n\n3. Regarding your comments on \"construction of the dataset raises several concerns,\" to clarify, we do not solely rely on LLMs for dataset construction. Human inspection and manual review by the authors are conducted to ensure data quality. Moreover, we have hired native English-speaking crowd workers to test the human capability of answering the questions. We will provide more detailed information in the revised manuscript.\n\n[1] Yu, W. et.al. (2023). Chain-of-note: Enhancing robustness in retrieval-augmented language models.\n\nShould you have further questions, we would be happy to answer them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper identifies a gap in current RAG systems, specifically their inefficiency in handling multi-hop queries. By focusing on this underexplored area, the paper employs GPT-4 to develop a benchmark and conducts experiments to evaluate both the retrieval and generation components of the RAG systems. However, there remain key issues that prompt reasons for rejection, see next part.", "reasons_to_reject": "The paper claims to present the first multi-hop RAG (Retrieval-Augmented Generation) dataset. However, this statement overlooks existing datasets such as HotpotQA and 2WikiMultiHop, which also focus on multi-hop reasoning and are cited in the related works section. However, the paper does not adequately differentiate its dataset from these predecessors in the main body of the text. A clarification or rephrasing to accurately position this work within the existing landscape of multi-hop datasets is necessary.\n\nThe evaluation methodology appears to inadequately address the core concept of multi-hop reasoning. The described approach primarily involves direct prompting to retrieve relevant documents and generate answers, which seems to neglect advanced multi-hop retrieval and reasoning techniques referenced in prior studies [1,2,3]. More rigorous testing focused on multi-hop reasoning capabilities is essential to substantiate the dataset's utility and effectiveness.\n\nThe construction of the dataset raises several concerns. The method involves using GPT-4 to generate answers, which is followed by a quality assurance process. However, the details of this process are vaguely described, such as the proportion of data points filtered out during quality checks and the accuracy of GPT-4 in generating correct answers. For a project aiming to establish a benchmark, the reliance on LLMs for generating dataset content without thorough validation may undermine the credibility and utility of the dataset. Typically, LLMs might be more suitably used for augmenting training sets rather than creating benchmarks intended for evaluating model capabilities.\n\n[1] Yu, W., Zhang, H., Pan, X., Ma, K., Wang, H., & Yu, D. (2023). Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210.\n\n[2] Zhang, J., Zhang, H., Zhang, D., Liu, Y., & Huang, S. (2023). Beam retrieval: General end-to-end retrieval for multi-hop question answering. arXiv preprint arXiv:2308.08973.\n\n[3] Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022). Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduce a new dataset, MultiHop-RAG, that uses GPT-4 to construct question-answer pairs based on synthesizing multiple news articles. Retrieving and integrating information from multiple sources is an important and topical challenge for present RAG systems, so the reviewers generally found this to be a timely contribution. However, concerns were raised regarding a lack of details in some places and especially the comparisons to prior work. While MultiHop-RAG is a useful contribution (e.g., it is not Wikipedia-based, so it helps to diversify our current suite of multi-hop QA benchmarks), the authors' claim that it is one of the first RAG datasets to focus on multi-hop queries seems overly strong. I encourage the authors to contextualize MultiHop-RAG appropriately and to incorporate their responses to the reviewers into the next version.

[comments from the PCs] Please refine your claim presentation following the AC recommendation.
Decision: Accept