MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

Authors: yixuan  tang
yi  yang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-16 15:38:41

Abstract: Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge, showing promising potential in mitigating LLM hallucinations and enhancing response quality, thereby facilitating the great adoption of LLMs in practice. However, we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. Furthermore, to our knowledge, no existing RAG benchmarking dataset focuses on multi-hop queries. In this paper, we develop a novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence. We detail the procedure of building the dataset, utilizing an English news article dataset as the underlying RAG knowledge base. We demonstrate the benchmarking utility of MultiHop-RAG in two experiments. The first experiment compares different embedding models for retrieving evidence for multi-hop queries. In the second experiment, we examine the capabilities of various state-of-the-art LLMs, including GPT-4, PaLM, and Llama2-70B, in reasoning and answering multi-hop queries given the evidence. Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries. We hope MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice. We make the dataset and benchmarking code publicly available via GitHub.

Main paper: 
 introduction: 
 The emergence of large language models (LLMs), such as ChatGPT, has fostered a wide range of innovations, powering intelligent chatbots and other natural language processing (NLP) applications (OpenAI, 2023) . One promising use case is Retrieval-Augmented Generation (RAG) (Asai et al., 2023) , which optimizes the output of a large language model by referencing an external knowledge base outside of the LLM training data sources before generating a response. RAG improves LLM's response (Borgeaud et al., 2022) and also mitigates the occurrence of hallucinations, thereby enhancing the models' credibility (Gao et al., 2023) . LLM-based frameworks, such as LlamaIndex (Liu, 2022) and LangChain (Chase, 2022) , specialize in supporting RAG pipelines.In real-world Retrieval-Augmented Generation (RAG) applications, a user's query often necessitates retrieving and reasoning over evidence from multiple documents, a process known as multi-hop query. For instance, consider financial analysis using a database of financial reports. A financial analyst might query, Which company among Google, Apple, and Nvidia reported the largest profit margins in their third-quarter reports for 2023? or inquire about a specific company's performance over time, such as How does Apple's sales trend look over the past three years? These queries require evidence from multiple documents to formulate an answer. Due to the multifaceted nature of such queries, involving information from various sources, traditional similarity matching methods like cosine similarity between query and financial report chunk embeddings might not yield optimal results. We demonstrate this multi-hop retrieval process in Figure 1 .However, existing RAG benchmarks, such as RGB (Chen et al., 2023) and RECALL (Liu et al., 2023) , mainly evaluate a simple case where the answer of a query can be retrieved and solved using one single piece of evidence. None of these benchmarks assess the retrieval and reasoning capability of LLMs for complex multi-hop queries. To address this gap and make RAG benchmarking more closely resemble real-world scenarios, in this paper, we introduce MultiHop-RAG. To our knowledge, MultiHop-RAG is one of the first RAG datasets focusing specifically on multi-hop queries.Based on the RAG queries commonly encountered in real-world scenarios, we first categorize multi-hop queries into four types: Inference query, Comparison query, Temporal query, and Null query. The first three types -Inference, Comparison, and Temporal -require the retrieval and analysis of evidence from multiple sources, encompassing tasks like inferring relationships, comparing data points, and sequencing events over time. The Null query represents a scenario where the query cannot be derived from the knowledge base. This category is crucial for assessing whether an LLM might hallucinate an answer to a multi-hop query when the retrieved text lacks relevance.We construct our RAG knowledge base using a collection of news articles. Using GPT-4 as a data generator, we then take an extensive procedure to construct a diverse set of multi-hop queries, each requiring the retrieval and reasoning over multiple documents. An example of query construction is shown in Table 1 . First, we begin by extracting factual sentences from each news article as evidence. For example, an extracted piece of evidence from an article may state: "Back then, just like today, home prices had boomed for years before Fed officials were ultimately forced to hike interest rates aggressively in an attempt to fight inflation." Second, we input each evidence piece into GPT-4, prompting it to rephrase the evidence into a claim. This claim is clarified with a disambiguated topic and entity. For instance, GPT-4 might rephrase the aforementioned evidence into: "Federal Reserve officials were forced to aggressively hike interest rates to combat inflation after years of booming home prices", identifying "Interest rate hikes to combat inflation" as the topic and "Federal Reserve" as the entity. These topics and entities act as bridges for constructing multi-hop queries, known as bridge-topic or bridge-entity. Next, we use GPT-4 to generate specific multi-hop queries related to the same bridge-topic or bridge-entity, accompanied by the correct answers. Lastly, we undertake a validation step to ensure the data quality.We demonstrate the benchmarking capabilities of MultiHop-RAG using two experiments, utilizing a RAG system implemented with LlamaIndex (Liu, 2022) . The first experiment involves a comparison of different embedding models for retrieving relevant evidence for multi-hop queries. In the second experiment, we assess the reasoning and answering abilities of various state-of-the-art LLMs, including GPT-4, GPT-3.5, PaLM, Claude-2, Llama2-70B, and Mixtral-8x7B, for multi-hop queries when retrieved text is provided. The results from both experiments indicate that the current RAG implementations are inadequate for effectively retrieving and answering multi-hop queries. We publicly release this challenging MultiHop-RAG dataset and hope it will be a valuable resource for the community in developing and benchmarking RAG systems, thereby unleashing the great potential of generative AI in practice. 
retrieval-augmented generation (rag): 
 In an RAG application, we utilize an external corpus, denoted as D, which comprises multiple documents and serves as the knowledge base. Each document within this corpus, represented as d i ∈ D, is segmented into a set of chunks.These chunks are then transformed into vector representations using an embedding model and stored in an embedding database. Given a user query q, the system typically retrieves the top-K chunks that best match the query. These chunks constitute the retrieval set for query q, represented as R q = {r 1 , r 2 , ..., r K }. The retrieved chunks, combined with the query and an optional prompt, are then fed into an LLM to generate a final answer, following the format: LLM(q, R q , prompt) → answer.We define a multi-hop query as one that requires retrieving and reasoning over multiple pieces of supporting evidence to provide an answer. In other words, for a multi-hop query q, the chunks in the retrieval set R q collectively provide an answer to q. For example, the query "Which company among Google, Apple, and Nvidia reported the largest profit margins in their third-quarter reports for 2023?" requires 1) retrieving relevant pieces of evidence related to profit margins from the reports of the three companies; 2) generating an answer by comparing and reasoning from the multiple pieces of retrieved evidence. This differs from a single-hop query such as "What is Google's profit margin in the third-quarter reports for 2023," where the answer can be directly derived from a single piece of evidence.Based on the queries commonly used in real-world RAG systems, we identify four types of multi-hop queries. For each type, we present a hypothetical query within the context of a financial RAG system, where the knowledge base consists of a collection of annual reports.Inference query: For such a query q, the answer is deduced through reasoning from the retrieval set R q . An example of an inference query might be: Which report discusses the supply chain risk of Apple, the 2019 annual report or the 2020 annual report?Comparison query: For such a query q, the answer requires a comparison of evidence within the retrieval set R q . For instance, a comparison query might ask: Did Netflix or Google report higher revenue for the year 2023?"Temporal query: For such a query q, the answer requires an analysis of the temporal information of the retrieved chunks. For example, a temporal query may ask: Did Apple introduce the AirTag tracking device before or after the launch of the 5th generation iPad Pro?Null query: For such as query q, the answer cannot be derived from the retrieved set R q .We include the null query to assess the generation quality, especially regarding the issue of hallucination. For a null query, even though a retrieved set is provided, an LLM should produce a null response instead of hallucinating an answer. For example, assuming ABCD is a non-existent company, a null query might ask: What are the sales of company ABCD as reported in its 2022 and 2023 annual reports?An RAG system handling multi-hop queries can be assessed from two key aspects: retrieval evaluation and generation evaluation.Retrieval Evaluation: Evidently, the quality of the retrieval set R q determines the final generation quality. We compare the retrieved set with the ground truth evidence associated with each query, except for the null queries, as they have no evidence to derive from. Assuming the top-K chunks are retrieved, i.e., |R q | = K, we use retrieval evaluation metrics including Mean Average Precision at K (MAP@K), Mean Reciprocal Rank at K (MRR@K), and Hit Rate at K (Hit@K). MAP@K measures the average top-K retrieval precision across all queries. MRR@K calculates the average of the reciprocal ranks of the first relevant chunk for each query, considering the top-K retrieved set. Hit@K metric measures the fraction of evidence that appears in the top-K retrieved set.Response Evaluation: Since the multi-hop query requires reasoning over multiple pieces of retrieved chunks, we can also evaluate the reasoning capability of the LLM by comparing the LLM response with the ground truth answer of the query. 
a benchmarking dataset: multihop-rag: 
 In this section, we provide detailed information on the construction of the MultiHop-RAG dataset. Specifically, we describe the process of creating a set of multi-hop queries, along with the corresponding ground truth evidence sets and answers derived from a collection of news articles.Step 1: Dataset Collection. We download a news dataset using the mediastack API2 , a REST API interface delivering worldwide news data. The news data source comprises various English-language websites covering a range of news categories: entertainment, business, sports, technology, health, and science. To mimic real-world RAG scenarios, where the knowledge base data, such as an enterprise's internal data, may differ from the LLMs' training data, we select news articles published from September 26, 2023, to December 26, 2023. This timeframe extends beyond the knowledge cutoff of some widely-used LLMs, including ChatGPT and LLaMA, as of the time of writing. This selection also helps in teasing out the possibility of the underlying LLM having been exposed to these news articles. We only keep articles with a token length greater than or equal to 1,024. Every news article is paired with metadata, including the title, publish date, author, category, URL, and news source.Step 2: Evidence Extraction. For each article, we extract factual or opinion sentences using a trained language model3 . These factual sentences are later used as evidence for answering multi-hop queries. We retain only those news articles containing evidence that may have overlapping keywords with other news articles. This allows us to later create multi-hop queries where the answer's evidences are drawn from multiple sources.Step 3: Claim, Bridge-Entity, Bridge-Topic Generation. Our goal is to use GPT-4 to automatically generate high-quality multi-hop queries using the evidence set. However, the raw evidence obtained from Step 2 is not ideal for query generation due to inconsistency in linguistic structure. For example, some pieces of evidence use pronouns to refer to subjects and lack the actual entity in the text. To address this, we employ GPT-4 to paraphrase the evidence, which we refer to as claims, given the original evidence and its context. To ensure consistency between the generated claim and the evidence, we further perform fact-checking using the UniEval (Zhong et al., 2022) framework to verify the alignment between the evidence and claim. Appendix A presents the prompt used for GPT-4 for claim generation.Bridge-Entity and Bridge-Topic: The shared entity or topic across pieces of evidence is referred to as the bridge-entity or bridge-topic. These bridge-entities or bridge-topics can be used to link different pieces of evidence from which a multi-hop query's answer is derived. For example, in a claim such as "Google reports its third-quarter results for 2023, showcasing a detailed overview of its financial performance, including revenue growth, profit margins", the term profit margin can be viewed as a bridge-topic and the term Google can be viewed as a bridge-entity that links the different pieces of evidence. We prompt GPT-4 to identify the bridge-entity and bridge-topic for each claim. Appendix A also presents the prompt used for GPT-4 for bridge generation.Step 4: Query and Answer Generation. In this step, we leverage the bridge-entity or bridge-topic to generate multi-hop queries. Specifically, we first group the claims having the same bridge-entity or bridge-topic into a claim set. We restrict the claim set to have at least two claims but no more than four claims. For each type of query, we feed the claim set to GPT-4 and prompt it with an instruction to generate a query with information from each claim. Below, we explain the specifications for different multi-hop query types. In the construction of each query, we also include the source of the news article where the supporting evidence is associated with mimicking real-world RAG scenarios. Appendix A presents the prompts used for GPT-4 for query generation.Inference Query: These queries are formulated by synthesizing the various characterizations of the bridge-entity across multiple claims, with the final answer being the identification of the entity itself.Comparison Query: These queries are structured to compare the similarities and differences related to the bridge entity or topic. The resultant answer to such queries is typically a definitive "yes" or "no", based on the comparison.Temporal Query: These queries explore the temporal ordering of events across different points in time. The answer to such queries is typically a "yes" or "no" or a single temporal indicator word like "before" or "after".Null Query: Null query is a query whose answer cannot be derived from the retrieved set. To create null queries, we generate multi-hop queries using entities that do not exist in the existing bridge-entities. To add complexity, we also include fictional news source metadata when formulating these questions, ensuring that the questions do not reference any contextually relevant content from the knowledge base. The answer to the null query should be "insufficient information" or similar.Step 5: Quality Assurance. Finally, we use two approaches to reassure the dataset quality. First, we manually review a subset sample of the generated multi-hop queries, their corresponding evidence sets, and the final answers. The results of the manual examination indicate a high degree of accuracy and data quality. Second, we utilize GPT-4 to assess each example in the dataset against the following criteria: 1) The generated query must utilize all provided evidence in formulating the response; 2) The query should be answerable solely based on the provided evidence; 3) The response to the generated query should be either a single word or a specific entity; 4) The query must conform to its designated query type.The MultiHop-RAG dataset contains six different types of news articles, covering 609 distinct news, with an average of 2,046 tokens. The distribution of the news categories is shown in Table 4 . MultiHop-RAG contains four types of multi-hop queries and the distribution of these queries is shown in Table 3 . In total, about 88% of queries in the dataset are non-null queries where answers can be retrieved and reasoned from the knowledge base. In addition, the form of queries exhibits considerable diversity. Approximately 27% of interrogative queries start with "does," around 15% initiate with "what," a similar proportion start "which," and 14% begin with "who," with the remainder incorporating a small percentage of other interrogative words such as "when." Moreover, the number of evidence required to answer a multi-hop query varies. Table 2 shows the distribution of evidence numbers for each query in the dataset. Around 42% of queries can be answered using two pieces of evidence, while approximately 30% and 15% of queries can be answered using three or four pieces of evidence, respectively.  
benchmarking rag system using multihop-rag: 
 MultiHop-RAG can be used as a benchmark for various RAG-related tasks. Broadly speaking, RAG-related tasks can be categorized as retrieval-related tasks and generation-related tasks.A retrieval-related task focuses on retrieving relevant text from the knowledge base, while a generation-related task focuses on generating high-quality responses given the retrieved text. In this section, we showcase two use cases for each task where MultiHop-RAG can be employed.An important design choice in an RAG system is the selection of the embedding model. An embedding model converts data into numerical vectors and subsequently stores these vectors in embedding databases. In this experiment, we evaluate different embedding models by examining their retrieval quality.We implement an RAG system using the LlamaIndex framework (Liu, 2022) . We partition the documents in the MultiHop-RAG knowledge base into chunks, each consisting of 256 tokens. We then convert the chunks using an embedding model and save the embeddings into a vector database. Similarly, in the retrieval step, we convert a query using the same embedding model and retrieve the top-K most relevant chunks that have the highest cosine similarity with the query embedding. In this experiment, we test a variety set of embedding models, including the ada-embeddings by OpenAI (text-embedding-ada-002, text-search-ada-query-001), voyage-024 , llm-embedder (Zhang et al., 2023) , bge-large-en-v1.5 (Xiao et al., 2023) , jina-embeddings-v2-base-en (G ünther et al., 2023), e5-base-v2 (Wang et al., 2022) , and instructor-large (Su et al., 2023) . NULL queries are excluded in this experiment because there is no matching evidence to the query. Additionally, we also include a Reranker module to examine the retrieval performance, using bge-reranker-large (Xiao et al., 2023) . After retrieving 20 related chunks using the embedding model, we further select the top-K chunks using the Reranker.Experiment Result: Table 5 shows the retrieval result of using different embedding models. It shows that there is still a significant gap in retrieving relevant evidence for the multi-hop queries. While Rerank can effectively improve retrieval relevance, the highest Hits@10 is only 0.7467 when the Reranker technique is used. Moreover, the drop in the highest Hits@4 to 0.6625 is worrisome. In practical RAG systems, the underlying LLM often has a context window limit. As a result, the number of retrieved chunks is usually restricted to a small number. The low values of the retrieval metrics highlight the challenges in retrieving relevant pieces of evidence for multi-hop queries when using direct similarity matching between the multi-hop query and text chunks.The underlying LLMs play a crucial role in generating responses in an RAG system. In this experiment, we evaluate the quality of generated responses under two different settings. InWith bge-reranker-large MRR@10 MAP@10 Hits@10 Hits@4 MRR@10 MAP@10 Hits@10 Hits@4 the first setting, we employ the best-performing retrieval model, namely voyage-02 with bge-reranker-large, as indicated in Table 5 , to retrieve the top-K texts and then feed them into the LLM. In the second setting, we use the ground-truth evidence associated with each query as the retrieved text for the LLM. This setting represents a ceiling performance for testing the LLM's response capabilities, as it utilizes the actual evidences.Experiment Setup: In the first experiment, we retrieve top-6 chunks so that the total length of the retrieved text does not exceed 2,048. All queries in MultiHop-RAG are tested in the experiment. In the second experiment, since the null queries do not have associated evidence, we exclude this type of query in the experiment. For the LLMs used in the experiment, we consider state-of-the-art commercial models, including GPT-4 (OpenAI, 2023) , GPT-3.5, Claude-2 (Anthropic, 2023) , and Google-PaLM (Google, 2023) . We obtain answers using the provided API of the respective models. We also assess some opensource models, including Mixtral-8x7b-instruct (Jiang et al., 2024) and Llama-2-70b-chat-hf (Touvron et al., 2023) .Experiment Results: Table 6 shows the response accuracy of different LLMs. First, we can see that the response accuracy rate using the retrieved chunks is not satisfactory, with the state-of-the-art GPT-4 model achieving only 0.56 accuracy. This is expected because the retrieval component falls short in retrieving relevant evidence from the knowledge base. Second, even when we provide the LLM with ground-truth evidence, we can see that the response accuracy is far from being perfect. Open source LLM, such as Llama02-70B and Mixtral-8x7B, only achieve an accuracy of 0.32 and 0.36, respectively. GPT-4 achieves strong reasoning capability with an accuracy of 0.89, followed by the second-based LLM Google-PaLM with an accuracy of 0.74.Figure 3 shows the detailed results of different query types for GPT-4 and Mixtral-8x7Binstruct. Both models show relatively high robustness on null queries, meaning they are generally good at determining when a query cannot be answered based on the retrieved text. This is encouraging because one benefit of RAG is to mitigate the LLM hallucination issue by augmenting LLM with retrieval knowledge. However, Mixtral-8x7B model performs significantly worse than the GPT-4 in comparison and temporal queries. Upon reviewing the incorrect responses, we find that Mixtral-8x7B fails to accurately handle logical negation, leading to misinterpretation of statements and thus a low performance in the comparison queries. In addition, Mixtral-8x7B often fails to correctly identify the chronological order of events, which is crucial for answering temporal queries where timing is a key factor. Taken together, this experiment demonstrates that there is still room for improvement in Beyond embedding models and LLM generation, there are other areas worth exploring. For example, query decomposition is a widely utilized technique in RAG frameworks, such as LLamaIndex. This process involves breaking down the query into sub-questions; it targets a single document for retrieval and integrates the information subsequently, thereby potentially enhancing retrieval accuracy. Another advanced and promising approach involves building LLM-based agents that can automatically plan and execute multi-hop queries, such as AutoGPT (Gravitas, 2023) . AutoGPT breaks down complex instructions into smaller steps. It analyzes feedback to continuously refine its response until the final decision. Another promising direction is the exploration of hybrid retrieval approaches. These methods combine the precision of keyword-based search with the contextual understanding afforded by embedding. By leveraging the strengths of both approaches, hybrid models aim to deliver more accurate and relevant search results. Overall, we believe that there are many potential areas for enhancing RAG's performance on multi-hop queries, and the curated dataset MultiHop-RAG can be a valuable resource to the community. 
related work: 
 RAG Evaluation: As RAG systems gain increasing popularity, a variety of RAG benchmarking datasets and evaluation tools have been developed. For instance, RGB (Chen et al., 2023) and RECALL (Liu et al., 2023) evaluate the performance of LLMs in generating responses for RAG systems under conditions involving noisy, integrative, and counterfactual queries. However, both datasets primarily focus on evaluating the generation aspect of RAG systems without specifically addressing their retrieval accuracy. In addition, recent advancements have been made in automated RAG evaluation tools, such as ARES (Saad-Falcon et al., 2023) and RAGAS (Es et al., 2023) . These tools utilize LLMs to automatically assess the quality of RAG generation, yet they do not introduce benchmarking datasets. Our work introduces one of the first RAG benchmarking datasets, consisting of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence, thereby complementing existing RAG evaluations.Retrieval datasets: Apart from the context of RAG, several benchmarking datasets exist for information retrieval evaluation. The FEVER (Fact Extraction and VERification) dataset, for instance, contains claims classified as Supported, Refuted, or NotEnoughInfo by the given Wikipedia article (Thorne et al., 2018) . Similarly, the SciFact dataset comprises scientific claims paired with evidence-containing abstracts (Wadden et al., 2020) . However, the claims in both datasets are single-hop statements, and the supporting evidence is from one single article, in contrast to the multi-hop queries discussed in this paper. Another dataset, HoVer, involves claims that require extracting and reasoning from multiple Wikipedia articles (Jiang et al., 2020) . However, unlike our dataset, HoVer focuses solely on classifying claims as either supported or not supported by the articles without evaluating an LLM generation step. Moreover, in HoVer, the Wikipedia articles from which evidence is drawn are given for claim verification, which is significantly different from our setting, where relevant pieces of evidence need to be extracted from a large knowledge base. Separately, (Kamalloo et al., 2023) evaluates a range of commercial embedding APIs for information retrieval, but this evaluation is not contextualized within the framework of RAG systems either.Multi-document QA datasets: Question-answering (QA) is a fundamental task in NLP, and several popular benchmarks, such as HotpotQA (Yang et al., 2018) , MultiRC (Khashabi et al., 2018) , and 2WikiMultiHopQA (Ho et al., 2020) , aim to answer the question with multiple sources. This task is similar to our multi-hop query RAG task, as both involve reasoning from multiple sources of information. However, existing datasets are not tailored for the RAG task as they lack a retrieval step. Even modifying a multi-hop query dataset to assess retrieval performance by integrating the input evidence into a corpus does not adequately reflect the demands of real-world RAG tasks. A primary concern is that these datasets, often sourced from Wikipedia, overlap with training data used for large language models (LLMs).In practical RAG scenarios, datasets typically comprise private corpora with metadata like titles or timestamps. This overlap can compromise the evaluation process, raising concerns that LLM responses may depend more on memorized training data rather than on reasoning from the retrieved knowledge base. The proposed dataset, in contrast, is specifically crafted for RAG tasks, including those designed to simulate real-life scenarios. It addresses these issues by introducing a very recent corpus paired with relevant metadata. This dataset can serve as a benchmark to evaluate multi-evidence retrieval and multi-hop evidence generation capabilities. 
conclusion: 
 In this work, we introduce MultiHop-RAG, a novel and unique dataset designed for queries that require retrieval and reasoning from multiple pieces of supporting evidence. These types of multi-hop queries represent user queries commonly encountered in real-world scenarios. MultiHop-RAG consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence. This paper details the creation process of MultiHop-RAG, employing a hybrid approach that integrates human effort with GPT-4. Additionally, we explore two use cases of MultiHop-RAG in the benchmarking of RAG systems, thereby highlighting the potential applications of this dataset. By publicly releasing MultiHop-RAG, we aim to provide a valuable resource to the community, contributing to the advancement and benchmarking of RAG systems. 
 
 Appendix: This work has several limitations that can be improved in future research. First, our ground truth answers are restricted to simple responses such as "yes", "no", entity names, or temporal indicators like "before" or "after" to facilitate the use of a straightforward accuracy metric for evaluating generation performance. Future work could consider allowing free text as answers and employing more sophisticated metrics to assess generation quality. Second, the current dataset limits supporting evidence for a query to a maximum of four pieces. Future work can extend the dataset by including queries that require retrieving and reasoning from even more evidence. Lastly, while our experiments utilize a basic RAG framework using LlamaIndex, future work could involve evaluating the answering of multi-hop queries using more advanced RAG frameworks or LLM-agent frameworks.We present the prompts used for guiding GPT-4 for data generation. Table 7 shows the prompt used for claim generation, along with the corresponding topics and entities within these claims. A multi-hop question is a query requiring multiple inferential leaps or accessing several pieces of information from different locations or sources to arrive at an answer. The following are news articles' metadata and claims come from the articles. All the claims from the article are related to a similar target. Your task is to generate one multi-hop inference question based on the claims. Here are some instructions: 1. Find the Connection: The connection between claims is ⟨target⟩, which is how these key pieces of information are related or how they can be combined to form a more complex idea.2. Formulate the Question: Create a question that cannot be answered by relying on just one of the sentences but instead requires understanding and linking the information from all of the sources. The answer is ⟨target⟩. Table 10 : Temporal Query Generation Prompting A multi-hop question is a query requiring multiple inferential leaps or accessing several pieces of information from different locations or sources to arrive at an answer. Considering you have read at least two news articles on ⟨entity⟩, construct a multi-hop question that incorporates all the news sources. The source of the news should be stated in the question. Also, ensure that the answer to the question is a single word/entity. Do not answer this question directly. Just give me the question: https://github.com/yixuantt/MultiHop-RAGhttps://mediastack.com/https://huggingface.co/lighteternal/fact-or-opinion-xlmr-elhttps://www.voyageai.com/ 


Decision: The authors introduce a new dataset, MultiHop-RAG, that uses GPT-4 to construct question-answer pairs based on synthesizing multiple news articles. Retrieving and integrating information from multiple sources is an important and topical challenge for present RAG systems, so the reviewers generally found this to be a timely contribution. However, concerns were raised regarding a lack of details in some places and especially the comparisons to prior work. While MultiHop-RAG is a useful contribution (e.g., it is not Wikipedia-based, so it helps to diversify our current suite of multi-hop QA benchmarks), the authors' claim that it is one of the first RAG datasets to focus on multi-hop queries seems overly strong. I encourage the authors to contextualize MultiHop-RAG appropriately and to incorporate their responses to the reviewers into the next version.

[comments from the PCs] Please refine your claim presentation following the AC recommendation.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel dataset named MultiHop-RAG, specifically designed to address the inadequacies of existing Retrieval-Augmented Generation (RAG) systems in managing multi-hop queries. The authors establish that current RAG setups fall short in accurately handling these complex queries and highlight the absence of a dedicated benchmarking framework to assess RAG performance on multi-hop inquiries. The developed dataset, MultiHop-RAG, comprises a knowledge base built from English news articles, numerous multi-hop queries, their correct answers, and identifiable supporting evidence.
strength_weakness: strength_weakness: While the paper introduces a new dataset and conducts experiments, it may lack a clear comparison to existing datasets. A comparative analysis demonstrating the unique challenges and benefits of MultiHop-RAG over previous datasets could strengthen the case for its necessity and utility.
In addition, the use of an English news article dataset as the sole source for the knowledge base might introduce bias or limit the scope of queries the dataset can effectively support. The paper could benefit from a discussion on these limitations and potential mitigation strategies.
paper_summary: paper_summary: In this paper, the authors propose a multi-hop queries datasets named MultiHop-RAG for evaluating the LLMs' retrieving and reasoning capabilities. The dataset contains various well-designed types including inference, comparison, temporal and null queries. The detailed data statistics also present the integrity and adequacy of the dataset. From the Table 5 and Table 6, we can observe that the various LLMs can not perform the retrieve and reason well in this difficulty scenario, indicating that the dataset is valuable for the future research.
strength_weakness: strength_weakness: 1. The number of query seems a little bit small, the author can collect more in the future.

2. For the Table 6, we can observe that the LLMs can not perform well in this dataset. However, the detailed reasons are not analyzed in the experiment results description.
paper_summary: paper_summary: This paper presents a new dataset to assess multihop RAG, with preliminary performance results by a number of available embeddings and models, with items to test inference, comparison, temporal relations, and a null option to assess hallucination. The dataset is recent (Sept-Dec 2023) so includes data that the tested models were not trained on. The abstract indicates that existing systems perform "unsatisfactorily", and the paper clarifies that conclusion by showing that best performance is 56% or less (table 6). 

The paper is well-situated and fairly clear. I think it will be useful. 

One shortcoming is in the quality control (step 5). The paper says "we manually review a subset sample", but we are not told how large that sample is, how it was selected, or whether reviewers agreed with each other. A related problem is the complete lack of individual item analysis in the paper. Accuracy per query type is reported, but across those types, were the same items difficult for all systems? And *those* difficult items are the ones that should be manually reviewed, and discussed as examples. A perhaps easier alternative sanity check could be provided with a quick check of human performance on the task. Another shortcoming is the lack of analysis of which features of existing systems best explain the range of performance results. None of these things are listed among the limitations that could be improved with future research.
strength_weakness: strength_weakness: I think some additional attention should be given to quality control and item analysis. If the dataset is widely used, these will be of interest.
questions: questions: Some informal item analysis must have been done, and could be reported in the quality section (step 5).
paper_summary: paper_summary: The paper introduces "MultiHop-RAG," a dataset designed to evaluate retrieval-augmented generation (RAG) systems by focusing on multi-hop queries. The author conducts experiments on the various embedding models, testing their efficacy in evidence retrieval for multi-hop queries and, furthermore, evaluates the ability of off-the-shelf LLMs such as GPT-4 and Llama2-70B,  indicating that direct RAG methods struggle with the multi-hop query demands.
strength_weakness: strength_weakness: The paper claims to present the first multi-hop RAG (Retrieval-Augmented Generation) dataset. However, this statement overlooks existing datasets such as HotpotQA and 2WikiMultiHop, which also focus on multi-hop reasoning and are cited in the related works section. However, the paper does not adequately differentiate its dataset from these predecessors in the main body of the text. A clarification or rephrasing to accurately position this work within the existing landscape of multi-hop datasets is necessary.

The evaluation methodology appears to inadequately address the core concept of multi-hop reasoning. The described approach primarily involves direct prompting to retrieve relevant documents and generate answers, which seems to neglect advanced multi-hop retrieval and reasoning techniques referenced in prior studies [1,2,3]. More rigorous testing focused on multi-hop reasoning capabilities is essential to substantiate the dataset's utility and effectiveness.

The construction of the dataset raises several concerns. The method involves using GPT-4 to generate answers, which is followed by a quality assurance process. However, the details of this process are vaguely described, such as the proportion of data points filtered out during quality checks and the accuracy of GPT-4 in generating correct answers. For a project aiming to establish a benchmark, the reliance on LLMs for generating dataset content without thorough validation may undermine the credibility and utility of the dataset. Typically, LLMs might be more suitably used for augmenting training sets rather than creating benchmarks intended for evaluating model capabilities.

[1] Yu, W., Zhang, H., Pan, X., Ma, K., Wang, H., & Yu, D. (2023). Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210.

[2] Zhang, J., Zhang, H., Zhang, D., Liu, Y., & Huang, S. (2023). Beam retrieval: General end-to-end retrieval for multi-hop question answering. arXiv preprint arXiv:2308.08973.

[3] Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022). Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509.

Comments: title: Any further questions?
Further clarification on the differences between MultiHop-RAG and existing multi-hop datasets
comment: Dear Reviewer rLY3,

We appreciate your constructive feedback. As noted in our engagement, we believe there is a misunderstanding regarding the contribution of our work. We want to reiterate that our developed dataset is specifically for RAG tasks, which differs from previous multi-hop QA datasets that primarily focus on the multi-hop reasoning aspect while overlooking the multi-hop retrieval aspects. In addition to this major difference, there are also other outstanding differences, which we do not repeat here in the interest of your time.

If we clear up this misunderstanding, we believe this paper constitutes a valuable resource for benchmarking real-world RAG systems, as most of the human queries in RAG systems involve multi-hop, multi-evidence retrieval, and reasoning, as rightly recognized by other reviewers. We hope we have clarified this point and would appreciate it if you could reconsider your rating. Should you have any further questions, we are very happy to address them.

Best,
Authors
Thanks for your insights!  Please allow us to further clarify about the quality control tests. 

First, the table in Q1 shows the difficulty distribution for each question type based on majority voting from crowdsourced annotations. The difficulty ranges from 1 (easiest) to 5 (most difficult). Scores below 3 are considered easy, while 3 and above are considered difficult. Temporal queries, which involve reasoning about time, are the most challenging for humans, followed by comparison queries that require making comparisons. Null queries, which have no answerable questions, are not evaluated and therefore excluded from this data. The results align with the model answer accuracy shown in Figure 3. As expected, the model exhibits the lowest accuracy on comparison and temporal queries, which are identified as the most difficult for humans. In contrast, the model achieves the highest accuracy on inference questions, which appear to be relatively easier based on the difficulty distribution.

Second, motivated by your comments, we reviewed the per-item correlations between human evaluators and model-generated responses. In our understanding, the per-item correlation you referred to may be the model's accuracy in answering each question compared to the human-annotated difficulty levels. If we have misunderstood, please correct us. Table 1 shows the accuracy of different models in answering the questions we used in human annotations. GPT-4 remains the best, followed by Google-PaLM. The results in Table 2 specifically analyze the questions the models answered incorrectly. For each model, the table shows the average difficulty scores of the questions it struggled with, as determined by human annotators. The 'nan' values indicate no incorrect questions. Thus, we can find that GPT-4 and PaLM can handle more challenging questions (failed on a high difficulty score), while open-source models struggle with simpler questions, as indicated by the average difficulty scores of their incorrectly answered questions in Table 2. This observation is consistent with the findings in Table 6 of the paper. Combining all the tables, we can clearly see how the difficulty of different types of questions varies for both humans and models. The stronger the model, the higher its accuracy in answering questions, and it can handle more challenging problems.

**Table 1: Generation accuracy of LLMs for the sampled questions.**

| Model  	| Comparison Query | Inference Query | Temporal Query | Avg. |
| ------------ | ----------------- | --------------- | -------------- |-------------- |
| GPT-4  				| 40.00%              	| 100%                 | 50.00%           	 | 63.00% | 
| Google-PaLM   	| 30.00%               	| 100%                 | 50.00%               | 60.00% | 
| ChatGPT	 		| 0.00%              	    | 100%                 | 40.00%               | 46.67% |
| Claude-2.1 		| 10.00%           	    | 80.00%            	| 10.00%               | 33.34% |
| Llama-2-70b-chat-hf  	| 0.00%             	 	| 90.00%              | 0.00%             | 30.00%|
| Mixtral-8x7B-Instruct  	| 0.00%               	    | 80.00%              |0.00%              | 26.67%|

**Table 2: Average human annotated difficult score of the inaccurate questions from different models.**

| Model  	| Comparison Query | Inference Query | Temporal Query |
| ------------ | ----------------- | --------------- | -------------- |
| GPT-4  				| 2.0              	| nan             | 4.6           	 |
| Google-PaLM   	| 2.43             	| nan             | 4.6             |
| ChatGPT	 		| 2.4              	| nan             | 4.16           |
| Claude-2.1 		| 2.55            	| 2.0             	 | 4.55           |
| Llama-2-70b-chat-hf  	| 2.4             	 	| 2.0              | 4.4             |
| Mixtral-8x7B-Instruct  	| 2.4              	| 2.0              | 4.4             |

The findings from our analysis clearly demonstrate that our dataset poses a substantial challenge due to its high difficulty and quality, making it an ideal testbed for RAG systems. 

We are available for further discussion if you have any more questions or need additional clarification.
Thanks, these statements clarify the quality control tests you did, though the statement of results here is still too brief for me to understand fully. Were per-item correlations with GPT-4 or any other calculated?
Please allow us to further clarify a key difference between MultiHop-RAG and existing datasets such as HotpotQA and 2WikiMultiHop. We acknowledge the existence of multihop QA datasets such as HotpotQA and 2WikiMultiHop, among others. However, these existing datasets are not designed for the RAG task and do not include a retrieval step. The input used in these datasets is already provided with multiple pieces of evidence, and they primarily examine QA models' performance in multi-hop reasoning. Even if one extends HotpotQA to evaluate retrieval performance by combining the HotpotQA input data (Wikipedia text) as a corpus, this extension does not align with real-world RAG tasks. In real-world RAG scenarios, the dataset is usually internal or proprietary corpus, not a corpus used in LLM training such as Wikipedia. This is why we mentioned the Wikipedia data source issue in our previous response.

Our developed dataset, on the other hand, is specifically designed for the RAG task. It can be used to benchmark multi-evidence retrieval capability (given a question that requires multiple pieces of evidence, how to retrieve relevant pieces of evidence from a database) as well as multi-hop evidence generation capability. Our experiments find that an off-the-shelf RAG solution cannot effectively handle multi-evidence retrieval, which subsequently affects downstream generation performance. We believe the focus on the retrieval aspect sets us apart from existing datasets.
Thank you very much for your reply! We appreciate your academic professionalism and willingness to engage in this discussion. Below, we provide our response to each of your comments in this round of discussion.

**Q1. The novelty of this work compared to existing multi-hop reasoning benchmarks**

In addition to the retrieval data source, we have designed certain questions to assess the reliability of the target RAG systems, particularly in scenarios involving null queries. This type of question lacks a relevant context in the corpus, necessitating the LLM to identify and reject responses rather than make up a wrong answer. This is another important difference. Since LLM struggles to say "I don’t know" [1] and hallucinations exist in the RAG framework [2], it is important to measure the robustness of RAGs. Thus, our dataset proposes a comprehensive testbed for the RAG framework. 

**Q2. No evaluation of the advanced framework**

We appreciate your insight into the RAG framework and its role in our research. While focusing on dataset construction, we utilized a basic RAG framework to demonstrate the dataset’s potential. We are excited about the more complex frameworks, which could provide further insights, and have acknowledged this as a promising area for future research in the limitation section. And we're still exploring new methods. For your reference, we have tested the performance of advanced RAGs based on generating follow-up questions (query decompose) and using these questions to retrieve relevant documents. We also tested the Hybird search [3]. The baseline in the following table is the basic RAG framework, and the retriever is the best model tested in our work (bge-large-en-v1.5). Here are our results: 

|    mode   | chunk size | MRR@10 | MAP@10 | Hits@4 |
|:---------:|------------|:------:|:------:|:------:|
| baseline  | 512        | 0.4975 | 0.2447 | 0.6718 |
| hybrid    | 512        | 0.7017 | 0.3682 | 0.8421 |
| decompose | 512        |  0.508 | 0.2509 | 0.6794 |
| baseline  | 256        | 0.4322 | 0.2017 | 0.6049 |
| hybrid    | 256        | 0.6779 | 0.3469 | 0.8018 |
| decompose | 256        | 0.4576 | 0.2218 | 0.6222 |
| baseline  | 128        | 0.3679 | 0.1728 | 0.4958 |
| hybrid    | 128        | 0.5988 | 0.3016 | 0.7135 |
| decompose | 128        | 0.3884 | 0.1923 | 0.5188 |

It is evident that the advanced framework's performance on our datasets is suboptimal. As a result, we believe Multihop-RAG can be a useful resource for benchmarking advanced RAG technology, and we are available for further discussion if you have any additional questions or need more clarification.

[1] Yin, Zhangyue, et al. Do Large Language Models Know What They Don't Know?

[2] Niu, Cheng, et al. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models.

[3] Gordon V. Cormack, et Reciprocal rank fusion outperforms condorcet and individual rank learning methods.
I remain unconvinced by the authors' rebuttal. The novelty of this work is still unclear. The response does not adequately address the fundamental difference between this work and existing multi-hop reasoning benchmarks, aside from the retrieval data source. My initial question was not intended to suggest that Wikipedia should be the data source. Rather, I am concerned with the conceptual differences, particularly given that previous literature has also evaluated retrieval performance and generation accuracy of large language models. If the primary contribution of this work is the introduction of a different data source, then the claim of presenting "the first multi-hop RAG (Retrieval-Augmented Generation) dataset" appears tenuous.

Regarding my second concern, the authors did not directly address why they did not evaluate using existing multi-hop reasoning baselines. The baselines I mentioned, along with several other widely used multi-hop reasoning methods(such as chain-of-note[1], react[2]), generate follow-up questions and use these questions to retrieve relevant documents before generating a final answer. This approach would seem to be more appropriate as a baseline for evaluating a multi-hop RAG dataset. The choice of data source should not preclude testing against these established methods.

In light of these points, I plan to maintain my score.

[1] Yu, W. et.al. (2023). Chain-of-note: Enhancing robustness in retrieval-augmented language models.

[2] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.