From Complex to Atomic: Enhancing Augmented Generation via Knowledge-Aware Dual Rewriting and Reasoning
Authors: Jinyu Wang
Jingjing Fu
Rui Wang
Lei Song
Jiang Bian
Venue: ICML 2025 poster (2025)
Keywords: RAG
Knowledge atomizing
Knowledge-aware task decomposition
Multihop QA
Primary area: applications->language_speech_and_dialog
TL;DR: We present an advanced RAG system with knowledge-aware dual rewriting and reasoning capabilities, designed to improve knowledge extraction and rationale formulation within specialized datasets.

Abstract: Recent advancements in Retrieval-Augmented Generation (RAG) systems have significantly enhanced the capabilities of large language models (LLMs) by incorporating external knowledge retrieval. However, the sole reliance on retrieval is often inadequate for mining deep, domain-specific knowledge and for performing logical reasoning from specialized datasets. To tackle these challenges, we present an approach, which is designed to extract, comprehend, and utilize domain knowledge while constructing a coherent rationale. At the heart of our approach lie four pivotal components: a knowledge atomizer that extracts atomic questions from raw data, a query proposer that generates subsequent questions to facilitate the original inquiry, an atomic retriever that locates knowledge based on atomic knowledge alignments, and an atomic selector that determines which follow-up questions to pose guided by the retrieved information. Through this approach, we implement a knowledge-aware task decomposition strategy that adeptly extracts multifaceted knowledge from segmented data and iteratively builds the rationale in alignment with the initial query and the acquired knowledge. We conduct comprehensive experiments to demonstrate the efficacy of our approach across various benchmarks, particularly those requiring multihop reasoning steps. The results indicate a significant enhancement in performance, up to 12.6\% over the second-best method, underscoring the potential of the approach in complex, knowledge-intensive applications.

From Complex to Atomic: Enhancing Augmented Generation via Knowledge-Aware Dual Rewriting and Reasoning
Jinyu Wang" Jingjing Fu"
Rui Wang'
Lei Song'
Jiang Bian
Abstract
Chain shaped Path
Tree
Dynamic Path Generation
Recent advancements in Retrieval-Augmenter Generation (RAG) systems have significantly en- hanced the capabilities of large language models (LLMs) by incorporating external knowledge re- trieval. However, the sole reliance on retrieval is often inadequate for mining deep, specialized knowledge and performing the logical reasoning necessary to tackle domain-specinn complex ques- tions. To address these challenges, we present an approach, which is designed to extract, com- prehend, and utilize specialized knowledge an atomic manner while simultaneously construct- ing coherent rationale. At the heart of our approach lie four pivotal components: knowl- edge atomizer that extracts atomic tags from raw data, query proposer that generates subsequent questions to facilitate the original inquiry, an atomic retriever that locates knowledge based on atomic knowledge alignments, and an atomic selector that determines which atomic tag and chunk pair query, guided by the retrieved in- imple- formation. Through this approach, ment knowledge-aware task decomposition strat- egy that iteratively builds the rationale in align- ment with the initial question and the acquired knowledge. We conduct comprehensive experi- ments demonstrate the efficacy of our approach across various benchmarks, particularly those re- quiring multihop reasoning steps. A substantial performance improvement of up +101 (20.4%) over the second-best method underscores the po- tential of the approach in complex, knowledge- intensive applications. The code is publicly avail- able http:://gttttu..ccmmmm/oooottftt
Stee-
Step22
Pare
Quedion
Clark
quania
Rotiicall
Klzmno Cuery Proposel from Questor))
Thy
Figure Complex questions are typically decomposed into sub- questions following either chain-shaped or tree-shaped path, with chunk retrieval used to gather relevant information for resolution. Unlike previous approaches, question decomposition with information retrieval through atomic knowledge alignment by dual rewriting upon questions and chunks, and dynamically determine follow-up sub-questions by atomic pair retrieval and selection, enabling an adaptive and interactive decomposition path that evalves hased on the retrieved knowledge.
method seamlessly integrate the
1. Introduction
Large Language Models (LLMs) have revolutionized the field of natural language processing by demonstrating the capability to generate coherent and contextually relevant text and the versatility execute diverse spectrum of lin- guistic tasks, ranging from text completion to translation and summarization (Achiam et al., 2023; Touvron et al., 2023). Despite their broad capabilities, LLMs exhibit pro- nounced limitations when tasked with specialized queries in professional domains (Ling et al., 2024; Wang et al., 2023a). This primarily arises from the scarcity of domain -specff training material (e.g., unpublished documents) and an in- complete understanding of specialized knowledge and ratio- nale within these domains (e.g., industry-specifee acronyns, company-specift operational rules). As a result, LLMs may produce responses that are not only potentially erro- neous but also lack the detail and precision required for expert-level engagement (Bender al., 2021).
'qqual contribution 'iiroooott Research Asia, Beijing, China. Correspondence to: Lei Song <lesong microsoft.com com.>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

From Complex to Atomic: Enhancing Augmented Generation
Question
Atomic Query Proposal
mother director Atomised
the are the parents of the director Oskar Rochter? nume biegraphy Reehlar's mether? fanily
(Fitm)?
Target Chunk
title: oshar Rochler
Atomic Retrieval
contents: He was born in Starnberg as the son of nriter Gisela Elsner and the ariter Klaus Rochler.
atomic query Who are the parents of the director Oskar Rothley?
atonit teg
chunk relezent?
Who are the paront Osker Rochler?
Fp Question
Where was Caker tochker born?
Who is the mother of Oskar Roehlar?
Whhn oun Caker nechler born?
Hisrarchical
ihfo wore paranta hare the stall Etrggenant
source chunk
chunk relevam??
She was the mother of Prince Maritago.
atomic query
Oakar
She wos the mother of Queen Morgaret Lejonhutrus and the second cousin and mother lawe
mother Prince Cruss Netherands, Nesherlands, Censort feotticef
abomic tag
clunk relevant?
ihoo was Werner Art langet huben' nuther?
Whlt was the name of Wandera mothes
to butner Mencer Partimmen Elasson
ihoo was Helling of Minsto leng Out's mutherr?
whee the name of Rucoff It's nother?
Self-Ask
KAR
Figure 2. Case demonstration of Self-Ask and KAR'. By proposing multiple atomic queries, KAR effectively retrieves the relevant knowledge chunk, whereas the single deterministic follow-up question approach employed by Self-Ask fails to align with the knowledge base's 's schema, resulting in a retrieval failure.
To mitigate these issues, RAG (Lewis et al., 2020) has emerged as promising solution, augmenting LLMs with external knowledge retrieval to anchor generated content. By supplementing or even replacing the knowledge encoded within LLMs, RAG frameworks aim to improve factual- ity and relevance. However, existing RAG systems often struggle with domain- domain-specifie complex tasks. For example, answering query like "please provide the product name of the latest biosimilar for HUMIRA that has been suc- cessfully approved." requires more than retrieving scattered specialized knowledge (c.g., biosimilar for HUMIRA) from multiple sources. It demands logical reasoning on eligible products and their approval timelines to synthesize a precise and reliable response. Current RAG methods predominantly rely plain text retrieval, which may fail to effectively capture correlations within domain domain-specific expression, and often employ question decomposition without considering available knowledge, leading to suboptimal sub-question generation, ineffective retrieval and reasoning failures.
cialized domains requires knowledge extraction and com- prehension to deeply understand both the user's information needs and the underlying context of the retrieved data. For example, specialized questions in fields like medicine, law, or finance often involve domain-specific terminology and logic, which generic LLMs may fail to grasp fully. Tra- ditional RAG systems that retrieve text passages based on keyword matching (Ram al., 2023; Jiang al., 2023) embedding similarity (Gao et al., 2023) may retrieve contextually relevant information, that may lack semantic precision, insufficient for answering intricate questions.
Iterative Reasoning for Complex Query Resolution Com- plex reasoning tasks, where the answer depends on synthe- sizing information from multiple sources, demand the de- composition of the original question into series of simpler, interrelated sub-questions (Press et al., 2023). Nonethe- less, this approach may face obstacles in domains where the knowledge is not readily accessible to LLMs. We argue that the decomposition such domains should be contextual, rather than a standalone operation, meaning that decom- posed queries can be answered with the retrieved knowledge and context progressively and evolve into refining subse- quent queries. This iterative approach allows the system to evolve its understanding of the user's 's inquiry, ensuring that follow-up questions are informed by the most recent retrieval results.
In this work, we argue that advancing RAG requires knowledge-aware processing, especially on question decom- position and information retrieval, and iterative reasoning to effectively address complex, multi-step questions in pro- fessional domains.
Knowledge-aware Processing for Domain-Specift Com- prehension Addressing complex, logic-driven tasks in spe-
2

---

From Complex to Atomic: Enhancing Augmented Generation
We introduce novel framework, KAR*-RAG, which em- ploys knowledge-aware dual rewriting and reasoning mechanism. Our approach features dynamic interaction between question rewriting and knowledge retrieval, en- abling the system to adaptively refine both the query and the retrieved context at each iteration, as illustrated in Fig- ure 1. The core components of our system include Knowl- edge atomizer, decomposing raw data into atomic lags for more granular retrieval, Query proposer, generating follow- up questions based on the evolving context, Atomic re- triever, identifying and retrieving relevant knowledge based on atomic knowledge alignments, and Atomic selector, de- termining the most relevant follow-up questions based on the retrieved information. More specially, the atomic tags are formulated as relevant inquiries that can be answered by the given chunk, thereby encompassing the multifaceted knowledge of the chunk and facilitating effective retrieval. Atomic query proposals are raised to inquiry the knowledge that is helpful to answer the question better. By leverag- ing these components, our system can iteratively refine its understanding of both the question and the retrieved knowl- edge, enabling more accurate and context-aware reasoning over multiple hops. We demonstrate the advantages of task decomposition and atomic retrieval on real-world case, shown in Figure 2. Our approach not only enables multi- faceted task decomposition, but also alleviates the misalign- ment between the corpus formulation and the query through atomic tagging of the corpus.
2023; Zhong et al., 2024), mixed retrieval (Yang, 2023) and re-ranking (Cohere, 2023). On one hand, efforts focus on query rewriting, either explicitly (Zheng et al., 2024) or implicitly (Gao et al., 2022), to enhance retrieval perfor- mance. On the other hand, several studies transform raw data sources into structured data, ultimately converting them into valuable knowledge for more effective retrieval and rea- soning (Wang et al., 2023b; Zheng et al., 2024; Raina & Gales, 2024; Liang al., 2024). In our system, we intro- duce atomic rewriting for both queries and chunks, which achieves multi-granularity question decomposition and com- prchensively extract inherent knowledge from chunks.
To tackle complex tasks such as summarization (Hayashi et al., 2021) and multihop reasoning (Ho et al., 2020), re- cent research focuses on developing advanced coordination schemes that leverage existing RAG modules to collabora- tively address these challenges. Iter-RetGen (Shao et al., 2023) and DSP (Khattab et al., 2023) employ retrieve-read iteration to leverage generation response as the context for next round retrieval. FLARE (Jiang et al., 2023) proposes a confidence-bass active retrieval mechanism. Our approach adopts an iteration-based pipeline that leverages context- aware reasoning process, enabling the adaptive formulation of follow-up questions for each iteration and reducing the difficulty of retrieval and reasoning of complex tasks.
2.2. Multihop QA
Multihop Question Answering (MHQA) (Yang et al., 2018) require reasoning over multiple pieces of information, often scattered across different sources. This task presents unique challenges as it necessitates not only retrieving relevant information but also effectively combining and reasoning over the retrieved pieces to arrive at a correct answer. The traditional graph-based methods in MHQA solves the prob- lem by building graphs and inferring on graph neural net- works(GNN) to predict answers (Qiu & other authors, 2019; Fang & other authors, 2020). With the advent of LLMs, recent graph-based methods (I.i & Du, 2023; Panda et al., 2024; Liang et al., 2024) have evolved to construct KGs for retrieval and generate response through LLMs. However, constructing a high-quality domain-specific KG is costly, and the structured triple format imposes inherent constraints on contextual representation. limiting its expressiveness. Self-RAG (Zhang et al., 2024a)and beam-retriewe (Asai et al., 2023) treating MHQA as a supervised problem, ne- cessitating labeled data and additional training.
Our key contributions are as follows: 1) We propose a knowledge-aware RAG framework that incorporates re- trieved knowledge into question decomposition, enabling iterative exploration of the reasoning path. 2) We introduce an atomic knowledge aligmment approach by dual writing that tightly couples query decomposition with retrieval, sig- nificantly improving retrieval efficiency. 3) We report on comprehensive experimental and ablation studies that vali- date the superior performance of our approach across mul- tiple benchmark datasets, achieving up to 20.4% increase over the second-best method.
2. Related work
2.1. RAG
RAG has emerged as a promising solution that effectively incorporates external knowledge to improve the generation of LLMs. Naive RAG systems retrieve pertinent informa- tion from external data sources and integrate it into the context of the question prompt as supplementing knowl- edge for contextually relevant generation (Ram et al., 2023). Advanced RAG approaches implement specific enhance- ments across the pre-retrieval, retrieval, and post-retrieval processes, including query optimization (Ma et al., 2023; Zheng al., 2023), multi-granularity chunking (Chen al.,
Another branch of methods decomposes multihop ques- tions into sub-questions following either chain-shaped path (Trivedi et al., 2023; Khattab et al., 2023; Feng et al., 2023; Xu et al., 2024) or tree-shaped path (Zhang et al., 2024b; Jiapeng et al., 2024; Cao et al., 2023), as depicted in Figure 1. The sub-questions guide sequential chunk re-
3

---

From Complex to Atomic: Enhancing Augmented Generation
Chunk
Chunk
Atomizer
Retrieved
Atumic Retriever
Atamic Selector
Quiry
Query Proposer
KDPIE JUSN pepuel
Answer
Generator
Conteet
Figure 3. Overview of the KAR*-RAG workflow, illustrating knowledge atomizing by the atomizer, and knowledge-aware task decompo- sition using the query proposer, atomic retrieval and atomic selector. The query proposer generates atomic query proposals based on the original question and reference context. These proposals are used to retrieve the relevant atomic tags, producing retrieved atomic pairs. The atomic selector chooses the most relevant pair and the corresponding chunk, which is added to the reference context task decomposition in the subsequent iteration. Once the atomic selector determines that further information is required and atomic pair selected, the original question reference context are passed to generator to produce final answer.
trieval, with the retrieved results subsequently facilitating the reasoning process. In chain-shaped decomposition, a single sub-question is generated, and its answer availability not guaranteed. potentially leading to answer failure. In contrast, tree-shaped decomposition requires exploring mul- tiple reasoning paths, necessitating sophisticated evidence verification and fusion for final response generation. Our approach explore the reasoning path by interactively select the sub-question from a set of query proposals based on the relevance of atomic retrieval. This allows for flexible decomposition by leveraging updated context and selecting query proposals with available knowledge.
rewriting is employed to bridge the semantic gap between the question and the chunks to be retrieved. The rewritten query is represented as q = fre( The workflow of the advanced RAG system is further improved as follows,
à = LLM(q, D9), where D = R(g. D)
(2)
This enhancement allows the system to better align queries with relevant document chunks, enhancing retrieval accu- racy and answer generation. However, addressing complex multihop questions remains challenging. These questions often require reasoning across multiple chunks and inte- grating information through several retrieval and generation steps-a process that single pass may not fully capture.
3. Methodology
3.2. Framework
3.1. Preliminary
To address complex multihop questions, we introduce an enhanced RAG system with Knowledge-Aware dual Rewriting and Reasoning, termed as KAR This system employs an iterative strieval-reasoming gengrgggeeeenonnn mech- anism that facilitates gradual collection of relevant infor- mation and progressive reasoning over incremental context. An overview of the proposed workflow is depicted in Fig- ure 3. In our framework, raw data chunks are broken down into atomic tags using a knowledge atomizer to construct an atomic knowledge base for the subsequent retrieval. Ques- tions are similarly atomized by query proposer to generate atomic query proposals, which are utilized retrieve the relevant atomic tags from the knowledge base. Both chunks and questions are rewritten to bridge the semantic gap and improve the alignment of knowledge. An atomic retriever then selects the top-k atomic pairs for each atomic query proposal. Based on these retrieved atomic pairs, an atomic selector, as a reasoner, identifies the most useful atomic pair
In a RAG system, the textual corpus is divided into a collec- tion of document chunks, denoted as D = {d1, d2,............. dn where d; represents the i-th document chunk. The original question is denoted as q. and its corresponding ground truth answer is represented by a. The retrieval phase involves evaluating the similarity between the question q and each document chunk d,, after which the top-k most relevant chunks are selected as retrieval results, forming the basis for subsequent generation.
R: topk Sim(g, - D)a d
Here, the retriever R selects the top-k most relevant chunks 1e based on the similarity function (-) Finally, the original question and retrieved chunks are fed into the large language model to generate the answer, denoted as LLM(g, D9). In the advanced RAG systems, query

---

From Complex to Atomic: Enhancing Augmented Generation
for problem-solving and adds the corresponding raw chunk to the context. This context is then aggregated with the original question for the task decomposition in next itera- tion. The iteration process may terminate earlier if it fails to retrieve suitable atomic tags, either due to the genera- 3: Generate atomic query proposals q' + fplg. Ct-1)
Algorithm Task Solving with Knowledge-Aware Decom- position
1: Initialize context Co Φ
2: for t 1,2,.... do
tion of low-quality query proposals or the lack of relevant atomic tag candidates. At this point, the original question and context are passed the generator produce the final answer.
4: Retrieve top-k atomic pairs for each atomic query proposal from knowledge base
p4'
K.B.
Ratom Sa())
3.3. Knowledge Atomizing
5: Select the most useful atomic tag or None when additional information is unnecessary
Chunked text often contains multifaceted information, and typically, only subset is needed to address specific task. Traditional information retrieval methods, which consoli- date all information within single chunk may not facilitate the efficient retrieval of the precise information required.
Ikgt. + CL-,
6:
if gk,t, is None then
7:
C, c C,-1
Recent research have explored the extraction of triple knowl- edge units from chunked text and constructing knowledge graphs to facilitate efficient information retrieval (Edge et al., 2024; Panda et al., 2024). However, the construction of these knowledge graphs is costly, and the inherent knowledge may not always be fully explored. To better present the knowl- edge embedded in documents, we propose atomizing the original documents for knowledge extraction, a process we refer as Knowledge Atomizing. This approach leverage the context understanding and content generation capabilities of LLMs automatically tag atomic knowledge pieces within each document chunk.
8: break
9: else
10: Fetch the relevant chunk cf corresponding to th,t
11: Update context C. C--1 U
12: end if
13: end for
14: Generate answer à +- LLM((,
demands the ability break down the original question into several sequential or parallel atomic tags for retrieval. We re- fer to this operation as Task Decomposition. By combining the extracted atomic knowledge with the original chunks, we construct an atomic knowledge base. Each time a task is decomposed, the atomic knowledge base provides insights into the available knowledge, enabling knowledge-awan task decomposition. We design the Knowledge-Aware Task Decomposition workflow, and the complete algorithm for solving task is detailed in Algorithm 1, and an example is illustrated in Figure 4(b).
The presentation of the atomic knowledge can be vari- ous. Instead of utilizing declarative sentences or subject- relationship-objet tuples, we propose using questions as knowledge indexes to further bridge the gap between stored knowledge and queries. In knowledge atomizing process, input the document chunk to LLM as context, ask it to generate relevant questions that can be answered by the given chunk as many as possible. These generated atomic tags are stored together with the given chunks. The knowl- edge atomizer applies atomizing operation on each chunk.
Initially, the reference context C. is initialized as an empty set. the first iteration, task decomposition relies solely on the original question generate atomic query proposals. As iterations progress, the accumulated context t-th iteration denoted as C,-1, consists of chunks retrieved from previous iterations. During the t-th iteration, the query proposer gen- erates atomic query proposals based on the original question and the accumulated context.
fa = {gk1. 9k2,
(3)
The atomic tags are generated by atomizer for every chunk, forming an atomic knowledge base, denoted as KB (fa(d)), ). An example of knowledge atomizing is il- lustrated in Figure 4(a), where the atomic tags encapsulate various aspects of the knowledge contained within the chunk. Since each chunk is tagged with multiple atomic tags, an atomic query can be used to locate relevant atomic tags, which then leads to the associated reference chunks.
Cr-1) = -00 16 ..
(4)
The query proposer fp(.) can be implemented as either an LLM or a learnable component. we leverage an LLM to gen- erate query proposals that are potentially beneficial for task completion, represented as q'=( During this process, the selected reference chunks C--1 are provided as con-
3.4. Knowledge-Aware Task Decomposition
Addressing complex multihop questions often requires in- tegrating multiple pieces of knowledge, which implicitly
5

---

From Complex to Atomic: Enhancing Augmented Generation
Atomic
Elerentary Partictes) is a 2816 Geeman fita baoed on the mexal Via Particales alemetatren)" by Mimes dizeccal oskar Sceller and peoduced by Dlover Bestion and
"chusle; "Mtoiised Calso
1. "Mho arote and directed the file Acandsest
2. "Nto prodaced the fite stcalsed?*
Atomizer
"Mhen std Anarteed have the preatece at the
Beclin fill restisal?"
Eschtrger...
"Is which month and yeur etd Atardsed
(a) Illustrative example of of knowledge atomizing
Question
Question
Quastion
"Mhe the nother of the director of file Atomiaed
"he the mather of the directse of film Atcmised (FS1m))-
"Mo is the notter of the director of #ite Atoadote (Filn)7*
6F11.OP*
Context
Contect
Content
El "soarce, chussn: "Agentaed (slas loean the noxel
Elewattary Partictes) 1s 2000 Gerean file
Query Proposer
Query Proposer
eriter Gisala Elanse and acitec Rlaus Roshlee
chuskm:
Atomic Query Proposal
5 Atomic Proporal
Query Proposer
1.1Ma full of the film 'Atowised
thare any altarnate titles trarslations
"Mht pacents the directer Caker Ler
for fite "atowised"
"Whut nase oskar Roebler's eather?"
Atomic Query Proposal
directed file "stcmisse!""
*Te there blagraphy proeite avallable includes fastly information atsut Doker Reehllec"
need additiinal Infermation nuteiide proxided
nationality
diswcter
<Ita "Atowised to
it "live Deker teen' av punltcly nesttared his nothee in inteevisms public appearencss?*
"Miua full name of the director of the <ite "Stoeised"
Atomic Retriever
Atomic Retriever
Question
Seo is the aother of the divectee of file Atomseed [Fita]*-
2
Retrieved Atomic
"atemic quary": "Whe directed the 8iin 'Atowised/?",
*atentc, quary": 'Ihz are the parents of dicestur Uskez teuhler!"
Contest
"ssurce_ chuner rapartoed Caloo known Un rovel Partictes) is a 2058 Gutmar +ll based
"atensc tag": "Whz noste and sirected the film stamisedr"
*atande tag": "Whe are the pazerts of Celor
"sasuce chask": "Atomised Calao kroms
*source, chusk"; "Osker feebley Obarn January 3003) is Gecman file cirector, acrvnsetcter and jouralist. vis in Starbberg Ginels Eltner arster Klass tounler.
de welter Giseta Elsnch ard the arbbor Raas booh bur
torticles)
film based
the rocel 1*Las Particales stamentaires"" by Hosiilebecs file vas vrittinn and direectd fnetler
soe
Generator
Atomic Selector
Atomic Selector
*eatimnale"; "The centext states that cskur Hachlar, the strector of the Hile laconteed, uae bem in weltee alaus hoshlor. Therefore, Gisela Elsner Is the nether daker Roshler.", *ansvey"; "oiseta Etsner!
"soTected": true,
"solected": true,
"cutionale": "I reed to select a subr siirqusstion that
"cationate": "he need to lassn the pacests of Oskar sbout the parents Oskaz foetler
help out who directed the
sul-pistians that
Atentzed
"chosen_inée" info") [
"chssen, infa")
"atonic sag": "Whe azote and directed the file
"atceic_ tag": "Mhe are the pacents of Oaker kozh?
Mtoeised""
(atap
Elonentary Partieles) is a 2008 Gersan file based the nsvel -*}
son of ucitur diseta Elaner and the neator Klass foenler ]
(b) Illustrative example of KAR3-RAG case
Figure 4. Illustrative examples of KAR -RAG cases: (a) Example of knowledge atomizing, (b) RAG case with knowledge-aware task decomposition. As iterations progress, the reference context is enriched by accumulating relevant chunks via atomic retrieval and selection. With the expansion of context, the number of atomic query proposals generated decreases until no further proposals are produced. Subsequently, the iteration process terminates, and the combined question and context are harnessed produce the final response.
text to avoid generating proposals linked to already known knowledge. Consequently, the query proposals evolve with each iteration, adapting to the updated context and aiming to explore additional knowledge beyond chunks in the context. For each atomic query proposal, we retrieve its top-k rele- vant atomic tag candidates along with their source chunks from the knowledge base. The atomic retrieval process is:
similarity of the corresponding embeddings to retrieve the top-k atomic tags, provided their similarity to a proposed atomic tag meets exceeds specified threshold 5. With the original question, the accumulated context, and the list of retrieved atomic pairs, the atomic selector employ an LLM to select the most useful atomic pair for problem-solving.
Ratom topk Sim(d qu)) KB pot (D)
(5)
LLM C,-, p4') (Gle Ikoto: di.))
(6)
where the atomic retriever, denoted as Ratoms produces set of retrieved atomic pairs for each atomic query proposal, represented as po = {Ø Ikate )- All the retrieved atomic pairs from each atomic query proposal are aggre-
The atomic selector, denoted as Satom further retrieve the relevant raw chunk of the atomic pair selected as the new context added in the t-th iteration, denoted as ct. This chunk corresponds to de, in equation 6. The chunk retrieval
gated to generate an overall set p4", We employ cosine
6

---

From Complex to Atomic: Enhancing Augmented Generation
process can be represented by the following formula,
4.1. Experimental Setup
Methods To thoroughly evaluate the performance of our proposed knowledge-aware decomposition approach, we have selected a variety of baseline methods that represent different strategies for task-solving with LLMs. We in- clude Zero-Shot CoT (Kojima et al., 2022) to assess the inherent reasoning capabilities and built-in knowledge of the underlying LLM without any additional context. Naive RAG (Lewis et al., 2020), which introduces external knowl- edge through retrieval, serves as a benchmark for evaluating the incremental benefits of augmented knowledge. The Self-Ask framework (Press et al., 2023) is employed to investigate the impact of an iterative question decompo- sition and answering strategy on task performance. The IRCoT (Trivedi et al., 2023), which iteratively generates the rationale to process the multihop questions, along with the Iter- (Shao et al., 2023), which iteratively uses the recent response sponse quality, and the Prob Tree (Cao et al., 2023), which explicitly decompose the complex QA into a search tree, are also conducted for performance comparison. Detailed descriptions of methods are provided in Appendix A.4.
Ct = Satom Caatom(aaoonn RRatom (ol
(7)
This retrieved chunk is aggregated into the reference con- text for the next round of decomposition, expressed as C, = U C.-1. Knowledge-aware decomposition can it- crate up to N times, where N is hyperparameter set to control computational cost. The iteration process may con- clude earlier if it fails to retrieve suitable atomic tags, either due to the generation of low-quality query proposals or the absence of relevant atomic tag candidates. Alternatively, the process can be halted if the LLM deems the accumu- lated knowledge adequate for task completion. This early termination mechanism allows the process to conclude be- fore completing all iterations, reducing computational costs without compromising accuracy. Finally, the accumulated context C₁ is utilized to generate answer à for the given question q in line 1.
retrieval query improve the re-
It is worth mentioning that knowledge-aware decomposition can be learnable component. For each specialized knowl- edge base, we can utilize the data collected in each decompo- sition iteration speecficlyyy (g, a, à, ( of. q', Pa" C.))- This trained proposer can then directly suggest atomic queries q' during inference, which means lines to in Algorithm can be replaced by single call to this learned proposer, thereby reducing both inference time and com- putational cost. We leave the exploration of training an efficient query proposer as future work.
In our experiments, we employ GPT-4 (1106-Preveew) and Llama-3.1 1.....1770-Inttrure across the methods outlined previ- ously. For the experiments presented in Section 4.2, the iteration number N is set to 5 for Self-Ask with Retrieval, IRCoT, Iter-RetGen and KAR Additionally, the atomic retriever is initialized with k = and = 0.5. A compre- hensive list of syper--arameten for the retrieval and LLM can be found in Appendix A.3. For brevity, Llama-3.1-70B. Instruct abbreviated Llama in the following content.
Metrics To ensure consistency with established bench- marks, we adopt F1 as a conventional metric in our ex- perimental evaluation. To more accurately assess the the alignment of responses with the intended beyond mere lexical matching-we introduce a novel evaluation metric employing GPT-4. In this process, GPT-4 acts as an evaluator, assessing the correctness of a response in relation to the question and the correct answer labels. We refer to this metric as Accuracy (Acc). Upon manual inspection of sample set, the judgments rendered by GPT-4 demonstrate complete agreement with human evaluators, affirming the reliability of this metric. Furthermore, full evaluation re- sults with Exact Match (EM), Recall and Precision can be found in Appendix A.4.
4. Evaluation and Metrics
Since KAR* is proposed to handle the challenges in special- ized domains, we have conducted experiments on both a Chinese legal benchmark named LawBench and the Open Australian Legal QA benchmark. The experimental results demonstrated that KAR$ have obtained significantly im- provement than baseline methods across all these bench- marks, and the accuracy on generation tasks can reach up to 10% and 98.59% in LawBench and Australian Legal QA respectively. The detailed introduction to these legal benchmarks and the experimental results can be found in Appendix A.S.
To better compare the proposed approach with baseline methods, we focus on the redolie-eeog open-domain benchmarks in this section. Section 4.1 and 4.2 outline the experimental setup and the primary experimental results respectively. Ablation studies are discussed in Section 4.3. Additionally, cost analysis and case studies are included in Appendix A.5 and A.6 due to content constraints.
Datasets To better compare with baseline methods, evaluation focuses on three widely-recognized multihop datasets: HotpoiQA (Yang et al., 2018), 2WikiMulti- HopQA (Ho et al., 2020), and MuSiQue (Trivedi et al., 2022). A brief introduction to these datasets can be found in Appendix A. For each dataset, we randomly sample
7

---

From Complex to Atomic: Enhancing Augmented Generation
mantic gap. The "proposal first, then select" framework, detailed in Algorithm 1, enables a dynamic decomposition path search, provides an opportunity to validate the intent of the question and rectify potential errors in the historical
Table 1. Performance comparison on multihop QA datasets with GPT-4. Best in bold, second- best underlined.
HolpoiQA
2Wiki
MuSiQue
Method
F1
Acc
F1
Acc
F1
Acc
rationale generation process. A practical application of this
Zero-Shot CoT
43.94
53.60
41.40
43.87
22.90
23.47
point can be seen in Case(a) of Appendix A.6. Consequently, the experimental results demonstrate that KAR consistently outperforms other methods with different models, validating not only its effectiveness but only its robustness and adapt- ability for different models in complex reasoning scenarios.
Naive RAG
72.67
82.60
59.74
62.80
43,31
44.40
Self-Ask w/ R.
71.40
80.00
69.06
75.00
46.76
$1.40
IRCoT
67.30
81.00
63.83
70.40
47.57
49.20
Tter-RetGen
75.27
86.60
67.21
73.60
52.48
55.60
ProbTree
62.41
73.40
69.42
80.00
43.26
52.86
KAR (Ours)
76.48
88,00
75.00
82.20
57.86
62.60
4.3. Ablation Study
Table 2. Performance comparison on multihop QA datasets with Llama 3. Best in bold, second-best underlined.
The selection of N. We first conducted experiments with the iteration upper bound N set to 1, 2, ..... 10, and the results are presented in Figure 5. Detailed performance metrics are available in Table 8 of Appendix A.4. Across all three datasets, there is a consistent uptrend in both Supporting
HotpotQA
2Wiki
MuSiQue
Method
F1
Acc
F1
Acc
F1
Acc
Zero-Shot CoT
40.10
54.80
38.54
43.20
15.69
19.80
Naive RAG
70.78
84.20
56.58
62.20
32.53
36.40
Fact Recall and Answer Accuracy. This pattern underscores the approach's 's capability to incrementally enhance its out- puts through additional iterations, particularly when more detailed and contextually relevant information is required to address problem.
Self-Ask w/ R.
70.25
83,00
66.25
74.00
38.19
44.20
IRCoT
74.59
88.00
69.49
77.60
43.12
49.60
Iter-RetGen
72.23
85,20
59.21
65.00
37.16
40.40
KAR (Ours)
75.27
88.20
72.79
81.00
50.68
59.70
Additionally, upon examining the relationship between the number of iterations and the observed growth in supporting fact recall, we note that for HotPotQA and 2Wiki datasets, the recall curves exhibit pronounced increase up to the fourth iteration. Conversely, the recall for the MuSiQue dataset continues to rise sharply beyond this point, even though the maximum number of hops per question is capped at four, as mentioned in Appendix A. This discrepancy implies that while KAR is adept at retrieving relevant and useful information within a limited number of iterations, it still has certain limitation: KAR relies on the reasoning capability of the used LLM, and further iterations may be re- quired to fully capture the necessary information, especially as the complexity of the questions increases.
500 QA data from the dev set, disregarding the question type and the number of hops to ensure randomness. We compile the context paragraphs from all sampled QA data into single knowledge base for each benchmark, creat- ing a more complex retrieval scenario. This design choice aims to rigorously assess the task decomposition and rele- vant context retrieval capabilities of our model. For brevity, 2WikiMuliHopQQA is abbreviated as 2Wiki.
4.2. Main Results
As demonstrated in Table 1 and Table 21, our approach achieves superior performance across all datasets with both GPT-4 and Llama 3. Specifically, with GPT-4, we observe increases of approximately +1.4(1.6%) +2.2(2.8% and +7.0(1..6%) in accuracy over the second-best results for HotpotQA, 2Wiki, and MuSiQue. Similarly, with Llama 3, we achieve increases of +0.)(0.2%) +3.4(4.4%), and +10.1 for three datasets, respectively. These en- hancements are statistically significant, underscoring the robustness of KAR in handling complex QA tasks.
A Although Algorithm I does incorporate early-stopping mechanisms, a higher N invariably leads to increased com- putational demands. Therefore, we choose N = 5 - - value slightly above the maximum number of hops for the exper- iments in Section 4.2 to achieve a delicate balance between computational resources and the expected enhancement in performance.
Our proposed approach, KAR3, emphasizes knowledge- aware task decomposition and differs from the spontaneous decomposition mechanism reliant on given demonstrations, as employed by belf-Ask. It performs decomposition with an awareness of available knowledge and effectively uses atomic tags as an intermediate medium to bridge the se-
Hepsens
MudiQuz
122
Since we encountered problem obtaining the logprobs from the Llama 3 endpoint, we leave the experiment of ProbTree with Llama 3 as future work.
Figure 5. Supporting fact recall (in blue) and answer accuracy (in orange) over iterations.
8

---

From Complex to Atomic: Enhancing Augmented Generation
Table 3. Ablation study of the components in KAR".
HotporQA
2Wiki
MuSiQue
Variable Compunent
Modification
F1
Acc
F1
Acc
F1
Acc
Knowledge Atomizer
Atomizing to questions > Atomizing to plain texts Proposing multiple queries + Proposing a single query
73,05
84,50
64.15
69.80
50.72
55.20
Query Proposer
75.06
85.60
70.19
76.40
49.67
52.20
Atomic Retriever
Retrieving (atomic tag. chunk) pairs Retrieving chunks
76.31
86.60
67.14
72.40
49.05
53,00
Atomic Selector
Selecting chunks by atomic tags Selecting chunks directly
72.80
83,20
61.65
65.80
49.31
53,40
KAR (Ours)
76.48
88,00
75.00
82.20
57.86
62.60
The contribution of the approach components. KAR is comprised of four key components: a knowledge atomizer, a query proposer, an atomic retriever, and an atomic selector. We conduct ablation studies to ascertain the individual and collective contributions of these components.
over the baseline methods, our approach requires higher token consumption compared to some of the methods evalu- ated, as detailed in Table 10 in Appendix A.5. Specifically, on MuSiQue, it uses fewer tokens than ProbTree and IR- CoT, but more than Iter-RetGen and Self-Ask with retrieval. This increased token usage could lead to higher costs when implemented with proprietary models like GPT-4.
by introducing several method variants with modification to these components one by one: (1) For the knowledge at- omizer, we change the atomic tag presentation from atomic questions to plain text sentences to explore the influence of atomic knowledge representation; (2) For the query pro- poser, we limit to generate only one query to evaluate the advantage of the originally designed multiple proposals mechanism; (3) For the atomic retriever, modified the components to let it retrieve chunks rather than (atomic tag, chunk) pairs; (4) For the atomic selector, instead of filtering chunks by atomic tags, we implemented a variant to select chunks directly. Since there is no atomic tag existing in this setting, the context selection later determined by the chunk directly.
5. Conclusion
We present an advanced RAG system, enhanced with knowledge-award dual rewriting and reasoning capabilities, designed to improve knowledge extraction and rationale formulation within specialized datasets. The comprehen- sive results of extensive experiments underscore the efficacy of our approach, particularly in scenarios involving bench- marks with multihop questions. For future work, we aim to refine the system's 's proficiency through the integration of in-context learning (Wei al., 2022). by adaptively select- ing demonstrations for the query proposer. This will further enhance its ability perform knowledge-aware question rewriting. Additionally, we are interested in developing a knowledge-awarc atomizer capable of incorporating feed- back from sample questions, thereby improving its under- standing of the most beneficial types of atomic knowledge.
As evidenced by the results in Table 3, the individual contri- butions of the components were evaluated. We observed that replacing the knowledge atomizer, query proposer, atomic retriever and atomic selector with their substitutes will lead to accuracy reductions up to 1%% 16.6%, 15.3% and 16.2%, respectively, over three datasets. These ablation studies imply that each designed component crucial for achieving optimal retrieval performance and coherent rea- soning traces.
Impact Statement
Our approach utilizes existing large language models to avoid additional training and minimize the introduction of new biases, generating responses from pre-processed knowl- edge base to ensure reliability. The process records cach step of question dccomposition, creating a transparent and interpretable reasoning chain, and can be privately deployed to enhance data security in sensitive environments. This approach advances the use of Retrieval- Augmented Gener- ation (RAG) technology in fields like legal research, medi- cal diagnostics, and technical support, improving decision- making quality and efficiency. The enhanced clarity, pre- cision, and logical coherence of information could lead to better healthcare outcomes, more accurate legal judgments, and improved technical assistance, contributing significantly to societal well-being and progress.
Limitation Discussion. Beyond the need for additional iterations to extract crucial information for complex ques- tions, our experiments with GPT-3.5 - detailed in Table 9 in Appendix A.4 indicate a limitation in relying on LLMs' reasoning capabilities. With GPT-3.5, the performance of KAR* does not significantly surpass that of methods like IRCoT and Self-Ask w/ Retrieval and occasionally falls short compared to Aelf-Ask w/ Retrieval. This highlights that KAR success hinges on its advanced reasoning skills and its ability to robustly follow complex instructions.
While the experimental results using the open-source model Llama demonstrate notable performance improvement

---

From Complex to Atomic: Enhancing Augmented Generation
References
Gao, L.,, Ma, X., Lin, J., and Callan, J. Precise zero-shot
dense retrieval without relevance lahels, 2022. URL https: / /arxiv www.wiwwovvaa.e.oeeee.o.e.oeeeoe wwtp:/iw...g//a///.2.......
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.1094 2023.
Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi. H. Self- rag: Learning retrieve, generate, and critique through self-reflection, 2023. URL https://arxiv. abs/2310.20 11511.
Hayashi, H., Budania, P., Wang, P., Ackerson, C., Neer- vannan, R., and Neubig, G. WikiAsp: A Dataset for Multi-domain Aspect-based Summarization. Transac- tions of the Association for Computational Linguistics, 9:
Bender, E. M., Gebru, T., McMillan--MMarrr A., and Mitchell, M. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Con- ference on Fairness, Accountability, and Transparency, pp. 610-623, ACM, 2021.
211 211-225, 2021.
Ho, X., Nguyen, A.-K. D., Sugawara, S., and Aizawa, A. Constructing a multi-hop qa dataset for compre- hensive evaluation of reasoning steps. arXiv preprint arXiv:201 1.01060, 2020.
Butler, U. Open australian legal qa. 2023. URL https:/ http:////gggaggfcccceeeeeee http://acc.cco/ttttttttteteeeeee aaaabbuuler/ppee--nssttaalaeaeee
/
Jiang. Z., Xu, F.F. Gao, L., Sun, Z., Liu, Q., Dwivedi- Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation, 2023. URL httth://taaaett org/ abs /2305 06983.
Cao. S., Zhang, J. Shi, J., Lv, X., Yao, Z. Tian, Q. Li, J.. and Hou, L. Probabilistic rre-off-houggt reasoning for answering knowledge-intensivve complex questions. arXiv preprint arXiv:231..13983 2023.
Jiapeng, L., Runze, L.,, Yabo, L., Tong, Z., Mingling, L., and Xiang, C. Tree of reviews: A tree-based dynamic iterative retrieval framework for multi-hop question an- swering, 2024. URL htwps: //apxii////xvvv////////s/ wor.ioooorrrreeeee.ooo /bbs/ 2404. 14464.
Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., and Yu, Dense x retrieval: What retrieval granularity should we use? arXiv preprint arXiv:2312.06648, 2023. URL https://arxiv. org/pdf/2311 0ww.066666.....
Khattab, O., Santhanam, K.,, Li. X. L., Hall, D., Liang, P., Potts, C., and Zaharia, M. Demonstrate-searce prrecc-- Composing retrieval and language models for knowledge- intensive nlp, 2023. URL
Cohere. Say goodbye to irrelevant search results: Co- here rerank is here. httpp::/txtt wwwere. com/ rerank/,2022 2023.
abs/2212 1 4024.
Accessed: 2023-08-28.
Kojima, T. Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., and Larson, J. From local global: A graph rag approach to query-focused summarization, 2024. URL petets::/aariie wttp:b//////////////////// 16130.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., Yih, W., Rocktaschel, T., et al. Retrieval-augmenne generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 33:9459-994 94449,944444- 2020.
Fang, Y. and other authors. Hierarchical graph network for multi-hop question answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Lin- guistics, 2020.
Li, R. and Du, X. Leveraging structured information for explainable multi-hop question answering and reason- ing, 2023. URL http:: /ttxxviiiivv.////////e/////aaava org/ abs/ 2311. 03734.
Fei, Z., Shen, X., Zhu, D., Zhou, F., Han, Z., Zhang, S., Chen, K., Shen, Z., and Ge, J. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309. 16289, 2023.
Liang, L., Sun, M., Gui, Z., Zhu, Z., Jiang, Z., Zhong, L., Qu, Y., Zhao, P., Bo, Z., Yang, J., Xiong, H., Yuan, L., Xu, J., Wang, Z., Zhang, Z., Zhang, W., Chen, H., Chen, W., and Zhou, J. Kag: Boosting llms in professional do- mains via knowledge augmented generation, 2024. URL https: http://wwwwvvoooo///////os
Feng. Z., Feng, X., Zhao, D., Yang, M. and Qin, B. Retrieval- -eeeeraatnneerttioonn synergy augmented large language models, 2023. URL https: /tte://aaree /aaxivviiv.. org/ abs/ 2310. 05149.
10

---

From Complex to Atomic: Enhancing Augmented Generation
Ling, C., Zhao, X., Lu, J., Deng, C., Zheng, C., Wang, J., Chowdhury, T., Li, Y., Cui, H., Zhang, X., Zhao, T., Panalkar, A., Mchta, D., Pasquali, S., Cheng, W., Wang, H., Liu, Y., Chen, Z., Chen, H., White, C., Gu, Q. Pei, J., Yang, C., and Zhao, I.. Domain specialization as the key to make large language models disruptive: A comprehensive survey, 2024. URL https: //apxii//axvv.. Etg/abs/2333 /305. 18703.
Wang, C., Liu, X., Yue, Y., Tang, X., Zhang, T., Jiayang, C., Yao, Y., Gao, W., Hu, X., Qi, Z.., Wang, Y., Yang. L.,, Wang. J., Xie, X., Zhang, Z., and Zhang, Y. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. 2023a.
Wang. Y., Lipka, N., Rossi, R. A., Siu, A., Zhang, R., and Derr, T. Knowledge graph prompting for multi-document question answering. 2023b. URL https: https://arxiv. org/ heg/abs/2300 111330.00.......
Ma, X., Gong, Y., He, Zhao, H., and Duan, N. Query rewriting for retrieed large language models. arXiv preprint arXiv:2305. /tti::223....283., 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le. Q. V., and Zhou, D. Chain-of-thought prompting clicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022.
Panda, P., Agarwal, A., Devaguptapu, C., Kaul, M., and P, A. Holmes: Hyper-relational knowledge graphs for multi-hop question answering using Ilms, 2024. URL
https:/ //tp///aavvv../.ee/.aaeeeae.aaia wwt.ooog/////.o.eo.ooo... ab// 6022..
Press, O., Zhang, M., Min, S., Schmidt, L.., Smith, N. A., and Lewis, M. Measuring and narrowing the composi- tionality gap in language models, 2023. URL https: // //arxiv.org/ abs/ /2100. 03350.
Xu, S., Pang, L., Shen, H., Cheng, X., and Chua, T.-S. Search-in-the-chain: Interactively enhancing large lan- guage models with search for knowledge-intensinn tasks. In Proceedings of the ACM on Web Conference 2024, pp. 1362-1373, 2024.
Qiu, M. and other authors. Dynamically fusing recurrent neural networks for multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2019.
Yang, S. trieval. com/ http://danceedrr---1----ee- -retrieval-1721113333 2023-08-28.
Advanced rag 01: Small-to-big re-
httt::://towrrdsdatee
Raina, and Gales, M. Question based retrieval using atomic units for enterprise rag, 2024. URL https:// arxiv. org/ abs/ / 2405 12363.
2023. Accessed:
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Beyton-Browwy K., and Shoham, Y. In-context retrieaal-augmenter language models. Transactions of the Association for Computational Linguistics, 11:1316- 1331, 2023.
Yang. Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. Hotpotqa: A dataset for diverse, explainable multi-hop question swering. arXiv preprint arXiv:1809 1809.09600, 2018.
Zhang. J., Zhang, H., Zhang, D., Liu, Y., and Huang, S. End-to-end beam retrieval for multi-hop question an- swering, 2024a. URL https: http://wwwvv.././....../... ors/abs/////aaaaoo/////o/ 3308.08973.
Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., and Chen, W. Enhancing retrieval-augmented large language models with iterative aatriieval-generattien synergy, 2023.
URL https::tp///mmm / //////aaaaa //txxvxxvvv./////a////////aaaava htrgg/abs/2330 15294.
Zhang. K., Zeng, J., Meng, F., Wang, Y., Sun, S., Bai, L., Shen, H., and Zhou, J. Tree-of rrre----raasnning question decomposition for complex question answering with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38 38(17):195001195
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
2024b.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Musique: Multihop questions via single-hop ques- tion composition. Transactions of the Association for Computational Linguistics, 10:539-5555 554, 2022.
Zheng, H. S., Mishra, S., Chen, X., Cheng, H.-T., Chi, E. H., Le, Q. V., and Zhou, D. Take a step back: Evoking reasoning via abstraction in large language models. arXiv preprint arXiv:2310.0611 2023.
Trivedi, H., Balasubramaniann N., Khot, T., and Sabharwal, A. Interleaving retrieval with thain-offthhuggee reasoning for knowledge-inten intensive mueti-step questions, 2023. URL https: / /arxivv www..v..o.....eeo..t.....oo.ee abs 2212 10509.
Zheng, H. S., Mishra, S., Chen, X., Cheng, H.-T., Chi, E. H., Le, Q. V., and Zhou, D. Take a step back: Evoking reasoning via abstraction in large language models, 2024. URL htt.st:/// //wxivv..vv//v wrgg/bbs/23100000 06117. http://////aaavooo/////////////

---

From Complex to Atomic: Enhancing Augmented Generation
Zhong, Z., Liu, H., Cui, X., Zhang, X., and Qin, Z. Mix- of-granularity: Optimize the chunking granularity for retrieval-augmented generation, 2024. URL https:// // arxiv. http.orr//abs/24..000222 00456
12

---

From Complex to Atomic: Enhancing Augmented Generation
A. Appendix
Appendix A.1 provides detailed introduction to the three open-domain benchmarks, Appendix A.3 enumerates the hyperpa- rameters utilzied in our experiments, and Appendix A.4 presents the comprehensive experimental results.
For more thorough understanding of our methodology, cost analysis is available in Appendix A.5. Additionally, an introduction and discussion of an alternative atomic tag presentation is also shown in this subsection.
Appendix A.6 explores three real case studies. The prompts employed across the four components in our approach are outlined in Appendix A.7, accompanied by a discussion on the influence of decomposition demonstration. Finally, the evaluation conducted on two legal benchmarks are detailed in Appendix A.8.
A.I. Introduction to Open-Domain Benchmarks
We provide a brief overview the multihop QA datasets used our experiments, noting that our method does not leverage the question type information nor the number of hops information during the solving process, as our approach is designed to be agnostic to such classifications. Table outlines the distribution of question types within our sampled sets, offering insight into the variety of reasoning challenges presented in our evaluation, though this does not directly impact our method.
The HotpotQA dataset is a well-known multihop QA benchmark primarily consisting of 2-hop questions, each associated with 10 Wikipedia paragraphs. Among these, some paragraphs contain supporting facts essential to answering the question, while the rest serve as distractors. The dataset also includes a question type field, which delineates the logical reasoning
HotpotQA
questions involve contrasting two entities, and bridge questions require inferring the bridge entity, or inferring the property of an entity through an intermediary entity, or locating the answer entity (Yang et al., 2018). Although method operates independently of these types, their description here is to exemplify the nature of questions within the dataset and to contextualize the expected performance variance across different benchmarks.
2WikiMultiHopOAA Inspired by HotporQA, 2WikiMultiHopQA expands the diversity of question types. It retains the comparison type from HotpotQA and introduces inference and compositional questions that evolve from the bridge type by focusing on entity attribute deduction and entity location, respectively. Additionally, the bridge comparison type is a novel category that requires synthesis of bridge and comparison reasoning. This dataset typically presents 2-hop to 4-hop questions, each accompanied by 10 Wikipedia paragraphs containing supporting facts and distractors. While these types inform the dataset's structure, they are not utilized by method, which treats all questions uniformly regardless of their categorization. For the sake of brevity, 2WikiMultHHpQAA is abbreviated to 2Wiki in this paper.
MuSiQue Addressing the issue that many multihop questions can be solved via shorteuts-arrivinng at correct answers without proper BartooinngMuSiQQA implements stringent filters and additional mechanisms specifically designed to encourage connected reasoning, reported by Trivedi et al. (Trivedi et al., 2022). Unlike the other datasets, MuSiQue does not categorize questions by type, but it does provide explicit information the number of hops required for each question, ranging from 2 to 4 hops. Each question is associated with 20 context paragraphs, which introduce mix of relevant and irrelevant information, further complicating the task of discerning the correct reasoning path. This explicit hop information, while not used by our method, underscores the complexity of the dataset and the robustness required by models to handle such challenges effectively.
Table 4. Distribution of question types across three distinct multihop QA datasets.
Type
Count
Ratio
comparison
132
26.4%
#Hops
Count
Ratio
Type
Count
Ratio
inference
64
12.8%
2
263
52.6%
comparison bridge
107
21.4%
compositional
196
39.2%
3
169
33.8%
393
78.6%
bridge_comparison
108
21.6%
68
13.6%
(a) HotPotQA
(b) 2WikMuliiHloQA
(c) MuSiQue
13

---

From Complex to Atomic: Enhancing Augmented Generation
A.2. Introduction to Evaluation Methods
In addition the methods outlined in Table I, we also conduct experiments with an iteratively search and decomposition method named SearChain (Xu et al., 2024) and a Knowledge Graph- based method named GraphRAG (Edge et al., 2024). The GraphRAG was inferred in both local and global modes. The methods evaluated in this study are listed as follows:
Zero-Shot CoT: Questions are addressed using solely the Chain-Of-Thought (CoT) technique, which prompts the LLMs to articulate its reasoning process step-by-step without the aid of example demonstrations or supplemental context. This method assesses the LLMs' intrinsic knowledge and reasoning capabiliites in zero-shot setting.
This approach employs dense retrieval from a flat knowledge base to procure relevant information for each question. The knowledge base consists of pre-embedded chunks are matched to the original question based on semantic similarity. The retrieval process is direct, without any intermediate task decomposition.
Naive RAG:
This method employs task decomposition strategy wherein the LLMs is prompted to iteratively generate and answer follow-up questions, thereby breaking down complex problems into more manageable sub-tasks. General demonstrations illustrating the logic and methodology of task decomposition are provided for all benchmarks to guide the L.MMs* reasoning process. Different to the original setting (Press et al., 2023), where the framework relies solely on LLM's own knowledge to answer each follow-up question, this setting, we introduces an additional retrieval component. Relevant chunks retrieved with the follow-up question the query from flat knowledge base to provide a reference context. What's more, we also limit the decomposition process to raise up to N follow-up questions to align with other methods.
Self-Ask w/ Retrieval:
This approach iteratively prompts LLMs to generate one more sentence of rationale with retrieved passages, and retrieves new passages with the newly generated reason. The original setting limit the process with maximum token number (Trivedi et al., 2023). In our experiments, we limit the total number of iterations to the constant N we used for our methods.
IRCoT:
This method iteratively answers questions with retrieved passages, and uses the newly generated rationale and answer for the next-round retrieval. In this setting, we also limit the total number of iterations to the same N.
Reer:RetGen:
SearChain:
This approach focuses on the interaction between L.L. and Information Retrieval (IR). SearChain starts from a -gee-rree reasoning chain named Chain-of-Query (CoQ) where each node consists of an IR-oriented query-answer pair. It then iteratively verifies the answer of each node of CoQ by IR and re-generate the CoQ for node that is not consistent with the retrieved information. The re-generation mechanism let SearChain forms novel reasoning path based on tree, which enables LLM to dynamically modify the direction of reasoning. Since the official code loads pre-trained models from local without uploading those models online, we find models with most-similar name from HuggingFace to adapt it. Besides, the experimental results shown in Table 6 are conducted with BAAl/bge-m3 instead of the ColBERT retriever due to environmental issues.
This approach is an explict tree search method. ProbTree starts from a L.L.-translaeed query tree for the given question, in which each non-root node denotes a sub-question of its parent node. Then, probabilistic reasoning is conducted over the tree, by solving questions from leaf to root considering the confidence of both question decomposing and answering.
Prob Tree:
GraphRAG Local: The knowledge base is processed to construct a knowledge graph in accordance with the public guidance. The evaluation is inferred in local mode.
GraphRAG Global: The knowledge base is pre-processed to construct knowledge graph in accordance with the public guidance. The evaluation is inferred in global mode.
KAR (Ours): The proposed knowledge-awan decomposition method iteratively decomposes complex questions into sub-questions and retrieves relevant knowledge up to a maximum of N iterations. This process limits the context for the final answer to the five most useful knowledge chunks.
To better illustrate the distinctions among the evaluation methods discussed, we have systematically detailed their character- isties in Table 5. This table classifies each method according to its approach to question decomposition, chunk retrieval, and
14

---

From Complex to Atomic: Enhancing Augmented Generation
Table 5. Method Comparison.
Decomposition
Generation Context
Method
Retrieval
demonstration
path
context
sub-answer
final answer
Zero-Shot CoT
N/A
N/A
N/A
N/A
Naive RAG
N/A
question + chunk
N/A
chunks
Self-Ask w/ R.
few-shot
chain
qa pairs
sub-question + chunk
chunks
qa pairs
IRCoT
few-shot
(implicit)
rationale, chunks
rationale sentence - chunk
N/A
rationale, chunks chunks
Iter-RetGen
zero-shot
(implicit)
chunks
whole rationale chunk
N/A
SearChain
few-shot
chain
0: qa pairs, chunks
sub-question - chunk
0: chunks
qu pairs
ProbTree
few-shot
0 selected chunks
sub-question - chunk
chunks
qa pairs
KAR (Ours)
zcro-shot
dynamic
sub-question + atomic question + chunk
N/A
selected chunks
the context used in answer generation. Specifically, delineates whether each method operates under zero-shot or few-shot conditions, the nature of its decomposition process (e.g., explicit or implicit decomposition; thain-shapeds tree-shaped, dynamically generated paths), and the context utilized during decomposition. The retrieval column clarifies the mechanisms each method employs to gather information, while the columns dedicated to the generation context-both for sub-answer and final answer ceneratioon-highiige the specific contexts each method leverages when generating answers.
As Table 5 demonstrates, the decomposition module KAR employs zero-shot, knowledge-an aware approach, maintaining accumulated selected chunks in context for iterative decomposition. Additionally, we discuss the potential benefits of incorporating demonstration in Appendix A.7, suggesting that this feature could further enhance performance. This possibility carmarked for future exploration. Notably, approach dynamically formulates decomposition path during iterations, allowing for adjustments based on new insights from the contextually provided knowledge. In the retrieval phase, uses atomic tags to bridge the semantic gap between the query and the information within the chunks. Importantly, during the generation phase, our method retains the selected chunks, ensuring that the generation remains knowledge-awan and mitigates the risk of error accumulation often seen in methods that rely solely on follow-up questions and answers for context.
A.3. Hlyper-Paramett
During the knowledge extraction phase, utilize temperature setting of 0.7 specifically for the Knowledge Atomizing process, promoting a balance between diversity and determinism in the generated atomic knowledge. Conversely, for all question-answering (QA) steps in each method, we implement a temperature of 0, ensuring consistent responses from the model.
Regarding the retrieval component, we engage the text-embedding -ada--dd--02 (version 2) our embedding model for both the general knowledge bases and the atomic knowledge bases. For the general knowledge bases used Naive RAG and Iter-RetGen, the retriever configured to fetch up 16 knowledge chunks, applying retrieval score threshold of 0.2. For the general knowledge bases used in Self-Ask w/ Retrieval and IRCoT, where the retrieval chunks are used for a single follow-up question answering or the generation of single continuous rationale sentence, the reference chunks for whole rationale or final question answering are accumulated. The system retrieves 4 relevant chunks per request, maintaining the same score threshold of 0.2. In the case of atomic knowledge bases, the retriever is set to retrieve 4 relevant atomic tags for each atomic query but with a higher threshold 0.5 due to the shorter content length.
A.4. Detailed Experimental Results
As for the evaluation metrics, three more metrics employed in Appendix. Exact Match (EM), which assesses whether the response is identical to predefined correct answer is applied as the community usually did, Furthermore, we encounter situations where a method achieves high accuracy (Acc) scores yet registers low Fl scores. To elucidate the underlying factors of such discrepancies, we also report on the Recall and Precision of the generated responses. Recall measures the proportion of relevant tokens from the answer labels that are captured in the response, while precision evaluates the relevance of the tokens in the generated answer with respect to the correct labels.
Evaluation Metrics
15

---

From Complex to Atomic: Enhancing Augmented Generation
Table 6. Detailed performance comparison on multihop QA datasets. Best in bold, second-best underlined.
Table 6. (a) HotPotQA
Method
FM
Acc
Precision
Recall
Zero-Shot CoT
32.60
43.94
53.60
46.56
43.97
Naive RAG
56.80
72.67
82.60
74.52
74.86
Self-Ask w/ Retrieval
$7.00
71.40
73.25
73.95
IRCoT
51.40
67,30
81.00
69.32
72.15
Iter-ReiGen
59.60
75.27
77.18
77.62
SearChain
28.60
40.48
74.40
40.77
66,63
ProbTree
47.00
62.41
73.40
64.83
64.95
GraphRAGi Local
0.00
10.66
89.00
5.90
83.07
GraphRAG Global
0.00
7.42
64.80
4.08
63.16
KAR (Ours)
61.40
76.48
88.00
78.53
78.96
Table (b) 2WikiMultiHopQA
Method
EM
Acc
Precision
Recall
Zero-Shot CoT
35.67
41,40
43.87
41.43
43.11
Naive RAG
51.20
59.74
59.06
62.30
Self-Ask w/ Retrieval
60.60
69.06
75.00
67.88
73.15
IRCoT
55.00
63.83
70.40
62.47
68.86
Iter-RetGen
57.80
67.21
73.60
66.10
71.09
ScarChain
7.00
15.67
68.40
11.91
66.74
ProbTree
57.00
69,42
80.00
67.61
76.89
GraphRAG Local
0.00
11.83
71.20
6.74
75.17
GraphRAG Global
0.00
7.35
45.00
4.09
55.43
KAR (Ours)
65.80
75.00
82.20
73.63
79.08
Table (c) MuSiQue
Method
EM
F1
Acc
Precision
Recall
Zero-Shot
12.93
22.90
23.47
24.40
24.10
Naive RAG
32.00
43.31
44.40
44.42
47.29
Self-Ask w/ Retricval
38.20
46.76
51.40
46.75
51.00
IRCoT
36.00
47.57
49.20
48.70
50.30
Iter- Reinen
40.20
52.48
55.60
53.51
56.45
SearChain
24.40
33.26
45.80
33.00
46.37
ProhTree
28.57
43.26
52.86
42.27
$4,70
GraphRAG Local
0.60
9.62
5.73
55.82
GraphRAG Global
0.00
5.16
44.60
2.82
52.19
KAR (Ours)
47.40
57.86
62.60
58.52
61.37
Detailed Main Results The detailed experimental results on multihop datasets HotpotQA, 2Wiki and MuSiQue are presented in Table 6. Besides the metrics shown in Table 1, the EM, Precision and Recall are provided here.
Discussion of Graph-Based Method Notably, knowledge graph-based method, GraphRAG Local, excels HotpotQA dataset predominantly comprised of 2-hop questions. However, in the other two datasets, which contain questions involving more hops, GraphRAG Local is merely on par with IRCoT. This highlights the challenge that knowledge graph-based methods face in addressing complex multihop questions. Regarding GraphRAG, originally designed for the query-focused summarization (QFS) task outlined by (Edge et al., 2024), observe its suboptimal performance in both local and global modes compared to our method. GraphRAG exhibits curious trend: it achieves higher accuracy and recall scores while performing lower on EM, FI, and Precision metrics. A closer analysis of GraphRAG's outputs reveals a tendency to echo the query and include meta-information about the answer within its graph structure. Despite attempts to refine its QA prompt, this behavior persists. An illustrative example is presented in Table which shows GraphRAG Local's response to a question from HotpotQA.
16

---

From Complex to Atomic: Enhancing Augmented Generation
Table An Example of GraphRAG Local output HotpotQA question. The table showcases the tendency to repeat the question and include meta- eeta-information in its response.
Question
Which country is home to Alsa Mall and Spencer Plaza?
Answer Labels
India
Answer of GraphRAG
Alsa Mall and Spencer Plaza are both located in Chennai, India [Data: In- dia and Chennai Community (2391); Entities (4901, 4904); Relationships (9479, 1687, 5215, 5217)].
Detailed Evaluation Results of N Selection Table 8 lists the granular performance metrics according to those we shown in Figure 5 for the ablation study on the iteration upper bound N. Different to the Recall we reported in Table 6, which indicates the recall tokens the answer labels, the Recall' here represents the recall of the supporting facts provided by these datasets.
Table 8, Ablation study hyper-parameter N. Recall" indicates the recall of supporting facts.
HotpotQA
2Wiki
MuSiQue
N
Recall
F1
Acc
Recall
Fl
Acc
Recall
F1
Acc
1 42.96
59.46
70.20
40.41
41.08
43.00
31.20
32.55
32.80
2 82.04
74.27
84.80
78.83
70.22
77.20
56.43
48.46
50.00
3
90.16
76.90
87.20
87.71
72.84
79.40
64.82
53.50
57.20
4
92.46
76.49
87.80
92.86
74.68
81.80
69.87
55.73
59.40
5
92.83
76.48
88.00
94.06
75.00
82.20
73.08
57.86
62.60
6
93.35
77.67
89.00
94.76
75.12
81.80
74.88
57.03
61.20
7
93.68
77.32
88.80
94.91
75.44
82.40
76.07
56.66
61.40
8
93.78
76.88
88.40
95.06
75.16
82.00
76.72
57.65
62.40
9 93.78
76.99
88.60
95.11
74.89
81.80
76.90
$7.17
61.40
10
93.78
77.52
89.00
95.16
75.09
82.00
77.20
$7.69
62.40
As introduced the limitation discussion section, we have carried out a series of experiments utilizing GPT-3.5. The outcomes of these experiments are delineated in Table For these specific trials, we substituted GPT-4 (1 (116--r)))www) with GPT-3.5 (1106-Preview) as the language model, while maintaining all other experimental settings identical to those employed in the experiments summarized in Table 1.
Evaluation Results with Less Advanced LLM
Table Performance comparison of implementations with GPT-3.5. 5. Best in bold, second-best underlined.
HotpotQA
2Wiki
MuSiQue
Method
F1
Acc
Fl
Acc
F1
Acc
Solf-Ask w/ Retrieval
49.52
61.40
53.83
60.00
31.05
35.20
IRCoT
56.39
68.40
40.31
46.00
33.93
34,40
Iter RetGen
48.63
66.80
44.32
55.20
25.77
37.80
KAR (Ours)
46.37
68.80
41,95
58.20
26.80
39.60
A.5. Cost Analysis and Discussion
In this section, we conduct comprehensive cost analysis to evaluate our model's s API consumption. We first evaluate and compare the inference cost to other baseline methods, later we further decompose the cost into components, and finally a cost summarization of the one-time data preprocessing step will be provided.
Inference Cost Comparison As Table 10 demonstrated, from the perspective token consumption per QA, our method utilizes fewer tokens than both ProbTree and IRCoT, and is comparable to Iter- ner-RetGen. However, our approach significantly outperforms these baselines on both F1 and Accuracy by considerable margin. This demonstrates the efficiency of our
17

---

From Complex to Atomic: Enhancing Augmented Generation
approach in balancing cost and performance. It important highlight that our method focuses on exploring potential reasoning chains, necessitating thoughtful analysis during question decomposition with context at each iteration. As result, completion token usage constitutes approximately one-quarter of the total consumption, distinguishing our approach from other baselines.
Table 10. Token consumption (average/QA) and performance comparison MuSiQue.
Token Consumption (4)
Performance (ft)
Method
Prompt
Completion
Total
F1
Acc
Zero-Shot CoT
85
105
191
22.90
23.47
Naive RAG
1765
103
1869
43.31
44.40
Self-Ask w/ Retrieval
5894
619
6514
46.76
51.40
IRCoT
9703
86
9789
47.57
49.20
Iter-RetGen
8140
473
8614
52.48
55.60
ProbTree
25225
650
25875
43.26
52.86
KAR (Oues)
6525
2295
8820
57.86
62.60
Token Consumption of Different Components In the experimental results presented so far, the same LLM is used for the decomposer, selector, and generator components. The prompts designed for these components are detailed in Appendix A.7. It is worth noting that these components can be configured to use different language models, we leave it as future works. The detailed token consumption of difference components on MuSiQue are illustrated in Table 11. The decomposition-selectio loop iterates up to 5 rounds, leading to the multiple calls for decomposer and selector for each QA. Consequently, the decomposer and selector constitute the majority of the total consumption.
Table 11. Token consumption (average/QA) MuSiQue.
Component
Prompt
Completion
Total
Query Proposer
2691
768
3459
Atomic Selector
3278
1429
4707
Answer Generator
556
98
654
KAR
(Ours)
6525
2295
8820
Token Consumption of Chunk Atomization The chunk atomization, as a one-time preprocessing step, for which the LLM API consumption scales linearly with the number of data chunks and constitutes an overhead that varies slightly across different benchmarks. As described in Section 4.1, all chunks are derived from the context paragraphs, and the number of LLM calls, which is equivalent to the chunk count, is listed in the last column of Table 12, together with the token consumption, for your reference. The input token size (i.e., Prompt in the table) is primarily determined by the chunk size, while the output token size (i.e., Completion in the table) depends on the size of generated atomic tags.
Table 12, Token consumption (average/chunk) and chunk count statistics.
Dataset
Prompt
Completion
Total
Calls
HotpotQA
209
129
338
4950
2Wiki
199
122
321
3410
MuSiQue
197
123
320
7120
An Alternative Presentation of Atomic Tag We recognize the significance of scalability when applying our method to extensive datasets. To enhance cost-efffcctiveness while maintaining scalability, we integrate the use of open-source language models like Llama 3, which significantly reduces preprocessing costs. Additionally, we explore alternative atomic tag presentation to further optimize resource usage. One promising approach is atomizing data into plain-text sentences, treating each sentence as an atomic tag. This method simplifies the preprocessing steps by utilizing the "spacy" library to segment the original data chunks into sentences, thereby avoiding the need for language model invocations. Our evaluations, as detailed in Table 13, show that this approach, while reducing performance to 55.2% on the MuSiQue dataset, still outperforms most
18

---

From Complex to Atomic: Enhancing Augmented Generation
baseline methods. This demonstrates its potential effectiveness in scenarios where lower-cost preprocessing is a priority, offering viable alternative that balances cost and performance efficiently.
Table 13. Performance of alternative atomic tags on MuSiQue.
LLM
Atomic Tags
F1
Acc
plain text sentence
45.88
54.20
Llama
atomic question (Ours)
50.68
59.70
plain text
50.72
55,20
GPT-4
atomic question (Ours)
57.86
62.60
A.6. Real Case Studies
This section presents there real case studies from our evaluation benchmark to illustrate the underlying principles of our proposed decomposition pipeline, as detailed Algorithm 1. Through these world examples, we to highlight the benefits of our systematic approach. These cases will shed light how each step of the pipeline contributes to improved performance and the insights gained from their implementation.
Question
Atomic Query Proposal
Who the mother of the director of film Atonised (Filn)?
parents
Rochler?
2. Mhat is the nane of Oskar Rochler's nother? there biography profile available that family information
Target Chunk
title: Oskar Roehler
Rochler?
contents: He was born in Starnberg the of writer Gisela Elsner the writer Klaus Roehler.
Atomic Retrieval
atomic query who are the parents of the director Oskar Roshler?
Follow-Up Question
atomic
chunk relevant?
Who is the nother of Oskar Roehler?
Who are the parents of Oskar Roebier? Where Rowhier Who were the parents of Hans Christoff?
Hierarchical Retrieval
Oskar bornž
source chunk
chunk relevant?
She was the mother of Prince Monnaga.
Mangaret
atomic query
Lejoonhufvud and second cousin and weether-in-lowe of
What is the name of Oskar Roehler's mother?
atomic tag
chunk relevant?
Netherlands, who was the Prince Consort of Queen Beatrix the Netherlands, thus making the mother-
Jaegerhuber's mother?
Waldemar V's mother?
the fother the later Member Parliament Anna Ellasson.
Who was leedwig of Münsterber''s mother?
What was the rumn Ruskol mother?
Self-Ask
KAR3
Figure 6. Case (a): Given the lesser-known film "What Women Love" opposed the more popular "What Women Want," single-path methods like Self-Ask on the left are predisposed to generating follow-up questions about the latter, leading to an incorrect final answer. Conversely, KAR* can effectively discern the intended meaning of the original question by positing several atomic queries and postpone the task understanding to atomic selection phase with relevant atomic tags provided, and subsequently arriving at an accurate conclusion.
19

---

From Complex to Atomic: Enhancing Augmented Generation
KAR outperforms single-path methods by effiectively discerning the intended meaning of the original questions. Our task decomposition strategy involves generating multiple atomic queries rather than producing single deterministic follow-up question, as demonstrated in the Self-Ask approach. Contemporary decomposition methods typically employ generative model to formulate singular follow-up question. However, this approach carries an intrinsic risk of generating erroneous questions, potentially leading to an incorrect decomposition pathway and, ultimately, an erroneous answer. Consider the Case (a) depicted in Figure 6, where the original question pertains to film titled "What Women Love." Due to existence of a more prominent film, "What Women Want," the employed language model tends to 'correct' the original question. Consequently, methods like Self-Ask (as shown on the left side of Figure 6) generate only one follow-up question related to this erroneously assumed object. In the illustrated instance, although the target chunk has been retrieved due the similarity in embeddings. 'false' intermediate answer is produced for the 'false' follow-up question, culminating in an incorrect final response. In contrast, our methodology posits atomic queries concerning both "What Women Love" and "What Women Want," thereby seeking to clarify the true intent of the initial question. With both films existing and relevant atomic tags being retrieved, our approach subsequently gains the advantage of verifying the question's 's intent and selecting the correct and most pertinent chunk during the atomic selection phase.
Question
Atomic Query Proposal
Mho is the mother of the director of file Atomised (Filn)?
Whe parents of the director Oskar Roshles?
2. What is the name of Osker Rochler's mother?
3. Is there bisgoappy ez pesfile available that incluses family information about Cakar Reehler?
Target Chunk
title: Rowhler
contents: He Starnberg of nriter
Gisela
klaus Rochlar.
query
Who are the pazents of the director Oskaz Rochlez?
atomic
chunk relevant?
Who the panants Diker Roabler?
Follow-Up Question
Where Caker Roekier born?
who is the mother Oskar Roehler?
When Coaa Reebler bomm
Hlrrarhical Reerivval
Who parerts of Dhrieot Kirigona uk?
relevam?
She
atomic query
What is the name of Oskar Roeh nother?
Gueen Lejuedralea)
coudin mother-te
was the mother of Prince Caus of ketherande,
atomic
chunk relevant?
iifu roo Prens Conount Guster bostrie of the Merthelandss making motherr
Who Werner Arton hasger mather?
Whlt the of Wandrrr Vt's motho??
who weo Hoziing Miniter berg mather?
whetwo name of Racolf I'' mether?
Self-Ask
KAR
Figure 7. Case (b): By proposing multiple atomic queries, KAR effectively retrieves the relevant knowledge chunk, whereas the single deterministic follow-up question approach employed by Self-Ask fails to align with the knowledge base's schema, resulting in retrieval failure.
KAR outperforms baseline methods through better knowledge schema alignment by atomic tags as bridge. The discrepancy between the formulation of the corpus and the query, is another critical factor ad advocating for multi-query approach over a singular deterministic one. The presentation gap can impede the retrieval process even when the generated follow-up question is semantically accurate, For instance, as illustrated in Case (b) in Figure 7, a single-path method such as Self-Ask on the left side might directly inquire 'Who the mother of Oskar Rochler?' However, the knowledge base articulates familial relationships using different schema, 'A is the son of B and C' this case, thus the retrieval process falters despite the correctness of the question. Even when applied the hierarchical retrieval to Self-Ask. the Self-Ask with Hierarchical Retrieval did not succeed in bridging this gap. In contrast, our approach, which generates multiple atomic queries, encompasses broader range of phrasings that correspond to the diverse representations in the knowledge base. In the depicted case, while the atomic query specifically asking for Oskar Roehler's mother encounters the same retrieval issue, an alternative query seeking information about his parents successfully retrieves the target chunk. This exemplifies how our method's s flexibility in query generation enhances the likelihood aligning with the knowledge base's 's structure and
20

---

From Complex to Atomic: Enhancing Augmented Generation
obtaining accurate information.
Our methodology emphasizes the retrieval of atomic tags rather than directly retrieving chunks. This design choice is exemplified in Case (b) depicted in Figure 7. The knowledge chunk in the corpus is structured using the pattern 'A as the son of B and C", which poses challenges for direct retrieval by queries such as 'Who is the mother of . In our specialized knowledge base, such direct queries tend to retrieve chunks conforming to the patterns 'A is the mother of B' or 'A is the father of B", By utilizing atomic tags as intermediaries for retrieval, our approach effectively narrows the gap between a single query and the multiple sentence structures found in the knowledge base. It facilitates bridging the expression pattern differences exemplified by 'the mother of" versus 'the son of" in this scenario.
KAR outperforms methods that rely on intermediate answers by maintaining concise and highly relevant context. In contrast to methods like Self-Ask, which only retains intermediate answers for subsequent processing, our method preserves the entire chunk as contextual information. During the atomic selection phase, we present list of atomic tags as candidate summaries of the relevant content from the original chunk. This strategy significantly reduces token usage and simplifies the process of selecting the pertinent information. Case (c) in Figure 8 demonstrates the dual benefits of our approach: first, by selecting from a curated list of atomic tags, we streamline the identification of relevant information; second, by retaining the entire selected chunk rather than just the intermediate answer, we ensure a rich context is maintained for accurate and comprehensive subsequent processing. While the Self-Ask method on the left retrieves the target chunk, it fails to correctly identify the pertinent 'Emni Watts' due to the excessive contextual information. Since retrieved chunks in Self-Ask discarded after generating an intermediate answer, the method potentially follows an incorrect pathway, leading to an inaccurate conclusion. In contrast, our approach can efficiently filter and select the appropriate atomic tag from a concise list. Although the atomic tag in this round pertains to the role of Emni Watts, there is no need to inquire further about his birthplace, as this information is encapsulated within the selected chunk, which remains available for context in subsequent rounds.
Case (b) Question
Atomic Query Proposal
In which district was Ernie Watts born?
the full Ereta latte?
Target Chunk
1. W.st ix the date of birth of tinte Watce?
titte:
Emie watts (lootballer, barn 1872)
Atamic Natrional
contents: Baen doolhanpton in Burksshiee his earliest known club was Reading, before he apved to botts County, where he made 17 appearances in The Football League. Ha bad a second spell with Reading and also played for
atomic query
Eznáe Natss?
Up Queston Ernie
khuze Ernis betts boze?
Atonic Selection
"selectee":
Bann in arodia Jacodiin Ni aur crllectioonn club was Brading belore ba roved tokum County, whire he made appasrrcees in foothal...
"rationale":
"cheeen Lifa'
"atomie, Say": "chece was Eiile sotts berett
deete Thougri fetter Walts Bowe
chessrs.hhss". Woothenttn ineen Seading. befoce nsved letta Countyy to sido acpsarances Formalt Lesger. le had a secaad played
eartiest
dankl, Partin-uutiee Arthoay bakien.
Goneration
"Quesiton":
"Cortoot": {
Internediats Answer: trrle Vatts assician the playes in the album Gencla Thought..
t any
"caance Benn Hollbaytta na Peading, befare be nevd fetts Cuuntyy shure sade apeasannes rxcond spall with Seacting and alss played for ...
Fulore Up Ouodin Mut is the bistiplace of time Villo
Gereration
Fint Inmae: congreentarat district Virgints.
"rationale":
*answer": "West Barishiire*
Self-Ask
KAR
Figure 8. Case (c): KAR' has the advantage of leveraging a concise list of atomic tags for targeted selection and retaining full chunks for rich contextual support. Conversely, Self-Ask's approach, although successful in retricving relevant chunks, compromised by its dependency on intermediate answers for context, which ultimately results in the generation of incorrect final answers.
21

---

From Complex to Atomic: Enhancing Augmented Generation
A.7. Prompt Design
Our approach employs four distinct prompts: (1) Atomic question tagging prompt: the one used to pre-processing the source paragraphs that linking each paragraphs with several atomic questions atomic tags; (2) Atomic query proposer prompt: the one used when generating multiple atomic query proposals, referring line 1 in Algorithm 1; (3) Atomic tag selection prompt: the one used when selecting the most useful atomic tag from the given question list, referring to line in Algorithm I; (4) Question answering prompt: the one applied upon exiting the decomposition loop to generate the final answer to the given question, as described line of Algorithm 1.
Atomic Question Tagging Prompt
Task
Your task is to extract as many questions as possible that are relevant and can be answered by the given content. Please try to be diverse and avoid extracting duplicated or similar questions. Make sure your question contain necessary entity and avoid to pronouns like it, he, she, they, the company, the person etc.
Output Format
Output your answers line by line, with each question new line, without itemized symbols or numbers.
Content (content)
Output
Atomic Query Proposer Prompt
Task
Your task is to analyse the provicing context then raise atomic sub sub-questions for the knowledge that help you the question better. Think in different ways and raise as many diverse questions as possible.
Output Format
Please output in following JSON format:
{{
"thinking": <your thinking for this task, including analysis to the question and the given context>,
"sub ssub.questions": <a list of sub-questions indicating what you needs
}}
Context
Ihhe cortext we al ready have: {chosen content)
Question
(content)
Your Output
22

---

From Complex to Atomic: Enhancing Augmented Generation
Atomic Tag Selection Prompt
Task
Your task to analyse the providing context then decide which sub-questions may be useful to be answered before you can answer the given question. select a most
relevant sub-question from the given question list, avoid selecting sub-question that can already be answered with the given context or with your own knowledge.
Output Format
Please output in following JSON format:
{{
"thinking"; syour thinking for this selection task?,
"question_idx": sub-question index, integer from to fnum.atom.questionn
}}
Context
The context we already have: (chosen.content)
Sub-Questions You can Choose From (st.st.uess.....sttt.sttt
Question {content]
Your Cutput
Question Answering Prompt
Task
Your task is to answer question referring to given context, if any. For answering the Question at the end, you need to first read the articles, reports, context provided, then give your final answer.
Output format Your output should strictly follow the format below. Make sure your output parsable by json in Python.
{
"answer": <Your Answer, format it as a string.>, "rationale": <rationale behind your choice?
}}
Context, if any (context_if.any)
Question
(yees or no limit)
Let's think step by step.
Demonstration Discussion In our current experiments, all prompts are zero-shot, meaning no demonstrations are provided illustrate the expected reasoning logic. To explore whether demonstrations could enhance performance, we designed ablation study. We adapted the Self-Ask w/ Retrieval and IRCoT methodologies previously employed, modifying the prompts and task descriptions to create zero-shot, demonstration-fies variants of these methods. These were denoted as Zero-Shot Self-Ask w/ Retrieval and Zero-Shot IRCoT. The results of the experiment are presented in Table 14. The experimental results reveal that the Zero-Shot Self-Ask w/ Retrieval method experiences a marginal decline in accuracy for the 2Wiki and MuSiQue datasets, potentially due to the inherent randomness in generation. However, the inclusion of demonstrations significantly improves all Fl scores and enhances the overall performance of the IRCoT method. This suggests that demonstrations could be particularly beneficial for methods that rely on step-by-step decomposition approach. Consequently, integrating demonstrations is identified as promising direction for future work within the KAR framework.
23

---

From Complex to Atomic: Enhancing Augmented Generation
Table 14, Performance comparison: Zero-Shot vs. Few-Shot.
HotpotQA
2Wiki
MuSiQue
Method
F1
Acc
F1
Acc
F1
Acc
Zero-Shot Self-Ask w/ Retrieval
55.76
76.20
54.98
76.20
40.97
50.40
Self-Ask w/ Retrieval
71.40
80.00
69.06
75.00
46.76
51.40
Zero-Shot IRCoT
58.22
75.80
49.69
60.20
37.17
43.00
IRCoT
67.30
81.00
63.83
70.40
47.57
49.20
A.8. Evaluation on Legal Benchmarks
In this subsection, present the performance of our approach two legal benchmarks: LawBench (Fei et al., 2023) and Open Australian Legal QA (Butler, 2023). Before doing we provide a brief description of each benchmark.
LawBench LawBench is comprehensive legal benchmark for Chinese laws. It comprises 20 meticulously designed tasks aimed at accurately assessing the legal capabilities of LLMs. Unlike some existing benchmarks that rely solely on multiple-choice questions, LawBench includes variety of task types that are closely related to real-world applications. These tasks encompass legal entity recognition, reading comprehension, crime amount calculation, and legal consulting, among others. Since not all tasks are RAG-oriented (c.g., reading comprchension), we have selected specific tasks, which are detailed in Table 15. The number of questions of each task 500.
Table 15. Overview of Law Bench tasks Task
Task
Type
Metric
1-1
Statute Recitation
Generation
F1
1-2
Legal Knowledge Q&A
Single Choice
EM
3-1
Statute Prediction (Fact-based)
Multiple Choices
EM
3-2
Statute Prediction siecnario-based
Generation
F1
3-6
Case Analysis Consultation
Single Choice
EM
3-8
Generation
F1
We also provide example questions of these tasks for the readers reference (translated using GPT-4).
1-1: Answer the following question by di rectll provi ding the content of the article:what
+ is the content of Article 76 of the Securities law?
1-2: According ng to the Securities Law', which of the following statements about stock incorrect? A: Without the permission of the stock exchange, no entity + or individual may publish real-time securities trading information; B: The stock securities accounts that exhibit major abnormal trading conditions needed, and report the securities regulatory authority
exchanges
exchange may restrict trading
The accumulated property stock exchange belongs to the members, and their rights are jointly enjoyed by the
under the State Council for record;
a member-based
members; during its existence, the accumulated property may not be distributed to the members; D: The stock exchange formulates listing rules, trading rules, member management rules, and other relevant rules in accordance with securities laws and administrative regulations, and reports to the securities regulatory authority under the State Council for record.
3-1: Based en the following facts and charges, provide the relevant articles of the Criminal Law. racts: The Yushu City, Jilin Province, accused that on November 15, 2015, the defendant He signed car rental agreement with Guo, the owner of taxi with license plate number xxx. The agreement stipulated a monthly rent of RMB
+ 3, 3,900.00, payable monthly. On January 19, 2016, without the knowledge of Guo, the
+ defendant He concealed the truth and falsely claimed to be the owner of the taxi.
He signed car rental agreement with the victim Ma, with monthly rent of RMB
total of RMB 50, 600.00 from Ma for one year's rent and vehicie deposit. On February 26, 2016, the taxi was
3, 800.00 and a rental period one year, collecting
retrieved by its owner Guo from the victim Ma. The victim Ma repeatedly asked the
defendant He to return the rent and deposit, but the defendant He refused to return
them. The prosecution provided evidence including the defendant's confession, the
victim's statement, witness testimonies, and documentary evidence, and believed
24

---

From Complex to Atomic: Enhancing Augmented Generation
Table 16. Evaluation Results on Legal Benchmarks (Metric EM
indicated in Table 15)
Task
Zero-Shoo CoT
GraphRAG Local
Ours (N=5)
1-1
21.31
23.27
78.58
1-2
54.24
62.60
70.60
3-1
53.32
74,60
83.16
LawBench
3-2
27.51
25.98
46.05
3-6
51.16
47.64
61.91
3-8
17.44
18.43
23.58
Open Australian Legal QA
25.10
34.35
63.34
Table 17. Evaluation Results on Legal Benchmarks (Metric Acc)
Task
Sero-Shot CoT
GraphRAG Local
Ours (N=5)
1.23
16.60
90.12
1-2
54.00
63.40
70.60
3-1
49.90
75.40
88.82
LawBench
3-2
15.83
27.60
67.54
3-6
51.12
57.00
62.73
3-8
49.70
58.80
61.72
Open Australian Legal QA
16.48
88.27
98.59
that the defendant He, with the purpose of illegal possession, defrauded others their property by fabricating facts and concealing the truth during the signing and performance of the contract. The amount was relatively large, and his actions violated the provisiens of Article xx of the Criminal Law of the People's Republic of China, and should be held criminally responsible for Charge: Contract Fraud,
3-2: Please provide the legal basis according to the specific scenario and question, only
4 the content of the specific legal provision is needed, each scenario involves only ene legal provisien. Scenario: A cargo ship arrives the consignee fails to arrive
the port of discharge, but time to collect the goods. Under which legal
provision can the captain unload the goods at another appropriate, place?
3-6: One year after the bar opened, the business envi ronment. changed drastically, and all partners held meeting to discuss countermeasures. According to the "Partnership considered valid votes: Zhang believes that the name 'Tongcheng' is not attractive and proposes to change it Tongsheng Bar.' Wang and Zhao agree, but Li opposes; B: In view of the sluggish business, Wang preposes to suspend operations for one month for renovation and reorganization. Zhang and Zhao agree, but Li opposes; C: Due to the urgent needs of the bar, Zhao proposes to sell a batch of coffee machines to the bar. Zhang and Wang agree, but Li opposes; D: Given the four partners' lack of experience bar management, Li proposes to appoint his friend Wang as the managing partner. Zhang and Wang agree, but Zhao opposes.
Enterprise Law,' the following voting matters
3-8: Resident rented out the house and sublet
With A's consent, renovated the rented house c. C unilaterally altered the load-bearing structure of the house. + Why can A request B to bear liability for breach of contract?
Open Australian Legal QA The benchmark consists 2,124 questions and answers synthesized by GPT-4 from Australian legal corpus. All questions are of the generation type. One example is: "What the landlord's general obligation under section 63 of the Act in the case of Anderson Armitage [2014] NSWCATCD 157 New South Wales?"
Evaluation results are listed in Table 16, where only compare to "GraphRAG Local", it generally performs better than "GraphRAG Global" on these tasks.
For the aforementioned reasons, we also use GPT-4 to evaluate all experimental results, reporting the accuracy (Acc) in Table 17. When comparing the results Table 16 and Table 17, we observe that the order the results preserved, even though some metrics change significantly. In the following section, we aim identify the reasons behind these changes,
25

---

From Complex to Atomic: Enhancing Augmented Generation
which may provide valuable insights for designing better metrics to evaluate RAG frameworks in the future.
1. The accuracy of our approach increases significantly for generation tasks (1-1,3-2 Open Australian Legal QA). For these tasks, our answers are often semantically equivalent but syntactically different from the golden answers. This explains the improved metric performance, as GPT-4 can compare the semantic content of the answers. This also applies to the "GraphRAG Local" results for the "Open Australian Legal QA" task.
2.
The accuracy of "GraphRAG Local" decreases for generation tasks 1-1 and 3-2. These tasks involve statute recitation and prediction, requiring the retrieval of specific articles. Upon detailed examination, We find that "GraphRAG Local" often fails retrieve the correct articles or references the wrong ones, but it tends to repeat the legal information. Therefore, token-level recall can be improved by simply rephrasing legal names and common prefixes, such as "According to XX law, XX articles..."
3. Both our approach and "GraphRAG Local" show significant accuracy improvements on task 3-8. Besides the reason mentioned in the first point, the quality of the golden answers may also contribute to this difference. The questions and golden answers in task 3-8 are sourced from a consulting website, resulting in varying quality, For example, one question asks "Do the children from the original marriage have obligation to support the father?" However, the provided golden answer includes an irrelevant article, "Article 1067," which pertains to parents' obligations to support minor children.
Question: In the case where both parents divorced and have formed their
+ families with new children, and according to the court's judgment, the father is
4 required to pay monthly child support to the mother until the child is 18 years
4 old. Do the children from the original marriage have an obligation to support
the father?
Reference Answer: In our country, biological children have an obligatien
support + their parents who are divorced. The relationship between children and parents
+ does not dissolve because of the divorce of the children or parents. Supporting
4 parents is a legal obligation of the children. If the children do not support
+ the elderly, the parents can directly file lawsuit in the People's Court
requesting the court to order the daughter to pay support. Legal basis: Article
+ 1067 of the Civil Code of the People's Republic of China states that if parents 4 do not fulfill their obligation to support, minor children or adult children who
4 cannot live independently have the right to request support from their parents.
4 If adult children do not fulfill their obligation to support, parents who are incapacitated or have difficulties in living have the right to request support + from adult children. Article 1084 states that the relationship between parents
4 and children does not dissolve because of the parents' divorce. After the
+ divorce, whether the children are di directly raised by the father or the mother,
they are still the children both parents. lawyer's explanation: After the parents are divorced, adult children still have the obligation to support both
+ parents. The obligation of adult children to support their parents does not
+ change because of changes in the parents' relationship. If adult children do not
4 fulfill their obligation to support, parents who are incapacitated or have
+ difficulties in living have the right to request support from adult children,
Our Answer: Yes, there is an obligation to support. Legal basis: According to Article
+ 1069 of the Civil Code of the People's Republic of China, it is stipulated that:
4 'The obligation of children to support their parents does not terminate because
4 of changes in the parents' marital relationship." Therefore, even if the
4 parents are divoreed and remarried with new chilaren, the original children
4 still have an obligation to support their parents.
4. The accuracy of all methods on choice tasks 1-2, 3-1, and 3-6 almost coincides with the F1 score, expected. An exception is task 3-1, where the difference is mainly due to GPT-4's capacity to understand Chinese, particularly in distinguishing numbers in Arabic and Chinese. In Chinese law, all numbers are written in Chinese, while in the golden answers, all numbers are given in Arabic.
26


Reviews:
[{"review_id": "355rPQKH2M", "reviewer": "Reviewer_9CDV", "summary": "The authors present a new RAG framework suitable for addressing complex questions with a focus on multi-hop. The main idea is based on an iterative process of collecting evidence and generating follow-up questions as required. To devise this iterative process, the authors describe four main components: (1) Knowledge Atomizer: mapping doc chunks to set of answerable questions, (2) Query Proposer: mapping user question and intermediate context to reformulated question candidates, (3) Atomic Retriever: a high-recall lightweight ranking of atomic questions and their respective docs, and (4) Atomic Selector: LLM-based selector of top atomic question and its respective doc to be added to context.\n\nAuthors test their method on several datasets corresponding to specialized domains, and compare it against several comparable yet differentiated benchmarks. The authors demonstrate significant enhancement across benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback from the reviewer. Below, we address each concern point by point.\n\n### *Q1.* How KAR-RAG system would fare on questions that require parallel subtasks. \n\nKAR³ is specifically designed to handle complex questions by decomposing them into multiple subqueries, enabling effective retrieval and iterative reasoning. This decomposition mechanism allows KAR³ to address both sequential and parallel subtasks. For instance, consider the parallel comparison question provided in Figure 5 of the Appendix (page 22): \"Which film came out first, What Women Love or Ramudu Kadu Krishnudu?\" KAR³ decomposes this question into atomic subqueries:\n\n    (a) \"What is the release date of What Women Love?\"\n    (b) \"What is the release date of Ramudu Kadu Krishnudu?\"\n\nIn the first iteration, KAR³ retrieves a chunk tagged with the atomic question \"In what year was the film 'What Women Love' released?\" relevant to subquery (a) and add it to the context. In the second iteration, using the updated context and the original question, the system regenerates subquery (b) and retrieves the relevant chunk tagged with the atomic question \"In what year was the film 'Ramudu Kadu Krishnudu' released?\" Through this iterative decomposition and retrieval, KAR³ resolves parallel subtasks.\n\n### *Q2.* What is the stopping criteria for the system when no high quality proposed questions (or possible answers) are available?\n\nWhen no high-quality proposed queries or no relevant atomic questions, the atomic selector may return an empty or out-of-range atomic question index after evaluating the provided context, which consists of the retrieved atomic questions. This means that no additional relevant chunks can meaningfully contribute to answering the original question. In such cases, the decomposition loop terminates, and the system generates a final answer based on the information already accumulated in the context.\n\n### *Q3.* The paper can use more analysis of results and anecdotes to drive home the advantage of the system presented over other baselines.\n\nThank you for the constructive suggestion. We agree that incorporating additional analysis and case studies will help to emphasize the advantages of our method over the baselines. In the revised version, we will include more baseline analysis and case studies to highlight the strengths of our method.\n\n### *Q4.* The claim on LLMs' unawareness to technical terminologies may be outdated given newer evidence and missing a key RAG application which is equipping LLMs with knowledge crossing its cutoff date.\n    \nWe appreciate the insightful feedback and the opportunity to clarify and strengthen our claims. \n\n**Potential outdated nature of our claim:** While we acknowledge that recent advancements have improved LLMs' performance in specialized fields, challenges persist in areas requiring precise understanding of technical terminologies, particularly in dynamic domains with evolving jargon. For example, in OLED-related technologies, the term *CSE* is often misunderstood by LLMs as \"Charge Spread Effect\" or \"Charge Sheet Effect\" when it actually refers to \"Channel-Shortening Effect\". Such examples highlight the ongoing limitations of LLMs in accurately handling domain-specific acronyms and terminology, especially when context-specific disambiguation is required. We will refine our claim to acknowledge progress in LLMs while addressing areas where challenges remain.\n\n**RAG’s role in addressing knowledge cutoff issues:** We agree that one of RAG’s critical advantages is mitigating knowledge cutoff limitations by retrieving up-to-date information. We regret the omission of this important point and will explicitly highlight it in the revised manuscript as an important benefit of RAG systems.\n\n### *Q5.* Differentiating between atomic questions from the original question and the atomic questions from the knowledge base.\n\nThanks for your valuable suggestions. **Atomic query proposals** decomposed from the original query are derived to break down the original query into subqueries that aid in addressing the query. In contrast, **atomic questions** generated from chunks (knowledge base) are the questions that are relevant and can be answered by the given chunk. In the revised manuscript, we will replace the term \"atomic questions\" with \"atomic tags\" to clearly distinguish it from \"atomic query proposals\" and include illustrative examples.\n\n### *Q6.* Suggestions: a) reducing unnecessary jargon, b) early use of running examples, c) properly organizing supplementary materials.\n\nThanks for your valuable suggestions. In the revised version, we will a) simplify technical language by reducing unnecessary jargon and using clear, concise terminology, b) introduce running examples early on to illustrate key concepts step-by-step, making the explanations more accessible and easier to follow, c) carefully review and reorganize the supplementary materials to address misalignments in tables and figures.", "claims_and_evidence": "All claims in the paper are well supported, except for the following one:\n\nThe paper claims that LLMs struggle with specialized fields and links RAG systems as a solution to address this shortcoming, attributing this shortcoming to reasons such as unawareness to technical terminologies. Although the claims were supported by references, such a claim might be outdated with contrasting evidence such as: https://www.frontiersin.org/journals/oncology/articles/10.3389/fonc.2023.1219326/full . The other issue with this claim is missing a main application of RAG which is equipping LLMs with knowledge crossing its cutoff date.\n\nAdditionally, the authors motivated this work on specialized fields but later demonstrated the system on pop culture questions which reflected a disconnect between the original claim/motivation and the later support. With all that being said, the demonstrated enhancement on specialized datasets closes the gap.", "methods_and_evaluation_criteria": "Yes they do. Benchmarks and baselines are well represented.", "theoretical_claims": "The paper does not make any theoretical claims but is rather a systems paper.", "experimental_designs_or_analyses": "No issues found with experimental design.", "relation_to_broader_scientific_literature": "This work establishes the connection to the broader scientific literature mainly through comparing against similar, yet differentiated, RAG setups, most notably: Self-Ask and IterRetGen. KAR-RAG mainly distinguishes itself by using context to generate follow-up questions and by representing the KB as appended by atomic questions for easier atomic retrieval. The paper also demonstrates enhanced performance over said baselines.", "essential_references_not_discussed": "None that I can think of.", "other_strengths_and_weaknesses": "Strengths:\n- comparison against a large set of representative baselines and across a diversified benchmark.\n- enhanced performance on benchmarks.\n- ablation study showing benefit of each component.\n- representing document chunks as a set of answerable questions, simplifying retrieval\n- helpful supplementary materials\n\nWeaknesses:\n- The paper can use more analysis of the results and anecdotes to drive home the advantage of the system presented over other baselines.\n- The paper is harder to read than necessary. It can be simplified by reducing unnecessary jargon and the use of running examples.", "other_comments_or_suggestions": "- Authors can do a better job differentiating between atomic questions coming from the original question, and the atomic questions coming from the knowledge base. It got confusing in the doc.\n\n- I highly recommend the authors to use a running example earlier in the doc. The paper starts off with crowded jargon that is unexplained. Having a running example to explain steps along the way would enhance the paper. The authors did provide examples later in the doc but I found them to have arrived a bit too late.", "questions_for_authors": "- the paper addresses and demonstrates KAR-RAG on complex questions that require iterative subtasks, but it is not clear how the system would fare on questions that require parallel subtasks. In other words, the question can be divided to atomic subquestions that are not dependent on each other. I would like to hear the authors' thoughts on how the KAR-RAG system would fare in such examples.\n\n- Although most of the pseudo algorithm is clear, it is unclear what is the stopping criteria for the system when no high quality proposed questions (or possible answers) are available. In other words, it is clear how the system would stop when hitting N, or when the system would deem available context sufficient, but it is unclear how it would deem the set of proposed question insufficient to find proper evidence/context.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H6pxIiUqF5", "reviewer": "Reviewer_a9wr", "summary": "This paper proposes a framework for handling multi-hop questions which require complex reasoning which has 4 main components; an atomizer which generates atomic questions from document chunks, a query processor which iteratively generates atomic queries using the input question and current context, an atomic retriever which maps the atomic queries with the atomic questions generated by the atomizer and hence also mapping the atomic questions with a document context and finally an atomic selector which selects the atomic question-query-chunk triplet to either append to the context or respond the the user question. The authors have evaluated their proposed methods against six existing methods and have showcased 20.4% improvements over the second best method. They have also discussed the limitation of their method which is mainly the reliance on the reasoning ability of the underlying LLM in use. They have also conducted ablation study to understand the impact of the individual components in their pipeline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback and insightful comments from the reviewer. Below, we address the raised concerns point by point.\n\n### *Q1.* Discussion on the important limitation of the method that it is a resource heavy method and can not be used practically when retrieving from dynamic data sources or large volume datasources such as the web.\n\nWhile resource demands are an important consideration, our approach incorporates several key features and optimizations that mitigate these challenges and enhance practicality.\n\n**Dynamic Integration of Data Sources:** \nOur method supports dynamic addition of data sources without requiring structural modifications. This contrasts with knowledge graph-based extraction methods, which necessitate updating the graph when expanding the corpus. This flexibility ensures seamless integration of new information. \n\n**Scalable Preprocessing Costs:** \nThe construction of our knowledge base involves a one-time preprocessing cost that scales linearly with the size of the corpus. This scalability ensures that our approach remains efficient and suitable for processing large-scale datasets. The detailed preprocessing cost of three benchmarks are provided in Table 12 in the Appendix (page 18).\n\n**Optimizations for Large-Scale Data:**\nWe acknowledge the need to further optimize preprocessing for extremely large datasets, such as web-scale data, where resource efficiency becomes critical. To address this, we propose alternative atomization strategies to reduce preprocessing costs while maintaining competitive performance.\n\n1. *Using Open-Source Models:* We explore replacing computationally expensive LLMs like GPT-4 with more resource-efficient open-source models such as LLaMA 3 during the chunk atomization step. In additional experiments, this substitution significantly reduced preprocessing costs, with only a minor accuracy drop (~3% on MuSiQue dataset), as shown in Table 1. \n    \n2. *Sentence-Level Segmentation:* for scenarios requiring even lower-cost preprocessing, we propose using sentence-level segmentation as atomic units for retrieval. Although this approach reduces performance (55.2% on MuSiQue), it still outperforms the majority of baselines presented in Table 1 of the main paper, demonstrating its practicality in resource-constrained settings.\n\nTable 1: Ablation study on the preprocessing method on MuSiQue\n\n|LLM Used|Variant|F1|Acc|\n|-|-|-|-|\n|Llama 3|KAR³ w/ plain-text|45.88|54.20|\n|Llama 3|**KAR³**|**50.68**|**59.70**|\n|GPT-4|KAR³ w/ plain-text|50.72|55.20|\n|GPT-4|**KAR³**|**57.86**|**62.60**|\n\n### *Q2.* Thoughts on the trade-off between resource usage and efficiency, and how to effectively weigh the advantages?\n\nThe trade-off between resource usage and efficiency is an important consideration for our method, and we have designed it to balance these aspects effectively while offering flexibility based on specific use cases. To this end, we propose alternative atomization strategies that allow users to tailor preprocessing costs to meet their needs. \n\nFor example, substituting computationally intensive models like GPT-4 with lighter open-source models such as LLaMA 3 significantly reduces preprocessing costs, with only a minor performance drop (~3% on MuSiQue). Additionally, for resource-constrained scenarios, sentence-level segmentation can be employed, reducing preprocessing overhead while still outperforming most baselines (55.2% on MuSiQue). \n\nThe choice of strategy depends on the specific application: high-accuracy requirements may justify the use of more powerful models, while dynamic or resource-limited settings can benefit from cost-effective alternatives. These considerations ensure that our method remains adaptable and effective across a range of scenarios, balancing resource usage and performance as needed. The detailed experiment results are provided in Table 2 of main paper and Table 1 in this response. \n\nIn the revised manuscript, we will incorporate a detailed discussion on the trade-off between resource consumption and efficiency, accompanied by experimental results of alternative preprocessing methods.\n\n### *Q3.* Evaluation with diverse set of tasks and domain should be done.\n\nThanks for the constructive suggestion. We agree that evaluating our method on a broader set of tasks and domains is crucial to demonstrating its robustness and generalizability. To address this, we have included evaluations on two legal benchmarks in Appendix A.5，which further validate the effectiveness of our approach. \n\nFurthermore, we have applied our method to develop RAG systems in specialized domains, such as manufacturing and healthcare, achieving consistent accuracy improvements of over 15% across a variety of tasks. Due to privacy restrictions, we are unable to release the data from these domains. To further validate and benchmark our method, we are actively exploring publicly available datasets in specialized domains to conduct more rigorous evaluations.", "claims_and_evidence": "Yes. The authors have successfully showcased that their method is effective in multi-hop question answering systems, and their method is especially effective when domain awareness is required for domain specific complex question answering. For effectiveness in multi-hop question answering they have demonstrated that the proposed method is better compared with existing methods such as standard RAG, Zero shot COT based step by step question answering, Self-Ask w/ Retrieval etc. However, for domain specific complex question answering they evaluated their method on just two datasets from a single domain where clearly the method has an advantage due to its inherent nature and how the task of the datasets are designed. Thus, a more holistic evaluation with diverse set of tasks and domain should be done.", "methods_and_evaluation_criteria": "Yes, The method discussed in the paper are mainly around multi-hop question answering. The evaluation datasets and metrics used for experimentation are aligned.", "theoretical_claims": "Yes, I verified the results shared in the paper and those are accurate.", "experimental_designs_or_analyses": "Yes, The experimental design of the paper is sound with respect of multi-hop question answering evaluation.", "relation_to_broader_scientific_literature": "The paper discusses how iterative context building and query to knowledge mapping improves the response generation of the RAG system. This is aligned with some of the other papers such as", "essential_references_not_discussed": "Nothing that I am aware of.", "other_strengths_and_weaknesses": "Strength: The paper highlights how static query to document chunk mapping followed by dynamic query to context and user question mapping improves the response generation in a multi-hop question answering scenario. The novelty is explained in a clear manner with clear evaluation datasets and methods. \n\nWeakness: The paper does not do a good job in terms of providing evidence for the claim that the method is effective for cases where domain knowledge is required for complex problem solving. The paper also does not discuss an important limitation of the method that it is a resource heavy method and can not be used practically when retrieving from dynamic datasources or large volume datasources such as the web.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "This method demands a lot of resources. Do you have any thoughts on the trade-off between resource usage and efficiency, and how to effectively weigh the advantages?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dmBc0TMWTD", "reviewer": "Reviewer_TVW1", "summary": "This paper addresses the challenge of solving complex, multi-hop queries in domain-specific contexts by introducing a method called KAR^3-RAG. Traditional Retrieval-Augmented Generation (RAG) techniques often rely on straightforward text retrieval methods, which can struggle when queries require multiple steps or hops to reach an answer.\n\nThe key idea behind KAR^3-RAG is to reorganize or “atomize” the knowledge base (KB) into smaller units of knowledge in the form of atomic question–answer pairs. From a query standpoint, the method then uses:\n\n1. A knowledge atomizer to decompose the large KB into these atomic QA pairs.\n2. A query proposer to break down the original complex question into more granular sub-queries (or atomic queries).\n3. An atomic retriever to efficiently retrieve relevant QA pairs.\n4. An atomic selector that uses these retrieved QAs to generate either an answer or a next-step question.\n\nEmpirical results suggest that KAR^3-RAG outperforms various baselines, including on legal question-answering benchmarks and general multi-hop QA tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer's feedback. All tables are accessible via hyperlinks. \n\n### *Q1-1.* How large was the knowledge base in the experiments?\n[Table 1](https://tinyurl.com/R3-tables) provides detailed statistics. All chunks are derived from the context paragraphs of the sampled QA, with chunk count varying by dataset. Additionally, the count info is included in Appendix A.1.\n\nTable 1: Chunk Statistics\n\n|Dataset|Avg. Len|Count|\n|-|-|-|\n|Hotpot|546|4950|\n|2Wiki|422|3410|\n|MuSiQue|484|7120|\n\n### *Q1-2.* How computationally intensive was the process of generating atomic questions?\n[Table 2](https://tinyurl.com/R3-tables) shows the computational cost. Per Appendix A.3, atomization is a one-time step requiring LLM calls equal to chunk count, with cost scaling linearly based on chunk sizes and generated questions.\n\nTable 2: Preprocessing Token Consumption\n\n|Dataset|Avg. Tokens|Calls|\n|-|-|-|\n|Hotpot|338|4950|\n|2Wiki|321|3410|\n|MuSiQue|320|7120|\n\n### *Q1-3.* Discussion of scalability and simpler preprocessing method.\nOur approach mitigates scalability for larger datasets through three key features:\n- Dynamic addition of atomic questions without structural changes\n- Linear preprocessing cost scaling with corpus size\n- Compatible with open-source LLMs (Table 2 in paper), reducing cost with only 3% performance drop on MuSiQue\n\nFollowing the valuable suggestion to **employ simpler preprocessing steps**, we tested using plain-text sentence splitting via *spacy* instead of LLM-based atomization. Each sentence serves as an atomic question. After revising the selection prompt (Appendix line 1375-1396), results in [Table 3](https://tinyurl.com/R3-tables) show that while performance drops 7% on MuSiQue, this method still outperforms most baselines from Tables 1 and 2 in paper. This demonstrates its effectiveness in case lower-cost preprocessing needed, offering a flexible performance-efficiency tradeoff alongside \"dynamic addition\".\n### *Q2.* Clarify whether baselines employed preprocessing or KB re-structuring? How might that affect the performance？\nMost baselines use standard retrieval methods without preprocessing (details in Appendix Table 5, page 14). Only Self-Ask explicitly generates sub-questions that can query atomic questions using a similar retrieval path (sub-question -> atomic question -> chunk).\n\nTesting atomic questions with Self-Ask, IRCoT, and Iter-RetGen shows slight performance improve for Self-Ask (1.6%) due to its natural retrieval path, but decreased performance for IRCoT and Iter-RetGen (see [Table 4](https://tinyurl.com/R3-tables)). This demonstrates that atomization alone cannot contribute significantly to performance.\n\nTable 4: Abalation study of baselines with atomic questions\n\n|Variant|Hotpot Acc|2Wiki Acc|MuSiQue Acc|\n|-|-|-|-|\n|Self-Ask w/ Atomic Question|80.00|77.60|53.00|\n|IRCoT w/ Atomic Question|77.80|65.20|47.20|\n|Iter-RetGen w/ Atomic Question|82.20|63.60|46.80|\n|**KAR³**|**88.00**|**82.20**|**62.60**|\n### *Q3.* Awareness of tree-based or partial KB re-structuring methods ([Sarthi et al., 24]). How might those compare to your approach？\nWe compared KAR³ with other KB-restructuring methods like GraphRAG and RAPTOR (Tables 6, 14, 15 in appendix, and [Table 5](https://tinyurl.com/R3-tables)). Both underperform in F1 scores, likely due to their focus on summarization which can introduce redundancy. KAR³'s precise chunk atomization enables more accurate retrieval and reasoning, leading to better performance on complex questions in 2Wiki and MuSiQue.\n\nAs for complexity, the preprocessing cost of both KAR³ and RAPTOR scales linearly with chunk size, while GraphRAG incurs significantly higher cost due to its hierarchical KG construction. This highlights that KAR³ achieves outstanding performance with reasonable computational cost compared to RAPTOR and GraphRAG.\n\nTable 5: Comparison of KB re-structuring methods\n\n|Method|Hotpot F1|Hotpot Acc|2Wiki F1|2Wiki Acc|MuSiQue F1|MuSiQue Acc|\n|-|-|-|-|-|-|-|\n|RAPTOR|12.46|81.40|10.03|69.80|6.86|55.00|\n|GraphRAG|10.66|**89.00**|11.83|71.20|9.62|49.80|\n|**KAR³**|**76.48**|88.00|**75.00**|**82.20**|**57.86**|**62.60**|\n\n### *Q4.* How can we ensure the quality of knowledge atomizer？\nKAR³ reduces dependence on high-quality atomization in two ways:\n- *Atomic Questions Mainly as Multi-aspect Indexing*: Generates multiple atomic questions from different perspectives, providing relatively comprehensive coverage without requiring perfect atomization.\n- *Separation of Retrieval and Reasoning*: Uses atomic questions only for retrieval, with downstream LLMs handling reasoning, making the system robust to imperfect retrieval.\n\nIn Table 3 of paper, using Llama3 instead of GPT-4 for all components only reduces MuSiQue performance by 2.9%, demonstrating KAR³'s robustness to atomizer quality. While better atomization would improve performance, KAR³ can achieve this incrementally since it naturally supports adding new atomic questions to existing databases on the fly.", "claims_and_evidence": "The authors claim that KAR^3-RAG:\n\n1. Improves multi-hop retrieval performance by decomposing the KB into atomic QA pairs.\n2. Outperforms competing retrieval-based methods on both general and domain-specific QA tasks.\n\nThese claims are generally supported by the experimental results reported. That said, the methodology for constructing the atomic QA pairs (the knowledge atomization process) provides KAR^3-RAG with an advantage that other methods may not share, so it would be helpful to see a discussion on this additional effort and how it compares to simpler or alternative preprocessing steps.", "methods_and_evaluation_criteria": "The proposed method makes sense for multi-hop retrieval because it directly tackles the need for stepwise reasoning by structuring the KB itself into smaller, more retrievable units. The evaluation spans both general multi-hop QA tasks and domain-specific (legal) QA, which aligns well with the claim that the approach is general yet especially useful for specialized domains.\n\nHowever, additional clarity on how much preprocessing effort is involved—and to what extent alternative approaches could achieve similar results with less overhead—would be beneficial. The evaluation is done via standard QA metrics, which is appropriate for comparing question-answering performance.", "theoretical_claims": "There is no theoretical claim in this paper.", "experimental_designs_or_analyses": "From the experiments, the methodology seems sound: the paper compares KAR^3-RAG against strong retrieval baselines on multi-hop QA datasets and domain-specific data (e.g., legal documents). The results show improvements in retrieval accuracy and final QA accuracy.\n\nHowever, one concern is the fairness of the comparison. Because KAR^3-RAG performs a knowledge atomization step, it is not entirely clear whether the other baselines had an equivalent chance to restructure their knowledge bases or if they simply used plain text as-is. The paper might benefit from more discussion on how to ensure each method has a similar knowledge “preparation” or from a deeper ablation study on the effect of atomization.", "relation_to_broader_scientific_literature": "The paper contributes to the ongoing research in multi-hop retrieval and question answering by introducing the idea of an atomic knowledge base—turning all or most of the raw text segments into smaller question–answer pairs that can be more directly retrieved. This idea builds on previous multi-hop QA work but takes it further by making the entire knowledge base “query-friendly.”\n\nThis approach relates to existing multi-hop retrieval frameworks that attempt to break down complex questions into sub-questions. However, instead of focusing purely on query decomposition, KAR^3-RAG also restructures the knowledge base itself. This is a novel angle worth discussing in comparison with other knowledge-base transformation methods.", "essential_references_not_discussed": "One relevant line of work is the transformation or rearrangement of knowledge bases for tree-based or structured retrieval. For example, there is a method described in:\n\n[Sarthi et al., 24] “RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval.”\n\nAlthough not widespread yet, this work (or similar) might provide a useful point of comparison. It would be good to see how KAR^3-RAG stands relative to these approaches in terms of complexity, performance, and scalability.", "other_strengths_and_weaknesses": "**Strengths**\n\n- Novel restructuring of the knowledge base into atomic QA pairs, which helps address the complexity of multi-hop queries.\n- Demonstrates strong improvements in both general and specialized (legal) QA tasks, which suggests broad applicability.\n- Brings a fresh perspective to retrieval by rethinking how knowledge is stored and accessed.\n\n**Weaknesses**\n\n- The cost and scalability of knowledge atomization are not explored in depth, leaving uncertainty about applying this method to large-scale knowledge resources (e.g., Wikipedia).\n- It is not fully clear how to compare the results fairly, as the proposed approach benefits from pre-processing the KB, while baselines typically rely on unstructured plain text.\n- Comparison to other potentially simpler or less costly knowledge-base transformation approaches is missing (e.g., [Sarthi et al., 24]).\n\n[Sarthi et al., 24] RAPTOR: RECURSIVE ABSTRACTIVE PROCESSING FOR TREE-ORGANIZED RETRIEVAL", "other_comments_or_suggestions": "- Including more details on the size of the knowledge base used in the experiments (e.g., line 371) would help contextualize the results.\n- A discussion on the computational cost and feasibility of creating atomic QA pairs for large-scale knowledge sources would be valuable.\n- It might be instructive to conduct an ablation or pilot study on partial or dynamic atomization to see how much the approach relies on the full-blown transformation.", "questions_for_authors": "- How large was the knowledge base in your experiments, and how computationally intensive was the process of generating atomic QA pairs? Do you foresee your approach being scalable to something the size of Wikipedia or large internal corporate/legal databases?\n- Could you clarify whether baselines also employed any form of preprocessing or KB re-structuring? If not, how might that affect the reported performance differences?\n- Are you aware of tree-based or partial KB re-structuring methods (e.g., [Sarthi et al., 24]) and how might those compare to your full-scale atomic approach, particularly in terms of complexity and performance?\n- How can we ensure the quality of knowledge atomizer? Considering its importance, it is highly crucial to perform the high-quality atomization.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jjC0ksrDJD", "reviewer": "Reviewer_fmKx", "summary": "The authors propose a knowledge aware rewriting and reasoning framework, a variant of retrieval augmented reneration (RAG), which is suitable for multi hop question answering tasks since it can aggregate knowledge from different documents. It consists of 4 steps: knowledge atomizer, query proposer, atomic retriever and atomic selector. During knowledge atomizing, a chunk is given to a LLM and asked to produce as many questions as possible whose answer is contained in the given chunk -- this will be included in the knowledge base together with the original chunk. In the next step, a LLM is prompted to break down the original question is broken down into subquestions (atomic query proposals) and if there are any chunks/question pairs from previous step in the knowledge base, these are passed in the context. The next step is the atomic retriever, which uses traditional similarity search with cosine similarity to retrieve top-k atomic questions/chunks that are most similar to the query proposals. Finally, a LLM is again used to select which question/chunk is most useful for the atomic query proposal. The experiments section showcase that their method is competitive with respect to other baselines ( RAG, zero shot CoT, Self-Ask, IRCoT and ProbTree) wrt F1 and using a LLM evaluator.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback and insightful comments from the reviewer. Below, we address each concern point by point. All tables are accessible via [hyperlinks](https://tinyurl.com/R4-tables). \n\n### *Q1.* Embedding model used for retrieval.\n\nThe text-embedding-ada-002 model is used across all experiments demonstrated in the paper. For more hyper-parameter settings, you can refer to Appendix A.2 (line 654 ~ 668).\n\n### *Q2.* Chunk size and number of chunks retrieved.\n\nAs introduced in line 370 ~ 373 in the paper, we compile the context paragraphs without additional chunking, resulting in a chunk size of around 500 chars. The detailed statistics are listed in [Table 1](https://tinyurl.com/R4-tables). In retrieval phase, the retriever is configured to retrieve 4 atomic questions per atomic query with a relevance threshold of 0.5. However, the actual number of chunks retrieved varies based on the number of queries proposed and their associated relevance scores. Additionally, since there is an atomic selection step incorporated in each decomposition round, a maximum of one chunk may remain after each round. Given decomposition loop constrained up to 5 rounds (N = 5) in our main experimental setup, no more than 5 chunks will be utilized during final answer generation phase.\n\n### *Q3.* Which work is the RAG approach (mentioned in L65 of page 2) that use question decomposition without considering available knowledge?\n\nThanks the reviewer for pointing it. One such example is Self-Ask ([Press et al., 2023]). We will clarify this in the revised version and explicitly refer to Self-Ask to ensure the statement is supported and precise.\n\n### *Q4.* Paper do not include other RAG techniques where the knowledge augmentation happens at training or fine-tuning but only at inference time.\n\nThanks for the suggestion regarding related work. In the revised version, we will include the key works that incorporate knowledge augmentation during training or fine-tuning, such as *REALM: Retrieval-Augmented Language Model Pre-Training ([Guu et al., 2020])*, and *LLaMA-Adapter: Efficient Fine-Tuning of LLaMA for RAG ([Zhang et al., 2023])*, among others.\n\n### *Q5.* Rename the metric Acc to LLM evaluator or similar.\n\nThanks for the suggestion! We understand how the term \"Acc\" might cause confusion. In the revised version, we will rename the metric to something more descriptive, such as \"LLM Evaluator\", to better reflect its purpose and avoid overloading the term.\n\n### *Q6.* Use a LLM with a context window sufficiently long to include the increasing context.\n\nTo explore the dependency of our approach on the context window size, we analyzed the token distribution on MuSiQue (using GPT-4), considering settings where N = 5 and N = 10 (N represents the decomposition round limit, detailed in line 2 of Algorithm 1). The findings are presented in [Table 2 and 3](https://tinyurl.com/R4-tables). Table 2 shows that the maximum number of input tokens is around 6K, while the maximum output tokens is around 0.5K across all LLM interactions. Additionally, this maxmium token consumption shows only a slight increase as N increases but remains within the same order of magnitude. This suggests that the maximun token requirement grows gradually. Futhermore, Table 3 demonstrates that over 99% of the LLM calls are accommodated by models with a token limit of 4096, a capacity commonly supported by existing LLMs. For models providing context windows of at least 8K tokens, all LLM calls can be handled without the need for any token truncation.\n\nTable 3: Prompt Token (per LLM call) Distribution of KAR³ on MuSiQue.\n\n|Prompt Token|Percentage of Calls (N = 5)|Percentage of Calls (N = 10)|\n|-|-|-|\n|<= 512|49.38%|34.11%|\n|<= 1024|91.87%|77.61%|\n|<= 2048|97.16%|97.10%|\n|<= 4096|99.78%|99.89%|\n|<= 8196|100.00%|100.00%|", "claims_and_evidence": "The claims made in the submission supported by clear and convincing evidence", "methods_and_evaluation_criteria": "The authors showcase their method's performance using three different multi hop QA datasets : hotpot QA, mystique and 2wiki. These benchmarks are relevant to assess performance.", "theoretical_claims": "There are no proofs in the paper and the claims are well supported.", "experimental_designs_or_analyses": "I checked and the experimental design and analysis are solid. The only detail I couldn't find is how many tokens per chunk are included or whether the chunks are at sentence or paragraph level.", "relation_to_broader_scientific_literature": "The contributions from the paper are related to the broader scientific literature for improving vanilla retrieval augmented generation when the knowledge source is included in more than one chunk.", "essential_references_not_discussed": "The authors do a good job including the relevant references to other works in this research area. Since retrieval augmented generation is a very active area of research, I would mention in the paper that the works included do not include other RAG techniques where the knowledge augmentation happens at training or fine-tuning but only at inference time.", "other_strengths_and_weaknesses": "The paper is well written and easy to follow with respect to all the algorithmic details. I would like to know which embedding model they used for the retriever step (or if they used BM25). I would also like to know the chunk size and number of chunks retrieved.", "other_comments_or_suggestions": "- It would be interesting to see an ablation of how the chunk size and number of chunks affects performance of the method.\n\n- I would rename the metric Acc to LLM evaluator or similar since Acc typically refers to another metric so it's confusing to overload the term.\n\n- I would ask the authors to flesh out the other requirement which is to have a LLM with a context window sufficiently long to include the - increasing context they provide.\n\n- When you reference Gao et al. 2023, since it's a survey, I would explicitly say \"and references therein\"", "questions_for_authors": "- Which embedding model they used for the retriever step (or did you use BM25)? \n\n- What is the chunk size and number of chunks retrieved? \n\n- You mentioned that there are RAG approaches that use question decomposition without considering available knowledge in L65 of page 2, which work is this? Please include a reference", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose an advanced RAG-based system, Knowledge-Aware dual Rewriting and Reasoning (KAR^3)-RAG, for handling multi-hop questions requiring reasoning with a motivation of supporting highly-technical verticals (e.g., medicine, legal). The key innovation is to 'atomize' the corpus into a knowledge-base like structure consisting of question-answer pairs which are then incrementally retrieved and consolidated to put into the LLM context window for answer generation. Specifically, the key subcomponents include (see Figure 3):
- Knowledge Atomizer: decomposes the larger corpus/KB into atomic QA pairs
- Query Proposer: decomposes the original 'complex' question into 'atomic' (i.e., more granular) sub-queries, possibly considering previously retrieved context
- Atomic Retriever: retrieves atomic QA pairs given the atomic queries
- Atomic Selector: takes the retrieved QA pairs and determines if they should be added to the context
At this point, either additional 'knowledge-decomposition' can occur or the context can be packed into the LLM context window for answer generation.

Experiments are conducted on widely used multi-hop QA datasets and specialized legal datasets relative to multiple recent strong baselines, showing solid empirical improvements in most settings with GPT-4 and Llama3 as the base LLM. Finally a sensitivity analysis is performed to address the search depth threshold and an ablation study is performed to show the relative contribution of various subcomponents.

Strengths of this work identified by the reviewers include:
- The proposed method is well-motivated, conceptually appealing, and shown to work well on widely-studied multi-hop QA datasets against strong recent baseline methods. The results are ostensibly the state-of-the-art for these datasets.
- The proposed method is particularly applicable to domain-specific knowledge RAG cases.

Conversely, limitations of this work identified by the reviewers include:
- A primary RAG motivation for the proposed method was reasoning within specialized domains. However, the primary evaluation was on widely used datasets and there were no experiments that directly addressed this motivating aspect of the method. It seems that machinery to index data by generated questions accounts for most of the empirical gains while technical terms, etc. aren't a key aspect (or at least it isn't demonstrated).
- It isn't clear that the proposed approach would scale to web-sized data for time-sensitive queries, etc. due to the preprocessing needed for chunking and question generation.
- The paper introduces a lot of technical terminology early in the paper (repeatedly) that isn't defined until later in the paper and isn't really well understood until presented in the context of the figures. As reviewer 9CDV pointed out, the paper would benefit from an earlier figure that introduced a running example. In my opinion, the first 2-3 pages could be tightened up to make room for presenting a subset of the legal results and/or addressing specific 'reasoning' experiments to support motivating aspects of the work.
- While not mentioned by the reviewers, it seems that this 'chunked' corpus may not be as useful for other LLM tasks (including non-multi-hop QA). Thus, it isn't clear if this is a solution specific to multi-hop, but may not be practical for the more general setting.
- There were several detailed questions that were largely answered during rebuttal.

In my assessment, the key technical innovation is the machinery introduced to support chunking of the data and query generation for associated indexing -- which is shown to work well for multi-hop QA requiring RAG operations. The empirical results in this setting are strong. However, my primary concerns are: (1) while this chunking machinery is useful for short-context snippets common to multi-hop QA, it may not be amenable to longer-form settings which is more common in general LLM QA traffic and (2) the paper is motivated by reasoning in vertical domains (e.g., legal, medical), but these experiments are limited (in Appendix A.5 with many results and little discussion). Thus, it seems to represent the state-of-the-art for the target setting, but it isn't clear it is generally deployable (which was discussed to some degree in rebuttal, but not yet incorporated in the paper) nor really solves a primary motivating case in the domain-specific case. This would be a stronger paper if these aspects were more clearly addressed.
Decision: Accept (poster)