From Complex to Atomic: Enhancing Augmented Generation via Knowledge-Aware Dual Rewriting and Reasoning

Authors: Jinyu Wang, Jingjing Fu, Lei Song, Jiang Bian
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: rag, knowledge atomizing, knowledge-aware task decomposition, multihop qa

Abstract:
Recent advancements in Retrieval-Augmented Generation (RAG) systems have significantly enhanced the capabilities of large language models (LLMs) by incorporating external knowledge retrieval. However, the sole reliance on retrieval is often inadequate for mining deep, domain-specific knowledge and for performing logical reasoning from specialized datasets. To tackle these challenges, we present an approach, which is designed to extract, comprehend, and utilize domain knowledge while constructing a coherent rationale. At the heart of our approach lie four pivotal components: a knowledge atomizer that extracts atomic questions from raw data, a query proposer that generates subsequent questions to facilitate the original inquiry, an atomic retriever that locates knowledge based on atomic knowledge alignments, and an atomic selector that determines which follow-up questions to pose guided by the retrieved information. Through this approach, we implement a knowledge-aware task decomposition strategy that adeptly extracts multifaceted knowledge from segmented data and iteratively builds the rationale in alignment with the initial query and the acquired knowledge. We conduct comprehensive experiments to demonstrate the efficacy of our approach across various benchmarks, particularly those requiring multihop reasoning steps. The results indicate a significant enhancement in performance, up to 12.6\% over the second-best method, underscoring the potential of the approach in complex, knowledge-intensive applications.

Decision: reject
Reviewer scores: 6, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730608361841

thread_type:
official_review

<thread_object>
creator: reviewer_vtwe

<content>
summary:
This paper points out that the semantic retrieval is inadequate for mining deep, domain-specific knowledge for performing logical reasoning. The authors propose a knowledge-aware task decomposition strategy that extracts, comprehends, and utilizes atomic domain knowledge while constructing a coherent rationale.

strengths:
S1.  Supplementing semantic retrieval-based RAG with fine-grained and atomic knowledge is a novel and realistic practice.

S2. The proposed strategy is easy to understand and follow.

S3. The experimental results are explicitly competitive.

weaknesses:
W1. Regardless of the novel motivation, the explanation of the motivation is poor. The authors claim that ‘Current RAG methods relying on semantic retrieval fall short in deep domain-specific reasoning’. However, they do not specify the main challenges in domain-specific reasoning and why current RAG methods fall short in this setting. It will make this paper clearer and more convincing to clarify the above points.

W2. The proposed strategy includes using GPT4 to atomize raw chunks into sub-queries, using GPT4 to decompose the incoming queries, and using GPT4 to select optimal contexts and judge the termination. It’s more like a pipeline. Please point out the main technical challenges through this pipeline and the main technical contributions to solve the challenges.

W3. The proposed strategy relies highly on calling LLMs, which costs much. The dependence on the capability of GPT4 makes the proposed strategy inefficient and cost-intensive. Is there any cost analysis comparing the proposed strategy to baselines?

questions:
The proposed strategy is dependent on GPT-4, which is expensive. Do cheaper LLMs fit in the pipeline? Are there any experiments or analyses with less expensive models? How might the performance be expected to change with different LLMs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648608208

thread_type:
official_review

<thread_object>
creator: reviewer_1t8b

<content>
summary:
This paper introduces a novel Retrieval-Augmented Generation (RAG) framework called KAR-RAG, which enhances retrieval efficacy by incorporating a knowledge-aware task decomposition strategy. KAR-RAG decomposes complex queries into atomic questions, enabling more effective retrieval and logical reasoning from specialized datasets. The framework consists of four key components: a knowledge atomizer, query proposer, atomic retriever, and atomic selector, which collaboratively extract and utilize domain-specific knowledge to construct coherent rationales, achieving up to a 12.6% performance improvement over existing methods in multihop reasoning tasks.

strengths:
1. This paper propose to enhance the RAG system with atomized knowledge retrieving mechnism. In particular, this paper designs knowledge atomizer, query proposer, atomic retriever, and atomic selector to construct the pipeline.
2. Experiments on complex question answering datasets show that the proposed method is effective.
3. The presentation of this paper is clear and easy to follow.

weaknesses:
1. The effectiveness of each individual components should be further explored. For example, what if the knowledge is mingled together, rather than presented in atomic simple questions.
2. Lack of related works and baselines. This paper should survey and compare with more powerful RAG methods that elaborates on question decomposition, for example:
    - https://arxiv.org/pdf/2404.14464
    - https://arxiv.org/pdf/2406.19215

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689376176

thread_type:
official_review

<thread_object>
creator: reviewer_4rfd

<content>
summary:
This paper introduces KAR-RAG, a novel RAG framework that improves retrieval efficacy through knowledge rewriting. Specifically, for each document chunk, KAR-RAG generates a set of atomic questions to label the available knowledge contained in the chunk, and use these atomic questions as indices for document chunk retrieval.

strengths:
1. The paper is well-organized, and main components of the method (knowledge atomizer, query proposer, atomic retriever, atomic selector) are explained smoothly. 
2. The paper proposes a novel rewrite-and-index approach to tackle retrieval-augmented generation.
3. Experiments conducted on both open-domain and domain-specific datasets demonstrate that KAR-RAG achieves state-of-the-art results.

weaknesses:
1. The construction of KAR-RAG's atomic knowledge base can be highly cost-intensive, as generating atomic questions for each document chunk for a large text corpus leads to extensive LLM API consumption. The authors did not include a cost analysis to address this.
2. The paper’s motivation is a bit ambiguous. While the authors claim that the work focuses on enhancing “domain-specific knowledge mining” and “complex logical reasoning”, the paper lacks clear explanation on how KAR-RAG resolves these issues. Instead, it seems like KAR-RAG is a framework enhancing knowledge retrieval in general, showing no specific improvement over either “domain-specific knowledge mining” or “complex logical reasoning”. 
3. The paper does not compare more recent and well-performing RAG baselines such as SearChain, ProbTree, etc, which may yield better results than KAR-RAG.

questions:
When discussing the paper’s motivation, the authors could consider to address the challenge of accurate retrieval in general, instead of claiming to improve domain-specific knowledge retrieval (which may lead to misunderstandings on the paper’s core motivation).
Refered to Weaknesses session.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711703467

thread_type:
official_review

<thread_object>
creator: reviewer_qwux

<content>
summary:
This paper describes an approach to enhance and use knowledge in RAG setting to improve the overall of reasoning performance of LLMs. Paper proposes atomiser, a question generation for a given query/chunk. It uses the questions generated from the query to generate questions using LLM and use it in retriever to retrieve the context chunks. These are further used to generate any missing context by iteratively decomposing queries to gather mode suitable context to answer the given query and the final answer is generated using this process.

strengths:
Paper is easy to follow and authors have done good job in explaining the overall method with good examples. Method shows consistent improvements across all the chosen benchmarks which require reasoning. Another strong point I see is knowledge augmentation is happening with query decomposition and context gathering, rather than going through KG generation etc, which can be more involved and can introduce another place for injecting errors.

weaknesses:
Authors have no talked about the overhead this method introduces in terms of generating the final answer in terms of response time/ cost of using LLM in the whole process.

questions:
Do any of the existing query decomposition techniques be used as is without using LLM here ? For example for the of interest in example presented in Figure 2, main questions needed are who is the direct of the film and who is the mother or parent of person X . Can't this be done using other methods ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732436259169

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses and Common Questions (part 3)

comment:
### CQ3. Lack of specific explanation over domain-specific knowledge and complex logical reasoning upon them. (for 4rFD and vTwe)

In our work, domain-specific knowledge pertains to information extracted from the corpus designated for RAG, which may not be fully covered by the pre-trained LLMs and can sometimes diverge from LLMs' intrinsic knowledge. For instance, within a particular domain, "ice cream" might be used as a unique project name rather than referencing the typical frozen dessert. This domain-specific knowledge serves as external knowledge to assist LLMs in generating responses that are consistent with the knowledge from the RAG corpus. **Nevertheless, integrating domain-specific knowledge to domain-aligned reasoning and generation introduces significant challenges, which include the difficulties of a) effectively extracting domain-specific knowledge, b) precisely retrieving relevant domain-specific knowledge with queries, and c) successfully constructing coherent reasoning chain based on it. To tackle these challenges, we propose a novel approach encompassing knowledge atomization, atomic pair retrieval, and query decomposition with reference context.**

Specially, our method employ the reference context as known domain-specific knowledge to influence the subsequent query decomposition, thereby tailoring the reasoning path according to the available corpus. To illustrate this approach, Table 4 presents an example to aid reviewers in understanding this concept. In this example, *A* and *B* are individuals with common names, which may confuse LLMs and may lead them to generate the information from the other person with the same name. *C* represents a company name.

Table 4: Query Example for query decomposition and logical reasoning .

| **Query**         | "Who joined the company *C* first, *A* or *B*?" |
| ----------------- | ---------------- |
| **Supporting facts** | *S1*. "After graduating from University, *A* found his first job as a product manager in *C*."  |
|      | *S2*. "As the company underwent spin-off litigation, *A* experienced uncertainty about the company's future and carefully considered his career options. |
|      | *S3*. "After a period of corporate turmoil caused by the spin-off lawsuit, *B* was invited to join *C* as a strategic consultant to help navigate the challenges and capitalize on new opportunities." |

Referring to the **part one** of example analysis below, given a query, LLMs that rely on common knowledge tend to decompose it into two straightforward subqueries often overlooking the availability of corresponding answers in the RAG data. Although two related facts *S1* and *S3* are retrieved, the provided information is insufficient to construct a reasoning chain needed to answer the question.
Our method, illustrated in the **part two** of example analysis, incorporates the reference context from previously retrieved data to guide the subsequent query decomposition. This approach generates potential reasoning paths based on the domain-specific knowledge retrieved from RAG data, and further utilize subqueries to retrieve the relevant knowledge, ultimately arriving at the correct answer with a complete reasoning chain.

In the revised version, we will elucidate the definition of domain-specific knowledge and delineate the challenges encountered when engaging in domain-specific logical reasoning to more clearly convey our motivation and highlight our contribution.

**Example analysis**

**Part one: Decompose and Reasoning with *common knowledge***

*Decompose query without context*:

    1. "When does A join the company C?"
    2. "When does B join the company C?"

*Retrieve*: *S1* and *S3*

*Reasoning*: insufficient information to answer.


**Part two: Decompose and Reasoning with *domain-specific knowledge***

*Context retrieved from previous iterations*: *S1* and *S3*

*Decompose query with context*:

    1. "Did A join the company while C was undergoing spin-off proceedings?"
    2. "When did A graduate from university?"
    3. "When did C become involved in a spin-off litigation?"

*Retrieve*: *S2*

*Reasoning*: Based on *S1*, *S2*, and *S3*, *A* joined *C* before the spin-off lawsuit, while *B* joined *C* after the spin-off lawsuit. Therefore, the answer is *A*.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732436566333

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses and Common Questions (part 2)

comment:
### CQ2. Comparison with more baselines. (for reviewer 4rFD and 1t8b)

We appreciate the reviewers' suggestion to incorporate performance comparisons with four additional baseline methods. In response, we have conducted a comprehensive evaluation against three of these baselines: SearChain, ProbTree, and Tree of Reviews (ToR). As shown in Table 3, our method consistently outperforms these baselines.

It is important to note that the baseline, SeaKR, relies on self-aware uncertainty calculated from hidden state embeddings of LLMs. The official implementation of SeaKR necessitates modifications to the inference function to access these embeddings, a requirement that is incompatible with GPT-4. Consequently, we have not included SeaKR in our comparison table. For the other works with available code (SearChain and ProbTree), we conduct experiments using our sampled datasets while making every effort to adhere to their default settings. And the metrics reported in Table 3 were calculated using our evaluator, which normalizes answers before evaluation. In the case of ToR, which has no available code, we compare our results directly with the performance reported in their paper. In addition to F1 metrics shown in Table 3, ToR reports supporting fact recall rates of 73.8, 79.4, and 48.5 for HotpotQA, 2Wiki, and MuSiQue, respectively. In contrast, our method achieves significantly higher supporting fact recall rates of 92.83, 94.06, and 73.8 on these datasets, as detailed in Table 6 of our paper.

Table 3. Performance comparison on multi-hop QA datasets. Best in bold.

| Method                | HotpotQA F1     | HotpotQA Acc    | 2Wiki F1        | 2Wiki Acc       | MuSiQue F1      | MuSiQue Acc     |
| --------------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- |
| Zero-Shot CoT         | 43.94           | 53.60           | 41.40           | 43.87           | 22.90           | 23.47           |
| Naive RAG             | 72.67           | 82.60           | 59.74           | 62.80           | 43.31           | 44.40           |
| Self-Ask w/ Retrieval | 71.40           | 80.00           | _69.06_       | _75.00_       | 46.76           | 51.40           |
| IRCoT                 | 67.30           | 81.00           | 63.83           | 70.40           | 47.57           | 49.20           |
| Iter-RetGen           | _75.27_       | _86.60_       | 67.21           | 73.60           | _52.48_       | _55.60_       |
| SeaChain\[1\]         | 40.48           | 74.40           | 15.67           | 68.40           | 33.26           | 45.80           |
| ProbTree\[2\]         | 62.41           | 73.40           | 69.42           | 80.00           | 47.01           | 54.20           |
| ToR\[3\]              | *63.1*        | *-*           | *62.9*        | *-*           | *43.6*        | *-*           |
| **KAR³ (Ours)**  | **76.48** | **88.00** | **75.00** | **82.20** | **57.86** | **62.60** |

[1]. **SeaChain**: Since the official code loads pre-trained models from local without uploading those models online, we find models with most-similar name from HuggingFace to adapt it. Besides, we use *BAAI/bge-m3* instead of the ColBERT retriever due to environmental issues.

[2]. **ProbTree**: The official code was executed with bug fixes.

[3]. **ToR**: No official code has been released for ToR. The performance metrics provided are those reported by the authors using GPT-4-Turbo. Please note that the sampled testing suite and the method for answer normalization before F1 calculation may differ from ours.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732436690097

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses and Common Questions (part 1)

comment:
We appreciate the valuable feedback and insightful comments from all reviewers. In this rebuttal, we will address all reviewer comments point by point and revise our paper accordingly.

### CQ1. Cost analysis on LLM API Consumption and implementation with open-source LLMs (for reviewer qWux, 4rFD and vTwe)

We have conducted a comprehensive cost analysis to evaluate our model's API consumption compared to other baseline methods, and token consumption per QA of various methods are shown in Table 1. Our analysis primarily focuses on the inference cost, as chunk atomization is a one-time preprocessing step. The LLM API consumption for this preprocessing scales linearly with the number of data chunks and constitutes an overhead that varies across different benchmarks. Specifically, the average token usage per chunk for the three benchmarks is 338 tokens/chunk (HotPotQA), 321 tokens/chunk (2Wiki), and 320 tokens/chunk (MuSiQue).

As demonstrated in Table 1, our method utilizes fewer tokens than both ProbTree and IRCoT, and is comparable to Iter-RetGen. However, our approach significantly outperforms these baselines on both F1 and Accuracy by a considerable margin. This demonstrates the efficiency of our approach in balancing cost and performance. It is important to highlight that our method focuses on exploring potential reasoning chains, necessitating a thoughtful analysis generation during query decomposition with context at each iteration. As a result, completion token usage constitutes approximately one-quarter of the total consumption, distinguishing our approach from other baselines.

Table1: Token consumption (average/QA) and performance comparison on MuSiQue.

| Method                 | Prompt | Completion | Total ⬇️  | F1 ⬆️    | Acc ⬆️   |
| ---------------------- | ------ | ---------- | --------- | ----------| ----------|
| Zero-Shot CoT          | 85     | 105        | 191       | 22.90     | 23.47     |
| Naive RAG              | 1765   | 103        | 1869      | 43.31     | 44.40     |
| Self-Ask w/ Retrieval  | 5894   | 619        | 6514      | 46.76     | 51.40     |
| IRCoT                  | 9703   | 86         | _9789_    | 47.57     | 49.20     |
| Iter-RetGen            | 8140   | 473        | 8614      | _52.48_   | _55.60_   |
| SeaChain               | 2247   | 627        | 2875      | 33.26     | 45.80     |
| ProbTree               | 25225  | 650        | **25875** | 47.01     | 54.20     |
| **KAR³ (Ours)**        | 6525   | 2295       | 8820      | **57.86** | **62.60** |

To further address reviewers' concerns regarding API costs, we implement our method and other baselines using the open-source LLM, meta-llama/Llama-3.1-70B-Instruct (including chunk atomization). The results, presented in Table 2, indicate that this implementation not only maintains excellent performance after replacing GPT-4 with an open-source LLM, but also significantly outperforms the other baselines.

Table 2: Performance comparison with different LLMs.

| Method | HotpotQA F1 | HotpotQA Acc | 2Wiki F1 | 2Wiki Acc | MuSiQue F1 | MuSiQue Acc |
| ----------------------------- | ----- | ----- | ----- | ----- | ----- | ----- |
| Zero-Shot CoT(Llama3)         | 40.10 | 54.80 | 38.54 | 43.20 | 15.69 | 19.80 |
| Naive RAG(Llama3)             | 70.78 | 84.20 | 56.58 | 62.20 | 32.53 | 36.40 |
| Self-Ask w/ Retrieval(Llama3) | 70.25 | 83.00 | 66.25 | 74.00 | 38.19 | 44.20 |
| IRCoT(Llama3)                 | 74.59 | 88.00 | 69.49 | 77.60 | 43.12 | 49.60 |
| Iter-RetGen(Llama3)           | 72.23 | 85.20 | 59.21 | 65.00 | 37.16 | 40.40 |
| **KAR³(Llama3)**       | **75.27** | **88.20** | **72.79** | **81.00** | **50.68** | **59.70** |
| **KAR³(GPT-4)**        | **76.48** | **88.00** | **75.00** | **82.20** | **57.86** | **62.60** |

We will incorporate the results from Tables 1 and 2 into the revised version to provide a more comprehensive view of the token consumption of our work and its performance with open-source LLMs.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces KAR, a novel RAG framework combining knowledge atomization, query decomposition, and retrieval optimization to enhance domain-specific reasoning in multi-hop QA. Experiments on HotpotQA, 2Wiki, and MuSiQue benchmarks show improvements over baselines.  

Strengths:   
-The framework is well-designed, with innovative components like the knowledge atomizer and atomic selector, as noted by reviewers (qWux, 4rFD).   
-The paper is well-organized and well written  
-The method shows clear empirical advantages.   

Weaknesses:  
-Cost-Intensive construction: Several reviewers (4rFD, vTwe, and qWux) highlight concerns about the high computational costs of constructing KAR's atomic knowledge base, which relies heavily on repeated LLM calls.  
-Ambiguous Motivation: Reviewers (4rFD and vTwe) note that the motivation for the paper is unclear, particularly in explaining how the proposed method effectively addresses the stated challenges of domain-specific reasoning and retrieval.  
-Missing Baseline Comparisons: Reviewers (4rFD and 1t8b) emphasize the need for additional comparisons with strong RAG baselines.  
-Component Effectiveness: The reviewers raised concerns about lack of ablations on different components of the method (1t8B).

Despite strong empirical results, there are several concerns regarding the experimental rigor and presentation of the work.