StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization

## Abstract

Abstract Retrieval-augmented generation (RAG) is a key means to effectively enhance large language models (LLMs) in many knowledge-based tasks.
However, existing RAG methods struggle with knowledge-intensive reasoning tasks, because useful information required to these tasks are badly scattered.
This characteristic makes it difficult for existing RAG methods to accurately identify key information and perform global reasoning with such noisy augmentation.
In this paper, motivated by the cognitive theories that humans convert raw information into various structured knowledge when tackling knowledge-intensive reasoning, we proposes a new framework, StructRAG, which can identify the optimal structure type for the task at hand, reconstruct original documents into this structured format, and infer answers based on the resulting structure.
Extensive experiments across various knowledge-intensive tasks show that StructRAG achieves state-of-the-art performance, particularly excelling in challenging scenarios, demonstrating its potential as an effective solution for enhancing LLMs in complex real-world applications. https://github.com/Li-Z-Q/StructRAG

## 1 Introduction

With the advancement of deep learning technology, large language models (LLMs) have demonstrated considerable strengths in natural language tasks and are extensively applied in complex real-world scenarios (OpenAI et al., ; Yang et al., ).
However, they still exhibit limitations in factual tasks due to a lack of domain-specific knowledge, real-time updated information, and proprietary knowledge (Huang et al., ; Sui et al., ).
To address this, retrieval-augmented generation (RAG) methods have been developed to effectively provide essential external knowledge (Yu et al., ; Gao et al., ).
Typically, RAG methods involve splitting original documents into shorter chunks, retrieving the most relevant ones based on the query, and then using these chunks to enable LLMs to generate reliable answers (Ma et al., ; Li et al., ).
Due to their strong performance through this straightforward process, RAG methods are commonly employed in various knowledge-based question-answering tasks (Shi et al., ; Wang et al., ).

Unfortunately, current RAG approaches cannot effectively handle knowledge-intensive reasoning tasks due to the scattered nature of related information needed to solve these tasks (Kuratov et al., ; Yang et al., ).
Specifically, knowledge-intensive reasoning tasks often require a large amount of useful information which is dispersed across many locations in the provided documents, meanwhile the model needs to perform integrated reasoning after retrieving useful information (Yang et al., ; Wang et al., ).
Taking financial report analysis as an example, given a large number of financial documents and the need to compare the development trends of multiple companies, LLMs need to dig out all relevant financial indicators scattered across original documents and then generate insights by carefully comparing and comprehensively analyzing these indicators.
In such scenarios, standard RAG methods face challenges in accurately retrieving all relevant textual chunks, which may contain substantial noise, and integrating multiple key pieces of information for reasoning, leading to unsatisfactory performance on these tasks.

Figure: Figure 1: The overview of StructRAG framework, including an hybrid structure router to select the optimal structure type based on task requirements, a scattered knowledge structurizer to convert raw documents into structured knowledge, and a structured knowledge utilizer to decompose complex question and then effectively using the structured knowledge to infer the final answer.
Refer to caption: x1.png

From a human perspective, people do not solve knowledge-intensive reasoning tasks by simply reading raw texts (Johnson-Laird, ; Paivio, ).
As suggested in cognitive load theory, humans typically summarize scattered information from documents into structured knowledge, which is then used to shorten the reasoning path and enable more accurate judgement (Sweller, ; Chandler & Sweller, ).
Furthermore, cognitive fit theory shows that humans prefer using different types of structured knowledge for various tasks, such as tables for statistical analysis tasks and graphs for long-chain inference (Vessey, ; Umanath & Vessey, ).
In recent years, the rapid development of LLMs has laid the foundation for directly using these models to construct various knowledge structures (Li et al., ; Jain et al., ).
Meanwhile, many studies suggest that LLMs share similarities with humans in how they utilize information and solve complex problems (Wei et al., ; Li et al., ).
These inspire us to explore whether LLMs can adopt human-like thinking processes to transform scattered information into various structure formats during inference, thereby better serving knowledge-intensive reasoning tasks.

Motivated by this, we propose StructRAG, which employs a hybrid information structuring mechanism to construct and utilize structured knowledge in the most suitable format based on task requirements.
As illustrated in Figure 1, the StructRAG framework consists of three modules designed to sequentially identify the most suitable structure type, construct structured knowledge in that format, and utilize that structured knowledge to infer the final answer.
First, recognizing that different structure types are suited for different tasks, a hybrid structure router is proposed to determine the most appropriate structure type based on the question and document information of the current task.
Second, given that constructing structured knowledge is complex and requires strong comprehension and generation abilities, an LLM-based scattered knowledge structurizer is employed to convert raw documents into structured knowledge in the optimal type.
Finally, since questions in knowledge-intensive reasoning tasks can often be a complex composite problems that are challenging to solve directly, a structured knowledge utilizer is used to perform question decomposition and precise knowledge extraction for more accurate answer inference.

The core aspect of StructRAG is the hybrid structure router’s ability to accurately select the most suitable structure type for each input task.
To equip the router with this capability, we propose a training method for the hybrid structure router.
Inspired by successful use of reinforcement learning in training LLMs for decision-making tasks (Havrilla et al., ; OpenAI, ), we employ the DPO algorithm to train the router module, which follows reinforcement learning principles without requiring additional reward models (Rafailov et al., ; Allam, ).
However, there is insufficient training data for the model to learn how to choose the optimal structure type, and collecting enough such data in the real world is also challenging.
To address this, we introduce a novel pipeline for constructing preference training data that involves task synthesis, solution simulation, and preference judgment to create high-quality synthetic data, thereby enhancing the router’s ability to select the appropriate structure type.

In our experiments, we evaluate StructRAG across various knowledge-intensive reasoning tasks and compare it with several strong RAG baselines.
The results demonstrate that StructRAG achieves state-of-the-art performance, with improvements becoming more pronounced as task complexity increases.
This confirms that StructRAG is a robust solution for addressing challenging knowledge-intensive tasks.
Additionally, compared to recent Graph RAG methods, StructRAG not only exhibits superior performance across a broader range of tasks but also operates significantly faster on average.

## 2 Related Work

### 2.1 Retrieval-Augmented Generation

RAG technology achieves good performance in the era of LLMs by providing external knowledge to assist answering questions and reducing hallucinations (Jiang et al., ; Asai et al., ; Gao et al., ; Chen et al., ).
The initial strategy of RAG involves using a retriever to search for and retain highly relevant chunks from a knowledge base based on a query, these chunks are then fed into the generation module as external knowledge, enhancing its performance (Qi et al., ; Lewis et al., ; Gur et al., ; Yu et al., ).
To improve RAG effectiveness, some approaches have introduced iterative RAG, proposing various enhancements such as query expansion and rewriting (Ma et al., ; Li et al., ; Chan et al., ; Shi et al., ), and others try to improve the corporation between retrieval and generation (Qian et al., ; Su et al., ; Luo et al., ; Zhang et al., ).
Although existing methods achieve strong performance on multi-hop tasks like HotpotQA, chunk-based RAG struggles with knowledge-intensive tasks (Wang et al., ). This is because chunks must contain excessive text noise and do not capture the interconnections among information , thus LLMs cannot effectively use augmented knowledge.

### 2.2 Graph Retrieval-Augmented Generation

Recently, to assist LLMs in handling complex question-answering tasks, some works introduce graph structures into RAG systems (Edge et al., ; Panda et al., ; Peng et al., ).
One kind of approach uses pre-built knowledge graphs, extracting subgraph based on queries, which are then encoded as soft prompts or flattened into plain text for the generation module (Tang et al., ; He et al., ; Guan et al., ).
Another kind of approach involves extracting entity-relation triples from given text documents based on query requirements to construct graph structures, which are then used for knowledge augmentation (Fang et al., ; Edge et al., ; Panda et al., ).
Although these approaches significantly improve performance on multi-hop question-answering tasks, they focus solely on graph-based knowledge via the format of triplets, thus limiting their practical applicability in various domain and application of knowledge-intensive reasoning tasks.

## 3 StructRAG via Hybrid Information Structurization

As mentioned, due to badly dispersed information in knowledge-intensive reasoning tasks, the traditional retrieval module in RAG could retrieve chunks containing substantial textual noise, making it difficult for the generation module to extract useful information for inference.
Drawing inspiration from cognitive theories on how humans tackle such tasks, this paper proposes StructRAG, which utilizes a hybrid information structurization mechanism to construct and leverage structured knowledge in its optimal form.
Specifically, as illustrated in Figure 1, StructRAG first employs a hybrid structure router to identify the most appropriate structure type for the given task, and then employs a scattered knowledge structurizer to transform raw documents into structured knowledge in that format, and finally incorporates a structured knowledge utilizer to break down complex questions into simpler sub-questions, enabling more accurate reasoning on structured knowledge.

##### Task Formulating.

Knowledge-intensive reasoning tasks involved in this paper provide a question $q$ and a large set of documents $D$ as input, with the goal of deriving an answer $a$ based on the provided documents, which can be expressed as follows:

$$ $a=\mathcal{F}(q,D),\text{where}~{}D=\{d^{(i)}\}_{i=1}^{m}~{}~{}$ (1) $$

where $m$ is the number of documents, which can exceed 20, resulting in a total token of up to 200K.
Thus, the most obvious characteristic of these tasks is that useful information is dispersed across the provided documents, requiring the model to engage in complex reasoning based on large-scale relevant data.
For example, when comparing the development trends of several companies using a batch of financial reports, the task necessitates retrieving various financial indicators scattered throughout the documents, followed by a detailed comparison of these indicators.
This involves considering factors such as the relative importance of different indicators and the magnitude of numerical differences. Consequently, knowledge-intensive reasoning tasks present significant challenges.

##### Hybrid Structure Router.

From a human perspective, when solving knowledge-intensive reasoning tasks, individuals tend to use the type of structured knowledge that best matches the specific requirements of faced task.
To this end, StructRAG incorporates a hybrid structure router $\mathcal{R}$ to select the optimal structure type.
Specifically, the router leverages the question $q$ and the core content $C$ of documents $D$ to make its decision and generate the most suitable structure type $t$, as it is impractical to process the entire set of documents at once.

$$ $t=\mathcal{R}(q,C),\text{where}~{}C=\{c^{(i)}\}_{i=1}^{m}$ (2) $$

The core content $C$ is the concentrate of the titles or the first few sentences from each document $d^{(i)}$.
In our work, there are five candidate structure types for five kinds of knowledge-intensive tasks: table for statistical tasks, graph for long-chain tasks, algorithm for planning tasks, catalogue for summarizing tasks, and chunk for simple single-hop tasks.
Considering the core effect of the router in the overall framework, our work designs a DPO-based training method to develop a router that excels in knowledge type decision, which is detailed in Section 4.

##### Scattered Knowledge Structurizer.

After identifying the most suitable structure type, StructRAG extracts the textual knowledge scattered across raw documents and reconstructs it into structured knowledge.
This process requires a comprehensive understanding of all raw documents and and precise formatting of the information, making it a challenging and flexible problem.
Therefore, StructRAG employs an LLM-based scattered knowledge structurizer to facilitate the structurization process.
Specifically, as shown in Eq. 3 the structurizer $\mathcal{S}$ takes the question $q$, the selected type $t$, and each raw document $d^{(i)}$ as input, and extract the structured knowledge $k_{t}^{(i)}$ from the document via the powerful understanding and generation ability of LLMs.
In addition, a description of the structured knowledge $k_{t}$ is also generated.

$$ $k^{(i)}_{t},b^{(i)}_{t}=\mathcal{S}(q,t,d^{(i)})$ (3) $$

After that, all output structured knowledge will be collected as the overall one $K_{t}=\{k^{(i)}_{t}\}_{i=1}^{m}$, and the overall description of the whole structured knowledge is constructed as $B_{t}=\{b^{(i)}_{t}\}_{i=1}^{m}$.
In term of detailed representation of each kind of structure, the table is by markdown, graph by in list of head-relationship-tail triplets, chunk is by regular text, algorithm is by pseudo code, and catalogue is by text with hierarchical number (e.g., Section One, 1.1, 1.1.2) as explicitly chapter identifier.

##### Structured Knowledge Utilizer.

After obtaining the structured knowledge in its optimal type, StructRAG performs reasoning to answer the question.
Given that the question can be highly combinatorial, this may hinder the identification and use of relevant information in the structured knowledge. Therefore, StructRAG employs an LLM-based structured knowledge utilizer to facilitate question decomposition, precise knowledge extraction, and final answer inference.
Specifically, the decomposition process of the utilizer takes the original question $q$ and the overall description of structured knowledge $B_{t}$ as input, breaking the question down into several simple and intuitive sub-questions $\hat{q}^{(j)}$.
Next, the extraction process aims to find out precise knowledge $\hat{k}^{(j)}_{t}$ for each sub-question $\hat{q}^{(j)}$ from the whole structured knowledge $K_{t}$.
Finally, the inference process integrates all the sub-questions and their extracted precise knowledge to generate a final answer $a$, which can be expressed as follows:

$$ $\hat{Q}=\mathcal{U}_{\text{decompose}}(q,B_{t})=\{\hat{q}^{(j)}\}_{j=1}^{n}$ (4) $$

$$ $\hat{K}_{t}=\{\mathcal{U}_{\text{extract}}(\hat{q}^{(j)},K_{t})\}_{j=1}^{n}=\{ \hat{k}^{(j)}_{t}\}_{j=1}^{n}$ (5) $$

$$ $a=\mathcal{U}_{\text{infer}}(q,\hat{Q},\hat{K}_{t})$ (6) $$

where $n$ is number of sub-questions, $\mathcal{\hat{Q}}$ is set of all sub-questions, $\hat{K}_{t}$ is whole precise knowledge for all sub-questions, and $\mathcal{U}_{\text{decompose}}$, $\mathcal{U}_{\text{extract}}$ and $\mathcal{U}_{\text{infer}}$ are process of decomposition, extraction and inference, respectively. More details about the utilizer are shown in Appendix A.3.

## 4 Hybrid Structure Router Training

In the StructRAG framework described above, the core factor is accurately determining the most suitable structure type based on the input task, and the performance of the hybrid structure router directly influences the overall effectiveness of the framework.
Therefore, to achieve a high-performance router, we propose a training method to enhance the ability of LLMs in identifying the suitable structure type of knowledge.
Specifically, given the strong capabilities of reinforcement learning in decision-making scenarios, we train the router using the DPO algorithm, which achieves results similar to reinforcement learning while avoiding the need for additional reward models. Regarding training data, since there is no existing preference data for the optimal structure type selection task, we design a synthesizing-simulating-judging method to efficiently construct preference pairs for training. A detailed explanation is provided in the following paragraphs, along with examples and prompts in the Appendix A.1.

Figure: Figure 2: The illustration of training data constructing. First use LLMs to synthesize knowledge-intensive tasks, and then simulate solutions by structured knowledge in different types, finally judge all possible solutions and get preference pairs about candidate structure types.
Refer to caption: x2.png

##### Data Constructing.

Due to the scarcity of training data for selecting the optimal structure type in the current NLP community, we employ a synthesizing-simulating-judging method to construct preference pairs for training the router.
Specifically, as illustrated in Figure 2, given several manually collected seed tasks that covering the possible structure types, we first use LLMs to synthesize a set of new tasks by the in-context learning method, where each task contains a question and core context for documents.
Then, for each synthetic task, LLMs is employed to simulate the process of addressing this task by structured knowledge in different types, thus getting different simulated solutions.
Finally, a LLM-based judge compares these simulated solutions for solving the task, generating preference pairs regarding the structure types.
Each constructed data entry includes a question, the core contents of documents, the chosen structure type, and the rejected structure type, as expressed as follows:

$$ $D_{\text{synthetic}}=\{q^{(k)},C^{(k)},t_{w}^{(k)},t_{l}^{(k)}\}_{k=1}^{N}$ (7) $$

where $t_{w}$ and $t_{l}$ are chosen structure type and rejected structure type, respectively. In addition, synthetic preference pairs include both English and Chinese data in order to improve the universality.

##### Preference Training.

Inspired by the success of using reinforcement learning to train LLMs for decision-making tasks, we employ the DPO algorithm to train the router module, which can get the same effectiveness as reinforcement learning without adding additional reward models.
Specifically, the input for training the router includes a question and the core contents of documents, and the output is one kind of structure type (e.g., “table”, “graph”).
As described in last paragraph, we simulate and construct a set of preference pairs for DPO training, which can be formulated as following:

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{E}_{(q,C,t_{w },t_{l})\sim D_{\text{synthetic}}}\left[\log\sigma\left(\beta\log\frac{\pi_{ \theta}(t_{w}\mid q,C)}{\pi_{\text{ref}}(t_{w}\mid q,C)}-\beta\log\frac{\pi_{ \theta}(t_{l}\mid q,C)}{\pi_{\text{ref}}(t_{l}\mid q,C)}\right)\right]$ (8) $$

where $\pi_{\theta}$ and $\pi_{\text{ref}}$ are target policy and reference policy, respectively, and $\beta$ is a hyperparameter.
As analyzed later, this preference training enables the model to differentiate between various types of knowledge and their suitability for a given task, resulting in better performance compared to zero-shot and few-shot settings.

## 5 Experiments

### 5.1 Experimental Settings

##### Evaluation Datasets.

This paper includes various knowledge-intensive reasoning tasks in evaluation.
First, this paper chooses the Loong benchmark (Wang et al., ), which includes four tasks (Spotlight Locating, Comparison, Clustering, and Chain of Reasoning) and four document length settings, as the length of the document increases, the useful information needed to solve the task becomes more dispersed. As for metrics, this paper adheres to original settings in Loong and use the official code repository(^1^11https://github.com/MozerWang/Loong) in evaluation, involving using LLMs to decide a score from 0 to 100 and the exact matching (EM) rate.
In addition, this paper chooses Podcast Transcripts, which is a query-focused summarization task reported by GraphRAG (Edge et al., ). As for metrics, this paper follows GraphRAG settings, involving head-to-head win rate by a LLM judgement, across four kinds of dimension, which are comprehensiveness, diversity, empowerment, and directness.

##### Implementation Details.

We build framework based on Qwen2 series models (Yang et al., ).
For the hybrid structure router, StructRAG uses Qwen2-7B-Instruct as the base model and implement DPO training by trl(^2^22https://github.com/huggingface/trl). As for the details of hybrid structure router training, StructRAG constructs and uses a total of 900 preference data, setting the learning rate as 1e-5, number of epochs as 3 and the $\beta$ as default in training.
For the scattered knowledge structurizer and strutured knowledge utilizer, StructRAG directly uses Qwen2-72B-Instruct as base model and deploy models as API using vllm(^3^33https://pypi.org/project/vllm/) following the same setting as in Loong (Wang et al., ).

##### Selected Baselines.

We select baselines from commonly used or recent methods for knowledge-based question-answering tasks. Specifically,
(1) Long-Context (Yang et al., ), which extends the input window of LLMs to up to 128K tokens through extrapolation techniques, allowing large-scale documents to be directly input into the model.
(2) RAG (Lewis et al., ), which splits the given documents into multiple short chunks and uses a retriever to retain only the most relevant chunks as augmentation based on the question.
(3) RQ-RAG (Chan et al., ), which uses a trained LLM to decompose and refine the original question to more accurately find the required chunk augmentation.
(4) GraphRAG (Edge et al., ), which extracts triples (head, relationship, tail) from raw documents and constructs into multi-layered graphs, then uses structured information in graphs to help the generation model answer questions.
In implement, for Long-Context and RAG, we directly follow the experimental settings reported in Loong (Wang et al., ), and for RQ-RAG(^4^44https://github.com/chanchimin/RQ-RAG) and GraphRAG(^5^55https://pypi.org/project/graphrag/), we evaluate the performance based on the official code repositories. Noting that, for fair comparison, we also set Qwen-72B-Instruct as base model of GraphRAG.

**Table 1: LLM-judged scores and exact matching rate in knowledge-intensive tasks of Loong benchmark. From Set 1 to Set 4, task complexity gradually increases, as reflected by the growing token number of documents. The table show two main conclusions: StructRAG get the SOTA performance in overall metrics. And the more complex the task, the greater the improvement of StructRAG.**
| Method | Spot. | Comp. | Clus. | Chain. | Overall |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLM Score | EM | LLM Score | EM | LLM Score | EM | LLM Score | EM | LLM Score | EM |  |
| Set 1  (10K-50K Tokens) |  |  |  |  |  |  |  |  |  |  |
| Long-context (Yang et al., ) | 68.49 | 0.55 | 60.60 | 0.37 | 47.08 | 0.08 | 70.39 | 0.36 | 60.11 | 0.29 |
| RAG (Lewis et al., ) | 51.08 | 0.35 | 44.53 | 0.27 | 37.96 | 0.05 | 53.95 | 0.35 | 46.11 | 0.23 |
| RQ-RAG (Chan et al., ) | 72.31 | 0.54 | 48.16 | 0.05 | 47.44 | 0.07 | 58.96 | 0.25 | 53.51 | 0.17 |
| GraphRAG (Edge et al., ) | 31.67 | 0.00 | 27.60 | 0.00 | 40.71 | 0.14 | 54.29 | 0.43 | 40.82 | 0.18 |
| StructRAG (Ours) | 74.53 | 0.47 | 75.58 | 0.47 | 65.13 | 0.23 | 67.84 | 0.34 | 69.43 | 0.35 |
| Set 2  (50K-100K Tokens) |  |  |  |  |  |  |  |  |  |  |
| Long-context (Yang et al., ) | 64.53 | 0.43 | 42.60 | 0.21 | 38.52 | 0.05 | 51.18 | 0.20 | 45.71 | 0.17 |
| RAG (Lewis et al., ) | 66.27 | 0.46 | 46.28 | 0.31 | 38.95 | 0.05 | 46.15 | 0.22 | 45.42 | 0.19 |
| RQ-RAG (Chan et al., ) | 57.35 | 0.35 | 50.83 | 0.16 | 42.85 | 0.03 | 47.60 | 0.10 | 47.09 | 0.10 |
| GraphRAG (Edge et al., ) | 24.80 | 0.00 | 14.29 | 0.00 | 37.86 | 0.00 | 46.25 | 0.12 | 33.06 | 0.03 |
| StructRAG (Ours) | 68.00 | 0.41 | 63.71 | 0.36 | 61.40 | 0.17 | 54.70 | 0.19 | 60.95 | 0.24 |
| Set 3  (100K-200K Tokens) |  |  |  |  |  |  |  |  |  |  |
| Long-context (Yang et al., ) | 46.99 | 0.27 | 37.06 | 0.13 | 31.50 | 0.02 | 35.01 | 0.07 | 35.94 | 0.09 |
| RAG (Lewis et al., ) | 73.69 | 0.55 | 42.20 | 0.27 | 32.78 | 0.02 | 37.65 | 0.13 | 42.60 | 0.18 |
| RQ-RAG (Chan et al., ) | 50.50 | 0.13 | 44.62 | 0.00 | 36.98 | 0.00 | 36.79 | 0.07 | 40.93 | 0.05 |
| GraphRAG (Edge et al., ) | 15.83 | 0.00 | 27.40 | 0.00 | 42.50 | 0.00 | 43.33 | 0.17 | 33.28 | 0.04 |
| StructRAG (Ours) | 68.62 | 0.44 | 57.74 | 0.35 | 58.27 | 0.10 | 49.73 | 0.13 | 57.92 | 0.21 |
| Set 4  (200K-250K Tokens) |  |  |  |  |  |  |  |  |  |  |
| Long-context (Yang et al., ) | 33.18 | 0.16 | 26.59 | 0.08 | 29.84 | 0.01 | 25.81 | 0.04 | 28.92 | 0.06 |
| RAG (Lewis et al., ) | 52.17 | 0.24 | 24.60 | 0.10 | 26.78 | 0.00 | 17.79 | 0.00 | 29.29 | 0.07 |
| RQ-RAG (Chan et al., ) | 29.17 | 0.08 | 40.36 | 0.00 | 26.92 | 0.00 | 34.69 | 0.00 | 31.91 | 0.01 |
| GraphRAG (Edge et al., ) | 17.50 | 0.00 | 26.67 | 0.00 | 20.91 | 0.00 | 33.67 | 0.33 | 23.47 | 0.05 |
| StructRAG (Ours) | 56.87 | 0.19 | 55.62 | 0.25 | 56.59 | 0.00 | 35.71 | 0.05 | 51.42 | 0.10 |

**Table 2: Win rate of StructRAG vs. baselines on Podcast Transcripts. StructRAG achieves the best average performance compared to all baselines, further conforming effectiveness of framework.**
| Compared Method Pair | Comprehensiveness | Diversity | Empowerment | Directness | Average |
| --- | --- | --- | --- | --- | --- |
| StructRAG vs. Long-context (Yang et al., ) | 98 | 96 | 97 | 92 | 95.75 |
| StructRAG vs. RAG (Lewis et al., ) | 67 | 78 | 51 | 52 | 62.00 |
| StructRAG vs. RQ-RAG (Chan et al., ) | 67 | 75 | 50 | 46 | 59.50 |
| StructRAG vs. GraphRAG (Edge et al., ) | 61 | 68 | 42 | 41 | 53.00 |

### 5.2 Overall Results

Results compared with baselines are shown in Table 1 and Table 2, there are two main conclusions:

1) StructRAG is a powerful solution to addressing knowledge-intensive reasoning tasks.
Based on the experimental results in Table 1, StructRAG outperforms the baselines in most tasks and document length settings, and in the overall metric, StructRAG exceeds all baselines in both LLM score and EM rate. In addition, as shown in Table 2, StructRAG achieves the best average performance compared to all baselines in Podcast Transcripts. All in all, these experimental findings demonstrate that StructRAG can effectively address knowledge-intensive reasoning tasks and improve a lot compared with previous long-context methods, and different kind of existing powerful RAG techniques.

2) StructRAG is particularly suitable for complex tasks, performance improvement becomes more significant in scenarios with more dispersed information.
Based on the overall performance in Table 1, the performance comparison between StructRAG and the long-context baseline shows that StructRAG achieves performance improvements of approximately 9, 15, 22, and 23 on Set 1, Set 2, Set 3, and Set 4, respectively. Similarly, comparing StructRAG with RAG shows performance improvements of around 15, 15, and 22 on Set 2, Set 3, and Set 4, respectively. Each set represents the total length of the given documents, with Set 1 being the shortest and Set 4 being the longest. This means that the information needed to answer the questions becomes more dispersed as the length of the documents increases, and making the reasoning process more challenging. Therefore, these results indicate that StructRAG shows more significant improvements over the baselines with longer documents and more scattered information, demonstrating that abilities of our framework to construct and use the optimal type of structured knowledge is especially effective for complex tasks.

### 5.3 Ablation Results of Modules

To validate the role of each module in StructRAG, we perform ablation experiments. As shown in Table 3, “w/o router” refers to random routing, “w/o structurizer” means using only chunks, and “w/o utilizer” refers to directly concatenating the structured knowledge with the original question for answer generation. There are following conclusions:

1) All three modules contribute positively to the overall framework.
The table shows that removing any of the three modules results in a noticeable performance decline. The overall performance will reduce from 60.38 to 45.33, 53.92 and 55.94 for the router, structurizer, and utilizer, respectively.
This proves that all three modules play an irreplaceable role, and StructRAG tightly and orderly combines these three modules to achieve excellent overall performance.

2) Choosing the suitable structure type and constructing documents into structured knowledge are more crucial than designing complex utilization methods.
A comparison in the table reveals that different modules are with different importance. The performance drop is most significant when the router is removed, with a decreasing from 60.38 to 45.33. In contrast, removing the utilizer leads to a smaller performance decline, from 60.38 to 55.94.
This indicates that simply question-refining as existing methods provides limited improvement for knowledge-intensive reasoning tasks, a more promising direction is constructing and using structured knowledge in suitable type.

**Table 3: Ablation results of three modules. The table shows that all three module are with positive contribution, and the most core module is the hybrid structure router.**
| Method | Set 1 | Set 2 | Set 3 | Set 4 | Overall |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLM Score | EM | LLM Score | EM | LLM Score | EM | LLM Score | EM | LLM Score | EM |  |
| StructRAG | 69.43 | 0.35 | 60.95 | 0.24 | 57.92 | 0.21 | 51.42 | 0.10 | 60.38 | 0.23 |
| w/o router | 51.09 | 0.28 | 48.28 | 0.17 | 39.52 | 0.13 | 41.83 | 0.10 | 45.33 | 0.17 |
| w/o structurizer | 64.97 | 0.29 | 52.17 | 0.17 | 53.18 | 0.19 | 44.24 | 0.10 | 53.92 | 0.19 |
| w/o utilizer | 68.23 | 0.29 | 59.73 | 0.24 | 53.29 | 0.19 | 35.77 | 0.10 | 55.94 | 0.22 |

Figure: Figure 3: Performance of StructRAG with different routers. The strong router shows obvious positive impact on the overall framework.
Refer to caption: x3.png

### 5.4 Detailed Analysis

In this section, we do some detailed analysis, including, performance and impact of the router, drawback of using fixed structure type, case study about EM rate performance, and efficiency reports.

#### 5.4.1 Effect of the Router

To explore the necessity of constructing data and conducting DPO training, and relationship between performance of hybrid structure router and overall StructRAG, we first compare our router with raw LLMs, and then draw curl of router and overall StructRAG score. There are following conclusions:

1) Selecting the optimal type of knowledge based on the task is challenging for raw LLMs without special training.
Based on the experimental results in Table 4, the router trained based on Qwen2-7B-Instruct model significantly outperforms the 72B model with few-shot setting. This
indicates that LLMs need some special training to get the ability of selecting the optimal structure type based on needs of the task, even when the model scale reaches 72B.

2) The performance of hybrid structure router is with significant relevance with the final performance of StructRAG.
As shown in Figure 4, we select Qwen2-72B-Instruct (zero-shot) as the weak router, and design a completely random router and a completely incorrect bad router. The curves in the figure clearly show a positive correlation between router accuracy and the overall performance of the StructRAG framework. This further demonstrates that selecting knowledge types that match the task needs for augmentation is crucial in knowledge-intensice reasoning tasks.

#### 5.4.2 Drawback of a Fixed Type of Knowledge

To further verify the importance of containing hybrid types of structure rather than a fixed type, we freeze the structure type used in the framework as either chunk, graph, table, algorithm or catalogue for all evaluation tasks. There are the following conclusions:

Using a single fixed type of knowledge cannot achieve good performance on diverse tasks.
Based on the experimental results in Table 4, it shows that for both scores and exact matching rate, using a single fixed type performs worse than selecting the optimal structure type for needs of the input task. In comparison, the performance degradation is least when only using chunk, with a reduction from 60.38 to 53.92, in other cases, the performance decline is more significant. This can demonstrate the effectiveness of cognitive fit theory in LLMs, meaning that using structured knowledge in the most suitable type can effectively enhance problem-solving abilities.

#### 5.4.3 Case Study about EM metric

According to the experimental results in Table 1, StructRAG surpasses baselines in general score, but falls short in seven sub-situations for the exact matching rate. Therefore, we analysis some cases that StructRAG method gets high score but fails exact matching.
The reason is mainly about structurization process may alter the textual format of original information. As shown in Table 6, there are some wording differences between structured knowledge and raw information (e.g. from original “$ 1,308,463” to “138463” in the table). Intuitively this aligns with the common sense, where structurizer is a probabilistic language model rather than rule-based model, thus some possible textual loss may be unavoidable, and output from GraphRAG method also show similar issue.

**Table 4: Results of only containing structured knowledge in one fixed type. It shows any single fixed type is insufficient, confirming the advance of StructRAG via hybrid information structurization.**
| Method | Set 1 | Set 2 | Set 3 | Set 4 | Overall |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLM Score | EM | LLM Score | EM | LLM Score | EM | LLM Score | EM | LLM Score | EM |  |
| StructRAG | 69.43 | 0.35 | 60.95 | 0.24 | 57.92 | 0.21 | 51.42 | 0.10 | 60.38 | 0.23 |
| w/ only table | 48.00 | 0.23 | 55.19 | 0.24 | 50.35 | 0.19 | 38.44 | 0.12 | 49.66 | 0.21 |
| w/ only graph | 30.59 | 0.09 | 24.05 | 0.05 | 17.46 | 0.03 | 20.96 | 0.04 | 22.71 | 0.05 |
| w/ only chunk | 64.97 | 0.29 | 52.17 | 0.17 | 53.18 | 0.19 | 44.24 | 0.10 | 53.92 | 0.19 |
| w/ only catalogue | 30.49 | 0.10 | 36.36 | 0.13 | 36.77 | 0.12 | 23.75 | 0.03 | 33.26 | 0.10 |
| w/ only algorithm | 43.53 | 0.24 | 32.86 | 0.08 | 31.59 | 0.13 | 16.67 | 0.04 | 32.32 | 0.12 |

Figure: Figure 5: Comparison of implementing latency (minute). The StructRAG has an available speed, which is a little slower than RQ-RAG, but is much faster than GraphRAG method.
Refer to caption: x4.png

#### 5.4.4 Efficiency Report

In this section, we report average latency of StructRAG and compare it with the RQ-RAG (Chan et al., ) and GraphRAG (Edge et al., ). The latency includes two components: The first part is constructing latency, referring to the process of iteratively retrieving chunks for RQ-RAG, constructing graphs for GraphRAG, and determining the optimal knowledge type and constructing corresponding structure for StructRAG. The second part is reading latency, referring to the process of using augmented knowledge to generate final answers. As shown in Table 6. StructRAG has slightly higher latency compared to RQ-RAG but is obviously faster than GraphRAG. Therefore, StructRAG is a kind of high-performance framework with available implementing speed.

## 6 Conclusion

In this paper, noticed the limitation of existing RAG methods in knowledge-intensive reasoning tasks, and inspired by cognitive theories about how human beings solve such tasks, we propose a new framework, StructRAG via hybrid information structurization, which can construct and utilize structured knowledge in the optimal format as augmentation.
StructRAG includes a hybrid structure router to precisely select the optimal structure type, then a scattered knowledge structurizer to convert raw documents into structured knowledge, and finally a structured knowledge utilizer to decompose complex questions and infer the final answer via the constructed structured knowledge. Furthermore, in order to get a high-performance hybrid structure router, we construct training data by a synthesizing-simulating-judging pipeline and then implement preference training via DPO algorithm.
Experiments on extensive knowledge-intensive reasoning tasks demonstrate that StructRAG is an effective solution, which reach the SOTA performance and can achieve large improvement in badly information-scattered scenarios.
Therefore, this paper offers a promising direction, focused on hybrid structured knowledge, for developing more powerful RAG systems in the future.

## Appendix A Prompts in StructRAG

### A.1 Prompts in Data Constructing

Considering the current lack of training data that determines the optimal structure type based on task requirements, and the difficulty of collecting such data in real-world scenarios, we designed a pipeline for synthesizing, simulating, and judging data for DPO training specifically aimed at a hybrid structure router. In terms of implementation, we created several prompts to drive the three processes of the LLM, as shown in Figure 9.

Figure: Figure 8: Prompts and examples used in data constructing for training the hybrid structure router, including synthesizing tasks, simulating solutions, and judging difference solutions.
Refer to caption: x6.png

### A.2 Prompts of Scattered Knowledge Structurizer

Considering that constructing various structured knowledge from a large scale of scattered information is a complex task that requires strong comprehension and generation abilities, and that LLMs have demonstrated a good capacity for integrating structured knowledge in previous work, we designed prompts to drive LLMs in achieving this process, as shown in Figure 8

Figure: Figure 9: Prompts used in convert raw information in original documents into structured knowledge via LLMs, different type of structured knowledge use different prompts, and the chunk is directly by splitting without prompt.
Refer to caption: x7.png

### A.3 Prompts of Structured Knowledge Utilizer

Considering that questions in knowledge-intensive reasoning can be complex combinatorial tasks, breaking them down into multiple simpler sub-questions can leverage the structured knowledge more effectively for reasoning. Therefore, we designed prompts to drive LLMs to achieve question decomposition and precise knowledge extraction, as shown in Figure 7.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: In Table 3 and Figure 3, the exact match (EM) rate for Set 4 appears close across methods or even equivalent in some cases. Could the authors clarify whether this is due to specific characteristics of the dataset (e.g., limited answer variation) or if this is a coincidence? If the dataset impacts EM rates, additional context would be helpful.
soundness: 3
strengths: The paper addresses a challenging problem in LLM and RAG research, focusing on improving knowledge-intensive reasoning for real-world tasks. The idea of information structurization for RAG is clear and innovative, offering a new approach in this area.

The architecture and its components are described clearly, with each part of the StructRAG model explained in a way that makes it easy to follow. The experiments are thorough and well-documented, including ablation studies that show the contributions of each module and support the effectiveness of the approach.

Overall, the work has practical applications and could be adapted to a range of RAG-related tasks, suggesting it has potential for further impact in the field.
summary: The paper introduces StructRAG, a novel framework designed to enhance LLMs for knowledge-intensive reasoning tasks by structuring information at inference time. StructRAG addresses limitations in existing retrieval-augmented generation (RAG) methods, which struggle when relevant knowledge is dispersed and reasoning demands are complex. Inspired by cognitive theories, StructRAG applies a hybrid structure router to identify the best structure type for each task, a scattered knowledge structurizer to transform raw documents into structured knowledge, and a structured knowledge utilizer to perform multi-step reasoning. This framework achieves state-of-the-art results across multiple benchmarks and tasks, particularly excelling in challenging scenarios where information is fragmented and extensive reasoning is required.
weaknesses: The study presents StructRAG’s effectiveness with a single structured knowledge type for each task, yet it is unclear how the model would perform on tasks requiring multiple types simultaneously. For example, an ablation study that evaluates combinations of two fixed structure types could shed light on whether using multiple structure types consistently improves performance across different task categories.

The paper does not provide details on the distribution of structure types chosen by the hybrid structure router in its dataset. Since some structure types, such as algorithmic pseudo code, may be more complex to extract compared to tables or catalogues, reporting this distribution would clarify the router’s efficacy across different structure types. Additionally, an analysis of the structurizer’s performance in cases involving algorithmic structuring would help assess whether these cases yield similar performance gains as simpler structures.

Table 5 highlights performance differences when using fixed structure types, but these differences may be influenced by the distribution of tasks that align better with certain types (e.g., most tasks in the dataset naturally suited to tables). A breakdown of the dataset showing the router’s selected structure types and a comparison to human-selected types would be useful. Such a comparison could indicate how well the router aligns with human judgment and whether adding certain structure types as a standard complement might enhance overall performance in diverse datasets.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Lines 458-460 highlight a challenge in real-world applications. How do the authors propose to address this challenge?
soundness: 3
strengths: This paper demonstrates several notable strengths:

- The novel integration of cognitive science principles into RAG systems. The authors establish meaningful connections between human information processing and computational approaches, making the cognitive science foundation both natural and practical.

- The paper addresses a well-recognized challenge in the field: efficient information retrieval from unstructured inputs. Their solution, particularly the Hybrid Structure Router that mimics human decision-making combined with DPO fine-tuning, offers an innovative yet implementable approach.

- The experimental evaluation is comprehensive, comparing StructRAG against current state-of-the-art approaches including long-context models, traditional RAG, RQ-RAG, and GraphRAG. The promising results across these various settings demonstrate the method's practical value.

- The ablation studies are particularly insightful, reflecting the authors' thorough analysis and iterative refinement of their methodology.
summary: This paper introduces StructRAG, a novel framework for improving retrieval-augmented generation (RAG) in knowledge-intensive reasoning tasks. The key insight is that existing RAG methods struggle when useful information is scattered across documents, making it difficult to identify and reason with key information.

Inspired by cognitive theories about how humans process information using different knowledge structures, StructRAG consists of three main components:

1. A hybrid structure router that determines the most appropriate structure type (e.g., tables, graphs) for a given task
2. A scattered knowledge structurizer that converts raw documents into the chosen structured format
3. A structured knowledge utilizer that decomposes complex questions and performs reasoning using the structured knowledge

To train the hybrid structure router effectively, the authors develop a novel pipeline for generating training data through task synthesis, solution simulation, and preference judgment, combined with the DPO algorithm for preference learning.

Experimental results show that StructRAG achieves state-of-the-art performance across various knowledge-intensive tasks, with particularly strong improvements on more complex tasks. The framework also demonstrates better performance and faster operation compared to existing Graph RAG methods.

The paper's main contribution is presenting a cognitively-inspired approach to structuring information during inference time in RAG systems, offering a promising direction for handling complex reasoning tasks where relevant information is scattered across multiple sources.
weaknesses: While not a significant weakness, one suggestion concerns the comparison with Graph RAG. Graph RAG typically excels with complex relational data rather than long-context scenarios, which might explain its relatively poor performance in many of the experimental settings. A more targeted comparison using relationship-heavy datasets could provide additional insights into the relative strengths of both approaches.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - What is the definition of LLM score
- Did you tested the new method StructRAG with other models, or models with larger amount of parameters?
soundness: 3
strengths: - It proposed a novel method
- It conducted thorough literature review
- It compared the method with other RAG methods including Long-context, RAG, RQ-RAG, GraphRAG
summary: This paper proposed a novel method to summarise the document with Hybrid Structure Router, and it dynamically converted documents into graph, chunk, catalogue, algorithm, table. It showed that the new method had better performance in terms of score and latency. The paper contains thorough literature review, and detailed experiment results. However, the paper only evaluates the performance on Qwen2-7B-Instruct model, which is a relatively small model. It did not evaluate on other main stream models like llama, and also did not evaluate the new method on models with larger amount of parameters. For example, the GraphRAG method used GPT-4-turbo in their original paper, and it is very hard to conclude that StructRAG is better than GraphRAG with the results of Qwen2-7B-Instruct presented in this paper.
weaknesses: - It did not provide the definition of LLM score
- It only tested the new model with Qwen2-7B-Instruct with relatively small amount of parameters, and lacks of experiment results with other models or models with larger amount of parameters

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Could you provide more detailed examples of tasks that are suitable for each of the five types of structured knowledge you propose? This would help clarify the rationale behind categorizing them into these five types.

2. When using LLMs to extract structured knowledge, do the LLMs have the capability to directly generate knowledge in the specified formats? Did you perform any checks or corrections on the outputs generated by the LLMs?
soundness: 4
strengths: **Originality**: This paper introduces a novel approach to addressing the complex issue of RAG from multiple knowledge sources. The proposed process includes summarizing structured knowledge into five categories and utilizing LLMs for the construction and retrieval of this structured knowledge. This approach demonstrates a high degree of originality.

**Quality**: The writing quality of the paper is commendable, with coherent reasoning and a logical flow that makes it easy for readers to understand. The authors have conducted detailed experiments across multiple datasets, effectively demonstrating the validity of StructRAG.

**Clarity**: The paper is logically and structurally well-organized, with a clear presentation of the proposed framework's structure and processes. The use of formulas and diagrams effectively illustrates the concepts and workflow.

**Significance**: This paper effectively addresses the challenge of handling RAG with complex documents by aggregating knowledge from different sources and structuring it into specified formats. This approach significantly enhances the performance and applicability of RAG systems.
summary: The paper addresses the challenge of handling RAG with complex documents, where information may be scattered across different parts of multiple documents. For RAG to be accurate, it must precisely retrieve and integrate all relevant texts, which presents a significant challenge. To tackle this, the authors propose a method that mimics human cognitive processes by using LLMs to convert dispersed information into various types of structured knowledge. Additionally, they employ a Hybrid Structure Router, trained with DPO, to select the appropriate structure type for each query. The implementation of this approach demonstrates its effectiveness in improving performance across most tasks.
weaknesses: The paper, while introducing the StructRAG framework, relies significantly on the inherent capabilities of large language models (LLMs) for structuring and utilizing raw knowledge. However, it could benefit from a more detailed discussion on the challenges encountered during knowledge processing and on strategies for more effectively leveraging LLMs in this context.

Meta-review: This paper proposes a novel framework for solving knowledge-intensive reasoning tasks, which are difficult for existing retrieval-augmentation generation methods to address in locating key information. The framework comprises three components: a structure router for structure type selection, a knowledge structurizer for knowledge extraction, and a knowledge utilizer for answer generation. This framework achieves state-of-the-art performance across various tasks, and each component's effectiveness is demonstrated through ablation studies.

The standout design in this framework is the structure router, which contributes the most to performance gains. The method of constructing synthetic data for training the router is innovative, effectively addressing the issue of data scarcity. Furthermore, the significant performance improvements observed in the experimental results demonstrate the effectiveness of StructRAG in knowledge-intensive tasks.

One area for potential improvement is that the effectiveness of StructRAG depends on the underlying language models used for structurization. An additional study on structurization across various language models would be valuable for broader application.

Decision: Accept (Poster)