StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization
Authors: Zhuoqun Li
Xuanang Chen
Haiyang Yu
Hongyu Lin
Yaojie Lu
Qiaoyu Tang
Fei Huang
Xianpei Han
Le Sun
Yongbin Li
Venue: ICLR 2025 Poster (2025)
Keywords: RAG
Structured Knowledge
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: To address knowledge intensive reasoning tasks, we propose StructRAG, constructing raw documents into hybrid structured knowledge as augmentation.

Abstract: Retrieval-augmented generation (RAG) is a key means to effectively enhance large language models (LLMs) in many knowledge-based tasks. 
However, existing RAG methods struggle with knowledge-intensive reasoning tasks, because useful information required to these tasks are badly scattered. 
This characteristic makes it difficult for existing RAG methods to accurately identify key information and perform global reasoning with such noisy augmentation.
In this paper, motivated by the cognitive theories that humans convert raw information into various structured knowledge when tackling knowledge-intensive reasoning, we proposes a new framework, StructRAG, which can identify the optimal structure type for the task at hand, reconstruct original documents into this structured format, and infer answers based on the resulting structure. 
Extensive experiments across various knowledge-intensive tasks show that StructRAG achieves state-of-the-art performance, particularly excelling in challenging scenarios, demonstrating its potential as an effective solution for enhancing LLMs in complex real-world applications.

Published as a conference paper at ICLR 2025
STRUCTRAG: BOOSTING KNOWLEDGE INTENSIVE REASONING OF LLMs VIA INFERENCE-TIME HYBRID
INFORMATION STRUCTURIZATION
Zhuoqun Li Xuanang Chen'; Haiyang Yu", Hongyu Lin', Yaojie Lu', Qiaoyu Tang-
Fei Huang', Xianpei Han Sun Yonghin Lj3* Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences Univerriity Chinese Academy of Sciences TTngyy Lab
(lizhuoqun2021, che hongyu, ytg. hogpapaaww....ooo.o..o.aoiiioaeo ac. (tanggiaoyu2020, xianpei, sus.eeeissssneeeessssssee.... en lyffei.yy. f huang, shuide wywp:e/libabbaiii.....omm com
ABSTRACT
Retrieval-augmented generation (RAG) is key means to effectively enhance large language models (LLMs) in many knowledge-base based tasks. However, existing RAG methods struggle with nnwlledgeiintnsiiv reasoning tasks, because useful information required to these tasks are badly scattered. This characteristic makes it difficult for existing RAG methods to accurately identify key information and perform global reasoning with such noisy augmentation. In this paper, motivated by the cognitive theories that humans convert raw information into various struc- reasoning, we proposes a
tured knowledge when tackling knowledge-intensee new framework, StructRAG, which can identify the optimal structure type for the task at hand, reconstruct original documents into this structured format, and infer answers based on the resulting structure. Extensive experiments across various -nowledge-intensive tasks show that StructR achieves state-of-the-art perfor- mance, particularly excelling in challenging scenarios, demonstrating its potential effective solution for enhancing LLMs in complex real-world applications. com wwtp://www......aaaat..........o wtt.caattttttttttaaaaa
https: htttp:://
INTRODUCTION
With the advancement of deep learning technology, large language models (LLMs) have demon- strated considerable strengths in natural language tasks and are extensively applied in complex real- world scenarios (OpenAl et al. 2024. Yang et al., 2024ap.. However, they still exhibit limitations in factual tasks due to lack of comain-specific knowledge, real-time updated information, and propri- etary knowledge (Huang et al. 2023 Sui et al. 2024). To address this, retrieval-augmennt gener- ation (RAG) methods have been developed to effectively provide essential external knowledge (Yu cct al. 2022 Gao et al., 2024). Typically, RAG methods involve splitting original documents into shorter chunks, retrieving the most relevant ones based on the query, and then using these chunks to enable LLMs to generate reliable answers (Ma et al. 2023 Li et al. (202a)) Due to their strong per- formance through this straightforward process, RAG methods commonly employed in various knowledge- based dnestion-answert tasks (Shi et al. 2024 Wang et al.. 2024b).
Unfortunately, current RAG approaches cannot effectively handle kmowledge-intensse reasoning tasks due to the scattered nature of related information needed to solve these tasks (Kuratov et al. 2024 Yang al. 2024b). Specifically, knowledge-intensive reasoning tasks often require large amount of useful information which is dispersed across many locations in the provided docu- ments, meanwhile the model needs perform integrated reasoning after retrieving useful infor- mation Yangg et al. 2024b) Wang et al. 2024a). Taking financial report analysis as an example, given a large number of financial documents and the need compare the development trends of
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Question: Compare develapment trends of company A, B and
Scattered Knowledge Structurizer
Structured Knowledge Utilizer
Documents: lnc get lotal profit 290 und with value 1.25
Core Content: Financial negurtt of Company A, B,C
Quaesiione Documents
Struckired Krotshagge
OB Decomposition: Sub-Questions
Knowwadge Descripion
-ues-uustiic Each comgany' Subogiestion company
H+
Structured Knowledge
Hybrid Structure Router
Dom
Income
(CRO)
Tooe
Talgen
Loc
Stari
1996
1:25
MB Extraction: Precise Knowledge
For subqquaction 2
Fery
Londin
1996
1.25
B
Question Care Content
Optimal Type Table]
New
201l
1/0
350
390
790
1:25
22
1/2
%% Graph
Chunk
Catalogue
Algorithm
Table
Knowledge Description
X Inforance: Final Answer
Table includes inforctation of incoome, CRD, volue, location, stort time, valuee...
Inc. B has lnill Jesa value, but has much thus with the best trend.
Figure 1: The overview of StructRAG framework, including hybrid structure router to select the optimal structure type based on task requirements, a scattered knowledge structurizer to convert raw documents into structured knowledge, and structured knowledge utilizer decompose complex question and then effectively using the structured knowledge to infer the final answer.
multiple companies, LLMs need to dig out all relevant financcal indicators scattered across original documents and then generate insights by carefully comparing and conprehensively analyzing these indicators. such scenarios, standard RAG methods face challenges in accurately retrieving all relevant textual chunks, which may contain substantial noise, and integrating multiple key pieces of information for reasoning. leading to unsatisfactory performance on these tasks.
From a human perspective, people do not solve www.ledeeintensivee reasoning tasks by simply read- ing raw texts (Johnson-Lairy 1986 Paivio 1990). As suggested in cognitive load theory, humans typically summarize scattered information from documents into structured knowledge, which is then used to shorten the reasoning path and enable more accurate judgement (Sweller) 1988 Chandler & Sweller 1991). Furthermore, cognitive fit theory shows that humans prefer using different types of structured knowledge for various tasks, such as tables for statistical analysis tasks and graphs for long-chain inference (Vessey) [991] Umanath & Vessey 1994). In recent years, the rapid develop- ment of LLMs has laid the foundation for directly using these models to construct various knowledge structures (Li al.. 2023 Jain et al. 2024). Meanwhile, many studies suggest that LLMs share sim- ilarities with humans in how they utilize information and solve complex problems (Wei et al.) [002]] Li al.] 2024b). These inspire us to explore whether LLMs can adopt human-liks thinking pro- cesses to transform scattered information into various structure formats during inference, thereby better serving Enowledge-intensing reasoning tasks.
Motivated by this, we propose StructRAG, which employs a hybrid information structuring mecha- nism to construct and utilize structured knowledge in the most suitable format based on task require- ments. As illustrated in Figure [ the StructRAG framework consists of three modules designed to sequentially identify the most suitable structure type, construct structured knowledge in that format, and utilize that structured knowledge to infer the final answer. First, recognizing that different struc- ture types are suited for different tasks, a hybrid structure router is proposed to determine the most appropriate structure type based on the question and document information of the current task. Sec- ond, given that constructing structured knowledge is complex and requires strong comprehension and generation abilities, an LLM-based scattered knowledge structurizer is employed convert raw documents into structured knowledge in the optimal type. Finally, since questions in knowledge- intensive reasoning tasks can often be a complex composite problems that are challenging solve directly, a structured knowledge utilizer is used to perform question decomposition and precise knowledge extraction for more accurate answer inference.
The core aspect of StructRAG is the hybrid structure router's 's ability to accurately select the most suitable structure type for each input task. To equip the router with this capability, we propose a training method for the hybrid structure router. Inspired by successful use of reinforcement learning in training LLMs for decision-making tasks (Havrilla et al., 2024 OpenAl 2024). we employ the DPO algorithm to train the router module, which follows reinforcement learning principles without requiring additional reward models (Rafailov et al. 2023 Allam 2024). However, there is insuffi- cient training data for the model to learn how choose the optimal structure type, and collecting enough such data in the real world is also challenging. To address this, we introduce a novel pipeline for constructing preference training data that involves task synthesis, solution simulation, and pref-
2

---

Published as a conference paper at ICLR 2025
erence judgment to create high-quality synthetic data, thereby enhancing the router's ability to select the appropriate structure type.
In our experiments, we evaluate StructRAG across various sewwledge-intensive reasoning tasks and compare with several strong RAG baselines. The results demonstrate that StructRAG achieves -ott-offtthe-rt performance, with improvements becoming more pronounced as task complexity increases. This confirms that StructRAG a robust solution for addressing challenging knowledge- intensive tasks. Additionally, compared to recent Graph RAG methods, StructRAG not only exhibits superior performance across a broader range of tasks but also operates significantly faster on average.
RELATED WORK
2.1 RETRIEVAL-AUGMENTER GENERATION
RAG technology achieves good performance in the era of LLMs by providing external knowledge to assist answering questions and reducing hallucinations (Jiang et al. 2023) Asai et al. 2023 Li et al.) 2024c Gao et al. 2024 Chen et al. 2024). The initial strategy of RAG involves using a retriever to search for and retain highly relevant chunks from knowledge base based query, these chunks are then fed into the generation module as external knowledge, enhancing its performance (Qi et al.] 2019 Lewis et al., 2020 Gur al. 2021 Yu al. 2022). To improve RAG effectiveness, some approaches have introduced iterative RAG, proposing various enhancements such as query expan- 2024a Chan al. 2024 Shi et al., 2024), and others try to improve the corporation between retrieval and generation (Qian et al. 2024 Su et al. 2024
sion and rewriting (Ma et al.
2023 Li al.
Luo et al. 2024 Zhang et al.
2024). Although existing methods achieve strong performance on
mopti-hop tasks like HotpotQA, chunk-based RAG struggles with moowledge-nnteniig tasks (Wang al., 2024a). This is because chunks must contain excessive text noise and do not capture the interconnections among information, thus LLMs cannot effectively use augmented knowledge.
2.2 GRAPH RETRIEVAL-AUGMENT GENERATION
Recently, to assist L.MMs handling complex question-answertt tasks, some works introduce graph structures into RAG systems (Edge et al. 2024 Panda et al., 2024 Peng et al.] 2024). One kind of approach uses pre-built knowledge graphs, extracting subgraph based on queries, which are then encoded as soft prompts flattened into plain text for the generation module (Tang al. 2024 He et al. 2024: Guan et al. 2024p. Another kind of approach involves extracting entity-relation triples from given text documents based query requirements to construct graph structures, which are then used for knowledge augmentation (Fang et al.) 2024] Edge et al.) 2024 Panda et al.) [024]] Gutierrez al. 2024). Although these approaches significantly improve performance
multi-hop
question-answeringg tasks, they focus solely on graph- knowledge via the format of triplets, thus limiting their practical applicability in various domain and application of knowledge-intensing reasoning tasks.
STRUCTRAG VIA HYBRID INFORMATION STRUCTURIZATION
As mentioned, due to badly dispersed information in knowledge-inten intensive reasoning tasks, the tradi- tional retrieval module in RAG could retrieve chunks containing substantial textual noise, making it difficult for the generation module to extract useful information for inference. Drawing inspiration from cognitive theories on how humans tackle such tasks, this paper proposes StructRAG, which utilizes a hybrid information structurization mechanism to construct and leverage structured knowl- edge in its optimal form. Specifically, as illustrated in Figure T. StructRAC first employs hybrid structure router to identify the most appropriate structure type for the given task, and then employs a scattered knowledge structurizer to transform raw documents into structured knowledge in that format, and finally incorporates structured knowledge utilizer to break down complex questions into simpler sub-questions, enabling more accurate reasoning on structured knowledge.
Task Formulating. Knowledge-intensive reasoning tasks involved in this paper provide a question q and large set of documents D input, with the goal of deriving answer based the
3

---

Published as a conference paper at ICLR 2025
provided documents, which can be expressed as follows:
a Flg. D), where D pm
where is the number of documents, which exceed 20, resulting in a total token of up 200K. Thus, the most obvious characteristic of these tasks is that useful information is dispersed across the provided documents, requiring the model engage in complex reasoning based on large-scale rele- vant data. For example, when comparing the development trends of several companies using a batch of financial reports, the task necessitates retrieving various financial indicators scattered throughout the documents, followed by a detailed comparison of these indicators. This involves considering factors such as the relative importance of different indicators and the magnitude of numerical differ- ences. Consequently, knowledge-intensiee reasoning tasks present significant challenges.
Hybrid Structure Router. From a human perspective, when solving mnowledge-intensing rea- soning tasks, individuals tend use the type of structured knowledge that best matches the specific requirements of faced task. To this end, StructRAG incorporates a hybrid structure router R to select the optimal structure type. Specifically, the router leverages the question q and the core content C of documents D to make its decision and generate the most suitable structure type t, as it is impractical to process the entire set of documents at once.
t= R(g, C) where C = (c(()
1 Ji=1
(2)
The core content C' is the concentrate of the titles or the first few sentences from each document d()) In our work, there are five candidate structure types for five kinds of knowledge-intensse tasks: table for statistical tasks, graph for long-chain tasks, algorithm for planning tasks, catalogue for summarizing tasks, and chunk for simple single-hop tasks. Considering the core effect of the router in the overall framework, our work designs a DPO-based training method to develop a router that excels in knowledge type decision, which detailed in Section 4
Scattered Knowledge Structurizer. After identifying the most suitable structure type, StructR extracts the textual knowledge scattered across raw documents and reconstructs into structured knowledge. This process requires a comprehensive understanding of all raw documents and and precise formatting of the information, making it a challenging and flexible problem. Therefore, StructRAG employs an LLM-based scattered knowledge structurizer to facilitate the structurization process. Specifically, as shown in Eq. 3 the structurizer S takes the question g. the selected type i, and each raw document d( as input, and extract the structured knowledge k.(²) from the document via the powerful understanding and generation ability of LLMs. In addition, a description of the structured knowledge h2 is also generated.
( 1 = ((),
After that, all output structured knowledge will be collected as the overall one K, = {k; pm
1==1 and
the overall description of the whole structured knowledge is constructed as B, [b] pm Ji In term of detailed representation of each kind of structure, the table is by markdown, graph by in list of head-relationship-taa triplets, chunk is by regular text, algorithm by pseudo code, and catalogue is by text with hierarchical number (e.g., Section One, 1.1, 1.1.2) as explicitly chapter identifier.
Structured Knowledge Utilizer. After obtaining the structured knowledge in its optimal type, StructRAG performs reasoning to answer the question. Given that the question can be highly combinatorial, this may hinder the identification and use of relevant information in the structured knowledge. Therefore, StructR employs an LLM--ased structured knowledge utilizer to facili- tate question decomposition, precise knowledge extraction, and final answer inference. Specifically, the decomposition process of the utilizer takes the original question q and the overall description of structured knowledge B, as input, breaking the question down into several simple and intuitive sub-questions g(t), Next, the extraction process aims to find out precise knowledge k,(7) for each sub-question g(ú) from the whole structured knowledge K₁. Finally, the inference process integrates all the sub-questions and their extracted precise knowledge to generate a final answer a, which can be expressed as follows:
Q Ustcoompase (1, Bi) 1000) 1
4

---

Published as a conference paper at ICLR 2025
Seeding
Synthesizing
Simulating
Judging
Sckst golles uslo coreiitg all posable syes.
Goande kacw itgettttieee
Desefite solutions lo task different type kolge
Compure oud soluiic and preamic mining pam.
sade via invoret lammy
Golden
Synthedi
Synthetic Simulated Solutions
Preferance Judgments
1. Eruzscal sopats Micoosed:
Synthelic
proppecs their reference repes,
fly aike valae be dea celealtee Solution,, By yaph, dineser central rede and collacorazed sctura linking nede, then do check Solutiond, By clirk, retiove and red all chunks comtningg dinettorattr colleboration record. then gat snswer
disectur
colluhveated dove filex
diseoly telyfil. Generated Preferance Data::
will
diector
9. Lives tatch moding secord. simmarite the mais point ofe meetng 4. Glem cumpater oorgorents monull
Teak2 Given the timzline
100 nisurical event the 300 centrry Bed the evers the
(Task), Takle, Churk)
[Tuil Grogh Churk)
Figure 2: The illustration of training data constructing. First use LLMs to synthesize knowledge- intensive tasks, and then simulate solutions by structured knowledge in different types, finally judge all possible solutions and get preference pairs about candidate structure types.
R. = {Mextract( Ki)] = (1) 13=1
(5)
= Minter (4, Q K.)
(6)
where n is number of sub-questions, ② is set of all sub-questions, R. is whole precise knowledge for all sub-questions, and Maecompuses Mextract and Linter process of decomposition, extraction and inference, respectively. More details about the utilizer are shown in Appendix [A.3
4 HYBRID STRUCTURE ROUTER TRAINING
In the StructRAG framework described above, the core factor is accurately determining the most suitable structure type based on the input task, and the performance of the hybrid structure router directly influences the overall effectiveness of the framework. Therefore, performance router, we propose training method to enhance the ability of LLMs in identifying the suitable structure type of knowledge, Specifically, given the strong capabilities of reinforce- ment learning in decision-making scenarios, we train the router using the DPO algorithm, which achieves results similar to reinforcement learning while avoiding the need for additional reward models. Regarding training data, since there is no existing preference data for the optimal structure type selection task, we design a preference pairs for training. A detailed explanation is provided in the following paragraphs, along with examples and prompts in the Appendix A.
achieve a high-
method to efficiently construct
Data Constructing. Due to the scarcity of training data for selecting the optimal structure type method to construct in the current NLP community, we employ a illustrated in Figure given several manually preference pairs for training the router. Specifically, collected seed tasks that covering the possible structure types, we first use LLMs to synthesize a set of new tasks by the in-context learning method, where each task contains question and core context for documents. Then, for each synthetic task, LLMs is employed to simulate the process of addressing this task by structured knowledge in different types, thus getting different simulated solutions. Finally, a L-b--a judge compares these simulated solutions for solving the task, generating preference pairs regarding the structure types. Each constructed data entry includes a question, the core contents of documents, the chosen structure type, and the rejected structure type, as expressed as follows:
Dsymthetic = (x) c(k) (())
where tu and t are chosen structure type and rejected structure type, respectively. In addition, synthetic preference pairs include both English and Chinese data in order to improve the universality.
Preference Training. Inspired by the success of using reinforcement learning train LLMs for decision-making tasks, we employ the DPO algorithm to train the router module, which can get the same effectiveness reinforcement learning without adding additional reward models. Specifically, the input for training the router includes a question and the core contents of documents, and the out- put is one kind of structure type (e.g., "table", "graph"). As described in last paragraph, we simulate and construct a set of preference pairs for DPO training, which can be formulated as following:
To C. - log
) Tee (t: q, C*)
( t 9. C.
CoPo(To: ) = - -Eig D log o [glog Ter 1, C)
(8)
5

---

Published as a conference paper at ICLR 2025
where Th and Tet are target policy and reference policy, respectively, and 3 is hyperparameter. As analyzed later, this preference training enables the model to differentiate between various types of knowledge and their suitability for given task, resulting in better performance compared to zero-shot and few-shot settings.
EXPERIMENTS
5.1 EXPERIMENTAL. SETTINGS
Evaluation Datasets. This paper includes various chowledge-intensive reasoning tasks in evalua- tion. First, this paper chooses the Loong benchmark (Wang et al. 2024a). which includes four tasks (Spotlight Locating, Comparison, Clustering, and Chain of Reasoning) and four document length settings, as the length of the document increases, the useful information needed to solve the task becomes more dispersed. As for metries, this paper adheres original settings in Loong and use the official code repository in evaluation, involving using LLMs to decide score from 0 to 100 and the exact matching (EM) rate. In addition, this paper chooses Podcast Transeripts, which is query-focused summarization task reported by GraphRAG (Edge et al., 2024). As for metrics, this paper follows GraphRAG settings, involving head-to-head win rate by a LLM judgement, across four kinds of dimension, which are comprehensiveness, diversity, empowerment, and directness.
Implementation Details. We build framework based on Qwen2 series models (Yang et al. 2024a) For the hybrid structure router, StructRAG uses Qwen2-7B- --Intrrctt the base model and imple- ment DPO training by tr As for the details of hybrid structure router training, StructRAG con- structs and uses total of 900 preference data, setting the learning as 3 and the 3 as default in training. For the scattered knowledge structurizer and strutured knowl- edge utilizer, StruetRAG directly uses Qwen2-72B-Instee Instructt as base model and deploy models as API using vllm following the same setting as in Loong (Wang et al. 2024a).
1e-5, number of epochs
Selected Baselines. We select baselines from commonly used or recent methods for knowledge- based question-answerin tasks. Specifically, (1) Long-Context (Yang et al. (2024a), which extends the input window of LLMS to up to 128K tokens through extrapolation techniques, allowing large- scale documents to be directly input into the model.
(2) RAG (Lewis et al.) 2020), which splits the given documents into multiple short chunks and uses a retriever to retain only the most relevant chunks as augmentation based on the question.
(3) RQ-RAG (Chan et al. 2024), which uses trained LLM to decompose and refine the original question to more accurately find the required chunk augmentation. (4) GraphRAG (Edge al. 2024). which extracts triples (head, relationship, tail) from raw documents and constructs into multi-layered graphs, then uses structured information in graphs to help the generation model answer questions. In implement, for Long-Context Context and RAG, we directly follow the experimental settings reported in Loong (Wang et al. (024a)). and for RQ- RAC and GraphRAG" we evaluate the performance based on the official code repositories. Noting that, for fair comparison, we also set Qwen-728 72B-IIstrud as base model of GraphRAG.
5.2 OVERALL RESULTS
Results compared with baselines are shown Table and Table there are two main conclusions:
1) StructRAG is powerful solution to addressing knowledge-intensive reasoning tasks. Based on the experimental results in Table StructRAG outperforms the baselines in most tasks and doc- ument length settings, and in the overall metric, StructRAG exceeds all baselines in both LLM score and EM rate. In addition, as shown in Table 2 StructRAG achieves the best average performance compared to all baselines in Podcast Transcripts. All in all, these experimental findings demonstrate that StructRAG can effectively address knowledge-intensin reasoning tasks and improve a lot com- pared with previous long-context methods, and different kind of existing powerful RAG techniques.
wtaprihhhh..oooooeeeeeeeeeeee https:/typi.oo//noojee/llll hhtp:://yppiagghhooce///apaaa///

---

Published as a conference paper at ICLR 2025
Method
Spot.
Comp.
Clus.
Chain.
LLM Score
EM
LLM Score
EM
LLM Score
EM
LLM Score
EM
LLM Score
EM
Set
(10K-50K Tokens)
Lang-contest Yang
2024g
68.49
0.55
60.60
0.37
47.08
0.08
70.39
0.36
60.11
0.29
RAG (Lewis of 20A
51.08
0.35
44.53
0.27
37.96
0.05
53.95
0.35
46.11
0.25
2024
53.51
GraphRAG [Lage et al 2024
31.67
0.00
27.60
0.00
40.71
0.14
$4.29
0.43
40.82
0.18
StrictRG CAARRAAGOO (007))
74.53
0.47
75.58
0.477
65.13
0.223
67.84
0.34
69.43
0.35
Set (50K- 1O0K
Long-pontext Yang et
2024g
64.53
01.43
42.66
0.21
38.52
0.05
$1.18
01.20
45.71
0.17
RAG (Lewis
66.27
0.46
46.28
0.51
38.95
0.05
46.15
0.22
45.42
0.19
RQ-RNG
47.09
GriphRA AG ArrPagge (Pgge et
24.80
0.00
14.29
0.00
37.86
0.00
46
0.12
33.06
0.03
StructRAG (005)
68.00
0.41
63.71
0.36
61.40
0.17
54,70
0.19
60.95
0.24
Set (100K-200K Tokens)
Long-coutett Yang et 2024a
46.99
0.27
37.06
0.13
31.50
0.02
35.01
0.07
35.94
0.09
RAG (Lewis
73.69
0.55
42.20
0.27
32.78
0.02
37.65
0.13
42,60
0.18
RQ-Rht Lal
0.13
GraphRAG (Blge et 2024
15.83
0.00
27.40
0.00
42.50
0.00
43.33
0.17
33.28
0.04
SmrieRAG (OUS)
68.62
0.44
57.74
0.35
58.27
49.73
57.92
0.21
Set (200K-255) 250K Tokens)
Long-comtest Yann et 20240
33.18
0.16
26.59
0.08
29.84 29.84
0.01
25.81
0.034
28.92
0.06
RAG (Lowis
RQ-RAG
ai 2024
29.17
0.08
40.36
0.00
26.92
0.00
34.69
0.00
31.91
0.01
GriphRAt [inge et 2024
17.50
0,00
26.67
0.00
20.91
0.00
33.67
0.33
23.47
0.05
StructRAG (O65))
56.59
35.71
0.10
Table 1: LLM-judged scores and exact matching rate in phowledge-intensivy tasks of Loong bench- mark. From Set to Set 4, task complexity gradually increases, as reflected by the growing token number of documents. The table show two main conclusions: StructRAG get the SOTA performance in overall metrics. And the more complex the task, the greater the improvement of StructRAG.
Compared Method Pair
Comprehemsivnees
Diversity
Empowerment
Direciness
Average
StructRAG vs. Long-contest Yane al. 2024g
98
97
92
95.75
StructRAG vs. RAG (Lewis et MAM
51
52
62.00
StructRGG RQ-RAU
2014
67
50
46
59.50
StructRAG vs. GraphRAG (Edge et
2024
42
41
53.00
Table 2: Win rate of StructRAG vs. baselines on Podcast Transcripts. StructRAG achieves the best average performance compared to all baselines, further conforming effectiveness of framework.
2) StructRAG is particularly suitable for complex tasks, performance improvement becomes more significant in scenarios with more dispersed information. Based on the overall performance in Table [ the performance comparison between StructRAG and the long-context baseline shows that StructRAG achieves performance improvements approximately Set 2, Set 3, and Set 4, respectively. Similarly, comparing StructRAG with RAG shows performance improvements of around 15, 15, and 22 on Set 2, Set 3, and Set 4, respectively. Each set represents the total length of the given documents, with Set 1 being the shortest and Set being the longest. This means that the information needed to answer the questions becomes more dispersed as the length of the documents increases, and making the reasoning process more challenging. Therefore, these results indicate that StructRAG shows more significant improvements over the baselines with longer documents and more scattered information, demonstrating that abilities of our framework to construct and use the optimal type of structured knowledge is especially effective for complex tasks.
15, 22, and 23 on Set 1,
5.3 ABLATION RESULTS OF MODULES
To validate the role of each module in StructRAG, we perform ablation experiments. As shown in Table 3 "w/o router" refers to random routing, "w/o structurizer" means using only chunks, and "w/o utilizer" refers to directly concatenating the structured knowledge with the original question for answer generation. There are following conclusions:
1) All three modules contribute positively to the overall framework. The table shows that remov- ing any of the three modules results in a noticeable performance decline. The overall performance will reduce from 60.38 to 45.33, 53.92 and 55.94 for the router, structurizer, and utilizer, respec-

---

Published as a conference paper at ICLR 2025
Method
Set 1
Set
Set 3
Set 4
Overall
LLM Score
EM
LLM Score
EM
LLM Score
EM
LLM
EM
LLM Score
EM
StreeRAG
69.43
0.35
60.95
0.24
57.92
0.21
51.42
0.10
60..88
0.23
w/o router
$1.09
0.28
48.28
0.17
39.52
0.13
41.83
0.10
45.33
0.17
structurizer
64.97
0.29
52.17
0.17
0.19
44.24
0.10
53.92
0.19
w/o utilizer
68.23
0.29
59.73
0.24
$3.29
0.19
35.77
0.10
55.94
0.22
Table 3: Ablation results of three modules. The table shows that all three module are with positive contribution, and the most core module is the hybrid structure router.
LLM Score
Se12 Se14
Method
F1
ACC
Sel S2t-
S21
0.31
Hybrid Structure Router
93.42
94.38
64
Owen2-7B-Instman (zero-shot)
50.04
54.25
Qwen2-7P 7B Instruct (few-shot)
65.59
66.
0.01
Qwen2-72B, 2212-722--Intruuct (zero-shot)
78.38
80.56
Randon
Weak
Rocie:
Our
Rosic:
Bul
Runkon
Rosic
Bouss
Week
Oue Bosser
Qwen2-72B-Ing Instruct (few-shot)
89.39
90.06
Figure 3: Performance of StructRAG with dif- ferent routers. The strong router shows obvious positive impact on the overall framework.
Table 4: Results of evaluating hybrid structure routers. The table shows that preference training is necessary for the routing ability.
tively. This proves that all three modules play irreplaceable role, and StructRAG tightly and orderly combines these three modules to achieve excellent overall performance.
2) Choosing the suitable structure type and constructing documents into structured knowledge are more crucial than designing complex utilization methods. comparison the table reveals that different modules are with different importance. The performance drop most significant when the router is removed, with a decreasing from 60.38 to 45.33. In contrast, removing the utilizer leads smaller performance decline, from 60.38 to 55.94. This indicates that simply question-refining as existing methods provides limited improvement for pnowledge-intensee reasoning tasks, more promising direction constructing and using structured knowledge in suitable type.
5.4 DETAILED ANALYSIS
In this section, we do some detailed analysis, including, performance and impact of the router, draw- back of using fixed structure type, case study about EM rate performance, and efficiency reports.
5.4.1 EFFECT OF THE ROUTER
To explore the necessity of constructing data and conducting DPO training, and relationship between performance of hybrid structure router and overall StructRAG, we first compare our router with raw LLMs, and then draw curl of router and overall StructRAG score. There are following conclusions:
1) Selecting the optimal type of knowledge based on the task is challenging for raw LLMs without special training. Based on the experimental results in Table the router trained based on D-wen2-7B-Insteee indicates that L.MMs need some special training to get the ability of selecting the optimal structure type based needs of the task. even when the model scale reaches 72B.
model significantly outperforms the 72B model with few-shot setting. This
2) The performance of bybrid structure router with significant relevance with the final per- formance of StructRAG. As shown in Figure 3 we select Sreen--72BInntruuu (zero-shot) as the weak router, and design a completely random router and completely incorrect bad router. The curves in the figure clearly show a positive correlation between router accuracy and the overall per- formance of the StructRAG framework. This further demonstrates that selecting knowledge types that match the task needs for augmentation is crucial in knowledge-intensice reasoning tasks.
8

---

Published as a conference paper at ICLR 2025
Method
Set 1
Set 2
Set 3
Set
Overall
Score
LLM Score
LLM Score
LLM Score
EM
Score
StretRAG
69.43
0.35
60.95
0.24
57.92
0.21
51.42
0.10
60.38
0.23
w/ only table
48.00
0.23
55.19
0.24
50.35
0.19
38.
0.12
49.66
0.21
only graph
w/ only chunk
64.97
0.29
52.17
0.17
53.18
0.19
44.24
0.10
53.92
0.19
w/ only catalogue
30.49
0.10
36.36
0.13
36.77
0.12
23.75
0.03
33.26
0.10
w/ only algoritlm
43.53
0.24
32.86
0.08
31.59
0.13
16.67
0.04
32.32
0.12
Table 5: Results of only containing structured knowledge in one fixed type. shows any single fixed type is insufficient, confirming the advance of StructRAG via bybrid information structurization.
Method
Constructing
Reading
Total Latency
depreciation 1,308,463 and share-haser compensation expense of $ 537,197) in 2024
Raw
RQ-RAG
7.8
1.2
9.0
GraphRAG
215.3
1.8
217.1
StreeR (Ours)
Structured
2006
July
133888
Figure 4: Comparison of implementing latency (minute). The StructRAG has an available speed, which is little slower than RQ-RAG, but is much faster than GraphRAG method.
Figure 5: Some cases to show the reason why EM rate not perfect, because of a little differ- ence in textual format after structurization.
5.4.2 DRAWBACK OF FIXED TYPE OF KNOWLEDGE
To further verify the importance of containing hybrid types of structure rather than fixed type, we freeze the structure type used in the framework as either chunk, graph, table, algorithm or catalogue for all evaluation tasks. There are the following conclusions:
Using single fixed type of knowledge cannot achieve good performance on diverse tasks. Based on the experimental results in Table shows that for both scores and exact matching rate, using single fixed type performs input task. In comparison, the performance degradation least when only using chunk, with a reduction from 60.38 to 53.92, in other cases, the performance decline is more significant. This can demonstrate the effectiveness of cognitive fit theory in LLMs,, meaning that using structured knowledge in the most suitable type can effectively enhance problem-solving abilities.
than selecting the optimal structure type for needs of the
5.4.3 CASE STUDY ABOUT EM METRIC
According to the experimental results in Table 1 StructRAG surpasses baselines in general score, falls short seven sub-situations for the exact matching rate. Therefore, we analysis some cases that StructRAG method gets high score but fails exact matching. The reason is mainly about structurization process may alter the textual format original information. As shown in Table 5 there are some wording differences between structured knowledge and information (e.g. from original "$ 1,308, 463" to "138463" in the table). Intuitively this aligns with the common sense, where structurizer is probabilistic language model rather than rule-based model, thus some possible textual loss may be unavoidable, and output from GraphR method also show similar issue.
5.4.4 EFFICIENCY REPORT
In this section, we report average latency of StructRAG and compare with the RQ-RAG CCan et al. (024) and GraphRAG (Edge et al. 2024). The latency includes two components: The first part is constructing latency, referring to the process iteratively retrieving chunks for RQ-RAG, constructing graphs for GraphRAG, and determining the optimal knowledge type and constructing corresponding structure for StructRAG. The second part is reading latency, referring to the process of using augmented knowledge to generate final answers. As shown in Table 4 StructRAG has slightly higher latency compared to RQ-RAG but is obviously faster than GraphRAG. Therefore, StructRAG is kind of high-performance framework with available implementing speed.

---

Published as a conference paper at ICLR 2025
CONCLUSION
In this paper, noticed the limitation of existing RAG methods knowledge-intensivee reasoning tasks, and inspired by cognitive theories about how human beings solve such tasks, we propose new framework, StructRAG via hybrid information structurization, which can construct and utilize structured knowledge in the optimal format as augmentation. StructRAG includes a hybrid struc- ture router to precisely select the optimal structure type, then a scattered knowledge structurizer convert raw documents into structured knowledge, and finally structured knowledge utilizer to decompose complex questions and infer the final answer via the constructed structured knowledge. Furthermore, in order to get a nigh-performance hybrid structure router, we construct training data by simpesiaiiin--mullaiiiinnngg
pipeline and then implement preference training via DPO algo- rithm. Experiments on extensive knowledge-intensiee reasoning tasks demonstrate that StructRAG an effective solution, which reach the SOTA performance and achieve large improvement in badly scattered scenarios. Therefore, this paper offers a promising direction, focused on hybrid structured knowledge, for developing more powerful RAG systems in the future.
ACKNOWLEDGMENTS
We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work was supported by the Natural Science Foundation of China (No. 62122077), Beijing Nat- ural Science Foundation (1.243006). Beijing Municipal Science and Technology Project (Nos.
1231100010323002) This work was supported by Alibaba Innovative Research Program.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ahmed Allam. BiasDPO: Mitigating bias in language models through direct preference opti- mization. In Xiyan Fu and Eve Fleisig (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp- 71-79. Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/w1/2024.aclss URL http:sss//aaaaatttollooooooo//os
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications. In Yun-Nung (Vivian) Chen, Margot Margot, and Siva Reddy (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tuto- rial Abstracts), pp. 41-46, Toronto, Canada, July 2023, Association for Computational Linguis- URL https://aclaattlllggg//o
tics. doi: 1/20333/1122222cc-uuuuao sol-tutorials..
Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. Rq-rag: Learning refine queries for retrieval augmented generation, 2024. URL https://arxii> org /bs/abs// abs
2404.044000 00610
Paul Chandler and John Sweller. Cognitive load theory and the format of instruction. Cognition and Instruction, 8(4):293-332, 1991, doi: 10.1207/s1532600xx66688
URL https:
2
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmeented generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38 (16):17754-17762, Mar. 2024. doi:
237216. URL httttt:://ossaaattt aaji.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summariza- tion, 2024. URL https::
org/ hbtpg//abs/44000 16130
Jinyuan Fang. Zaiqiao Meng, and Craig MacDonald. REANO:
Optimising retrievallaugmmentdd reader models through knowledge graph generation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. Association for Computational Linguistics. URL
Bangkok, Thailand, August 2024. ortph//0
acl ong. 115
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augnmented generation for large language models: A survey, 2024. BtLLbttss: httpps://aariiv.org/aa/////11/1/
Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le Sun. Miti- gating large language model ballucinations via autonomous knowledge graph- based retrofitting. Proceedings of the AAAI Conference on Artificial Intelligence, 18116):18126-881 1813,, Mar. 2024. doi: v0.1609/aai..v88ll 29770.
URL https:: //tp:///////// aaai wot.oorgiiiieee.e
http:////iaaa....////e.ee... phpp /AAII/
article/view/22770
Shir Gur, Natalia Neverova, Chris Stauffer, Ser-Nam Lim, Douwe Kiela, and Austin Reiter. Cross- modal retrieval augmentation for multi-modal classification. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.). Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pp. 111-123, Punta Cana, Dominican Republic, Novem- ber 2021. Association for Computational Linguisties. doi: 122.86653/112021.nnmmmm findings-emnlp.111 URL httphhpppp/acccaattooootcoooooee hwwp:/hwwwoooooooooooooooooooooe 2021. .indingss-enll htt.://ow.....o.o...oooo..oonnna
Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. HippoRAG: Neuro- biologically inspired long-tec term memory for large language models. In The Thirty-ighth Anmual Conference on Neural Information Processing Systems, 2024. URL ht h/tppp.ttttstt //tp::////peeeiiew.. met./oorumiid=haee jvAPVsg
11

---

Published as a conference paper at ICLR 2025
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning, 2024. URL https://arxi::
org/abs/ 2403. 04642
Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bres- son, and Bryan Hooi. G-retriever: Retrieval-augmeentd generation for textual graph understand- ing and question answering. 2024. UtLhttps::/ pttttpps///axiiv..oo org/ abs / 220. 076600
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023. URL https:
05232
Parag Jain, Andreea Marzoca, and Francesco Piccinno. STRUCTSUM generation for faster text comprehension. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7876-7896, Bangkok, Thailand, August 2024. Association for Computational Lin- guisties. doi: 10. 1/tp:33//1/2222lllllol ac long.o2g666 426
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 1969-7992, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10. 10.18////112023 wttp:///.2..mmmmmai..... wwn.l-maii.mnn9
URL https:
//aclanthology/ wol.antthoooooy.orr/22222223 emn P main. 495
P. N. Johnson-Laird. Mental models: towards cognitive science of language, inference, and con- sciousness. Harvard University Press, USA, 1986. ISBN 0674568826.
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of Ilms with long context saapainn-ii---hysstaee
2024. URL httpt:::////// //arxiv org/ abs/ 22./244....1 10149
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kicla.
generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 9459-9474. Curran Associates, Inc., 2020. URL https::/ppoceedd.nggg/. neurips wwtpccppppppp..eeeeeeee www.ffllppppa halllee/ppper/22222 File/ /ttp:/66982200017000026344444843
Tong Li, Zhihao Wang, Liangying Shao, Xuling Zheng, Xiaoli Wang, and Jinsong Su. A sequence- so-sequence&&set model for text-to-table generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10.18653/1122233 findings-acl.333 Findings acl. 330
2023, pp.
WRLhttts:://ad hhttts:://aeaatthlllogg.. hotg://coooroorrrr/222.....3 org/2023.
Xiaoxi Li, Zhicheng Dou, Yujia Zhou, and Fangchao Liu. Corpusim: Towards unified language model on corpus for knowledge-intensive tasks. In Proceedings of the 47th International ACM
SIGIR Conference Research and Development in Information Retrieval, SIGIR '24, pp. 26-37, New York, NY, USA, 2024a. Association for Computing Machinery, ISBN 9798400704314, doi: 10.11457366772.365777 URL URLpttpss: //ooi. www.oo.../......... 0tt.///11.11453366767171 76577777777667777887
Zhuoqun Li, Hongyu Lin, Yaojie Lu, Hao Xiang, Xianpei Han, and Le Sun. Meta-cognitive analysis: Evaluating declarative and procedural knowledge in datasets and large language models. arXiv preprint arXiv:2403 203.00000 2024b.
Zhuoqun Li, Hongyu Lin, Tianshu Wang. Boxi Cao, Yaojie Lu, Weixiang Zhou, Hao Wang, Zhenyu Zeng, Le Sun, and Xianpei Han. Url: Universal referential knowledge linking via iastiinstructed representation compression, arXiv preprint 4404v:2404. 16248, 2024c.
12

---

Published as a conference paper at ICLR 2025
Kun Luo, Zheng Liu, Shitao Xiao, Tong Zhou, Yubo Chen, Jun Zhao, and Kang Liu. Landmark embedding: A chunking-fire free embedding method for retrieval augmented long-conteds large lan- guage models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp.
Bangkok, Thailand, August 2024. Association for Computational Linguisties. org/ 2024 acl long. 180
URL Mittps:/accaatthot
Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg. and Jianfeng Gao. Chain-of- skills: A configurable model for open-domain question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (cds.). Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 1599-1618, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.186653/112222.addloon..g..
URL
2223g/2023aa2 acl- 89
OpenAl. Openai ol system card. preprint, 2024. URL
card/
OpenAI, Josh Achiam, at al. Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. Gpt-4 technical report, 2024. URL
wwg. abs attp///s.//33..../// 2303308777
Allan Paivio. Mental Representations: A dual coding approach. Oxford University Press, 09 1990. doi: 10. 00.00////cootss00777001166666660 100000000066...00000000...0.0...
ISBN 9780195066661.
URL https://
doi ooi.org/10.. 10.000.1199//.. 1ttp://0....3///cooffoooo :o0.: 0707778011066666111 0010 0001
Pranoy Panda, Ankush Agarwal, Chaitanya Devaguptapu, Manohar Kaul, and Prathosh Ap- HOLMES: Hyper-relational knowledge graphs for multi-hop question answering using LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13263- 13282. Bangkok. Thailand, August 2024. Association for Computational Linguisties. URL
acl-ooooo. 717
Boci Peng. Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong. Yan Zhang, and Siliang Tang. Graph retreeeel-augmeeted generation: A survey, 2024. URL https: //arxiv./
org/abs/2408 08921
Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. Answering complex open-domain questions through iterative query generation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing DOMNLP-IJCNPP) pp. 2590-2600 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1tt/:////w//1911111 1261. URL
brg/D19-1267
Hongjin Qian, Zheng Liu, Kelong Mao, Yujia Zhou, and Zhicheng Dou. Grounding language model with chunking-free free in-context retrieval. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1311-131,, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL Ltl.ittsss/acclantooooyyyy www.aaawwwhoooooo..eee/e2/ee/e acl- 71
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 13728-53774 Curran Associates, Inc., 2023. URL https::/proccest
neurips cc cctccc///aaaa http//aper___ee httpprrsfillsspppe//22222 Saaprea---ppp-C-Ceeeeeeeeeeeeee F
file
Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. Generate-then-grouud in retrieval-augmeented generation for multi-hop question answer- ing. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7339-7353, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://eet antho log.. w0w.or//222202 acll244ccll--- lon....oooooooooooooooooo 397
13

---

Published as a conference paper at ICLR 2025
Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Lun- the 62nd Annual Meeting
Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12991-13013, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https:
//aclanthology. www.ollgy..gg////0. ael- 702
Peiqi Sui, Eamon Duede, Sophie Wu, and Richard So. Confabulation: The surprising value of large language model hallucinations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pp. 14274-14284, Bangkok, Thailand, August 2024. As- sociation for Computational Linguistics. doi: 10.1 11118://1/2222..allloee -ono24,.cclooo.77000 770. URL https:
//aclanthology.
770
John Sweller. Cognitive load during problem solving: Effects on learning. Cognitive Sci- ence, 12(2):257-285 1988. ISSN 0364-0213. doi: https://o.loog/00000000000444440
90023-7. URL http:://www....eneeeeeeee..e.ees
0364021 90020337
Jiabin Tang. Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuming for large language models. Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '24, pp. 491-500, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704314. doi: 10.11457326777.366777
URL https://Gci.org: 1145/
3626772. 3657775
Narayan S. Umanath and Iris Vessey. Multiattribute data presentation and human judgment: A cognitive fit perspective. Decision Sciences, 25(5-6):795-824, 1994. doi: http:://doi.or//// www.wwwww....................... cwm/ doi/
URL httpptt:/oonnieeeelllee abs/10. 1011// 5915--9595,..0 1994 tb01870.x
Iris Vessey. Cognitive fit: theory-based analysis of the graphs versus tables litera- ture. Decision Sciences, 22(2):219-24)) 1991. doi: http://dbb.org//..1111114444 1111/
1991.bb00344.x URL htttps::/coninelllee j.1540 5915. 1991. 100914..00344..
wiley.
Minzheng Wang. Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yonghin Li. Leave no document behind: Benchmarking long-context Ilms with extended multi-doc qa, 2024a. URL
bettps:/arxxv.orxt abs/ 1.406.17411
Zheng Wang, Shu Teo, Jieer Ouyang, Yongjun Xu, and Wei Shi. M-RAG: Reinforcing large lan- guage model performance through aatrivaalauugmette generation with multiple partitions. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meet- ing of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1966-1999999 1978, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 18653/ v1/2024. acloo4..l--oog.1100 108. URL https: ftttps:/aaclathaate http://aaaaaattooooo.oo.o wwt.oogg//2222 2024 acl moce-llong.. 108
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Lc, and Denny Zhou.
prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824 24837. Curran Associates, Inc., 2022. URL lattps://proceddinn wwwponnggnnuuueuee nwwrips ctcessccc/ppee wwtpppppppppppeceeecccccceeee..e httppaaper_ieeepppeeeeeeeeee www.s//aape/ee/./22//222 abcat Comppeerccoffeencceee.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng
14

---

Published as a conference paper at ICLR 2025
Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL httpss:/aaza
org/ http/a///////oo///t////////e 0tos/2407.106.
Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nico- las Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar. Wen tau Yih, and Xin Luna Dong. Crag comprehensive rag benchmark, 2024b. URL https:://eett https///xxvvvoo///ee// 2406. 04744
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. A survey knowledge-enheee enhannee text generation. ACM Comput. Surv., 54(1ss), nov 2022. ISSN 0360-0300. doi: 10.1 145/35124677
//doi httg///oooor////////////oo 0. 1114535512467
Peitian Zhang, Zheng Liu, Shitao Xiao, Zhicheng Dou, and Jian- Nie. A multi-task embedder for retrieval augmented LLMs. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Pro- ceedings of the 62nd Annual Meeting the Association for Computational Linguisties (Volume 1: Long Pupers), pp. 3537-3553, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL ht https: //acl www.clanntthlooooooooooooooooooo wwwethhwoo......o ogy. www.yorrg/022..e.m acl long. 194
15

---

Published as a conference paper at ICLR 2025
Prompta in Decompasing Sub-queation
Isiratio::
Toselre die complix donsncs prokan, you neel a book dovn in siver qaay ino makkys rullicly axpli incoonnionn wippposkoo.
Kaawildge lnfo douripiis ofe aradtral kanwkeig inden aieeaaion, wisc refe lo for bookiown. 21r givin quesion aboady ungu earagh er canon todkre doon aral thin sa braaution recestry.
Kecelodge inee
causo "ludgrent Decanent "Malgnuu Docuraee "Jadgrenn Decunent 'udgpren Docunent "Jalgnor Documne "Jadgrent Decanen: "Juggnett....
Paase charty the sbove selazee decanent accerding as de so cakgrria Uheennkip Dippai; Advmattrtiee Baby Labee and Socal Seaurty Adszzimttton (Laber, socal Socurtyf "Eaactien Caame- Nen dagatice Adbusurmaie Esceator Erahoukkenn Britary; Esuezies Caase Chasp,, Adtenimann Actien Aditammiie Prynent cupua the jadanes: docunnn. comggaal wotal julpeet fastal Admiriatation (taker, Secial Secvtty? ("hoggrer Decementa" "Jalgnen
roperd
(//rere Dégoe ["ladgrew
"Jlgger (P]
Cepuo
puggrees
gives jegrure
docanveff Tracctior Crese New Miriiventive Encorter desmii whether jadgurn doconiifscca lirdborte and fithery conerine oheter the jelgnine docmmetf case Tiocattoo Chase Other Cresed, cournces cheturrte givee foommett case interoorati Action Adriseeratia syrean
Socal Securty desime
Knce lafe (kaznkolee.iii)
Quey (gec))
Prompts Extracting Prectse Knowledge
Larndt thes erveneed incededge hased
sth-quamtaa:: quecton)
Outra::
Prompts Inferencing Final Answer
Meging wlisett-gggeee precse lowwledge to Ginl unowe.
(paesios)
Preise Knowlelge (pensse_lwwwkktee kawklggr)
Figure 6: Prompts used in structured knowledge utilizer, including decomposing sub-questions, extracting precise knowledge, and infering final answer.
A PROMPTS IN STRUCTRAG
A.1 PROMPTS IN DATA CONSTRUCTING
Considering the current lack of training data that determines the optimal structure type based on task requirements, and the difficulty of collecting such data in real-world scenarios, we designed a pipeline for synthesizing, simulating, and judging data for DPO training specifically aimed at a hybrid structure router. In terms of implementation, created several prompts to drive the three processes of the LLM, as shown in Figure g
A.2 PROMPTS OF SCATTERED KNOWLEDGE STRUCTURIZER
Considering that constructing various structured knowledge from large scale of scattered informa- tion is complex task that requires strong comprehension and generation abilities, and that LLMs have demonstrated good capacity for integrating structured knowledge in previous work, we de- signed prompts to drive LLMs in achieving this process, as shown in Figure
A.3 PROMPTS OF STRUCTURED KNOWLEDGE UTILIZER
Considering that questions in knowledge-intensive reasoning can be complex combinatorial tasks, breaking them down into multiple simpler sub-questions can leverage the structured knowledge more effectively for reasoning. Therefore, we designed prompts to drive LLMs to achieve question decomposition and precise knowledge extraction, as shown Figure
16

---

Published as a conference paper at ICLR 2025
Prompts in Constructing Table
Acpording recul
deseribed hote that whtn consmucting the tooic, shouls include tible the and source informaiion such as which company report the tabls corres from.
complets
Content.
First, identity by the seywords in the Recuirement, including antity names and ory buto na namcs,, and thon use Theso kowwords cxtract information from the Fow Content. the Row ntain information required Requroment, cxtroct only priount information moiitain
relovint Roqul
extracting
longuoge..
Exomolss
pott tnsorn bearing depoate hunked eo madta cash apasaleem 122 epsccantin aratlall sall, Burrvatae valadd 203, 340 neaer hald trvotrairs, alloorancce
aed and stes 935 AV' Accomad uasme mochsabuut Ai Feddral Uores Lean Beas (TTIL slcckss (aakerrs Rosrva Blacc ("RI vlock 3.543aaiiiic prrpertio and aq aquipeaaee, sut te ownnd txurmoe, oets 0,26k apDuttmed su sacts, RJ27/auCon depont trangiii,, not SJATTpa je fite awurags J40qu auutgg 1.701-644
Row Decameet: (dcereee))
(ppnry
Prompts in Constructing Graph
Acconting nog roguteeaan desorbeed Rege hoesee
Crener
The ohorit he curpet cn one tow in rimai (Thead" oi tibtree toath (-- 1-11-113 Newe seractin reples thee tms, andlyre mntati pudi go entrien mentceed the Recaborarr and only sareu milenmmple Adtesuuiity,
jas siggle phosse ray paograple
whthee attoovtaton..
Exemmlles:
(1"ame" "LLMEEVid: Uniited trolaaton Franwwert dhe Decropting trzhaacia LLMd) Vulurantity Aearcning" "webatee" "sefirmace" "aal" "Way Ean GPPT Lean Cicheou Langrage Moddle Inpek Parfon Crrlien Dewere es Mexa- Mex-Cytinsiens "Cas Lirge Languege Models Be Alteraaiie Horon Bookutions"]] sa6as-ce
Row Decameet:
(Aoerera))
Outna:
Prompts Constructing Algorithm
Intrution:
Requienent cstrad accosery alasilas pecelooo Cieket
Yeu dout docclled cassut soulvel using pendnoods de net foree Inkedd, single informtiie that sddros regarcnees.
Evee origial desrent.
Sote If rerepieenntteee
anstette
Ivlaator corpeeees
frad ligh Semunnce mabiicor CPC
thavedi "Havei Alpenverz masttasing tharral ranagemaat"
thavei Maxac Fue. 128mma "1200mm slantad cooling tas"
Pue. :4lane- "jlean auheellsknee lax"
Evakati roqurcnsss:
high astrod daoley
adoaal CPU last Anaeniing to origiaa oxtos, laad CPU perfammnce
Raw Decument:
(docnarra)
Constructing Catalogue
Instaction:
Aucenting sogutemans dosntied Requieeent suctine, pornet saceomar dractry emcars benechical auarary tevds and number of sodes deurrnee
Yeu raqaied tellvr dexild
and prosided the Etuupplos. Make that cuch lovel the
Meratil distngaasiing detirne lends, sumunary
New: extnut inlenani informatin pouitle lives Ras Conteri, incidina exity namen chanacer runce mentoond Rouitmmen, prnruct campltt dieeasey struciass.
Examoles:
sacasace
Sanenay Al Tedinnaage Hequaaser Chalienazs
pentippleen Fiercial Services
reietin legal fronowaros, with paricoie privacy
rechodlggii
Seowet Summrry Regubarry wit as Modiide hightiges concame the the carser reg laver antinmnee fiurcia onn losiiss is sur wall salied addiow snagu thitinngs paoad All dartc oncemainyy commenting apolianilin eccinng regitatioe.
Regaboary
Meeilii dicosse how lack of clartyy regabaiine shiins
abitty iedattrss mtnnca daaly hamwir poseenal secoollggio.
(docerurac
Query (pers)
Figure 7: Prompts and examples used in data constructing for training the hybrid structure router, including synthesizing tasks, simulating solutions, and judging difference solutions.
17

---

Published as a conference paper at ICLR 2025
Prompts Synthestzing
Inssutio::
docannee guotkneesoveiing tolo, caupary quotian kn sispookpp sooiiiaa epicad smrig sehring topo cooiie tropl docancn ina maklgh indeyondas
vaery gpodiees sal camlying cospeiing frascal dat of (fallipp ceppriii optinal sinagg provee dee insersaiinn docunnnt the fore take, for een gureden u eilets, sudics eacling ancebli conquán foor lege beck diffeet mokks and anspehot speiffc wa socls (he (hcepiinal stragg in prowet the pnteratton decerser the form alpottere suneary quanton,, sith senzurtong a large menig rocret mostin eammary, optraa srraugy is otganane
pepes. sptiriaal
informiie ie dooanen le furn goopl
Recsisranne
A diventt contan to penamited conetina varkea and socnatio.
2 Uao DOCUMEENTS INP0 and QUiSIty maaiors, with suparate botwees
Eaumple
DOCT MINTS Ihityy
Thachmeen Life: " uue"" hn aee. "D Vincce Liee "Gaiteon LIB", "Noreink L1o"
QUIRY::
sconding glonn
DDC MINTS INDO-
2001 Fusncea Repor Nartte theancia Report Altato zuct louncaal Report Teazrri
QUERY:
with
lightnoone compesics hiliiee los
DOCUMENTSS, INPO:
Aagrocied
Kaowkeige linessiee NLP Deno Phosege Revivel for Qusilon Arowering DeDe
Undesradnng Exploring Treesfel
Unifiel Tou Tee Traesfrmee kudERTe HERT ALBERTI BERT
seperiied sarnin Languee Pouge Regessntatios
QURRY
refornce and adjt anceg the pivce decements
Osgea:
Prompts Simulating
Incuutio:
publess
ope laneniddg..
Requismatt
The of indeda prati dgarillss cnalage takk cerreppeding ooloodldggs
subiiin dailyy Fullow the ougut fered the Eurplees with arch willice suppasto and cadosbiiinn beginning witl del chesen topo laengg
Earrpio:
Buad en the decarieno abono, sammunter the devdopnuns bonds of the company over the puat tve yarr.
Guppat
bext compuay's
developpuus trzad. Fauily will nanamatic the campuay's devcaprent toails ever pual
and intermation Ean aductaal daakaa
2 Grsps: To baild and uas anapleii laan adas. lnce to proce, de infermiii foon the dicenceo the dees st's snads, with the ouspaty sand the beal aods, finacid cattica is the relaccenkko. und
orneppociing coocill
modes Fine
veloours company wigraya buad the qaery mone to congany's droolgwee aeed is over the part two yoars haudd the inforra
Table The hyroph
nal knoededge,
prnsee inforaatis
decoramas sable Some,
compuny rume aow wate riancia marros the calonnas The volla
ach call sell be the comepaatnng flnancii dies lieally; bill cnnmarte the compaay' development tondk over pai two years hand the infernatie in the silke
Algoratess hulld alpuniten rype loave edgs meed compuny mumettoe as jungrert crianta, with euch decitios the gracess denemiiinng whither cousder the informatic thene
comppayy's Chralogar: build canuogaan
sned di drow concladon head on all the mnaes informatiin the carieggae.
diticaee flnancal insprtance, Elach livel ortha will include informaticn about the runnis
Ouguat
Prompts Judging
For the given talk thore so curredyy mallipe soltioes wo oad utiling oi alffrre typo beesdelg You need rask Bo Beelllaiinnss of those seldiensced ldently the seballl ore. Firs preient coepp and eeyyyis
of cach ondlyss selatoos thes esgur mekking toalls, ssng SC dooooo resiing reditoonipp
mes tefo
ideennent
Quays
Schaken:
boldiee!
Ouged:
Figure 8: Prompts used in convert raw information in original documents into structured knowledge via LLMs, different type of structured knowledge different prompts, and the chunk directly by splitting without prompt.
18


Reviews:
[{"review_id": "Q3n3ZTdaRE", "reviewer": "Reviewer_Q2p8", "summary": "The paper introduces StructRAG, a novel framework designed to enhance LLMs for knowledge-intensive reasoning tasks by structuring information at inference time. StructRAG addresses limitations in existing retrieval-augmented generation (RAG) methods, which struggle when relevant knowledge is dispersed and reasoning demands are complex. Inspired by cognitive theories, StructRAG applies a hybrid structure router to identify the best structure type for each task, a scattered knowledge structurizer to transform raw documents into structured knowledge, and a structured knowledge utilizer to perform multi-step reasoning. This framework achieves state-of-the-art results across multiple benchmarks and tasks, particularly excelling in challenging scenarios where information is fragmented and extensive reasoning is required.", "questions": "In Table 3 and Figure 3, the exact match (EM) rate for Set 4 appears close across methods or even equivalent in some cases. Could the authors clarify whether this is due to specific characteristics of the dataset (e.g., limited answer variation) or if this is a coincidence? If the dataset impacts EM rates, additional context would be helpful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a challenging problem in LLM and RAG research, focusing on improving knowledge-intensive reasoning for real-world tasks. The idea of information structurization for RAG is clear and innovative, offering a new approach in this area.\n\nThe architecture and its components are described clearly, with each part of the StructRAG model explained in a way that makes it easy to follow. The experiments are thorough and well-documented, including ablation studies that show the contributions of each module and support the effectiveness of the approach.\n\nOverall, the work has practical applications and could be adapted to a range of RAG-related tasks, suggesting it has potential for further impact in the field.", "weaknesses": "The study presents StructRAG’s effectiveness with a single structured knowledge type for each task, yet it is unclear how the model would perform on tasks requiring multiple types simultaneously. For example, an ablation study that evaluates combinations of two fixed structure types could shed light on whether using multiple structure types consistently improves performance across different task categories.\n\nThe paper does not provide details on the distribution of structure types chosen by the hybrid structure router in its dataset. Since some structure types, such as algorithmic pseudo code, may be more complex to extract compared to tables or catalogues, reporting this distribution would clarify the router’s efficacy across different structure types. Additionally, an analysis of the structurizer’s performance in cases involving algorithmic structuring would help assess whether these cases yield similar performance gains as simpler structures.\n\nTable 5 highlights performance differences when using fixed structure types, but these differences may be influenced by the distribution of tasks that align better with certain types (e.g., most tasks in the dataset naturally suited to tables). A breakdown of the dataset showing the router’s selected structure types and a comparison to human-selected types would be useful. Such a comparison could indicate how well the router aligns with human judgment and whether adding certain structure types as a standard complement might enhance overall performance in diverse datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2B7VS5UXns", "reviewer": "Reviewer_WHkD", "summary": "This paper introduces StructRAG, a novel framework for improving retrieval-augmented generation (RAG) in knowledge-intensive reasoning tasks. The key insight is that existing RAG methods struggle when useful information is scattered across documents, making it difficult to identify and reason with key information.\n\nInspired by cognitive theories about how humans process information using different knowledge structures, StructRAG consists of three main components:\n\n1. A hybrid structure router that determines the most appropriate structure type (e.g., tables, graphs) for a given task\n2. A scattered knowledge structurizer that converts raw documents into the chosen structured format\n3. A structured knowledge utilizer that decomposes complex questions and performs reasoning using the structured knowledge\n\nTo train the hybrid structure router effectively, the authors develop a novel pipeline for generating training data through task synthesis, solution simulation, and preference judgment, combined with the DPO algorithm for preference learning.\n\nExperimental results show that StructRAG achieves state-of-the-art performance across various knowledge-intensive tasks, with particularly strong improvements on more complex tasks. The framework also demonstrates better performance and faster operation compared to existing Graph RAG methods.\n\nThe paper's main contribution is presenting a cognitively-inspired approach to structuring information during inference time in RAG systems, offering a promising direction for handling complex reasoning tasks where relevant information is scattered across multiple sources.", "questions": "Lines 458-460 highlight a challenge in real-world applications. How do the authors propose to address this challenge?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper demonstrates several notable strengths:\n\n- The novel integration of cognitive science principles into RAG systems. The authors establish meaningful connections between human information processing and computational approaches, making the cognitive science foundation both natural and practical.\n\n- The paper addresses a well-recognized challenge in the field: efficient information retrieval from unstructured inputs. Their solution, particularly the Hybrid Structure Router that mimics human decision-making combined with DPO fine-tuning, offers an innovative yet implementable approach.\n\n- The experimental evaluation is comprehensive, comparing StructRAG against current state-of-the-art approaches including long-context models, traditional RAG, RQ-RAG, and GraphRAG. The promising results across these various settings demonstrate the method's practical value.\n\n- The ablation studies are particularly insightful, reflecting the authors' thorough analysis and iterative refinement of their methodology.", "weaknesses": "While not a significant weakness, one suggestion concerns the comparison with Graph RAG. Graph RAG typically excels with complex relational data rather than long-context scenarios, which might explain its relatively poor performance in many of the experimental settings. A more targeted comparison using relationship-heavy datasets could provide additional insights into the relative strengths of both approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Agui4vfX2M", "reviewer": "Reviewer_EKh5", "summary": "This paper proposed a novel method to summarise the document with Hybrid Structure Router, and it dynamically converted documents into graph, chunk, catalogue, algorithm, table. It showed that the new method had better performance in terms of score and latency. The paper contains thorough literature review, and detailed experiment results. However, the paper only evaluates the performance on Qwen2-7B-Instruct model, which is a relatively small model. It did not evaluate on other main stream models like llama, and also did not evaluate the new method on models with larger amount of parameters. For example, the GraphRAG method used GPT-4-turbo in their original paper, and it is very hard to conclude that StructRAG is better than GraphRAG with the results of Qwen2-7B-Instruct presented in this paper.", "questions": "- What is the definition of LLM score\n- Did you tested the new method StructRAG with other models, or models with larger amount of parameters?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It proposed a novel method\n- It conducted thorough literature review\n- It compared the method with other RAG methods including Long-context, RAG, RQ-RAG, GraphRAG", "weaknesses": "- It did not provide the definition of LLM score\n- It only tested the new model with Qwen2-7B-Instruct with relatively small amount of parameters, and lacks of experiment results with other models or models with larger amount of parameters", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wse1fiNitx", "reviewer": "Reviewer_QD9m", "summary": "The paper addresses the challenge of handling RAG with complex documents, where information may be scattered across different parts of multiple documents. For RAG to be accurate, it must precisely retrieve and integrate all relevant texts, which presents a significant challenge. To tackle this, the authors propose a method that mimics human cognitive processes by using LLMs to convert dispersed information into various types of structured knowledge. Additionally, they employ a Hybrid Structure Router, trained with DPO, to select the appropriate structure type for each query. The implementation of this approach demonstrates its effectiveness in improving performance across most tasks.", "questions": "1. Could you provide more detailed examples of tasks that are suitable for each of the five types of structured knowledge you propose? This would help clarify the rationale behind categorizing them into these five types.\n\n2. When using LLMs to extract structured knowledge, do the LLMs have the capability to directly generate knowledge in the specified formats? Did you perform any checks or corrections on the outputs generated by the LLMs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality**: This paper introduces a novel approach to addressing the complex issue of RAG from multiple knowledge sources. The proposed process includes summarizing structured knowledge into five categories and utilizing LLMs for the construction and retrieval of this structured knowledge. This approach demonstrates a high degree of originality.\n\n**Quality**: The writing quality of the paper is commendable, with coherent reasoning and a logical flow that makes it easy for readers to understand. The authors have conducted detailed experiments across multiple datasets, effectively demonstrating the validity of StructRAG.\n\n**Clarity**: The paper is logically and structurally well-organized, with a clear presentation of the proposed framework's structure and processes. The use of formulas and diagrams effectively illustrates the concepts and workflow.\n\n**Significance**: This paper effectively addresses the challenge of handling RAG with complex documents by aggregating knowledge from different sources and structuring it into specified formats. This approach significantly enhances the performance and applicability of RAG systems.", "weaknesses": "The paper, while introducing the StructRAG framework, relies significantly on the inherent capabilities of large language models (LLMs) for structuring and utilizing raw knowledge. However, it could benefit from a more detailed discussion on the challenges encountered during knowledge processing and on strategies for more effectively leveraging LLMs in this context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)