RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation

## Abstract

Abstract Large language models (LLMs) have achieved impressive performance on knowledge-intensive tasks, yet they often struggle with multi-step reasoning due to the unstructured nature of retrieved context. While retrieval-augmented generation (RAG) methods provide external information, the lack of explicit organization among retrieved passages limits their effectiveness, leading to brittle reasoning pathways. Recent interpretability studies highlighting the importance of structured intermediate reasoning further align with this perspective.
We propose Retrieval-And-Structuring (RAS), a framework that dynamically constructs query-specific knowledge graphs through iterative retrieval and structured knowledge building. RAS interleaves targeted retrieval planning with incremental graph construction, enabling models to assemble and reason over evolving knowledge structures tailored to each query. On seven knowledge-intensive benchmarks, RAS consistently outperforms strong baselines, achieving up to 6.4% and 7.0% gains with open-source and proprietary LLMs, respectively. Our results demonstrate that dynamic, query-specific knowledge structuring offers a robust path to improving reasoning accuracy and robustness in language model generation. Our data and code can be found at https://github.com/pat-jj/RAS .

## 1 Introduction

Complex reasoning tasks such as scientific analysis or multi-hop question answering demand both comprehensive knowledge and structured logical thinking [1]. While large language models (LLMs) have achieved remarkable performance across a wide range of natural language processing tasks [2, 3, 4], they often struggle with knowledge-intensive reasoning due to the absence of precise, logically organized information [5, 6]. This limitation has motivated growing research into augmenting LLMs with structured knowledge to enhance their reasoning capabilities [7].

Retrieval-augmented generation (RAG) approaches have emerged as a promising direction by providing LLMs with additional context from retrieved passages [8, 9, 10, 11]. However, these methods frequently face hallucination challenges [12, 13], where generated content deviates from retrieved information. This issue stems primarily from the unstructured nature of retrieved passages, requiring the model to implicitly bridge gaps during generation.
Recent research in model interpretability further highlights this issue [14]. Attribution graph analyses have revealed that LLMs internally attempt multi-hop reasoning by chaining facts across different parts of context, but failures in these reasoning pathways are correlated with hallucinations and factual inconsistencies. These insights suggest that explicitly providing structured intermediate knowledge could significantly enhance the reliability and transparency of LLM reasoning.
Recent efforts have explored integrating knowledge graphs (KGs) with LLMs [15, 16, 11, 17] to provide compact, relational representations that enable more interpretable reasoning [18, 19, 20]. However, existing approaches largely rely on static, corpus-wide graphs, which may not adapt well to the specific reasoning needs of individual queries. Constructing and maintaining large-scale KGs also incurs substantial costs in open-domain settings.

To address these challenges, we propose Retrieval-And-Structuring (RAS), a framework that dynamically constructs and reasons over query-specific knowledge graphs through iterative retrieval and structured knowledge building. RAS interleaves three key components: (1) a planning module that strategically identifies knowledge gaps and generates targeted sub-queries, (2) a retrieval-and-structuring module that extracts factual triples from retrieved passages and incrementally builds a query-specific graph, and (3) a knowledge-augmented answering module that generates outputs conditioned on the accumulated structured knowledge.

RAS addresses several critical limitations of prior methods. Unlike traditional RAG approaches that rely on single-pass retrieval [8, 9, 10], RAS employs iterative planning to actively identify and fill knowledge gaps during inference. In contrast to static KG-based approaches [11, 17], RAS dynamically constructs query-specific knowledge graphs tailored to the evolving reasoning needs of each query. This targeted construction captures only task-relevant information, avoiding the inefficiencies and noise associated with static, corpus-level graphs, which enables more precise and robust language generation.

Through extensive evaluations across seven benchmarks spanning open-domain QA, closed-set QA, and long-form generation, RAS consistently outperforms strong baselines by 6.4% with open-source LLMs and 7.0% with proprietary models. Our main contributions are:

- •
We introduce RAS, a novel framework for dynamic knowledge structuring through iterative retrieval and graph construction, tailored to each query.
- •
We develop a structure-aware planning and answering model that operates over evolving knowledge graphs.
- •
We demonstrate RAS’s effectiveness through comprehensive experiments and ablation studies across diverse knowledge-intensive tasks.

## 2 Related Work

Retrieval-Augmented Generation (RAG) [8, 9] enhances language model performance on knowledge-intensive tasks by incorporating retrieved passages into the model input, improving factual accuracy and grounding. Early RAG approaches retrieved a fixed number of passages once before generation [21, 22, 23], while later methods explored adaptive retrieval [24], where models dynamically issue retrieval queries based on uncertainty signals, or retrieval evaluation [25] to improve relevance. However, these techniques often struggle with complex or multi-hop reasoning due to their linear designs.
Recent advances aim to strengthen retrieval quality by iterative refinement. Iterative retrieval-generation approaches [21, 26] leverage historical context to guide subsequent retrievals, while targeted retrieval strategies based on subqueries [27] or follow-up questions [28, 29] progressively enrich the retrieved evidence. Self-RAG [30] introduced self-reflective retrieval, allowing LLMs to selectively query for missing information and evaluate the quality of retrievals. RPG [31] further refined retrieval by extracting fine-grained paragraphs to enhance relevance. Despite these advances, existing methods face persistent challenges: retrieved context often contains redundant or irrelevant information, or conversely, discards essential facts during refinement.
To address these issues, we propose converting retrieved content into a structured graph format and maintaining an iteratively enriched, query-specific knowledge graph, allowing the model to reason over an organized and progressively expanded knowledge structure.

Graph as Context for LLMs.
Integrating structured graphs into language model workflows has emerged as a powerful approach for enhancing reasoning and factual accuracy [32, 33, 34, 35, 36, 37]. Graphs provide explicit, relational representations that help models go beyond flat text by making entity connections and multi-hop relationships more tractable. Recent work, such as GraphToken [38], shows that LLMs can process serialized graph inputs through token-level prompts [39] without modifying model parameters. G-Retriever [11] leverages global knowledge graphs (KGs) to retrieve entity-centric subgraphs, while GraphRAG-style methods [17, 40] construct corpus-level graphs and generate summaries to enhance retrieval quality.
Despite their benefits, these methods rely heavily on static, global graphs, which often include irrelevant or redundant information and cannot adapt to the needs of individual queries. Moreover, building and maintaining large-scale KGs is computationally expensive, particularly in open-domain settings. This motivates a shift toward dynamic, inference-time graph construction.
Our approach embraces this direction by proposing a retrieval-and-structuring framework that builds *query-specific knowledge graphs* on the fly. Rather than relying on fixed knowledge coverage, we iteratively retrieve and structure information into task-relevant graphs that evolve with the reasoning process. This dynamic graph construction aligns with recent interpretability findings [14], which suggest that LLM failures often stem from breakdowns in implicit reasoning chains—failures that explicit graph structuring can help mitigate.

Figure: Figure 1: Overview of the Retrieval-And-Structuring (RAS) framework. RAS operates through three stages: (1) Planning strategically determines knowledge retrieval needs and generates focused sub-queries based on the current knowledge state; (2) Text Retrieval and Knowledge Structuring retrieves text based on the sub-query and transforms the retrieved content into a graph, which is emerged to an evolving knowledge graph that expands based on reasoning needs; and (3) Answering leverages the accumulated structured knowledge to produce the final output. The framework employs parameter-efficient training to train the Graph LLM [38, 11], fine-tuning only the graph encoder, projector components, optionally with LoRA. We provide a step-by-step running example in Figure [24](https://arxiv.org/html/2502.10996v2#A7.F24) to concretize the framework.
Refer to caption: x1.png

## 3 Retrieval-And-Structuring (RAS) Framework

Effective knowledge-intensive language generation requires not only retrieving relevant information, but also structuring and reasoning over it systematically. We introduce Retrieval-And-Structuring (RAS), a framework that interleaves iterative retrieval planning with dynamic knowledge graph construction, enabling large language models (LLMs) to reason over progressively organized knowledge tailored to each query. Figure [1](https://arxiv.org/html/2502.10996v2#S2.F1) illustrates the overall workflow.

Key Definitions. We define the core concepts used in the RAS framework as follows.
The Main Query ($Q$) denotes the original task input. A Subquery ($q_{i}$) is a focused retrieval query generated at iteration $i$ to obtain supporting evidence. Retrieved Passages ($t_{i}$) are the top-$k$ documents selected for each subquery. A text-to-triples model $f_{t2t}$ converts retrieved text into Extracted Triples ($g_{i}$), structured as subject-predicate-object facts. These triples are incrementally accumulated into an evolving Knowledge Graph ($G_{Q}$), representing organized evidence about $Q$. The model $\mathcal{M}$ produces an Action Plan ($p_{i}$) at each step, determining whether to continue retrieval ([SUBQ]), stop retrieval ([SUFFICIENT]), or, initially, answer directly without retrieval ([NO_RETRIEVAL]).

### 3.1 Knowledge State-Aware Planning

The Planning stage initiates and controls the retrieval-and-structuring process by dynamically assessing the current knowledge state.

Initial Planning. Formally, given an input query $Q$, the model $\mathcal{M}$ generates an initial plan $p_{0}$:

$$ $p_{0}\leftarrow\mathcal{M}(\emptyset;\texttt{INST}_{\texttt{Plan}};\emptyset;Q)$ (1) $$

where $\texttt{INST}_{\texttt{Plan}}$ is the planning instruction (as shown in Figure [15](https://arxiv.org/html/2502.10996v2#A3.F15)). $p_{0}$ can take one of two forms:

- $\blacksquare$
[SUBQ] $q_{0}=Q$: If $\mathcal{M}$ assesses that the query cannot be satisfactorily answered with its own knowledge, we start the iteration with the main query $Q$ as the initial subquery, and move to the next stage (§[3.2](https://arxiv.org/html/2502.10996v2#S3.SS2)).
- $\blacksquare$
[NO_RETRIEVAL]: If $\mathcal{M}$ determines that $Q$ can be answered directly without requiring any additional knowledge, the planning process terminates, and the framework proceeds directly to the final Answering stage (§[3.3](https://arxiv.org/html/2502.10996v2#S3.SS3)).

Iterative Planning. At iteration $i>0$, given the accumulated knowledge $G_{i}$ and the subquery-triples history $[q_{0},g_{0},\ldots,q_{i},g_{i}]$, the model updates the plan:

$$ $p_{i+1}\leftarrow\mathcal{M}(\text{GNN}(G_{i});\texttt{INST}_{\texttt{Plan}};[ q_{0},g_{0},...,q_{i},g_{i}];Q)$ (2) $$

where GNN is a graph neural network for encoding and projecting the evolving KG $G_{i}$; $q_{k}$ is the subquery at iteration $k$, and $g_{k}$ is the extracted graph information (a list of triples) from the retrieved context $t_{k}$.

The output $p_{i+1}$ at each iteration can be either:

- $\blacksquare$
[SUBQ] $q_{i+1}$: The model generates a new subquery $q_{i+1}$ to guide the retrieval of additional relevant knowledge. The subquery is designed to fill specific gaps in the current knowledge state with respect to answering $Q$. The framework proceeds to the next stage (§[3.2](https://arxiv.org/html/2502.10996v2#S3.SS2)).
- $\blacksquare$
[SUFFICIENT]: The accumulated knowledge $G_{i}$ is deemed sufficient to comprehensively address the main query $Q$. The iterative retrieval process terminates, and the framework proceeds to the Answering stage (§[3.3](https://arxiv.org/html/2502.10996v2#S3.SS3)).

The Planning stage serves as a key driver of the RAS framework’s iterative retrieval and refinement process. By dynamically assessing the adequacy of the retrieved knowledge and generating targeted sub-queries, it enables the efficient acquisition of query-relevant information.

### 3.2 Text Retrieval and Knowledge Structuring

This stage transforms retrieved text $t_{i}$ into structured knowledge $g_{i}$ and progressively builds the query-specific graph $G_{Q}$.

Text Retrieval.
We use a text retriever to retrieve the top-$k$ semantically relevant passages $t_{i}$ from the corpus $C$ for each subquery $q_{i}$:

$$ $t_{i}\leftarrow\text{DenseRetrieval}(q_{i},C,k)$ (3) $$

We use a standard dense retriever by default but note that RAS is compatible with more advanced information retrieval methods [41, 42, 43].

Text-to-Triples (Graph) Conversion.
To extract essential factual information from the retrieved passages $t_{i}$, we employ a text-to-triples model $f_{t2t}$. This model is trained on the full WikiOfGraph dataset [44], which is a high-quality, LLM-curated text-to-triples corpus. Details of the training process are provided in Appendix [C.1](https://arxiv.org/html/2502.10996v2#A3.SS1). The model generates structured triples in the following format:

$$ $g_{i}\leftarrow f_{t2t}(t_{i})=[(s_{0},r_{0},o_{0}),...,(s_{|g_{i}|},r_{|g_{i} |},o_{|g_{i}|})]$ (4) $$

where each triple $(s_{j},r_{j},o_{j})$ represents a subject-predicate-object fact extracted from the text. This structured representation enables efficient downstream reasoning and facilitates integration with external knowledge graphs. Although $f_{t2t}$ is a lightweight LLM capable of fast inference using techniques such as quantization [45] and optimized inference frameworks like vLLM [46], the text-to-triples conversion can also be precomputed offline when maximal efficiency is required.

Iterative Knowledge Enrichment to Query-Specific KG.
The extracted triples $g_{i}$ are then converted into a graph structure $g_{i}^{\prime}=(V_{i},E_{i})$, where $V_{i}$ and $E_{i}$ denote the sets of nodes and edges, respectively. Each node $v\in V_{i}$ corresponds to a unique subject or object entity in $g_{i}$, while each edge $e\in E_{i}$ represents a predicate connecting two entities.
To enrich the graph with semantic information, the attributes of nodes and edges are obtained through a text encoder (Sentence-BERT [47]):

$$ $\text{emb}(v)\leftarrow\text{encode}(v),\forall v\in V_{i};\quad\quad\text{emb }(e)\leftarrow\text{encode}(e),\forall e\in E_{i}$ (5) $$

These semantic embeddings enable the model to capture the nuanced relationships between entities and facilitate reasoning over the knowledge graph.

To progressively enrich the query-related knowledge in response to the evolving sub-queries, the structured graph $g_{i}^{\prime}$ at each iteration $i$ is merged into an evolving KG $G_{Q}=(V_{Q},E_{Q})$ specific to the main query $Q$:

$$ $G_{Q}\leftarrow G_{Q}\cup g_{i}^{\prime}$ (6) $$

After enriching $G_{Q}$ with the new knowledge, the Planning module (§[3.1](https://arxiv.org/html/2502.10996v2#S3.SS1)) reassesses the current knowledge to determine the next step. Based on $G_{Q}$ and the chain of previous subqueries and their associated graph information, the model decides whether to generate another focused subquery for additional retrieval or to proceed with answering (§[3.3](https://arxiv.org/html/2502.10996v2#S3.SS3)) if the accumulated knowledge is sufficient.

### 3.3 Knowledge-Augmented Answering

At the final stage, the model $\mathcal{M}$ generates an answer $A$ to the main query $Q$ either conditioned on knowledge graph $G_{Q}$ and subquery chain $(q_{0},g_{0}),...,(q_{i},g_{i})$ when retrieval-and-structuring was processed, or directly when no retrieval is needed.

If no retrieval is needed ($p_{0}=\text{{[NO_RETRIEVAL]}}$), the answer is generated directly:

$$ $A\leftarrow\mathcal{M}(\emptyset;\texttt{INST}_{\texttt{Ans}};\emptyset;Q)$ (7) $$

Otherwise, after iterative knowledge enrichment concludes with [SUFFICIENT] plan, the answer is generated using encoded KG $(G_{Q})$ and subquery chain:

$$ $A\leftarrow\mathcal{M}(\text{GNN}(G_{Q});\texttt{INST}_{\texttt{Ans}};[q_{0},g _{0},...,q_{i},g_{i}];Q)$ (8) $$

where $\texttt{INST}_{\texttt{Ans}}$ is the answering instruction (as shown in Figure [16](https://arxiv.org/html/2502.10996v2#A3.F16)). $\mathcal{M}$ attends to knowledge in $G_{Q}$ and subquery chain to generate accurate, coherent answers. This structured conditioning enables systematic reasoning grounded in the assembled knowledge.

### 3.4 Sturcture-Aware Multitask Learning

The RAS framework is trained through a multitask setup that unifies knowledge planning and knowledge-augmented answering under a standard next-token prediction objective.

Each training instance corresponds to either a planning task or an answering task, selected randomly:

- $\diamond$
Planning Task: Input: (encoded $G_{i}$), planning instruction $\texttt{INST}_{\texttt{Plan}}$, subquery-triples history, main query $Q$. Output: next plan $p_{i+1}$.
- $\diamond$
Answering Task: Input: (encoded $G_{Q}$), answering instruction $\texttt{INST}_{\texttt{Ans}}$, full subquery-triples history, main query $Q$. Output: final answer $A$.

The model $\mathcal{M}$ used in RAS is Graph LLM, an architecture adapted from prior work [38, 11]. It is trained to perform both planning and answering tasks via standard next-token prediction, conditioned on the instruction type and input structure. To the best of our knowledge, this is the first application of Graph LLMs to general language generation tasks such as open question answering.

## 4 Experiments

Training Data & Setting. We develop HotpotQA-SUBQ, a dataset constructed based on HotpotQA [1], to train our model’s action planning and answering capabilities.
Our dataset creation begins with document filtering: using Claude-3.5-Sonnet [48], we identify and retain only the supporting documents necessary for answering the main query, removing irrelevant content. For each supporting document $d_{j}$, we then iteratively generate a subquery $q_{j}$, considering the main query, previous subqueries, and supporting documents. During iteration $j$, when more supporting documents remain, we create training samples with input $\{q_{0},g_{0},...,q_{j},g_{j},Q\}$ and output label “[SUBQ] $q_{j+1}$”, where $g_{k}$ represents triples extracted from document $d_{k}$, and $Q$ is the main query. For the final supporting document, we label the sample as “[SUFFICIENT]”.
To identify queries that can be answered directly, we test our base LLM (LLaMA-2-7B) on HotpotQA’s main queries without context. For correctly answered queries, we create training samples with the main query $Q$ as input and “[NO_RETRIEVAL]” as the output label. Additionally, to ensure fair comparison with existing approaches, we incorporate the subset of Arc-Easy (2,147 samples) and ASQA (3,897 samples) from Self-RAG’s training data, resulting in 208k training samples in total. We place detailed training data processing, dataset statistics, and data samples in Appendix [B](https://arxiv.org/html/2502.10996v2#A2).
For efficient inference, we train a text-to-triples model $f_{t2t}$ on the WikiOFGraph dataset [44] using LLaMA-3.2-3B-Instruct as the base model (see Appendix [C](https://arxiv.org/html/2502.10996v2#A3)), and deploy it using vLLM for optimized runtime performance. We present our hyperparameter study of each component in Appendix [H](https://arxiv.org/html/2502.10996v2#A8).

Knowledge Sources.
Due to the large sizes of wiki corpora, we employ faiss [49] for efficient vector searching over the dense index. Following the Self-RAG [30], we utilize the Wikipedia 2018 [50] by default, while specifically using the Wikipedia 2020 for PopQA to access more recent information. To optimize retrieval efficiency, we partition the index into five segments.

Test Datasets & Metrics & Compared Baselines. We conduct comprehensive evaluations on diverse knowledge-intensive tasks following previous studies [30, 31]. The evaluation encompasses three categories of datasets: (1) open-domain short-form generation datasets: TriviaQA [51], PopQA [52], and 2WikiMultihopQA [53]; (2) closed-set task datasets: PubHealth [54] and ARC-Challenge [55]; and (3) long-form generation datasets: ALCE-ASQA [56, 57] and ELI5 [58]. For evaluation metrics, we maintain consistency with prior work [30, 52, 31], employing “golden match” accuracy for PopQA and TriviaQA, token-level F1 score for 2WikiMultihopQA, accuracy for PubHealth and ARC-Challenge, and ROUGE-LSum alongside MAUVE score [59] for ASQA and ELI5. Our comparative analysis includes three baseline categories: models without retrieval augmentation, incorporating Claude 3.5 Sonnet as a state-of-the-art closed-source baseline; models with single retrieval over top-5 documents, including Claude 3.5 Sonnet, and SuRe [25], a leading retrieve-and-summarize method; and models with self-reflective retrieval, including leading approaches Self-RAG [30], RPG [31], and Search-R1 [60]. We place more details of metrics/datasets in Appendix [D](https://arxiv.org/html/2502.10996v2#A4).

**Table 1: Performance Comparison. We highlight the top-2 closed-source models and top-2 open-source 7B models, underlining the best closed-/open-source (7B) model for each dataset. Note: RAS is designed for general open-domain question answering and is not tailored for Knowledge Graph Question Answering (KGQA) tasks [62]; therefore, we do not compare it to KGQA-focused methods [20, 63, 64].**
|  | Short-form | Closed-set | Long-form Generation |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | TQA | 2WQA | PopQA | Pub | ARC | ASQA | ELI5 |  |  |
|  | (acc) | (F1) | (acc) | (acc) | (acc) | (rouge) | (mauve) | (rouge) | (mauve) |
| Baselines w/o Retrieval |  |  |  |  |  |  |  |  |  |
| Llama2${}_{7\text{B}}$ | 30.5 | 18.9 | 14.7 | 34.2 | 21.8 | 15.3 | 19.0 | 18.3 | 32.4 |
| Alpaca${}_{7\text{B}}$ | 54.5 | – | 23.6 | 49.8 | 45.0 | 29.4 | 61.7 | – | – |
| Llama2${}_{13\text{B}}$ | 38.5 | 20.2 | 14.7 | 29.4 | 29.4 | 12.4 | 16.0 | 18.2 | 41.4 |
| Alpaca${}_{13\text{B}}$ | 61.3 | – | 24.4 | 55.5 | 54.9 | 32.0 | 70.6 | – | – |
| Llama3${}_{8\text{B}}$ | 56.1 | 21.2 | 26.7 | 33.2 | 42.2 | 17.6 | 25.0 | 18.2 | 39.7 |
| ChatGPT${}_{\sim 175\text{B}}$ | 74.3 | 24.8 | 29.3 | 70.1 | 75.3 | 36.2 | 68.8 | 22.8 | 32.6 |
| Sonnet-3.5${}_{\sim 175\text{B}}$ | 78.4 | 40.0 | 30.2 | 83.7 | 88.5 | 37.0 | 39.1 | 21.8 | 26.5 |
| Baselines w/ Retrieval (#docs=5 by default) |  |  |  |  |  |  |  |  |  |
| Llama2${}_{7\text{B}}$ | 42.5 | 21.0 | 38.2 | 30.0 | 48.0 | 22.1 | 32.0 | 18.6 | 35.3 |
| Alpaca${}_{7\text{B}}$ | 64.1 | – | 46.7 | 40.2 | 48.0 | 33.3 | 57.9 | – | – |
| Llama2${}_{13\text{B}}$ | 47.0 | 31.2 | 45.7 | 30.2 | 26.0 | 20.5 | 24.7 | 18.6 | 42.3 |
| Llama3${}_{8\text{B}}$ | 60.4 | 33.4 | 48.6 | 36.5 | 40.1 | 23.9 | 52.1 | 18.8 | 40.7 |
| Alpaca${}_{13\text{B}}$ | 66.9 | – | 46.1 | 51.1 | 57.6 | 36.7 | 56.6 | – | – |
| RECOMP${}_{7\text{B}}$ [61] | – | 32.4 | – | – | – | 36.5 | 76.0 | – | – |
| ChatGPT | 65.7 | 32.8 | 50.8 | 54.7 | 75.3 | 39.9 | 79.7 | 20.6 | 57.2 |
| Sonnet-3.5${}_{\#\text{docs}=1}$ | 69.1 | 41.9 | 51.5 | 49.1 | 88.6 | – | – | – | – |
| Sonnet-3.5${}_{\#\text{docs}=5}$ | 72.5 | 53.7 | 57.3 | 53.9 | 87.1 | 38.8 | 61.6 | 20.2 | 32.3 |
| SuRe${}_{7\text{B}}$ [25] | 51.2 | 20.6 | 39.0 | 36.2 | 52.7 | 35.8 | 76.2 | 16.1 | 26.6 |
| SuRe${}_{\text{GPT}}$ | 72.3 | 38.1 | 53.6 | 57.2 | 79.6 | 36.0 | 74.2 | 19.2 | 51.6 |
| SuRe${}_{\text{Sonnet-3.5}}$ | 76.8 | 37.6 | 41.2 | 62.8 | 91.6 | 30.2 | 69.9 | 15.4 | 27.2 |
| Baselines w/ Self-Reflective Retrieval |  |  |  |  |  |  |  |  |  |
| Self-RAG${}_{7\text{B}}$ [30] | 66.4 | 25.1 | 54.9 | 72.4 | 67.3 | 35.7 | 74.3 | 17.9 | 35.6 |
| Self-RAG${}_{13\text{B}}$ | 69.3 | – | 55.8 | 74.5 | 73.1 | 37.0 | 71.6 | – | – |
| RPG${}_{7\text{B}}$ [31] | 65.1 | 33.6 | 56.0 | 73.4 | 65.4 | 37.6 | 84.4 | 19.1 | 46.4 |
| Search-R1${}_{7\text{B}}$ [60] | 63.8 | 35.2 | 45.7 | 69.4 | 64.4 | 35.2 | 72.3 | 18.4 | 37.1 |
| Retrieval-And-Structuring (RAS) |  |  |  |  |  |  |  |  |  |
| RAS${}_{\text{Sonnet-3.5}}$ | 77.6 | 57.7 | 62.3 | 71.3 | 93.9 | 39.1 | 70.5 | 23.3 | 37.7 |
| RAS${}_{8\text{B}}$ | 73.8 | 44.2 | 57.7 | 77.6 | 71.4 | 37.6 | 96.2 | 20.1 | 54.4 |
| RAS${}_{7\text{B}}$ | 72.7 | 42.1 | 58.3 | 74.7 | 68.5 | 37.2 | 95.2 | 19.7 | 47.8 |

Inference Setting.
We evaluate RAS using both our trained open-source models (RAS${}_{7\text{B}/8\text{B}}$, based on LLaMA-2-7B/LLaMA-3-8B and a Graph Transformer encoder [65]) and the closed-source Claude-3.5-Sonnet model under varied inference settings. For open-source models (RAS${}_{7\text{B}/8\text{B}}$), we follow prior work [30, 31] and adopt zero-shot inference across all datasets. For the closed-source model (RAS${}_{\text{Sonnet-3.5}}$), we apply few-shot inference with two exemplars for ASQA and ELI5, while using zero-shot inference for all other datasets. More details are provided in Appendix [E.1](https://arxiv.org/html/2502.10996v2#A5.SS1) and [E.2](https://arxiv.org/html/2502.10996v2#A5.SS2).

During inference, if the initial planning output is not [NO_RETRIEVAL], we utilize the main query $Q$ as the initial subquery $q_{0}$. For PopQA and TriviaQA evaluation, we follow established settings [30, 66], incorporating top-five web search engine results as initial retrieved context $t_{0}$. For ASQA and ELI5, we maintain methodological consistency with prior work [31, 30, 56], utilizing their predetermined five-document context. Under these conditions, we omit plan generation and text retrieval phases, implementing static inference [30] with five fixed iterations. For remaining datasets, we establish a maximum iteration count of five. Following previous studies, we employ Contriever-MS MARCO [67] as the primary dense retriever, with BM25 [68] serving as the retrieval mechanism for 2WikiMultihopQA. Across all retrieval processes, we maintain a consistent top-$k$ document selection of five. We present more details of inference settings in Appendix [E](https://arxiv.org/html/2502.10996v2#A5).

### 4.1 Main Results

Our performance evaluation, presented in Table [1](https://arxiv.org/html/2502.10996v2#S4.T1), demonstrates that the LLaMA-2-7B/LLaMA-3-8B model fine-tuned with RAS outperforms existing open-source solutions, including Self-RAG [30] and RPG [31]. Notably, compared to the previous SOTA models, RAS${}_{7\text{B}}$ shows a 9.7% improvement in short-form question-answering and a 7.9% gain in long-form generation tasks. Additionally, when applied to closed-source Sonnet-3.5 model, RAS consistently achieves superior results compared to single retrieval RAG approaches, including retrieve-and-summarize approach SuRe [25].
We find that sometimes (e.g., on TriviaQA and PubHealth) single-hop retrieval could not boost LLM’s performance, and even makes it worse, which demonstrates the necessity of on-demand retrieval, aligning with previous findings.

Our “role-swapping” study in Figure [3](https://arxiv.org/html/2502.10996v2#S4.F3.4), where Sonnet-3.5 and RAS${}_{7\text{B}}$ alternately perform the planning and answering tasks, reveals that performance is primarily bounded by the answering capability. For example, when Sonnet-3.5 handles planning and RAS${}_{7\text{B}}$ handles answering, the system achieves 62.4% accuracy on the ARC-Challenge dataset. However, when Sonnet-3.5 is responsible for both planning and answering, accuracy increases to 93.9%. Notably, despite having 60$\times$ fewer parameters, our trained RAS${}_{7\text{B}}$ achieves planning performance comparable to Sonnet-3.5, and even outperforms it in some cases.

Also, for long-form generation, our trained model achieved notably high MAUVE scores, which can be majorly attributed to our “text-to-triple” step, as demonstrated by our ablation study in Table [2](https://arxiv.org/html/2502.10996v2#S4.T2).

### 4.2 Ablations & More Analysis

We analyze the impact of each component of RAS in Table [2](https://arxiv.org/html/2502.10996v2#S4.T2) and provide detailed discussion below.

Effect of Iterative Planning and Retrieval. Comparing the base model with “No Planning” variant shows that iterative planning provides consistent improvements across all metrics (e.g., +8.8% on TQA, +9.0% on 2WQA). This demonstrates the importance of dynamically determining retrieval needs and generating focused sub-queries. Without planning, the model relies on single-pass retrieval, which may miss crucial information needed for complex reasoning. Also, when turning off retrieval, the performance degradation is more obvious, due to the knowledge-intensive nature of those datasets.

Figure: Table 2: Ablations in Training and Inference (with RAS${}_{7\text{B}}$). Training: “No GraphEncode” removes the graph encoder during training, using only LoRA-based LLM fine-tuning. “No LoRA” uses graph token optimization without low-rank adaptation. “No Text-to-Triple” keeps the original retrieved texts instead of converting them into triples. “No Multi-Task” trains two models separately handling planning and answering. Inference: “No Retrieval” tests direct query answering without any context. “No GraphEncode” removes graph encoding and projection during inference, using only textual context. “No Planning” removes the planning module and runs single-pass retrieval-structuring-answering pipeline.
Refer to caption: x2.png

Effect of Graph Construction and Encoding. The impact of structured knowledge representation is evident from two ablations. First, “No Text-to-Triple” degrades performance significantly on all metrics (e.g., -9.0% on 2WQA, -22.2% MAUVE on ASQA), showing the value of essential information extraction via converting retrieved text into structured triples. Second, removing the graph encoder (“No GraphToken”) during training or inference consistently hurts performance across datasets, with particularly large drops on PubHealth (-11.2% and -10.0% respectively). This suggests that the graph structure encoding helps the model better leverage the knowledge relationships.

Effect of LoRA and Multi-task Learning. Our experiments reveal that parameter-efficient training strategies significantly impact model performance. Using only graph token optimization without LoRA leads to substantial degradation (-11.8% on average). A similar observation can be made for “No Multi-Task” (e.g., 12.3% accuracy degradation on PubHealth), indicating the significance of jointly training the model on both action planning and answer generation tasks rather than optimizing for each task separately, supporting findings from prior work [31]. The complementary effects suggest that while graph-based knowledge representation is valuable, it needs to be combined with careful parameter tuning and multi-task learning to achieve optimal performance.

Impact of Graph Information Abundance. Figure [3](https://arxiv.org/html/2502.10996v2#S4.F3.4) shows that increasing the amount of structured graph information consistently improves RAS’s performance across tasks. For TQA, both RAS${}_{7\text{B}}$ and RAS${}_{8\text{B}}$ exhibit approximately linear gains as more triples are retained, with no clear saturation even at 100% information. For ASQA-MV, RAS${}_{7\text{B}}$ benefits from a sharp improvement between 10% and 30% graph information, followed by steady increases, while RAS${}_{8\text{B}}$ maintains a smoother and more stable growth pattern throughout. These results confirm that RAS effectively leverages structured knowledge at all levels of availability, and that more complete knowledge graphs consistently translate to stronger reasoning and generation quality. Additionally, the larger 8B model consistently outperforms the 7B variant under all conditions, suggesting that scaling the model size further enhances RAS’s ability to utilize structured knowledge. Interestingly, even partial graphs containing only 30%–50% of triples already deliver substantial gains over low-information baselines, highlighting RAS’s robustness to incomplete or partially retrieved knowledge.

Figure: Figure 4: Impact of Training Data Volume on Model Performance. Results for RAS${}_{\text{7B}}$ (top) and RAS${}_{\text{8B}}$ (bottom) illustrate how performance scales with increasing training data.
Refer to caption: x4.png

Impact of Training Data Volume. Figure [4](https://arxiv.org/html/2502.10996v2#S4.F4) demonstrates how training dataset size influences model performance across different tasks. Considering the computational efficiency, we sampled 2,000 instances each from TQA and 2WQA for evaluation, while maintaining the original sizes of other datasets. The results indicate that model performance generally improves with increased training data volume.
Notably, our model achieves competitive performance even with limited training data - using only 5% (10K instances) of our full dataset, it surpasses previous state-of-the-art models on TQA, 2WQA, and ELI5. These results suggest both the robustness of our architectural design and the effectiveness of our data curation methodology.
However, we observed an exception with the ELI5 dataset, where performance were inconsistent. This irregularity can be attributed to the inclusion of ASQA training data in our training set, following established setting from previous research [30, 31]. Among our test datasets, ASQA and ELI5 are unique in requiring long-form response generation. The periodic decline in ELI5 performance suggests that the model’s response generation began to align more closely with ASQA’s training data distribution, potentially at the expense of ELI5’s distinct characteristics.

**Table 3: Impact of Triple Extraction Quality. Higher-quality text-to-triples models lead to better answer generation in RAS. Claude 3.5 Sonnet achieves the best performance on TriviaQA, 2WikiMultihopQA, and PopQA. LLaMA-3.2-3B is used in our experiments, selected for its strong balance between accuracy and efficiency.**
| Triple Extractor $f_{t2t}$ | TQA | 2WQA | PopQA |
| --- | --- | --- | --- |
| Flan-T5-Large | 70.3 | 40.7 | 56.7 |
| LLaMA-3.2-3B | 72.7 | 42.1 | 58.3 |
| Claude-3.5-Sonnet | 73.8 | 44.5 | 60.1 |

Impact of Triple Extraction Quality
We further examine how the quality of triple extraction influences the overall effectiveness of RAS. Specifically, we test three text-to-triples models: Flan-T5-Large, LLaMA-3.2-3B model and Claude-3.5-Sonnet where the first two are trained on WikiOfGraph data. As shown in Table [3](https://arxiv.org/html/2502.10996v2#S4.T3), higher-quality triple extraction leads to consistent performance gains across all evaluated datasets.
Notably, using Claude-3.5-Sonnet triples improves TriviaQA accuracy from 72.7% to 73.8%, boosts 2WikiMultihopQA F1 from 42.1 to 44.5, and raises PopQA accuracy from 58.3% to 60.1%. These results underscore that while RAS achieves strong performance even with lightweight extraction models, its effectiveness can be further enhanced by supplying more precise and complete structured knowledge during inference.

## 5 Conclusion

We presented RAS (Retrieval-And-Structuring), a novel framework that improves knowledge-intensive language generation by interleaving iterative retrieval with dynamic knowledge graph construction. RAS introduces key innovations, including structure-aware action planning and graph-augmented answering, enabling models to reason over progressively organized information tailored to each query. Extensive experiments across seven benchmarks demonstrate that RAS consistently outperforms existing methods, particularly in tasks requiring multi-step reasoning and factual consistency. Furthermore, the modular, graph-based architecture of RAS allows seamless integration with external knowledge sources and human feedback, making it well-suited for high-stakes domains where transparency and adaptability are essential. Limitations and directions for future work are discussed in Appendix [A](https://arxiv.org/html/2502.10996v2#A1).

## References

- [1]
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
arXiv preprint arXiv:1809.09600, 2018.
- [2]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding.
arXiv preprint arXiv:1810.04805, 2018.
- [3]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
Advances in neural information processing systems, 33:1877–1901, 2020.
- [4]
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.
Palm: Scaling language modeling with pathways.
Journal of Machine Learning Research, 24(240):1–113, 2023.
- [5]
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.
Scaling language models: Methods, analysis & insights from training gopher.
arXiv preprint arXiv:2112.11446, 2021.
- [6]
Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su.
Deductive verification of chain-of-thought reasoning.
Advances in Neural Information Processing Systems, 36, 2024.
- [7]
Xuezhi Wang, Haohan Wang, and Diyi Yang.
Measure and improve robustness in nlp models: A survey.
arXiv preprint arXiv:2112.08313, 2021.
- [8]
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang.
Retrieval augmented language model pre-training.
In International conference on machine learning, pages 3929–3938. PMLR, 2020.
- [9]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
- [10]
Gautier Izacard and Edouard Grave.
Leveraging passage retrieval with generative models for open domain question answering.
In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880, Online, April 2021. Association for Computational Linguistics.
- [11]
Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi.
G-retriever: Retrieval-augmented generation for textual graph understanding and question answering.
arXiv preprint arXiv:2402.07630, 2024.
- [12]
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald.
On faithfulness and factuality in abstractive summarization.
In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online, July 2020. Association for Computational Linguistics.
- [13]
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al.
Siren’s song in the ai ocean: a survey on hallucination in large language models.
arXiv preprint arXiv:2309.01219, 2023.
- [14]
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson.
On the biology of a large language model.
Transformer Circuits Thread, 2025.
- [15]
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu.
Ernie: Enhanced representation through knowledge integration.
arXiv preprint arXiv:1904.09223, 2019.
- [16]
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang.
A survey of knowledge-enhanced text generation.
ACM Computing Survey (CSUR), 2022.
- [17]
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson.
From local to global: A graph rag approach to query-focused summarization.
arXiv preprint arXiv:2404.16130, 2024.
- [18]
Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al.
Knowledge graphs.
ACM Computing Surveys (Csur), 54(4):1–37, 2021.
- [19]
Pengcheng Jiang, Lang Cao, Cao Xiao, Parminder Bhatia, Jimeng Sun, and Jiawei Han.
Kg-fit: Knowledge graph fine-tuning upon open-world knowledge.
In The Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024.
- [20]
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M Ni, Heung-Yeung Shum, and Jian Guo.
Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph.
arXiv preprint arXiv:2307.07697, 2023.
- [21]
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen.
Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.
arXiv preprint arXiv:2305.15294, 2023.
- [22]
Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert.
RAGAs: Automated evaluation of retrieval augmented generation.
In Nikolaos Aletras and Orphee De Clercq, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta, March 2024. Association for Computational Linguistics.
- [23]
Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen.
Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models, 2024.
- [24]
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig.
Active retrieval augmented generation.
arXiv preprint arXiv:2305.06983, 2023.
- [25]
Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin.
Sure: Summarizing retrievals using answer candidates for open-domain qa of llms.
arXiv preprint arXiv:2404.13081, 2024.
- [26]
Jian Guan, Wei Wu, Zujie Wen, Peng Xu, Hongning Wang, and Minlie Huang.
Amor: A recipe for building adaptable modular knowledge agents through process feedback.
arXiv preprint arXiv:2402.01469, 2024.
- [27]
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia.
Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp.
arXiv preprint arXiv:2212.14024, 2023.
- [28]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
ReAct: Synergizing reasoning and acting in language models.
arXiv:2210.03629, 2023.
- [29]
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis.
Measuring and narrowing the compositionality gap in language models.
arXiv preprint arXiv:2210.03350, 2022.
- [30]
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi.
Self-rag: Learning to retrieve, generate, and critique through self-reflection.
In The Twelfth International Conference on Learning Representations, 2023.
- [31]
Yuanjie Lyu, Zihan Niu, Zheyong Xie, Chao Zhang, Tong Xu, Yang Wang, and Enhong Chen.
Retrieve-plan-generation: An iterative planning and answering framework for knowledge-intensive llm generation.
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4683–4702, 2024.
- [32]
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec.
Qa-gnn: Reasoning with language models and knowledge graphs for question answering.
arXiv preprint arXiv:2104.06378, 2021.
- [33]
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy S Liang, and Jure Leskovec.
Deep bidirectional language-knowledge graph pretraining.
Advances in Neural Information Processing Systems, 35:37309–37323, 2022.
- [34]
Wenhao Yu.
Retrieval-augmented generation across heterogeneous knowledge.
In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 52–58, 2022.
- [35]
Mingxuan Ju, Wenhao Yu, Tong Zhao, Chuxu Zhang, and Yanfang Ye.
Grape: Knowledge graph enhanced passage reader for open-domain question answering.
arXiv preprint arXiv:2210.02933, 2022.
- [36]
Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, and Chengwei Pan.
REALM: Rag-driven enhancement of multimodal electronic health records analysis via large language models.
arXiv:2402.07016, 2024.
- [37]
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su.
Hipporag: Neurobiologically inspired long-term memory for large language models.
arXiv preprint arXiv:2405.14831, 2024.
- [38]
Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan Halcrow.
Let your graph do the talking: Encoding structured data for llms.
arXiv preprint arXiv:2402.05862, 2024.
- [39]
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang.
P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.
arXiv preprint arXiv:2110.07602, 2021.
- [40]
Pengcheng Jiang, Cao Xiao, Minhao Jiang, Parminder Bhatia, Taha Kass-Hout, Jimeng Sun, and Jiawei Han.
Reasoning-enhanced healthcare predictions with knowledge graph community retrieval.
In The Thirteenth International Conference on Learning Representations, 2025.
- [41]
Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, Kazuma Hashimoto, Mike Bendersky, and Marc Najork.
Exploring the viability of synthetic query generation for relevance prediction.
arXiv preprint arXiv:2305.11944, 2023.
- [42]
SeongKu Kang, Shivam Agarwal, Bowen Jin, Dongha Lee, Hwanjo Yu, and Jiawei Han.
Improving retrieval in theme-specific applications using a corpus topical taxonomy.
In Proceedings of the ACM Web Conference 2024, pages 1497–1508, 2024.
- [43]
Pengcheng Jiang, Jiacheng Lin, Lang Cao, R. Tian, S. Kang, Z. Wang, Jimeng Sun, and Jiawei Han.
Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning.
arXiv preprint arXiv: 2503.00223, 2025.
- [44]
Daehee Kim, Deokhyung Kang, Sangwon Ryu, and Gary Geunbae Lee.
Ontology-free general-domain knowledge graph-to-text generation dataset synthesis using large language model.
arXiv preprint arXiv:2409.07088, 2024.
- [45]
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.
Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.
Advances in neural information processing systems, 35:30318–30332, 2022.
- [46]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
- [47]
N Reimers.
Sentence-bert: Sentence embeddings using siamese bert-networks.
arXiv preprint arXiv:1908.10084, 2019.
- [48]
Anthropic.
Introducing claude 3.5 sonnet.
[https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet), 2024.
Accessed: 2025-01-06.
- [49]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou.
The faiss library.
arXiv preprint arXiv:2401.08281, 2024.
- [50]
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave.
Atlas: Few-shot learning with retrieval augmented language models.
Journal of Machine Learning Research, 24(251):1–43, 2023.
- [51]
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.
arXiv preprint arXiv:1705.03551, 2017.
- [52]
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi.
When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.
arXiv preprint arXiv:2212.10511, 2022.
- [53]
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa.
Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.
arXiv preprint arXiv:2011.01060, 2020.
- [54]
Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng, and James Glass.
Interpretable unified language checking.
arXiv preprint arXiv:2304.03728, 2023.
- [55]
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
arXiv preprint arXiv:1803.05457, 2018.
- [56]
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen.
Enabling large language models to generate text with citations.
arXiv preprint arXiv:2305.14627, 2023.
- [57]
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang.
Asqa: Factoid questions meet long-form answers.
arXiv preprint arXiv:2204.06092, 2022.
- [58]
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli.
Eli5: Long form question answering.
arXiv preprint arXiv:1907.09190, 2019.
- [59]
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui.
Mauve: Measuring the gap between neural text and human text using divergence frontiers.
Advances in Neural Information Processing Systems, 34:4816–4828, 2021.
- [60]
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
arXiv preprint arXiv:2503.09516, 2025.
- [61]
Fangyuan Xu, Weijia Shi, and Eunsol Choi.
Recomp: Improving retrieval-augmented lms with compression and selective augmentation.
arXiv preprint arXiv:2310.04408, 2023.
- [62]
Aleksandr Perevalov, Xi Yan, Liubov Kovriguina, Longquan Jiang, Andreas Both, and Ricardo Usbeck.
Knowledge graph question answering leaderboard: A community resource to prevent a replication crisis.
In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2998–3007, Marseille, France, June 2022. European Language Resources Association.
- [63]
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan.
Reasoning on graphs: Faithful and interpretable large language model reasoning.
arXiv preprint arXiv:2310.01061, 2023.
- [64]
Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, Cehao Yang, Jiaxin Mao, and Jian Guo.
Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation.
arXiv preprint arXiv:2407.10805, 2024.
- [65]
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun.
Masked label prediction: Unified message passing model for semi-supervised classification.
arXiv preprint arXiv:2009.03509, 2020.
- [66]
Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass.
Sail: Search-augmented instruction learning.
arXiv preprint arXiv:2305.15225, 2023.
- [67]
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave.
Unsupervised dense information retrieval with contrastive learning.
arXiv preprint arXiv:2112.09118, 2021.
- [68]
Stephen Robertson, Hugo Zaragoza, et al.
The probabilistic relevance framework: Bm25 and beyond.
Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009.
- [69]
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.
Scaling instruction-finetuned language models.
Journal of Machine Learning Research, 25(70):1–53, 2024.
- [70]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.
- [71]
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer.
A discrete hard em approach for weakly supervised question answering.
arXiv preprint arXiv:1909.04849, 2019.
- [72]
Chin-Yew Lin.
Rouge: A package for automatic evaluation of summaries.
In Text summarization branches out, pages 74–81, 2004.

## Appendix A Limitations & Future Work

Figure: Figure 5: Potential Future Extensions of RAS.
Refer to caption: x5.png

While RAS demonstrates strong performance, several key limitations warrant consideration for future improvements:

- •
Knowledge Extraction Quality: The effectiveness of both planning and answer generation depends on the quality of text-to-triple extraction. Despite using an end-to-end text-to-triples model for triple extraction, implementing a more sophisticated approach – such as a dedicated Named Entity Recognition and Relation Extraction (NER-RE) pipeline – could potentially enhance RAS’s reasoning capabilities and knowledge representation.
- •
Graph Evolution Strategy: The current approach to evolving knowledge graphs could be enhanced by incorporating more sophisticated graph pruning and merging strategies. Future work could explore dynamic graph summarization techniques to maintain the most relevant information while preventing excessive graph growth during iterations.
- •
Cross-Domain Adaptability: While RAS performs well on the evaluated tasks, its effectiveness across highly specialized domains or multilingual settings remains to be investigated. Future research could focus on developing domain-adaptation techniques and multilingual knowledge structuring approaches.
- •
Interactive Refinement: The current implementation lacks mechanisms for interactive refinement of the constructed knowledge graphs. Future versions could incorporate human feedback loops to improve the quality of knowledge structuring and reasoning paths.

These limitations suggest several promising directions for future research, including improved knowledge extraction techniques and enhanced graph evolution strategies. A particularly promising direction is the integration of external knowledge graphs into RAS’s iterative loop, as its graph-based architecture naturally accommodates the incorporation of established KGs like Wikidata or domain-specific knowledge bases. This could significantly enhance the system’s reasoning capabilities by combining dynamically constructed knowledge with curated structured information. We summarize the potential future directions of extension in Figure [5](https://arxiv.org/html/2502.10996v2#A1.F5).

## Appendix B HotpotQA-SUBQ Dataset

The source of our dataset is HotpotQA [1].(^1^11We use the version hotpot_train_v1.1 from [https://github.com/hotpotqa/hotpot](https://github.com/hotpotqa/hotpot).) Our goal is to create a dataset that can let the model learn the capabilities of subquery generation and answering with intensive knowledge. We name the dataset as HotpotQA-SUBQ.

The construction of HotpotQA-SUBQ includes three steps: (1) filter out irrelevant/unnecessary “supporting docs” (§[B.1](https://arxiv.org/html/2502.10996v2#A2.SS1)), (2) generate one subquery for each supporting document (§[B.2](https://arxiv.org/html/2502.10996v2#A2.SS2)), and (3) sample labeling (§[B.3](https://arxiv.org/html/2502.10996v2#A2.SS3)).

### B.1 Supporting Document Filtering

To identify and filter out irrelevant supporting documents from HotpotQA, we employ an instruction-based approach using Claude 3.5 Sonnet, as shown in Figure [6](https://arxiv.org/html/2502.10996v2#A2.F6).

Figure: Figure 6: Prompt used for filtering supporting documents in HotpotQA. The prompt includes examples to demonstrate the difference between helpful and irrelevant documents. The input parts to the prompt are highlighted.

The filtering prompt is designed with clear examples that illustrate the criteria for document relevance. For each HotpotQA sample, we enumerate all supporting documents and use Claude to identify only those that contribute directly to answering the question. After filtering out irrelevant documents, we filter the question with no supporting documents. This filtering step reduces noise in the training data and helps focus the model on truly relevant information during sub-query generation.

### B.2 Sub-query Generation

For sub-query generation, we use the template as follows to generate one subquery per (filtered) document:

Figure: Figure 7: Prompt used for subquery generation from HotpotQA. The input parts to the prompt are highlighted.

The prompt template enforces document-specificity, goal-orientation, and conciseness in sub-query generation. For iterative querying, we maintain a list of previously generated sub-queries to avoid redundancy and encourage progressive information gathering.

### B.3 Sample Labeling

The sample labeling process transforms the filtered and subquery-augmented HotpotQA examples into training instances for both the Planning and Answering components. We formalize this process in Algorithm 1.

Figure: Algorithm 1 HotpotQA-SUBQ Sample Labeling

The algorithm takes as input the filtered HotpotQA dataset $\mathcal{D}$, base language model $\mathcal{M}$, and text-to-triple conversion model $f_{t2t}$. For each example, we first attempt direct answer generation using the base LLM (Line 3). If successful, we create [NO_RETRIEVAL] training instances for both the planning and answering (Lines 4-6).

For examples requiring retrieval, we process each supporting document sequentially:

- 1.
Convert document text to structured triples using $f_{t2t}$ (Line 12)
- 2.
For intermediate documents ($i<n$):
•
Construct input by concatenating previous subquery-graph pairs
•
Label with [SUBQ] and next subquery
- 3.
For final document ($i=n$):
•
Include all accumulated context
•
Label planning sample as [SUFFICIENT]
•
Create answering training instance with ground truth answer

The input formatting function (Lines 14 and 17) follows the template:

This process generates two datasets:

- •
$\mathcal{T}_{plan}$: Trains the planning ability to determine retrieval needs and generate targeted subqueries
- •
$\mathcal{T}_{ans}$: Trains the answering ability to synthesize final responses from accumulated graph context

Table [4](https://arxiv.org/html/2502.10996v2#A2.T4) illustrates the distribution of our generated datasets. The planning dataset demonstrates a balanced distribution across three label types: 35% [SUFFICIENT], 55% [SUBQ], and 10% [NO_RETRIEVAL]. The answering dataset incorporates both no-retrieval and sufficient cases from multiple sources. The size of the answering dataset exceeds the combined total of no-retrieval and sufficient cases from the planning dataset for two reasons: first, we incorporated additional data from ASQA and Arc-Easy datasets; second, we included no-retrieval cases that were initially filtered out in step 1 of our process. This comprehensive approach ensures a robust training set for the answering component.

Our labeling approach ensures that models learn both the iterative nature of complex question answering and the importance of structured knowledge representation. The complementary training objectives help develop robust reasoning capabilities while maintaining retrievability of supporting evidence.

### B.4 Dataset Statistics

**Table 4: Statistics of our constructed HotpotQA-SUBQ dataset.**
|  |  | Planning Data | Answering Data |
| --- | --- | --- | --- |
| # Queries |  | 129,902 | 78,164 (w/ 3,897 ASQA & 2,147 Arc-Easy) |
| # Input Tokens | (Mean, Median, Max) | (338, 301, 1,910) | (475, 466, 2,214) |
| # Output Tokens | (Mean, Median, Max) | (13, 12, 62) | (13, 4, 2,332) |
| # Subqueries | (Min, Mean, Max) | (0, 0.8, 5) | (0, 1.2, 6) |
| # [SUFFICIENT] |  | 45,722 | – |
| # [SUBQ] |  | 71,676 | – |
| # [NO_RETRIEVAL] |  | 12,504 | – |
| # [SUBQ] Tokens | (Min, Mean, Max) | (6, 18.3, 62) | – |
| # Nodes | (Mean, Median, Max) | (11.6, 11, 56) | (11.7, 11, 84) |
| # Edges | (Mean, Median, Max) | (21.2, 20, 116) | (21.3, 20, 160) |

### B.5 Data Examples

Figure: Figure 8: Training Data Example (Planning – [NO_RETRIEVAL])
Refer to caption: x6.png

Figure: Figure 9: Training Data Example (Planning – [SUBQ])
Refer to caption: x7.png

Figure: Figure 10: Training Data Example (Planning – [SUFFICIENT])
Refer to caption: x8.png

Figure: Figure 11: Training Data Example (Answering – [NO_RETRIEVAL])
Refer to caption: x9.png

Figure: Figure 12: Training Data Example (Answering – [SUFFICIENT])
Refer to caption: x10.png

## Appendix C Training Details

### C.1 Text-to-Triples Model Training

For our text-to-triple conversion model, we fine-tuned a Flan-T5-Large model [69](^2^22[https://huggingface.co/pat-jj/text2triple-flan-t5](https://huggingface.co/pat-jj/text2triple-flan-t5)) and a LLaMA-3.2-3B-Instruct model [70](^3^33[https://huggingface.co/pat-jj/text2graph-llama-3.2-3b](https://huggingface.co/pat-jj/text2graph-llama-3.2-3b)) to transform raw text passages into structured knowledge triples. The model processes input text sequences up to 512 tokens in length and generates structured triples in a standardized format “(S $>$ subject$|$ P$>$ predicate$|$ O$>$ object)”.

Our training dataset WikiOFGraph [44], curated by ChatGPT, comprises 5,851,776 text-triple pairs, with an additional 5,000 samples reserved for validation. Each training instance consists of a natural language text passage paired with its corresponding set of comma-separated triples.
For example:

Figure: Figure 13: Example of WikiOFGraph data
Refer to caption: x11.png

We implemented the training using the AdamW optimizer with a learning rate of 2e-5, incorporating linear warmup and decay schedules. The training process ran for 500,000 steps with a batch size of 32 per GPU and gradient accumulation over 4 steps. To optimize training efficiency and memory usage, we employed mixed-precision training using bfloat16 and applied weight decay at 0.01. The maximum source and target sequence lengths were set to 512 and 256 tokens respectively.

For evaluation, we primarily relied on ROUGE-1, ROUGE-2, and ROUGE-L scores to assess the quality of triple generation. We supplemented these metrics with custom triple matching accuracy measures that consider subject matching, predicate normalization, and object entity alignment. Validation metrics were computed at 5,000-step intervals throughout training.

The training curves for various metrics are shown in Figure [14](https://arxiv.org/html/2502.10996v2#A3.F14), demonstrating steady improvement in the model’s ability to extract structured knowledge from text. The training of this model was conducted on eight NVIDIA RTX 6000 with 48GB memory.

Figure: Figure 14: Text-to-Triple Model (base model: Flan-T5-Large) Training Curves.
Refer to caption: x12.png

### C.2 Multitask Training of Planning and Answering Capabilities

We employ Graph LLM [38, 11] as our foundational model architecture and utilize our constructed HotpotQA-SUBQ dataset to train a unified model capable of performing both graph-conditioned action planning and answer generation through multitask learning.

Since both tasks share an identical architectural foundation, as illustrated in Figure [1](https://arxiv.org/html/2502.10996v2#S2.F1), we differentiate their functionality through specialized instruction sets. For Planning, we employ the following instruction template:

Figure: Figure 15: Instruction $\texttt{INST}_{\texttt{Plan}}$ for Planning.

For the Answering, we utilize this distinct instruction set:

Figure: Figure 16: Instruction $\texttt{INST}_{\texttt{Ans}}$ for Answering.

Figure: Figure 17: Training loss comparison over one epoch. The plot compares RAS-7B (green) and RAS-8B (orange) training trajectories. Two additional RAS-7B variants are shown: “continual”: a continual learning approach where Answering training precedes Planning training, and “ptune”: a parameter-efficient variant that only tunes graph tokens without LoRA. The lower and more stable loss curves of the standard RAS variants demonstrate the effectiveness of joint training with LoRA. We use the last 100 steps for loss smoothing.
Refer to caption: extracted/6448654/figures/training_curve.png

For graph encoding, we implement Graph Transformer [65], selected for its robust capability in handling non-fully-connected graphs—a common characteristic of text-extracted triples. Our base language models comprise LLaMA-2-7B and LLaMA-3-8B, chosen both to maintain consistency with previous research [30, 31] and to investigate our framework’s performance scaling across different model capacities.

Our implementation of GraphLLM differs from G-Retriever [11] primarily due to the distinct nature of our graph structures. While G-Retriever operates on single interconnected graphs, our framework processes multiple potentially disconnected subgraphs, each corresponding to different subqueries. To address this architectural difference, we adopt a sequential encoding strategy: rather than encoding the entire graph at once, we process each subgraph individually using Graph Transformer, followed by mean pooling across all subgraph embeddings to produce the final encoded representation.

The training process utilizes 4 NVIDIA RTX 6000 Ada Generation GPUs, each with 48GB memory. We train all models for 2 epochs using a batch size of 2 and gradient accumulation steps of 2, implementing a peak learning rate of 1e-5 with a 0.15 warmup ratio and 0.01 decay rate. The maximum sequence lengths are set to 300 tokens for generation and 2,500 tokens for input, with training conducted in BFloat16 precision.

## Appendix D Evaluation Datasets & Metrics

### D.1 Test Datasets

We evaluate RAS on diverse benchmark datasets spanning short-form QA, closed-set tasks, and long-form generation in the zero-shot setting, aligning with [30, 31]. Below we describe each dataset:

Short-form Generation Datasets:

- •
TriviaQA-unfiltered (TQA) [51]: A large-scale QA dataset containing 11,313 question-answer pairs in our test set. The questions are sourced from trivia enthusiasts and cover diverse topics.
- •
2WikiMultiHopQA (2WQA) [53]: A multi-hop question answering dataset (with 12,576 samples in test set) that requires models to combine information from multiple Wikipedia articles to answer questions.
- •
PopQA [52]: A dataset focusing on questions about long-tail entities, containing 1,399 queries where the monthly Wikipedia page views are less than 100. These questions test models’ ability to handle queries about less common entities.

Closed-set Tasks:

- •
PubHealth (Pub) [54]: A dataset for verifying health-related claims. Models must classify statements as "true" or "false" based on scientific evidence. The dataset contains 987 test samples.
- •
ARC-Challenge (ARC) [55]: A multiple-choice science question dataset designed to be challenging for models, requiring multi-hop reasoning and background knowledge. The dataset contains 1,172 test samples.

Long-form Generation Datasets:

- •
ASQA [57]: A long-form question answering dataset that requires models to generate comprehensive answers with proper citation of sources. Models must balance information completeness with factual accuracy. The dataset contains 948 test samples.
- •
ELI5 [58]: A dataset derived from the "Explain Like I’m Five" subreddit, containing questions seeking straightforward explanations of complex topics. The dataset contains 1,000 test samples.

### D.2 Evaluation Metrics

We employ different evaluation metrics appropriate for each task category:

Short-form Generation:

- •
For PopQA and TriviaQA, we use the "golden match" metric [30, 71, 8](^4^44Implementation aligned with [https://github.com/AkariAsai/self-rag/blob/main/retrieval_lm/metrics.py](https://github.com/AkariAsai/self-rag/blob/main/retrieval_lm/metrics.py)), where a prediction $p$ is considered correct if it contains any normalized version of the ground truth answers $G=\{g_{1},...,g_{n}\}$:
$\text{match}(p,G)=\begin{cases}1&\text{if }\exists g\in G:\text{norm}(g)
\subseteq\text{norm}(p)\\
0&\text{otherwise}\end{cases}$
(9)
where $\text{norm}(\cdot)$ normalizes text by lowercasing, removing articles and punctuation.
- •
For 2WikiMultiHopQA, we follow RPG [31](^5^55Implementation aligned with [https://github.com/haruhi-sudo/RPG/blob/main/retriever/src/evaluation.py](https://github.com/haruhi-sudo/RPG/blob/main/retriever/src/evaluation.py)) to use token-level F1 score between prediction $p$ and ground truth $g$:
$\text{F1}(p,g)=2\cdot\frac{\text{precision}\cdot\text{recall}}{\text{precision
}+\text{recall}}$
(10)
where precision and recall are computed over normalized token overlap:
$\text{precision}=\frac{|\text{tokens}(p)\cap\text{tokens}(g)|}{|\text{tokens}(
p)|}$
(11)
$\text{recall}=\frac{|\text{tokens}(p)\cap\text{tokens}(g)|}{|\text{tokens}(g)|}$
(12)

Closed-set Tasks: For both PubHealth and ARC-Challenge, we use accuracy [30], computed as:

$$ $\text{accuracy}=\frac{|\{i:\text{norm}(p_{i})=\text{norm}(g_{i})\}|}{N}\times 100$ (13) $$

where $N$ is the total number of examples.

Long-form Generation: For ASQA and ELI5, we use multiple metrics(^6^66Implementation aligned with [https://github.com/princeton-nlp/ALCE/blob/main/eval.py](https://github.com/princeton-nlp/ALCE/blob/main/eval.py)):

- •
ROUGE-L score to measure the longest common subsequence between prediction and reference [72]
- •
MAUVE score [59] to test the generation fluency by comparing the distribution of generated text against human references

All scores are reported as percentages, multiplied by 100.

## Appendix E Inference & Evaluation Details

We evaluated RAS using both closed-source and open-source language models. For closed-source evaluation, we used Claude-3.5-Sonnet (Sonnet-3.5) as our base model. For open-source evaluation, we tested with both LLaMA-2-7B and LLaMA-3-8B, as shown in the performance table (Table [1](https://arxiv.org/html/2502.10996v2#S4.T1)).

For the open-source models, we first fine-tuned the GraphLLM architecture on our HotpotQA-SUBQ dataset (see Section [C.2](https://arxiv.org/html/2502.10996v2#A3.SS2)). We then conducted zero-shot knowledge-intensive generation tests. With Claude-3.5-Sonnet, we used few-shot prompting for both action planning and answer generation phases. Below, we detail our approach for each model type.

### E.1 Closed-Source Model Settings

For Text-to-Triple Conversion (Figure [18](https://arxiv.org/html/2502.10996v2#A5.F18)), we instruct the model to extract structured knowledge triples following specific formatting rules. The prompt precisely defines the triple format as $(S\mathbin{>}subject\mathbin{|}P\mathbin{>}predicate\mathbin{|}O\mathbin{>}object)$ and provides comprehensive guidelines to ensure consistent knowledge representation. Key rules include extracting maximal meaningful relationships, maintaining original entity casing, avoiding pronoun usage, and ensuring clear predicate specification. The example demonstrates the conversion of a biographical text into structured triples, showing how complex information about an individual’s life, career, and temporal details can be systematically decomposed into atomic facts. This standardized format enables reliable knowledge accumulation and reasoning in subsequent stages of the RAS framework.

Figure: Figure 18: Few-shot prompt for text-to-triples transformation with closed-source LLM.
Refer to caption: x13.png

For Planning (Figure [19](https://arxiv.org/html/2502.10996v2#A5.F19)), we design a comprehensive prompt that guides the model in making strategic decisions about information retrieval needs. The prompt instructs the model to output one of three labels based on careful analysis of the current knowledge state: (1) [NO_RETRIEVAL] when the question can be answered directly, either due to the model’s inherent knowledge or the question’s nature, (2) [SUBQ] accompanied by a focused subquery when additional information is needed, or (3) [SUFFICIENT] when the accumulated knowledge is adequate to answer the question.
The prompt includes diverse examples demonstrating different scenarios:

(1) Generating follow-up queries based on partial information
(2) Creating new queries when relevant information is missing
(3) Recognizing when accumulated information is sufficient
(4) Identifying questions that don’t require external knowledge
(5) Handling common knowledge questions

A key feature of the prompt is its emphasis on query efficiency - it explicitly prohibits generating redundant subqueries that might retrieve already-known information. This design helps maintain the system’s efficiency while ensuring comprehensive knowledge gathering.

Figure: Figure 19: Few-shot prompt for action planning with closed-source LLM.
Refer to caption: x14.png

For Answering (Figure [20](https://arxiv.org/html/2502.10996v2#A5.F20)), we design a prompt that focuses on synthesizing precise answers from structured knowledge graphs. The prompt emphasizes selective use of retrieved information - instructing the model to utilize subqueries and graph information only when relevant to the question at hand. A distinctive feature of this prompt is its requirement for definitive answers even under uncertainty or incomplete information, ensuring the model always provides a response.

The prompt includes carefully selected examples demonstrating two key scenarios: direct fact retrieval (Nobel Prize winner) and complex reasoning across multiple knowledge pieces (athlete’s sport classification). These examples illustrate how to effectively combine information from multiple subqueries and their associated graph information to construct accurate answers. The prompt strictly enforces concise answer generation, requiring only the essential information without additional explanation or commentary.

Figure: Figure 20: Few-shot prompt for answer generation with closed-source LLM.
Refer to caption: x15.png

In addition, for ASQA and ELI5 with closed-source models (RAS${}_{\text{Sonnet-3.5}}$ and all the other baselines (e.g., SuRe) using Sonnet-3.5), we conduct few-shot inference with two in-context learning demonstrations, aligning with previous study’s [56] implementation.(^7^77[https://github.com/princeton-nlp/ALCE/blob/main/run.py](https://github.com/princeton-nlp/ALCE/blob/main/run.py))(^8^88Exemplars can be found at [https://github.com/princeton-nlp/ALCE/tree/main/prompts](https://github.com/princeton-nlp/ALCE/tree/main/prompts))

### E.2 Open-Source Model Settings

For inference with open-source model, we use our GraphLLM trained by Hotpot-SUBQ (see Appendix [C.2](https://arxiv.org/html/2502.10996v2#A3.SS2)), we use 8 NVIDIA RTX 6000 Ada Generation with 48GB memory and CUDA version 12.4. We use Python 3.10, PyTorch 2.1.2, and torch-geometric 2.6.1 throughout all experiments.

For more efficient text retrieval, we split the
dense index of the corpus into eight splits, and load them on eight GPUs with faiss-gpu 1.7.2.(^9^99All the Wikipedia dumps (2018, 2020) we used are downloaded from [https://github.com/facebookresearch/atlas](https://github.com/facebookresearch/atlas))

We set maximum new tokens as 100 for PopQA, TriviaQA, and 2WikiMultihopQA, as 50 for PubHealth and ARC-Challenge, and as 300 for ASQA and ELI5, aligning with previous study [30, 31]. All the generation are configured with batch size of 1.

As we always set the main query as the first subquery, we pre-extract the triples from the context at the first iteration using our text-to-triple model for batch triple extraction.

We use exact same instructions for planning and answering as we used in training phase, as shown in Figures [15](https://arxiv.org/html/2502.10996v2#A3.F15) and [16](https://arxiv.org/html/2502.10996v2#A3.F16).

### E.3 Instructions

We use the following task-specific instructions for zero-shot inference:

**Table 5: Task-specific instructions. For short-form QA, we do not use instructions and use the original questions only.**
| Dataset | Instruction |
| --- | --- |
| ARC-C (baseline) | Given four answer candidates, A, B, C and D, choose the best answer choice. Please answer with the capitalized |
|  | alphabet only, without adding any extra phrase or period. |
| ARC-C | Which is true? Output A, B, C, or D. |
| PubHealth (baseline) | Is the following statement correct or not? Say true if it’s correct; otherwise, say false. Don’t capitalize or add |
|  | periods, just say “true” or “false”. |
| PubHealth | Is statement ’true’ or ’false’? Only output ’true’ or ’false’. |
| ASQA (baseline) | Instruction: Write an ACCURATE, ENGAGING, and CONCISE answer for the given question using the retrieved |
|  | graph information (some of which might be irrelevant). Use an unbiased and journalistic tone. |
| ASQA | Answer the following question. The question may be ambiguous and have multiple correct answers, |
|  | and in that case, you have to provide a long-form answer including all correct answers. [Long Form] |
| ELI5 (baseline) | Instruction: Write an ACCURATE, ENGAGING, and CONCISE answer for the given question using the retrieved |
|  | graph information (some of which might be irrelevant). Use an unbiased and journalistic tone. |
| ELI5 | Provide a paragraph-length response using simple words to answer the following question. [Long Form] |

## Appendix F Supplementary Results and Analysis

Higher Performance with More Contexts?

We further examine the performance of Sonnet-3.5 when increasing the number of retrieved documents. Table [6](https://arxiv.org/html/2502.10996v2#A6.T6) compares Sonnet-3.5 under different retrieval depths with RAS-Sonnet.

**Table 6: Performance comparison between Sonnet-3.5 under varying context sizes and RAS-Sonnet.**
| Method | 2WikiMultihopQA (F1) | PopQA (Acc) | PubHealth (Acc) | ARC (Acc) |
| --- | --- | --- | --- | --- |
| Sonnet-3.5 (#docs=5) | 53.7 | 57.3 | 53.9 | 87.1 |
| Sonnet-3.5 (#docs=10) | 55.1 | 58.5 | 56.6 | 87.9 |
| Sonnet-3.5 (#docs=15) | 55.4 | 58.0 | 56.9 | 86.9 |
| RAS (Sonnet) | 57.7 | 62.3 | 71.4 | 93.8 |

Increasing retrieval depth yields diminishing returns for Sonnet-3.5, while RAS-Sonnet achieves significantly higher performance with structured, targeted context. This highlights the limitations of passive context scaling and the benefits of RAS’s organized retrieval strategy.

Error Analysis on Graph Construction

We present two representative cases where the quality of retrieved structured knowledge significantly impacts final answer generation. These examples are distinct from the qualitative examples shown in Section [G](https://arxiv.org/html/2502.10996v2#A7) and highlight both successful and failed reasoning paths.

Case 1: Accurate Reasoning with Well-Structured Graph
Question: Who directed the film adaptation of the novel “Tinker Tailor Soldier Spy”?
Subqueries and Graphs:

- •
[SUBQ] Who directed the film “Tinker Tailor Soldier Spy”?
(S> Tinker Tailor Soldier Spy | P> Film adaptation director | O> Tomas Alfredson),
(S> Tinker Tailor Soldier Spy | P> Release year | O> 2011)
- •
[SUBQ] Who is Tomas Alfredson?
(S> Tomas Alfredson | P> Occupation | O> Film director)

Answer: Tomas Alfredson (Correct)
Analysis: The subquery identifies the correct adaptation and its director. The structured triple precisely encodes this relation, allowing the model to reason accurately.

Case 2: Reasoning Failure Due to Spurious or Missing Triples
Question: What invention is Nikola Tesla best known for?
Subqueries and Graphs:

- •
[SUBQ] What are Nikola Tesla’s major inventions?
(S> Nikola Tesla | P> Known for | O> Earthquake machine),
(S> Nikola Tesla | P> Proposed | O> Death ray),
(S> Nikola Tesla | P> Created | O> Tesla coil)
- •
[SUBQ] What is a Tesla coil used for?
(S> Tesla coil | P> Used in | O> High-frequency experiments)

Answer: Earthquake machine or death ray (Incorrect)
Analysis: The graph omits the key triple (Nikola Tesla | Invented | Alternating current system), instead surfacing fringe inventions. This misleads the model into emphasizing less impactful or speculative work, leading to an incorrect answer.

These examples reveal that RAS makes reasoning pathways more interpretable by structuring retrieved knowledge explicitly. When failures occur, they can often be traced to identifiable issues in the extracted triples, such as missing key relations or overemphasis on fringe facts, rather than opaque model behavior. This transparency not only facilitates error analysis but also suggests concrete avenues for improving upstream components like triple extraction or subquery generation, making RAS a flexible foundation for future refinement.

## Appendix G Qualitative Examples

In this section, we show some qualitative examples and running examples of RAS.

Figure: Figure 21: An inference example in PopQA dataset.
Refer to caption: x16.png

Figure: Figure 22: An inference example in TriviaQA dataset.
Refer to caption: x17.png

Figure: Figure 23: Examples of long-form generation on ASQA and ELI5.
Refer to caption: x18.png

Figure: Figure 24: A Running Example of RAS.
Refer to caption: x19.png

## Appendix H Hyperparameter Study

**Table 7: Hyper-parameter study. We highlight the setting used in experiments.**
| Hyper-parameter | Studied Values |
| --- | --- |
| Text-to-Triples Model |  |
| Batch size | {2, 4, 8, 16, 32} |
| Learning rate | {1e-5, 2e-5, 5e-5, 1e-4, 2e-4} |
| GraphLLM (Action Planner & Answerer) |  |
| \hdashline     GNN Setting (for Graph Token) |  |
| GNN architecture | {GCN, GAT, Graph Transformer} |
| Hidden dimension | {512, 768, 1024, 2048} |
| Number of layers | {2, 3, 4, 5} |
| Number of heads | {4, 6, 8, 12} |
| Dropout rate | {0.05, 0.1, 0.2, 0.3} |
| Projector intermediate dimension | {1024, 2048, 4096} |
| Projector output dimension | 4096 |
| \hdashlineLoRA Setting |  |
| LoRA rank (r) | {4, 8, 16, 32} |
| LoRA alpha | {8, 16, 32} |
| LoRA dropout | {0.01, 0.05, 0.1, 0.15} |
| \hdashline     General Setting |  |
| Learning rate | {1e-6, 2e-6, 5e-5, 1e-4} |
| Batch size (training) | {2, 4, 8, 16, 32} |
| Batch size (inference) | 1 |
| Weight decay | {0.001, 0.01, 0.05, 0.1} |
| Gradient accumulation steps | {2, 4, 8, 16} |
| Gradient clipping | {0.1, 0.3, 0.5, 1.0} |
| Warmup ratio | {0.05, 0.1, 0.15, 0.2} |
| Max text length | 2500 |
| Max new tokens | Task-specific (see Appendix [E.2](https://arxiv.org/html/2502.10996v2#A5.SS2)) |
| Others |  |
| Dense retrieval top-$k$ | 5 |

## Appendix I Broader Impacts, Safeguards, and Assets

Broader Impacts.
The RAS framework presents several promising impacts for both research and practical applications. By transforming unstructured retrieved content into structured, query-specific knowledge graphs, RAS improves reasoning transparency and factual reliability—qualities that are especially valuable in education, scientific research, and technical writing. The ability to explicitly organize retrieved knowledge into interpretable structures also opens up new possibilities for human-in-the-loop AI systems, enabling users to trace and verify model reasoning paths.

However, several ethical considerations remain. RAS’s reliance on structured knowledge representations means its effectiveness may be limited in low-resource languages or domains with sparse data. Additionally, biases present in training data can be preserved or even amplified in structured forms. Addressing these concerns will require rigorous evaluation across diverse user groups and careful curation of source corpora. Promoting fairness, transparency, and inclusivity should be core design goals for future RAS-based systems.

Safeguards.
To mitigate potential harms, we propose several safeguards during the development and deployment of RAS-based systems. First, developers should routinely audit training and retrieval corpora for biases and factual inconsistencies, and apply data augmentation or counterfactual generation techniques where appropriate. Second, regular evaluations should be conducted across a diverse range of tasks, languages, and user demographics to assess any unintended performance disparities. Third, when applied in sensitive domains, human-in-the-loop oversight should be incorporated to validate the structured knowledge outputs and final generations. Lastly, transparency measures, such as providing users with access to intermediate structured knowledge (e.g., the constructed knowledge graph), can help foster trust and enable error analysis.

Assets Used.
To support the development and evaluation of the RAS framework, we utilized a wide range of publicly available datasets, open-source tools, and pretrained models. All datasets were selected based on relevance to knowledge-intensive tasks, including question answering and long-form generation, and were licensed for academic research or distributed under permissive open-source terms such as MIT, Apache 2.0, and CC BY. Our training and evaluation relied on benchmark datasets such as HotpotQA, ASQA, TriviaQA, and ARC, while structured knowledge was derived using models trained on WikiOfGraph. We used open-source infrastructure including FAISS for retrieval, vLLM for inference optimization, and Graph LLM as our model backbone. For baseline comparisons, we incorporated implementations from Self-RAG, RPG, and ALCE. Closed-source models like Claude-3.5-Sonnet were used for performance comparison and triple extraction under API constraints.
Full licensing and usage details are provided in Table [8](https://arxiv.org/html/2502.10996v2#A9.T8).

**Table 8: Assets used in RAS. All assets are licensed or cited for research use or open-source compatibility.**
| Asset | Source / URL | License | Usage |
| --- | --- | --- | --- |
| HotpotQA | [https://hotpotqa.github.io/](https://hotpotqa.github.io/) | MIT | Base QA dataset |
| HotpotQA-SUBQ | We constructed from HotpotQA | MIT (inherited) | Subquery training dataset |
| ASQA | [https://github.com/google-research/language/tree/master/language/asqa](https://github.com/google-research/language/tree/master/language/asqa) | Apache 2.0 | Training and evaluation |
| Arc-Easy | [https://allenai.org/data/arc](https://allenai.org/data/arc) | AI2 Research License | Training (subset) |
| TriviaQA | [https://nlp.cs.washington.edu/triviaqa/](https://nlp.cs.washington.edu/triviaqa/) | CC BY-SA 4.0 | Evaluation |
| PopQA | [https://huggingface.co/datasets/akariasai/PopQA](https://huggingface.co/datasets/akariasai/PopQA) | CC BY 4.0 | Evaluation |
| 2WikiMultihopQA | [https://github.com/Alab-NII/2wikimultihop](https://github.com/Alab-NII/2wikimultihop) | Apache 2.0 | Evaluation |
| PubHealth | [https://huggingface.co/datasets/bigbio/pubhealth](https://huggingface.co/datasets/bigbio/pubhealth) | MIT | Evaluation |
| ARC-Challenge | [https://allenai.org/data/arc](https://allenai.org/data/arc) | AI2 Research License | Evaluation |
| ELI5 | [https://huggingface.co/datasets/eli5](https://huggingface.co/datasets/eli5) | Apache 2.0 | Evaluation |
| Wikipedia 2018/2020 | [https://github.com/facebookresearch/atlas](https://github.com/facebookresearch/atlas) | Creative Commons Public Licenses | Knowledge source for retrieval |
| WikiOfGraph | [https://github.com/daehuikim/WikiOFGraph](https://github.com/daehuikim/WikiOFGraph) | CC BY 4.0 | Triple model training |
| Claude-3.5-Sonnet | [https://www.anthropic.com](https://www.anthropic.com) | Proprietary | Default closed source model |
| LLaMA-2/3 | [https://ai.meta.com/llama](https://ai.meta.com/llama) | Meta LLaMA Community License | Default open source model |
| LLaMA-3.2-3B | Derived from Meta LLaMA | Meta LLaMA Community License | Triple extraction |
| Flan-T5-Large | [https://huggingface.co/google/flan-t5-large](https://huggingface.co/google/flan-t5-large) | Apache 2.0 | Triple extractor |
| Graph LLM | Modified from [https://github.com/XiaoxinHe/G-Retriever](https://github.com/XiaoxinHe/G-Retriever) | Research use | Model backbone |
| Faiss | [https://github.com/facebookresearch/faiss](https://github.com/facebookresearch/faiss) | MIT | Dense retrieval |
| vLLM | [https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm) | Apache 2.0 | Optimized inference |
| Sentence-BERT | [https://www.sbert.net](https://www.sbert.net) | Apache 2.0 | Graph node/edge embedding |
| Self-RAG Code | [https://github.com/AkariAsai/self-rag](https://github.com/AkariAsai/self-rag) | MIT | Baseline eval |
| RPG Code | [https://github.com/haruhi-sudo/RPG](https://github.com/haruhi-sudo/RPG) | Research use | Baseline eval |
| ALCE Code | [https://github.com/princeton-nlp/ALCE](https://github.com/princeton-nlp/ALCE) | MIT | Eval metrics |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Please refer to the weakenesses.
soundness: 2
strengths: 1. The paper is easy to follow.
2. The studied problem is important.
3. The proposed method is evaluated on an extensive set of datasets.
summary: LLMs perform well on knowledge-intensive tasks but struggle with multi-step reasoning due to unstructured retrieved context, a limitation RAG shares. Aligning with structured intermediate reasoning insights, the authors propose RAS—a framework that dynamically builds question-specific knowledge graphs via iterative retrieval and incremental graph construction. Evaluated on seven benchmarks, RAS outperforms baselines by up to 8.7% (proprietary LLMs) and 7.0% (open-source LLMs), validating dynamic, query-tailored knowledge structuring as effective for enhancing reasoning accuracy/robustness. Data and code are publicly available.
weaknesses: 1. Important related work is neglected. Constructing dynamic knowlege graph for sovling complex reasoning tasks in RAG is a well-studied area. Representative work, such as SG-Prompt, ERA-CoT, and KnowTrace, is not analyzed in the paper, especially KnowTrace. Compared with these existing work, the novelty and technical contribution are limited.
2. Important baselines are missing in the experiments. It is not clear the true performance of the proposed method among existing work.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: Since graph extraction can be viewed as a form of information organization and denoising, complex sentences with multiple conditions might lose contextual nuances when converted to triples.
Why not combine graph representations and original retrieved text during answering, to preserve both structure and completeness?
soundness: 2
strengths: + The paper introduces RAS, a dynamic, query-specific knowledge graph construction framework that avoids inefficiencies of global KG indexing and eliminates costly offline graph building, achieving pay-per-query scalability.
+ The method is evaluated comprehensively on multiple datasets across both open-source and closed-source LLMs, demonstrating consistent and generalizable effectiveness.
+ The framework improves reasoning transparency and provides a structured way to bridge retrieval and reasoning in a unified pipeline.
summary: The paper proposes Retrieval-And-Structuring (RAS), a framework designed to enhances reasoning in large language models for knowledge-intensive tasks. Unlike traditional RAG approaches that retrieve but fail to organize context, RAS dynamically builds question-specific knowledge graphs through iterative retrieval and structured knowledge synthesis. The framework consists of three key stages: planning (to identify knowledge gaps), retrieval and structuring (to form factual triples), and graph-based answering. Experiments across seven diverse benchmarks demonstrate gains across multiple datasets.
weaknesses: - The paper lacks concrete examples demonstrating how graph representations outperform plain text in enhancing reasoning accuracy. It remains unclear which aspects of reasoning (e.g., factual grounding, compositional reasoning, or logical chaining) benefit most from graph structuring.
- Moreover, since RAS and RPG are trained on different datasets of different scales, and the appendix shows that training set size directly affects performance, the gains of RAS over RPG on open-source settings (insignificant on half of the datasets) might stem from data scale rather than the graph mechanism itself.
- In principle, clearer graph structures should lead to larger improvements on more complex reasoning tasks, yet the paper does not analyze this correlation or provide corresponding comparisons to validate the claim that graph structuring enhances reasoning.
- There are several presentation issues, such as missing “top-2” formatting in Table 1 (best/second-best not consistently marked) and missing figure references in the appendix.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: See weakness!
soundness: 3
strengths: 1. I appreciate this paper for utilizing the graph form representing knowledge and an elegant workflow to address complex and knowledge-intensive tasks. 
2. The experiments are solid across many classic knowledge-intensive tasks, and the improvement is good.
3. This framework is general for closed and open-source LLMs, where the code is open-source.
summary: This paper proposes a RAG framework for structuring the context in a graph form for improving knowledge-intensive tasks. In detail, this framework is composed of sub-question generation, retrieval-and-structuring, and terminal-and-answering. Additionally, this paper also proposes a training framework to enhance the model to be compatible with this framework. Then the comprehensive evaluation over five tasks shows the advantages of new framework over closed and open source models.
weaknesses: 1. This paper lacks the motivation for why organizing context in a graph form improves the performance of RAG. 

2. The involved benchmarks are outdated. There are many new graph-rag datasets for reasoning on domain knowledge. The results on such datasets are necessary to improve the quality.

[1] GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation
[2] When to use Graphs in RAG: A Comprehensive Benchmark and Analysis for Graph Retrieval-Augmented Generation

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses above. Please respond to these cons.
soundness: 3
strengths: 1. The paper is clear and well-written, with a reasonable motivation and considerable insight.

2. The experiments are sufficient and provide a comprehensive analysis.

3. The reproduction is excellent, with code and extensive reproduction details provided.
summary: This paper proposes a planning-based iterative RAG method for knowledge-intensive LLM generation. It conducts experiments on several datasets to verify its effectiveness.
weaknesses: 1. The backbone used in the experiments appears somewhat outdated. We recommend using a more recent backbone, such as Claude-4 and LLama-3.1. Alternatively, consider the GPT-4/5 series or the Qwen3 series.

2. Please provide a more detailed analysis of the core differences between this paper and other iterative RAG methods.


Decision: Accept (Poster)