Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation
Authors: Prafulla Kumar Choubey
Xin Su
Man Luo
XIANGYU PENG
Caiming Xiong
Tiep Le
Shachar Rosenman
Vasudev Lal
Phil L Mui
Ricky Ho
Phillip Howard
Chien-Sheng Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Graph
RAG
Synthetic Data Generation
Knowledge Distillation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose SynthKG, a multi-step data synthesis pipeline for document-level KG construction, and Distill-SynthKG, a distilled single-step model enabling efficient, high-quality KG generation and retrieval.

Abstract: Document-level knowledge graph (KG) construction faces a fundamental scaling challenge: existing methods either rely on expensive large language models (LLMs), making them economically nonviable for large-scale corpora, or employ smaller models that produce incomplete and inconsistent graphs. We find that this limitation stems not from model capabilities but from insufficient training on high-quality document-level KG data. To address this gap, we introduce SynthKG, a multi-step data synthesis pipeline that generates high-quality document-KG pairs through systematic chunking, decontextualization, and structured extraction using LLMs. By fine-tuning a smaller LLM on synthesized document-KG pairs, we streamline the multi-step process into a single-step KG generation approach called Distill-SynthKG. Furthermore, we repurpose existing question-answering datasets to construct KG evaluation datasets and introduce new evaluation metrics. Using KGs produced by Distill-SynthKG, we also design a novel graph-based retrieval framework for RAG. Experimental results demonstrate that Distill-SynthKG not only surpasses all baseline models in KG quality (including models up to eight times larger) but also consistently improves in retrieval and question-answering tasks. Additionally, our proposed graph retrieval framework outperforms all KG-retrieval methods across multiple benchmark datasets.

Published as a conference paper at ICLR 2026
SCALING KNOWLEDGE GRAPH CONSTRUCTION THROUGH SYNTHETIC DATA GENERATION AND DIS-
TILLATION
Prafulla Kumar Choubey"1, Xin Su Man Luo"3, Xiangyu Peng! Caiming Xiong'. Tiep Le Shachar Rosenman', Vasudev Lal* Phil Mui', Ricky Ho Phillip Howard Chien-Sheng Wui! 'Salesforce Research, TTouughtworks. 'Abridge, *Oracle, 'Intel Labs
Correspondence: pchoubey salesforce.com. xin.su thoughtwwors..com com, m aoo@ariidgecom
MMain authors contributed equally, (Senior authors
ABSTRACT
Document-level knowledge graph (KG) construction faces a fundamental scal- ing challenge: existing methods either rely on expensive large language models (LLMs), making them economically nonviable for large-scale corpora, or employ smaller models that produce incomplete and inconsistent graphs. We find that this limitation stems not from model capabilities but from insufficient training on high-quality document-level KG data. To address this gap, we introduce SynthKG, a multi-step data synthesis pipeline that generates high-quality document-KC G pairs through systematic chunking, decontextualizzttio, and structured extraction using LLMs. By fine-tuning a smaller LLM on synthesized document- KG pairs, we streamline the multi-step process into a single-step KG generation approach called Distill-SynthKCG Furthermore, we repurpose existing to construct KG evaluation datasets and introduce new evaluation metrics. Using KGs produced by Dystill-SynthKGG we also design a novel graph-based retrieval framework for RAG. Experimental results demonstrate that Distill- Distill-SynthKG not only surpasses all baseline models in KG quality (including models up to eight times larger) but also consistently improves in retrieval and guestion-answering tasks. Additionally, proposed graph retrieval framework outperforms all KG- retricval methods across multiple benchmark datasets.
datasets
INTRODUCTION
Retrieval Augmented Generation (RAG) has been widely adopted for effectively connecting large language models (LLMss) with external knowledge sources. Recently, Knowledge Graph (KG)- augmented RAG methods have demonstrated strong potential, offering several advantages such as effective corpus-level information summarization (Edge et al. (2024), improved reasoning capabilities
Guutéérezz et al. 2024 Li et al. 2024). and accurate modeling of historical customer issue resolutions
QA (Xu al. 2024).
Recent works (Edge et al. 2024 Gutiérrez et al. 2024) have begun exploring the use of LLMs to automate the construction KGs, which then serve knowledge sources for tasks such as question answering or building intelligent agentic frameworks. However, these approaches face a fundamental scaling challenge: they rely on simple zero-shot or few-shot prompting to construct KGs in a single step using L.MSs such as GPT-4o (OpenAI, 2024). Consequently, they can incur significant inference costs when applied to large corpora due to the need for many commercial API calls. These methods also lack a rigorous, reliable design tailored to document-level KG construction, and processing entire documents with LLMs, particularly for long texts, can lead to information loss (Edge et al. 2024). Finally, the lack of datasets and evaluation methods for document-lever ontology- ffree KGs makes it difficult to determine whether errors in RAG systems stem from failures in specific reasoning components or from low-quality KGs that propagate errors throughout the system.

---

Published as a conference paper at ICLR 2026
We find that these limitations stem not from model capabilities but from the absence of high- training data for document-level KG construction. While other structured extraction tasks benefit from supervised datasets, ontology-free free KG construction lacks substantial training corpora, forcing reliance on expensive zero-shot inference. To address this data gap, we introduce SynthKG, a data synthesis pipeline for KG construction. We further distill this pipeline into smaller L.M,, Distill-SynthKe which enables efficient one-step generation of high-quality document-level KGs. In SynthKG, we begin by splitting the input document into manageable, semantically complete text chunks. Each chunk is then processed through decontextualization step that performs entity disambiguation using prior context, making each chunk a self-contained unit. We then prompt the LLM to extract entities, relations, and relevant propositions from each chunk, which are combined to form the final KG. By fine-tuning Distill-SynthKG on the synthetic document-KG pairs produced by SynthKG, smaller models can generate high-quality KGs for a given document in a single inference step.
Additionally, we propose a method for constructing evaluation datasets for document-level, ontology- free KGs, along with a corresponding evaluation framework. Specifically, we repurpose existing multihop QA datasets by converting questions and answers into ground-truth relation triplets, where the answer appears the head, tail, predicate of triplet. Using these ground-trut truth triplets for each document, we introduce semantic-similant and keyword- -bsddd metrics to assess KG triplet coverage. Finally, we present graph-based retrieval framework built KGs generated by Distill-SynthKGG We design progressive retrieval method that begins with proposition retrieval and leverages the graph structure retrieve related triplets. propositions, and text chunks relevant the query. Our retriever outperforms state-of- -f-th-rrt methods in both retrieval and question guestion-answering accuracy across three multihop QA datasets: MuSiQue (Trivedi al., 2022), 2WikiMultHHoPAA (Ho al. 2020), and HotpotQA (Yang et al.) 2018). Moreover, our KG coverage evaluation framework correlates strongly with both QA and retrieval performance, demonstrating its effectiveness for evaluating document-level KG coverage.
In summary, our contributions are as follows:
1. We introduce SynthKG, a systematic data synthesis pipeline that generates high quality document- level ontology-free KGs, addressing the training data scarcity in KG construction.
2. We present Distill-SynthKG; demonstrating that smaller LLMs can achieve large-model KG construction quality when trained on appropriate synthetic data.
3. We establish comprehensive KG evaluation datasets and metrics by re-purposing existing multi-hop
QA datasets.
4. We design novel graph-based retrieval framework that effectively leverages KG structure for RAG.
5. Our experiments show that Distill-SynthKG produces higher-quality KGs than all baselines, including models up to eight times larger, while consistently improving retrieval and question- answering performance.
RELATED WORK
Recently, there has been growing interest in using KGs for different Retrieval-Augmented Gen- eration (RAG) applications. For instance, GraphRAG (Edge et al.) 2024) shows the advantages of using KGs over text corpus for answering global queries that require summarizing information from multiple documents. HippoRAG (Gutiérrez et al. 2024) demonstrates that applying personalized PageRank algorithms on LLM-derived KG can enhance retrieval accuracy for complex multi-hop reasoning questions. GraphReader (Li et al. 2024) shows how KGs can enable LLM agents to plan and reason long context to answer complex questions. These approaches focus on maximizing KG utility.
All the above work, along with many others such as Chia et al. (2022) ) T Trajanoska et al. (023))
Chen & Bertozzi 2023); Kai Zhang (202)); Nayak & Timmapathini (2023) ; Mihinduku lasooriya ct al. (2023); Zhu et al. (2024) ; Jiao et al. (2023), Khorashadizadch et al. (2023); Han et al. (2024); Yao et al. (2024); Bi et al. (2024) ; Ding et al. (2024); Sanmartin (2024); Sun et al. (2024); Yao et al. (2023): ; Chase (222) have used LLM prompting to build KGs or extract semantic relation triplets from text. However, all prior works have overlooked improving the efficiency of ontology-free KG construction. We are the first to develop a specialized LLM for KG construction, enhancing efficiency by shifting from large models to smaller, more efficient models without sacrificing performance.
2

---

Published as a conference paper at ICLR 2026
3 DISTILL-SYNTHKG
We present Distill-SynthKG. a data-centric approach for scaling KG construction. Rather than relying on increasingly larger models, generate high-quality training data through a systematic pipeline (SynthKG) and use it to train smaller models for document-level KG construction. This enables efficient single-step KG generation from documents using smaller-scale LLMs while matching the quality of expensive multi-step pipelines.
Synth30: Mail ti-stap Pranwook
Sangle-stup Tranwwock
blati Sy-nnbba
Tenn
proferse statt to
Raraask Dup
Step 3
Eutky Extructias
Long
Disiilation
Smallr
Nurmretteal Eassunk Lap
Stap ard Extacic
Propoilton Entily
KG Costrtation
Glmph Bettieni
Figure 1: Our SynthKG data synthesis method (left) generates high-coverage, ontology-free, document-level KGs. We distill this synthetic data into Distill-SynthKG (right), which is applied to multiple downstream applications. Long document refers to multi- multi--prragrph documents.
3.1 SYNTHKG: A DATA SYNTHESIS ENGINE
Traditional prompt-based KG extraction treats each document as an isolated zero-shot problem, leading to inconsistent outputs and preventing systematic learning. SynthKG addresses this by decomposing KG construction into reproducible stages that generate consistent, high-quality training data. The pipeline consists of two main steps: (1) document chunking and decontextualizztionn followed by (2) entity, relation and proposition extraction. These steps ensure high coverage of extracted entities and relations while minimizing information loss. We present overview of SynthKG Figure [ with detailed prompts in Section
Document Chunking and Decontextualization Directly inputting long texts into LLM has been shown to result in information loss (Edge et al., 2024). To mitigate this risk, we first split each input document into smaller, more manageable chunks before processing them with the LLM in subsequent steps. This chunking is done along sentence boundaries, without overlap, to preserve semantic coherence and avoid redundancy.
However, processing each chunk in isolation can lead to a loss of prior context. For example, "John Doe" appears in one chunk and "John" in another, we might lose track of who "John" refers to. To prevent this, we apply "decontextualization" step, where we prompt the LLM to rewrite each chunk, replacing all entity mentions with their most informative form based on the context of the preceding chunk. This step serves critical dual purpose: ensuring entity consistency across chunks and creating self-contained text units that can be independently processed. For example, "John Doe" is introduced in a previous chunk, subsequent mentions of "John D." "John," related pronouns are replaced with "John Doe." This not only preserves context but also prevents the same entity from being represented in different forms, which could lead to redundancy, discontinuous KG paths, and reduced accuracy at inference time. The first chunk of a document is not decontextualized, chunking does not lead to context loss in this case. We provide an example of decontextualized chunk in Figure
To verify that the preceding chunk is sufficient for decontextualization, calculate the average chunk distance for the same entity within each document in our generated dataset of 100K samples (details of this dataset described in Section 6.1.. Specifically, we measure the distance between

---

Published as a conference paper at ICLR 2026
the first occurrence of each entity and its subsequent mentions. The overall average chunk distance per entity is 0.9, indicating that, on average, entities are mentioned again within less than one chunk after their first mention. This suggests that using only the preceding chunk is sufficient and that no significant number of entities remain unresolved due to the chunk-hased decontextualizztion process.
A potential drawback of prompting an LLM rewrite chunks is that the rewritten text drift from the source, introducing information loss or hallucinations. To mitigate this risk, we compute ROUGE scores between the original and decontextualized chunks and filter out cases with substantial divergence. Full experimental settings are reported in Section 6.1
To further evaluate decontextualizatioon accuracy, manually annotate 75 randomly selected rewrit- ten chunks. Three authors each label 25 chunks, with access to both the original chunk and the full document, For each chunk, annotators assess whether any modifications were made, whether those modifications are correct, and whether any information was lost.
Across the 75 chunks, we record 593 edits in total, of which only six involve incorrect modifications and four exhibit information loss. Overall, these results suggest that decontextualization produces high-quality, self-contained text. Qualitatively, most edits increase specificity, for example, replacing generic term like "scientists" with "Darwinian scientists," indicating that the rewritten chunks are typically understandable without additional context.
Entity and Relation Extraction Similar Edge et (0024) and Gutiérrez et al. (2024). we first prompt the LLM to extract all entities and their corresponding types from each text chunk, as shown Step 3 of Figure Then, we prompt the LLM again to generate all propositions and corresponding relation triplets based on the text chunk and previously extracted entities. Each relation is represented by quadruplets consisting of source entity, predicate, target entity, and proposition (see Figure for examples). The proposition is sentence that describes the semantic relation between the source and target entities, encapsulating all key details of that relation.
We extend traditional KG triples by adding proposition component, which functions inter- mediate chain of thought (Wei et al. 2022) enabling the LLM to first articulate the relevant context coherently before extracting the corresponding triplets. This approach therefore better leverages contextual information. Additionally, the proposition acts as a fine-grained, self-contained retrieval unit, which facilitates the construction of KG-based retrieval indices. Beyond triplets and text chunks, our final KG incorporates these clear, independent propositions. For example, the proposition "OWC Pharmaceutical Research Corp preferred stock convertible to common stock $0.20 per share." provides important contextual details, such as the "conversion price $0.20 per share," and also serves as a precise, indexable unit.
Crucially, this multi-step decomposition creates consistent patterns that can be learned. Unlike single-step prompting which produces variable outputs. our pipeline generates KGs with predictable structure: consistent entity naming from deconiextualizatton. systematic coverage from chunk- by-chunk processing, and interpretable intermediate representations through propositions. These properties make the outputs suitable as training data for smaller models.
3.2 DISTILLING SYNTHKG
While SynthKG's detailed, chunk-by-chunk workflow enables high-quality KG construction with LLMs, it creates efficiency challenges. Building a KG from single document requires multiple LLM calls, which increases computation or API cost and limits scalability. For instance, processing 1,000-word document requires 12 inference calls: the document split into four chunks, and each chunk triggers three calls for decontextualization. entity extraction, and relation extraction.
To scale KG construction, we distill the multi-step SynthKG pipeline into a single-step model, as shown in Figure [I] The key insight is that once KG construction is decomposed into systematic stages, it can be learned as a pattern recognition problem rather than re-solved through multi-step prompting at inference time. Concretely, we fine-tune a smaller LLM the document- KG pairs generated by SynthKG, enabling it to process full documents end-to-end and produce high-quality KGs in single forward pass.
This distillation approach overcomes the limitations direct prompting because the synthetic training data provides two essential signals: (1) consistent examples of handling long documents without
4

---

Published as a conference paper at ICLR 2026
losing information, learned from the chunk-aggregater outputs; and (2) an implicit encoding of the multi-step procedure into the model parameters. As a result, the model learns not only to extract triplets, but also to preserve entity consistency and the coverage patterns demonstrated by SynthKG. Notably, no large-scale dataset currently exists for this form of dooumenn-to-KK training, making SynthKG crucial for generating the data required enable effective distillation.
4 KG COVERAGE EVALUATION
Evaluating the quality of the extracted KG is essential for our data-centric approach, as it enables systematic assessment of both the synthetic training data and the distilled models. However, there is lack of document-leved evaluation resources for ontology-free KGs. Although DocRED (Yao et al.) 2019) is one existing dataset, is limited to just 96 relations, making it unsuitable for open-domain KGs that require diverse, unconstrained relations. To address this gap and enable scalable evaluation, we propose framework that re-p re-purposes multihop QA datasets to create proxy ground truth triplets, providing the first systematic evaluation method for document-level ontology-free KG construction.
Proxy Triplets Generation We leverage multihop QA datasets because they naturally encode the structured facts needed to answer complex questions, which is exactly the kind of information KGs should capture. We use GPT-4o to generate triplets from QA pairs, since each multihop question implicitly contains multiple interconnected facts. When dataset provides these facts as subquestion- -nsweer pairs, we form triplets directly from those pairs and ensure that the final answer appears the head, relation, tail of at least one triplet. When subquestions supporting facts not available, we first prompt GPT-4o to generate the necessary subquestions and then generate the corresponding triplets.
Although this procedure is synthetic, yields a scalable and consistent evaluation framework that would be impractical to obtain through manual annotation at the scale required for training data validation. The prompts for triplet generation and question decomposition, along with illustrative examples, are provided in Appendix D.I Human evaluation shows that the resulting triplets largely valid (86% accuracy); see Appendix D.2 for details.
KG Coverage Evaluation Metrics Existing KG evaluation metrics typically rely exact match or token-level F1, often assuming relations come from a predefined schema. This assumption breaks down for ontology-free KGs, where entity and relation strings are unconstrained and surface-fom variation is common. To handle this setting, we perform semantic matching to align extracted triplets with ground-truth triplets and report three complementary measures: semantic scores, triplet coverage, and F1.
We emphasize coverage over precision for practical reason. In RAG applications, missing critical information (low recall) is usually more harmful than extracting additional facts (lower precision): irrelevant triplets can often be ignored during retrieval, whereas absent triplets can make relevant questions unanswerable. Accordingly, our metrics focus on whether the graph includes the key triplets needed to answer questions. As proxy for overall comprehensiveness, we also report the total number of extracted triplets.
Our three proposed metries are defined follows:
Semantic score: We calculate the cosine similarity between the vector representation of each ground truth triplet and the triplets in the KG, taking the highest similarity score as the semantic score for that ground truth triplet. higher semantic score indicates closer match between the ground truth and the extracted graph.
Tripler Coverage: If the semantic score for ground truth triplet exceeds cutoff threshold, marked as covered (coverage 1); otherwise, the triplet is not covered (coverage = 0).
F/ score: We use the semantic score to identify the triplet from the KG that most closely matches the ground truth triplet. Then, we compute the F1 score by comparing the text of the extracted and ground truth triplets.
These metrics collectively provide a comprehensive view of KG quality: semantic score captures conceptual alignment, coverage measures recall of critical information, and F1 score assesses surface-
5

---

Published as a conference paper at ICLR 2026
level accuracy. Together, they enable systematic comparison across different KG construction methods and validate the effectiveness of our data-centric approach.
PROPOSITION-ENTITI GRAPH RETRIEVER
We introduce a retrieval framework that leverages the structure of our synthesized KGs, in particular the proposition-entiin bipartite graph (Figure 2 Unlike prior graph-based retrieval methods that operate over sparse triplets, our retriever treats propositions the primary retrieval units. Be- cause propositions are rich, self-contained text statements, they serve as higher-quality candidates for RAG while the bipartite topology provides ex- plicit links through shared entities, promoting log- ical connectivity across retrieved evidence. This design is made possible by the consistent KG struc- ture produced by our data-centric pipeline.
Graph
Reicant
Query
Step 1
Propostton satly
E2
Quary Enilyy
Given question, we first retrieve the top-M most relevant propositions from the KG using embed- ding similarity, narrowing the search space to a smaller subset of relevant information. This initial semantic retrieval is crucial because propositions, unlike raw triplets, contain sufficient context for meaningful similarity computation. In step 2, we construct sub- graph consisting of these proposi- tions and their linked entities, capturing the rela- tions among the retrieved propositions. In step 3, we traverse the sub-graph starting from the entities mentioned in the question, selecting only propo- sitions within their N-hop neighborhood. This graph-constrained filtering addresses key chal- lenge in multi-hop reasoning: distinguishing between semantically related but logically disconnected information. For instance, propositions about "Washington" the president versus "Washington" the state may have high embedding similarity but different graph neighborhoods. We then include text chunks corresponding to the selected propositions within N-hop distance to question entities, ranked by their embedding similarity to the query, until the top-K chunks selected. We call this approach Graph Retriever.
PS
Propesiton linkad Query Ensty Sutggrahh
PLLP3
Step Re-reiinn
Figure 2:
Our Proposition-Entiiy Graph Re- triever for multi-hop reasoning retrieves seman- tically similar propositions, uses graph traversal to select those connected through query entities, and then re-ranks selected propositions using LLMs.
Additionally, as shown in Step 4 of Figure we augment retrieval with ALM-based re-ranking. Given the candidate propositions returned by the Graph Retriever, we prompt an LLM to select the propositions required to answer the question, using its reasoning ability to re-rank the evidence. This is feasible because propositions are interpretable text units that an LLM can directly judge for relevance, unlike isolated triplets.
After re-ranking, we prioritize the chunks associated with the LLM-selected propositions, and then fall back the Graph Retriever to fill the remaining budget until the top-K chunks are selected. We refer to this hybrid method as Graph+LLM in Section 6
The design of this retrieval framework tightly coupled with our data synthesis approach: the consis- tent entity naming from deconntetuullization ensures reliable graph traversal, while the proposition generation provides semantically rich retrieval units.
6 EXPERIMENTS
Our experiments designed to validate the data-centric approach across three dimensions: (1) the quality of synthetic training data, (2) the effectiveness distillation, and (3) the performance of downstream applications. We address the following research questions: (RQ1) Does the multi-step

---

Published as a conference paper at ICLR 2026
SynthKG pipeline produce higher-quality KGs than direct prompting? (RQ2) Can 8B model achieve large-model performance through distillation on synthetic data? (RQ3) How does data scale affect the performance of distilled models? (RQ4) Does improved KG quality translate to better retrieval and QA performance?
6.1 KG SYNTHESIS AND DISTILLATION SETTINGS
SynthKG Dataset and Model:
To create a large-scale training resource, we use -lama-3.1-700 Instruct (AI@ Meta 2024) to synthesize KGs from 100K documents from IndustryCorpus (by BAAI). We ensure domain diversity by sampling equally from ten categories: politics, news, medicine, literature, finance, film & TV, computer science, automotive, technology, and education. We use the SentenceSplitter from the Llama-Index (Liu 2022) framework to split documents into chunks, setting the chunk size to 256 tokens and chunk overlap to 0 tokens. We apply filtering criterion based on the ROUGE-1 Fl score (Lin 2004), setting a threshold of 0.70 to minimize the risk of hallucinations from decontextualizatton. We perform the KG synthesis using VLLM (Kwon et al.) 2023) on 160 Intel Gaudi 2 AI accelerators in the Intel Tiber" TM AI Cloud. Our 100K generated document KeuumentKKG pairs represent the first large-scale training resource for ontology-free KG construction and will be publicly released.
SynthKG Distillation: We validate that synthetic data enables effective distillation by training (AT@ Meta 2024 30K synthesized documents. The model learns to directly generate corresponding KGs for entire input documents in a single pass. Training uses 8 lntel Gaudi 2 AI accelerators with a learning rate of Sc-5, batch size of 32, for one epoch. We name our model Distill-SynthKG and refer to it subsequently as D-SynthKG-8b.
6.2 EVALUATION SETTINGS
Datasets We evaluate on three multi-hop reasoning benchmarks that require integrating information across multiple facts: MuSiQue, 2WikiMultiHopQA (2Wiki), and HotpotQA. These datasets are well suited for KG evaluation because answering their questions depends on structured, multi-fact reasoning, which is precisely the behavior KGs are intended support. Following the experimental protocol of HippoRAG (Gutiérrez et al.] (024)), we use the same 1,000 questions and the same candidate passage sets, which include both supporting and distractor passages, to enable a fair comparison. For KG coverage evaluation, we generate proxy ground-truth triplets with GPT-4o as described in Section 4
Baselines
We compare against multiple baseline categories for comprehensive evaluations:
Direct prompting baselines: Llama-3-8b and Llama-3-70t for single-step KG extraction
Multi-step baselines: Full SynthKG pipeline with Llama-3-88 (SynthKG-8b) and Llama-3-706 (SynthKG-70b) assess distillation effectiveness
Retrieval baselines: Standard dense vector retrieval and a two-stage approach combining dense retrieval with L.---bseed re-ranking (Dense+LLM) to establish non-KG performance benchmarks KG-based retrieval: Retrieval using GPT-4o-based KGs
KG-RAG systems: GraphRAG and HippoRAG using GPT-4o-extracted KGs, representing state-of-
the-art KG-based approaches
Closed- book QA: LLM using only parametric knowledge, establishing the lower bound
We provide full experimental details including hyperparameters in Section F
Multihop QA Frameworks To demonstrate the versatility of our KGs, we evaluate using three distinct retrieval frameworks using LlamaIndex's Srmmarieee with GPT-4o for answer generation:
We use the Instruct variants of both models throughout
"To improve GraphRAG performance, we append instruction: "Only provide the answer without any context. For yes/no questions, just mention yes or no. Do not cite data sources." at the end of each query. For GraphRAG, we report results using the local and drift modes, which yield the best performance; the global mode is excluded. For HippoRAG, use GPT-4o for KG construction and apply our query synthesizer to the retrieved text chunks generate the final answer, ensuring fair comparison.
7

---

Published as a conference paper at ICLR 2026
LlamaIndex: Standard KG-based retrieval using LlamaIndex's KnowledgeGraphhldex with hybrid keyword and semantic search
Direct KG reasoning by decomposing questions into sub-queries and retrieving
matching triplets (details in Appendix
L.L. Our proposition-entity graph retriever with LLM re-ranking, leveraging the unique structure of our synthesized KGs
7 RESULTS
MuSiQue
2wiki
HotpoiQA
Source
Tripicts
Semantic
Coverage
F1
Triplets
Semantic
Coverage
F1
Triplcts
Semanic
Coverage
Fl
Llama-3.88
0.51
41384
41.79
SynthKG--hh
125197
0.8341
38.84
0.55
56178
0,8275
44.56
0.34
108051
0.8448
47.72
0.60
Llama-3-70b
102119
0.56
0.8475
82948
0.8440
47.20
SynthKGG70b
140527
0,8859
47.18
0.59
71305
0.8778
63.30
0.61
124460
0.8633
54.54
0.63
K-SynthKG-Sh
139376
0.8546
46.90
0.59
68800
0.8693
58.27
0.59
123458
0.8693
55.26
0.64
Table I: KG coverage performance. The best scores are bolded, and the second-best scores are underlined.
We present experimental results that validate our data-centric approach to KG construction, directly addressing the research questions posed in Section 6
7.1 KG COVERAGE RESULTS (RQ1: MULTI-STEP VS DIRECT PROMPTING)
The multi-step SynthKG pipeline consistently generates more triplets and achieves higher coverage than the commonly used single-step LLM prompting approach across all three datasets, for both LLaMA-3-8b and 706 models (Table T This validates our hypothesis that decomposing KG construction into systematic steps prevents the information loss inherent in single-pass processing. Furthermore, our D-SynthKG-8b model outperforms the untrained Llama-3-8b, Llama-3 1aama-3-7700 3-70b, and SynthKG-8b baselines, demonstrating the benefit of distilling the SynthKG pipeline using Llama- 3-70b as the teacher. Remarkably, D-SynthKG-8b is also highly competitive with SynthKG-70b, despite being approximately ~8x smaller and relying on a single-step inference process. These results underscore that high-quality training data can bridge the gap between model sizes. As our KG coverage metric emphasizes recall of critical information, we also verified precision through manual inspection. Among 150 randomly sampled triplets from D-SynthKG-B8 Ghb''s predictions, only 4 were incorrect and 5 meaningless, indicating that the generated triplets largely align with the source content while maintaining high coverage.
7.2
RETRIEVAL (RQ2& RQ4: DISTILLATION EFFECTIVENESS AND DOWNSTREAM IMPACT)
KG Source
Retrlever
MuSiQue
2wiki
HorperQa
Hitdd 10)
MRR
MAP
Hire02
Hisvde
MRR
MAP
Hiss88
Hisee 10
MRR
MAP
None
74.72
Noor
DenseLLLM
47.60
67.02
84.44
44.26
72.63
76.70
97.77
38.65
83,10
92.10
96.79
67.58
Llama-3-8b
Graph e LLM
31.33
42.68
60.67
29.29
41.55
45.60
66.53
36,70
50.65
57.45
73.72
45.06
SynhKGG88
Graph- LLM
50.62
65.17
86.65
45.43
65.25
69.65
95,54
54.79
76.55
86.35
92.69
63.44
1aama-3-70b
Graph
48.64
68.73
74.47
97.32
57.42
79,10
93.75
93.27
65.7%
SymthiKG770b
Graph L1M
53.70
72.23
39.81
48.52
25.23
78.80
98.80
60.09
81.90
94.40
04.62
66.93
SyattaKG-88 Sb
Graph LLM
53.35
72.78
87.11
48.04
73.15
78.57
98.74
59.91
81.85
94.70
91,53
67.22
GPT--o
Graph
70.38
79.25
93.98
Table 2: Retrieval performance. The best scores are bolded, and the second-best scores are underlined.
Our D-SynthKG-8h model improves Hits @2 by an average of 28.27 points over the pre-trained Llama-3-8b, 5.31 points over SynthKG-8b. and 3.96 points over the larger Llama-3-706 (Table 2 These gains suggest that training synthetic data produces a qualitative shift in capability rather than a marginal tuning effect. Notably, D-SynthKG-S3 remains highly competitive with the full SynthKG 70b pipeline, despite being much smaller and requiring only single-step inference, indicating that the
8

---

Published as a conference paper at ICLR 2026
multi-step procedure can be effectively internalized through distillation. It also performs comparably to GPT-4o (see Appendix B.2 for details).
In addition, GrMPh+LLM retriever improves Hits@2 by 12.75 points over standard dense retrieval and by 1.67 points over dense retrieval augmented with an LLM re-ranker, demonstrating that structured KGs enable more effective retrieval strategies.
MuSiQue
2wiki
HorporQA
Average
KG Source
Retrieval
EM
F1
EM
F1
EM
F1
EM
F1
None
None
0.100
0.220
0.190
0.340
0.290
0.440
0.193
0.333
Nome
Dense Retriever
0.237
0.376
0.380
0.497
0441
0.641
0.363
0.505
None
Dense: L.M
0.260
0.398
0.414
0.531
0.509
0.678
0.394
0.536
GPT-to
GraphRAG (local)
0.291
0.112
0.132
0.191
0.448
0.569
0.390
0.191
GPT-4o
GraphRAGi (dril))
0.222
0.350
0.497
0.629
0.434
0.384
GPT-4o
HippoRAG
0.224
0.368
0.493
0.627
0.492
0.644
0.403
0.546
Ours
Llama 3
LlamaIndex
0.155
0.259
0.366
0.461
0.405
0.555
0.308
0.425
Llama-3-706
Llamalndex Llamalndex
0.202
0.309
0.117
0.507
0.121
0.563
0.317
0.159
D- D-SymthKG-88 8b
0.212
0.320
0.435
0.528
0.451
0.608
0.362
0.485
Llama-3-8b Sbb
Chain of Triple
0.131
0.244
0.305
0.381
0.278
0.469
0.258
0.365
1-70---700h
Chuin-offTTiplet
0.188
0.299
0.42%
0.370
0.303
D-SymhKGGBb
Chain-offTTrile
0.243
0.383
0.410
0.507
0.400
0.579
0.354
0.490
Llama-3-8h
Graph + LLM
0.181
0.299
0.291
0.394
0.373
0.515
0.281
0.402
Lhmma-3-706
Graph -LM
0.297
0.437
0,400
0.501
0.544
0.705
0.413
0,548
D-SymihKG-Bb
Graph -.M
0,459
0.706
0.569
Table 3: Multi-hop QA evaluation (EM and Fl score). Best scores for each framework are underlined.
7.3 MULTI-HOP QA RESULTS (RQ4: TO--TO-END PERFORMANCE)
D-SynthKG-Sb delivers the best overall results across all three frameworks: LlamaIndex, Chain- of-Triplet, and Graph+LLM, surpassing both Llama-3-8h and the larger Llama-3-70b (Table 3 The consistency of these gains across diverse RAG pipelines highlights the robustness and broad applicability of our synthesizedd KGs. The largest improvement appears in the Graph+LLMM setting, where D-SynthKG-8i achieves a +15.2% absolute EM gain over Llama-3-8h and a +2.0% gain over Llama-3-70b, yielding the best overall performance. Notably, it also outperforms GraphRAG and HippoRAG, two strong KG- ba-daaeddlae RAG systems built on KGs generated by GPT-4o, underscoring that our data-centric approach enables a smaller model to compete with substantially larger systems in practical applications.
7.4 ANALYSIS (RQ3: SCALING AND DESIGN VALIDATION)
How effective is multi-step SynthKG in processing documents of increasing length? We com- pare our multi-step SynthKG framework with single-step LLM prompting approach by examining the number of triples generated per 100 words for documents of varying lengths (Figure 4 Ap- pendix B}. For the single-step method, the proportion of extracted relations decreases as document length increases, with triple density dropping by about 60% when moving from 100-word documents to 1,200-word documents. In contrast, SynthKG's s triple density remains nearly constant across all lengths. This validates our chunking strategy and demonstrates why the multi-step approach is essential for creating consistent training data.
What is the optimal retrieval source? Our KGs support multiple retrieval strategies. In Fig- aa) Appendix B we compare retrieval using triples, propositions, and the full graph structure. Proposition retrieval outperforms triples (+0.89 Hits 10) due to richer context, while graph-based retrieval achieves the best performance (+2.50 over propositions). This validates our design choice of including propositions first-class components of the KG, not just intermediate representations.
How can our KG improve RAG beyond retrieval? Our ablation study (TigureBib) in Appendix (
shows that enriching retrieved context with structured signals improves QA performance. Adding propositions (+2 EM) or 2-hop paths (+2.7 EM) to retrieved chunks improves accuracy, with 2- hop paths offering slightly better gains due to their ability to capture complex relationships. This demonstrates that our KGs provide value beyond simple retrieval, enabling structured reasoning.

---

Published as a conference paper at ICLR 2026
8 CONCLUSION
We presented SynthKG, a data synthesis pipeline that generates high-quality document- KG pairs, and Distill-SynthKGG which distills this multi-step process into efficient single-step model. Our experiments demonstrate that an 8B model trained on synthetic data matches or exceeds the perfor- mance of models eight times larger across KG quality, retrieval, and QA tasks. This work shifts the paradigm from scaling models to generating training data, showing that structured extraction capabilities can be transferred through synthetic examples rather than parameter count. We release our 100K document-KG pairs to enable future research in data-centric approaches to knowledge extraction.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide detailed experimental settings and hyperparameters Section and Appendix [F] All prompts used in the SynthKG pipeline are included in Appendix (C) The 100K synthetic document-Ke -G pairs dataset will be made publicly available. The core implementation code, including the KG synthesis pipeline and evaluation metrics, will be released to facilitate the reproduction of our results.
REFERENCES
AI@ Meta. Llama 3 model card. 2024. URL http:///hhtt..o.mmmmmeeeea/aaa// Main/MODEL_CARD md]
Zhen Bi, Jing Chen, Yinuo Jiang, Feiyu Xiong, Wei Guo, Huajun Chen, and Ningyu Zhang. Codekge:
Code language model for generative knowledge graph construction, 2024. URL https://arxiv
org/ brg/abs/2n 230.20024400000
Harrison Chase. Langchain
LangChain, October 2022.
URL https://ggihbb..om//aaaggiiiii
Bohan Chen and Andrea L. Bertozzi. Autokg: Efficient automated knowledge graph generation for language models, 2023. URL http:pss//aaaxi..or//////3111111 14740
Yew Ken Chia, Lidong Bing. Soujanya Poria, and Luo Si. Relationprompt: Leveraging prompts to generate synthetic data for zero-shot relation triplet extraction, 2022. URL httpss:/s//// orrg abs/2203. 091001
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint
2023.
Linyi Ding, Sizhe Zhou, Jinfeng Xiao, and Jiawei Han. Automated construction of theme-specific 19146)
knowledge graphs, 2024.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local global: A graph rag approach to query-focused summarization. arXiv preprint wrXiv:2404.16138 2024.
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobio- logically inspired long-term memory for large language models. arXiv preprint 14831. 2024. URL hnttps://arxinn http://awxvv.g....aa///// abs/ 2405. 14831
Jiuzhou Han, Nigel Collier, Wray Buntine, and Ehsan Shareghi. Pive: Prompting with iterative verification improving graph-based generative capability of 1lms, 2024. URL /https://arxivv org/abs/2305. 12ti//2225511.22222. 12392
Xanh Ho. Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference Computational Linguisties, pp. 6609-66255 Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. URL https: //www www.no-main..n...m
aclweb. org/ www.b.org/antholooooo http://attooolllooooeo http://///.......ooooooaooaaaaaa
10

---

Published as a conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Ame Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. /ttp:htts://aaaivv.oo abs/2310. 06825
Yizhu Jiao, Ming Zhong, Sha Li, Ruining Zhao, Siru Ouyang, Heng Ji, and Jiawei Han. In- struct and extract: Instruction tuning for on-demand information extraction. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 10030-10051, Singapore, December 2023. Associ- ation for Computational Linguistics. doi: 10. 10.18653//1/2023.. http://///.....mmmmamm......mmm //aclanthology.co org wwwg//22....... ennIp-main. 620
URL https
Yu Su Kai Zhang, Bernal Jiménez Gutiérrez. Aligning instruction tasks unlocks large language models as zero-shot relation extractors. In Findings of ACL, 2023.
Hanieh Khorashadizadeh, Nandana Mihindukulasoorrya, Sanju Tiwari, Jinghua Groppe, and Sven Groppe. Exploring in-context learning capabillities of foundation models for generating knowledge graphs from text, 2023. URL http:h/ss//aaaw..og////////////e /
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, Wenbo Su, and Bo Zheng. Graphreader: Building graph- based agent to enhance long-conters abilities of large language models, 2024. URL https 14550
//arxiv.org/abs//2
Chin-Yes Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https:://aclantho..ogg org. org/W04-1013
Jerry Liu. LlamaIndex, 11 2022. URL https: www..tt..................e..teee wttp.//jrrrrryllillaaaaa...
Nandana Mihinduukulssooriya,, Sanju Tiwari, Carlos F Enguix, and Kusum Lata. Text2kgbench: A benchmark for ontology-driven knowledge graph generation from text. In International Semantic Web Conference, pp. 247-265. Springer, 2023.
Anmol Nayak and Hari Prasad Timmapathini. Llm2kb: Constructing knowledge bases using instruction tuned context aware large language models, 2023. URL htttp:://arxis https//////vvv.oo//ab//b org/abs/ 2308. 13207
OpenAl. Hello gpt-4o!
2024. 2024.
Diego Sanmartin. Kg rag: Bridging the gap between knowledge and creativity, 2024. https:
//arxiv.org' www./bs////aaeeeee /ww./////555 2405 12035
Qi Sun, Kun Huang, Xiaocui Yang, Rong Tong, Kun Zhang, and Soujanya Poria. Consistency guided knowledge retrieval and denoising in Iims for zero-shot document-level relation triplet extraction. In Proceedings of the ACM on Web Conference 2024, WWW '24, pp. 4407-4416, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400701719. doi: 10.1 414444558833333455655
WRLLhhttss///oi.oorr org/ /111/3189334444 3645678
Milena Trajanoska, Riste Stojanov, and Dimitar Trajanov. Enhancing knowledge graph construction using large language models, 2023.
04676.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 554, 2022. doi: 16.1/62/ a 00475. URL / htatt::://alaanthoooyyy. org/ 2022. 1.31
11

---

Published as a conference paper at ICLR 2026
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. thoougt prompting elicits reasoning in large language models. Advances in neural information processing systems, 01224424-24833 2022.
Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang. Manasi Deshpande, Xiaofeng Wang. generation with knowledge graphs for customer service question answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024. ACM, July 2024. doi: 10.1 10.1145/3626772. 3661370. URL http://ddk
and Zheng Li.
1145/3666772.6611377
Zhilin Yang, Peng Qi, Saizheng Zhang. Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: dataset for diverse, explainable multi-hop question answer- ing. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, Brussels, Belgium, October-November 2018. Association for Computational Linguisties. doi: 10.18653/v1/DI8-111 1259. URL /tttt/tts///lllanoooloy D1gp:///0ooo.o/111--///
Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. Exploring large language models for knowledge graph completion, 2024. httlhtttss//axxi.orgrgg// 23300 13916
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang. Jie Zhou, and Maosong Sun. DocRED: A large-scale document-level relation extraction dataset. In Anna Korhonen, David Traum, and Lluís Märquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 764-777, Florence, Italy. July 2019, Association for Computational Linguistics. doi: 10. 10.18653/11/11910000 URL https www.aanhhoooooog/g//i...
Yunzhi Yao, Shengyu Mao, Ningyu Zhang, Xiang Chen, Shumin Deng, Xi Chen, and Huajun Chen. Schema-aware reference prompt improves data-efficient knowledge graph construction. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, pp. 91 911-921, New York. NY, USA, 2023. Association doi: 10.11457539618835917765 URL
for Computing Machinery. ISBN 9781450394086.
https://doi. 000/ 1145/353961111 3591763
Yuqi Zhu, Xiaohan Wang. Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Hua- jun Chen, and Ningyu Zhang. Lims for knowledge graph construction and reasoning: Recent capabilities and future opportunities, 2024. Utthhttss://arrt wwwpaawwwww..r////bb/b5..../ 13168 13168
A APPENDIX
B
ADDITIONAL ANALYSES
70-17
6.7
30
0.1
NLV
LUN
100
25g pello
Pags
2 Prgs
hipkb
(a) Retrieval methods comparison
(b) Input context combinations
Figure 3: Ablation studies: (a) Performance comparison of different KG-based retrieval methods on
multi-hop QA. (b) Results on different combinations of input context for multi-hop QA.
12

---

Published as a conference paper at ICLR 2026
SynthKG Dacument-Levet Extraction
Trplets 100 Words
200
800
1000
Number of Words Decument
Figure 4: SynthKG maintains the triplet density consistently across documents of different lengths.
Retrieval Evaluation
QA Evaluation
KG Source
@its@2
Hits@
Hits@1 10
MRR
MAP
EM
F1
D-SythhKG77byy GPT
Mistral
0.680
0.776
0.809
0,932
0.575
0.417
0.556
D-SymthKG-7hyy Mistral
Caant Llama-3
0.685
0.780
0.811
0.931
0.578
0.433
0.565
D-SynthKG-&b
0.695
0.792
0.820
0.936
0.584
0.433
0.569
Table 4: Efficiency and Generalizability results for Dystill-SynthKG. The results show aver- age performance across MuSiQue, 2wiki, and HotpotQA datasets. D-SynthKG-7bMiah and D-SynthKK-7byyie Lamma- are Mistrrll77--Instruet--.. models fine-tuned using QLoRA on 1000 document-Ki KG pairs annotated by GPT-4o and 1-mma-.1-700bInsttei (respectively).
B. EFFICIENCY AND GENERALIZABILITY OF DISTILL-SYNTHKG
In Table study three key important questions for developing Distill-SnntKGG: 1. the efficiency of training Distill-SynthKG, the effectiveness of various powerful LLMs for synthesizing training data, and 3. the generalizability of fine-tuning other smaller LLMs synthesized data. To address these questions, we employ QLoRA (Dettmers et al. (0223) fine-tuning on approximately 1,000 synthetic Document-KG pairs, generated using GPTT-4 and 1armma--.--70-Issttttte
We provide fine-tuning configure configurations Section E.1 Additionally, to answer the third question, we fine-tune another well-known base LLM,,
(Jiang et al. 2023).
We observe that both QLoRA fine-tuned models perform slightly below the fully fine-tuned model on retrieval and multi-hop QA tasks. However, the performance gap is minimal, demonstrating that QLoRA fine-tuning, even on a small dataset, remains competitive while requiring significantly fewer compute resources. The model fine-tuned on GPT-4o synthesized KGs shows slightly lower performance, which we attribute to more abstractive and atomic propositions in the synthesized data.
Dataset
Model
Hits@2
Hits@ 10
MAP
MRR
GPT-4o (OpenAl)
74.35
79.25
60.52
99.02
2Wiki
D-SynthKG-Sb
73.15
78.57
59.91
98.74
GPT-4o (OpenAl)
82.90
94.95
67.15
93.98
HotpotQA
D-SynthKG-B8 8bb
81.85
94.70
67.22
94.53
GPT-4o (OpenAT) D-SynthKG-8h
53.90
70.38
48.66
90.46
MuSiQue
53.35
72.78
48.04
87.41
Table 5: Retrieval performance comparison between D-SynthKG-8b and GPT-4o across three multihop QA datasets.
13

---

Published as a conference paper at ICLR 2026
B.2 COMPARISON OF RETRIEVAL PERFORMANCE WITH PROPRIETARY FOUNDATION MODELS
We include additional comparisons between distilled model D-SynthKG-S8 and state-of-the-a proprietary foundation model, GPT-4o from OpenAI. We evaluate retrieval performance on three multihop QA benchmarks: 2Wiki, HotpotQA, and MuSiQue. We present the results in Table 5 The results show that D-SynthKG-8b achieves retrieval performance that is highly competitive with GPT-4o across all datasets. Notably, D-SynthKG-88 yields a slightly higher average Hits @ 10 (82.02 vs. 81.52), despite being significantly more cost-efficient. GPT-4o incurs $2.50 per million input tokens and $10 per million output tokens, while D-SynthKG-8h operates at only $0.20 per million tokens (input or output), representing roughly 3% of the inference cost. These findings highlight the practical advantages of our approach cost-sensitive deployment scenarios.
B.3 ENTITY DISTRIBUTION ANALYSIS
When analyzing entiyy references across document chunks, find that the overall average chunk distance per entity is 0.9. This metric represents the average number of chunks between an entity mention and its most recent previous mention. Further analysis reveals that 80.03% of entities have their most recent mention within single paragraph, indicating that the majority of entity references are relatively localized within the document.
These findings support our design decision to only reference the immediately preceding chunk during decontextualization, as this approach effectively balances computational efficiency with adequate contextual information.
C LLM PROMPTS FOR SYNTHKG
We use prompts Figure Figure and Figure decontextualization, entity extraction and relation extraction respectively within SynthKG. We also provide an example decontextualized chunk in Figure g
D KG COVERAGE EVALUATION
D.1 PROMPTS AND EXAMPLES
Figure 9 shows the prompt that we used to generate the triplets, and Figure the the prompts that we used to instruct the model to generate the decomposed questions. Table 6 shows question from HotpotQA dataset, and the generated decomposed questions and the triplet for the question answer pair.
D.2 HUMAN EVALUATION OF EXTRACTED TRIPLETS
To evaluate the quality of the GPT-4-generatee KG coverage evaluation data, three authors of this work reviewed and validated both the decomposed questions and the proxy triplets. A random sample of 50 instances from each dataset was selected for human assessment, where evaluators rated the validity of the generated outputs. For decomposed questions from the HotpotQA dataset, the validity rate was found to be 85%, while the generated triplets for both the Musique and HotpotQA datasets showed validity rate of 86%. Annotators also provided reasons for any invalid ratings. Common issues with decomposed questions included the presence previously unseen entities in the first qub--quetion poorly structured second sub-questi sub-question. For triplets. the most frequent problem was the omission of minor details, such as dates, which did not necessarily make the triplet incorrect but affected its completeness. Only 4% of the cases involved an incorrect relation being extracted.
14

---

Published as a conference paper at ICLR 2026
Previous paragraph from Document:
towa with name leaves visioos tongue didd, Men's s Journal magazine deseribes Guahia as an ouppett of adventure lifestlem its latest edition. which goes on sale today, The magazine describes Gualiala (pronounced wn- un-LA-la by locals) as one the helow-dte -edarrr plazes to make mooe on before the word gets out. There were five such citiss. The others Homer, Alaskn, Newport Vt, Logus. Utal: and Walla Walla, Wash. Kolling Stune maguzine's paid circulition about 620,000 Gualala joined thece other Californis communities on magazise's lis:: Santa Cruz, Mamuott
Gualala, isolated Mendocino
list the places live the Unised States,
Wenner publishes Men's Journal which has
and Bishop. were looking for places that combined afforcability, proximity to ourdoor adventure and a generally undiscovered quality of Sadd Erica Kestenbaum. spokesweman for Men's Journal. Instruction:
Rewnite below paragraph resalving eatity cocefevences with proceding paragrago from document. Resolve all inter sentence pronoun references.
Make sure that all prosouns in seréence refers to some named cstity with the same sentence. ambiguity from sentence. Remember make sentence clear unaniiquouu.
Explicitly mention emiity
wherever necessary For entiyy use only the one mosi informative name.
Do not genemae asything exeept the cowittnn paragraph. Paragraph:
She said isolation played facter. In Noethern Califoenis, potticulall difficulr find beantifull crastal setting
emtieel Ahe Gaialala being to logging town tourist center. coastal
magazine listing the aneetion resadents chackled about any notion of affocdability, given an inflx of newcomers who" 'e driven the median housing price to SSM00000 compared the median family income of 778, Ottices rocalled an when the Guslala repion was better knows for the ligggin of oncient redwoods, marijuana growing and boisterous drinking the historic Guslals Hotel, Still there
residents Monday were largely unsware
the magazine's designation. Yvent
25-ycar resideut who woeks
the Guslista Spoet, Thckle shop, said proud lown made it on Output:
Erica Kestenhaum said isolation played a factor. in Northen Califommi, it's particularly difficult find besunital coastal setting that isa't entirely Frica Kesienhuum said. Gualala miiidnts Monday largely residents of Guulala the Gualzilas silia's median housing price to $550000 compaeed to the median family income of $47.778. Other Gualala residents recalled an ema whea the Gusiata regicn was botter known for the logging of sncient rafwoods. marijaans growing and boisterons hear drinking the bistoric Gislala Hotel. Still there was the Mon's Journal magazine's designation. Yvetie White 25-year Gualals resident who woeks the Gualala Sport; Tackle shop, said she's yroud her town made on the list. Previous poragraph from Document: [peevion paragnph]
the Men's Journal magazine listnn sitemtion could bring the old about any
logging town tirsed tourist conter. few
affordabiiity, given influx newcomer who' ve driven
Instruction:
Rewrite
paragraph resolving entiyy coreferences with proceding puragaah from Resolive intersentence proseun references.
Make sure that all prosouns in sentence refers to some named astity with the same sentence.
Explicitly mention emitty names wherever necessary remove ambignity from senience. Remember to make each sentence clear and unambiguous. For enityy only the
informative Do not genenite asything except the cowrition paragraph.
Paragraph:
[pongraph] ]
Output:
Figure 5: The prompt for chunk decontextuulizzatinn.
Extract all samed entinies from the document. Also generate the type for each eniity. Instructions
Generate only the informative for named eniity Example: John Parker, John Parker coreferemial, only generate John Parker. Use your understanding best domain of paragraph decide appropriate enitty types. Respond using jouu fonuat provided below.
"nl":(""aae" "naas"" "ent"iyynim"" "type": " "ent ty.t "n2":O,
Below example reference.
Parzgraph: Tucked Lilly 's year earnings report, the company revealed positive Synengy-NAe NASII-ASSI-- lts pluse 2 study tinzeputide adults in nonalecholic stratohepuznis (NASH), also known as metabolic dysfunction seunchepaiiss (MASH). Ouiput:
"ni": ("nase": "ET: LiTly,, "type": "Organization").
"n2": ("nase": "Synergy-uuSH" "type": 'Clinical Trial"), ("naxe"; "tirzepatide" "type": "Orug"). ("haxe"; "noraiccholli ie steatohepatitis" "type": "Disease"), "0ypes=""issesse") "),
"né":
"nS":
"n6": ("haset: "ustabolic pasfunttoorsssoeiat steatolepatitis"", "type": "Disease"),
"n7": ("naxe": "year-end earnings report", "type": "Document")
Figure 6: The prompt for graph node extraction
E EXPERIMENTAL SETTINGS
E. QLORA FINE-TUNING SETUP
In our experiments detailed in Section B. we employ the QLoRA fine-tuning. The training configuration used is as follows: we train models for 3 epochs, with an alpha value of 256 and a rank of 128. The learning rate, warmup steps and batch size are set to 0.00003, 50 and respectively.
15

---

Published as a conference paper at ICLR 2026
faets from
each fact, generate semantic triplets.
Instructions
Consistently use the must informative name for each numed entity in all facts and tripltt..
Avoud pronouns ambiguous references facts and tiplets. Instcod, directly include all relevant named entities facts. head predicase, and tuil eutity. one (preferably both) entiyy semantic triplet present the given entiiies Respond using joon fonmat provided below:
Eisure that semantic tripltt
"M
Tact"; "A factual statement deseribing Liportant information (oreferably about some entities) from the paragraph", "triplats: CC"entity 17, "prediicate", "entity 2"1, l'entity "predicate", "entity
"r2":()
Below is an exumppe reference. Paragnph: Locked a heated butle with Novo Nordick's semaglutide franchise, Bi Lilly tiroppuiiee is beginning to come into its own both with regards sales and amid attempts show the dual (TPACI P-1 agonist can strike out heyond disbetes and chesity. As Mounjaro, tirzepatide won its finst FDA nod in Novenieer,
Type diabetes back May Zephound 2023's fourth quarter. Mounjaro generseed a whopping $2.2 billion sales, a nearly eight-fold increase over the $379 million pulled down during the samo statch in 2022. Yea-kodase,, the dng bmught home awoud $5 billion in recomss 1illy said an eirnings release Thesday. Zophound, for is penerated 5175 million during is fiss quaste the marker Overall, Lilly reclad croud $2.4 billion fouttiqquster sales, growing 28% over the 57.3 billion Fatities: Fli Lilly, Novo Noedisk, Troepatide, Somaghtrde, Gl 2-1, GIP, Mounjaro, Zephound Ouiput:
An obesity approva followed
fonsulation of tirzepatide adopting
moniler
quatteria 2022.
"#1": {
"fatt"; "Eli Lilly's tirzepatide corpeting with Novo Nordisk's seroglutide franchise.
"triplets": (C'ElL Lilly", "corpeting with", Nordisk"), ("Tirzepatide" competing with", "Senaglutide")
"P2t: {
"fact"; "Els Lilly "triplets": [["Eli Lilly", "is trying show', "Tarzepatide") ["Tarzepatide" a", "dual GIP/ELP-1 agonist"), ["Tirzepatide", "can treat beyone". "Diabetes"), 1, ["Tärzepatid", "con trest beyond", "Obesity
tirzesstide, cual GIP/GLP-1 agonist, strike
diabetes and obesity.".
"ract:; "Tirzegatide, under brand Nounsaro, recesved its first approval for Type diabetes May 2022.", 'triplets:: [["rrreepttie"' "branded as", "Hounjaro"), ('Hounjaro", "won", "FDA approval "], [MTDA azproval", 'for". "Type diabetes"), ', ["TDA approval", " 'was in", "May 2622"]]
"fact": "Firzepatide, under the brand nore Zepbound, received an obesity approval in November 2022." "triplets": [L"Tarzeaatde", "was branded as", "Zapbound"), ("Zestound", "recesved", "Obesity approval"), ("Obesity approval", "was in". "hovenber 2922"]]
"1gt: [
"fctt": "Mounjaro generated $2.2 billion in sales in the fourth quarter of 2023, the $279 millson "triplets": [[*Mounjaro", "2023's fourth quarter sales", "52.2 billion sales"). ["Hounjaro", "2022's fourth quarter sales", "5279 million"]]
the pertod in 2022.",
"ig":
"fact": "Nounjaro brought around 55.2 billion revenues year
2023, Lilly said an eamnings release Tuesday", 'triplats": [L'Mounjaro", "2023 sales year-to-date", "$5.2 billion revenues")
"17": {
'fact": 'Zephound generated $175.8 milliion in its first quarter on narket.", 'triplets": (['Zepbound", "first quarter sales", "5175.8 million")
"ig":
'fact": Lilly's fourth-quarter sales were around billion, a 28% increase ever the $7.3 hillion during the same pertod in 2022.", "triplets": [L'ELL Lilly", "2023 fourth-quarter sales", "39.4 billion,") "]. ['F11 191ly", "2072 fourth-qurrter sales", "37.3 billion,
Figure The prompt for relation extraction
F TASK-SPECIFIC EVALUATION SETTINGS
F.1 KG COVERAGE TASK
We use the 'all-MiniLM-L-- checkpoint to embed the triplets for semantic matching. For the coverage, we use threshold 0.88 as we manually check that this threshold representing a desirable semantic match.
16

---

Published as a conference paper at ICLR 2026
The Supreme Court H96t on July 18 directed the Union of India to come up with proper guidelines within the time frame of two weeks to blacklist those builders, contractors and architects who are found to have constructed buildings against the sanctioned plan. \nihe se Supreme Court said that the sealing and demolition of unauthorised constructions in the national capital will continue. continue.\nThe AnThe sourt Supzeme Court has passed the verdict after the Centre said that it had not asked the authorities in Delhi to stop the sealing drive against illegal constructions. \nd se Supreme Court division banch, headed by Justice Madan Rhimrao Lokur told the Centre that hereafter, owners of illegal encroached constructions would only be given 48 hours show cause notice to why the building should not be sealed demolithed, \nThe opex-court Supreme Court informed today that Municipal Councillor, Mukesh Suryan, Chairman of Najafgarh wards committee, had allegedly prevented officers from carrying out sealing drives the area, to which the top sought Mukesh Suryan's personal presence. AnThe bench also asked the Najafgazh authority file affidavit Vishvendra Singh transfersed the bebest of Municipal Councillor Mukesh Suryan.
allegation that Deputy Commissioner
Figure 8: An example of decontextuullization chunk.
You given questoon answer puir pleas genenate relation triplet to repeesent the relationchip.
Geeeeate oupput the fornst deserihed helow. head rolation tail Note: Must include relation, head eatiry and tail carity. Fasure head eniity subject rohtion, and tiil entiyy is direct object of relation. You must use the given unower as the head or tail enity. Specific entlyy is more preferable than generic encity Do NOT genentt pronoums or references in hend and tail entiies. - Esample I:
Question: To whom Mess's goal in first leg the Copa del Rey compared? Answer. Diego Maradona compured Diego Maradonu
Quiput: Messi's goal
Example 2:
Question: The father of Chiang Hiiio wen is whom!? Answer: Chisng Ching kuo Ouiput: Chiang Ching kuo the father of lI Chiang Hsiau-wen [question]
Answer (answer)
Ouiput:
Figure 9: The prompt for generating triplct given a question and the answer.
You multihop guestion, facts that used reach correct and the correct Your goal decompose the question
cobo-qustion, and the correpunding unswer each question.
Example
Question: What relationship does Fred Gehrke have to the 23rd overall pick in the 2010 Major Leugue Baseball Draft?
Facts: He the great-grandee graniatradeether of Miumi Marlin Christiun Yelich Yelich was drafled out of high school by the Marlins in the Ist round (23rd overall) the
2010 Major League Baseball Draft.
Answer: great grandfatber
Decompuse quectioo snower pairs:
Who 23rd overall
the 2010 Major League Baschall Draff? Chrtstian Yelich Whst relationship does Fred Gehrke have to Christian Yelich? grandfather
Question (question)
Facts: [facts]
Answer: (answer)
Decompose question unswer pairs:
Figure 10: The prompt for decomposing question into sub-questions and answers.
F.2 RETRIEVAL TASK
We use rext-embbedding---smdde for both dense retrieval and embedding propositions in KG-based retrievers. For both the graph and graph + LLM retrievers, we first construct the sub-graph by selecting the 200 propositions (M = 200) most similar to the question based on embedding similarity. Within the sub-graph, we traverse the KG, starting from the question entity, and select propositions within a 5-hop neighborhood (N = 5). For re-ranking the propositions in the L--M-b retriever and also for re-ranking chunks in Dense+I.LM, retriever, we use the GPT-4o model. The Dense+ILLM retriever uses LlamaIndex's 's implementation of the LLMRerank post-processor.
We evaluate retrieval performance at the passage level, following the setup used in HippoRAG. For each query, we evaluate whether the retrieved passages contain all ground-truth information required to answer the question. Retrieval metrics include Hits and Mean Reciprocal Rank (MRR), computed based on the rank positions of the passages associated with ground-truth triplets. Specifically, a passage is considered relevant it contains all the facts (triplets or propositions) needed for correct multi-hop reasoning. The evaluation code is directly adopted from Ai.
17

---

Published as a conference paper at ICLR 2026
To ensure comparability, use the same benchmark datasets and experimental setup HippoRAG, including 1,000 questions and a fixed set of candidate passages. The number of ground passages per question is consistent with the original annotations. Our experiments are conducted three multi-hop QA datasets: MuSiQue (11,656 passages), 2WikiMultHHoPQA 16,119 passages), and HotpotQA (9,221 passages).
F.3 MULTI-HOP QUESTION ANSWERING TASK
F.3.1 LLAMAINDEX CONFIGURATION
Table presents the complete configuration of our Llamalndex query engine setup.
F.3.2 CHAINNOP-TRIPLET
We design triplet retrieval method that first breaks down the question into sub-queries in a triplet format. These triplet queries are then used to retrieve the most semantically matching triplet facts from the extracted KG. Specifically, includes three steps to generate the final answer.
Generate the Chain of Triplet Queries: given a question, we convert into a series of triplet queries. Specifically, since our downstream task involves multi-hop QA, instead of generating a single triplet, we prompt the model to generate chain of triplets. The generated triplets may contain placeholders that represent unknown entities. The prompt is shown in Figure
Step 1:
There is knowlodge graph (enities and relations). Now, are given question, tock is to decompase this question into a chain tripless used for faet from grah.. ciplet should
format:
Note: - Easure that head entity is sthject of relation, and mill aatity is direct object of relarion. - Do NOT gesenate prenouns references in head and tail cuttiee. Do NOT generate eutities that are not oppeared the question. If an rriplet inciaadss an intermediate answer the final anewer, you can use triplets order should the onder retrieving facts knowledge graph.
followed by digit for reference.
Example
Question: Who is oldes, Hampton Del Ruth or Ted Kotcheff?
Decompose Triplet::
Hampton Del
Kotcheff bom
Example 2:
Question: In what town is Suffolk county humlet that was served by the Suffolk Traction Compony?
Decompase Triplet:
is locosed Now please generate dasompose Triplets for the question: (question)
Suffolk Traction Compuny
Decompase Triplets:
Figure II: The prompt for generating chain of triplet query given a question, which are then used for triplet retrieval.
Step 2: Triplet Retrieval: once the chain of triplet queries is generated, we retrieve the top 20 triplets for each query. During retrieval, if any of the triplets contain placeholders for uncertain entities, we attempt to resolve those entities by filling them with entities or relations from the previously retrieved triplets. For subsequent triplet queries in the chain, placeholders are updated with these resolved entities, thus refining the triplet queries progressively.
Step 3: Question Answering: with the question, the chain of triplet queries, and the retrieved triplets, we prompt the model to generate the answer. If the graph extraction method also retrieves associated propositions alongside the triplets, these propositions are provided to the model further enhance the answer generation. The prompt is shown in Figure 12
You are given satural language question, tripkes chain for this question, of retrieved uipletes, and set of facts, please answer the question.
Questio:: [question]
Questtion Triplets
Iguestion triplees)
Retrieved Triplets: fretrieedd tiplets]
Renieved Facts: fretrived facts)
Short Answer
words:
Figure 12: The prompt for generating the final answer given the original question, chain of triplet
query, retrieved triplets and the facts.
18

---

for 3 epochs to Published as a conference paper at ICLR 2026
Graph LLM We the same graph LLM retriever
as in section F2
G
DATA RELEASE AND TRAINING/ INFERENCE COST CONSIDERATIONS
We will make our annotated 100K data samples publicly available to support future research. With the rapid udvancements in LLMs, researchers may choose to resynthesize data to better align with their specific applications. In such cases, we recommend using our cost-efficient approach, detailed Section B.I which provides practical balance between performance and computational cost.
Below, we present the detailed training and inference costs, highlighting the efficiency of our SynthKG and Gysthll-SyttKKG methods.
Cost of Data Synthesis: With the Llama-3.. 1.mmma-3.1770 -0--ruc--nttruct model, running the entire SynthKG
pipeline on a single document requires processing an average of 11,849 input tokens. This results in a total of 2,675 average output tokens, distributed across intermediate steps such decontextualization and the final entities, relations, and proposition generation. At a cost of $0.90 per million tokens the total annotation cost per document is $0.0131. This translates to $13.08 for synthesizing training data for the D-SynthKG-7bbmmi
Llama-3 model and $392.28 for the D-SynthKG-8h model. Mistral
Cost of Model Training: After consolidating the data synthesized by SynthKG, each document contains an average of 1,723 input tokens (including prompts) and 1 1,248 output tokens, totaling 2,971 tokens per document. For a dataset of 30,000 documents, the total training token count is 89.13 million tokens. Fine-tuning (lama-3.1-8b-Instie D-SynthKG-80 would cost $36.65. Additionally, fine-tuning obtain the D-SynthKG-70 Liama-3 model would cost $3.67.
for one epoch on this dataset to obtain
Combining data synthesis and fine-tuning costs, training D-SynthKG-8h would cost $428.93, while training D-SynthKGG-77 Mistall
Llama-3 would cost $16.75.
Inference Cost: As mentioned in the cost of data synthesis, processing a single document using the SynthKG pipeline requires an average of 11,849 input tokens and 2,675 output tokens, totaling 14,524 tokens per document. At cost of $0.90 per million tokens, this amounts to $0.031 per document. In contrast, with D-SynthKG-8k and D-SynthKG-7b Laama-3 each document requires 1,723 input tokens and 1, 1,248 output tokens, totaling 2,971 tokens, with a cost of $0.00267 per document. This is only 8.6% of the cost of SynthKG, demonstrating the significant efficiency gains from fine-tuning distilled model.
Question
Decomposed Question and Answer
Triplet (head relation I| tail)
The birthplace of George McCall Theal a port city of what bay?
Where was George McCall Theal horn? Saint John, New Brunswick
George McCall Theal Il was born in II Saint John, New Brunswick
Saint John is port city of what bay? Bay of Fundy
Saint John II is a port city of Bay of Fundy
Table Example from HotpotQA dataset: the generated decomposed question answer pair and the triplet.
H LICENSE INFORMATION
We respect the license and intended use of all models and datasets employed in this study. Detailed license information is provided below.
hhmps///www.nngethersi//nnnnggg
19

---

Published as a conference paper at ICLR 2026
Parameter
Value
You are an expert Q&A system that is trusted around the world. Always answer the query using the provided context information, and not prior knowledge.
Some rules to follow:
1. Never directly reference the given context in your answer.
QA Prompt
2. Avoid statements like 'Based on the context, ...' or 'The context information
...' anything along those lines.
Provide only the essential information. Answer as briefly possible, using
keywords, phrases, or dates. Avoid sentences or unnecessary details.
include text
True
response_mone m_ddc retriever_model siin_chunks_ppp
tree summarize hybrid
10
similarity __tp___p__________
2
graph_ strr_sttre___eeeee store_query eeeer_uerreeeee depth
2
Table 7: Llamalndex query engine parameter settings.
Models. The Llama-3 models utilized our study are licensed under the Meta Llama 3 Community License Agreement The Llama-3. models utilized in our study are licensed under the Llama 3.
Community License Agreement The Mistral-7b-v0.3 model is licensed under the Apache 2.0 license
Datasets. The BAAI/IndustryCopgg dataset used for extracting our synthetic training data is available under the Apache 2.0 license, The 2WikiMultihopQA dataset used in evaluations is available under the Apache 2.0 license The Musique dataset used in our evaluations is available under the Creative Commons Attribution 4.0 International license The HotpotQA dataset used in our evaluations is available under the Apache 2.0 license We will make our synthetic dataset publicly available under the MIT license, subject terms and conditions of the Llama 3.1 Community License) Agreement related to the use of Llama-3.1 outputs.
1 DISCUSSION ON THE SEGMENTATION STRATEGY AND DOCUMENT STRUCTURE PRESERVATION
While our segmentation and de-ssmmaniiciztion approach may appear simple, our design choice is guided by both empirical findings and practical considerations. First, our analysis (see Figure shows that model performance consistently degrades as document length increases. Preserving structural associations in segmentation might result in longer input spans, which, based our experiments, would still harm performance. Second, while we agree that capturing structural dependencies across multiple pages is meaningful goal, it remains open research challenge particularly in open- ontology settings. Even state-of-the-aa models like GPT-4o lack robust, generalizable pipelines for reliably preserving cross-page structure in way that could be distilled into smaller model. Given these limitations, we prioritize practicality and scalability by adopting fixed-size 1K token) chunking approach. This method aligns with the constraints of retrieval-augmented generation (RAG) and enables effective, document-level KG construction at scale. We believe our approach provides a strong balance between empirical performance and real-world applicability under current technological constraints.
J DISCUSSION ON MANUAL HYPERPARAMETER SELECTION
We manually tune two key hyperparameters in our framework: the semantic match score threshold for triplet coverage evaluation and the ROUGE score threshold for decontextualization. For the semantic match score, we intentionally select higher threshold to ensure accuracy when determining whether two triplets are semantically equivalent. It is important to note that this threshold is only used for evaluation purposes and does not affect model training, inference, RAG evaluation, thus having impact on the model's 's learning or predictions. While the threshold manually adjusted, downstream task performance reflects the reliability of our model.
20

---

Published as a conference paper at ICLR 2026
For the ROUGE threshold used in decontextualization, we conduct careful manual analysis to ensure its effectiveness. For both Llama-3.1-27 and GPT-4o, we examine a small subset of chunks with low ROUGE scores and find that most rewritings are accurate. We identify 593 edits in this subset, with only six containing incorrect modifications and four showing some loss of information. These results suggest that the decontextualization process generally yields high-quality, self-contained text.
The low ROUGE scores typically result from document footers or metadata, which are removed during the LLM's decontextualization process, and not from factual errors or information loss. Our analysis shows that upproximately 72% of the chunks achieve ROUGE score of 90 or higher, reflecting strong alignment with the original content. Chunks with scores below 0.70 make up less than 3%, and these are filtered out during the process to avoid any omission extreme paraphrasing.
K LLM USAGE STATEMENT
We used ChatGPT and Claude as writing assistants to improve the clarity and grammar of our manuscript. All scientific content, experimental design, and technical contributions are the original work of the authors.
21


Reviews:
[{"review_id": "h1Uj27h5G8", "reviewer": "Reviewer_Q3Mt", "summary": "The paper introduces SynthKG, a multi-step data synthesis pipeline that creates high-quality document–KG pairs via document chunking, decontextualization, and LLM-based structured extraction of entities, relations, propositions.   The authors repurpose multi-hop QA datasets to build KG evaluation datasets and propose coverage metrics (semantic similarity and keyword-based) for KG assessment. Based on KGs produced by Distill-SynthKG, they design a graph-based retrieval framework that progressively retrieves propositions, related triplets, and text chunks leveraging the graph structure for RAG. Experiments show Distill-SynthKG surpasses baselines.", "questions": "Q1: How are edge weights defined and optimized during distillation? Are they learned with retrieval feedback or heuristic scores? \n\nQ2: Does the system include cross-document coreference and alias normalization? How does this impact graph compactness and recall?\n\nQ3: How are temporal or conditional relations handled (time-scoped facts, event causality) to avoid misleading retrieval when collapsing during distillation?\n\nQ4: What are the observed trade-off curves across corpora sizes and domains? How stable are results under parameter changes?\n\nQ5: How does the distilled KG integrate with text retrieval and rerankers? Are there benchmarks showing gains over strong vector-only baselines under equal resources?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1: Pipeline-level clarity makes the approach reproducible with standard components.\n\nS2: Empirical emphasis on compositional queries aligns with KG advantages over pure vector search.", "weaknesses": "W1: Distillation objectives and training signals need precise specification; optimization of edge weights/pruning thresholds should be detailed and justified.\n\nW2: The mechanisms for error correction and robustness are underexplored.\n\nW3: Limited failure-mode analysis (rare entities, emergent relations, long-tail concepts) and cross-domain robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IiugB5Itiu", "reviewer": "Reviewer_hKxC", "summary": "This paper tackles the scalability bottleneck in document-level, ontology-free knowledge graph (KG) construction for retrieval-augmented generation (RAG) systems. Current approaches either rely on costly large language models (LLMs) or use smaller models that yield incomplete and inconsistent KGs—limitations attributed to a lack of high-quality training data rather than intrinsic model capabilities. To address this, the authors propose SynthKG, a multi-step pipeline for data synthesis that creates high-quality document-KG pairs through chunking, decontextualization, and structured extraction with LLMs. They introduce Distill-SynthKG, a smaller LLM fine-tuned on this synthetic data, capable of end-to-end document-level KG generation in a single step. The authors also present a framework for evaluating KGs, repurposing multi-hop QA datasets as KG evaluation benchmarks, and introduce semantic matching metrics designed for ontology-free KGs. Furthermore, they propose a novel graph-based retrieval methodology utilizing propositions as rich semantic retrieval units within the KG.", "questions": "Refer to the Weaknesses section above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The writing and figures in the paper are clear.\n- The discussion around knowledge graph (KG) construction in the context of GraphRAG is meaningful, especially considering the high API cost of current LLM-based construction methods.\n- The proposal to fine-tune a smaller LLM for accurate and efficient KG construction is novel.", "weaknesses": "- The main concern is the potential accumulation of errors throughout the pipeline. In the proposed method, an LLM is first prompted to extract entities, relations, and relevant propositions to construct the knowledge graph. This process is used to distill and fine-tune a smaller model. However, the data used for distillation is not from ground truth, but rather generated by the initial LLM, which may contain errors. As the smaller model is then trained on this potentially noisy data, it may inherit these biases and inaccuracies.\n- The generalization ability of the trained KG construction model in real-world scenarios is uncertain. Since the training relies solely on the IndustryCorpus dataset, there is no guarantee that the model will generalize well to new domains. As KG construction is a key part of the GraphRAG pipeline, it is important to discuss the model’s generalizability to out-of-domain scenarios.\n- The experiments do not include a comparison with more representative baselines for KG construction, such as HIPPORAG2 [1].\n\n[1] From RAG to Memory: Non-Parametric Continual Learning for Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZJ5NPkVbiU", "reviewer": "Reviewer_vie4", "summary": "This paper introduces SynthKG, a data synthesis pipeline that creates high-quality document-KG pairs to address the lack of training data for document-level KG construction. They then fine-tune a smaller LLM on this synthesized data, resulting in Distill-SynthKG, which excels at single-step KG generation. Distill-SynthKG significantly outperforms existing models in KG quality and downstream tasks like retrieval and question-answering. The authors also propose new KG evaluation methods and a novel graph-based retrieval framework for RAG. While this paper introduces SynthKG for data synthesis and Distill-SynthKG for efficient KG generation with promising results, its core innovation appears limited to leveraging LLMs for decontextualization. Furthermore, the experimental validation is insufficient, lacking crucial details on the Distill-SynthKG fine-tuning process, an ablation study on decontextualization's impact, and comprehensive comparisons with a wider range of LLMs and advanced methods, thus undermining the robustness of its claims.", "questions": "1. Could you elaborate on the fine-tuning process for Distill-SynthKG, specifically detailing the inputs provided to the large language model and the expected outputs during this phase?\n\n2. Given that decontextualization is a central tenet of SynthKG, what is the observed impact on performance when this step is removed or altered? A detailed analysis of this would be highly beneficial.\n\n3. The current comparisons are primarily against unprocessed large language models. How does your method fare against more advanced, state-of-the-art approaches in various tasks? For instance, what are the comparative results for QA tasks with different established methods, and what is the performance impact when decontextualization is removed from your proposed method in these contexts?\n\n4. Please clarify the fundamental distinctions between your proposed approach and methods relying on sophisticated prompting techniques. Could comparable performance be achieved solely through advanced prompting strategies without the multi-step SynthKG pipeline?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Please see summary", "weaknesses": "1. The description of how Distill-SynthKG is fine-tuned from a large language model is insufficiently detailed. Crucially, the specific inputs and outputs of this fine-tuning process are not clearly defined.\n\n2. Decontextualization is presented as a core component of SynthKG, yet the paper lacks an ablation study demonstrating its specific impact on performance. The absence of such an analysis makes it difficult to assess its contribution.\n\n3. The experimental section omits comparisons with a broader range of contemporary large language models, limiting the generalizability and competitive standing of the proposed method.\n\n4. The current experimental setup and results do not provide robust or comprehensive evidence to fully substantiate the paper's claims and conclusions.\n\n5. The overall innovation presented in this paper appears limited, primarily leveraging existing large language models for a decontextualization operation rather than introducing fundamentally new paradigms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rA7qx0BDIS", "reviewer": "Reviewer_Tntm", "summary": "The paper proposes a data-centric pipeline, SynthKG, to construct document-level, ontology-free KGs via (i) sentence-boundary chunking, (ii) decontextualization to enforce consistent entity mentions across chunks, and (iii) LLM-based extraction of entities, propositions, and triplets. It then distills the multi-step pipeline into a smaller LLM, Distill-SynthKG (D-SynthKG-8B), capable of single-step KG generation for a whole document. The authors also repurpose multi-hop QA datasets to create proxy triplets and propose semantic coverage metrics for KG evaluation, and design a Proposition–Entity bipartite graph retriever with optional LLM re-ranking for RAG. Experiments on MuSiQue, 2WikiMultiHopQA, and HotpotQA show higher KG coverage and competitive or better retrieval/QA versus larger baselines and GPT-4o KGs, while using a much smaller model.", "questions": "Please see my above w1-w5 for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "S1. One-step distilled model (8B) achieves large-model quality on KG coverage/retrieval, improving efficiency (fewer LLM calls).\n\nS2. Proposition–Entity graph retriever with LLM re-ranking shows consistent gains and strong comparisons to dense retrieval and prior KG-RAG systems.\n\nS3. Data-centric pipeline that converts a fragile prompt-only process into learnable patterns; propositions act as interpretable, retrievable units.", "weaknesses": "W.1 Decontextualization locality assumption (preceding-chunk context suffices) is empirically supported but may break on long-range coreference or cross-section entity normalization (e.g., legal/biomedical corpora). A stress-test on documents with longer entity re-mention distances would strengthen generality claims.\n\nW2. How robust is decontextualization when entities are not re-mentioned locally (e.g., reappear after several pages)? Any fallback using document-level coreference?\n\nW3. For evaluation via proxy triplets, how do you control GPT-4o style drift (atomic vs abstractive propositions) that may advantage a particular retriever? Any normalization?\n\nW4. In Graph+LLM, how sensitive are results to M (initial propositions), N-hop, and K (final chunks)? Please provide stability ranges / guidance.\n\nw5. Please include more recent GraphRAG baselines in recent years.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)