KGGen: Extracting Knowledge Graphs from Plain Text with Language Models
Authors: Belinda Mo
Kyssen Yu
Joshua Kazdan
Proud Mpala
Lisa Yu
Charilaos I. Kanatsoulis
Sanmi Koyejo
Venue: NeurIPS 2025 poster ()
Keywords: knowledge graph
KG
synthetic data
data generation
Primary area: applications
TL;DR: KGGen is a text-to-knowledge-graph generator that uses language models to extract high-quality graphs from plain text.

Abstract: Recent interest in building foundation models for knowledge graphs has highlighted
a fundamental challenge: knowledge graph data is scarce. The best-known knowl-
edge graphs are primarily human-labeled, created by pattern-matching, or extracted
using early NLP techniques. While human-generated knowledge graphs are in
short supply, automatically extracted ones are of questionable quality. We present
KGGen, a novel text-to-knowledge-graph generator that uses language models to
extract high-quality graphs from plain text with a novel entity resolution approach
that clusters related entities, significantly reducing the sparsity problem that plagues
existing extractors. Unlike other KG generators, KGGen clusters and de-duplicates
related entities to reduce sparsity in extracted KGs. Along with KGGen, we release
Measure of Information in Nodes and Edges (MINE), the first benchmark to test an
extractor’s ability to produce a useful KG from plain text. We benchmark our new
tool against leading existing generators such as Microsoft’s GraphRAG; we achieve
comparable retrieval accuracy on the generated graphs and better information re-
tention. Moreover, our graphs exhibit more concise and generalizable entities and
relations. Our code is open-sourced at https://github.com/stair-lab/kg-gen/.

KGGen: Extracting Knowledge Graphs from Plain Text with Language Models
Belinda Mo Kyssen Yuu?, Joshua Kazdan" Joan Cabezas, Proud Mpala', Lisa Yu², Chris Cundy Charilaos Kanatsoulis', Sanmi Koyejo
1 Stanford University
University of Toronto
FAR Al
Abstract
Recent interest in building foundation models for knowledge graphs has highlighted a fundamental challenge: knowledge graph data is scarce. The best-known knowl- edge graphs are primarily human-labeled, created by pattern-matchin or extracted using early NLP techniques. While human-gent -gnneratd knowledge graphs are in short supply, automatically extracted ones are of questionable quality. We present KGGen, novel sott-to-knoweegge-ee
generator that uses language models extract high-quality graphs from plain text with a novel entity resolution approach that clusters related entities, significantly reducing the sparsity problem that plagues existing extractors. Unlike other KG generators. KGGen clusters and de-duplicates related entities reduce sparsity in extracted KGs. Along with KGGen, release Measure of Information in Nodes and Edges (MINE), the first benchmark to test an extractor's 'ss ability produce useful KG from plain text. We benchmark our new tool against leading existing generators such as Microsoft's GraphRAG; we achieve comparable retrieval accuracy the generated graphs and better information tention. Moreover, our graphs exhibit more concise and generalizable entities and relations. Our code is open-sourced at https//ggtuub.coo//taaiallaa
Introduction
Knowledge graph (KG) applications and graph-based Retrieval-Augmenner Generation (RAG) systems are increasingly bottlenecked by the scarcity and incompleteness of available KGs. KGs consist of a set of subject-predicate-obe
triples, and have become fundamental data structure for information retrieval [Schneider] 1973]. Most real-world KGs, including Wikidata (contributors) 2024]. DBpedia [Lehmann et al. 2015], and YAGO [Suchanek et al. 2007 J, are far from complete, with many missing relations between entities [Shenoy et al.) 202 1]. The lack of domain-specific and
verified graph data poses serious challenge for downstream tasks such as KG embeddings, graph RAG, and synthetic graph training data.
Several synthetic plain-text-to-Ke KG extractors have been proposed to address KG scarcity, most prominently including OpenlE [Angeli al., 2015) and Microsoft's GraphRAG [Larson and Truitt 2024]. Both OpenlE and GraphRAG extract entities and relations directly from text, but they lack effective mechanisms for entity resolution and relation normalization. This leads to graphs with nearly as many unique relation types as edges, resulting in sparse, disconnected knowledge representations that limit their utility for downstream tasks. solve this problem, we propose KGGen, a text- wekknowledge-geee
generator that leverages language models (LMs) and an algorithm for entity and edge resolution to extract high-quality, dense KGs from text. First, KGGen uses an L.M-based extractor to read unstructured text and predict relations; after extracting the triples, it applies a novel, iterative clustering algorithm to refine the raw graph. Inspired by crowd-sourcing strategies for entity resolution [Wang al. 2012 KGGen
triples to capture entities and
"Euual Contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

identifies nodes that refer to the same underlying entities, and consolidates edges that have equivalent meanings.
The nascent field of sein--text-to-kowllede graph extraction currently lacks benchmarks to measure the fidelity of KG generation from text. To close this gap, we provide two new benchmarks: the first captures information retention from short texts; the second, based on WikiQA, measures knowledge retrieval capabilities for graphs generated from multi-million token, web-based knowledge databases. On these benchmarks, KGGen performs comparably to GraphRAG. However, KGGen exhibits far better scaling properties with respect to information compression and graph sparsity as the plain-text database length increases.
To summarize contributions:
1. We introduce KGGen, an open-source package that uses LMs to extract high-quality KGs from plain text. Our package is available as Python library.
2. We develop benchmarks to drive improvements in plain-text-to- -oretttoo-owwweee-eeee -gaph extraction, and measure KGGen's 's performance on these benchmarks.
3. We show that KGGen exhibits improved scaling with respect to the size of the text source relative to past methods.
2
Related Work
Interest in automated methods to produce structured text to store ontologies dates back to at least 2001
when large volumes of plain text began to flood the fledgling internet [Maedche and Staab 2001]. KG
extraction from unstructured text has seen significant advances through rule-based and -.-powered approaches in the last 15 years. Early work (Suchanek et al., [007] used hard-coded rules to develop YAGO, a KG extracted from Wikipedia containing over five million facts, and rules-based extraction
still has appeal for those producing KGs in multi-modal domains today (Norabid and Fauzi 2022
Oramas et al., 2013]. With the development of modern natural language processing, hard-coded rules generally ceded to more advanced approaches based on neural networks. For instance, OpenIE [Angeli et al. 20155) provides two-tiered extraction system: first, self-contained clauses are identified by a classifier; then, [Angeli et al.) 2013) run natural logic inference to extract the most representative entities and relations from the identified clauses. Stanford KBP (Angeli et al. 2013 presents another seminal early approach to using deep networks for entity extraction.
As early as 2015, some hypothesized that extracting KGs would go hand-in-hand with developing better language models (Domeniconi et al.) 2015]. More recently, evidence has emerged that transformer-based architectures can identify complex relationships between entities, leading to wave of transformer-basee KG extraction techniques, which range from fully automatic [Qiao et al.,
2022 Arsenyan al. 2023 Zhang and Soh 2024 to human-assister [Kommineni et al. 2024 Our contribution to the extraction literature is to build KGs conducive to embedding algorithms such as TransE and TransR [Bordes et al. 2013 Lin et al. 2015). We observed that when one extracts KGs from plaintext, the nodes and relations are often so specific that they are unique. This causes the estimation of embeddings to be under-specified. We develop a method for automatic KG extraction from plain text that clusters similar nodes and edges to prevent this sedde--peciiiiationn This leads to KG with better connectivity and more functional nodes and edges.
Evaluating the quality of knowledge graphs important to ensure usefulness and reliability in downstream applications. Farly evaluation methods focused primarily on directly assessing aspects such as completeness and connectivity or using rule- based statistical methods, while recent approaches emphasize usability in downstream applications and incorporation of semantic coherence Xue and
Zou 2023
In the late 2000s, research focused on assessing the correctness and consistency of KGs. The evaluations relied on expert annotations by selecting random facts from the generated KG and then calculating the accuracy of those facts. [Suchanek et al.) 2007] This proved to be laborious and prone to errors. This led to accuracy approximation methods like KGEval (Ojha and Talukdar 2017) and Two-State Weight Clustering Sampling(TWCS) [aao et al. 2018). which employed sampling methods with statistical guarantees as well as use less annotation labor. As the KGs became larger and more diverse, particularly with the rise of automated extraction techniques from web data, this

---

generated more pressure on annotators, leading to methods like Monte-Cate search being used for the interactive annotation of triples [Qi al. 2022]. Furthermore, because accuracy alone did not fully capture the complexity of the knowledge graph, more evaluation metrics like completeness were used characterize the quality of knowledge graphs. [Issa et al. 202T).
In recent years, the evaluation of knowledge graphs (KGs) has increasingly focused on their role in downstream AI applications, such augmenting language models [Schneider al. 2022 and recommendation systems [He et al. 2020]. As a result, semantic coherence and usability have become key criteria for assessing the quality of extracted knowledge graphs.
Two notable approaches to KG evaluation are the LP-Measure and the triple trustworthiness mea- surement (KGTtm) model. LP- LP-Measure assesses the quality of KG through link prediction tasks, eliminating the need for human labor or a gold standard (Zhu et al. 2023] This method evaluates KGs based their consistency and redundancy by removing portion of the graph and testing whether the removed triples can be recovered through link prediction tools. Empirical evidence suggests that LP-Measure can effectively distinguish between "good" and "bad" KGs. The KGTim model, on the other hand, evaluates the coherence of triples within a knowledge graph Jia et al. [2019). Based on these evaluation methods, frameworks like Knowledge Graph Evaluation via Downstream Dosks(KGrEaTV and DiffQ(diifferentill testing) emerged. KGrEaT provides a comprehensive assess- ment of KGs by evaluating their performance on downstream tasks such as classification, clustering. and recommendation (Heist et al. [2023] rather than focusing solely on correctness or completeness. In contrast, DifiQ uses embedding models evaluate the KG's quality and assign a DiffQ Score, resulting in improved KG quality assessment Tan et al. 2024
3 Existing Methods
Before describing KGGen, we explain two popular existing methods for extracting KGs from plain text, which will serve as a basis for comparison throughout the rest of this paper.
3.1 OpenIE
Open Information Extraction (OpenIE) implemented by Stanford CoreNLP based on Angeli et al. 2015). It first generates a 'dependency parse" for each sentence using the Stanford CoreNLP pipeline. A trained classifier then traverses each edge in the dependency parse, deciding whether to create, continue, or stop processing a clause. These decisions split complex sentences into shorter, self-contained clauses. From these clauses, the system produces (subject, relation, object) tuples, each accompanied by a confidence score. Because OpenIE does not require its input text to have a specific structure, OpenIE can handle text in any format.
3.2 GraphRAG
GraphRAG, developed by Microsoft in 2024, integrates graph-based knowledge retrieval with language models (LMs) [Larson and Truitt 2024). As a first step, GraphRAG provides functionality for generating KGs from plain text, which serve its database for retrieval. In this process, GraphRAG creates a graph by prompting LMs to extract node-entities and relationships between these entities. Throughout this extraction, few-shot prompting provides the LM with examples of desireable extractions. GraphRAG aggregates well-connected nodes into communities and generates summary for each community. The final graph consists of the nodes and their relationships along with communities their summaries.
4 KGGen: Knowledge Graphs From Plain Text
Unlike most previous methods LLM-based KG extraction, we rely on a multi-stage approach (1) extract entity and relations from each source text using LLM, (2) aggregate graphs across sources, and (3) iteratively resolve duplicate entities and edges using a hybrid LLM and traditional informational retrieval methods.
We impose strong constraints on the LLM via prompting to prevent it from incorrectly grouping together entities edges that are similar in meaning but not actually the same for example, conflating
3

---

"Type I diabetes" and "Type 2 diabetes," "hypertension" and "stress," "MRI" and "CT scan.". We introduce multiple passes through our extracted edges and edges resolve similar entities and consolidate the number of edge types. Entity and edge resolution prevents the formation of sparse KGs, which may produce meaningless KG embeddings under standard algorithms such as TransF.
Our extraction method involves several steps, which we outline below. The exact prompts for each step can be found in Appendix A and the process is illustrated in Figure [
4.1 Entity and Relation Extraction
The first stage takes unstructured text as input and produces initial knowledge graph extracted triples. We use Google's s Gemini 2.0 Flash as the language model to provide structured output via DSPy signatures. The first step takes in source text and extracts a list of entities. Given the source text and entities list, the second step outputs a list of corresponds to DSPy signature that specifies instructions for the LLM to follow in its docstring. We find this 2-step approach works better to ensure consistency between entities.
relations. Each step
4.2 Aggregation
After extracting triples from each source text, we collect all the unique entities and edges across all source graphs and combine them into single graph. All entities and edges are normalized to in lowercase letters only. The aggregation step reduces redundancy in the KG. Note that the aggregation step does not require an LLM.
4.3 Entity and Edge Resolution
After extraction and aggregation, we typically have a raw graph containing duplicate or synonymous entities and possibly redundant edges. The resolution stage is key innovation in our KG extraction methodology that merges nodes and edges representing the same real-world entity or concept. Our resolution process employs two-stage approach combining ambedding-based clustering with LLM- based de-duplication efficiently handle large knowledge graphs. The approach is applied both entity and edge items separately:
First, all items in the graph are clustered. We get the semantic embeddings of every item using S-BERT and cluster using k-means into clusters of 128 items.
1. For each item in cluster, we retrieve the top-k most semantically similar items, where k=16, using a fused BM25 and semantic embedding approach.
2. Then, the LLM is prompted identify exact duplicates from this set, considering variations in tense, plurality, case, abbreviations, and shorthand forms.
3. For each set of duplicates, the LLM selects canonical representative that best captures the shared meaning, similar to aliases that Wikidata uses. Cluster maps track which entities belong to which alias.
4. The item and its duplicates are removed from the cluster and steps 1-3 repeat until no items remain in the cluster.
This approach enables effective de-duplication even for very large knowledge graphs by processing semantic clusters in parallel. When processing our largest 20M chunk dataset, this method success- fully consolidated entities like "Olympic Winter Games", "Winter Olympics", and "winter Olympic games" into single canonical representation.
5 Benchmarks for Extraction Performance
Although handful of existing methods attempt to extract KGs from plain text, it is difficult to measure progress on this task due to the lack of existing benchmarks. As a remedy, we produce the Measure of Information in Nodes and Edges (MINE), the first benchmark that directly measures knowledge-granl graph extractor's ability to capture and distill a body of text into KG. MINE contains two tasks: the first assesses how well a KG extractor captures the information in short, page-length text;

---

extraction using L.M
betwees extraeted entiies using LLM
Split into sets of 128 semantic similarity using BM25 and
Aggresaate initis
tnudfommr imboiddnn
Inital Kis
Aggregated KG
Select mode from each of 128 and the 15 nodes in the most
LJM combine duplicates in of clustens
Final
limes
Figure KGGen extraction method
the second measures downstream RAG performance on multi-million token datasets. call these tasks MINE-1 and MINE-2 respectively. MINE-1 ensures that a KG extractor accurately represents source text, and MINE-2 gauges the practicality the knowledge graphs for realistic applications.
5.1 MINE-1: Knowledge Retention
MINE-1 approximates the fraction of information a KG extractor is able to capture from an article without relying downstream tasks, which obscure whether performance gains stem from the KG extractor itself or from aspects of the extraction process.
MINE-1 consists of 100 articles, each accompanied by 15 facts that are known to be present in the article. The dataset has the following characteristics: articles have a mean length of 592 words (std. 85 words, range: 440-976 words) and cover diverse topics including Arts, Culture & Society (24 articles), Science (27 articles), Technology (19 articles), Psychology/Human Experience (18 articles), and History & Civilization (17 articles). Articles generated by LLM to ensure balanced coverage across these domains. For each article, MINE-1 generates a corresponding KG using the extractor being evaluated.
To assess the quality of these KGs, we extract 15 facts from each article using an LLM- extraction prompt found in Appendix B We manually verify that the 15 facts are accurate and contained in the article. To measure performance on MINE-1, the KG extractor first extracts a KG from each article. Then, MINE-1 contains process to verify whether each fact can be recovered from corresponding KG.
Verification occurs via a semantic query process: both the 15 facts and all KG nodes are embedded using the all-MiniLM-L6-1 model from CentenceTransformee the top-k most semantically similar nodes in the KG, then expands the result to include all nodes within two relations those top-k nodes, reflecting the fact that KG's are often used for multi-hop reasoning tasks. The subgraph induced by these nodes is passed to an LLM, which is prompted to output a binary score: the fact can be inferred from the retrieved nodes and relations alone, and otherwise. The prompt is detailed in Appendix B The final MINE-1 score for KG extractor is the percentage of the 15 facts scored as 1, averaged across all 100 articles. While LLM-based evaluation introduces potential biases, we validated its reliability by manually scoring 60 randomly selected fact-KG pairs and comparing them L.M judgments, achieving 90.2% agreement and a correlation of 0.80. The full evaluation pipeline is illustrated in Figure]
For each fact, the verifier retrieves
5

---

Essay
Fomral and vectorize
being evalusted
of cisep
KG query
Procentage
Vectonized KG
costailse
Dvaloation
Top reletions
Figure Evaluation process used in MINE-1
5.2 MINE-2: KG-Assisted RAG Description
The RAG evaluation is based on the WikiQA dataset Yang et al. 2015 which contains 20,400 questions based on 1,995 Wikipedia articles. Using the method under evaluation, construct single KG that aggregates information from all articles referenced in WikiQA. For each question in the dataset, we retrieve the top 10 most relevant triples by embedding both the question and all KG triples using the All-MiniLMM-L6- similarity between the question and each triple, alongside relevance score produced by BM25. The final similarity score is obtained by combining BM25 relevance score and the cosine similarity score, weighted equally. The 10 triples with the highest combined scores are selected, and further expand this set by appending 10 additional triples that fall within two hops of the nodes in the top 10 triples enable multi-hop reasoning.
model from
We then compute the cosine
Since each relation is linked to a source text chunk during generation by KGGen and GraphRAG, we provide the full set of 20 retrieved triples, their associated text chunks, and the original question to LM, which synthesizes an answer based on these inputs. The complete prompt used can be found in Appendix B Finally, the LM responses evaluated using LLMM-as-a-Judge to determine whether they contain the correct answer to the question. The prompt used for this final verification step is also included in Appendix B OpenIE excluded from this comparison, as it cannot produce KGs that link relations to the original text chunks.
6 Results
We use MINE to benchmark KGGen against leading existing methods of plain-text-to-K KG extraction: Apenll/Angelii et al. (2055] and GraphRAG and Truitt (2024) After providing this quantitative comparison of extraction fidelity, we present qualitative results that demonstrate the advantages of KGGen over past methods.
6.1 Evaluations on MINE-1
Figure displays accuracies from KGGen, OpenIE, and GraphRAG on MINE. Note that KGGen outperforms competing methods. Figure 4 shows example query from MINE-1 and relevant relations extracted by KGGen, OpenlE, and GraphRAG.
6.1.1 Generalization Across Language Models
To evaluate KGGen's robustness across different foundation models, we tested its performance on MINE-1 using multiple state-of-the-art L.LMs. Table [ shows that KGGen maintains strong performance across different models, with Claude Sonnet 3.5 achieving the highest score of 73%.

---

GraphRAG
OpenIE:
KGKiin
lreqquenyy (Artieles)
15
70
902
Facts captured (%)
Figure 3: Distribution of MINE-I scores across 100 articles for GraphRAG, OpenIE, and KGGen. Dotted vertical lines show average performance. KGGen scored 66.07% on average, significantly outperforming GraphRag 47.80 80% and OpenIE 29.84%.
Fact being queried for:
"Decentralization provides users with more control over their funds in cryptocumencies."
Extractor
Sample of triples queried
securityl (oryptomureeniies decentralized, coryptoourrencter tocurrencies,
(oryptocurrencles,
control funde)
privacy) iesyptocuzzeniee, teer-to-peez (czyptocurnncies, transaotions: (blockchain, ensures, traneparency)
Cryptomatrenctes are having a protoune impact on the Financial world by intraduring new ways of thinking about noney and rinanes, FIRANITAL WORLD) (BLOCRCHATN, cayptocucrencts operate soing Blookehain tachnology which provides scoure and tronopareht CRYPROCURRONCTGG)
(CRFPTCCURRENCCED
GosbRRAG
cerptoouzzencies. alloving tranaacLiona oucus between upсed, beed intermediaries) (oryptoouErencles, allowing, transactions to sccur disectly: [Cryptomureencies. have taker Finsrntal world in, storm) (Blockehain, ix, ledger tachnology! [Blockebain, iu distribated, ensures)
OpenIE
Figure 4: An example query from the MINE-1 benchmark. along with relevant relations in the KGs extracted by KGGen, GraphRAG, and OpenIE. Note that the relation triples extracted by KGGen contain the fact being queried for, whereas the KGs extracted by GraphRAG and OpenIE do not. The relation types extracted by KGGen are more concise and generalize more easily than those from GraphRAG and OpenIE, The full article that these relations were extracted from can be found in Appendix C
Table 1: Performance comparison of KGGen
(a) KGGGe performance MINE-1 across dif- ferent language models
(b) Validity of extracted triples across different
methods
Model
MINE-1 Score (%)
Method
Valid Triples (%)
Claude Sonnet 3.5
73
KGGen
98/100 (98%)
GPT-4o
66
GraphRAG
0/100 (0%)
Gemini 2.0 Flash
44
OpenIE
55/100 (55%)
KGGen's extraction methodology generalizes well across different foundation models. Although Claude Sonnet 3.5 achieves the highest score of 73%, tested models maintain reasonable extraction quality, making KGGen adaptable to users' preferred LLM providers.
7

---

6.2 Evaluations on MINE-2: RAG performance
Figure shows comparable performance between KGGen and GraphRag on MINE-2.
Grspiit
81.
K00cn
80.63%
Fully (%
Figure 5: Comparison between KGGen and GraphRag on MINE-2, The two methods perform comparably.
6.3 Evaluation on Human-Annotated Data: SemEval-2010
To evaluate KGGen's 's extraction quality against human-annotatee ground truth, conducted experi- ments using the SemEval-2010 Task 8 dataset. We randomly selected 100 sentences from the dataset, each containing two manually labeled target entities. After removing entity markup tags to ensure unbiased extraction, we applied KGGen to extract entities and relationships from each sentence.
Our evaluation focused on entity extraction accuracy, as the dataset's relation labels consist of only 5 broad categorical types rather than specific semantic relations. We assessed whether both human- annotated target entities appeared in KGGen's extracted entities, allowing for more specific entity descriptions (e.g., "Eurasia exhibition" instead of just "exhibition") that still referred the same object.
Results show that KGGen successfully captured both target entities in 96% cases (96/100). The method consistently extracted more detailed entity descriptions compared to human annotations, often identifying additional relevant entities in longer sentences. For example, given the sentence "The ambitious Eurasia exhibition arose from an idea by Achille Bonito Oliva," with target entities 'exhibition' and 'idea', KGGen extracted ['Eurasia exhibition', 'idea', 'Achille Bonito Oliva'), ]. providing more specific and complete entity identification.
6.4 Qualitative Results
We first evaluated the fundamental quality of extracted knowledge graphs by examining whether the extracted triples conform to the basic definition of knowledge graph: subject- prejictt-prddccat--ojjeee triples where subjects and objects are entities (nodes) and predicates are relationships (edges). We randomly selected 100 triples from each method and manually evaluated their validity. The results are shown Table [
Despite GraphRAG's comparable performance on downstream tasks, does not extract structures that closely resemble traditional knowledge graphs, which major strength of KGGen. As seen in Figures 66 and [ee GraphRAG often extracts very few nodes and connections for an entire article. This sparsity results the omission of critical relationships and information. For compression. Figure 6a and 6d illustrate sections of the KGs generated by KGGen for the same articles. Figure 6c illustrates one of many issues in OpenlE's KGs. Firstly, most node types are hyperspecific, incoherent phrases. Many of these nodes are redundant near-copies of each other, adding unnecessary complexity the graph. Additionally, as seen of OpenIE primarily uses pattern matching to identify entities. and frequently produces generic nodes such as "it" and "are". Due to their frequency, these nodes, which contain no useful information, often end up as some of most well-connected nodes the graph. Consequently, unrelated concepts end up being just two hops apart, linked by paths through nodes like "it" or "are". By contrast, KGGen consistently generates KGs that are informative and coherent, effectively capturing critical relationships and information from the articles.

---

MuLE
sssciiated
eicempte
indiation
Jiseenrg
ele elegance
aften
reado
ausoci socianed
pang
feelings of elegance
example offen with feelings
me reahi
sha ppcce
(a) Section of KG generated by KGGen on "How Music Influ- ences Mood"
(b) Full KG generated by GraphRAG on "How Music Influences Mood"
Section of KG generated by OpenIE, on "How Music Influ- ences Mood", with most node la- bels omitted for readability.
MAILLARD REACTION
prporates
OTe meatry
EMULSFICCTTON
cemtro
Precainge
TEMPERATURE CONTROL
Watel
Inseitius
PICKLING
retymote
Gitrery
FERMENTATION
brita
LEMON
JUICE
saggan
(d) Section of KG generated by KGGen "The Chemistry of Cooking"
(e) Full KG generated by
(f) Section of KG generated by OpenlE "The Chemistry of Cooking"
GraphRAG "The Chemistry of Cooking"
Figure 6: Visual comparison of KGs generated using KGGen, GraphRAG, and OpenIE. Results show that KGGen discovers more informative nodes to estimate a richer graph compared to GraphRAG, and collapses synonyms discover a more informative graph than OpenlE.
6.5 A Note on Scaling
A major motivation for the creation of KGGen was to produce graphs where edge types are gen- eralizable, and used more than once when the corpus grows large. test the re-usability of our relations, we generate three knowledge graphs from text of different sizes: 10000 characters, 100 0000 characters, and 000 000 characters and plot the number of edges divided by the number unique relations. As one can see from Figure 7 KGGen reuses each relation-tyy type average of 10 times, and the average number of occurrences of each relation-type increases with the size the corpus. By contrast, GraphRAG reuses each relation type average of 2 times regardless of the size of the graph. This suggests that the relations in GraphRAG do not generalize as the corpus grows.
6.6 Efficiency and Cost Analysis
To evaluate the practical applicability of KGGin, we analyzed its computational efficiency and cost on a large-scale extraction task. We extracted a KG from the novel The Name of the Wind by Patrick Rothfuss, processing text corpora of increasing sizes. Table 2 demonstrates KGGen's de-duplication effectiveness across different scales.
Table 2: KGGen scaling characteristics showing entity and relation de-duplication
Corpus Size (chars)
Pre- Enfities
Post- Entities
Entity De-dup Ratio
Pre- Relations
Post- Relations
Relation De-dup Ratio
Pre- Edges
Post- Edles
Edge De- Ratio
100
1
1.000
1
1
1.000
1
I
1.000
1,000
20
18
0.900
12
12
1.000
9
9
1,000
10,000
78
75
0,962
62
57
0,919
100,000
727
604
0.831
926
924
0.998
498
424
0.851
1,000,000
4,602
3,573
0.776
8,137
8,094
0.995
3,180
2,448
0.770
The de-duplication ratios improve with scale, demonstrating the effectiveness our clustering algorithm. For the complete novel (IM characters), KGGen achieved 22.4% reduction in entities

---

and 23% reduction in edges through intelligent consolidation. This analysis highlights the contribution
of each module: the extraction phase (Steps 1-2) captures comprehensive information from text, while the resolution phase (Step 3) significantly reduces redundancy without information loss. Table
presents the computational cost. breakdown for processing the entire novel:
Table 3: KGGen cost and throughput analysis for 1M character corpus
Step
Prompt Tokens
Completion Tokens
Total Tokens
Time
Throughput (tokens/s)
Cost
(s)
(S)
KG Extraction (Steps 1-2)
1.59M
0.63M
2.22M
273
8,139
0.46
Entity/Edge Resolution (Step 3)
2.93M
0.22M
3.15M
279
11,304
0.38
Total
4.52M
0.85M
5.37M
551
9,739
0.84
For comparison, we evaluated GraphRAG on the same corpus: results can be found in Table
Table 4: GraphRAG scaling characteristics the same corpus
Corpus Size (chars)
Entities
Relations
Edge Types
Time (s)
100
2
1.89
1,000
4
3
3.01
10,000
16
20
20
29.71
100,000
80
100
99
205.12
1,000,000
514
981
966
2,079.17
While GraphRAG is faster short corpora, its execution time scales superlinearly. The complete extraction time comparison reveals significant difference: GraphR requires 2,319 seconds for the extraction phase alone on the 1M character corpus, compared to KGGen's total processing time of 551 seconds (including both extraction and resolution). Additionally, GraphRAG produces nearly as many relation types (966) as edges (981), indicating minimal relation reuse and poor generalization compared to KGGen's 's efficient consolidation.
7 Broader Impact and Community Adoption
Our work produces a KG-from (GG-fro--plii-teee plan---lii-etttt extractor that helps to solve the KG-scarcity problem. Improved knowledge-graphe extractors could lead to the prevalence of structured text, which can help improve factuality and reliability of information retrieval systems. Our open-source implementation has already enjoyed widespread community adoption. The package has received over 700 Github stars and has been downloaded over 12,000 times since its release.
8 Limitations and Future Work
Although KGGen holds many advantages over past extraction methods, its graphs still exhibit problems, like over or under de-duplication of entities and relations. Further research into entity resolution could improve the quality our KGs. Additionally, our benchmarks measure corpora of up to 5M tokens, which does not reflect the size of web-scale text that would be necessary to produce KG foundation model. Future expansions of our benchmark could focus on larger corpora to better measure the practicality of different extraction techniques.
Domain-specific knowledge extraction presents additional challenges. Fields like medicine and finance require specialized domain knowledge that general-purpose LLMMs may lack, potentially limit- ing extraction quality compared to human experts. While MINE-2 demonstrates KGGen's capability across diverse domains, incorporating -omain-specificc ontologies could improve extraction precision. Future work could explore adaptive ontology integration to balance structure with completeness.
Acknowledgments and Disclosure of Funding
We acknowledge support from NSF grant numbers DGE-1656518, NSF 2046795 and 2205329, the Archur Foundation, Stanford HAL, and Google Incorporated.
10

---

References
Gabor Angeli, Arun Tejasvi Chaganty, Angel X. Chang, Kevin Scott Reschke, Julie Tibshirani, Jean Wu, Osbert Bastani, Keith Siilats, and Christopher D. Manning. Stanford's 2013 kbp system. Theory and Applications of Categories, 2013. URL
http:/pp...emaaaacccccooo..oooea org/
CorpuaID:1427363)
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. Leveraging linguistic structure for open domain information extraction. In Chengqing Zong and Michael Strube, editors, Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 344-354, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/P15- 1034. URRLhtttss://clanbboo wwcellggy.rgg/P11 org/P15-1030
Vahan Arsenyan, Spartak Bughdaryan, Fadi Shaya, Kent Small, and Davit Shahnazaryan. Large language models for biomedical knowledge graph construction: Information extraction from cmr notes. In Workshop on Biomedical Natural Language Processing, 2023. URL https
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 2, NIPS' 13, page 2787-2795, Red Hook, NY, USA, 2013. Curran Associates Inc.
Wikidata contributors. Wikidata: free collaborative knowledge base, 2024. URL phttps://w/
wikidata. org.
Giacomo Domeniconi, Gianluca Moro, Roberto Pasolini, and Claudio Sartori. study on term weighting for text categorization: A novel supervised variant of tf.idf. In Proceedings of the 4th International Conference on Data Management Technologies and Applications (DATA 2015), 07 2015. doi: 10.5220/000551 Ttt///0005511900000 1900260037.
Junyang Gao, Xian Li, Yifan Ethan Xu, Bunyamin Sisman, Xin Luna Dong, and Jun Yang. Efficient knowledge graph accuracy evaluation. ACM Transactions on Information Systems, 3-(2):1-21, 2018. doi: 10.14778/342263.344266 3342263. 3342642
htt////i///ff//// wtoi//pf/11..1100 14777/ URL htttp:://dd acm. orto/oig//ooooooooooooooooo Duke University and Amazon.com.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, YongDong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 20), pages 639-648. ACM, 2020. doi: 10.1145/399221.344106. URL https://doi. org/oog// 10.
3401066
Nicolas Heist, Sven Hertling, and Heiko Paulheim. Kgreat: A framework to evaluate knowledge graphs via downstream tasks. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management (CIKM '23), pages 3938-3942, ACM, 2023. doi: 10.1145/35833880331111
URL/https:// http:tpps///oi.orr//1...1..///// wttp//0..11///6337800011111 36152411 Published
on 21 October 2023.
Subhi Issa, Onaopepo Adekunle, Fayçal Hamdi, Samira Si-Said Cherfi, Michel Dumontier, and Amrapali Zaveri. Knowledge graph completeness: A systematic literature review. JEEE Access, 31322-31339, 2021. doi: 10.1109/ACCESS. WWW.:CCCCESS222.1355566600 URL htttps::/ieeexpporree ieee org/ htt//document/9944460
Shengbin Jia, Yang Xiang, Xiaojun Chen, Kun Wang, and Shijia. Triple trustworthiness measurement for knowledge graph. Proceedings of the World Wide Web Conference (WWW '19). pages 2271. ACM, May 2019. doi: 10. 10.1145333885583311555 3333338885883313866 URL https://doi...... org/ 1145/3308588 30313588888838888866888
Vamsi Krishna Kommineni, Birgitta König-Ries and Sheeba Samuel. From human experts to machines: Ilm supported approach to ontology and knowledge graph construction. ArXiv, abs/2403.083455, 2024. https: x httpc/hholllaoorrrrrapprseeee org/CorppsID:226337..0

---

Jonathan Larson and Steven Truitt.
Graphrag: Unlocking 1l discovery on nar-
rative private
graphrag-unlockingg 11m https://swws http:///www.iccoooot.coo com/// http://awro..ooooooooe/////eea//
data.
uu/research/blog!
2024. Microsoft
Research Blog, published Feb 13, 2024.
Jens Lehmann, Robert Iscle, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Scbastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. DBpedia A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia. Semantic Web Journal, 62):167-195, 2015. doi: 10.3233/SW-140.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI' 15, page 2181-2187. AAAI Press, 2015. ISBN 0262511290
Alexander Maedche and Steffen Staab. Ontology learning for the semantic web. IEEE Intelligent Systems, 16:72-79, 03 2001. doi: 10.1100955544920622
Idza Aisara Norabid and Fariza Fauzi. Rule-based text extraction for multimodal knowledge graph. International Journal of Advanced Computer Science and Applications, 2022. URL https
7/api. www.aeaaaaannicccho..... org/ Sttg//oopuuIDD224334730
Prakhar Ojha and Partha Talukdar. KGEval: Accuracy estimation of automatically constructed knowledge graphs. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1741- 1750, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/w//DI7-118 URL URLhttpp::/acclaattoolgg org -11/////11///
Sergio Oramas, Mohamed Sordo, and Luis Espinosa-Anke. A rule-based approach to extracting relations from music tidbits. In Proceedings of the 24th International Conference on World Wide Web, WWW "15 Companion, page 661-666, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450334730. doi: 10.1 00.145/2740900.22477700 URL
httpp:://doi.orr/// 10. 1145/2740908 2741709
Yifan Qi. Weiguo Zheng, Liang Hong, and Lei Zou. Evaluating knowledge graph accuracy powered by optimized human-machinn machine collaboration. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining (KDD '22), pages 1368-1378. ACM, 2022. doi:
10. 10.1145/3534678.3533 335392333..... URL httLhttp:://dde htt.:/o....og..r/oooooo 10 0011111533366888 3539233
Bin Qiao, Zhiliang Zou, Yurong Huang. Buyue Wang, and Changlong Yu. A joint model for entity and relation extraction based on BERT. Neural Computing and Applications, 34(5):3471-3483 2022. ISSN 1433-3058. doi: 10.1007/-00521-001--55888
https::////aao http:///ooooo////.///// 1007/ 1007/
000521-021-0581---
Edward W. Schneider. Course modularization applied: The interface system and its implications for sequence control and data analysis. In Association for the Development of Instructional Systems (ADIS), Chicago, Illinois, April 1973. Presented in April 1972.
Phillip Schneider, Tim Schopf. Juraj Vladika, Mikhail Galkin, Elena Simperl, and Florian Matthes. A decade of knowledge graphs in natural language processing: A survey. In Proceedings of the 2nd Chapter the ACL and the 12th International Joint Conference
Conference of the Natural Language Processing (AACL-IJCNLP 2022), 2022. doi: 10.18653/w1/2022.aaelll main. 46.
Kartik Shenoy, Filip Ilievski, Daniel Garijo, Daniel Schwabe, and Pedro Szekely. A study of the quality of wikidata. Journal of Web Semantics, Special Issue on Community-Baan Based Knowledge Bases, 06 2021. doi: 10448550/axx..22077000.666
Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: a core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web, WWW '07, page 697-706. New York, NY, USA, 2007. Association for Computing Machinery. ISBN 978 595936547.. doi: 10. 10.1145/1242572 1242667. URL https: doi org/ 11t11/1.111///222722222222
12

---

Jiajun Tan. Dong Wang, Jingyu Sun, Zixi Liu, Xiaoruo Li, and Yang Feng. Towards assessing the quality of knowledge graphs via differential testing. Available online, Version of Record, 2024. URL pttpss//dol: org/10 10. 1016/j jss 2024. 07 .005, Received 3 October 2023, Revised 15 June 2024, Accepted 26 June 2024, Available online June 2024.
Jiannan Wang, Tim Kraska, Michael J. Franklin, and Jianhua Feng. Crowder: crowdsourcing entity resolution. Proc. VLDB Endow., 5(11): 1494, July 2012. ISSN 2150-8097. 10.147782235022223355222
doi:
URL attps:///oii 14t.://..0000111111//////////000 14778/2350229 2350263
Bingcong Xue and Lei Zou. Knowledge graph quality management: A comprehensive survey. IEEE
Transactions on Knowledge and Data Engineering, 35(5):4969-4988 May 2023. ISSN 1041-4347. doi: 10. 109/TKDE
httpttttt///.i..... http:/dii..g/11111111111 TKDE. 2022. 3150080
Published on 10 February 2022.
Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open domain question answering. In Lluís Mårquez, Chris Callison-Buree and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013-2018, Lishon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.1865/v1/D1151220 URL
1237/
Bowen Zhang and Harold Soh. Extract, define, canonicalize: An Ilm-based framework for knowledge graph construction. In Conference Empirical Methods in Natural Language Processing, 2024. URL htttps://app
semanticscholar org/ : 268987666
Ruiqi Zhu, Alan Bundy, Jeff Pan, Kwabena Nuamah, Fangrong Wang, Xue Li, Lei Xu, and Ste- fano Mauceri. Assessing the quality of knowledge graph via link prediction tasks. In Pro- ceedings of the 7th International Conference on Natural Language Processing and Information Retrieval (NLPIR 2023), pages 1-10, Scoul, Republic of Korea, December 2023, ACM. doi: 10.11455/63323333333333 URL https: //doi/ org 10. 1145/3639233 3639357 School of Informaties, University of Edinburgh, United Kingdom; Huawei Ireland Research Centre, Ireland.
13

---

A Prompts for KG Extraction
This section provides the exact prompts used extract KG's from the text.
The initial KG was extracted using the following two prompts passed as DSPy signature descriptions.
Prompt for extracting entities: Extract key entities from the source text. Extracted entities are subjects or objects. This for an extraction task, please be thorough and accurate to the reference text.
Prompt for extracting relations: Extract subject-prest preettt triples from the source text. Subject and object must be from entities list. Entities provided were previously extracted from the same source text. This is for an extraction task, please be thorough, accurate, and faithful to the reference text.
After extracting the entities and relations from each unit of text, we begin the de-duplication process, which is performed using the following prompts.
Prompt for entity or edge resolution:
Find duplicate satem_type) for the item and alias that best represents duplicates. Duplicates are those that are the same in meaning, such as with variation in tense, plural form, stem form, case, abbreviation, shorthand. Return an empty list if there are none.
B Prompts for MINE
This section provides the LLM prompts used by MINE to evaluate KGs.
Prompt for extracting fact from article: Extract 15 basic, single pieces of information from the following text that describe how object relates to another. Present the pieces of info in short sentences and DO NOT include info not directly present in the text. Your output should be of the form "infol", "info2" ,..., "info15" ]. "Make sure the strings are valid Python strings."
Prompt for evaluating if a fact is contained in the query result: ROLE: "You are an evaluator that checks if the correct answer can be deduced from the information in the context.
TASK: Determine whether the context contains the information stated in the correct answer.
Respond with "1" if yes, and "0" if no. Do not provide any explanation, just the number.
14

---

Prompt for RAG response:
Use the following knowledge graph triples and text evidence to answer the question.
Triples: (triples_textl Text Evidence: bext_block)
Question: {query) Answer:
Prompt for Evaluating containment of WikiQA answer:
You are a fact-checking assistant
Question: question
Expected answer: expected
Model's 's response: response
Does the model's 's response contain the information in the expected answer?
Respond with one word: Yes or No.
C Example Article from MINE
This section provides the article that the example fact is from.
15

---

Title:
The Rise of Cryptocurrencies
Content: Cryptocurrencies have taken the financial
world by storm in recent years, revolutionizing the way we think about money and transactions. From the creation of Bitcoin in 2009 by an anonymous individual or group known as Satoshi Nakamoto, to the thousands of altcoins that have since emerged, cryptocurrencies have become a significant player in the global economy. One of the key factors contributing to the rise of cryptocurrencies is the decentralized nature of these digital assets. Unlike traditional fiat currencies that and central banks, cryptocurrencies operate on a peer-to-peer network, allowing for transactions to occur directly between users without the need for intermediaries. This decentralization not only provides users with more control over their funds but also enhances security and privacy. Another driving force behind the popularity of cryptocurrencies is the technology that underpins them blockchain. Blockchain is a distributed ledger technology that ensures the transparency and immutability of transactions on network. Each transaction is recorded in block and linked to the previous block, forming a chain of blocks that cannot be altered once validated by the network. This technology has been instrumental in building trust and confidence in cryptocurrencies, as it eliminates the need for trusted third party to oversee transactions. The concept of decentralization and blockchain technology has also paved the way for various applications beyond just digital currencies. Smart contracts, for example, are self-executing contracts with the terms of the agreement directly written into code. These contracts
controlled by governments
automatically enforce and execute themselves when predefined conditions met, eliminating the need for intermediaries and streaml streamlining processes in various industries. Cryptocurrencies have also gained traction due to their potential for financial inclusion. In many parts of the world, traditional banking services are inaccessible or too costly for significant portion the population. Cryptocurrencies offer a way for individuals to access financial services, such as transferring money without the need for traditional bank account. This has the potential to empower individuals in underserved communities and drive economic growth. The volatile nature of cryptocurrencies has attracted both investors seeking high returns and speculators looking to capitalize
making payments,
price fluctuations. The rapid appreciation led to surge in of certain cryptocurrencies, such as Bitcoin, alike. While this interest from retail and institutional investors volatility presents opportunities for profit, it also poses risks, prices fluctuate dramatically in short period. Regulation has been contentious issue in the cryptocurrency space, with governments and regulatory bodies grappling with how to oversee this emerging asset class.
16

---

Some countries have embraced cryptocurrencies and blockchain technology, recognizing their potential for innovation and economic growth. Others have taken a more cautious approach, citing concerns about money laundering, tax evasion, and consumer protection. Despite the challenges and uncertainties surrounding cryptocurrencies, their rise has been undeniable. As more
individuals and businesses adopt digital currencies for transactions and investments, the landscape finance is evolving rapidly. The future of cryptocurrencies remains uncertain, but their impact the financial world is already profound. In conclusion, the rise of cryptocurrencies can be attributed to their decentralized nature, blockchain technology, financial inclusion potential, investment opportunities, and regulatory challenges. As these digital assets continue to gain acceptance and adoption, they are reshaping the way think about money and finance. Whether cryptocurrencies will become mainstream or remain on the fringes of the financial system remains be seen, but their impact is undeniable and will likely continue to unfold in the years to come.
D
Additional Graphs
Average Number of Times that Each Relation is Occurs Text Size
KGGen
GraphRAG
Average Time Each Edge Used
105
Corpus Size (Tokens)
Figure 7: As graphs increase in size, KGGen tends each unique relation type frequently, while GraphRAG maintains consistent average usage of about 2 instances per relation type regardless of graph size.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We support all claims made in the abstract: we build knowledge- geggleggeeaaphhhhh extractor and benchmark its performance against existing mnwwedge-grappe extractors.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
We do not provide theoretical claims in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clearly describe methodology and will release code with the final version along with python module to perform KG extraction.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
We will provide code in the supplementary material, and release all code publicly.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https sonideSuumisssoooP.e
//nips. //nips.c//pil wwwppbblccgguiicuceccccc
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all information necessary replicate the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report statistical error bars where possible for our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experiments do not require special hardware, and can be most laptops with production models. They require only an API key from a model provider- this is clear from the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research is very ethical, and does not breach any of the NeurIPS ethics guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include broader impacts section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release new models or sensitive data. Our code uses mostly existing data, and we release small number of new data that we have checked harmless.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We credit everyone whose code ideas use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our assets include meticulous documentation for our new package and dataset, which will be released with the final paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
We do not crowd source.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA].
Justification:
We do have human test subjects and do not require IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
23

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We describe the role of LLMs in our methodology very clearly.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: for what should or should not be described.
24


Reviews:
[{"review_id": "IdEXNKh2vq", "reviewer": "Reviewer_vhqp", "summary": "The paper introduces a software package for extracting knowledge graphs directly from text using Large Language Models (LLMs). Specifically, Gemini 2.0 Flash is used to extract entities (nodes) and relationships (edges) between nodes. Subsequently, a clustering approach is used to perform entity resolution and thus reduce the sparsity of the graph. To evaluate the proposed approach, the authors also introduce two datasets. Experimental results show that the proposed approach has performance comparable to that of Microsoft’s\nGraphRAG, while being more scalable.", "questions": "Why could't you evaluate the proposed approach on existing triple datasets that are manually extracted?", "limitations": "The authors discuss some limitations.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The authors focus on the important task of leveraging LLMs to extract knowledge graphs from plain text. \n\nTwo novel datasets for evaluating approaches for extracting knowledge graphs are introduced. \n\nThe approach proposed is very basic and not well-explained. \n\nThe datasets introduced are not well described. Some statistics about the datasets would be useful. \n\nThe experimental evaluation is done by verifying facts. It would also be good to evaluate the actual triples extracted. In fact, it is not clear why the authors can't evaluate this approach on existing knowledge graphs. \n\nThe results of the proposed approach are not better than those of one of the baselines, despite the improved scalability of the proposed approach.\n\nThe paper is not very well written. The Limitations section is included in two places.", "was_revised": true, "final_justification": "The authors responded to my questions and concerns. However, I still think the evaluation of the approach, in its current form, may not be very reliable. Nevertheless, I increased my original score.", "rebuttal": "Thank you for taking the time to review our paper.  While we appreciate and will integrate your suggestions for improvement, we disagree with your overall characterization of our paper.  We address the identified weaknesses and questions below, and we hope that you'll reconsider your score.\n\n\n>The datasets introduced are not well described. Some statistics about the datasets would be useful.\n\n\n We are not exactly sure what statistics would be interesting to readers.  To describe our datasets better:\n\n\n**MINE1:**\nNumber of articles: 105\nMean Length: 592 Words\nStd. Length: 85 Words\nMin Length: 440 Words\nMax Length: 976 Words\nTopic Breakdown:\n- Arts, Culture & Society: 24 essays\n- Science: 27 essays\n- Technology: 19 essays\n- Psychology/Human Experience: 18 essays\n- History & Civilization: 17 essays \n\n\n**Mine 2**:\nMine 2 is evaluated on the public WikiQA dataset, which contains 20,400 questions based on 1,995 wikipedia articles.  We are happy to provide more specific details about this dataset from Microsoft if the reviewer is interested.\n\n\n\n\n>The experimental evaluation is done by verifying facts. It would also be good to evaluate the actual triples extracted. In fact, it is not clear why the authors can't evaluate this approach on existing knowledge graphs.\n\n\nWe can’t evaluate this approach on existing knowledge graphs because there is no good metric for whether two knowledge graphs are similar.  As you point out in your review, there are no knowledge-graph foundation models due to lack of data, which leaves embedding similarity out of the question.  While there exist methods of graph embeddings, these only compare graph structure without taking node-level data into account.  Moreover, human-labeled knowledge graphs are not infallible: it is well-known that even the largest human-labeled knowledge graphs, such as WikiQA, have millions of missing nodes and relations.  Fact-verification is an efficient and objective way to measure a KG-extractor’s effectiveness.  We also note that past works, like GraphRAG and OpenIE did not perform verification via triples either.  In fact, our evaluations are already far more extensive than past works in the field, which generally verify based on metrics like information-compression.\n\n\n\n\n> The results of the proposed approach are not better than those of one of the baselines, despite the improved scalability of the proposed approach.\n\n\nWe first note that improved scalability is very important for a method to be practical, and should not be downplayed.  \n\n\nAlthough GraphRAG performed similarly on retrieval benchmarks, it does not conform to the basic tenet of knowledge graphs: a knowledge graph consists of subject-predicate-object triples, where the subject and object are nodes and the predicate is an edge.  We chose 100 node-triple-node pairs from each of OpenIE, GraphRAG, and KGGen and manually evaluated the percentage of these triples that conform to the definition of a knowledge graph.  The results were as follows: \n\n\n| Method | Fraction Valid |\n|--------|-------|\n| KGGen | 98/100 |\n| GraphRAG | 0/100 |\n| OpenIE | 55/100 |\n\nDespite GraphRAG’s comparable performance, it doesn’t extract something that closely resembles a knowledge graph, which is another major strength of KGGen over GraphRAG.\n\n\n>The paper is not very well written. The Limitations section is included in two places.\n\n\nWe apologize for this mistake.  We forgot to reformat the paper until very near the deadline, and we were rushing to adapt the paper for the NeurIPS template: mistakes were made.  Please be assured that if our work is accepted, we will ensure that the writing is cleaner and limitations only appear once.\n>Why couldn't you evaluate the proposed approach on existing triple datasets that are manually extracted?\n\n\nManually extracted KG’s are not unique, as two different humans will likely extract different KGs from the same text.  Even if there were agreement on the “correct” KG for a given text, it is very difficult to measure similarity between two knowledge graphs given that there are no foundation models to provide embeddings."}, {"review_id": "Id3TAJxlMD", "reviewer": "Reviewer_6cS2", "summary": "The authors in this paper introduce KGGen, a novel system for extracting KGs from plain text using LMs and entity resolution – identifying nodes that refer to the same underlying entities and consolidating similar/equivalent edges. The authors propose that KGGen addresses graph sparsity with this approach, unlike OpenIE and GraphRAG by utilizing LLM-guided clustering and resolution. They also propose MINE-1 for information retention and MINE-2 for RAG-style retrieval performance for KG evaluation. KGGen is an open-source package available on Github.", "questions": "Q1: Does KGGen’s core components generalize better across other LLMs (Claude, GPT)?\n\nQ2: What are some challenges observed with building KG from text that is domain-specific? Has this been explored by the authors? Domains like medical, financial can be quite challenging to build a KG on. This can also be thought of as a limitation since building a KG requires domain knowledge and often times the context human-evaluators/annotators carry.\n\nQ3: What happens if the clusterings are incorrect or inaccurate?\n\nQ4: Does KGGen take into account ontologies or ontology-aware extractions?", "limitations": "Aside from the limitation addresses in Q2, I am interested in knowing from the authors if they have observed any biases in the LLMs and if they observed this compounding during the growth of the KGs.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "S1: The paper is well-written and is clear to follow.\n\nS2: KGGen combines S-BERT embeddings with k-means clustering for entity and edge resolution which is quite significant to improve the coherence of the KG.\n\nS3: The system also uses multi-stage approach for LLM-based KG extraction and enforce strong constraints on the model through prompting. This primarily aids in preventing incorrect entity groupings which is a strong step to strengthen the objective of the framework.\n\nS4: The benchmarks proposed (MINE-1 and MINE-2) is a novel evaluation approach.\n\n\nW1: Both the evaluation approaches rely on LLMs which introduces the concern of hallucinations and bias.\n\nW2: While the work states the corpora size, it is the first occurrence of this and there is missing context on the corpora, its contents and how KGGen generalizes with the growth of the corpus/corpora.", "was_revised": true, "final_justification": "I will maintain my score as the proposed work because of the impact and the novelty in the domain.", "rebuttal": "REBUTTAL:\n\n\nThank you for the positive view of our work.  We are glad that you appreciated the advantages of KGGen over past methods!\n\n\nWe hope that we can address some of your remaining concerns below:\n\n\n> W1: Both the evaluation approaches rely on LLMs which introduces the concern of hallucinations and bias.\n\nThat is a fair point.  However, potential bias is a drawback of __any__ method that uses LLMs or even NLP, so it is not a unique weakness of KGGen.  \n\n> W2: While the work states the corpora size, it is the first occurrence of this and there is missing context on the corpora, its contents and how KGGen generalizes with the growth of the corpus/corpora.\n\nThanks for pointing this out.  MINE-2 was evaluated on the public WikiQA dataset, which consists of 20,400 questions about 1,995 Wikipedia articles.  The details of this dataset can be found publicly.  MINE-1 was evaluated on 105 synthetically generated articles.  We provide some general statistics about these articles and a breakdown of their topics below:\nMean Length: 592 Words\nStd. Length: 85 Words\nMin Length: 440 Words\nMax Length: 976 Words\nTopic Breakdown:\n- Arts, Culture & Society: 24 essays\n- Science: 27 essays\n- Technology: 19 essays\n- Psychology/Human Experience: 18 essays\n- History & Civilization: 17 essays \n\n\n\n\n> Q1: Does KGGen’s core components generalize better across other LLMs (Claude, GPT)?\n\n\nTo answer this question, we measured the performance of KGGen on MINE-1 using 2 other models:\n| Model | MINE-1 Score |\n|-------|---------------|\n| Sonnet 4 | 73% |\n| GPT-4o | 66% |\n| Gemini | 44% |\n\n\nAs you can see, the KGGen generalizes well across many different foundation models.  \n\n\n> Q2: What are some challenges observed with building KG from text that is domain-specific? Has this been explored by the authors? Domains like medical, financial can be quite challenging to build a KG on. This can also be thought of as a limitation since building a KG requires domain knowledge and often times the context human-evaluators/annotators carry.\n\n\nMINE-2 aims to address this question by testing KGGen on factual articles that span many different topics.  KGGen performed well on MINE-2, demonstrating its capability across domains.  We agree that sometimes human domain-experts will outperform LLMs on their areas of expertise, but this is an inherent challenge in building a KG-extractor that generalizes to arbitrary text.  Unfortunately, there are not enough domain experts to construct large-scale and comprehensive KGs, leaving the task up to methods like KGGen.\n\n\n> Q3: What happens if the clusterings are incorrect or inaccurate?\n\nDuring the clustering process, sub-nodes of each cluster are stored.  Therefore, if one suspects that a cluster contains an inappropriate member, one can easily remove and correct the error.  \n\n> Q4: Does KGGen take into account ontologies or ontology-aware extractions?\n\nThanks for asking this question! \n\nWe conducted a preliminary experiment that found that specifying an ontology allows the extractor to deterministically adhere to the ontology, but led to a less thorough extraction. Specifically, we ran the extractor over a family tree description and found that only 14/39 were identified with the family ontology whereas 39/39 related people were identified without one.  For this reason, a version of KGGen that allows for pre-specified ontologies is currently the subject of a follow-up work."}, {"review_id": "vRHBMwduyT", "reviewer": "Reviewer_fE3b", "summary": "This paper introduces KGGen, a generator for extracting knowledge graphs from unstructured plain text. KGGen uses language\nmodels to extract high-quality graphs from plain text with a novel entity resolution approach. To evaluate their generator, the authors propose a new two-part benchmark, MINE (Measure of Information in Nodes and Edges). The results show that KGGen outperforms baselines like GraphRAG and OpenIE on MINE-1 and achieves comparable performance to GraphRAG on MINE-2, The work is also presented as an open-source Python package to encourage adoption.", "questions": "**Suggestions/Questions**:\n\n1. What is the evaluation accuracy of the LLM-based scoring in the benchmark MINE-1?\n2. Providing a detailed comparision of efficiency and cost between different methods.\n3. Paying attention to the paper writting (repeated sections).\n4. Providing more detailed analysis to key modules of the proposed method.\n5. How to process knowledge conflict when creating KGs?\n\nI'm willing to raise my score if these suggestions/questions are solved.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is generally well-written and clearly structured.\n2. The authors have not only proposed the KGGen method for the important text-to-knowledge-graph task, but also implemented it as an useful open-source Python package.\n3. The proposal of benchmark MINE is a valuable conceptual contribution. \n\n**Weaknesses:**\n1. The MINE-1 scoring mechanism relies on an LLM-as-a-Judge to determine if a fact can be inferred from a KG subgraph. The reliability, bias, and accuracy of this judge are unknown and uncalibrated. \n2. The paper's ultimate goal is to create more useful KGs. However, on the MINE-2 RAG task, KGGen only performs \"comparably\" to GraphRAG. \n3. The paper is missing several crucial analyses. There are no ablation studies to isolate the impact of the key modules of the KGGen. Additionally, there is no discussion or comparison of efficiency or cost (e.g., inference time, LLM API tokens/cost), which are critical factors for a system designed to operate on large text corpora.\n4. The manuscript contains format errors. There are two \"Limitations and Future Work\" sections (Section 6 and Section 9) and two \"NeurIPS Paper Checklist\". \n5. While the KGGen system is a well-designed piece of engineering, its core components are largely combinations of existing techniques.", "was_revised": true, "final_justification": "My concerns have been solved.", "rebuttal": "Thanks for your helpful suggestions for improvement.  We address your concerns below, and we hope that you'll reconsider your score.  \n\n\n**Weaknesses and Questions:**\n\n\n>The paper's ultimate goal is to create more useful KGs. However, on the MINE-2 RAG task, KGGen only performs \"comparably\" to GraphRAG.\n\n\nAlthough GraphRAG performed similarly on retrieval benchmarks, it does not conform to the basic tenet of knowledge graphs: a knowledge graph consists of subject-predicate-object triples, where the subject and object are nodes and the predicate is an edge.  We chose 100 node-triple-node pairs from each of OpenIE, GraphRAG, and KGGen and manually evaluated the percentage of these triples that conform to the definition of a knowledge graph.  The results were as follows: \n\n\nKGGen: 98/100\nGraphRAG: 0/100\nOpenIE: 55/100\n\n\nDespite GraphRAG’s comparable performance, it doesn’t extract something that closely resembles a knowledge graph, which is another major strength of KGGen over GraphRAG.  As discussed in the paper, KGGen exhibits better scaling properties than GraphRAG as well.\n\n\n> What is the evaluation accuracy of the LLM-based scoring in the benchmark MINE-1?\n\n\nThis is a reasonable question.  Scoring by hand is extremely labor intensive.  However, we scored 60 questions by hand and compared our judgements to those of GPT-4o.  We found a 90.2% agreement rate and a 0.80 correlation.  Therefore, LLM-as a judge is fairly accurate for this application.\n\n\n> Providing a detailed comparison of efficiency and cost between different methods.\nAND\n>Providing more detailed analysis to key modules of the proposed method.\n\n\nThis is a fair point. While OpenIE is not LLM-based and thus cost comparison is not applicable, we can provide some insight into how costs and graph characteristics scale for KGGen as well as GraphRAG. To provide some variety in terms of genre, we extract a KG using KGGen with Gemini from the novel __Name of the Wind__, by Patrick Rothfuss.  As one can see, the number of relations grows sub-linearly, and the entity de-duplication ratio improves with scale, showing the effect of the clustering algorithm.  \n\n\n| Corpus Size (chars) | Pre Entities | Post Entities | Entity Dedup Ratio | Pre Relations | Post Relations | Relation Dedup Ratio | Pre Edges | Post Edges | Edge Dedup Ratio |\n|---------------------|--------------|---------------|-------------------|---------------|----------------|---------------------|-----------|------------|------------------|\n| 100                 | 1            | 1             | 1.000000          | 1             | 1              | 1.000000            | 1         | 1          | 1.000000         |\n| 1000                | 20           | 18            | 0.900000          | 12            | 12             | 1.000000            | 9         | 9          | 1.000000         |\n| 10000               | 90           | 78            | 0.866667          | 78            | 75             | 0.961538            | 62        | 57         | 0.919355         |\n| 100000              | 727          | 604           | 0.830812          | 926           | 924            | 0.997840            | 498       | 424        | 0.851406         |\n| 1000000             | 4602         | 3573          | 0.776402          | 8137          | 8094           | 0.994715            | 3180      | 2448       | 0.769811         |\n\n\nThe costs for each phase of KGGen on the entire novel can be found in the table below. We hope that the reviewer will agree that these costs are quite reasonable given the corpus size.\n\n\n| Step | Prompt Tokens | Completion Tokens | Total Tokens | Time (s) | Tokens/s | Cost ($) |\n|------|---------------|-------------------|--------------|----------|----------|----------|\n| KG Extraction (Steps 1–2) | 1592718 | 626481 | 2219199 | 272.65 | 8139.4 | 0.462726 |\n| Entity/Edge Resolution (Step 3) | 2925846 | 222790 | 3148636 | 278.54 | 11304.2 | 0.381701 |\n| TOTAL | 4518564 | 849271 | 5367835 | 551.19 | 9738.7 | 0.844427 |\n\nFor comparison, we also evaluated GraphRAG using the same corpus and model. Note that we do not have accurate token usage and cost data for GraphRAG since it makes LLM calls internally.\n\n| Corpus Size (chars) | Entities | Relations | Edge Types | Time (s) |\n|---------------------|----------|-----------|------------|----------|\n| 100                 | 2        | 1         | 1          | 1.89     |\n| 1000                | 4        | 3         | 3          | 3.01     |\n| 10000               | 16       | 20        | 20         | 29.71    |\n| 100000              | 80       | 100       | 99         | 205.12   |\n| 1000000             | 514      | 981       | 966        | 2079.17  |\n\n>How to process knowledge conflict when creating KGs?\n\n\nCurrently we don’t have a general solution to this problem.  Resolving knowledge conflict would likely require post-processing whose specifics vary based on the corpus.\n\n\n> Paying attention to the paper writing (repeated sections).\n\n\nWe apologize for this mistake.  We forgot to reformat the paper until very near the deadline, and we were rushing to adapt the paper for the NeurIPS template: mistakes were made.  While we are not allowed to upload a new manuscript during the rebuttal period, please be assured that if our work is accepted, we will ensure that the writing is cleaner and limitations only appear once."}, {"review_id": "cwCZYcdPb6", "reviewer": "Reviewer_S7Xx", "summary": "KGGen introduces a novel language model-based method for extracting knowledge graphs (KGs) from plain text, addressing KG sparsity through entity/relation resolution. Key contributions include: (1) A three-stage pipeline (entity/relation extraction → aggregation → iterative resolution) that clusters synonymous entities/edges using hybrid embedding+LLM deduplication; (2) MINE, the first benchmark for text-to-KG extraction, with tasks for knowledge retention (MINE-1) and downstream RAG performance (MINE-2); (3) Empirical validation showing KGGen outperforms GraphRAG by 18% and OpenIE by 36% on MINE-1, achieves comparable RAG results to GraphRAG on MINE-2, and demonstrates superior relation reuse during scaling. The tool is released as a Python library.", "questions": "1. Could the resolution stage incorporate domain-specific constraints (e.g., medical ontologies) to reduce under-clustering of specialized terms?\n\n2. What is the throughput (tokens/second) of KGGen on 1M-token corpora, and how does cost scale vs. GraphRAG?\n\n3. In MINE-1, why use an LLM for fact verification instead of human annotation? Could model biases affect scores?\n\n4. The authors claims KGGen has already enjoyed widespread community adoption. KGGen has received over 450 Github stars and has been downloaded as a python module over 11,000 times since its release in January. Thus, KGGen has already made an impact in the community that studies knowledge graphs. Does this violate the double-blind policy?", "limitations": "1. Generalizability: Evaluation primarily on Wikipedia-style articles; performance on conversational or technical text is unknown.\n\n2. Black-box dependencies: Entity merging relies on Gemini without error analysis (e.g., confusion rates for ambiguous entities).", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Innovative resolution technique: Hybrid clustering (S-BERT + k-means) followed by LLM-powered canonicalization effectively reduces sparsity, a critical bottleneck in KG extraction.\n\n2. Benchmark contribution: MINE fills a methodological gap by directly evaluating KG fidelity to source text, avoiding conflated downstream-task metrics.\n\n3. Rigorous evaluation: Comprehensive comparisons against GraphRAG/OpenIE across metrics (information retention, sparsity, scalability) with statistical significance reporting.\n\nWeaknesses:\n\n1.Computational cost: Heavy reliance on Gemini 2.0 Flash and iterative clustering may limit web-scale deployment (acknowledged but not quantified).\n\n2. Under/over-clustering issues noted without analysis of failure modes (e.g., handling complex entities like \"MRI vs. CT scan\").", "was_revised": true, "final_justification": "Thanks for the authors' response. I have read it and decided to maintain my positive score.", "rebuttal": "Thank you for your diligent assessment of our contributions.  We address the weaknesses and limitations that you identified below.  \n\n\n**Weaknesses**\n\n\n>Could the resolution stage incorporate domain-specific constraints (e.g., medical ontologies) to reduce under-clustering of specialized terms?\n\nThanks for asking! \n\nWe conducted an experiment that found that while specifying an ontology allows the extractor to deterministically adhere to the ontology, it led to a less thorough extraction. Specifically, we ran the extractor over a family tree description and found that only 14/39 were identified with the family ontology whereas 39/39 related people were identified without one. This is currently the subject of a follow-up work, in which we are developing a version of KGGen that can ingest an ontology to better guide its extraction.  \n\n>What is the throughput (tokens/second) of KGGen on 1M-token corpora, and how does cost scale vs. GraphRAG?\n\n\nWe currently do not have accurate cost numbers for GraphRAG aside from time required for extraction since GraphRAG makes LLM calls internally. However, we can provide this data for KGGen with Gemini. On a 1M-token corpus, here is the breakdown for cost and time, along with the throughput:\n\n\n| Step | Prompt Tokens | Completion Tokens | Total Tokens | Time (s) | Tokens/s | Cost ($) |\n|----------------------------------------------------------|---------------|-------------------|--------------|----------|----------|----------|\n| KG Extraction (Steps 1–2) | 1592718 | 626481 | 2219199 | 272.65 | 8139.4 | 0.462726 |\n| Entity/Edge Resolution (Step 3)   | 2925846 | 222790 | 3148636 | 278.54 | 11304.2 | 0.381701 |\n| KGGen TOTAL | 4518564 | 849271 | 5367835 | 551.19 | 9738.7 | 0.844427 |\n| GraphRAG extraction |- |- |- | 2318.90|-|-|\n\n>In MINE-1, why use an LLM for fact verification instead of human annotation? Could model biases affect scores?\n\n\nThis is a reasonable question.  Scoring by hand is extremely labor intensive.  However, we scored 60 questions by hand and compared our judgements to those of GPT-4o.  We found a 90.2% agreement rate and a 0.80 correlation.  Therefore, LLM-as a judge is fairly accurate for this application.\n\n\n\n\n\n\n\n\n>The authors claims KGGen has already enjoyed widespread community adoption. KGGen has received over 450 Github stars and has been downloaded as a python module over 11,000 times since its release in January. Thus, KGGen has already made an impact in the community that studies knowledge graphs. Does this violate the double-blind policy?\n\n\nWe deliberately redacted the package name.  There are many python modules that have been downloaded 11,000 times in the last several months, so this is not a unique identifier of KGGen.\n\n\n**Limitations**\n\n\n>Generalizability: Evaluation primarily on Wikipedia-style articles; performance on conversational or technical text is unknown.\n\n\nWe ran KGGen on some fictional works that contain dialogue as well.  However, it is much more difficult to evaluate performance on non-factual genres.  Moreover, the primary use-cases for knowledge-graphs involve factual text and databases.  MINE-1 and MINE-2 incorporate a broad array of texts to help assess KGGen holistically.  Mine 2 is evaluated on the public WikiQA dataset, which contains 20,400 questions based on 1,995 wikipedia articles, and the details of MINE-1 can be found below.\n\n\n**MINE1:**\nNumber of articles: 105\nMean Length: 592 Words\nStd. Length: 85 Words\nMin Length: 440 Words\nMax Length: 976 Words\nTopic Breakdown:\n- Arts, Culture & Society: 24 essays\n- Science: 27 essays\n- Technology: 19 essays\n- Psychology/Human Experience: 18 essays\n- History & Civilization: 17 essays \n>Black-box dependencies: Entity merging relies on Gemini without error analysis (e.g., confusion rates for ambiguous entities).\n\n\nThis is a fair criticism, which is common to many LLM-applications today.  We are committed to improving error analysis, and we believe that we will have more insight into the necessary steps towards this goal as KGGen enjoys more widespread adoption."}]

Author rebuttal:

Meta-review: This paper introduces KGGen, a language-model-based framework for extracting knowledge graphs from plain text with entity/relation resolution, accompanied by the new MINE benchmarks for evaluating information retention and downstream retrieval performance. Reviewers appreciated the novelty, clarity, and practical impact of the work, highlighting its open-source release, strong performance against GraphRAG and OpenIE, and meaningful contributions through benchmark design. While some concerns were raised regarding evaluation reliability (LLM-as-a-judge), dataset descriptions, and missing ablations, the authors provided additional experiments and clarifications during rebuttal that addressed most issues. Overall, the paper makes a solid methodological and resource contribution to the growing area of text-to-KG extraction and is recommended for acceptance.
Decision: Accept (poster)