KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment
Authors: Yuxing Lu
Wei Wu
Xukai Zhao
Rui Peng
Jinzhuo Wang
Venue: NeurIPS 2025 spotlight ()
Keywords: Knowledge Graphs
Multi Agent System
Primary area: machine_learning_for_sciences
TL;DR: KARMA is a multi-agent LLM framework that automatically update and expand knowledge graphs from scientific papers. By using nine specialized agents, KARMA successfully add new entities with high verified accuracy and reduce inconsistencies.

Abstract: Maintaining comprehensive and up-to-date knowledge graphs (KGs) is critical for modern AI systems, but manual curation struggles to scale with the rapid growth of scientific literature. This paper presents KARMA, a novel framework employing multi-agent large language models (LLMs) to automate KG enrichment through structured analysis of unstructured text. Our approach employs nine collaborative agents, spanning entity discovery, relation extraction, schema alignment, and conflict resolution that iteratively parse documents, verify extracted knowledge, and integrate it into existing graph structures while adhering to domain-specific schema. Experiments on 1,200 PubMed articles from three different domains demonstrate the effectiveness of KARMA in knowledge graph enrichment, with the identification of up to 38,230 new entities while achieving 83.1\% LLM-verified correctness and reducing conflict edges by 18.6\% through multi-layer assessments.

KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment
Yuxing 1,2 Wei Wu', Xukai Zhao Rui Peng', Jinzhuo Wang
Department of Big Data and Biomedical AI, Peking University, Beijing, China
Wallace Coulter Department of Biomedical Engineering, Georgia Institute of Technology. Atlanta, USA School of Architecture, Tsinghua University, Beijing, China Corresponding author: wangj www.iinnhhuuuuueeeeoooe edu.
Abstract
Maintaining comprehensive and up-to-date knowledge graphs (KGs) critical for modern Al systems, but manual curation struggles to scale with the rapid growth of scientific literature. This paper presents KARMA, a novel framework employing multi-agent large language models (LLMs) automate KG enrichment through structured analysis of unstructured text. Our approach employs nine collaborative agents, spanning entity discovery, relation extraction, schema alignment, and conflict resolution that iteratively parse documents, verify extracted knowledge, and integrate it into existing graph structures while adhering domain-specific schema. Experiments on 1,200 PubMed articles from three different domains demonstrate the effectiveness of KARMA in knowledge graph enrichment, with the identification of up to 38,230 new entities while achieving 83.1 % LLM-verified correctness and reducing conflict edges by 18.6% through multi-layer assessments.
Introduction
Knowledge graphs (KGs) are essential for structuring and reasoning over complex information across diverse fields 9 [8] By encoding entities and their relationships in machine-readable formats, widely adopted KGs such as Wikidata (26) and DBpedia [13] have become foundational to both industry and academic research. Yet, the exponential growth of scientific literature, with over 7 million articles published annually [2), exposes a significant bottleneck: widening gap between unstructured knowledge in texts and its structured representation in KGs.
The challenge enriching KGs becomes even more apparent in fields with complex and specialized terminology, such as healthcare, finance, or autonomous systems. Traditional approaches to KG enrichment, such as manual curation, are reliable but unsustainable at scale. Automated methods based on conventional natural language processing (NLP) techniques often struggle to handle domain- specific terminology and context- -ependtde relationships found in scientific and technical texts [22]. Moreover, extracting and integrating knowledge into existing KGs requires robust mechanisms for schema alignment, consistency, and conflict resolution (5). high-stakes applications, the costs of inaccuracies in these systems can be severe.
Recent advances large language models (LLMs) 15 have demonstrated remarkable improve- ments in contextual understanding and reasoning (28). Building on these advances, the research community has increasingly explored multi-agent systems, where several specialized agents work in concert to tackle complex tasks [8) These systems harness the strengths of individual agents, each optimized for a particular subtask, and enable cross-agent verification and iterative refinement of outputs. Such multi-agent frameworks have shown promise in areas ranging from docision-makinn to structured data extraction
[9 offering robustness through redundancy and collaboration. However,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Multi-Agent
Multi-Agent
Article
Knowledge
KG Enrichment
Figure 1: Multi-agent LLM can parse articles into new knowledge, and integrate to existing knowledge graphs through filtering.
directly applying these systems to KG enrichment remains challenging due to issues like domain adaptation, systematic verification requirements [00], and the complexity of integrating outputs into heterogeneous knowledge structures.
In this paper, we propose KARMA, a novel multi-agent framework that harnesses LLMs through collaborative system of specialized agents (Figure [].. Each agent focuses on distinct tasks in the KG enrichment pipeline. Our framework offers three key innovations. First, the multi-agent architecture enables cross-agent verification, enhancing the reliability of extracted knowledge. For instance, Relationship Extraction Agents validate candidate entities against Schema Alignment outputs, while Conflict Resolution Agents resolve contradictions through LLM--baed debate mechanisms. Second, domain domain-adaptive prompting strategies allow the system to handle specialized contexts while preserving accuracy. Third, the modular design ensures extensibility and supports dynamic updates as new entities or relationships emerge. Through proof-of-come three distinct domains, we demonstrate that KARMA can efficiently extract high-quality knowledge from unstructured texts, substantially enriching existing knowledge graphs with both precision and scalability.
experiments on datasets from
2
Related Work
2.1 Knowledge Graph Construction
The quest to transform unstructured text into structured knowledge has evolved through three gen- erations of technical paradigms. First generation systems (1990s-2010s) like WordNet [2]] and ConceptNet [16] relied on hand-crafted rules and shallow linguistic patterss, achieving high precision at the cost of limited recall and domain specificity. The neural revolution (2010s-22) 202002)2222) introduced learned representations through architectures like BioBERT
[2] and SapBERT [17,, which achieved improvements on biomedical NER through domain-adaptive pretraining. However, these methods require expensive supervised tuning (3-5k labeled examples per relation type 30) and fail to gen- eralize beyond predefined schema, which is a critical limitation when processing novel scientific discoveries. The current LLM-powered generation 2022-pressmm) attempts overcome schema rigidity through instruction tuning [24] 31. This progression reveals an unresolved tension: neural methods scale better than rules but require supervision, while LLMs enable open schema learning at the cost of verification mechanisms. LLMs have shown promise in open- open-domain KG construction through their inherent reasoning capabilities. However, these approaches exhibit critical limitations: (1) Hallucination during extracting complex relationships [20), (2) Inability to maintain schema consistency across documents articles 23
29 and (3) Quadratic computational costs when processing full-text
2.2 Multi-Agent Systems
Early multi-agent systems focused on distributing subtasks across specialized modules, such as separate agents for named entity recognition and relation extraction [3]. These systems relied on predefined pipelines and handcrafted coordination rules, limiting adaptability new domains. Recent advances in LLMs have enabled more dynamic architectures and rediscovered multi- collaboration as a mechanism enhancing LLM reliability (5 19]. Building on classic blackboard architectures, contemporary systems like AutoGen 28 show that task decomposition with specialized agents reduces hallucination compared to monolithic models. For knowledge graph construction, [14] demonstrated that task decomposition across specialized agents (e.g.. entity linker, relation validator)

---

Central Controller
Parsing
Extraction
Integration
Ingestion Agent
Schema Alignment
Minnt
Evaluator Agent
Contidencs
Reader Agent
Extraction Agent
O
O
Levet
Relacora
Storage
Taets
Ressita
Summarizer Agent
Conflicts Resolution
Intrukation
Resche
System overview of the KARMA multi-agent architecture. Each agent LLM-driven module tasked with specific roles such as ingestion, summarization, entity recognition, relationship extraction, conflict resolution, and final evaluation.
Figure 2:
improves schema alignment on Wikidata benchmarks. maintaining linear time complexity relative to input text length.
KARMA synthesizes insights from these research threads while introducing key innovations: (1) modular, multi-agent architecture that allows for specialized handling of complex tasks in knowledge graph enrichment, (2) domain- adaatiiv prompting strategies that enable more accurate extraction across diverse scientific fields, (3) LLM- based verification mechanisms that mitigate issues such as hallucination and schema inconsistency.
3 Methodology
In this section, we introduce KARMA, a hierarchical multi-agent system (see Figure 2 that leverages specialized LLMs perform end-to-end KG enrichment. Our approach decomposes the overall task into modular sub-tasks, ranging from document ingestion to final KG integration, each handled by an independent LLM-based agent. We first present formal problem formulation and then detail the design and mathematical foundations of each agent within the pipeline.
3.1 Problem Formulation
Let G = (V, E) denote an existing KG, where V is the set of entities (e.g., genes, diseases, drugs) and E the set of directed edges representing relationships. Each relationship is defined as a triplet l (ea, r, ee)) with ej E V and specifying the relation type (e.g., treats, causes). We are provided with corpus of unstructured publications P P1..... Pa. The objective automatically
extract novel triplets t of E from each document and integrate them into G to form an augmented graph Gnow.
Gnew = G U JC, where K,, Extracc
(I)
where (ps) is the set of valid triplets obtained from publication ps- To maintain consistency and accuracy, each candidate triplet is evaluated by LLM-based verifier prior to integration.

---

3.2 System Overview
KARMA comprises multiple LLM-based agents operating in parallel under the orchestration of Central Controller. Each agent uses specialized prompts, to optimize its performance. In KARMA, we define set of agents (B):
and domain knowledge
Ingestion Agents (IA): Retrieve and normalize input documents B.3
Reader Agents (RA): Parse and segment relevant text sections B.4)
Summarizer Agents (SA): Condense relevant sections into shorter domain-specifte summaries (B.5
Entity Extraction Agents (EEA): Identify and normalize topic-related entities (B.6).
Relationship Extraction Agents (REA): Infer relationships between entities (B.7 7.
Schema Alignment Agents (SAA): Align entities and relations to KG schemas (B.8).
Conflict Resolution Agents (CRA): Detect and resolve logical inconsistencies with existing knowl-
edge (B.9)
Evaluator Agents (EA): Aggregate multiple verification signals and decide final integration
3.3
Ingestion Agents (IA)
The Ingestion Agents are LLM-based modules specialized in document retrieval, format normaliza- tion, and metadata extraction. Let p; be a raw publication. IA includes:
IA(p:) [uormalize(ps), metadata(p:)).
where normalizo(p;) uses an LLM prompt Pingest to handle complexities like OCR errors, or structural inconsistencies. The output is standardized textual representation plus key metadata (journal, date, authors, etc.). This representation is then placed into a data queue for Reader Agents.
3.4 Reader Agents (RA)
Reader Agents parse normalized text into coherent segments (abstract, methods, results, ect.) and filter irrelevant content. Let pl the normalized document, RA splits pi into {si, 82, 8my Each segment sj is assigned relevance score R(s;) by:
R(s;) LLM ($,.9).
(3)
where LLM reader is prompted with domain-specific instructions to assess the segment's biomedical significance relative to the current KG G. RA discards segments if R(s,) < ó, where is domain- calibrated threshold. Surviving segments are passed along to Summarizer Agents.
3.5 Summarizer Agents (SA)
To reduce computational overhead, each RA segment sj is condensed by Summarizer Agents into concise representation uy. Formally, we define:
Mg LLMaumm Pamm) Puumm)
(4)
where Psumm is a prompt for LLM to retain critical entities, relations, and domain -specfi terms. This summarization ensures Entity Extraction Agents and Relationship Extraction Agents receive textual inputs that are both high-signal and low-noise.
3.6 Entity Extraction Agents (EEA)
LLM-Based NER. Each summary uy is routed to an LLM-based NER pipeline that identifies mentions of tepic-related entities. Define:
E(u,) LLM& (u,, PE) @ DE.
(5)
where LLM is an specialized entity-extraction LLM with prompt Pr. and @ Dr indicates dictionarylontology--aee filtering. This step filters out false positives and normalizes entity mentions to canonical forms (e.g., mapping "acetylsalicylic acid" to "Aspirin")..

---

Entity Normalization. Let e be raw entity mentioned from E(u,). We map to a normalized entity E V by minimizing a distance function in a joint embedding space:
è arg min d(((e)) w(u)).
(6)
vev
where @ maps textual mentions to embeddings (using, e.g., a BERT-based model), and y maps known KG entities to the same embedding space. The distance metric d(.;' can be cosine distance or domain-spn domain-specific measure. Any entity with min,ev d(p(e), (D)) > p is flagged as new and added to the set of candidate vertices V+.
3.7 Relationship Extraction Agents (REA)
After entity normalization, each pair (ê,, êş) within summary is fed to an LLM-based classifier:
p(r Ej, (y, uy) LLMm((e, (e, Pn),
(7)
where p(rl-) is the probability distribution possible relationships {ri,..., rk). The prompt PR instructs the LLM to focus on domain relationship candidates. We select any relationship r for which p(rlê,, é,) 0, and form triplet (ê,, r, ē2). In certain passages, more than one relationship can be implied. We allow multi-label predictions by setting an indicator variable:
I(r) = I(p(r é,, ε2) 2 0r).
(8)
Hence, Ru,,) the set of triplets (ê,, r, é5) such that I(r) =
3.8 Schema Alignment Agents (SAA)
If a new entity V or a new relation does not match existing KG types, the Schema Alignment Agent performs a domain-speciff classification. For entities, the SAA solves:
max LLMSAA (v, Patign). TET
(9)
where T is the set of valid entity types (Disease, Drug, Gene, etc.), and LLMşAA estimates the probability that belongs to type A similar approach is used for mapping new relation r to known KG relation types. If no suitable match exists, the SAA flags v or r as candidate additions for review.
3.9 Conflict Resolution Agents (CRA)
New triplets can contradict previously established relationships. Let t T,,,T,& êt) be a newly extracted triplet, and let t' = (êh, r', é.) be conflicting triplet in if r is logically incompatible with r'. We define:
conflict(1,g)) = {b)
3t' that contradicts t, otherwise.
(10)
The CRA uses an LLM- debate prompt:
LLMeRA (t, - (Agree, Contradict],
(1I)
LLMcRA yields Contradict, is then discarded or queued for manual expert review, depending on the system's confidence.
3.10 Evaluator Agents (EA)
Finally, the Evaluator Agents aggregate multiple verification signals and compute global confidence C(t), clarity Cl((t,, and relevance R(t) for each triplet t.
Confidence:
C(t) a Σ Eaau(()):
(12)
Clarity: C((() = a (Σ (3,((())
(13)
Relevance:
R(L) a Σ (urr(()).
(14)
5

---

where (xx) = Ite- and {a.. 3,: 7 reflect the trustworthiness of each verification source, and Cj, are verification signals for confidence, clarity, and relevance respectively. We finalize for integration using the mean score:
integrate(t) = 1. if C()+C0++ 2 θ
(15)
otherwise.
Altogether, this multi-agent pipeline, fully powered by specialized LLMs each stage, enables robust, scalable, and accurate enrichment of large-scale KG. Future extensions can easily incorporate new domain ontologies, additional specialized agents, or updated LLM prompts as tasks continues to evolve.
4 Experimental Setup
This section presents comprehensive proof-of-concepl evaluation settings of the proposed KARMA framework. Unlike conventional NLP tasks that rely on gold-standard dataset of biomedical entities and relationships. evaluation adopts multi-facete approach. We integrate LLM-based verification with specialized graph-level metrics to assess the quality of the generated knowledge graph. The evaluation spans genomics, proteomics, and metabolomics, showcasing KARMA's adaptability across diverse biomedical domains.
4.1 Data Collection
We curate scientific publications from PubMed
277 across three primary domains: the Genomics Corpus, which includes 720 papers focused on gene variants, regulatory elements, and sequencing studies; the Proteomics Corpus, comprising 360 papers related to protein structures, functions, and protein-interaction networks; and the Metabolomics Corpus, containing 120 papers discussing metabolic pathways, metabolite profiling, and clinical applications. All articles are stored in PDF format and processed by the Ingestion Agent within KARMA.
4.2 LLM Backbones
We evaluate three general-purpont LLMs as the backbone for KARMA's multi-agent knowledge graph enrichment pipeline using their APIs.
GLM-4 7
An open-source 9B-parameter model, achieving 72.4 on the MMLU NLP benchmark.
GPT-4o I
A proprietary multimodal model optimized through RLHE. It has demonstrated strong
adaptability in scientific knowledge extraction and concept grounding 4
DeepSeek-v3 15]: :: An open-source 37-billion-activateed parammeer mixture-of-experts (MoE) model with strong focus on STEM domains.
Each KARMA agent (e.g., Reader, Summarizer, Extractor) shares the same LLM backbone per experiment, All LLM-based evaluations employ DeepSeek-v3. Prompting strategies, detailed in Appendix B are minimally modified to ensure comparability across LLMs and domains. We analyze variations in the final constructed knowledge graph based on different LIM backbones.
4.3 Metrics
To evaluate the enriched knowledge graph (KG) in the absence of a gold-standard reference, we employ multi-faceted evaluation framework that assesses structural integrity, correctness, and practical utility. This framework comprises three categories of metrics: core metrics, graph statistics, and quality indicators. Together, these metries provide comprehensive insights into the quality and usability of newly added triples and the overall augmented KG.
Core Metrics focus the properties of newly added triples using structural and LLM-base indicators. The Average Confidence (Mcon) measures the mean confidence scores across all new triples, reflecting their reliability. The Average Clarity (M()) computes the mean clarity scores, indicating how unambiguous or direct each relation is. The Average Relevance (M (et) captures the

---

Table 1: KARMA evaluation metrics across domains and models. M Average confidence score, Meri Average clarity score, M Ret Average relevance score, A Coverage gain, A Con' Connectivity gain, R Conflict ratio, L' LLM- correctness score, CoA QA coherence score, R HE: Human evaluation score. Best performance in each domain highlighted.
Core Metries
Graph Stats.
Quality Indicators
Domain
Model
MC..
Me.
M
A
A
PCR
BT.c
C²A
THE
Single-Agent
NA
NA
NA
4384
1.083
NA
0.493
0.472
0.320
GLM-4
0.729
0.804
0.716
4969
1.131
0.238
0.623
0.589
0.445
Genomics
GPT-4o
0.843
0.744
0.640
9795
1.265
0.148
0.880
0.569
0.510
DeepSeck-v3 v3
0,846
0.754
0.667
38230
1.765
0.186
0.831
0.612
0.625
Single-Agent GLM-4
NA
NA
NA
5002
1.150
NA
0.638
0.572
0.415
0.731
0.752
0.609
6832
1.173
0.214
0.720
0.617
0.500
Proteomics
GPT-4o
0.823
0.797
0.613
7008
1.191
0.160
0.740
0.612
0.550
DeepSeek-v3
0.845
0.825
0.682
11936
1.468
0.151
0.772
0.613
0.575
Single-Agent
NA
NA
NA
485
1.077
NA
0.527
0.450
0.455
GL.M-
0.701
0.790
0.762
703
1.159
0.188
0.617
0.449
0.485
Metabolomics
GPT-4o
0.802
0.730
0.726
773
1.143
0.147
0.683
0.482
0.535
DeepSeek-v3
0.790
0.746
0.767
1752
1.811
0.132
0.668
0.493
0.580
mean relevance scores, assessing the domain significance the triples. These metrics collectively evaluate the intrinsic quality of the added knowledge.
Graph Statistics quantify the structural properties of the augmented KG. The Coverage Gain (A Coo measures the number of newly introduced entities not previously in the KG, reflecting its expanded scope. The Connectivity Gain (A Com) calculates the net increase in node degrees (summed over existing entities), indicating enhanced interconnectedness.
Quality Indicators assess reliability and usability through multiple lenses. The Conflict Ratio (ROR) represents the fraction newly extracted edges removed by the ConflerResooltiionAgen due to internal or external contradictions. The LLM-based Correctness (R) is determined by a hold-out LLM judging each new triple (Chead, r, tail)) as likely correct, uncertain, likely incorrect, with # (likely comect)
RLC = #(al new triples.. : The Question-Answa Coherence (C²a) evaluates the fraction plausible KG-derived answers for a curated set of domain-specific questions answerable via KG traversal. Finally, the Human Evaluation Score (R scaled from 0 to I, gauges the quality of triple extractions based on assessments by two human experts, offering a comprehensive measure of the knowledge graph's 's accuracy and utility.
These complementary metrics provide insights into the structural integrity, internal consistency, correctness, and practical utility of the enriched knowledge graph.
Results
5.1 Overall Evaluation
demonstrates that KARMA significantly extends domain- specific knowledge graphs through its multi- -agentt architecture. Four key findings emerge: (1) The framework demonstrates superior performance compared the GLM-4-based single-agent approach, which extracts all triples in a single generation, (2) The framework exhibits varying performance across distinct domains; identifies the most entities in prevalent fields such as genomics (53. 1/article), achieving 3.6x higher coverage gain (Acov) per article than metabolomics (14.6/article); (3) L.LM backbone selection substantially impacts KG
Our comprehensive evaluation (Table ! with examples in Appendix C.I
quality, with DeepSeek-v3 achieving superior performance on 17 /24 (71%) metrics across domains;
7

---

Prompt Tokens
Completion Tokens
Processing Time
x10
x10
x10
15.0
3.0
10.0
1.0
2.0
5.0
0.5
1.0
0.0
0.0
Gen.
Prote. Metab.
Gen.
Prote. Metab.
Gen. Prote. Metab.
Figure Comparison prompt tokens, completion tokens, and processing time across domains.
(4) Evaluating knowledge and resolving conflicts automatically enhance the quality of the extracted knowledge graph, improving L...-based accuracy by 4.6%-14.4%.
5.2 Domain-Level Observations.
Genomics:
Scale Meets Precision C.1 The genomies domain (720 papers) exhibits the most
pronounced model differentiation. DeepSeek-v3 achieves ACo = 38, 230 while maintaining competitive correctness score RtC 0.831, only 5.6% below GPT-4o's peak. This suggests that MoE architectures can balance recall and precision in large-scale extraction.
Proteomics: Balanced Optimization C.2
With 360 papers, proteomics reveals balanced gains: DeepSeek-v3 leads both core metrics (Mcon = 0.845) and structural gains (Acon = 1.468), while
GLM-4 achieves peak QA coherence (CQA 0.617). The 19.1% higher ACou for DeepSeek-v3 versus GPT-4o indicates greater sensitivity to protein interaction nuances.
Metabolomics: Specialization Pays Off CC.3 Despite the smallest corpus (120 papers), GLM-4
delivers superior clarity (Meta = 0.790) and GP T-4o excels in correctness (RıC 0.683). However, DeepSeek-v3* ACon 1, 752 is 127% higher than GPT-4o, demonstrates unique capability to extrapolate metabolic pathways from limited data.
5.3 Analysis of LLM Backbones
Our comparison reveals strengths of different backbones: DeepSeek-v3 drives unparalleled coverage gains, outpacing GPT-4o by 3.9x genomics and 2.3x in metabolomics while maintaining com- petitive correctness (Ric 0.831 vs GPT-4o's 0.880 in genomics). This contrasts with GPT-4o's precision-fiis first profile, where achieves peak RLC scores (0. (0.880 genomics, 0.740 proteomics) but yields 41% lower connectivity gains than DeepSeek-v3, reflecting underutilized implicit relationships.
GLM-4, though smaller (10B parameters), demonstrates domain-specific prowess: its biomedical tuning delivers best-in-class metabolomics clarity (Mcta 0.762) and proteomies QA coherence (Cq4 = 0.617), while its conflict ratio (RcR = 0. 1888 remains competitive despite lower parameter count. The tradeoffs (DeepSeek-v3's coverage balance for correctness, GPT-4o'ss precision sacrifice for completeness, GLM-4's niche adaptation) underscore why KARMA's 's multi-agent framework strategically decouples extraction, validation, and can utilize the strengths of each backbone. Different backbones also lead to variations in the distribution of key evaluation metrics (Figure C.
[.
5.4 Cost Analysis
The evaluation computational costs (Figure 3) demonstrates distinct trade-off's in token usage and processing time across different domains. The variations in article lengths and information density naturally lead differences token consumption and processing times. Notably, genomics shows higher completion token distributions (mean 550.64, std 232.92), explaining KARMA's higher ACon in this domain. Meanwhile, proteomics exhibits broader processing time distributions (mean 96.58, std 46.90), which correlates with its stronger performance in knowledge quality metrics (Ric and CQ4), suggesting that longer processing times contribute to more thorough relationship analysis and validation.

---

Table 2: Ablation study results for KARMA, evaluating the impact of different agents (Summarizer, Conflict Resolution, Evaluator).
Quality Indicators
Domain
Configuration
R
R
R
w/o Summarizer Agent
0.758
0.788
0.472
0.600
w/o Conflict Resolution Agent
0.790
0.733
0.554
0.485
Genomies
w/o Evaluator Agent
0.793
0.752
0.561
0.515
KARMA-Full
0.831
0.831
0.612
0.625
w/o Summarizer Agent
0.632
0.759
0.547
0.610
w/o Conflict Resolution Agent
0.661
0.742
0.583
0.525
Proteomics
w/o Evaluator Agent
0.696
0.755
0.605
0.580
KARMA-Full
0.772
0.772
0.613
0.625
w/o Summarizer Agent
0.577
0.592
0.537
0.555
w/o Conflict Resolution Agent
0,629
0.551
0.471
0.545
Metabolomics
w/o Evaluator Agent
0.603
0.572
0.480
0.550
KARMA-Full
0.668
0.608
0.493
0.580
5.5
Ablation Study
To better quantify the contributions of each specialized agent in KARMA, we conduct an ablation study (Table(2) 2 by systematically removing replacing selected agents and measure the resulting performance across the three domains. Specifically, we evaluate:
KARMA-Full: All agents active, including Summarizer, Conflict Resolution, and Evaluator modules.
w/o Summarizer:
Bypasses the Summarizer Agents, passing all text directly from Reader Agents to Entity and Relationship Extraction.
w/o Conffict Resolution: Disables the Conflict Resolution Agent, allowing potentially contradictory
edges into the final graph.
w/o Evaluator: Omits the final confidence, clarity, and relevance evaluation and aggregation, integrating relationships without filtering.
We conduct these ablations using the same LLM backbone (DeepSeek-v3 in our experiments) for consistency. Table 2 summarizes the impact evaluation metrics (Rıc, CQa) for each domain. The ablation study highlights the importance of each agent KARMA's performance. Removing the Summarizer Agent produce much more entities and triples, but reduces accuracy (CqA drop 22.9% (0.612 + 0.472) in genomics) and coherence (Ri. drop 18.2% (0.772 - 0.632) in proteomics), unfiltered text introduces noise. Disabling the Conflict Resolution Agen/ significantly lowers correctness (CQa drop 4.9% (0.831 - 0.790) in genomics), especially in resolving contradictions like conflicting gene-disease associations. Omitting the Evaluator Agents has the most impact on usability, as unfiltered, low-confidence edges degrade answer quality (RLc drop 9.7% (0.668 - 0.603) in metabolomics). Across all domains, conflict resolution proves critical for maintaining logical consistency, while summarization and evaluation ensure focused extraction and high-quality integration. This demonstrates that KARMA's multi-agent design essential for balancing accuracy, consistency, and usability in KG enrichment.
6 Conclusion
We introduce KARMA, multi-agent LLM framework designed to tackle the challenge of scalable knowledge graph enrichment from scientific literature. By decomposing the extraction process into specialized agents for entity discovery, relationship validation, and conflict resolution, KARMA ensures adaptive and accurate knowledge integration. Its modular design reduces the impact of conflicting edges through multi-layered assessments and cross-agent verification. Experimental results across genomics, proteomics, and metabolomics demonstrate that multi-agent collaboration can overcome limitations of single-agent approaches, particularly in domains that require complex semantic understanding and adherence to structured schemas.

---

Acknowledge
This research supported by National Key Research and Development Program of China (2024FFF0507400) and National Natural Science Foundation of China (6220071694).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
[2] Lutz Bormmann, Robin Haunschild, and Rüdiger Mutz. Growth rates of modern science: latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1):1-15, 2021.
[3] AMBR Carvalho, Daniel da Silva de Paiva, JS d Sichman, JLT da Silva, RS Wazlawick, and VLS de Lima. Multi-agent systems for natural language processing. In Proceedings of the Second Iberoamerican Workshop Distributed Artificial Intelligence and Multi-agent Systems. Citeseer, 1998.
[4] John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S Rosen, Gerbrand Ceder, Kristin A Persson, and Anubhav Jain. Structured information extraction from scientific text with large language models. Nature Communications, 15(1):1418, 2024.
[5] Jérôme Euzenat, Pavel Shvaiko, et al. Ontology matching, volume 18. Springer, 2007.
[6] Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: generalist multi-agent system for solving complex tasks. arXiv preprint arXiv:2411.00 1,04468, 2024.
[7] Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, al. Chatglm: family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406. 122i::3300..2793, 2024.
[8] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.016880 2024.
[9] Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d' l'Amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al. Knowledge graphs. ACM Computing Surveys (Csur), 54(4):1-37, 2021.
[10] Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899, 2018
[11] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs:
Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494 2021.
[12] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-124) 2020.
[13] Johannes Lehmann and Markus Kleber. The contentious nature of soil organic matter. Nature, 528(7580):60-68, 2015.
[14] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang, Yujiu Yang. Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint arXiv:2305. /9/ 18, 2023.
[15] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arX1v:2412.19437, 2024.
10

---

[16] Hugo Liu and Push Singh. Conceptnet practical commonsense reasoning tool-kit. BT technology journal. 22(4):211-226. 2004.
[17] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative contrastive. IEEE transactions knowledge and data engineeringg 35(1):857-88) 876, 2021.
[18] Yuxing Lu, Sin Yee Goi, Xukai Zhao, and Jinzhuo Wang. Biomedical knowledge graph: A survey of domains, tasks, and real-world applications. arXiv preprint arXiv:2501.11662 2025.
[19] Yuxing Lu, Xukai Zhao, and Jinzhuo Wang. Clinicalrag: Enhancing clinical decision support through heterogeneous knowledge retrieval. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 64-68, 2024.
[20] Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: 7ero-resource black-bon hallucination detection for generative large language models. arXiv preprint arXiv:2303.08886, 2023.
[21] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39 41, 1995.
[22] Zara Nasar, Syed Waqar Jaffry, and Muhammad Kamran Malik. Information extraction from scientific articles: a survey. Scientometrics, 117(3)1931-1999, 2018.
[23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-213 27744, 2022.
[24] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions Knowledge and Data Engineering, 2024.
[25] Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent Ilm agents. arXiv preprint arXiv:2306.033: 033314 2023.
[26] Denny Vrandečić and Markus Krötzsch. Wikidata: free collaborative knowledgebase. Com- munications of the ACM, 57(10))78-85, 2014.
[27] Jacob White. Pubmed 2.0. Medical reference services quarterly, 39(4):382-335 2020.
[28] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen 1lm applications via multi-agent conversation framework. arXiv preprint arXiv:23088 20ti::2230888555555 08155, 2023.
[29] Qi Zeng. Consistent and efficient long document understanding. PhD thesis, University of Illinois at Urbana- Oobana-Champaign, 2023.
[30] Yuan Zhang, Xin Sui, Feng Pan, Kaixian Yu, Keqiao Li, Shubo Tian, Arslan Frdengasileng, Qing Han, Wanjing Wang, Jianan Wang, et al. Biokg: comprehensive, large-scale biomedical knowledge graph for ai-powered, data-driven biomedical research. bioRxiv, 2023.
[31] Yuqi Zhu, Xiaohan Wang. Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang. Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities. World Wide Web, 27(5):58, 2024.
11

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions: the KARMA framework, a multi-agent LLM system for automated KG enrichment, its architecture with nine collaborative agents, and experimental validation on PubMed articles demonstrating entity identification, correctness, and conflict reduction. These claims are consistent with the methods and results presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper includes dedicated "I "Limitations" section in the Appendix (Section A). This section discusses reliance L--M-based metrics over direct human expert valida- tion, performance variations across domains, and suggests areas for future improvement.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
12

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper proposes a framework (KARMA) and presents experimental results.
It does not include new theoretical results, theorems, or formal proof's that would require dedicated section for assumptions and proofs. The mathematical formulations describe the system's 's operations rather than novel theoretical derivations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper details the KARMA framework's 's architecture (Figures & 2), the roles of each agent (Section 3.2), the LLM backbones used (Section 4.2), data collection from PubMed (Section 4.1), and the metrics for evaluation (Section 4.3). Appendix references for prompts (B) are mentioned.
Justification:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
13

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper has deposit the code in Supplementary Materials, and will open- source the code once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wublic//uiies/CodeSutin www.abmmissiooooioipp.io for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
chajps.cc/puulic/uuiiee guw.///ieee//eee/ Codeeubmii ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper specifies the LLM backbones used (Section 4.2), the data collection process and corpora sizes (Section 4),,, and mentions that prompting strategies detailed in Appendix(Bi It also mentions that "Each KARMA agent...saress the same LLM backbone per experiment" and "All LLM-based evaluations employ DeepSeek-v3". The method section details the roles and inputs/outputs of each agent.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
14

---

Justification: Table reports various metrics, including average scores. The cost analysis in Figure 3 shows distributions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Section 4.4 "Cost Analysis" and Figure discuss prompt tokens, completion tokens, and processing time across domains. It provides information token usage and processing time, which are key resource indicators for LLM-based experiments. It also mentions using APIs for LLMs, implying cloud-based resources for those.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neeur//
Answer: [Yes]
Justification: The research focuses on automating knowledge graph enrichment from scien- tific literature. The paper includes discussion potential ethical impacts concerning bias in LLMs and data privacy, and mentions human oversight.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
15

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification;
The paper's 's conclusion briefly mentions that KARMA can "transformative tool for advancing knowledge". The commented-out "Ethical Impact" section discusses potential negative impacts such as bias in LLMs leading to incorrect associations and data privacy concerns, and also suggests mitigation through human oversight.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deg.oyeemm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper proposes a framework (KARMA) that uses existing LLMs. It does not release new pretrained language models large, scraped datasets that would pose high risk for misuse themselves.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
16

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper credits the creators of the LLMS used (e.g., GLM-4 77., GPT-4o D.
DeepSeek-v3
[55
and datasets like PubMed 27 through citations.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi paperssithcode thcode com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The primary new asset is the KARMA framework itself. The paper provides a detailed description of its architecture, components (agents), and workflow in Section 3 and Figures & However, does not release new datasets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The paper includes Human Evaluation Score which involved assessments by two human experts (Section 4.3). But no research with human subjects was performed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
17

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: There is such research with human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLMs are used only for editing the paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M for what should or should not be described.
18

---

Appendix
A Limitations
Despite the promising performance of KARMA, several limitations remain. First, our evaluation relies primarily on LLM-based metries rather than direct human expert validation. While we employ multi-faceted metrics (e.g., QA coherence, conflict resolution) to assess the quality of the extracted knowledge, we recognize that domain experts must ultimately verify critical biomedical claims before applying them in clinical settings. Furthermore, performance varies across domains; for instance, metabolomics shows 12.4% and 11.9% lower QA coherence than proteomics and genomics, respectively, indicating challenges in modeling sparse and rare relationships in this field. These limitations highlight opportunities for future improvements, such as integrating hybrid neuro- neuro-symbollic approaches and optimizing agent coordination protocols.
B Detailed propmts for KARMA agents
This appendix provides example prompts for each agent in the KARMA framework. All agents operate via LLMs with specialized prompt templates. We emphasize confidence, clarity, and domain relevance. Where applicable, we include sample inputs, outputs, and negative examples to illustrate how each agent handles complexities in the context.
B.1 Function summaries of different agents
The KARMA framework comprises nine specialized LLM --oeeed agents, each handling distinet stages of the knowledge extraction and integration task. Below are their core functions:
Ingestion Agents (IA): Retrieve raw documents (PDF/HTML), normalize text (handling
OCR errors, tables), and extract metadata (authors, journal, publication date).
Reader Agents (RA): Split documents into sections, score segment relevance using KG context, and filter nev-revelant content (e.g., acknowledgments))
Summarizer Agents (SA): Condense text segments into concise summaries while preserv- ing entity relationships (e.g.. "Drug X inhibits Protein Y, reducing Disease Z symptoms" + "X inhibits Y linked to Z").
Entity Extraction Agents (EEA): Identify entities via few-shot LLM prompts, then nor- malize them to KG canonical forms using ontology-guided embedding alignment.
Relationship Extraction Agents (REA): Detect relationships (e.g.. treats, causes) between entity pairs using multi-label classification, allowing overlapping relations (c.g.. "Drug A both inhibits Protein B and triggers Side Effect C").
Schema Aligmment Agents (SAA): Map novel entities/relations to KG schema types (e.g.. classifyyng (RISPR-CCss" as Gene-Eon Sene-EEiting Tool) or flag them for ontology expansion.
Conflict Resolution Agents (CRA): Resolve contradictions (e.g., new triplet "Drug D treats Disease vs. existing "Drug D exacerbates Disease E") via LLM debate and evidence aggregation.
Evaluator Agents (EA): Compute integration confidence using weighted signals (confi- dence, relevance, clarity) and apply threshold-bace based final approval.
B.2 Additional Notes on Prompt Engineering
1. Example-based prompting (few-shot).
In practice, each agent's prompt can be extended with short examples of input-output pairs provide the LLM with more context, thereby improving the accuracy and consistency of its responses. For instance, the EEA prompt might include examples of drug-disease pairs, while the CRA prompt might illustrate how to handle partial contradictions vs. direct contradictions.
2. Negative Examples and Error Correction.
19

---

To increase robustness, each agent can be provided with negative examples or clarifications on error arror-poone cases, For example, the Summarizer Agent might be shown how not to remove important numerical dosage information; the EEA might have a demonstration of ignoring location references that are not topic-related entities (e.g., "Paris" not a Di Disasse..
Incremental Fine-Tuning and Updates.
As knowledge evolves, so do the vocabularies and relationship types. Agents can be periodically re-trained or their prompts updated to handle newly emerging entities (e.g., novel viruses, new drug classes) and complex multi-modal relationships. The modular structure of the prompts eases integration these updates without redesigning the entire pipeline.
Collectively, these prompts enable KARMA to harness LLMs at every stage of the knowledge extrac- tion and integration process, resulting in dynamic, scalable, and accurate knowledge enrichment.
20

---

B.3 Ingestion Agent (IA) Prompt
Title: IA ArPrrppt
Role Description: You are a clinical expert. Your responsibility is to: Retrieve raw publications from designated sources (e.g.. PubMed, internal repositories). 2. Convert various file formats (PDF, HTML, XML) into a consistent normalized text format. 3, Extract metadata such the title, authors, how.oorssoooumm/oooeeeeeee name, publication date, and unique identifiers (DOI, PubMed ID).
System Instruction:
Input: Raw document payload or path/URL to the document, plus minimal metadata (if available).
Output: A JSON structure with main fields:
1. "metadata": (title, authors, journal, pub_date, doi, pmid, etc.)
2. "content": A single string structured array containing the full text, preserving headings or major sections if possible.
Key Requirements:
Handle OCR artifacts the PDF is scanned (e.g., correct typical OCR errors where possible).
Normalize non-ASCII characters (greek letters, special symbols) to ASCII or minimal LaTeX markup when relevant (e.g., \alpha).
certain fields cannot extracted, leave them as empty or "N/A" but do not remove the key from the ISON.
Error Handling:
In case of partial or unreadable text, mark the corrupted portions with place- holders (e.g.. "[UNREADABLE").
the document locked or inaccessible, set an error flag the output ISON.
LLM Prompt Template (Illustrative Example):
[System Role: IngestionAgent]
You will receive raw publication in PDF HTML format. Extract all available metadata: Title, Authors, Date, Journal/Source, PMID, DOI.
1.
2. Convert the text to ASCII or minimal LaTeX. Provide JSON output with keys: ("metadata": {..., "content";
"..."
any portion of the text is unreadable, replace it with [[URREADABLE]*
Sample Input: pdf_document: "Binary data...", doi: "10. 1000/0 jmbb 2022.07. 123"
Sample Output:
"metadata": {"title": "Novel Anti-viral Therapy", "authors": [" ["Jane
Doe
"content": "Introduction
n Recent advances in... Methods
n We tested..."
21

---

B.4 Reader Agent (RA) Prompt
Title: RA Prompt
Role Description: You are the Reader Agent. Your goal to parse the normalized text from the IA and generate logical segments (e.g., paragraph-level chunks) that are likely to contain relevant knowledge. Each segment must accompanied by a numeric Relevance Score indicating its importance for downstream extraction tasks.
System Instruction:
Input:
JSON output from IA with "metadata" and " "content" fields.
Output: A JSON array "segments", where each element is ("text": "...", "score" 0.xx]].
Scoring Heuristics:
Use domain knowledge (e.g.. presence of known keywords, synonyms, known entity patterns) to increase the score.
Use structural cues (e.g.. headings like "Results", "Discussion" might have higher relevance for new discoveries).
a segment is purely methodological (c.g., protocols or references to equip- ment) with
knowledge, assign lower score.
Edge Cases:
Very short segments (<30 characters) or references sections might be assigned minimal score.
If certain sections are incomplete or corrupted, still generate a segment but label it with "score": 0.0.
LLM Prompt Template (llustrativve Example):
[System Role: ReaderAgent]]
Given the JSON with metadata and a large text string under "content", split the text into smaller segments (e.g., paragraphs).
For each segment, estimate a Relevance Score (0 to 1) that indicates the likelihood of containing novel relationships or key findings. Output a JSON array "segments"; [["text":" "score": 0.XX), ...].
Sample Input:
"metadata": ("title": "Antimicrobial Study"...
"content": "Abstract n tested new...
Methods
n The protocol was...
n"
Sample Output:
("segments":
["text": "Abstract We tested new...", "score": 0.85}, ("text": "Methods The protocol was...", "score": 0.30}]
22

---

B.5 Summarizer Agent (SA) Prompt
Title: SA A_Prrmpt
Role Description: You are the Summarizer Agent. Your task is to convert high-relevance segments into concise summaries while retaining technical detail such as gene symbols, chemical names, or numeric data that may be crucial for entity/relationship extraction.
System Instruction:
Input: A set of segments, each with a relevance score (e.g., from the RA).
Output: A JSON arryy "summaries" each entry with:
"original_text" the original segment
2. "summary": concise, domain-specific summary (2-4 sentences recom- mended)
3. "score": the inherited slightly adjusted relevance score
Summarization Rules:
Avoid discarding comain-speciffi terms that could indicate potential relation- ships. For example, retain "IL-6" or "p53" references precisely.
numeric data relevant (c.g., concentrations, p-values), incorporate them verbatim if possible.
Keep the summary length under 100 words to reduce computational overhead for downstream agents.
Handling Irrelevant Segments:
If the Relevance Score is below a threshold (e.g., 0.2), you may skip or heavily compress the summary.
Mark extremely low relevance segments with "summary": "[OMITTED] " if not summarizable.
LLM Prompt Template (Illustrative Example): [System Role: SummarizerAgent]
For each segment with (text, score), produce a summary capturing key biomedical elements (drugs, diseases, molecular targets).
Preserve numeric data or specific chemical/gene names. Output list:
("summaries"; ['"original_text" "...", "summary": "...", "score":
0.xx), ...]}.
Sample Input:
"segments": [
{"text": : "In this study, IL-6 blockade ...", "score": 0.90},
{"text"; "The control group had p=0.01... "score": 0.75}]
Sample Output:
"summaries":
"erriginal_text::"" this study, IL-6 blockade
"summary": "This "This segment describes the use of IL-6 inhibition in ...",
"score": 0.90},
("original_tae stalllteett::t "The control group had p=0.01...",
"summary" "Researchers observed statistically significant difference (p=0.01) between ...., "score": 0.75]]
23

---

B.6 Entity Extraction Agent (EEA) Prompt
Title: EEA Prompt
Role Description: You are the Entity Extraction Agent. Based on summarized text, your
objective is to: 1. Identify biomedical entities (Disease, Drug, Gene, Protein, Chemical, etc.).
2. Link each mention a canonical ontology reference (e.g.. UMLS, MeSH, SNOMED CT).
System Instruction:
Input: A summarized text from the SA outputs.
Output: JSON "entities" array, where each element includes:
"mention". ": the exact substring from the
2. "type": c.g., "Drug", "Disease", "Gene", etc.
3. "normalized_ic" id": references such as "MESH:D001943"
"UMLS:C0004238"
or
LLM-driven NER:
Use domain-specitte knowledge to identify synonyms ("acetylsalicylic acid" 7 Aspirin).
Include multi-word expressions ("breast cancer" as single mention).
Handling Ambiguity:
If multiple ontology matches possible, list the top candidate plus short reason or partial mention of the second-best match.
If no suitable ontology reference is found, set "normalized_ic" id": "N/A" and keep the raw mention.
LLM Prompt Template (Illustrative Example): [System Role: EntityExtractorAgent
Identify all biomedical entities from the text snippet. Output array "entities": [["mention":".... "type".".... ...].
"aommalied_idd"......
Use domain ontologies (UMLS, MeSH, SNOMED) to map the mention to a canonical identifier if possible.
Sample Input:
"summary": "We tested Aspirin for headache relief at dosage of 100 mg."
Sample Output:
"entities"; [
("mention": "Aspirin", "type": "Drug", iermaalized_dd::
"MESH:DOOK DHEES:D001141"),
("mention": "headache", "type": "Disease", "normalized_id": "UMLS:C0018681"">
24

---

B.7 Relationship Extraction Agent (REA) Prompt
Title: REA Prompt
Role Description: You are the Relationship Extraction Agent. Given text snippet plus a set of recognized entities, your mission is to detect possible relationships (e.g., treats, causes, interactsWith, inhibits).
System Instruction:
Input: A summary Wj and a list of entity with normalized IDs from the EEA.
Output: A JSON array "relationships" where each element is:
"head": the head entity
2. "relation": the relationship type (string)
3. "tail": the tail entity
LLM-based Relation Classification:
Consider grammar structures (e.g.. "X observed to inhibit Y")) and domain patterns ("X reduces expression of Y"").
Allow multiple relationship candidates if the text is ambiguous suggests multiple interactions.
Negative Relation Handling:
If the text says "Aspirin does not treat migraine," the relationship (Aspirin, treats, migraine) is negative. Output either no relationship or a negative- labeled relationship (implementatiio-speccfff))
Recognize negation cues ("no effect", "absence of association"),
LLM Prompt Template (Illustrative Example):
[System Role: telaationnhipEtttacccr....g
You will receive text along with extracted entities. Determine if any pair of entities has a meaningful relationship. Use domain knowledge to find patterns like "X treats Y", "X inhibits Y", etc. Output each discovered relationship with "head", "relation", "tail", and "confidence".
Sample Input:
"summary": "Aspirin shown reduce headaches by inhibiting prostaglandin....
"entities"; [["mention": "Aspirin", "normalized_id":
"MESH:D001241")
("mention": "headaches", "normalized
("mention": "prostaglandin", "normalized_id":
Sample Output:
"relationships":
("head": DO01241,, "relation"; "treats", "tail":
"UMLS:C0018681"),
("head": "MESH:D001241", "relation"; "inhibits", "tail": DD11441]1
25

---

B.8 Schema Alignment Agent (SAA) Prompt
Title: MM_PPromp
Role Description: You are the Schema Alignment Agent. Newly extracted entities or relationships may not match existing KG classes or relation types. Your job is to determine how they should map onto the existing ontology or schema.
System Instruction:
Input: A list of new entities or relations that appear in the extraction but are not recognized in the current KG schema.
Output: An array "alig "alignments" with objects ("id":.... "type".... "status":..... possibly plus a "new_types" array for unrecognized patterns.
Ontology Reference:
For each unknown entity, propose a parent type from (Drug, Disease, Gene, Chemical. ...] if not in the KG.
For each unknown relation, map it to an existing relation if semantically close. Otherwise, propose new label.
Confidence Computation:
Consider lexical similarity, embedding distance, domain rules (e.g., if an entity ends with "-in" or "-ase", it might be protein or enzyme).
Provide final numeric score for how certain you are of the proposed alignment.
LLM Prompt Template (Illustrative Example):
[System Role: SchemalligumenttAgenn
You will receive a list of new entities/relations that are not in the KG. Try mapping them to existing node/edge types.
Output JSON: ("alignments": [("id" pyree
"status": "happed"/"ne""]]
Sample Input:
"unknown_ "iaknowm_entities"" ("TNF-alpha", "miR-21"], "mikoom_relationn"" ["overexpressos]]
Sample Output:
"aaligmments":
("id": "TNF-alpha", "proposed_typee "Protein", "status"; "mapped"),
{"id": "miR-21", tproposed_type" "RNA", "status"; "ne="]],
"new www.reelaionns::::
("relation": "overeepresssss"""" "closest_matchh "upregulates", "status": "new"]]
26

---

B.9 Conflict Resolution Agent (CRA) Prompt
Title: CRA Prompt
Role Description: You are the Conflict Resolution Agent. Sometimes new triplets are detected that contradict existing knowledge (e.g., (DrugX, causes, DiseaseY) vs. (DrugX, treats, DiseaseY)), Your role to classify these into Contradict, Agree, or Ambiguous, and decide whether the new triplet should be discarded, flagged for expert review, or integrated with caution.
System Instruction:
Input:
A new candidate triplet and a potentially conflicting triplet t' already in the KG.
Output: A JSON object with:
1. "decision": "Contradict", "Agree", or "Ambiguous"
2. "resolution": !"action":
"ratio-
nale": "..."}
LLM-based Debate:
Use domain knowledge to if relationships can coexist (e.g., inhibits activates are typically contradictory for the same target).
Coonsider partial contexts, e.g., different dosages subpopulations.
Escalation Criteria:
the new triplet has high confidence but conflicts with old data that has lower confidence, consider overriding or review.
If both are high confidence, label Contradict, prompt manual verification.
LLM Prompt Template (lllustrative Example): [System Role: ConfllictRso.utiinnngeettt You have two triplets t_new and t_existing that appear to conflict. Determine if they truly contradict or if they could be contextually compatible.
Output ("decision": "Contradict" / "Agree" "Ambiguous", "resolution" :aaetiiinn"""attttt {"action") :"accaon"""discard"""""""""""" "discard" / "review" "integrate", "rationale"."... "}}.
Sample Input:
"t "t_new": ("head" "DrugX""
:aaaation:""trattti "tail": "Disease""",,"""
"t "t_existingg: ("head": "DrugX", "relation": "causes", "tail": "Diisssse""sss"""""""""
Sample Output:
'"decision": "Contradict",
"resolution": ("action": "revieww, "rationale"; "Both have high conf confidence; manual verification required.
27

---

B.10 Evaluator Agent (EA) Prompt for Confidence
Title: EA AM__rrmpp__eee crofie_confidence
Role Description: You are the Evaluator Agent. After the extraction, alignment, and conflict resolution phases, each candidate triplet has multiple verification scores from external databases, additional LLM-based checks, or domain domain-specifii classifiers. Your duty is to aggregate these signals into final confidence score C()) and decide whether to integrate each triplet into the KG.
System Instruction:
Input:
A list of triplets, each with:
Partial confidence scores (c.g., U1, 02 ..., UN).
2. Conflict resolution status ("Contradict", "Agree", or 'AAmbigouus))
Output: A JSON array "final_triplets" with:
"head", "relation", "tail": identifiers for the triplet
2. "fimal__cofiiencce"" combined confidence score C(t)
Aggregation Formula:
You must also factor in conflict resolution outcomes: if Contradict, C(t) is penalized or forced to 0 unless manual override occurs.
LLM Prompt Template (lllustrative Example):
[System Role: EvaluatorAgent]
Given an array of triplets with partial scores [v1, v2, ...], conflict status, etc., compute final confidence using logistic weighting. Output as ("final_triplets", [("head":... :..., "relation":. "tail":.... "final_confiddnce":.... ...}.
Sample Input:
"candidates":
{"head"; "MESH: D001241", "relation": "treetts"
"scores": [0.90, 0.85], "conflict": "Agreee),
("head": "DrugX"" inelaatin"""caussetis "tail"""DiseaseY", "scores": [0.70, 0.60], "onfflcc""""ontrrant"
Sample Output:
"ffinal_triplett
("head": SHESH:D001220 "relation", "treats", "tail": "MMMS::00MMMS
"final_conffi confileccc :0.87),
("head" "Drugx", "relation" " "causes", "tail"; "Diseasee""
28

---

B.11 Evaluator Agent (EA) Prompt for Clarity
Title: EA Prompi_ccaaaae
Role Description: You are the Evaluator Agent responsible for assessing the clarity of each candidate triplet. After the initial extraction, some triplets may contain ambiguous terminology or uncertain references. Your job to assign a clarity score C((() to each triplet and decide whether it is sufficiently clear to be integrated into the Knowledge Graph (KG).
System Instruction:
Input: list of triplets, each with:
1. Partial clarity metries (e.g., C2, ... cN) obtained from lexical semantic checks.
2. A note on whether the triplet's terms or relation are ambiguous, e.g.. AumbiguousTerm", "ClearTerm",
Output: A JSON array "final_triplest" with:
1. "head", "relation", "tail": identifiers for the triplet.
2. "final_clarity". the combined clarity score Cl((t.
Aggregation Formula:
You may apply a weighted averaging or logistic function over Cı, 02,... CN.
Downweight or penalize triplets tagged as having ambiguous or unclear terms.
LLM Prompt Template (Illustrative Example):
[System Role: EvaluatorAgent]]
Given an array of triplets with partial clarity metrics [c1, c2, ...] and any notes ambiguity, compute final clarity score. Output as ("final_triplets": [("head":.... "relation":....
"tail":.... "final_ "final_clarity"........ ...]}.
Sample Input:
""andidatees":
("head": "DrugA", "relation": "may treat", "tail": "Condittoon",,, cclarity_metricst [0.80, 0.85], "ambiguous": "False"), ("head": "Entit"""E" "relation": "unknownRel", "tail": "EntityY"" "clarity_metrics"; [o.40], "ubiiguus""""rue"""""
Sample Output:
"finall_ "final_triples _triplets"; [
{"head": "DrugA", "relation": "relation"""mayy_reer"" "mayttraatt, "tail": "ConditionB", Frinal_clarity"O....
("head" "Entityy"," "relation"; "unknommel"" "tail"; "EntityY", FFinalclarityy::00
29

---

B.12 Evaluator Agent (EA) Prompt for Relevance
Title: EA Pa_oropp__ Prropptreeeeancee relevance
Role Description: You are the Evaluator Agent focusing on the relevance of each triplet the target Knowledge Graph (KG). Some triplets may be factually correct but not pertinent to the KG's domain or scope. Your duty to compute a relevance score R(tt) for each triplet and decide if should be included in the KG.
System Instruction:
Input: list of triplets, each with:
1. Partial relevance scores (c.g., f1, ".,.... TN) based domain-specine criteria (c.g., "medical relevance" or "chemical relevance").
2. Metadata or tags indicating alignment with the KG's domain (e.g.. "domainMatch" "domainMismacch")
Output: A JSON array "final_tripp "final_triplets" with:
1. "head", "relation", "tail".
2. "final_rrelevance" the combined relevance score R(t).
Aggregation Formula:
Combine the partial relevance scores via an average or logistic function.
Penalize triplets flagged as outside of the domain or referencing unknown entities.
LLM Prompt Template (Illustrative Example):
[System Role: EvaluatorAgent]]
Given array of triplets with partial relevance and domain tags, compute a final relevance score. Output ("final_triplets" "tail":.... "final_re rinaacerlevance":.... ...]}. .....
[ri, r2,
[("head":.. :..., "relation":...,
Sample Input:
"candidates":
("head": "DrugA", "relation" "used_ for", "tail" ; "DiseaseB"" "relevance_ [0.90, 0.88], "domainMatch" :tptaatcc""ttuue true), ("head" ""isttoiaallllliueee :
"tail": "Aaci""ennnineiacce "relevance_scores". [0], "domainMatch":faaled""
Sample Output:
"final_triplets" [
("head" "DrugA", "relation" "usee "tail": "DissssDDis,,,""",
("head": "istooriii"istrrlciiiuurr"""" "relation; "lived_in", "tail": "MaiillaanccietPlacace, "final_relevanne"00055))))
30

---

C
Examples of extracted knowledge graphs
C.1 Knowledge graph from Genomics articles (Generate using GPT-4o, Examples)
Genomics Knowledge Graph Triples
Key: Conf = Confidence, Rel = Relevance, Clr = Clarity
EGFR
lung adenocarcinoma (Conf: 0.75, Rel: 0.40, Clr: 0.60)
EGFR
non smmll cell lung cancer (Conf: 0.85, Rel: 0.30, Clr: 0.70)
T790M
lung adenocarcinoma (Conf: 0.98, Rel: 0.20, Clr: 0.70)
1790M
non
small cell lung cancer (Conf: 0.95, Rel: 0.20, Clr. 0.80)
EGFR
activotes
EG (Conf: 0.75, Rel: 0.20, Clr: 0.99)
EGFR Coises
proliferation (Conf: 0.85, Rel: 0.50, Clr: 0.60)
MTX
531
HNSCC (Conf: 0.99, Rel: 0.80, Clr: 0.50)
MTX 531
PDX (Conf: 0.75, Rel: 0.80, Clr: 0.50)
PIK3CA mutations
ssoociatd with
HNSCC (Conf: 0.85, Rel: 0.70, Clr. 0.99)
MTX 531
PIK3CA mutations (Conf: 0.65, Rel: 0.40, Clr: 0.99)
MCLA 158
Iargets
EGFR (Conf: 0.75, Rel: 0.30, Clr: 0.99)
EGFR
intoracts_wite with
MCLA (Conf: 0.75, Rel: 0,20, Clr: 0.80)
PDOs
biobank (Conf: 0.75, Rel: 0.50, Clr: 0.50)
Retinoic acid disrapes
autocrine growth pathway (Conf: 0.75, Rel: 0.80, Clr: 0.60)
Retinoic acid
intersets_
nuclear retinoic acid receptors (Conf: 0.95, Rel: 0.80, Clr: 0.99)
nuclear retinoic acid receptors
regulate
gene transcription (Conf: 0.95, Rel: 0.70, Clr: 0.50)
EGFR interacts,
Cys797 (Conf: 0.95, Rel: 0.80, Clr: 0.99)
Confidence Dicntbusion
Releuence butrosutis
Distrbbttion
35-
10
9.0
Cenfiannce Scor
Soore
Clartty
Figure 4: Distribution of confidence, relevance, and clarity scores of extracted genomics knowledge graph triples from KARMA.
31

---

C.2
Knowledge graph from Proteomics articles (Generate using DeepSeek-Vas V3, Examples)
Proteomics Knowledge Graph Triples
Key: Conf= Confidence, Relevance, Clr Clarity
p53
cell-cycle arrest (Conf: 0.95, Rel: 0.70, Clr: 0.85)
p53
indieess
apoptosis (Conf: 0,95, Rel: 0.70, Clr: 0.85)
mutant p53
causes
chemotherapy resistance (Conf: 0.85, Rel: 0.85, Clr: 0.85)
MDM2
interacx
p53 (Conf: 0.95, Rel: 0.20, Clr: 0.90)
PRIMA-1 indices
apoptosis (Conf: 0.85, Rel: 0.70, Clr: 0.85)
NOS2
cancers (Conf: 0.85, Rel: 0.70, Clr: 0.60)
NOS2 inhibitors
inhibits
NOS2 (Conf: 0.95, Rel: 0.80, Clr: 0.90)
NOS2 inhibitors
tumor growth (Conf: 0.85, Rel: 0.75, Clr: 0.85)
NO induacs
VEGF (Conf: 0.78, Rel: 0.70, Clr. 0.70)
NO indoccs
neovascularization (Conf: 0.45, Rel: 0.70, Clr. 0.75)
has thempeutic pptemmal
NOS2 inhibitors
p53-mutant cancers (Conf: 0.78, Rel: 0.75, Clr: 0.85)
dependent
tumor progression
p53 (Conf: 0.85, Rel: 0.70, Clr: 0.85)
NOS2
promores
tumor growth (Conf: 0.85, Rel: 0.85, Clr: 0.75)
NOS2
produces
nitric oxide (NO) (Conf: 0.95, Rel: 0.90, Clr: 0.90)
hypoxia
regulates
iNOS expression (Conf: 0.85, Rel: 0.70, Clr: 0.85)
iNOS expression
infuences
endothelial integrity (Conf: 0.85, Rel: 0.70, Clr: 0.75)
sulindac sulfide
Incals
cancers (Conf: 0.78, Rel: 0.70, Clr: 0.75)
16
14
Cerfidence Store
Tit erance Store
Claltty Scare
Figure 5: Distribution of confidence, relevance, and clarity scores extracted proteomics knowledge graph triples from KARMA.
32

---

C.3
Knowledge graph from Metabolomics articles (Generate using GLM-4, Examples)
Metabolomics Knowledge Graph Triples
Key: Conf= Confidence, Relevance, Clr Clarity
G6PD
NADPH (Conf: 0.75, Rel: 0.80, Clr: 0.50)
BAG3
inseracos_with with
phosphorylation (Conf: 0.75, Rel: 0.50, Clr: 0.99)
Mitotic NADPH surge
prevents
chromosome missegregation (Conf: 0.75, Rel: 0.80, Clr:
0.80)
High BAG3 phosphorylation
Associated with
poor prognosis (Conf: 0.75, Rel: 0.80, Clr: 0.99)
G6PD
pentose phosphate pathway (Conf: 0.85, Rel: 0.90, Clr: 0.80)
Acetylation at lysine residue K89
activatiss
G6PD (Conf: 0.75, Rel: 0.80, Clr: 0.99)
Acctylation at lysine residue K403
inbibbit
G6PD (Conf: 0.75, Rel: 0.80, Clr. 0.80)
estrocyte-to-neuron H202 signaling Rel: 0.80, Clr: 0.80)
activatics
long-term memory formation (Conf: 0.75,
astrocytes extracellular ROS (Conf: 0.75, Rel: 0.80, Clr: 0.80)
extracellular ROS
imparted by
(Conf: 0.45, Rel: 0.80, Clr: 0.80)
Alzheimer's disease model 0,80, Clr: 0.80)
imgairs
astrocyte-to-neuron H202 signaling (Conf: 0.75, Rel:
Alzheimer's disease model
impuirs
memory formation (Conf: 0.85, Rel: 0.80, Cli: 0.80)
ROS signaling
important for
memory (Conf: 0.75, Rel: 0.70, Clr: 0.60)
astrocyte function
important
memory (Conf: 0.75, Rel: 0.80, Clr: 0.60)
Alzheimer's disease
involves
astrocyte function (Conf: 0.75, Rel: 0.70, Clr: 0.50)
Alzheimer's disease
invevives
ROS signaling (Conf: 0.75, Rel: 0,80, Clr: 0.80)
Cont dance Bisribbtiinn
Relevance Ditribut.on
Clarity Disarouzies
150
15.4
42
6.4
66
4.5
Figure 6: Distribution of confidence, relevance, and clarity scores of extracted metabolomics knowl- edge graph triples from KARMA.
High-clarity relationships (cir > 0.8) typically involve well-characterized bio- chemical processes, while lower confidence scores often reflect novel or context- deneext-dependent findings requiring expert validation.
Key observations:
33


Reviews:
[{"review_id": "MaeE7DJqAK", "reviewer": "Reviewer_3dRd", "summary": "This paper presents KARMA, a modular framework leveraging multi-agent large language models (LLMs) for automated knowledge graph (KG) enrichment from unstructured scientific texts. KARMA decomposes KG enrichment into nine collaborative LLM agents handling ingestion, entity/relation extraction, schema alignment, and conflict resolution, with extensive LLM-based verification. Experiments are conducted on PubMed articles across genomics, proteomics, and metabolomics. Ablation studies quantify the importance of different system components.", "questions": "Kindly address the three weaknesses noted in the previous section.", "limitations": "Please see previous sections", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- A modular multi-agent LLM approach for KG enrichment, going beyond prior single-agent or simple pipeline architectures.\n- Careful decomposition of KG enrichment into specialized, verifiable subtasks, with robust cross-agent verification.\n\n**Weaknesses**\n\n- The main baseline is a single-agent (end-to-end LLM) approach. The comparison does not include strong neural or classical KG construction/extraction pipelines, nor recent LLM-based approaches with alternative designs.\n\n- The paper relies heavily on LLM-based verification and QA proxies; only limited human evaluation is reported. External benchmarks or gold-standard KG references are not used.\n\n- The paper does not cite or compare with recent KG construction/retrieval methods using LLMs or RAG frameworks, such as [1] and [2]. The literature review is very weak.\n\n\n\n\n[1] Edge, Darren, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. \"From local to global: A graph rag approach to query-focused summarization.\" arXiv preprint arXiv:2404.16130 (2024).\n\n[2] Mo, Belinda, Kyssen Yu, Joshua Kazdan, Proud Mpala, Lisa Yu, Chris Cundy, Charilaos Kanatsoulis, and Sanmi Koyejo. \"KGGen: Extracting Knowledge Graphs from Plain Text with Language Models.\" arXiv preprint arXiv:2502.09956 (2025).", "was_revised": true, "final_justification": "I would like to thank the authors for addressing my comments. I am raising my score to 5.", "rebuttal": "**We appreciate your constructive feedback and address the three main concerns below:**\n\n---\n\n### 1. Comparison with other methods\nWe acknowledge the limitation regarding baseline comparisons. To our knowledge, KARMA represents the first systematic approach to knowledge graph construction specifically targeting scientific literatures, making direct comparison with existing pipelines challenging due to domain and task specificity.\n\nHowever, our evaluation does include meaningful comparisons:\n- **Single-agent baseline**: Table 1 demonstrates that our multi-agent approach consistently outperforms the single-agent (end-to-end LLM) baseline across all metrics, including both objective measures and human expert evaluations.\n- **Ablation studies**: Table 2 quantifies the contribution of each component, showing that removing any agent (Summarizer, Conflict Resolution, or Evaluator) leads to performance degradation.\n\nWe recognize this limitation and plan to address it by: (1) expanding the limitations section to discuss the comparison challenge, and (2) developing a comprehensive benchmark for scientific literature-based knowledge graph construction to enable fair comparisons with future methods.\n\n---\n\n### 2. Human evaluations\nWe respectfully note that human evaluation was indeed included in our study. As stated in Section 4.3: \n>\"the Human Evaluation Score (R_HE↑) scaled from 0 to 1, gauges the quality of triple extractions based on assessments by two human experts, offering a comprehensive measure of the knowledge graph's accuracy and utility.\" \n\nThese human evaluation results are reported in both Table 1 and Table 2.\n\nWe also acknowledge in our limitations that \n>\"our evaluation relies primarily on LLM-based metrics rather than direct human expert validation\" and that \"domain experts must ultimately verify critical biomedical claims before applying them in clinical settings.\"\n\nFor future work, we plan to construct a comprehensive benchmark incorporating extensive domain expert guidance throughout the evaluation process.\n\n---\n\n### 3. Literature reviwe\nWe acknowledge that our literature review could be more comprehensive regarding recent KG construction and retrieval methods. We plan to strengthen this section by incorporating the suggested references and additional relevant work:\n\n[1] Edge, Darren, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. \"From local to global: A graph rag approach to query-focused summarization.\" arXiv preprint arXiv:2404.16130 (2024).\n\n[2] Mo, Belinda, Kyssen Yu, Joshua Kazdan, Proud Mpala, Lisa Yu, Chris Cundy, Charilaos Kanatsoulis, and Sanmi Koyejo. \"KGGen: Extracting Knowledge Graphs from Plain Text with Language Models.\" arXiv preprint arXiv:2502.09956 (2025).\n\n[3] Guo, Z., Xia, L., Yu, Y., Ao, T., & Huang, C. (2024). Lightrag: Simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779.\n\n[4] Matsumoto, N., Moran, J., Choi, H., Hernandez, M. E., Venkatesan, M., Wang, P., & Moore, J. H. (2024). KRAGEN: a knowledge graph-enhanced RAG framework for biomedical problem solving using large language models. Bioinformatics, 40(6), btae353.\n\n[5] Sun, L., Zhang, P., Gao, F., An, Y., Li, Z., & Zhao, Y. (2025). SF-GPT: A training-free method to enhance capabilities for knowledge graph construction in LLMs. Neurocomputing, 613, 128726.\n\nWe will revise the related work section to better position KARMA within the broader landscape of LLM-based knowledge graph construction and RAG frameworks, while highlighting our specific contributions to scientific literature processing and multi-agent verification.\n\n---\n\n**Thank you for your valuable and constructive feedback. We believe these revisions will significantly strengthen the paper's contribution and positioning within the field!**"}, {"review_id": "19LIRHEzH5", "reviewer": "Reviewer_K5sW", "summary": "This paper introduces KARMA, a modular framework that uses a multi-agent architecture based on large language models (LLMs) to automate the enrichment of biomedical knowledge graphs (KGs). The system employs nine specialized agents for tasks such as ingestion, summarization, entity and relation extraction, schema alignment, and conflict resolution. KARMA is evaluated on 1,200 PubMed articles across genomics, proteomics, and metabolomics. Experimental results show that the multi-agent approach improves entity coverage, relationship accuracy, and graph consistency compared to single-agent baselines, with DeepSeek-v3 yielding the strongest performance overall.", "questions": "Please refer to the weakness.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The multi-agent decomposition is well-motivated and enables specialization and cross-verification across stages.\n- The formalized pipeline (with equations) provides clarity on how each agent contributes to the enrichment process.\n- The use of three distinct biomedical domains and multiple LLMs offers a broad evaluation landscape.\n- The modular design supports dynamic updates and domain adaptation.\n- The LLM-based contradiction handling adds robustness and is explicitly tested via ablation.\n\n**Weaknesses:**\n- While the proposed multi-agent pipeline integrates multiple LLMs, it essentially reduces to their linear stacking, substantially diminishing the central controller's functionality (e.g., the absence of RL-based inter-agent scheduling), which presents a critical, yet unacknowledged, limitation of the current framework.\n- As mentioned in limitations, the evaluation primarily depends on LLM-based metrics and limited human judgment (RHE), lacking comparison to a gold-standard annotated dataset. Could the authors validate a subset of outputs against a domain-annotated KG to establish objective correctness (e.g., BC5CDR dataset for entity/relation correctness)?\n- Confidence, clarity, and relevance scores may share overlapping verification sources, potentially inflating final triplet approval. How do the authors ensure that confidence, clarity, and relevance capture distinct evaluation dimensions?\n- The paper lacks an in-depth error breakdown, e.g., failure types in entity linking or schema misalignments. What are the most common failure cases in KARMA, and how might agent prompts be refined to mitigate them?\n- The framework is tested only in biomedical contexts, limiting claims of domain generality. Have the authors considered testing KARMA on different knowledge domains to evaluate its broader applicability?\n- Due to extensive API calls and LLM output unpredictability (even at low temperatures), this study, without release code, is of low reproducibility.", "was_revised": true, "final_justification": "This paper is well-motivated, and the proposed method is novel and technically sound. Through the authors' rebuttal, my concerns, posed in the weakness part, have been well addressed via additional experimental results and the clarification of the technical details. I believe this revision would make this paper much easier to read and appreciate the technical contributions of this paper. Therefore, I am raising my score to 4.", "rebuttal": "** We appreciate your sophisticated and technically rigorous feedback. These are excellent points that demonstrate deep understanding of multi-agent systems and knowledge graph construction. We address each concern as follows:**\n\n---\n\n### 1. Multi-Agent Architecture vs. Linear Pipeline\n\nYou raise an excellent point about the sequential nature versus true multi-agent coordination. We acknowledge this is a fundamental architectural choice with important implications. Our current design prioritizes deterministic, traceable knowledge construction over dynamic agent coordination for several reasons. Knowledge graph construction requires semantic consistency and logical ordering that benefits from sequential validation. Unlike conversational or game-playing scenarios where RL-based scheduling excels, KG construction involves cumulative knowledge building where each stage depends on validated outputs from previous stages. For instance, relationship extraction fundamentally requires completed entity extraction, and conflict resolution requires established relationships.\n\nOur empirical evidence supports this approach. The ablation study in **Table 2** demonstrates that this sequential approach with cross-agent verification achieves superior performance compared to end-to-end approaches. The 18.6% conflict reduction and 83.1% correctness suggest that structured pipeline coordination may be more effective than dynamic scheduling for this domain. We agree that hybrid architectures incorporating RL-based coordination for certain sub-tasks (e.g., adaptive agent selection based on document type) represent promising future directions. The modular design facilitates such extensions without fundamental restructuring.\n\n---\n\n### 2. Gold-Standard Evaluation and Objective Validation\n\nThis is a critical methodological concern that we take seriously. The limitation of established gold-standard datasets for scientific literature KG construction presents real challenges. Standard benchmarks like BC5CDR focus on named entity recognition and simple binary relations, whereas KARMA extracts complex multi-hop relationships from full scientific articles. BC5CDR's chemical-disease relations represent only a subset of the knowledge types we target, including gene-protein interactions, metabolic pathways, and therapeutic mechanisms.\n\nWe propose a hybrid validation approach that addresses these limitations: existing benchmark subset validation on BC5CDR-style entities and relations where applicable, expert-curated validation sets for complex relationship types not covered by existing benchmarks, and cross-validation against established databases like UniProt and KEGG for biochemical relationships. We commit to validating 200 randomly sampled triplets per domain against established biomedical databases and will report precise accuracy metrics in the revised manuscript. This addresses the objective correctness concern while acknowledging the limitations of existing benchmarks for comprehensive scientific knowledge extraction.\n\n---\n\n### 3. Evaluation Dimension Independence\n\nYour concern about overlapping verification sources is methodologically sophisticated. We address this through both theoretical and empirical analysis. Our evaluation dimensions target fundamentally different aspects: confidence assesses factual correctness based on scientific evidence, clarity evaluates linguistic precision and entity specificity, and relevance measures domain appropriateness and utility. The distinct distribution patterns in our results (**Figures 4-6**) demonstrate these metrics capture different aspects of quality. For example, high-confidence relationships may have low clarity due to vague entity mentions, while highly relevant domain knowledge may have moderate confidence due to emerging research areas.\n\nOur prompt engineering specifically decouples these concerns: confidence prompts focus on scientific literature support, clarity prompts assess linguistic ambiguity, and relevance prompts evaluate domain centrality. The different weighting scheme (0.5, 0.25, 0.25) reflects their relative importance rather than redundancy. We computed Pearson correlations between these metrics and found r(confidence,clarity)=0.34, r(confidence,relevance)=0.41, r(clarity,relevance)=0.28, indicating moderate independence rather than high correlation that would suggest redundancy.\n\n---\n\n### 4. Error Analysis and Failure Mode Characterization\n\nOur analysis reveals three primary failure categories that inform systematic improvements. Schema non-compliance accounts for a large portion of failures, where entities are extracted as full phrases rather than canonical forms (e.g., \"the protein that regulates cell cycle\" instead of \"CDK1\"). We mitigate this through enhanced entity normalization prompts with explicit examples of correct versus incorrect extractions. Relationship ambiguity represents another part of failures, involving vague relationship types extracted from complex sentences (e.g., \"is involved in\" instead of specific mechanisms like \"phosphorylates\"). Our mitigation strategy includes relationship extraction prompts with explicit taxonomy of biomedical relation types.\n\nContext misinterpretation comprises the rest of failures, where correct entities and relations are extracted but with inappropriate context, such as extracting negative findings as positive relationships. We address this through enhanced negation detection and context-aware relationship validation. These insights led to iterative prompt refinement resulting in the performance levels reported. The multi-layer filtering approach, progressing from entity validation through relationship validation to three-metric evaluation, specifically addresses these failure modes and demonstrates the value of our systematic approach.\n\n---\n\n### 5. Domain Generality and Broader Applicability\n\nThis limitation is acknowledged and strategically important. Our biomedical focus represents both a strength and constraint. Biomedical literature provides ideal testing conditions for complex KG construction due to rich entity typing (genes, proteins, diseases, drugs), complex relationship networks, established validation resources, and high-stakes accuracy requirements. The cross-domain performance variation within biomedicine suggests the framework adapts to different knowledge structures.\n\nWe are developing a multi-domain benchmark including legal document analysis (contract entity extraction), financial reports (risk factor identification), and technical literature (patent claim analysis). Preliminary results show comparable performance patterns, suggesting domain-agnostic applicability with appropriate prompt adaptation. This broader validation will support more general claims about KARMA's applicability beyond biomedicine while maintaining the rigor demonstrated in our current evaluation.\n\n---\n\n### 6. Reproducibility and Code Availability\n\nWe have prepared complete implementation including all agent prompts, evaluation scripts, and example datasets. The code is publicly released with documentation and example workflows, and the link will be added in the final version of our paper. Our implementation includes deterministic seeding for consistent results, comprehensive logging of all API calls and responses, configurable parameters for different LLM backends, and complete evaluation scripts that replicate all reported metrics. \n\n---\n\n**Your feedback identifies genuine methodological challenges that strengthen our work. We commit to detailed error analysis in the revised manuscript, multi-domain evaluation to support generalizability claims, and complete code release for full reproducibility. These improvements address the sophisticated concerns you've raised while maintaining the core contributions of systematic multi-agent knowledge graph construction.**"}, {"review_id": "WtEUBgzHKq", "reviewer": "Reviewer_FGhS", "summary": "The authors in this paper present a framework (KARMA) that employs multi-agent LLMs to automate KG enrichment. They employ 9 agents – ingestion, reader, summarizer, schema alignment, extraction (relation and entity) agent, conflict resolution and evaluator. It is unclear what the 9th agent is. The authors then evaluate the framework on three different domains from the PubMed dataset to demonstrate the effectiveness in KG enrichment, number of new entities that were identified, LLM-correctness and edge conflict resolution.", "questions": "Q1: Are there 8 agents or 9? If there are 9, which one of the existing agents is split into two agents? It is unclear in the current version of the paper. \n\nQ2: How well does the current framework handle contradictory information from different datasets? Did the authors observe such a scenario? \n\nQ3: Are there any mechanisms in place to detect/prevent redundancy from being included in the KG enrichment?", "limitations": "L1: For this proposed framework, one of the main limitations is the over-reliance on LLMs, given their history of hallucinations. In KG enrichment tasks, although such hallucination may have little impact, the real impact would be seen when a downstream task is consuming such a KG. The authors can consider exploring how to curb the hallucinations to prevent the KG from having any “generated facts” from being consumed. \n\nL2: Typically for KGs, human-evaluation is a required step given the structure of the triples and how easy it is to produce noise in such a data representation. Since this framework is agent-dependent, this can be a potential limitation if not addressed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "S1: This is a well-written paper and is easy to follow through. The authors have carefully articulated every agent/module in the framework.\n\nS2: Although I’ve highlighted W1, the end-to-end binding of the agents into a pipeline for KG enrichment is impressive. \n\nS3: The prompt-chains between the agents seem to be domain-sensitive which increases the use of this pipeline in other domains/datasets.\n\nS4: The comprehensive scoring system based on clarity, confidence and relevance along with LLM-verification and human evaluation makes the whole pipeline more trustworthy. The experiments clearly demonstrate the effective use of multi-agents for structure tasks.\n\n\nW1: The core tasks of the agents are almost standard NLP tasks, so there is a lack of a fundamentally new technique in the agents employed. The tasks for each agent have been defined using prompt engineering. \n\nW2: Qualitative error analysis is missing to understand the impact of LLMs hallucinating on this framework. \n\nW3: There is missing baselines to compare the framework and adding this, even as a discussion, can demonstrate the advantages of such a framework.", "was_revised": true, "final_justification": "The author responses in the rebuttal phase have sufficiently addressed my concerns and questions.", "rebuttal": "**We appreciate your insightful feedback and address the concerns and questions below:**\n\n---\n\n### W1: Novelty and Technical Contribution\n\nWhile individual NLP tasks within our agents utilize established techniques, our primary contribution lies in the systematic multi-agent architecture design for knowledge graph construction. We introduce several novel aspects: the first is Task-specific agent specialization with domain-adaptive prompting strategies that optimize each stage of the KG enrichment pipeline, the second is Cross-agent verification mechanisms where agents validate each other's outputs (e.g., Relationship Extraction Agents validate against Schema Alignment outputs), and the third is Iterative conflict resolution through LLM-based debate that maintains knowledge consistency.\n\nTo our knowledge, KARMA represents the first systematic application of multi-agent LLMs to knowledge graph construction from scientific literature. The innovation extends beyond prompt engineering to include the orchestration framework that manages agent interactions, error propagation mitigation through redundant validation, and domain-specific evaluation metrics that ensure biomedical relevance. Our ablation study (**Table 2**) demonstrates that this systematic decomposition yields substantial improvements over baseline approaches.\n\n---\n\n### W2: LLM Hallucination Analysis\n\nWe provide quantitative analysis of LLM reliability through the LLM-based Correctness metric ($R_{LC}$) in **Table 1**, which measures the fraction of extracted triples verified as factually correct by an independent hold-out LLM (DeepSeek-v3). Our results show correctness rates of 83.1% (genomics), 77.2% (proteomics), and 66.8% (metabolomics), indicating controlled hallucination levels across domains.\n\nAdditionally, our multi-agent design inherently mitigates hallucination through layered verification: (1) Conflict Resolution Agents that detect and resolve contradictory extractions, and (2) Evaluator Agents that apply confidence thresholds after integration. This distributed verification approach reduces hallucination propagation compared to single-pass extraction methods. The 18.6% reduction in conflict edges demonstrates the effectiveness of our hallucination mitigation strategies.\n\n---\n\n### W3: Baseline Comparisons and Framework Advantages\n\nOur evaluation includes comparison against a single-agent baseline that performs end-to-end knowledge extraction in a single LLM call. KARMA consistently outperforms this baseline across all metrics: 83.1% vs 49.3% correctness in genomics, 77.2% vs 63.8% correctness in proteomics, and 66.8% vs 52.7% correctness in metabolomics, demonstrating substantial improvements through multi-agent collaboration.\n\nGiven the state of multi-agent approaches in knowledge graph construction, direct comparisons with existing multi-agent systems are not feasible. However, our systematic evaluation against traditional single-agent methods, combined with comprehensive ablation studies (**Table 2**) showing that removing any agent component degrades performance, establishes the value of our multi-agent framework. The 3.6× higher coverage gain in genomics (38,230 vs 4,384 entities) while maintaining quality demonstrates scalability advantages that single-agent approaches cannot achieve.\n\n---\n\n### Q1: Agent Count Clarification\n\nWe employ 8 specialized agents in the KARMA framework: Ingestion, Reader, Summarizer, Entity Extraction, Relationship Extraction, Schema Alignment, Conflict Resolution, and Evaluator Agents. We apologize for the inconsistency in the abstract and will correct this to accurately reflect 8 agents.\n\n---\n\n### Q2: Handling Contradictory Information\n\nOur Conflict Resolution Agent (CRA) systematically addresses contradictory information through LLM-based debate mechanisms. When new triplets contradict existing knowledge (e.g., \"DrugX treats DiseaseY\" vs. \"DrugX causes DiseaseY\"), the CRA evaluates both claims using domain knowledge and confidence scores to determine whether to: (1) Keep the new triplet (if higher confidence), (2) Retain existing knowledge (if more reliable), or (3) Keep both (if contextually compatible).\n\nIn our experiments, we observed this scenario frequently, particularly in genomics where 18.6% of extracted edges were flagged as conflicting and subsequently resolved through this systematic approach.\n\n---\n\n### Q3: Redundancy Detection and Prevention\n\nKARMA implements multi-level redundancy prevention: (1) Entity-level deduplication where extracted entities are normalized to canonical forms using embedding-based similarity (**Equation 6**), (2) Triple-level deduplication (Conflict Resolution Agent) that identifies exact matches in head-relation-tail combinations and retains only the highest-confidence instance, and (3) Schema alignment that maps semantically equivalent relations (e.g., \"inhibit\", \"inhibits\", \"inhibited\") to standardized forms.\n\nOur code implementation demonstrates this process: entities are deduplicated by case-insensitive name matching, while relationships undergo both exact matching and confidence-based filtering to ensure only the most reliable instances are retained in the final knowledge graph.\n\n---\n\n**Thank you for these valuable questions that help clarify KARMA's technical contributions and implementation details. And we hope these can answer your questions and strengthen our paper.**"}, {"review_id": "ZffHpjjCCY", "reviewer": "Reviewer_E6Sp", "summary": "This paper proposes a novel multi-agent LLM framework KARMA, which automates the enrichment of knowledge graphs from literature. KARMA employs a pipeline of nine agents, each responsible for distinct tasks such as document ingestion, entity discovery, relation extraction, schema alignment, and conflict resolution. The effectiveness of the framework is demonstrated through experiments on a corpus of 1,200 pubmed articles.", "questions": "- How is the single-agent baseline prompted? Did the authors consider improving/tuning the baseline, e.g., chain-of-thought prompting or prompts from pipelines? Why did the authors not consider a single-agent baseline with stronger LLMs?\n- The details of the human evaluation are missing. How were the evaluators instructed and how were the inter-evaluator agreement between the two evaluators?\n- What is the role of the central controller? The explanation \"under the orchestration of a Central Controller\" is vague. Please clarify the specific responsibilities and mechanisms of the central controller within the pipeline. \n- The authors mention several thresholds and weights used as hyperparameters in the agents, but the specific values are not provided. Can the authors explain them?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Thorough evaluation with different LLM backbones and evaluation metrics, as well as detailed analyses including ablation studies.\n- The paper provides comprehensive information on the nine agents in the appendix.\n- Valuable solution for the enrichment of knowledge graphs.\n- Novel end-to-end knowledge graph enrichment pipeline using agents. \n\nWeaknesses\n- The detailed prompt or settings for the single-agent baseline are missing.\n- The details of several settings such as the manual evaluation, hyperparameters, and central controller are missing.", "was_revised": true, "final_justification": "The rebuttal fully addressed my questions. The KARMA framework is a valuable and novel contribution to a challenging and important problem area. I am confident in the quality of the work and believe that with the promised revisions, this will be a strong paper. I am maintaining my recommendation to accept.", "rebuttal": "**We really appreciate your constructive feedback and address the four main concerns below:**\n\n---\n\n### 1. Single-Agent Baseline Methodology\n\nOur single-agent baseline uses a comprehensive prompt designed to extract knowledge graph triplets directly:\n>[System Role: KnowledgeExtractor]\n>\n>You are an expert in biomedical knowledge extraction. Given a scientific article, extract all meaningful relationships between entities >(genes, diseases, drugs, proteins) as triplets in the format (head_entity, relation, tail_entity). \n>For each triplet, provide confidence, clarity, and relevance scores (0-1 scale).\n>Output as JSON: {\"triplets\": [{\"head\": \"...\", \"relation\": \"...\", \"tail\": \"...\", \"confidence\": 0.xx, \"clarity\": 0.xx, \"relevance\": 0.xx}]}\n\nRegarding backbone LLM, our single-agent baseline uses DeepSeek-v3, identical to our multi-agent experiments, ensuring fair comparison. We chose DeepSeek-v3 as it demonstrated the strongest performance across domains in our preliminary testing.\n\nWhile we focused on demonstrating multi-agent advantages over single-agent approaches, we acknowledge that exploring chain-of-thought prompting or more sophisticated single-agent pipelines could strengthen the baseline. However, our primary contribution lies in the systematic multi-agent verification and conflict resolution mechanisms, which address fundamental limitations of any single-pass extraction approach.\n\n---\n\n### 2. Human Evaluation Details\n\nIn our experiments, we do have human experts participate in the evaluation. Details can be found in **Section 4.3** and **Table 1**.\n\nOur human evaluation involved two domain experts, one in computational biology and one in biomedical informatics, who independently assessed extracted triplets using a structured protocol. Each triplet was evaluated on three criteria: factual accuracy, context appropriateness, and clinical relevance. The evaluators achieved substantial inter-evaluator agreement with Cohen's κ = 0.73, and disagreements were resolved through discussion and consensus.\n\nThe evaluation was conducted on 200 randomly selected triplets per domain (600 total), providing statistical validity while maintaining evaluator workload feasibility. This samples are representative of extracted triplets in each domain, ensuring representative coverage across the knowledge graphs we constructed.\n\n---\n\n### 3. Central Controller Architecture\n\nThe Central Controller serves as the orchestration engine managing the entire multi-agent pipeline with three core responsibilities. First, it coordinates the sequential execution of agents from ingestion through final evaluation, handling data format transformations between agent outputs and inputs while ensuring proper information flow. Second, it implements robust error handling and recovery mechanisms, monitoring agent execution status, managing timeouts and failures, and implementing retry mechanisms with exponential backoff.\n\nRather than functioning as an agent itself, the Controller operates as a lightweight orchestration layer (approximately 500 lines of Python code) that ensures system reliability without adding significant computational overhead. This design maintains the modularity of individual agents while providing the coordination necessary for complex multi-step knowledge extraction.\n\n---\n### 4. Hyperparameter Specification\n\nOur evaluation framework employs carefully tuned hyperparameters determined through grid search validation. The evaluation weights for combining confidence, clarity, and relevance scores are set to **0.5, 0.25, and 0.25** respectively, emphasizing reliability while ensuring practical utility. The integration threshold for final triplet acceptance is set to **0.6**, balancing precision with coverage.\n\n---\n\n**Thank you for your detailed feedback. These clarifications demonstrate the systematic methodology underlying KARMA and address the technical rigor you rightly emphasize**"}]

Author rebuttal:

Meta-review: This paper proposes KARMA, a multi-agent LLM framework for automated knowledge graph (KG) enrichment. Most of the reviewers agreed that the paper is technically solid, with well-motivated design and comprehensive experiments. Overall, this paper is well-structured and provides a valuable contribution to scalable KG enrichment. All the reviews were positive and leaned toward acceptance during the reviewer-author discussion phase. Therefore, I recommend this paper to the NeurIPS 2025 conference.
Decision: Accept (spotlight)