Title: KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment
Venue: NeurIPS 2025
Accepted: yes
Keywords: Knowledge Graphs

Multi Agent System
Abstract: Maintaining comprehensive and up-to-date knowledge graphs (KGs) is critical for modern AI systems, but manual curation struggles to scale with the rapid growth of scientific literature. This paper presents KARMA, a novel framework employing multi-agent large language models (LLMs) to automate KG enrichment through structured analysis of unstructured text. Our approach employs nine collaborative agents, spanning entity discovery, relation extraction, schema alignment, and conflict resolution that iteratively parse documents, verify extracted knowledge, and integrate it into existing graph structures while adhering to domain-specific schema. Experiments on 1,200 PubMed articles from three different domains demonstrate the effectiveness of KARMA in knowledge graph enrichment, with the identification of up to 38,230 new entities while achieving 83.1\% LLM-verified correctness and reducing conflict edges by 18.6\% through multi-layer assessments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel multi-agent LLM framework KARMA, which automates the enrichment of knowledge graphs from literature. KARMA employs a pipeline of nine agents, each responsible for distinct tasks such as document ingestion, entity discovery, relation extraction, schema alignment, and conflict resolution. The effectiveness of the framework is demonstrated through experiments on a corpus of 1,200 pubmed articles.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Thorough evaluation with different LLM backbones and evaluation metrics, as well as detailed analyses including ablation studies.\n- The paper provides comprehensive information on the nine agents in the appendix.\n- Valuable solution for the enrichment of knowledge graphs.\n- Novel end-to-end knowledge graph enrichment pipeline using agents. \n\nWeaknesses\n- The detailed prompt or settings for the single-agent baseline are missing.\n- The details of several settings such as the manual evaluation, hyperparameters, and central controller are missing.\n\nQUESTIONS:\n- How is the single-agent baseline prompted? Did the authors consider improving/tuning the baseline, e.g., chain-of-thought prompting or prompts from pipelines? Why did the authors not consider a single-agent baseline with stronger LLMs?\n- The details of the human evaluation are missing. How were the evaluators instructed and how were the inter-evaluator agreement between the two evaluators?\n- What is the role of the central controller? The explanation \"under the orchestration of a Central Controller\" is vague. Please clarify the specific responsibilities and mechanisms of the central controller within the pipeline. \n- The authors mention several thresholds and weights used as hyperparameters in the agents, but the specific values are not provided. Can the authors explain them?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in this paper present a framework (KARMA) that employs multi-agent LLMs to automate KG enrichment. They employ 9 agents – ingestion, reader, summarizer, schema alignment, extraction (relation and entity) agent, conflict resolution and evaluator. It is unclear what the 9th agent is. The authors then evaluate the framework on three different domains from the PubMed dataset to demonstrate the effectiveness in KG enrichment, number of new entities that were identified, LLM-correctness and edge conflict resolution.\n\nSTRENGTHS AND WEAKNESSES:\nS1: This is a well-written paper and is easy to follow through. The authors have carefully articulated every agent/module in the framework.\n\nS2: Although I’ve highlighted W1, the end-to-end binding of the agents into a pipeline for KG enrichment is impressive. \n\nS3: The prompt-chains between the agents seem to be domain-sensitive which increases the use of this pipeline in other domains/datasets.\n\nS4: The comprehensive scoring system based on clarity, confidence and relevance along with LLM-verification and human evaluation makes the whole pipeline more trustworthy. The experiments clearly demonstrate the effective use of multi-agents for structure tasks.\n\n\nW1: The core tasks of the agents are almost standard NLP tasks, so there is a lack of a fundamentally new technique in the agents employed. The tasks for each agent have been defined using prompt engineering. \n\nW2: Qualitative error analysis is missing to understand the impact of LLMs hallucinating on this framework. \n\nW3: There is missing baselines to compare the framework and adding this, even as a discussion, can demonstrate the advantages of such a framework.\n\nQUESTIONS:\nQ1: Are there 8 agents or 9? If there are 9, which one of the existing agents is split into two agents? It is unclear in the current version of the paper. \n\nQ2: How well does the current framework handle contradictory information from different datasets? Did the authors observe such a scenario? \n\nQ3: Are there any mechanisms in place to detect/prevent redundancy from being included in the KG enrichment?\n\nLIMITATIONS:\nL1: For this proposed framework, one of the main limitations is the over-reliance on LLMs, given their history of hallucinations. In KG enrichment tasks, although such hallucination may have little impact, the real impact would be seen when a downstream task is consuming such a KG. The authors can consider exploring how to curb the hallucinations to prevent the KG from having any “generated facts” from being consumed. \n\nL2: Typically for KGs, human-evaluation is a required step given the structure of the triples and how easy it is to produce noise in such a data representation. Since this framework is agent-dependent, this can be a potential limitation if not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces KARMA, a modular framework that uses a multi-agent architecture based on large language models (LLMs) to automate the enrichment of biomedical knowledge graphs (KGs). The system employs nine specialized agents for tasks such as ingestion, summarization, entity and relation extraction, schema alignment, and conflict resolution. KARMA is evaluated on 1,200 PubMed articles across genomics, proteomics, and metabolomics. Experimental results show that the multi-agent approach improves entity coverage, relationship accuracy, and graph consistency compared to single-agent baselines, with DeepSeek-v3 yielding the strongest performance overall.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The multi-agent decomposition is well-motivated and enables specialization and cross-verification across stages.\n- The formalized pipeline (with equations) provides clarity on how each agent contributes to the enrichment process.\n- The use of three distinct biomedical domains and multiple LLMs offers a broad evaluation landscape.\n- The modular design supports dynamic updates and domain adaptation.\n- The LLM-based contradiction handling adds robustness and is explicitly tested via ablation.\n\n**Weaknesses:**\n- While the proposed multi-agent pipeline integrates multiple LLMs, it essentially reduces to their linear stacking, substantially diminishing the central controller's functionality (e.g., the absence of RL-based inter-agent scheduling), which presents a critical, yet unacknowledged, limitation of the current framework.\n- As mentioned in limitations, the evaluation primarily depends on LLM-based metrics and limited human judgment (RHE), lacking comparison to a gold-standard annotated dataset. Could the authors validate a subset of outputs against a domain-annotated KG to establish objective correctness (e.g., BC5CDR dataset for entity/relation correctness)?\n- Confidence, clarity, and relevance scores may share overlapping verification sources, potentially inflating final triplet approval. How do the authors ensure that confidence, clarity, and relevance capture distinct evaluation dimensions?\n- The paper lacks an in-depth error breakdown, e.g., failure types in entity linking or schema misalignments. What are the most common failure cases in KARMA, and how might agent prompts be refined to mitigate them?\n- The framework is tested only in biomedical contexts, limiting claims of domain generality. Have the authors considered testing KARMA on different knowledge domains to evaluate its broader applicability?\n- Due to extensive API calls and LLM output unpredictability (even at low temperatures), this study, without release code, is of low reproducibility.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents KARMA, a modular framework leveraging multi-agent large language models (LLMs) for automated knowledge graph (KG) enrichment from unstructured scientific texts. KARMA decomposes KG enrichment into nine collaborative LLM agents handling ingestion, entity/relation extraction, schema alignment, and conflict resolution, with extensive LLM-based verification. Experiments are conducted on PubMed articles across genomics, proteomics, and metabolomics. Ablation studies quantify the importance of different system components.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- A modular multi-agent LLM approach for KG enrichment, going beyond prior single-agent or simple pipeline architectures.\n- Careful decomposition of KG enrichment into specialized, verifiable subtasks, with robust cross-agent verification.\n\n**Weaknesses**\n\n- The main baseline is a single-agent (end-to-end LLM) approach. The comparison does not include strong neural or classical KG construction/extraction pipelines, nor recent LLM-based approaches with alternative designs.\n\n- The paper relies heavily on LLM-based verification and QA proxies; only limited human evaluation is reported. External benchmarks or gold-standard KG references are not used.\n\n- The paper does not cite or compare with recent KG construction/retrieval methods using LLMs or RAG frameworks, such as [1] and [2]. The literature review is very weak.\n\n\n\n\n[1] Edge, Darren, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. \"From local to global: A graph rag approach to query-focused summarization.\" arXiv preprint arXiv:2404.16130 (2024).\n\n[2] Mo, Belinda, Kyssen Yu, Joshua Kazdan, Proud Mpala, Lisa Yu, Chris Cundy, Charilaos Kanatsoulis, and Sanmi Koyejo. \"KGGen: Extracting Knowledge Graphs from Plain Text with Language Models.\" arXiv preprint arXiv:2502.09956 (2025).\n\nQUESTIONS:\nKindly address the three weaknesses noted in the previous section.\n\nLIMITATIONS:\nPlease see previous sections", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-08"}