KGARevion: An AI Agent for Knowledge-Intensive Biomedical QA

Authors: Xiaorui Su, Yibo Wang, Shanghua Gao, Xiaolong Liu, Valentina Giunchiglia, Djork-Arné Clevert, Marinka Zitnik
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Medical Reasoning, Medical QA, Agent, Knowledge Graph, LLM

Abstract:
Biomedical reasoning integrates structured, codified knowledge with tacit, experience-driven insights. Depending on the context, quantity, and nature of available evidence, researchers and clinicians use diverse strategies, including rule-based, prototype-based, and case-based reasoning. Effective medical AI models must handle this complexity while ensuring reliability and adaptability. We introduce KGARevion, a knowledge graph-based agent that answers knowledge-intensive questions. Upon receiving a query, KGARevion generates relevant triplets by leveraging the latent knowledge embedded in a large language model. It then verifies these triplets against a grounded knowledge graph, filtering out errors and retaining only accurate, contextually relevant information for the final answer. This multi-step process strengthens reasoning, adapts to different models of medical inference, and outperforms retrieval-augmented generation-based approaches that lack effective verification mechanisms. Evaluations on medical QA benchmarks show that KGARevion improves accuracy by over 5.2% over 15 models in handling complex medical queries. To further assess its effectiveness, we curated three new medical QA datasets with varying levels of semantic complexity, where KGARevion improved accuracy by 10.4%. The agent integrates with different LLMs and biomedical knowledge graphs for broad applicability across knowledge-intensive tasks. We evaluated KGARevion on AfriMed-QA, a newly introduced dataset focused on African healthcare, demonstrating its strong zero-shot generalization to underrepresented medical contexts.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730257639846

thread_type:
official_review

<thread_object>
creator: reviewer_pkdu

<content>
summary:
This paper introduces KGAREVION, a novel approach that combines the non-codified knowledge of LLMs with the structured knowledge  in KGs to tackle the complexities of knowledge-intensive medical question answering. The method involves extracting relevant triples, using a fine-tuned LLM to verify the correctness of these triples, revising incorrect triples, and generating answers. This approach effectively addresses the challenges in the medical domain that require analogies and the integration of multi-source evidence, demonstrating improved performance across multiple datasets.

strengths:
**Originality**: This paper proposes a new KG-based agent KGAREVION, which can handle complex and knowledge-intensive question answering tasks in the medical field. This work is original because it combines multi-source medical knowledge such as LLMs and KG, and the integration of revise into the processing flow is innovative, especially when dealing with complex problems that require precise domain knowledge.

**Quality**: The quality of the paper is reliable, with each step in the KGAREVION process clearly described and supported by appropriate methodologies. The authors also validate the model's effectiveness through experiments on multiple datasets, including new and challenging ones.

**Clarity**: The paper demonstrates a high level of clarity in presenting the KGAREVION model and its experimental results. The methodology section provides a clear description of the process, with well-organized main diagrams. Additionally, the tables and figures illustrating the experimental results are easy to understand.

**Significance**: This research holds significant importance in the field of medical question answering. It offers an effective tool that integrates data from multiple sources, such as LLMs and KGs, thereby enhancing the ability to accurately address complex medical questions.

weaknesses:
In the description of the methodology, certain parts lack coherence and completeness. For instance, the revise and answer stages do not specify the exact processing methods used.

questions:
**1.** What model is used for the revise process? A more detailed description should be provided. If an unfine-tuned LLM is used, does it have the capability to correct erroneous triples?

**2.** How are triples involving entities not included in the knowledge graph (Incomplete Knowledge) handled? Are they further processed or directly accepted as credible knowledge?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730274865544

thread_type:
official_review

<thread_object>
creator: reviewer_yzkb

<content>
summary:
The paper proposes a novel knowledge graph-based prompting technique for question answering in the medical domain. The approach consiststs of four steps: (1) Given a question in natural language, an LLM is used to generate subject-predicate-object triples from the question. (2) The triples are checked by means of a knowledge graph. Triples that are not clearly wrong are kept. (3) Then an LLM is asked to revise the triples. (4) Finally, the question is answered by means of an LLM and the revised triples.

strengths:
- The paper tackles an important problem: question answering in the medical domain
- Original, novel question answering architecture: (1) generate triples, (2) review triples, (3) revise triples, (4) answer question
- Strong evaluation: multiple datasets, evaluation depending on question complexity, and ablation study
- Clarity: high-quality figures that help understanding

weaknesses:
- Section 3.2 "fine-tuning stage" I find this part rather hard to understand. Figure 2b definitely helps, but it might be good to clarify this part even further by providing an example.
- Section 4.4: It suprises me that the results barely depend on the knowledge graph. I am wondering whether the knowledge graph is required at all then. What do the results look like for a knowledge graph that is not specialized to medicine (e.g., Freebase15k-237)? If the results for Freebase15k-237 are almost the same as for PrimeKG, then the KG would be irrelevant.
- Related work on fact-checking seems to be missing. 

Ciampaglia, Giovanni Luca, Prashant Shiralkar, Luis M. Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. "Computational fact checking from knowledge networks." PloS one 10, no. 6 (2015): e0128193.

Shi, Baoxu, and Tim Weninger. "Discriminative predicate path mining for fact checking in knowledge graphs." Knowledge-based systems 104 (2016): 123-133.

questions:
- Can you give an example for the "fine-tuning stage" in Section 3.2?
- Section 3.2 "inference stage": How often does each case occur (triple removed vs triple kept) for the given dataset?
- Can you run your question answering approach with a knowledge graph that is mostly unrelated to the questions (e.g., Freebase15k-237)? Do the results get significantly worse or do they stay approximately the same?
- In how far does each component of the approach handle and support multi-hop paths in knowledge graphs? It seems, the generate and revise steps do not support paths. The review step might support paths indirectly via KG embeddings.
- How does the review phase of the approach compare to fact-checking approaches?
- The approach seems rather general. Wouldn't it also work on other domains besides medicine?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730286486651

thread_type:
official_review

<thread_object>
creator: reviewer_jrr1

<content>
summary:
This article introduces KGARevion, an LLM which performs medical Q&A by verifying generated information (in the form of triplets) against the information in a knowledge graph (provided as knowledge graph embeddings). By doing so, it achieves competitive performance on several medical Q&A benchmarks. Additionally, the authors provide a new biomedical Q&A benchmark, which they call MedDDx. MedDDx leverages information from another Q&A dataset, but it is supplemented with several incorrect answers which are semantically similar to the correct answers.  KGARevion can be used with a variety of base LLMs and KGs, making it not only competitive but also versatile.

strengths:
- **Figures:** the figures are nice for understanding the paper; in particular, figure 2 is great. 

- **Creative Benchmark:** the idea to create a benchmark with incorrect answers that are semantically similar to correct answers is very creative. In addition, separating questions into levels of difficulty is also quite clever. 

- **Reproducibility:** I commend the authors for providing code. Although I did not test it, it looks well documented and well written. 

- **Versatility:** the KGARevion approach seems versatile, able to be used with other base LLMs, KGs, and other domains in general.

weaknesses:
Following the authors' rebuttal, I feel that my concerns below were sufficiently addressed. The authors have added an additional benchmark dataset from a sufficiently different data lineage, and they have also clarified several points of confusion. I have updated my score accordingly.

-------------------------------------------------------------------------------

I have a few concerns about the scientific robustness and the presentation of this work: 

1. Firstly, my greatest concern regards the overlap between information in the knowledge graph(s) (KGs) used and the information used to create the benchmarks. Specifically, the authors designed their new QA benchmark, MedDDx, based upon the STaRK-Prime benchmark. **However, according to the corresponding STaRK-Prime paper (https://arxiv.org/abs/2404.13207), the biomedical QA from STaRK-Prime was built upon the PrimeKG, the exact KG which the authors used to fine-tune their approach, KGARevion.** Ultimately, this is not scientifically robust as the authors are providing their LLM with the same, structured data on which their benchmark was built. 

    I understand that the authors have also used other "gold-standard" benchmarks for comparison. However, there is also likely data leakage in some of these cases as well. For example, the PrimeKG is built from structured data derived from other sources, including the Disease Ontology (DO), the Gene Ontology (GO), UMLS, and DrugBank (Fig. 2, https://www.nature.com/articles/s41597-023-01960-3), all of which are also included in the creation of the BioASQ benchmark (https://pmc.ncbi.nlm.nih.gov/articles/PMC10042099/). Unfortunately, the other KG used within Section 4.4, ogb-bioKG, does not have any provenance information (https://github.com/snap-stanford/ogb/issues/111) but it was built by the same person (M. Zitnik) as PrimeKG, so it likely comes from similar sources. Ultimately, I would still avoid using obg-bioKG if the provenance can not be verified. 

    **I recommend that the authors research the data lineage of each of the benchmarks and KGs used. Perhaps they can present this as supplementary material. Then, I recommend that the authors only present the results in which there is no clear overlap between the structured data included within the KGs for fine-tuning and the benchmark datasets. The authors should omit all results where data leakage between the KGs and benchmarks is certain or likely.**



2. Secondly, I found this paper generally difficult to follow. Specifically, I believe the study could be more clearly motivated within the introduction, and Section 3 ("Approach") could be more clearly explained. Below, I list some specific points for improvement: 

- Paragraph 1 and the first half of paragraph 2 of the introduction seem loosely connected to the rest of the motivation; I suggest cutting these paragraphs and skipping more directly to the points in paragraph 3. 

- Figure 1 is problematic for several reasons. First, it is presented in the middle of the introduction, which is an inappropriate place to present results. Secondly, it is presented with no experimental context. Finally, it is not clear whether the figure even supports the claims being made in the introduction. I suggest moving or removing this figure. 

- RAG systems are presented as less desirable alternatives due to a reliance upon the quality of the documents provided, but the authors acknowledge that KGs may also have incorrect or incomplete information. Therefore, the motivation for the work seems unclear. 

- Section 3.2 on the Review section is extremely difficult to read. I believe this could be made clearer through more structure within the subsection (sub-headings or numbered steps). Perhaps the textual structure could correspond to flagposts or landmarks within Figure 2, so that the reader can refer back and forth between the text and Figure 2 easily. 

- Although fine-tuning with a KG is an integral part of the methodology, PrimeKG is not mentioned anywhere within Section 3. Until I read the Supplementary materials, I had understood that the KG being used for fine-tuning was the set of triplets derived from the "Generate" action. The authors should make it clearer that the KG comes from another, external source.

questions:
1. page 3, Section 3: The authors give an example of a possible question and a set of possible answers. However, wouldn't this question warrant a yes/no response, rather than the set of answers provided? 

2. page 5: On which prediction task was TransE trained?  

3. page 5: Why have the authors chosen TransE for the KG embeddings? 

4. page 5: Where does the description dictionary D(r) come from? I do not see this in the appendix, and there is no appendix section 8. 

5. page 6, section 3.3: How, specifically, does the Revise action "adjust the triplets in F to include more knowledge"? 

6. page 8, Fig. 3: What do the blue v. orange plots mean? The only y-axis label I see is accuracy, and I see no key for blue v. orange.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730796723856

thread_type:
official_review

<thread_object>
creator: reviewer_xbfb

<content>
summary:
This paper studies the integration of factual knowledge with free text information into LLMs. It introduces KGAREVION, a medical Question Answering (QA) agent, which additionally fine-tunes a LLM on a knowledge base completion task to improve its performance on evaluating medical facts. For an input question the agent can perform three actions:
 1. **Generate** relevant medical fact candidates 
 1. **Review** the generated fact candidates to check factual incorrect facts
 1. **Revise** to add more facts and change the factual incorrect candidate from Review
The outputs of the last action are used as part of a prompt to generate the final answer. The agent is compared on multiple choice and open ended questions against several baselines. It is able to reach state of the art performance.

strengths:
**Medical Application**

The paper study the application of an LLM-based agent to a complex and difficult domain, viz. medicine. I appreciate that the authors took up this challenge to make a relevant contribution. 

**Evaluation**

The presented approach is evaluated thoroughly. It covers a comparison against a large number of competitors on open-ended and multi-choice reasoning. It shows that the presented approach is able to reach state of the art performance and moves the bar.  Apart from this performance analysis the paper also offers an ablation study and sensitivity analyses.

weaknesses:
Complex approach: 

The presented approach is complex and requires a lot of steps and technical components. It depends on a separate entity recognizer to map entities to UMLS, two LLMs performing different tasks (generation and revision, review), and a knowledge graph for fine-tuning. This reduces reusability, since the entire system would need to be replicated. Reducing all this into one LLM and dataset would make the contribution for accessible for others.

questions:
Did you consider using the facts and relations from UMLS for the KB completion task?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731591289472

thread_type:
public_comment

<thread_object>
creator: ~mingyu_huang1

<content>
title:
A Public Comment

comment:
I have taken some time reading this manuscript as I am myself interested in AI4Healthcare. While I found certain aspects of this work to be valuable, I however had several concerns as listed below that I want to discuss with the authors.

**Evaluation (data leakage).** After reading this manuscript I have the same concern as **reviewer jrR1**. To add to his/her comments, using a fine-tuned LLM to enhance reasoning, while intuitive, would pose significant concern to the fairness of the evaluation results on general medical benchmarks due to potential data leakage. While I understand that this is too big a question to be thoroughly addressed in a conference paper, the authors could have either $i)$ attempted to quantitatively measure the extent of the data leakage [1], $ii)$ more carefully designed the experiments to avoid this, or $iii)$ at least thoroughly discussed this as an important thread to validity of the results (though given the direct impact of these on the validity of the evaluation, this may not be sufficiently convincing) [2]. Yet it seems that these are not presented in the current manuscript. This makes it hard for me to interpret the results: whether the reported improvement by KGARevion is actually because of the its technical advancement, or has been simply benefited from the data leakage?

**Reliability 1 (KG quality).** The authors criticized existing RAG-based approaches for being susceptible to the quality of retrieved information. However, KGARevion also heavily relies on an external KG to fine-tune the LLM, whose quality would affect the performance as well as reliability of the approach. This makes the point that the authors intend to argue seem questionable. Notably, there has been recent works showing that existing KG can be easily "poisoned" by factually incorrect information presented in the source data used for constructing it [3]. In light of this and the giant scale of existing KGs, it is highly possible that the KG used in KGARevion also contain various unreliable information. It remains unclear how prevalent are such errors and how would they effect the performance of KGARevion. I am interested in these questions as they are important for a system targeted at the health domain.

**Reliability 2 (LLM for entity extraction).** Another concern I have regarding reliability is the generation step (Section 3.1). Here the authors used an LLM for extracting medical triplets from the questions. Considering the well-known hallucination issues of the LLMs, it is very likely that the LLM used here may "invent" some entities that do not actually exist in the original question. The current manuscript does not touch on this. Also, how could you ensure that the LLM can extract EVERY relevant triplet in the questions, and if not, how this will impact the performance of KGARevion? To compare, LLM-based entity extraction is the core of [4], and the authors have employed specific strategies (i.e., multi-round prompting with follow-up questions) to avoid hallucination and ensure completeness.

**Explainability & trustworthiness.** Another concern I have regarding KGARevion is the lack of explainability. Since the veracity checking of triplets in KGARevion heavily relies on LLMs and KG embeddings, which are both black-boxes themselves, it would make it hard to justify why a provided answer is favored by KGARevion. This issue is especially critical in the medical context given its high-stakes nature. From my experience talking to clinicians in the UK, they would prefer a system that could faithfully justify the rational behind a decision rather than a high-performance black-box that you never know how it makes decisions. In this aspect, I think more conventionally yet transparent computational fact-checking methods like the Knowledge Linker (KL) or KG-Miner may offer more desirable behaviors and could at least be discussed in the paper. I do not think SOTA performance is the only measure for the success of a computational approach in the medical domain. Given these, I am wondering whether the authors have considered graph topology-based methods like KL or KM. How would they perform with KGARevion? Is there any relevant experiments data? And, how did the authors balance between performance and explainability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732490831197

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
Thank you to all the reviewers for their thoughtful and insightful feedback\! We are pleased that reviewers are excited about the novel contributions of our work. Reviewers remark that “**KGARevion takes up the challenge of complex and difficult medical reasoning, making a relevant contribution**” \[XbfB\], “**KGARevion approach seems versatile, able to be used other base LLMs, KGs, and other domains in general**” \[jrR1\], “**KGARevion tackles an important problem and has strong evaluation and clarity**” \[yzKB\], and “**KGARevion holds significant importance in the field of medical question answering**” \[pkDu\] . We thank the reviewers for the strong praise of our work and contributions.

We now highlight a few important points raised by reviewers that warrant inclusion in the general response:

### Highlights of New Experiments

In response to reviewers' comments, we have run four additional experiments. We have included these experimental details and results in individual rebuttals. We now briefly describe these experiments:

* **New MedQA Dataset** \[jrR1\]**:** We included an external medical QA dataset (e.g., afrimedqa\_v2, released Sep. 12, 2024\) and tested KGARevion on this new dataset. This analysis is important because it is verifiably free from data leakage, and we find that KGARevion improves the accuracy by 4.7%, compared with LLaMA3.1-8B.  
* **Additional Knowledge Graph** \[XbfB\]**:** We included an external KG (UMLS) and implemented the Review action in KGARevion with UMLS. We find that KGARevion is robust and compatible with any medical-related KGs since KGARevion (UMLS) improves the accuracy of LLaMA3.1-8B on all datasets.  
* **Removed Triplets vs. Kept Triplets** \[yzKB\]**:** We conduct experiments to verify the number of triplets removed vs. kept by the Review action. We find that KGARevion is capable of improving the accuracy of backbone LLM (e.g., LLaMA3.1-8B) and enhancing the medical reasoning by removing those identified ‘False’ triplets.  
* **Impact of irrelevant knowledge on KGARevion agent performance** \[yzKB\]: We add experiments to examine the impact of an out-domain KG (general-purpose KG with no medical domain-specialized knowledge) adopted in KGARevion. Even if KGARevion is used with an irrelevant or out-of-domain KG, KGARevion’s performance does not degrade substantially.

### Improved Writing for the 'Review' Action  
In response to reviewers' comments \[jrR1,yzKB\], we revised sections on ‘Review’ action to enhance readability and clarity. In addition, we provided more details about the implementation of ‘Review’ and ‘Revise’ actions. The modifications can be found in the Appendix.

### Broader Context  
In response to reviewers' comments \[yzKB\], we added related work on ‘fact-checking’ to make our discussion of background and related work more comprehensive. These additions can be found in the Related Work section.

### Enhanced Figure  
We updated Figure 1 and Figure 2 to improve clarity and show the entire process reasoning process in the agent.

All updates are highlighted in blue in the revised PDF.

We thank all reviewers for their thoughtful commentary. We worked hard to improve our paper, and we sincerely hope the reviewers find our responses informative and helpful. If you feel the responses have not addressed your concerns to motivate increasing your score, we would love to hear what points of concern remain and how we can improve our work. Thank you again\!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors introduce "KGARevion" for medical question answering. The method entails first generating structured triplets conditioned on a question and then verifying these against trusted knowledge bases. There was consensus this approach is novel and may have application beyond the immediate task. Moreover, the empirical results are promising (and concerns about leakage appear to have been adequately assuaged in discussion). Further, MEDDDX, which includes decoy options which are "semantically similar" to reference answers is a nice contribution unto itself. 

I would appreciate discussion around the envisioned use of the model and associated risks. Even if KGARevion improves model factuality, it will still sometimes make mistakes; how should we think about using such a method in practice, especially in the absence of transparency (an issue raised by yzKB and the public commenter)? Some discussion of such issues and risks of using medical QA systems in practice should really be included in the work.