KGARevion: An AI Agent for Knowledge-Intensive Biomedical QA
Authors: Xiaorui Su
Yibo Wang
Shanghua Gao
Xiaolong Liu
Valentina Giunchiglia
Djork-Arné Clevert
Marinka Zitnik
Venue: ICLR 2025 Poster (2025)
Keywords: Medical Reasoning
Medical QA
Agent
Knowledge Graph
LLM
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: A knowledge graph based agent for knowledge-intensive biomedical question answering

Abstract: Biomedical reasoning integrates structured, codified knowledge with tacit, experience-driven insights. Depending on the context, quantity, and nature of available evidence, researchers and clinicians use diverse strategies, including rule-based, prototype-based, and case-based reasoning. Effective medical AI models must handle this complexity while ensuring reliability and adaptability. We introduce KGARevion, a knowledge graph-based agent that answers knowledge-intensive questions. Upon receiving a query, KGARevion generates relevant triplets by leveraging the latent knowledge embedded in a large language model. It then verifies these triplets against a grounded knowledge graph, filtering out errors and retaining only accurate, contextually relevant information for the final answer. This multi-step process strengthens reasoning, adapts to different models of medical inference, and outperforms retrieval-augmented generation-based approaches that lack effective verification mechanisms. Evaluations on medical QA benchmarks show that KGARevion improves accuracy by over 5.2% over 15 models in handling complex medical queries. To further assess its effectiveness, we curated three new medical QA datasets with varying levels of semantic complexity, where KGARevion improved accuracy by 10.4%. The agent integrates with different LLMs and biomedical knowledge graphs for broad applicability across knowledge-intensive tasks. We evaluated KGARevion on AfriMed-QA, a newly introduced dataset focused on African healthcare, demonstrating its strong zero-shot generalization to underrepresented medical contexts.

Published as a conference paper at ICLR 2025
KGAREVION: AN AI AGENT FOR KNOWLEDGE-INTENSIVE BIOMEDICAL QA
Xiaorui Su! Yibo Wang² Shanghua Gao"
Xiaolong Liu?
Valentina Giunchiglia
Arné Clevert Marinka Zitnik
'Harvard University *University of Illimois Chicago 'Imperial College London xiaorui suthms harvard. edu, ywang6338uic.
Pfizer
shanghua gao@hms harees..aaaaaae hhavard..dde wiw.duixxii222ei...ee. v. w.giunchigliaa@e wwteeeeeeerrraa...mmme ac. Djork-Are -rnne. Cweeert com, marinka haw.r
harvard.
ABSTRACT
Biomedical reasoning integrates structured, codified knowledge with tacit, experience-driven insights. Depending on the context, quantity, and nature available evidence, researchers and clinicians diverse strategies, including rule-based, prototype-based and case-based reasoning. Effective medical Al models must handle this complexity while ensuring reliability and adaptabil- ity. We introduce KGAREVION, a knowledge graph-based agent that answers knowledge-intensie
questions. Upon receiving a query, KGAREVION generates relevant triplets by leveraging the latent knowledge embedded in large language model. It then verifies these triplets against grounded knowledge graph, filtering out errors and retaining only accurate, contextually relevant information for the final answer. This multi-step process strengthens reasoning, adapts to different models of medical inference, and outperforms retrieval-augmenned generation- based approaches that lack effective verification mechanisms. Evaluations on medical QA benchmarks show that KGAREVION improves accuracy by over 5.2% over 15 models in handling complex medical queries. To further assess its effectiveness, curated three new medical QA datasets with varying levels semantic complexity, where KGAREVION improved accuracy by 10.4%. The agent integrates with different LLMs and biomedical knowledge graphs for broad applicability across knowledge-intensin tasks. We evaluated KGAREVION on AfriMed-QA, newly introduced dataset focused on African healthcare, demon- strating its strong zero-shot generalization to underrepresented medical contexts.
INTRODUCTION
Biomedical reasoning requires integrating diagnostic and therapeutic decision-making with derstanding of the biology and chemistry of the disease of drugs (Patel et al.) (2005). Large language models (Opnn)) 2024; Dubey al. [222 Gao et al. 2024) demonstrate strong general capabilities, but their responses to medical questions often suffer from incorrect retrieval, omission of key information, and misalignment with current scientific and clinical knowledge. These models can struggle to generate contextually relevant answers that account for local factors, such as patient demographics, geographic variations, and specialized biomedical domains 2023). A limi- tation arises from their inability to integrate multiple types of evidence by combining structured, codified scientifie knowledge with tacit, noncodified expertise-clinical intuition, case-based ex- perience, and learned heuristics, which essential for contextualizing scientific evidence within real-world medical decision decision-making (Harris 2023).
LLM-powered QA models often lack the multi-source and grounded knowledge required for effec- tive medical reasoning. The answer to complex medical questions requires an understanding of the intricate and highly specialized nature of biomedical concepts. However, LLMs trained in general- purpose data struggle to solve medical problems that require specialized knowledge in the domain. This challenge stems from their inability differentiate subtle, domain-specific nuances of medical

---

Published as a conference paper at ICLR 2025
b
Basic
Intermediato
Expert
0.6
Basic
Toaeerioi balveam
Internidate Expert
eccrouaci the leaal
0.5
godci palcka
060200
saleer
Accurac
2 Cuprc do
(1 Mroa
0.4
0.3
X
KG
LLaMA3-88
LL-MA3.1-88 GPT-4-Turbo
LIN
X
UN
X
Figure 1: a) Performance of existing LLMs three new datasets (MedDDx Basic. MedDDx- Intermediate, MedDDx-Expert) introduced in this paper with questions of varying difficulty. b) Sample questions from the new datasets.
significance. As a result, LLMs often do not reason effectively in complex scenarios, where success- ful inference requires recognizing and reasoning about dependencies between multiple interrelated medical concepts within a single question and interpreting highly similar yet semantically distinct biomedical entities with precision, as illustrated in Fig. !
To address these limitations, researchers have turned to vetrieval-augmented generation (RAG), which follows a shriiev--then-nnswer paradigm (Shi et al. 2024). These methods enrich LLMS with external knowledge sources, allowing them to retrieve information from biomedical databases and structured repositories (Fan et al. 2024). However, the accuracy of RAG-based answers heav- ily dependent on the quality of the retrieved content, making them prone to errors (Karpukhin al.) 2020). Many biomedical knowledge bases contain incomplete, outdated, or incorrect information, leading to unreliable retrieval Adlakha et al. (2024); Thakur et al. 2023). Furthermore, approaches based on RAG lack
verification mechanisms, leaving them unable to assess whether retrieved information is factually accurate, contextually relevant, or if it omits essential informa-
tion (Zhao et al. 2023).
Knowledge graphs (KG) of medical concepts serve as grounded knowledge bases that provide pre- cise and specialized in-domain knowledge for medical QA models (Qi et al. 20224] Murali et al.) 2023 Chandaa et al. 2023). Although KGs improve these models, they often contain gaps, limiting their ability to capture biomedical relationships. Approaches that retrieve medical concepts based solely on direct associations (edges) in a KG fail account for implicit or complex relationships. For example, two proteins with different biological functions may lack a direct connection in KG, even though they share biological similarity at the molecular level (Menche et al. 2015). Advancing L..M-powered models for knowledge-intene intensive medical QA requires systems that simultaneously capture complex associations among multiple medical concepts, integrate multi-sourge knowledge systematically, and verify retrieved information to ensure contextual accuracy and relevance.
Present work. We introduce KGAREVION, a KG-based LLM agent designed for complex biomed- ical QA that integrates the non-codified knowledge of LLMs with the structured, codified knowledge found in KGs. As illustrated in Fig. 2 KGAREVION executes four key actions to ensure accurate and context-aware biomedical reasoning. First, it prompts the LLM generate relevant triplets based on the input question. To effectively leverage structured KG data, KGAREVION fine-tunes the LLM on KG completion task, incorporating pre-trained structural embeddings of triplets as prefix tokens. The fine-tuned model then evaluates the correctness of the generated triplets. Next, KGAREVION performs a 'Revise' action to correct erroneous triplets, refining the knowledge base before selecting the final answer. Given the complexity of medical reasoning, KGAREVION adap- tively chooses the most appropriate reasoning approach for each question, allowing for nuanced and context-aware QA. This flexibility enables KGAREVION to handle both multi-choice and open- ended QA tasks.
Our key contributions include: ! Developing KGAREVION, a versatile KG agent that dynami- cally adjusts reasoning strategies, achieving a 6.75% improvement over 15 baseline models in seven datasets, including three challenging newly curated benchmarks. ( Demonstrating that ground- ing through generated triplets significantly enhances KGAREVION's capabilities across multiple KGs. 2 Showing that KGAREVION effectively answers complex, knowledge-intensive medi- cal queries in both multi-choice and open-ended QA formats. 4 Evaluating KGAREVION on African healthcare datasets: We benchmark KGAREVION on AfriMed-QA, newly introduced
2

---

Published as a conference paper at ICLR 2025
dataset focused on African healthcare. The results highlight KGAREVION's strong zero-shot generalization, demonstrating its ability to reason effectively in undertepresented medical con- texts. ( Analyzing robustness to input variations: We analyze KGAREVION's sensitivity to changes in question structure, answer ordering, and answer relabeling. Unlike LLMs, which ex- hibit high variance when answer choices are reordered, KGAREVION maintains stable perfor- mance, demonstrating its stronger robustness in real-world settings. KGAREVION is available at hutp:://gicuu..com/n/n
http:///wwooootaaaaaaaaaaaAArar harvard/KGARevion
RELATED WORK
LLM-based reasoning. General- phurras- LLMs (GPT (OpenAl 2024),
LLaMA family (Dubey
et al. 2024 Touvron et al. 2023). Mistral Jiang et al., 2023)) and LLMs fine-tuned on biomedi-
cal data (BioMedLM (Venigalla et al. 2022). Codex (Liévin al.
2024). MedAlpaca (Han et al.
2023). Med-PaLM (Singhal et al. 2023). PMC-LLAMA (Wu et al.
2024a ) leverage their vast em-
bedded knowledge to perform medical reasoning. Some models enhance reasoning by decomposing complex queries into sub-tasks, solving them step by step using structured prompts, as seen in Chain-of-Thought (CoT) (Wei et al., (024) and CODEX CoT (Gramopadhye et al.) 2024). How- ever, these methods struggle with domain-specific knowledge, leading to gaps in accuracy and completeness.
medical queries that require multi-source
RAG-based models. Self-RAG (Asai al. 2024 is a framework that enhances LLM performance through retrieval and self-reflection. LLM-AM' (Wang et al.) 2023b) improves medical question answering by integrating authoritative medical textbooks into LLMs with specialized knowledge retrieval and self-refinement techniques. Adaptive-RAG (Jeong et al. 2024) introduces a dynamic RAG framework that adapts retrieval strategies based the complexity of the questions. However, its performance is restricted by the quality of the knowledge retrieved (Zhang et al. 2024).
KG-based models. Models, such as QAGNN (Yasunaga et al., 2021). JointLK (Sun et al. 2022 and Dragon (Yasunaga et al., 2022). handle medical questions solely using KGs in an end-to-end manner. However, these methods cannot be easily applied to questions involving unseen nodes or incomplete knowledge within the graphs. In addition, structured KGs have driven research toward graph-based RAG models, motivating models such as GraphRAG (Edge at. 2024). KG-RAG (So- man et al.] 2023), and MedGraphR AG (Wu et al.) 2024b). KG-Rank (Yang et al.) 2024) ranks the retrieved triplets and filters out irrelevant knowledge to improve the accuracy of the search. Addi- tionally, GenGround (Shi et al. 2024) uses a teerratttthe-Grruunt pipeline that grounds answers by prompting LLMs to validate retrieved knowledge, However, these approaches rely heavily on semantic dependencies, overlooking the rich structural information within the KGs.
3 KGAREVION AGENT
Given set of biomedical questions Q each question consists of question stem q and set of candidate answers C. For example, in Fig. (2h, the sample question has the stem q q= "Which gene interacts with the Heat Shock Protein 70 family that acts as molecular chaperone and is implicated in Retinitis Pigmentosa 59 due to DHDDDS mutation?" along with a set of semantically related candidate answers C {HSPA4, HSPA8, HSPAIB, HSPAIA). The goal to identify the correct answer using both an LLM (denoted as P) and a knowledge graph (KG, denoted as G). Here, a KG represented as a of triplets G [(h,r, r), where each triplet consists of head entity (h), a relationship (r), and tail entity (t). Table H2 provides summary of the notation. We consider both multiple-choic choice and open- reasoning settings (see Results).
KGAREVION is an LLM-powered agent (Wu et al. 2023 Li ct al. 2023) that defines agentic ac- Shen et al. 2023 Nakano et al. 202 1) to co collaboratively perform complex
tions (Schick et al. 2023
tasks (Tang et al. 2023: Bran et al. 2023) Boiko et al. (023). Fig. provides an overview of
KGAREVION, which operates through four key actions: Generate ($3.) Generates triplets rele- vant to the input question), Review ($3.) Assesses the correctness of each generated triplet), Revise ($3 Corrects any triplet identified as incorrect), and Answer ($3.3) Produces the final answer based on the triplets verified by the Review action). This structured reasoning process allows KGARE- VION to integrate both --g-gerrrtted knowledge and structured KG-based validation, improving accuracy and robustness in Knowledge-intensive medical QA.
3

---

Published as a conference paper at ICLR 2025
Watch grew intenace with Heat Shook and in irpiiated in Rediinils growntoss cou mutesion??
Review
[MEIAA interact,, DR1205)
MMM
Algredd pribodiing
A HSRP TI: HETPI E:
HSSPARR IB DI ISPATA
Know odge graph
Dewcrtation dedlorary
FFN
KGARevion
Dasccipptios arnoues with ppreesi
broding moduladion or other form
* Genciale
aution
Prosiorned eribodting
Iran protpr a
Disanigraa tokena
Matrull
() 0.5AWiA Intrnciias D-600S)
Aligning Embedding
100A3 bearncs, DIEDOS
Natrrctioon Gles Viee hors a browlldge graat
Matmul
proliita
HORAAIA anvondiion CHERSS
Projecton
Tokko eriooddnng
ambbddingg
Instruction tokera
LUN
Premmined embrcding
Description tokend
AnMARE
Fire-tuning LLM
Loss
+ Frozen
Trainable
Jayy rubrita pigaeeioo)
Figure 2: a) Overview of KGAREVION agent. b) Overview of fine-tuning in the Review action.
3.1 GENERATE ACTION
The Generate action aims gather comprehensive structured knowledge from input questions. Specifically, this action first identifies all medical concepts involved in the input question stem and then generates a set of triplets related to the question based on the extracted medical concepts.
Depending on the content of the answer candidate, the input questions can be broadly categorized into two types: choice-aware and non-choice-aware The answer candidates in the choice-aware group have specific contents, whereas the ones in the non-choice-aware group only contain yes-or-no options (as shown in Appendix Table[4). These different types of questions require distinct reasoning processes: choice-aware questions involve analyzing the content of each answer candidate, while non-choice-awar questions only require focusing on the question stem.
To handle this, this action is designed to prompt the LLM (uuyng al. 2022 Wang et al. 2023a) to follow different procedures for generating relevant triplets according to the input question type.
For choice-aware questions, the Generate action generates triplets based on the contents of each answer candidate and extracted medical concepts in question stem For non-choice-awan questions, the Generation action directly generates triplets based on medical concepts presented in question stem
The rationale behind this design is that LLMs have inherent biases in their knowledge, often gener- ating more detailed information on familiar topics compared less familiar ones when all answer candidates are presented simultaneously (Dai et al. 2024). Additionally, this approach helps reduce the impact of the order in which the answer candidates are presented. The process of the Generate action can be formulated as:
T aP)..a)). s [C], if (Yes, No, Maybe} P(g),
if S (Yes, No, Maybe]
where a; denotes the candidate answer in C, and the LLM P(-) prompted extracts triplets from the medical concepts involved in its input.
3.2 REVIEW ACTION
To enable LLMs to accurately judge the correctness of generated triplets, beyond relying solely on semantic dependencies inferred by LLMs (Shinn et al. 2023), the Review action also leverages the connections and relationships among various medical concepts contained KGs. This is achieved by fine-tuning the LLM on a KG completion task, explicitly integrating entity embeddings learned from KGs into the LLM. Then the Review action is performed by the fine-tuned LLM to assess the correctness of triplets generated by the Generate action, shown in Fig. 2.
4

---

Published as a conference paper at ICLR 2025
3.2.1 FINE-TUNING STAGE
Generating KG Embeddings and Triplet Descriptions. We use the well-known KG representa- tion learning method, TransE (Bordes et al. 2013), to learn structural embeddings for both entities and relations in G. For a triplet rt, G. the learned pre-trained embeddings are denoted as ej E IR E IR and e, E R" where d represents the embedding dimension. These embeddings are kept fixed during LLM fine-tuning. In addition, we instruct the LLM to generate description template for each relation E G. We store these descriptions in dictionary D(.), which can be found in Appendix Table 8
Aligning Embedding. Since the embeddings in LLMs are based on token vocabularies (Radford et al. 2019), LLMS cannot directly interpret pre-trained structural embeddings, as they lack se- mantic meaning. To make use of the pre-trained structural embeddings, we align them with the corresponding descriptions to generate new embeddings for the input triplet.
Specifically, given the description Drr) for the input triplet (h, r,,, we denote the embedding of Dr)) obtained from the LLM as X E Rlll xd1, where ||] is the maximum number of tokens and dy is the embedding dimension in the LLM. Next, we concatenate the embeddings of the head entity, the relation, and the tail entity, denoted as V = [g(en): g ((
33xx2, where g(.) R' - IR We then apply an attention block (Vaswani 2017), followed by a two-layer feed-forward network (FFN, shown in Fig. 2b) obtain the aligned triplet embedding matrix Z Rx3dd as follows:
V = V + (VX" X
(2)
Z= V (V)())
(3)
where (-) is the Softmax function, cp(.) represents layer normalization, W1 Rddx xda and W2 E Rdd xdz are trainable parameters in the FFN, and da is the dimension the hidden layer in the FFN.
Fine-tuning LLM. After obtaining the aligned embedding Z. we add it to the beginning of the instruction and fine-tune the LLM using LoRA (Hu et al.) (022) with the next-token prediction loss (Radford] 2018). The instruction is: 'Given triple from a knowledge graph. Each triple consists of a head entity, a relation, and a tail entity. Please determine the correctness of the triple and response with True False." The output should be either True False
3.2.2 INFERENCE STAGE
The fine-tuned LLM is then integrated to the Review action check the accuracy of each triplet in T. which was generated by the Generate action (3.I). Specifically, we first use UMLS codes (Bo- denreider 2004) to map entities in the KG and obtain pre-trained structural embeddings for the head entity, relation, and tail entity, respectively. These embeddings, along with their descriptions and instructions, are fed into the fine-tuned LLM to determine whether the triplet is correct not.
However, not all entities in the generated triplet (h, r, t) E T can be mapped to entities in KGs. To address this, the Review action applies soft constraint rule to distinguish whether the generated triplet is factually wrong or the result incomplete knowledge in KGs, as follows:
Factually Wrong: if we can map h and entities in KGs and the output of fine-tuned LLM is False, then the triplet (h, r, 1) is factually wrong and is removed from T.
to entities in KGs, then the triplet (h,r, is considered incomplete knowledge and is kept.
Incomplete Knowledge: if we cannot map either
In this way, the triplet in T' can be grouped into two categories, i.e., the True triplet set V and False triplet set F. where T = V U F and V F F
3.3 REVISE AND ANSWER ACTIONS
If F has triplets, KGAREVION calls the Revise action to adjust the triplets in to include more triplets covering more medical concepts that help with the answering of the input question. The new generated revised triplets are then reviewed by the Review action to make sure that they are correct and related to the input question. If the Review action outputs "True", then the revised triplets are added to the set of True triplets V. Otherwise, KGAREVION continues to call the Revise action until the max round (k 2 is achieved.
5

---

Published as a conference paper at ICLR 2025
After obtaining the set of True triplets from the Review or Revise actions, KGAREVION finally calls the Answer action to prompt the LLM to generate the answer to input question based on the verified triplets in V.
4 RESULTS
Datasets. We first start with four multi-choice medical QA benchmarks (Xiong al., 2024a) (Table D. In addition, we introduce a new benchmark for multi-choice complex medical QA fo- cused on differential diagnosis (DDx), named MedDDx. We begin by collecting questions and corresponding answers from STaRK-Prime (Wu et al. [024c). For each question, we then se- lect the top three entities with the highest semantic similarity to serve as additional answer can- didates. MedDDx comprises a total of 1,769 multi-choie QA samples. Based on the standard deviation of semantic similarity between answer candidates and the correct answer, we catego- rize the dataset into three difficulty levels: MedDDx-Base MedDDx-Intermedine Expert (The samples each dataset are shown in Fig. and details are available in Appendix (5)
and MedDDx-
Baselines. We consider eight LLM-based reason- ing models, four RAG-based models, and three KG-based models. The LLM-based reasoning models include LLaMA (2-7B/13B, 3-8B, 3.1-8B) 2024)) Mistral
Table I: QA benchmarks and three new Med- DDx datasets. 'OR' indicates whether the open-ended reasoning evaluation is done.
(Tourron et al.) 2023; Dubey et al.
Datasets
Size
QA
OR
Jiang al. (002),, MedAlpaca (7B) (Han et al.
MMLU-Med
1,089
A/B/CID
2023). PMC-LLaMA (7B) (Wu et al.,
2024a), and 2024). The
MedQA-US
PubMcdQA*
500
Yes/No/Maybe
LLaMA3-OpenBioLLM
(Ankit Pal
BieASQ-Y/N
618
Yea/No
(Asai et al. RAG-based models include RAG
McdDDx-Basic
483
A/B/CID
2024 MedRAG (Xiong et al.) 2024b). KG-RAG
leteDDx-Inermmddite
A/B/CID
(Soaan et al.] 2023). and KG-Rank (Yang et al.)
McdDDs Expert
245
A/B/CID
2024]. The KG-based models include QAGNN
Yasunaga et al. 2021), JointLK (Sun et al. 2022). and Dragon Yasunaga et al. 2022)
Evaluation setup. We consider two evaluation settings. Multi-choice reasoning: This setting eval- uates the model's performances on all collected multi-choice QA datasets. The model is tasked to select the correct answer to a user input question from a set of candidate answers. Open-ended reasoning: All candidate answers are masked, meaning that the model has generate response to the input question independently without being presented with a set of candidate answers. The model produces an answer solely on its own generated response. Additionally, evaluation scenarios for each setting to test model abilities in solving complex medical questions by considering the number of medical concepts and the semantic similarity among answer candidates. Query complexity scenario (QSS): This is hard evaluation scenario to test how the model per- forms with the increase of the number of medical concepts present in a question since the question often becomes more intricate, requiring more nuanced inferences between concepts to achieve the correct answer, with the increase of medical concepts. Semantic complexity scenario (CSS): This is harder evaluation scenario that tests the model's ability to identify the correct answer among semantically similar and closely medically related candidate answers.
design two new
4.1 MULTI-CHOICE QUESTION-
Table shows the accuracy and variance of KGAREVION and all baselines on all datasets. Eval- uation on four gold standard medical QA datasets shows that KGAREVION improves the average accuracy by over 4.8%, outperforming all basclines in handling medical queries.
Results under QSS. Fig. 3h illustrates the trends of KGAREVION alongside the top-performing baselines the number of medical concepts increases. It can first be observed that KGAREVION outperforms baselines of the same size, regardless of the number of medical concepts involved. Moreover, KGAREVION maintains stable performance the number of medical concepts increases and even improves when processing questions with n = 6, compared to those with n 5. In contrast, baseline models struggle with complex questions containing 5 or 6 medical concepts. Such an

---

Published as a conference paper at ICLR 2025
Table 2:
The accuracy of KGAREVION and all baselines on four gold standard and three newly created datasets under multi-choice reasoning settings. The value highlighted in Blue indicates the best result among LLM-based reasoning models with a size smaller than 8B, including LLaMA3-
8B, while Red marks the top value among .LLMA3.1-8B and other types of baselines. std means the standard deviation under three runs.
Multi-choice Reasoning
Established Medical QA Benehmarks
McdDDx
Mothod
MMLU-Mai
McdQA-US
PubMedaQ*
Basic
Intermedistic
Export
Metrics
Acc. (std)
Acc. (std)
Acc. (std)
Acc. (std)
Acc. (std)
Acc. (sid)
Acc. (sid)
LLaMAZ-78
(.376(008)
0.281 (004)
0,448 (018)010)
0.215(030) (.565(000)
0.198 (004)
(.192(012)
11 aMA2277 (CoT)
(005)
0.251 (002)
0.465 (011)
(011)
0.289 [010)
0.265 (005)
0.229 (025)
Mistral-88
0.634 (004)
0.477 (007)
0.400 (002)
0.644 (,00))
0.412
0.356 (003)
0.575 (007)
Mistral7BB (CoT)
(X033
(X05)
0 (.002)
(010)
0.368 (1123)
(027)
MedAlppaca-7B
0.500 (004)
0:101 (.01)
0.333 (0155)
(.193)
0.399 (012)
0.325 (004)
0.511 (.09))
MedAlppac-7B (CoT)
0.603 (0044)
0.399 (003)
0.315 (015)
0.485 (.125)
0.395 (107)
0.321 (011)
(.312
MMC-LLaMA-7B
0.207
0.247
0.179
0.346
0.057
0.086
0.079
PMC-IlaMA-7B (CoT)
0.204
0.208 (.002)
(.14)
(.208
(002)
0.077 (004)
(005)
1LaMA3-8B
0.634 (005)
0.566 (1004)
0.586 (000))
0.654 (,005)
0.428 (015)
0.319 (0022
0.306 (.009)
11MMA3888 (CoT)
0.651 (003))
0,552 (003)
0.574 (007)
0.681 (000)
0.434 (010)
0.368 (004)
0.313 (003)
1lame3-OpenBiolliMMM.
0.636 (005)
0.383
0.350
0.623
0.238
0.235
0.229
1amms--QpenBiollMMMM88 (C6T)
0.571 (003)
0.295 (002)
0.283 (001)
(.646((004)
(017)
(005)
0.527 (011)
LLaMA311-3
0.677 (007)
0.563 (1005)
0.396 (009)
0.687 (,000)
0.434 (018)
0.365 (0002
0.306 (021)
LLaMA331-SB (CoT)
0,681 (005)
0.549 (003)
0.600 (005)
0,705 (.002)
0.439 (017)
0.393
0.522 (014)
LLaMA2-13B
0,442 (X022)
(004))
0,252 (204)
0.455 (,002)
(00E)
0.338 (109)
(006)
LLaMA2-13B (CoT)
0.415 (002)
0.354 (005)
0.252 (006)
0.422 (.003)
0.309 (008)
0.263 (013)
0.243 (016))
QAGINN
0.317 (003)
0.450 (005))
0,439 (003))
0.644(000)
0.295 (001)
0 0.265 (007)
0.253 (003)
J K
0.288 (005)
0.472 (.005)
0.468 (007)
0.640
0.247 (0044)
0.250 (005)
0.244 (004)
Dragon
0,319 (003)
0.475
0,472
0,646 (.00)
0.286 (003)
0.247
0.240 (004)
Self-RAG (7B)
0.322 (019)
0.580 (028)
0.534 (028)
0,594 (.012)
0.238 (000))
0.199 (057)
0.221 (045)
Bale-RAG/ (13B)
0.502 (005)
0.408
0.331
0.646 (.100)
0.249
0.290 (015)
0.266
Rank (13B)
0.362
(.019)
(.015)
(021)
0.256 (013)
(010)
KG-RAG (8B)
0,516 (005)
0.343 (.001)
0,429 (011))
0.662 (.005)
0.434 (002)
0,413 (007))
0.391 (0044)
MedRAG (70B)
0,579 (015)
0.487 (0144)
0,574 (022)
0,719 (.008)
0.365 (0085)
0.348 (013)
0.527 (003)
KGAREVION (LLaMAS, wio Reviow)
0.621 (002)
0.528 (003)
0.356 (002)
0.713
0.310 (106)
0.354 (004)
0.313 (008)
KGAREVION (LLaMA33, wio Revise)
0.657 (004)
0.594 (006))
0.562 (.02))
((05))
0.386 (008)
0.372 (004)
0.327 (003)
KGAREVION (LLaMA3,
0.562
(.003)
(008)
0.404 (005)
(003)
KGAREVION (LLaMA3,, A ==
0.696 (006)
0.616 (0088)
0.566 (002)
0.723
0.457 (.006)
0.414 (005)
0.395 (000))
KGAREVION (LLaMA3, A 3)
0.678 (006)
0.628 (802)
0.590 (oU5)
0,737 (.007)
0.469 (.005)
0.451 (1044)
0,411 (005))
Improvement best baseline
+5,2%
+6,2%
+0,4%
+6.3%
+3,9%
+8.3%
+3,2%
KGAREVION (IL.MAS wio Review)
0.695 (006)
(002)
0.560 (004))
(.736(000))
0.298 (015)
0.299 (003)
(006)
KGAREVION (LLaMA3.11 wlo Revise)
0.716 (006)
0.573 (0005)
0.568 (.011)
0.749 (.003)
0.392 (.012)
0.357 (106)
0.352 (0055)
KGAREVION (LLLMA3,,1 A ==
0.734 (0044)
01618 (.002)
0.619 (.04))
0.763 (.01))
0.483 (013)
0.457 (010)
0.409 (005)
KGAREVION (LL.MA3.11.
0.720 (003)
0.616 (.05))
0.656 (006))
0:745 (0005)
0.395
0.454 (008)
0.342
KGAREVION (LL.MA3.1, =
0.716 0.716.0004
0.620 (.03))
0.638 (004)
0.749..000
0.469 (012)
0.411 (010)
0.447 (005)
Improvement over best baseline
+5.3%
+5.7%
+3.8%
+4.4%%
+4.4%
+4.4%
+5.6%
observation indicates that KGAREVION advances in handling complex medical questions involving
multiple medical concepts.
Results under CSS. As seen in Table 2 evaluations on three difficult levels in MedDDx indicate that KGAREVION exhibits a strong ability in handling differential diagnosis questions that request professional and accurate knowledge. In addition, the obtained results also show that KGAREVION excels in identifying the correct answer among semantically similar answer candidates since it im- proves the accuracy on MedDDx MedDDx-Expert Expert by 3.2% and 5.6% with LLaMA3-8B and LLaMA3.1-8B as the backbone LLM, respectively.
4.2 OPEN-ENDED REASONING
We transform multiple-choice questions into descriptive, open-ended ones to better simulate real- world medical scenarios, where such inquiries are more common (see details in Appendix B.4). This adjustment requires our model to generate responses without predefined choices, encouraging holistic reasoning and the integration of diverse knowledge sources. By removing answer choices, we can more effectively assess the reasoning ability of KGAREVION in complex medical situations, resulting in a more realistic evaluation of its capabilities. Table 3 shows the accuracy and variance across all datasets. The variance here denotes the difference in accuracy compared with that in the multi-choice reasoning setting.
Results under QSS. Fig. 3D shows the accuracy obtained by pure LLM and KGAREVION with the increase of medical concepts under open-ended reasoning setting. Compared to pure LLMs in the open-ended reasoning setting,
shows significant improvement in handling complex medical reasoning tasks involving more than 4 medical concepts and a comparable performance in questions with less than 3 medical concepts.
7

---

Published as a conference paper at ICLR 2025
Mull-choice reasoring setting
Open reasoring setting
0.8
08
0.8
0.6
0.7-
0.7-
0.-
EG004399 91
0.4-
L1
KOARerion (LLMAA-00)
05-
0.5
5
the number of medical concapts
the number of medical concepts
Figure 3: The accuracy of KGAREVION and pure LLMs with the medical concepts increase under a) multi-choice reasoning setting and b) open-ended reasoning setting.
Table 3:
The accuracy of KGAREVION and baselines on four gold standard and three newly created
datasets under open-ended reasoning settings. The value highlighted in Blue indicates the best result among LLM-based reasoning models with a size smaller than 8B, including LLaMA3-8B, while Red marks the top value among LLaMA3.1 1-8B and other types of baselines. AAcc denotes the difference in performance between the open-ended and multiple-choice reasoning settings.
Open-ended Reasoning
Open-ended Inquiries without pre-delined cholces
ModDDx-ho-OOg
(Open-ended)
Method
MMLU-Med
MedIAA-US
PubMedgAA
BloASQ-Y/N
Bosic
Insermediate
Expert
Metries
Ass. AAce
Aec. AAAc.
Auc. J.Ace.
Aoc. AAAc
Auc. AAcc.
Acc. AAcc
Aec. J.Ace
LLaMA2-7B
0.328 (40045)
0.302 (401011)
0.546 (+0099)
0.625 (-0055)
0.256.40007)
0.305 (-0 (09)
0,302 (-0 100
LLaMA2-7B (CoT)
0,362 (-0.044)
0.243 (0.005)
0.418(0,077)
0.642 (x0.255)
0.265 (00024)
0.270 co0.005)
0.250 (+0,055)
Mistral78
0.422 (4:065)
0.344
(40015)
0.349
0.228
0.273
Mistra-7B (CoT)
0.583 (.0051)
0.39% (40076)
0.122 (4.160)
0.657 (+0.006)
0.345 (40.159)
0232 (-0.16)
0.286 (4.003)
PMC-I1aMA-7B
0.033 (4)
0.882 (4:165)
0.090 (4008)
0.139 (4:201)
0.349 (611
0.191 (40.105)
0.232 (+0 150)
PMC MMC--LLMM-7BBBC (CoT)
0.080 (0.122)
0.079 (0.129)
0.092 (0.083)
0.125 (0.083)
0.220 (60.12)
0.245 (o0.168)
0.228 (+0.165)
LLaMA3-SB SB
0.595 (0:009)
0.458 0.005)
0.532 (0.05)
0.672 (+0.018)
0.343 (0005)
0.314 00005)
0.317 (-0,11)
LLaMA3-SB (Col)
0.608 (-0.44))
0.449 (003003
0.562 (4.002)
0.714 (.0.033)
0.289 (0.145)
0.304 (-0004)
0.327 (+0.014)
0.324 (4..11)
0.157 (4:126)
0,157 (4.199)
0.324 (4) 199)
0.16 (40222)
0.06 (4:220))
0.008 (4:222))
Lymma--OyenBill11MM-- (CWT)
0.398
0.146
0.100
0.098 (402
0.84
0.108 (4.319)
LLaMA3.1-8
0.607 (40.070)
0.551 (40003)
0.514 (.0.02)
0.694 (40.007)
(20.113)
0299 (40000)
0.338 (40.029)
11aMA3.1-8B (CiT)
(-01010)
0.563 (-00144)
0.572:0.00)
0,706 (4:000)
0.306 (40
0.294 (40090)
0.315 (6:00)
LLaMA2-13B
0,348 (.0.94))
0.283 (+0.000)
0,218 (0.034)
0,421 (00054))
0.190 (0.090)
0.123 (0.215)
0,153 (..164)
LLaMA2-13BB (CoT)
0 (0,104)
0,267 0007700:00 (0008)
0,266 (+00044)
0.471 (+01049)
0.269 (0:040)
0.268 (+0,005)
0.282 (+0,039)
SoIf-RAG (7R)
0.256 (400065)
0.235 (40
0.916 (4:218)
(4:2115)
(40001)
(40.047)
0.213 (6.011)
Self RAG (130)
0,309 (0.193)
0.257 (0.113)
0.488 (+0.107)
0,539 (00107)
0,212 (.000)
0.32 (00088)
0.226 ( (U040)
KG- Rank (138)
0.151 (-0.301)
0.199 (00177)
0.203 (4.112)
0.188 (40.215)
0.127 (4. 126)
0.139 (-0.125)
0.133 (40.101)
KG RAG (HB)
0.310 (402005)
0.290 (00003))
0,316 (4.113)
0,359 (.0.203)
0.216 (0.218)
0.220 (40.993)
0.213 (40175))
KGAREVION (LLaMA3, w/o Review)
0.645 (-0.024)
0.609 (+0081)
(0004)
0,701 (0012)
0,400 (-0,000)
0.360 (40.000)
0.356 (+0.083)
KGAREVION (LLaMA3 w/o Revise)
0.008 (-0.011)
0.626 (40:0522
0.572 (40.010)
0.716 (40.00)
0.426 (40.040)
0.403 (40.031)
0.412 (40.085)
KGAREVION (I...MA3,
1)
0.687 (-0.055)
0.678 (-0101))
0.578 (-0.016)
0.730 (40.014)
0.465 (40000)
0.430 (00.000)
0.478 (-0.05))
KGAREVION (LLaMA33, J
2)
0.682 (4b.014)
0U638 (-0.022)
0.566 (40,000)
0.736 (-0.013)
0L527 (40.000)
0.463 (40.049)
(L489 (-0.094)
KGAREVION (LLaMA3,
3)
0.696 (-0.015)
0.632 (+01004)
0,572 (40015)
0,733 (0000)
0.489 (00,000)
0.11 (00000))
0.429 (-0,01)
Improvement
bascline
+8.8%
+18.0%
+1.6%
+18.4%
+14.9%
+16.2%
KGAREVION L.LaMAA,,
Review)
0.659 (-0000))
0.526 (40:000)
0.556 (4/00)
(40010)
(401136)
0:439 (-0.112)
KGAREVION (LLaMA3.1, I, Rovise)
0.655 (.0.21))
0.626 (40.053)
0,556 (0.012)
(400013)
0.489 (10.007)
0,436 (40.099)
0.451 (+0.099)
KGAREVION (LLaMA3.1,
0.720 (..014)
0.644 (+0:220)
0,560 (0.09)
0.757 (0,000)
0.469 (0.014)
0.454 (0,00)
0.437 (+0,22)
KGAREVIoX (LLaMA3.1,
0,704 (-0.015)
0.636 (-0.020)
0.572 (40.04))
0.734 (40.011)
0 0.469 (40.073)
0.446 (-0.005)
0.432 (40.090)
KGAREVION (LLaMA3 F 3)
0.712 (400014)
0.639 (-00159)
0,562 (40075)
0,748 (40,00)
0:449 (-:000)
0.470 (40.009)
0.451 (-11204)
Improvment over best baseline Time
+2.4%
+8.1%
+0.0%
+4.9%
+16.7%
+17.6%
+11.6%
Results under CSS. To evaluate the model's ability to solve complex medical QA with differential diagnosis, we compare KGAREVION and all baselines on newly created datasets, as shown in Table 3 KGAREVION still achieves the best performance in the open-ended reasoning setting. In ad- dition, compared with the results in multi-choice reasoning setting, KGAREVION performs better. On the one hand, such a result demonstrates the strong ability of KGAREVION in the open-ended reasoning setting. On the other hand, it also indicates that these semantically candidates may affect the reasoning process to a certain extent.
4.3 ABLATION ANALYSES
Effect of the 'Review' action. As shown in Tables 2 and 3 and in Fig. (KGAREVION (w/o Re- view) vs. KGAREVION (w/o Revise)), the Review action plays a crucial role in answering medical questions across both reasoning settings. It improves the average accuracy by 9% in the multiple- choice setting and by 4% in the open-ended setting. Fig. I that the Review action has more pronounced effect on the MedDDx dataset than the other four datasets in both reasoning settings. This suggests that incorporating the Review action enhances KGAREVIONT performance com- plex medical questions. Furthermore, the Review action leads to greater accuracy improvements in the gold-standard datasets under the open-ended reasoning setting than in the multiple-choice setting. These findings emphasize the importance of verifying candidate answers for open-ended reasoning, where retrieval and knowledge validation critical for accuracy.
8

---

Published as a conference paper at ICLR 2025
a
Mul chaice reasoning setting b
Open reasening setting
Effect of refinement rounds in the 'Revise' action. The Revise action enhances accuracy by iteratively correcting erroneous triplets un- til the Review action verifies them as correct. "Talles 2 and 3 along with Fig- 4 demonstrate that KGAREVION benefits from this refine- ment process across both reasoning settings. Fig. shows that the Revise action significantly improves performance on the MedDDx dataset, yielding accuracy gains of 3.3% and 3.0% in the multiple-choice and open-ended settings, respectively. We further analyze the impact of the number of revision rounds across all datasets
We Revdew
Reviee
Resise
KGAlledon
0.
692
0.68
Accunss
0.93
5.05 3.07
0.-
C.8-
0.42
3.46
0.4
0:31
0.4-
Cold standard dilesets ModDOx
Cold standard Colanets MezDON
Figure 4: datasets under two settings.
The results of ablation studies across all
in both settings, as detailed Tables 2 and 3 The results indicate that KGAREVION achieves optimal performance with & 1 in multiplechhoiee reasoning datasets. However, it benefits from additional rounds of refinement when handling complex questions, such as those in the MedDDx- Expert dataset. KGAREVION typically requires more iterations refine and verify the correct answer for open-ended reasoning, highlighting the importance of iterative knowledge validation in knowledge-intensivee medical QA.
4.4 VERSATILITY OF KGAREVION
KGAREVION integrates with different LLMs. KGAREVION a versatile agent that operates effectively with various LLMs. We implement KGAREVION using three models: LLaMA3-8B, LLaMA3.1-8B, and GPT-4-Turbo. The averaged results across all datasets, shown in Fig. 5 demonstrate that KGAREVION consistently enhances the performance of its backbone LLMs, im- proving accuracy by 6% with LLaMA3-SB, 7% with LLaMA3.1-8B, and 2% with GPT-4-Turbo These improvements highlight the adaptability of KGAREVION across different model architec- tures, reinforcing its effectiveness in knowledge- intensivee QA.
KGAREVION adapts to different KGs. The Review action in KGAREVION verifies gener- ated triplets using structured knowledge from KGs. To evaluate how different KGs im- pact performance (see Appendix C.5 for de- tails), we implement KGAREVION with two KGs and assess its performance across all datasets, as shown in Fig. 5p. The results show that KGAREVION maintains strong per- formance regardless of the KG used, demon- strating its robustness and generalizability. De- spite PrimeKG (Chandak et al. 2023) being
Puin Mi
10
GOG sandar Salee
KCAParbe
+7%
625
9.0
MecDOk
E.IT
DA--
GPT-4-Tutt
Priveess
Figure 5: a) Performance of KGAREVION with different backbone LLMs across all datasets. b) Performance of KGAREVION with different KGs used in the fine-tuning stage of the Review action.
significantly larger than OGB-biokg (Hu et al. 2020), KGAREVION's performance remains high.
This stability arises because KGAREVION generated triplets rather than as a primary knowledge retrieval source. Unlike KG-based RAG mod- els, which depend heavily the structure and completeness of their KGs, KGAREVION only em- ploys KGs to ensure that generated triplets align with medical knowledge. This verification-ban approach makes KGAREVION more resilient and less dependent on any single knowledge source, allowing it to outperform RAG models that rely extensively on specific KGs for retrieval.
KGs exclusively in the Review action to validate
4.5 EVALUATION OF KGAREVION ON AFRICAN HEALTHCARE
To assess whether KGAREVION relies on the extensive pre-trained knowledge of LLMs from exist- ing datasets, we evaluate on AfriMed- -A, a newly published QA dataset released after all baseline models in this study (Olatunji et al. 2024). We benchmark KGAREVION and five baselines this dataset under a multiple-choice setting. Further details on AfriMed-QA are provided in Appendix
B
Fig. 6 prrreeents the accuracy of KGAREVION compared to the baselines on all expert-level multiple- choice questions in AfriMed- QA. Among the baselines, GPT--4-Turbo achieves the highest accuracy
9

---

Published as a conference paper at ICLR 2025
at 65.0%, while LLaMA3.1-8B reaches 55.5%. To demonstrate KGAREVION's effectiveness, implement with both LLaMA3.1-88 and GPT-4-urbb. The results show that KGAREVION improves accuracy by 5.2% when using LLaMA3.1-8B and by 4.6% when using GPT-4-Turbo. These findings highlight KGAREVION's robustness on previously unseen dataset and its strong zero-shot generalization capabilities in underrepresented medical contexts.
4.6 SENSITIVITY ANALYSES
LLMs often exhibit sensitivity the ordering and indexing of candidate answers in multiple- choice setups (Zheng al., 2023; Pezeshkpour & Hruschka 2023). Prior studies have shown that LLMs are not robust multiple-choice selec- tors and tend to favor answers appearing in the first position (Li et al.j 2024). To examine this issue, we analyze how variations in answer order and indexing impact model performance. We evaluate KGAREVION using LLaMA3-SB and LLaMA3.1 as backbones and compare its performance with their LLM-only counterparts across all datasets (detailed results Appendix C.4.1) Fig. presents the accuracy changes when altering the order or relabeling candidate answers.
80
+4.6%
KGNReeion
5%)
60-
$5 68
$0.00
30,20
KG-Rank SaI--RACLLAWA--58 LLaMAS.1-68 1-89
GPT-4Tubba
Figure 6: Performance of baselines and KGAREVION with LLaMA3.1-8B and GPT-4-- Turbo on AfriMed-QA expert multichoice ques- tions.
Impact of answer order in multiple-choice tups. Fig. 7h shows that pure LLMs exhibit strong sensitivity to answer order, with average accuracy shifts of 8.4% for LLaMA3-8B and 16.0% for LLaMA3.1-888 In contrast, KGARE- VION demonstrates significantly greater robust- ness, reducing this sensitivity. This stability arises from KGAREVION's evaluations of an- swers using its Generate action that mitigates the impact of answer order performance.
Changini
uboling
KGARerion
KOARevio!
Msboosee (5)
(S)
15-
12:90
10x
#11
1.50
2.48
2:51
LLAMA3-I5
LLaMAS 1-46
LLaMA3 86
LLaMA6.155
Figure 7: AAAcurayy of LLMs and KGARE- VION when changing order or relabeling index.
Impact of answer indexing in multiple-choice setups. LLMs also show substantial accuracy shifts when relabeling answers from ABCD EFGH, as illustrated in Fig. 7p. LLaMA3-8B experi- ences an average accuracy shift of 8.1%, while LLaMA3.1-88 shows a shift of 12.9%. In contrast, KGAREVION significantly stabilizes these LLMs, reducing accuracy loss 2.59% and 3.86%, respectively. These results highlight KGAREVION's ability to enhance robustness in knowledge- intensive medical reasoning.
5 CONCLUSION
Biomedical reasoning requires integrating multi-source, grounded, and specialized domain knowl- edge to ensure accurate and reliable decision-making. In this work, we introduced KGAREVION, KG-based LLM agent that addresses these challenges by combining the non-codified knowledge embedded in LLMs with the structured, codified knowledge of medical concepts stored in KGs. By dynamically generating, verifying, and refining knowledge, KGAREVION adapts the com- plexity of biomedical QA to generate accurate and contextually relevant answers. Evaluations across multiple-choice and open-ended tasks, including newly curated challenging benchmarks, demonstrate that KGAREVION consistently improves accuracy over existing methods. By ground- ing --g--rreeteedd knowledge in KGs, KGAREVION enhances reasoning robustness and ensures alignment with biomedical knowledge. These results highlight the potential of KGAREVION to advance wwwledde-intensiie medical QA, establishing the foundation for future applications in clinical decision support, biomedical research, and complex medical inference.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We gratefully acknowledge the support of NIH RO1-HD108794, NSF CAREER 2339524, US DoD FA8702-15-D-000) Harvard Data Science Initiative, Amazon Faculty Research, Google Research Scholar Program, AstraZeneca Research, Roche Alliance with Distinguished Scientists, Sanofi iDEA-iTECH, Pfizer Research, Gates Foundation (INV-079038), Chan Zuckerberg Initiative, John and Virginia Kaneb Fellowship at Harvard Medical School, Biswas Computational Biology Initiative in partnership with the Milken Institute, Harvard Medical School Dean's 's Innovation Fund for the Use of Artificial Intelligence, and Kempner Institute for the Study of Natural and Artificial Intelligence Harvard University. V.G. is supported by the UK Medical Research Council, MR/W007 10X/1. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funders.
ETHICS STATEMENT
This study evaluates LLMs benchmark datasets for medical QA. Although LLMs demonstrate promising capabilities in processing and generating medical information, their use in real-world medical decision-making carries inherent risks. LLMs may produce hallucinations (factually in- correct or misleading information), which can lead to inaccurate medical advice. In addition, these models are trained on datasets that can contain biases, which can lead disparities in healthcare rec- ommendations. Ensuring transparency. interpretability, and fairness in medical Al systems remains an ongoing challenge. Further, medical L.MMs should not be considered substitute for professional medical advice. The deployment of such models must be approached with caution, ensuring align- ment with medical cthics, patient safety guidelines, and regulatory standards. Users must be aware of these limitations, and appropriate human oversight is necessary when applying LLMs in clinical or consumer- facinng applications.
REFERENCES
Vaibhav Adlakha, Parishad BehnamGhader Xing Han Lu, Nicholas Meade, and Siva Reddy. Evalu- ating correctness and faithfulness of instruction-folloviii models for question answering. Trans- actions of the Association for Computational Linguistics, 12:775-793, 2024.
Malaikannan Sankarasubbu Ankit Pal. Openbiollms: Advancing open- large language mod- cls for healthcare and life sciences. 2024.
Llama3--Laa3---- 70B
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024. URL https://ppenevvie..ee///ouuuu Pid-SSy W5go0v8
Olivier Bodenreider. The unified medical language system (umls): integrating biomedical terminol- ogy. Nucleic acids research, 32(suppl (rcuupll_)::220000 (1:1)))::2722711 2004.
Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes, Autonomous chemical research with large language models. Nature, 624(7992):50--578 2023.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Dural Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in nearal information pro- cessing systems, 26, 2013.
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew White, and Philippe Schwaller. Augmenting large language models with chemistry tools. In NeurlPS 2023 Al for Science Work- shop, 2023. URL httpss//pppneeeevwweee///////dco
Payal Chandak, Kexin Huang, and Marinka Zitik. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1);67, 2023.
Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis M Rocha, Johan Bollen, Filippo Menezer, and Alessandro Flammini. Computational fact checking from knowledge networks. PloS one, 10(6): e01 28193, 2015.
11

---

Published as a conference paper at ICLR 2025
Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, and Jun Xu. Bias and unfairness in information retrieval systems: New challenges in the Ilm era. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 6437-6447, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan. The llama herd of models, 2024. URL https//arxii.orggabb2///////
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summariza- tion. arXiv preprint arXiv:2404.161:1 2024.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting Ilms: Towards retrieval-augmentee large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6491-6501, 2024.
Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik. Empowering biomedical discovery with ai agents. arXiv preprint arXiv:2404.00 02831, 2024.
Ojas Gramopadhye, Saeel Sandeep Nachane, Prateek Chanda, Ganesh Ramakrishnan, Kshi- tij Sharad Jadhav, Yatin Nandwani, Dinesh Raghu, and Sachindra Joshi. Few shot chain-of- thought driven reasoning prompt Ilms for open ended medical question answering. arXiv preprint arXiv:2403.0443 2024.
Lucas Graves and Tom Glaisyer. The fact-checking universe in spring 2012. New America, 2012.
Tianyu Han, Lisa C Adams, Mech-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. Medalpaca-an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247, 2023.
Emily Harris. Large language models answer medical questions accurately, but can't match clini- cians' knowledge, JAMA, 2023.
Naeemul Hassan, Afroza Sultana, You Wu, Gensheng Zhang, Chengkai Li, Jun Yang, and Cong Yu. Data in, fact out: Automated monitoring of facts by factwatcher. Proceedings of the VLDB Endowment, 7(13): 1557-1 2014.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Interna- lional Conference on Learning Representations, 2022. URL https://ppeneevii..ee/fommm KeeVKe)
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 1822218-2200 2-21133. 2020.
Soycong Jeong, Jinheon Back, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-rag: large language models through question complexity. arXiv
Learning adapt preprint Xrxiv:2403. /4403, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ana Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://aaxxv.orr/abb/ 2310.06825
Xiao Jiang, Chengkai Li, Ping Luo, Min Wang, and Yong Yu. Prominent streak discovery in quence data. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1288- 2011.
12

---

Published as a conference paper at ICLR 2025
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. Can multiple-choice questions really be useful detecting the abilities of Ilms? arXiv preprint arXiv:2403. 17752, 2024.
Xian Li, Weiyi Meng, and Clement Yu. T-verifier: Verifying truthfulness of fact statements. In 20/ IEEE 27th International conference on data engineering, pp. 63-74. IEEE, 2011.
Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions? Patterns, 5(3), 2024.
Jörg Menche, Amitabh Sharma, Maksim Kitsak, Susan Dina Ghiassian, Marc Vidal, Joseph Loscalzo, and Albzrt-László Barabási. Uncovering discase-disease relationships through the in- complete interactome. Science, 347(6222) 2015.
Lino Murali, G Gopakumar, Daleesha M Viswanathan, and Prema Nedungadi. Towards electronic health record-based medical knowledge graph construction, completion, and applications: lit- erature study. Journal of biomedical informatics, 143:104403, 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted with human feedback. arXiv preprint a1Xiv:2112.09332. 2021.
Tobi Olatunji, Charles Nimo, Abraham Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mard- hiyah Sanni, Chinemelu Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, et al. Afrimed-qa: -a: A pan-african, spelii-ppecilty, medical question-answering benchmark dataset. arXiv preprint arXiv:2411.15640 2024.
OpenAl. Gpt-4 technical report, 2024. URL https///arii.og////b/223.....77
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
Add=TG8KACXEON
Vimla I. Patel, José F Arocha, and Jiajie Zhang. Thinking and reasoning in medicine. The Cam- bridge handbook of thinking and reasoning, 14:727-750, 2005.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity the order of op- tions in multple-chooie questions. arXiv preprint arXiv:2308. 14431, 2023.
Kunxun Qi, Jianfeng Du, and Hai Wan. Learning from both structural and textual knowledge for inductive knowledge graph completion. Advances in Neural Information Processing Systems, 36, 2024.
Alec Radford. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Thirty Conference on Neural Information Processing Systems, 2023. URL http::/wwyeneeeeeeeeeeeeoeeeeeep
13

---

Published as a conference paper at ICLR 2025
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- GPT: Solving Al tasks with chatGPT and its friends in hugging face. In Thirty- Confer-
Neural Information Processing Systems, 2023. URL https://ppnrrviee.n///foumm y6dT'c
Baoxu Shi and Tim Weninger. Discriminative predicate path mining for fact checking in knowledge graphs. Knowledge-bed systems, 104:123-133 2016.
Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. Generate-then--roundd retrieval-augmented generation for multi-hop question answering. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7339- 7353, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https:
/aclanthologg.oog22224o .397
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972) 2023.
Karthik Soman, Peter W Rose, John Morris, Rabia E Akbas, Brett Smith, Braian Peetoom, Catalina Villouta-Reyes, Gabriel Cerono, Yongmei Shi, Angela Rizk-Jackson, al. Biomed- ical knowledge graph-enhanced prompt generation for large language models. arXiv preprint 1rX1v:2311.17330, 2023.
Yueqing Sun, Qi Shi, Le Qi, and Yu Zhang. Jointlk: Joint reasoning with language models and knowledge graphs for commonsense question answering. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies. NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 1044950060, Asso- ciation for Computational Linguistics. 2022. doi: http:/.....11/////0 https://doi.or//10. 18633 /6t55//17202222mmmmmm....33333.
372. URL
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. arXiv preprint arXiv:2311.105: 2023.
Nandan Thakur, Luiz Bonifacio, Xinyu Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, and Jimmy Lin. Nomiracl: Knowing when you don't know for robust multilingual retrieval-augmented generation. CORR, abs/2312. 11361, 2023.
doi: 10.44550/ARXIV231221136 URL https:
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, and
Nikolay Bashlykov. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https:
V/axiv.org/abs22307.0923
William Tunstall-Pedoe.
True knowledge: Open-domain question answering using structured knowledge and inference. Ai Magazine, 31(3):80-92, 2010.
A Vaswani. Attention all you need. Advances in Neural Information Processing Systems. 2017.
A Venigalla, Jonathan Frankle, and M Carbin. BiomedIm: a domain-specific large language model for biomedical text. MosaicMI. Accessed: Dec, 23(3):2, 2022.
Guanzhi Wang. Yugi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. In Intrinsically- Motivvated and Open-Ended Learning Workshop @NeurIPS2023, 2023a. URL
14

---

Published as a conference paper at ICLR 2025
Yubo Wang, Xueguang Ma. and Wenhu Chen. Augmenting black-box Ilms with medical textbooks for clinical question answering. arXiv preprint arXiv:2309.02233) 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 978171387 1088.
Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. Pmc-Ilama: toward building open-source language models for medicine. Journal of the American Medical Informatics Association, pp. ocae045, 2024a.
Junde Wu, Jiayuan Zhu, and Yunli Qi. Medical graph rag. Towards safe medical large language model via graph retrieval- augmenttd generation. arXiv preprint arXiv:2408. 04187, 2024b.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi- agent conversation framework. arXiv preprint wrXiv:2308.08/55 2023.
Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Zou, and Jure Leskovec. Stark: Benchmarking Ilm retrieval on textual and relational knowledge bases. 2024c.
You Wu, Pankaj K Agarwal, Chengkai Li, Jun Yang, and Cong Yu. On" one of the few" objects. In Proceedings of the 18th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 1487-1495, 2012.
Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking strieval-augmentt generation for medicine. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 6233-6251, Bangkok, Thailand and virtual meeting, August 2024a. Association for Computational Linguistics. URL hhtpps/// icclanhologg.oor/22044 htw.:logo.rrrr222222minnn------o ac1.372
Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval augmentl generation for medicine, 2024b. URL
Rui Yang, Haoran Liu, Edison Marrese-Taylor, Qingcheng Zeng, Yuhe Ke, Wanxin Li, Lechao Cheng, Qingyu Chen, James Caverlee, Yutaka Matsuo, and Irene Li. KG-rank: Enhancing large language models for medical QA with knowledge graphs and ranking techniques. In Dina Demner-Fushman, Sophia Ananiadou, Makoto Miwa, Kirk Roberts, and Junichi Tsujii (eds.). Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pp. 155-166, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL wwwp:/hhooooggooo22/....o..m 1.13.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tammoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 535-546, Online, June 2021. Association for Computational Linguis- tics. doi: 10. 10.18653/v 1/2021 maacl-main.45 URL https://cclaatoolog..or//////
maacl-main.45 main.45)
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang, and Jure Leskovec. Deep bidirectional Ennuage--nowledde graph pretraining. In Neural Information Processing Systems (NeurIPS). 2022.
Jiahao Zhang, Haiyang Zhang, Dongmei Zhang, Liu Yong, and Shen Huang. End-to-end beam retrieval for multi-hop question answering. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp- 1718-1731, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1//3//////22..maaaoooooooo URL http:///////caaaoooooo.////////.
15

---

Published as a conference paper at ICLR 2025
Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. Verify-and-ede A knowledge-enhanced Chain-of-thoughee framework. In Anna Rogers, Jordan Boyd-Graber, Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 5823-5840, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.188653/112023.al-lomooo
URL
https:/accantbboog..or/22222233 act.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors, The Twelfth International Conference on Learning Representations, 2023.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
Extended Related Work
18
Datasets
18
Gold Standard Multi-choice Medical QA Dataset
18
McdDDx
18
AfriMed-QU QA
19
Conversion of Multi-Choice Type Questions To Descriptive Type
20
Implementation Details
20
Experiment Environments
20
................................
Fine-tuning Details
21
Implementation Details of Baseline Models
Implementation Details of Sensitivity Analysis
22
Implementation Details of Adaptability Analysis
22
KGARevion
22
Implementation Details in Revise Action
22
Implementation Details in Answer Action
23
Prompts
23
Prompt Template for Evaluating Baseline Models Under Multi-choice Reasoning Setting
Prompt Template for Evaluating Baseline Models Under Open Reasoning Setting
23
Prompt Template KGAREVION
24
Tables
27
Description Template
27
Notations
28
17

---

Published as a conference paper at ICLR 2025
A EXTENDED RELATED WORK
Fact-checking Models, The vast amount of generated text has created a growing need for fact- checking (Graves & Glaisyer 2012). Early methods relied manually defined rules to verify generated statements (Hassan et al. 2014 Wu et al.) 2012 Jiang et al. 2011). KGs, as structured and reliable knowledge bases, have also been utilized for fact-checking by analyzing graph con- nectivity. These methods (Tunstall-Pedoe 2010, Shh & Weninger 2016 Ciampaglia et al., 2015) validate generated statements by identifying paths between entities mentioned the statement. Ad- ditionally, some approaches use popular KG representation learning algorithms for KG completion tasks to verify the accuracy of statements
201T). With the rise LLMs, new fact-checking approaches, e.g. RAG-based models. have emerged to ensure the consistency and reliability of gen- crated outputs by integrating external evidence retrieval for verifying and supporting LLM -ggeereee content (Asai et al. 2024).
B DATASETS
B. GOLD STANDARD MULTI-CHOICE MEDICAL QA DATASET
In this work, four well-known multi-choice medical QA datasets to evaluate the model per- formance, including two medical examination QA datasets (MMLU-Med, MedQA-US) and two biomedical research QA datasets (PubMedQA*, BioASQ- Y/NN. These datasets derived from (Xiong et al. 2024a). The samples in these datasets are shown in Table[4]
Dataset Name
Sample
Which of the following best describes the structure that collects urine in the body?
MMLU-Med
A: Bladder B: Kidney C: Ureter D: Urethra
A microbiologist is studying the emergence of a virulent strain of the virus. After a detailed study of the virus and its life cycle, he proposes theory: Initially, a host cell is co-infected with 2 viruses from the same virus family. Within the host cell, concomitant production of various genome segments from both viruses occurs. Ultimately, the different genome segments from the viruses are packaged into unique and novel virus particle. The newly formed virus particle is both stable and viable and is new strain from the virus family that caused the outbreak of infection. Which of the following viruses is capable of undergoing the anlove-mentione"s process?
MedQA-US
A: Epptein-Bart Barr virus B: Human immunodeficiency virus C: Rotavirus D: Vac- cinia virus
Is anorectal endosonography valuable in dyschesia?
PubMedQA
A: yes B: no C: maybe
Can losartan reduce brain atrophy in Alzheimer's 's disease?
BioASQ- Y/N
A: yes B: no
Table 4: Examples of four widely used medical QA datasets
B.2 MEDDDX
MedDDx a newly constructed dataset designed to test model performance on semantically com- plex answers. The motivation behind creating this dataset is twofold:
18

---

Published as a conference paper at ICLR 2025
While LLMs perform QA tasks, they often rely heavily semantic dependencies, making difficult for them to identify the correct answer among semantically similar an- swer candidates;
In real-world medical scenarios, researchers often focus identifying subtle differences between similar molecules, particularly in treatment or diagnostic settings. For instance, proteins may share similar names but have significantly different structures and functions, making it crucial to distinguish these differences to be able to provide accurate answers (see Table for an example).
Dataset Name
Sample
Can you recommend medications effective against peptic ulcer disease that also suppress Helicobacter pylori in the stomach?
Basic
A: Rebamipide Ecabet C: Bendazac D: Nepafenac
Can you recommend medications that treat both cosinophilic pneumonia and a parasitic worm infection?
Intermediate
A: Thiabendazole B: Albendazole C: Diethylcarbamazine D: Triclabendazole
Which genes or proteins are expressed exclusively in the pericardium and either the dorsal or ventral regions of the thalamus?
Expert
A: ADHIA B: ADHIC C: ADH4 D: ADHIB
Table 5: Examples of four widely used medical QA datasets.
Because of these reasons, we construct MedDDx, a multi-choice medical QA dataset that focuses on answering semantically complex multi-choice QA. These questions sourced from STaRK-Prine (Wu et al. 2024c). which provides both the questions and their corresponding answers. We extract questions with a single correct answer from the STaRK-Primt testing set and transform them into the multi-choice format. To generate three strong alternative answer candidates, we use semantic similarity to increase the difficulty, selecting the top three entities that have the highest semantic similarity as the correct answer. The semantic embeddings used for this process are derived from text-embeddi ada-002 model from OpenAl.
The semantic similarity is calculated using cosine similarity. We also compute the standard devia- tion of semantic similarity between the correct answer and the other three candidates. The density distribution of these values is shown in Fig. X Based on this distribution, we divide the queries into three complexity groups using quantile analysis: MedDDx-Exe -xppett (0-0.02), MedDDx (0.02-0.04), and MedDDx-Basis (>0.04).
B.3 AFRIMED-OR QQA
AfriMed-QA creates novel multispecialty open-source dataset of 15,275 pan-African clinically di- verse QA to rigorously evaluate LLMs African healthcare. The dataset is sourced from over 500 clinical and non-clinical contributors across countries and covers 32 clinical specialties. First, because the dataset is sourced from healthcare systems whose data are not online, it means that none of the multiple-choice and open-ended questions and answers can be in the knowledgebase of the LLM. Because of that, this benchmark has not been leaked into KGARevion' 's LLM. Furthermore, the knowledge cut-off date for KGARevion's LLM was before AfriMed-QA was released. Sec- ond, AfriMed-QA is sourced from clinical, private healthcare systems whereas PrimeKG, STaRK- Prime, and more, PrimeKG, STaRK-Prime Primee, and OGB-b were all released several years prior when the AfriMed- QA benchmark was released. As such, AfriMed-QA has certainly not been leaked into KGARevion's KG.
are sourced from non-clin con-clinical, public, biological data repositories. Further-
19

---

Published as a conference paper at ICLR 2025
Probability Density Function
30-
25
23-
0.-
Answer Similanty Std
Figure 8: The distribution of the standard deviation of semantic similarities between answer candi- dates and the correct answer. A lower value indicates greater similarity among the answers.
B.4 CONVERSION OF MULTI-CHOICE TYPE QUESTIONS TO DESCRIPTIVE TYPE
The conversion of multi-choice questions to descriptive ones aimed to evaluate scenarios where open-ended inquiries are prevalent. To achieve this, we modify the question by adding more descriptive terms, as shown in T
medical
Multi- Choice type
Open-ended type
Which of the following best describes the structure that collects urine in the body?
What best describes the structure that col- lects urine in the body?
A: Bladder B: Kidney C: Ureter D: Urethra
A microbiologist is studying the emergence of virulent strain of the virus. Af- ter detailed study of the virus and its life cycle, he proposes a theory: Ini- tially, host cell is co-infected with viruses from the same virus family.
A microbiologist is studying the emergence of a virulent strain of the virus. After a de- tailed study of the virus and its life cycle, he proposes theory: Initially, host cell is co- infected with 2 viruses from the same virus family. Which virus is capable of un- dergoing the above-mentioned process?
Which of the following viruses is capable of undergoing the
process?
A: Epstein-Bant Barr virus B: Human immunode- ficiency virus C: Rotavirus D: Vaccinia virus
Table 6: Examples of conversation of multi-choice type question to descriptive type.
C IMPLEMENTATION DETAILS
C.1 EXPERIMENT ENVIRONMENTS
Hardware. All experiments are conducted on a machine equipped with NVIDIA H100. We use NVIDIA H100 to implement baselines with small LLMs. In the fine-tuning stage, use NVIDIA H100 to fine-tune the review module.
Software. We implement KGAREVION using Python 3.9.19. Py Torch 2.3.1, Transformers 4.43.1, and Tokenizers 0.19.1. All LLMs adopted in this study are downloaded from Hugging Face, except for OpenAl models.
20

---

Published as a conference paper at ICLR 2025
C.2 FINE-TUNING DETAILS
During the fine-tuning stage, we first split PrimeKG Chandak al. (2023) into two parts: training set and testing set, in a ratio of 8:2. We use LoRA to fine-tune the LLMs on a single machine equipped with 4 NVIDIA H100 GPUs for knowledge graph completion tasks in the Review action.
For hyperparameter tuning, we use grid search to identify the optimal parameter combinations by evaluating the fine-tuned model's performance testing set. Specifically, we focus on the parameter r in LoRA training and the batch size during the fine-tuning stage. The values explored for are 16, 32, 64, 128, while the tested batch sizes be 128, 256, 512, 1024. The best parameters identified are r = 32, bz = 256.
the knowledge graph completion task using the
C.3 IMPLEMENTATION DETAILS OF BASELINE MODELS
All the results reported in this paper are averaged over multiple runs with different random seeds. For consistency and reproducibility, we select three commonly used seeds: 42, 777, and 1234.
C.3.1 IMPLEMENTATION DETAILS OF BASELINE MODELS UNDER MULTI-CHOOCE SETTING
LLM-based Reasoning Models. We evaluate these LLMs across all datasets in two distinct set- tings: the traditional inference setting and the CoT inference setting, using the Transformers package (v 4.31.1) with default parameters. The prompts used for evaluation are provided in E.1
KG-based Models. Since the baseline models are trained in an end-to-end way, they cannot be applied directly. To ensure fairness, train these three models collective dataset called MedM- CQA, which contains 4182 question-answeringg samples, and then evaluate them on all datasets used in this study. In addition, we utilize PrimeKG the knowledge base for all three methods con- struct subgraphs for each sample in both the training and evaluation stages. For knowledge graph processing. we follow the same procedure as JointL.,, converting each entity in the KG to its cor- responding UMLS code and retrieving entities that match those in the question to construct the subgraph for each question. In the case of Dragon, since it requires a pre-training stage, we first complete its pre-training adopted this study. We set the epoch number for all three models as 20. All other parameters were derived from the original publications.
the MedMCQA dataset and then directly infer the model on all datasets
RAG-based Models. We implement the Self-RAG with Llama2-7B/13B with VLLM (v 0.4.3). We set the temperature in SamplingParams as the default value as their original LLMs. MedRAG is implemented using the original code repository. We adopt Llama2-70B as its backbone model, "textbooks' as corpus, and MedCPT as retriever. KG-RAG is implemented with the same KG as in KGAREVION, which is PrimeKG and is utilized only during the inference stage. For KG-Rank, report the results using its original backbone model, which is Llama2-133 13B and adopt the MedCPT as the ranking method due limitations with the Cohere API.
C.3.2 IMPLEMENTATION DETAILS OF BASELINE MODELS UNDER OPEN-ENDED REASONING SETTING
To test bascline models under the open-ended reasoning setting, we add new module for both LLM-based reasoning models and RAG-based models. In terms of KG-based models, they could not be deployed under the open-ended reasoning setting due their model architectures, shown
For each model in the other two groups, first get response to the input descriptive question and then adopt the same LLM as their original design to match the response to the content of the correct answer without considering the input question.
21

---

Published as a conference paper at ICLR 2025
C.4 IMPLEMENTATION DETAILS OF SENSITIVITY ANALYSIS
C.4.1 EFFECT OF ANSWER ORDER/INDEX
To assess the sensitivity of each model to the answers' order, swap the positions of two options. For datasets with four options, the order is changed from ABCD to BCAD. For three-option datasets, the order is adjusted from ABC to CAB, and for two-option datasets, it is reversed from AB to BA. It important to note that we alter the order along with the corresponding content, not just the answer labels. To test the model sensitivity to the answers' index, relabel the answer indices from ABCD to EFGH. The question obtained after changing the order or the index is presented in Table 7
Sample
Which of the following best describes the structure that collects urine in the body?
Original
A: Bladder B: Kidney C: Ureter D: Urethra
Which of the following best describes the structure that collects urine in the body?
Changing Order
B: Kidney C: Ureter A: Bladder D: Urethra
Which of the following best describes the structure that collects urine in the body?
Relabeling Index
E: Bladder F: Kidney G: Ureter H: Urethra
Table 7: Examples of four widely used medical QA datasets.
C.5 IMPLEMENTATION DETAILS OF ADAPTABILITY ANALYSIS
To show the flexibility of KGAREVION, we further conduct adaptability analysis by implementing the model with different backbones and KGs. Overall, in this work, we test the KGAREVION with two KGs (i.e., PrimeKG and OGB-biokg! Their details are as follows:
PrimeKG (Chnndak et al.
2023) is precision medicine-orienttee knowledge graph that provides a holistic view of diseases. PrimeKG integrates 20 high- resources describe 17,080 diseases with 4,050,249 relationships representing ten major biological scales, including discase-associated protein perturbations, biological processes and pathways, anatomical and phenotypic scale, and the entire range of approved and experimental drugs with their therapeutic action, considerably expand- ing previous efforts discase-rooted knowledge graphs.
The OGB-biokg (Hu et al. 2020) dataset contains 5 types of entities: diseases (10,687 nodes), proteins (17,499), drugs (10,533 nodes), side effects (9,969 nodes), and protein functions (45, (45,085 nodes). There are 51 types of directed relations connecting two types of entities, including 38 kinds of drug-drug interactions, kinds of protein-protein interaction, as well as drug-protein, drug-side effect, and function-funciio relations.
D KGAREVION
D.1 IMPLEMENTATION DETAILS IN REVISE ACTION
The Revise action is carried out by providing an instruction to an unfine-tuned LLM, such as LLaMA3.1-8R 1-8B, instead of retraining the model. The instruction directs the LLM to correct the "False" triplet identified by the Review action by modifying either the head or tail entity.
Input of False Triplets The Revise action begins by receiving False triplets, such as (HSPAIA, interactions, DHDDS), which were flagged as False by the Review action.
Instruction to the LLM The system then provides an instruction to the LLM, telling that the given triplet is incorrect. The instruction asks the LLM to revise the triplet, replac- ing either the head or tail entity, and to generate a corrected triplet that is relevant to the input question. For example, the instruction could be: "The following triplet is incorrect:
22

---

Published as a conference paper at ICLR 2025
(HSPAIA, interactions, DHDDS). Please revise it correct triplet related the imput query." The details Instruction available in E.3
Revised Triplet Generation Simultaneously, the LLM is prompted to output the revised triplet in specified format, such JSON structure: ('Revised Triplets': (HSPAIB, interactions, DHDDS))
Review of Revised Triplet Once the revised triplet is generated, KGARevion sends it to the Review action for validation. The Review action checks the correctness of the newly generated triplet.
Iterative Revision Process If the revised triplet is identified as True by the Review action, it passed on to the Answer action for final output. the revised triplet still identified as False, it is sent back to the Revise action for further modification. This process repeats until either a True triplet is identified or the maximum number of revision rounds reached.
D.2 IMPLEMENTATION DETAILS IN ANSWER ACTION
The Answer action is performed by prompting an unfine-tuned LLM, such LLaMA3.1 instead of retraining the model. This approach mirrors the Revise action, allowing the system to leverage pre-trained models without the need for additional training.
Input of Verified Triplets The input to the Answer action consists of all triplets that have been verified as True by the Review actions.
Instruction to the LLM Once the verified triplets received, KGARevion sends instruction to the LLM. This instruction directs the LLM to generate the final answer based the provided triplets.
Final Answer Generation The LLM receives all True triplets and outputs the final answer based on those verified True triplets.
E PROMPTS
E.1 PROMPT TEMPLATE FOR EVALUATING BASELINE MODELS UNDER MULTI-CHOICE REASONING SETTING
Prompts for Evaluating LLMs
The following multiple-choice medical question. Please select and provide the correct answer from options 'A', 'B', 'C' or DD'.
Question: {question}
Answer:
Prompts for Evaluating LLMs with CoT
The following is multiple-choice medical question. Let's think step by step. Please select and provide the correct answer from options 'A', 'B', 'C' or DD"
Question: {question]
Answer:
E.2 PROMPT TEMPLATE FOR EVALUATING BASELINE MODELS UNDER OPEN REASONING SETTING
23

---

Published as a conference paper at ICLR 2025
Prompts for Evaluating LLMs
The following are medical questions. Please generate a response for input question.
Question: {question)
Answer:
Prompts for Evaluating LLMs with CoT
The following are medical questions. Let's think step by step. Please generate response for input question.
Question: {question]
Answer:
Prompts for Evaluating LLMs
Given a context, please select the most match answer from options by using 'A', BB, 'C', and
'D'.
Context:
{context}
Options: {options)
Answer:
E.3
PROMPT TEMPLATE IN KGAREVION
The Generate action is implemented with two prompts. One is responsible for identifying medical concepts involved in question stem, the other is for generating triplets.
Prompts for Generate Action
### Instruction:
Given the following multiple-choice question, extract all relevant medical entities contained within the question stem. Identify and extract all medical entities, such diseases, proteins, genes, drugs, phenotypes, anatomical regions, treatments, or other relevant medical entities. Ensure that the extracted entities are specific and medically relevant. If medical entities are found particular part, return an empty list for that section. Only return the extracted entities in JSON format with the key "medical terminologies" and the value a list of extracted entities.
### Input:
Question: {question]
#### Response:
24

---

Published as a conference paper at ICLR 2025
Prompts for Generate Action
### Instruction:
Given the following question stem, medical terminologies, and options, generate set of re- lated undirected triplets. Each triplet should consist of head entity, relation, and a tail entity. The relations should describe meaningful interactions or associations between the en- tities, particularly in medical or biomedical context. Use the question stem and the medical entities contained each option to extract triplets that are relevant to the query and can answer the query correctly. Each triplet should be in the format: (Head Entity, Relationship, Tail En- tity). Since the triplets are undirected, the order of Head Entity and Tail Entity does not imply any directional relationship between them. The relationship should be of the following: prootein_poteiiin 'carrier', 'enzyme", "target', 'transporter', "contraindication" 'indication', off-llbel use", 'synergistic interaction', "associated with', 'parentchild", 'phenotype absent', ppennotype present', 'side effect', 'interacts with', 'linked to', 'expression present', "expres- sion absent") Ensure that each entity the triplet is specific and concise, such as diseases, proteins, conditions, symptoms, drugs, treatments, anatomical parts, or other relevant medical entities.
Generate 1-3 triplets for each option, focusing the ones most relevant to answering the query.
Only return the generated triplets in structured JSON format with the key as "Triplets" and the value as a list of triplets. The format should be: "Triplets": [(Head Entity, Relationship. Tail Entity), (Head Entity, Relationship, Tail Entity)]
### Input:
Question: (query_stem)
(medical terminologies}
Options: {option}
### Response:
Prompts for Review Action
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
Given triple from a knowledge graph. Each triple consists of head entity, a relation, and tail entity. Taking (PHYHIP, protein_protein KIF15) example, it means that protein PHYHIP has an interaction with protein KIF15. Please determine the correctness of the triple and response True False. Please directly output 'Truc' or 'False..
### Input:
(triplet]
### Response:
25

---

Published as a conference paper at ICLR 2025
Prompts for Revise Action
### Instruction:
Given the following triplet consisting of a head entity, relation, and tail entity, please review and revise the triplet to ensure it is correct and helpful for answering given question. The revision should focus on correcting the head entity, relation, or tail entity as needed to make the triplet accurate and relevant. The triplet should follow the format (head entity, relation, tail entity). Ensure that the revised triplct is factually accurate and contextually appropriate. The relation should clearly define the relationship between the head entity and the tail entity. If no changes are necessary, return the original triplet.
Only return the revised triplet in JSON format with the key 'ripieed.Tripltts' and the value the corrected triplet. The format should be: "Revised Triplets": [(Head Entity, Relationship, Tail Entity)]
### Input:
Triplets: (triplets}
Questions: {query}
### Response:
26

---

Published as a conference paper at ICLR 2025
F TABLES
F.1 DESCRIPTION TEMPLATE
Relation
Description
protein-proteii
Protein {A] interacts with protein (B), indicating that the two proteins directly
or indirectly associate with each other to perform a biological function.
carrier
(A) acts as carrier for {B), facilitating its transport delivery to specific locations within the body or within a cell
enzyme
(A) functions as enzyme that catalyzes reaction involving
(B), converting
it into a different molecule or modifying its structure
target
{A} serves as a target for {B), meaning that {B} binds to or interacts with {A} to exert its biological effect. {A} is a transporter that facilitates the movement of {B} across cellular mem- branes or within different compartments of the body.
transporter
contraindication
The interaction between {A} and (B) contraindicated, meaning that the pres- ence of one molecule may have adverse effects or reduce the efficacy of the other {A} is indicated for the treatment or management of a condition associated with
indication
(B), suggesting that [A} has therapeutic role related to {B}
off-label use
(A) is used off-label in relation to (B), meaning it is utilized in a manner not specifically approved but based clinical judgment.
synergistic interaction
{A} and {B} interact synergistically, where their combined effect is greater
than the of their individual effects
associated with
{A} is associated with {B}. indicating a relationship or correlation between the two, often in the context of disease or biological processes
parent-child
{A} is related to {B} in a parent-child relationship, where {-A} gives rise to or
influences the formation of {B}
phenotype absent
The interaction between {-) and {B} } results in the absence of a specific phe- notype, indicating that the normal trait is not expressed
phenotype present
The interaction between { and {B} results in the presence of a specific phe- notype, indicating that a particular trait is expressed
side effect
The interaction between {-}} and {B} can cause side effect, where the pres- ence of one molecule leads to unintended and possibly adverse effects
interacts with
(A) interacts with BB). indicating general interaction that may involve bind- ing, modulation, or other forms of molecular communication.
linked
{A} is linked (B). suggesting a connection or association between the two molecules, often in a biological or pathological context.
expression present
{A) is expressed in the presence of {B), indicating that the existence or activity
of ( [ leads to or correlates with the expression of {A}
expression absent
(A) is not expressed in the presence of {B), indicating that the existence or
activity of {B} suppresses or does not correlate with the expression of (A)
Table 8: The description templates
27

---

Published as a conference paper at ICLR 2025
F.2 NOTATIONS
Variable
Description
set of medical queries
One question stemn in
set of amswer candidates for one question stem
Correct answer
the question
A large language model
Knowledge graph
A head eniiyy in one triplet
A relationship one triplet
A tail entity
triplet
set of triplets generated the Generation action
candidate C A set of medical concepts in
M
embeddings Pre-trained embeddings of
Pre trained embeddings of
Dimension of pre trained embedddings
The fine-tuned model used in Review action
Bool value that determining the correctuess triplct
The description dictionary for relationship
length description tokens
dg
The dimension of token embeddings in LLM
Token embedding matrix
shape of d, obtained from
linear layer map the dimension of pre pre-trained embeddings d to that of token embecklings dp The triplct embodding matrix
Aligned triplet embedding matrix
(
The Softmax function
layer normalization function
W, and W
Trainable parameters the two layer forward neural network
da
The dimension of hidden layers in the two layer forward neural network
instruction to the LLM
True triplct set desermined by Review action
False triplet determined
action
Max round of iterative review and revise actions Predicted answer from Answer
Table 9: Additional notation.
28


Reviews:
[{"review_id": "UFkZT4xH86", "reviewer": "Reviewer_XbfB", "summary": "This paper studies the integration of factual knowledge with free text information into LLMs. It introduces KGAREVION, a medical Question Answering (QA) agent, which additionally fine-tunes a LLM on a knowledge base completion task to improve its performance on evaluating medical facts. For an input question the agent can perform three actions:\n 1. **Generate** relevant medical fact candidates \n 1. **Review** the generated fact candidates to check factual incorrect facts\n 1. **Revise** to add more facts and change the factual incorrect candidate from Review\nThe outputs of the last action are used as part of a prompt to generate the final answer. The agent is compared on multiple choice and open ended questions against several baselines. It is able to reach state of the art performance.", "questions": "Did you consider using the facts and relations from UMLS for the KB completion task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Medical Application**\n\nThe paper study the application of an LLM-based agent to a complex and difficult domain, viz. medicine. I appreciate that the authors took up this challenge to make a relevant contribution. \n\n**Evaluation**\n\nThe presented approach is evaluated thoroughly. It covers a comparison against a large number of competitors on open-ended and multi-choice reasoning. It shows that the presented approach is able to reach state of the art performance and moves the bar.  Apart from this performance analysis the paper also offers an ablation study and sensitivity analyses.", "weaknesses": "Complex approach: \n\nThe presented approach is complex and requires a lot of steps and technical components. It depends on a separate entity recognizer to map entities to UMLS, two LLMs performing different tasks (generation and revision, review), and a knowledge graph for fine-tuning. This reduces reusability, since the entire system would need to be replicated. Reducing all this into one LLM and dataset would make the contribution for accessible for others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gU241DjYAQ", "reviewer": "Reviewer_jrR1", "summary": "This article introduces KGARevion, an LLM which performs medical Q&A by verifying generated information (in the form of triplets) against the information in a knowledge graph (provided as knowledge graph embeddings). By doing so, it achieves competitive performance on several medical Q&A benchmarks. Additionally, the authors provide a new biomedical Q&A benchmark, which they call MedDDx. MedDDx leverages information from another Q&A dataset, but it is supplemented with several incorrect answers which are semantically similar to the correct answers.  KGARevion can be used with a variety of base LLMs and KGs, making it not only competitive but also versatile.", "questions": "1. page 3, Section 3: The authors give an example of a possible question and a set of possible answers. However, wouldn't this question warrant a yes/no response, rather than the set of answers provided? \n\n2. page 5: On which prediction task was TransE trained?  \n\n3. page 5: Why have the authors chosen TransE for the KG embeddings? \n\n4. page 5: Where does the description dictionary D(r) come from? I do not see this in the appendix, and there is no appendix section 8. \n\n5. page 6, section 3.3: How, specifically, does the Revise action \"adjust the triplets in F to include more knowledge\"? \n\n6. page 8, Fig. 3: What do the blue v. orange plots mean? The only y-axis label I see is accuracy, and I see no key for blue v. orange.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Figures:** the figures are nice for understanding the paper; in particular, figure 2 is great. \n\n- **Creative Benchmark:** the idea to create a benchmark with incorrect answers that are semantically similar to correct answers is very creative. In addition, separating questions into levels of difficulty is also quite clever. \n\n- **Reproducibility:** I commend the authors for providing code. Although I did not test it, it looks well documented and well written. \n\n- **Versatility:** the KGARevion approach seems versatile, able to be used with other base LLMs, KGs, and other domains in general.", "weaknesses": "Following the authors' rebuttal, I feel that my concerns below were sufficiently addressed. The authors have added an additional benchmark dataset from a sufficiently different data lineage, and they have also clarified several points of confusion. I have updated my score accordingly.\n\n-------------------------------------------------------------------------------\n\nI have a few concerns about the scientific robustness and the presentation of this work: \n\n1. Firstly, my greatest concern regards the overlap between information in the knowledge graph(s) (KGs) used and the information used to create the benchmarks. Specifically, the authors designed their new QA benchmark, MedDDx, based upon the STaRK-Prime benchmark. **However, according to the corresponding STaRK-Prime paper (https://arxiv.org/abs/2404.13207), the biomedical QA from STaRK-Prime was built upon the PrimeKG, the exact KG which the authors used to fine-tune their approach, KGARevion.** Ultimately, this is not scientifically robust as the authors are providing their LLM with the same, structured data on which their benchmark was built. \n\n    I understand that the authors have also used other \"gold-standard\" benchmarks for comparison. However, there is also likely data leakage in some of these cases as well. For example, the PrimeKG is built from structured data derived from other sources, including the Disease Ontology (DO), the Gene Ontology (GO), UMLS, and DrugBank (Fig. 2, https://www.nature.com/articles/s41597-023-01960-3), all of which are also included in the creation of the BioASQ benchmark (https://pmc.ncbi.nlm.nih.gov/articles/PMC10042099/). Unfortunately, the other KG used within Section 4.4, ogb-bioKG, does not have any provenance information (https://github.com/snap-stanford/ogb/issues/111) but it was built by the same person (M. Zitnik) as PrimeKG, so it likely comes from similar sources. Ultimately, I would still avoid using obg-bioKG if the provenance can not be verified. \n\n    **I recommend that the authors research the data lineage of each of the benchmarks and KGs used. Perhaps they can present this as supplementary material. Then, I recommend that the authors only present the results in which there is no clear overlap between the structured data included within the KGs for fine-tuning and the benchmark datasets. The authors should omit all results where data leakage between the KGs and benchmarks is certain or likely.**\n\n\n\n2. Secondly, I found this paper generally difficult to follow. Specifically, I believe the study could be more clearly motivated within the introduction, and Section 3 (\"Approach\") could be more clearly explained. Below, I list some specific points for improvement: \n\n- Paragraph 1 and the first half of paragraph 2 of the introduction seem loosely connected to the rest of the motivation; I suggest cutting these paragraphs and skipping more directly to the points in paragraph 3. \n\n- Figure 1 is problematic for several reasons. First, it is presented in the middle of the introduction, which is an inappropriate place to present results. Secondly, it is presented with no experimental context. Finally, it is not clear whether the figure even supports the claims being made in the introduction. I suggest moving or removing this figure. \n\n- RAG systems are presented as less desirable alternatives due to a reliance upon the quality of the documents provided, but the authors acknowledge that KGs may also have incorrect or incomplete information. Therefore, the motivation for the work seems unclear. \n\n- Section 3.2 on the Review section is extremely difficult to read. I believe this could be made clearer through more structure within the subsection (sub-headings or numbered steps). Perhaps the textual structure could correspond to flagposts or landmarks within Figure 2, so that the reader can refer back and forth between the text and Figure 2 easily. \n\n- Although fine-tuning with a KG is an integral part of the methodology, PrimeKG is not mentioned anywhere within Section 3. Until I read the Supplementary materials, I had understood that the KG being used for fine-tuning was the set of triplets derived from the \"Generate\" action. The authors should make it clearer that the KG comes from another, external source.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "33QBvuaCGR", "reviewer": "Reviewer_yzKB", "summary": "The paper proposes a novel knowledge graph-based prompting technique for question answering in the medical domain. The approach consiststs of four steps: (1) Given a question in natural language, an LLM is used to generate subject-predicate-object triples from the question. (2) The triples are checked by means of a knowledge graph. Triples that are not clearly wrong are kept. (3) Then an LLM is asked to revise the triples. (4) Finally, the question is answered by means of an LLM and the revised triples.", "questions": "- Can you give an example for the \"fine-tuning stage\" in Section 3.2?\n- Section 3.2 \"inference stage\": How often does each case occur (triple removed vs triple kept) for the given dataset?\n- Can you run your question answering approach with a knowledge graph that is mostly unrelated to the questions (e.g., Freebase15k-237)? Do the results get significantly worse or do they stay approximately the same?\n- In how far does each component of the approach handle and support multi-hop paths in knowledge graphs? It seems, the generate and revise steps do not support paths. The review step might support paths indirectly via KG embeddings.\n- How does the review phase of the approach compare to fact-checking approaches?\n- The approach seems rather general. Wouldn't it also work on other domains besides medicine?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an important problem: question answering in the medical domain\n- Original, novel question answering architecture: (1) generate triples, (2) review triples, (3) revise triples, (4) answer question\n- Strong evaluation: multiple datasets, evaluation depending on question complexity, and ablation study\n- Clarity: high-quality figures that help understanding", "weaknesses": "- Section 3.2 \"fine-tuning stage\" I find this part rather hard to understand. Figure 2b definitely helps, but it might be good to clarify this part even further by providing an example.\n- Section 4.4: It suprises me that the results barely depend on the knowledge graph. I am wondering whether the knowledge graph is required at all then. What do the results look like for a knowledge graph that is not specialized to medicine (e.g., Freebase15k-237)? If the results for Freebase15k-237 are almost the same as for PrimeKG, then the KG would be irrelevant.\n- Related work on fact-checking seems to be missing. \n\nCiampaglia, Giovanni Luca, Prashant Shiralkar, Luis M. Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. \"Computational fact checking from knowledge networks.\" PloS one 10, no. 6 (2015): e0128193.\n\nShi, Baoxu, and Tim Weninger. \"Discriminative predicate path mining for fact checking in knowledge graphs.\" Knowledge-based systems 104 (2016): 123-133.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AzizvOc6Am", "reviewer": "Reviewer_pKDu", "summary": "This paper introduces KGAREVION, a novel approach that combines the non-codified knowledge of LLMs with the structured knowledge  in KGs to tackle the complexities of knowledge-intensive medical question answering. The method involves extracting relevant triples, using a fine-tuned LLM to verify the correctness of these triples, revising incorrect triples, and generating answers. This approach effectively addresses the challenges in the medical domain that require analogies and the integration of multi-source evidence, demonstrating improved performance across multiple datasets.", "questions": "**1.** What model is used for the revise process? A more detailed description should be provided. If an unfine-tuned LLM is used, does it have the capability to correct erroneous triples?\n\n**2.** How are triples involving entities not included in the knowledge graph (Incomplete Knowledge) handled? Are they further processed or directly accepted as credible knowledge?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality**: This paper proposes a new KG-based agent KGAREVION, which can handle complex and knowledge-intensive question answering tasks in the medical field. This work is original because it combines multi-source medical knowledge such as LLMs and KG, and the integration of revise into the processing flow is innovative, especially when dealing with complex problems that require precise domain knowledge.\n\n**Quality**: The quality of the paper is reliable, with each step in the KGAREVION process clearly described and supported by appropriate methodologies. The authors also validate the model's effectiveness through experiments on multiple datasets, including new and challenging ones.\n\n**Clarity**: The paper demonstrates a high level of clarity in presenting the KGAREVION model and its experimental results. The methodology section provides a clear description of the process, with well-organized main diagrams. Additionally, the tables and figures illustrating the experimental results are easy to understand.\n\n**Significance**: This research holds significant importance in the field of medical question answering. It offers an effective tool that integrates data from multiple sources, such as LLMs and KGs, thereby enhancing the ability to accurately address complex medical questions.", "weaknesses": "In the description of the methodology, certain parts lack coherence and completeness. For instance, the revise and answer stages do not specify the exact processing methods used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)