BioKGBench: A Knowledge Graph Checking Benchmark of AI Agent for Biomedical Science

## Abstract

Abstract Pursuing artificial intelligence for biomedical science, a.k.a. AI Scientist, draws increasing attention, where one common approach is to build a copilot agent driven by Large Language Models (LLMs).
However, to evaluate such systems, people either rely on direct Question-Answering (QA) to the LLM itself, or in a biomedical experimental manner. How to precisely benchmark biomedical agents from an AI Scientist perspective remains largely unexplored. To this end, we draw inspiration from one most important abilities of scientists, understanding the literature, and introduce BioKGBench. In contrast to traditional evaluation benchmark that only focuses on factual QA, where the LLMs are known to have hallucination issues, we first disentangle “Understanding Literature” into two atomic abilities, i) “Understanding” the unstructured text from research papers by performing scientific claim verification, and ii) Ability to interact with structured Knowledge-Graph Question-Answering (KGQA) as a form of “Literature” grounding. We then formulate a novel agent task, dubbed KGCheck, using KGQA and domain-based Retrieval-Augmented Generation (RAG) to identify the factual errors of existing large-scale knowledge graph databases. We collect over two thousand data for two atomic tasks and 225 high-quality annotated data for the agent task. Surprisingly, we discover that state-of-the-art agents, both daily scenarios and biomedical ones, have either failed or inferior performance on our benchmark. We then introduce a simple yet effective baseline, dubbed BKGAgent. On the widely used popular knowledge graph, we discover over 90 factual errors which provide scenarios for agents to make discoveries and demonstrate the effectiveness of our approach. The code and data are available at https://github.com/westlake-autolab/BioKGBench .

### 1 Introduction

Large Language Models (LLMs) are so powerful that they facilitate nearly every aspect of daily life and work right now, even research [, , , ]. Observing their marvelous successes in text generation [, ], text summarization [, ], and other tasks [, ], along with their consistent failures such as hallucinations [, ], one can conclude that LLMs are powerful in certain tasks involving large-scale unstructured data like daily text or images, but relatively powerless when dealing with data-hungry scenarios. As such, researchers then construct AI agents [, ] assisting LLMs with external tools to extend the capabilities of LLMs. These attempts are fruitful in many fields, including autonomous computer [], shopping web-agent [], code developing [], society simulation [, ], etc. A natural subsequent attempt is to develop AI agents to simulate scientists, aiding or even taking over the process of scientific discovery [].

As in Figure 1, existing attempts can be grouped into two categories: i) to build an AI agent for a specific task, such as Question Answering (QA) in a specific domain []; ii) to encompass multiple AI agents to formulate a multi-agent system as the copilot of scientists, automating certain scientific activities, such as experiment result analysis [, ].

Figure: Figure 1: (Left) Previous benchmarks for domain-specific AI Agents either focus on the low-level tasks like question answering or are embedded in a complicated pipeline as a scientist copilot. (Right) We close the gap by constructing a knowledge graph checking task that consists of two atomic sub-tasks: Knowledge Graph Question Answering (KGQA) and Scientific Claim Verification (SCV), to provide a better evaluation of AI Agents in biomedical science domain.
Refer to caption: x1.png

Literature review is the most critical ability that a scientist should possess [, ]. It does not only involve reading and memorizing, but also requires scientists to understand and critically analyze. Researchers and scientists widely spend a significant amount of time in reading recent works. To save human efforts in scientific discovery, it is necessary for AI scientists to be able to accurately understand and analyze the existing research. Many researchers have dedicated to literature understanding in AI agents [, ], while a systematic evaluation system is missing and even underexplored. The current finest evaluation system [] consists of multiple-choice questions extracted from literature, which cannot fully reveal the underlying reasoning regime of an agent’s success or failure, leaving no clue for future advancement nor indicating whether the agent understands the reasoning rationale or merely memorizes data patterns.

On the other hand, another crucial research direction is to help AI agents capture the underlying logic of literature through domain-specific Knowledge Graphs (KGs) [, ].
KGs store massive knowledge triples in a graph-structured format [, ], complementing LLMs with external knowledge while providing frameworks for interpretation and reasoning []. However, manually constructing such KGs is both intellectually and physically intensive. These domain-specific KGs require annotators with profound domain-specific knowledge, leading to high costs to create or maintain the knowledge graphs. As such, we observe that the existing and well-known biomedical KGs [, ] are not fully reliable due to outdated information. We attribute such discrepancy to the static nature of KGs, which lack mechanisms for dynamic updates to align with the evolution of external knowledge sources.

In this paper, we propose a novel agent evaluation benchmark BioKGBench to address both challenges simultaneously. As in Figure 1 (right), the ultimate goal of our benchmark is to verify the correctness of nodes and triples in the knowledge graph based on various information, including papers and well-maintained databases. We dub this task Knowledge Graph Checking (KGCheck). Agents need to first query the information recorded on the KGs as directed, then cross-reference this information with external literature or databases to combat hallucinations. This task evaluates the agents’ capacities to both process and understand structured data (like KGs) and unstructured data (like literature). It is worth mentioning that the process of verifying knowledge within KGs closely mirrors the methodology of human scientific research, including database queries and extensive literature reviews. This similarity not only underscores the task’s relevance to real-world scientific inquiry but also provides intriguing insights. Furthermore, we decompose this task into two more atomic subtasks: Knowledge Graph Question Answering (KGQA) and Scientific Claim Verification (SCV), enabling a more detailed evaluation of the agents’ capabilities in processing and understanding of structured and unstructured data, respectively.

We extensively analyze existing AI agents on our benchmark and find that none of the existing agents can accomplish our tasks without moderate adaptation. Therefore, we introduce our agent BKGAgent, the first agent framework to interact with external knowledge graphs as well as research papers. Experiments demonstrate fascinating results that our agent is capable of discovering real conflicts in the existing large-scale datasets. Within 225 professional-annotated data in Clinical Knowledge Graph (CKG) [], our agent BKGAgent successfully identified some conflicting or missing pairs. This evidence further supports the academic value of our agent by providing researchers with a tool to update their own knowledge bases, offering substantial potential in both academic and commercial markets.

### 2 Related Work

Science Agent. The swift progression of large language models (LLMs) has catalyzed the widespread deployment of intelligent agents across diverse fields, notably within the science domain. Notable examples include ChemCrow [] and Coscientist [] in the field of chemistry, DoInstruct [] in ocean science, and GeneGPT [], Almanac [], MedAgents [] in biomedical domain, etc. Among them, biomedical agents, in particular, have garnered significant attention due to its critical importance. Biomedical agents [] impact areas ranging from hybrid cell simulation [], the design of cellular circuits [] to the development of new therapies [] and so on. We posit that biomedical agents will emerge as a focal point of research. However, the current benchmark in this field remains inadequate. For instance, MedAgents is evaluated in MedQA [], MedMCQA [], PubMedQA [], relying heavily on inherent knowledge of LLMs, which leads to hallucinations easily. Our proposed BioKGBench is a dynamic benchmark that evaluates the capabilities of agents in utilizing external tools and knowledge retrieval, thereby addressing this gap.

Agent Benchmark. As agents are progressively applied across various domains, the urgency to construct corresponding benchmarks is escalating. Currently, the majority of benchmarks for evaluating agents adopt the approach of evaluating LLM-as-Agent [], linking LLMs to external frameworks to assess their performance on specific tasks. For instance, AgentBench [] is a general benchmark for evaluating an agent’s reasoning and decision-making capabilities, SWE-bench [] assesses an agent’s proficiency in software engineering, and AgentClinic [] examines an agent’s performance in a simulated clinical environment. However, a benchmark in AI Scientist perspective remains largely unexplored. Our benchmark originates from this perspective, taking the processing and understanding of large-scale data scenarios as the entry point, representing an initial attempt in this direction.

Agent Integrating LLMs and KGs. The collaborative use of LLM and KG has become one of the leading methodologies in contemporary agent design, aimed at alleviating uncertainties stemming from the intrinsic mechanisms of LLMs [, , ]. This paradigm not only capitalizes on the generalization ability of LLMs but also employs KGs as an external, trustworthy, and structured data source, thereby achieving reasoning proficiency that strikingly emulates human intellect[]. For instance, StructGPT [] boosts an LLM’s performance on general questions by tapping into the information from a supplied KG. Similarly, KG-Agent [] leverages knowledge from KGs, synthesizing instruction data for fine-tuning an open-sourced LLM, thereby achieving competitive performance on general question-answering tasks. However, to our knowledge, while this paradigm has been widely applied to the general question-answering area, its potential remains untapped in the biomedical field. BKGAgent, hence, is poised to fill this gap.

### 3 BioKGBench

**Table 1: Statistics of our BioKGBench.**
| Task | Metric | Scope | Data |  |  |
| --- | --- | --- | --- | --- | --- |
| Dev | Test | All |  |  |  |
| KGQA | F1 | KG | 60 | 638 | 698 |
| SCV | Acc. | Text | 120 | 1,265 | 1,385 |
| KGCheck | Acc. | KG + T | 20 | 205 | 225 |
|  |  |  |  |  |  |

Here, we present our benchmark in detail. As aforementioned, one key ability of “AI Scientists” is to understand domain knowledge. However, current LLM-driven agent systems inevitably suffer from hallucinations as a consequence of the statistical nature of LLMs along with the lack of scientific training data compared to data from daily scenarios. We notice that a recent trend in research is to use AI agents to leverage external tools to address these limitations [, ]. Drawing inspiration from this, we design two atomic abilities to evaluate AI scientists, i) Knowledge Graph Question Answering (KGQA) aiming to address the hallucination issue by grounding the knowledge with structured knowledge graphs; and ii) Scientific Claim Verification (SCV) based on retrieved text from peer-reviewed research papers. In addition, we propose an encompassing task combining these two atomic abilities, to perform Knowledge Graph Checking (KGCheck) as shown in Figure 1. The motivation behind this stems from our interviews with experts from biomedical domains. Their answers to the question “What is the most expected AI agent you would like to use in your daily research?” often included an AI agent that helps in extensive literature review and claim verification. We report the statistics over the scopes of knowledge search, including knowledge graphs and academic literature, in Table 1.

Figure: Figure 2: The sub-graph of the Clinical Knowledge Graph (CKG) retains 12 types of nodes and 18 kinds of relationships.
Refer to caption: extracted/5699777/figs/subKG.jpg

#### 3.1 Atomic Ability

##### 3.1.1 Knowledge Graph Question Answering

This atomic task in the benchmark is to evaluate the agents’ ability to interact with structured Knowledge Graph Question Answering as a grounding of academic literature. Without loss of generality, we choose Clinical Knowledge Graph (CKG) [] as the source of our data, which is one of the most popular large-scale knowledge graph databases in the biomedical domain.
CKG is a knowledge graph database with data imported from diverse biomedical databases, aimed at streamlining automated knowledge discovery through the graph’s extensive information.

**Table 2: Statics of three different reasoning types of KGQA dataset.**
| Reasoning Type | Graph | Example Question | Question Types | % |
| --- | --- | --- | --- | --- |
| One-hop |  | What proteins does the protein O94842 act on? | 8 | 56.0 |
| Multi-hop |  | What diseases are associated with the protein encoded by the gene KCNS1? | 4 | 28.7 |
| Conjunction |  | Which pathway are the proteins P02778 and P25106 both annotated in? | 4 | 15.3 |
|  |  |  |  |  |

As the original database is unnecessarily large, we focus on a sub-graph to mitigate the challenge while preserving all relevant information. Starting from the origin of CKG—protein, we select the sub-graph to contain exactly 12 categories of biological entities, as indicated in Figure 2. Thus, the sub-graph consists of 484,955 entities (nodes) across 12 categories (Biologically defined) and 18,959,943 relationships (edges) of 18 types, with each type consisting of relationships between a unique pair of entity categories.

After the sub-graph is ready, we construct the question set for the Question Answering (QA) database in two steps. We first handcraft question templates by selecting biomedical fields and pinpointing entities and relations in the CKG. Natural language questions were constructed in various formats, ensuring their accuracy through peer reviews and expert consultations. We then expand our dataset with autogenerated questions by matching CKG data to constructed QA templates, resulting in the generation of 698 questions across three reasoning types and 16 question categories (refer to Table 2).

In this task, we outfit LLMs with a set of atomic KG-querying tools and ask them to answer biomedical questions by querying the provided KG. The responses will be compared with the gold answers and evaluated using the F1 score, where the gold answer to the input question is typically characterized by a set of KG entities. It is noteworthy that our KGQA is built upon a biomedical KG rather than a common sense KG, the former being characterized by its highly dense information storage. This compactness greatly increases the complexity for agents to employ tools and perform reasoning on the structured graph to complete QA tasks. This task enables the development of assessing the robustness and tool learning ability of agents built upon various LLMs, and hopefully it would aid in guiding agents to leverage the extensive biomedical knowledge within the KG, thereby propelling scientific discovery.

##### 3.1.2 Scientific Claim Verification

**Table 3: Examples of reconstructed dataset for SCV, where data from PubMedQA is converted from QA to declarative claims. "NEI" stands for "Not Enough Information".**
| Example Claim | Label | % |
| --- | --- | --- |
| A deficiency of folate increases blood levels of homocysteine. | Support | 65.2 |
| Therapeutic anticoagulation in the trauma patient is safe. | Refute | 33.1 |
| Sternal fracture in growing children is a rare and often overlooked fracture. | NEI | 1.7 |
|  |  |  |

This task is designed to evaluate LLMs’ understanding of unstructured text from research papers in a retrieval-augmented generation manner. Following the definition in [], the task is to identify evidence related to the claim from the research literature and give a verdict of “Support”, “Refute”, or “NEI” (Not Enough Information) based on it. We reconstruct two high-quality biomedical datasets, PubMedQA [] and SciFact [], into one dataset for SCV, yielding a corpus constituted of abstracts derived from 5,664 scholarly articles, alongside a dataset comprising 1,385 biomedical claims, as shown in Table 3.

#### 3.2 Agent Task

Building upon the atomic abilities, we propose a novel and comprehensive task, KGCheck. This task necessitates the initial application of the tool-query functionality to extract information from the KG. Subsequently, it employs the RAG approach or database access to procure evidence pertaining to the queried information, facilitating a determination of either “Support” or “Refute”. This methodology enables agents to scrutinize the knowledge encapsulated within a large-scale KG, a venture of particular importance considering the prevalence of inaccuracies within numerous datasets, including prominent ones such as ImageNet [].

**Table 4: Four different checking types of KGCheck.**
| Check Type | Graph | % | Support: Refute |  |
| --- | --- | --- | --- | --- |
| Node | Existence | ? | 20.0 | 71.0:29.0 |
|  | Attribute | ? | 24.4 |  |
| Triple | Existing | ab? | 25.8 | 46.4:53.6 |
|  | Potential | ab? | 29.8 |  |
|  |  |  |  |  |

For this task, we collect 225 high-quality annotated data, as illustrated in Table 4. Given the massive data encapsulated within KGs via triples, we delineate the inspection process into two distinct categories: single-node and triple-based. The single-node inspection is divided into node existence and attribute value assessments, while the triple inspection encompasses scenarios with and without edges between two nodes:

- •
Existence: We note that databases may excise entries during updates due to inaccuracies or redundancies, whereas KGs remain static post-construction, similar to LLMs in some respects. If nodes corresponding to obsolete entities persist in the KG, the label is “Refute”; if they are congruent with real-time updated external databases, the label is “Support”.
- •
Attribute: Our KG is characterized by high information density, with each node and edge encapsulating numerous attribute values, which we scrutinize for accuracy and completeness.
- •
Existing Relationship: We check whether existing edges contradict information from external, real-time updated databases and literature. If external knowledge corroborates the relationship, the label is “Support”; conversely, it is “Refute”.
- •
Potential Relationship: If a relationship is confirmed by databases or literature but is not represented in the KG, the label is “Refute”; otherwise, it is “Support”.

Despite utilizing the latest databases (as of May 2024), we identified errors within the KG, evidenced by 96 “Refute” annotations. These data are valuable and provide scenarios for agents to comprehend knowledge from heterogeneous sources and make discoveries.

#### 3.3 BKGAgent: A Simple Baseline

We propose a biomedical knowledge-graph agent (BKGAgent), as shown in Figure 3. It’s a multi-agent framework based on *langgraph* [], capable of retrieving information from knowledge graph and cross-validating its correctness with multiple information sources. Our framework is comprised of three agents: the team leader for the progress control, the KG agent for information retrieval from KG, and the validation agent for checking the correctness of the information from KG. This setup simulates the workflow of a human research team, where a leader supervises the assistants’ work and makes the final decision based on their feedback. Additionally, the tool executor is solely responsible for executing functions, and is not based on LLMs.

Figure: Figure 3: Framework of our BKGAgent.
Refer to caption: extracted/5699777/figs/BKGAgent.jpg

When a user assigns a task, the leader initially breaks down the task and announces the plan. Then the KG agent is activated to retrieve task-related information from the KG. This involves specifying the tool and its arguments to the tool executor, interpreting the tool result, and communicating it back to the leader. After that, the validation agent is called to verify the information with a workflow similar to that of the KG agent. Finally, the leader will draw a conclusion and return it to the user.

### 4 Experiments

#### 4.1 Main Results and Analysis: Atomic Abilities

**Table 5: Test set (standard) results of two easy tasks: KGQA, SCV. Bold/underline and red/blue indicate the best and second in the subgroup and overall.**
| LLM<br>Type | Models | KGQA | SCV |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| F1 | EM | Executability | Accuracy | Right Quotes | Error |  |  |
| API | GPT-4 [] | 81.8 | 79.2 | 88.4 | 83.9 | 87.7 | 0.4 |
| GLM-4 [] | 72.4 | 70.4 | 82.7 | 86.9 | 86.5 | 0.6 |  |
| OSS<br>(Large) | Qwen1.5-72B-Chat [] | 74.7 | 72.2 | 96.1 | 85.7 | 83.3 | 0.1 |
| Llama-3-70B-Instruct [] | 80.7 | 77.8 | 97.0 | 85.9 | 86.6 | 0.2 |  |
| DeepSeek-LLM-67B-Chat [] | 69.6 | 66.8 | 86.3 | 76.6 | 82.6 | 0.4 |  |
| OSS<br>(Medium) | Qwen1.5-32B-Chat [] | 64.6 | 62.1 | 83.0 | 79.7 | 83.0 | 0.4 |
| Qwen1.5-14B-Chat [] | 66.0 | 61.6 | 78.7 | 66.1 | 67.4 | 0.2 |  |
| Baichuan2-13B-Chat [] | 43.7 | 42.0 | 82.2 | 26.3 | 35.8 | 33.6 |  |
| OSS<br>(Small) | Llama-3-8B-Instruct [] | 54.7 | 51.3 | 84.8 | 78.5 | 83.3 | 0.5 |
| Qwen1.5-7B-chat [] | 44.5 | 40.3 | 77.9 | 72.5 | 39.1 | 2.2 |  |
| OSS<br>(MoE) | Mixtral-8x7B-Instruct-v0.1 [] | 70.1 | 67.9 | 84.7 | 77.8 | 82.5 | 2.3 |
| Starling-LM-alpha-8x7B-MoE-GPTQ [] | 12.4 | 10.9 | 30.7 | 55.0 | 56.2 | 0.1 |  |
| Qwen1.5-MoE-A2.7B-Chat [] | 28.7 | 26.7 | 71.9 | 55.0 | 57.8 | 3.0 |  |

The detailed experimental results of atomic abilities evaluation on LLMs are shown in Table 5, and we summarize our key findings as follows:

- •
Disparity between open-source and commercial API models. Commercial API models like GPT-4 and GLM-4 generally outperform open-source models in several key metrics. GPT-4, for example, consistently achieves higher scores in both KGQA and SCV tasks, highlighting the advantage of proprietary training techniques and larger computational resources.
- •
Strong performance of open-source large models. Some large OSS models, such as Llama-3-70B-Instruct, perform competitively, sometimes surpassing API models in specific metrics. Llama-3-70B-Instruct, in particular, excels in KGQA executability, suggesting that optimized training can enable open-source models to rival or exceed commercial counterparts.
- •
Model parameters do not always correlate with better performance. In the OSS (Medium) and OSS (Small) categories, smaller models like Llama-3-8B-Instruct sometimes outperform larger models like Qwen1.5-32B-Chat in SCV tasks, indicating that model architecture, training data quality, and fine-tuning strategies significantly impact performance. Notably, Qwen1.5-14B-Chat outperforms Qwen1.5-32B-Chat in KGQA, suggesting the latter’s pre-training may be insufficient.
- •
Domain-specific models lack transferability. DeepSeek-LLM-67B-Chat excells in mathematical problems [], but underperforms in biomedical-related tasks, highlighting its lack of cross-domain transferability. This suggests that specialization in one area may compromise generalizability.
- •
Inconsistent performance of MoE models. While Mixtral-8x7B-Instruct-v0.1 performs well in both KGQA and SCV tasks, other MoE models like Starling-LM-alpha-8x7B-MoE-GPTQ and Qwen1.5-MoE-A2.7B-Chat show significantly lower scores. This inconsistency suggests that the effectiveness of MoE models heavily depends on the implementation and integration of the expert models. Additionally, Mixtral-8x7B-Instruct-v0.1, though strong in main metrics, struggles with controlling response format, indicating that individual expert models still require improvement.
- •
Biomedical knowledge embedded in model parameters. The new metric “right quote” for SCV assesses the alignment of retrieved quotes with ground truth evidence. Some models, such as GLM-4, Qwen1.5-72B-Chat, and Qwen1.5-7B-Chat, exhibit higher accuracy metrics than “right quote” metrics. This suggests these models can accurately assess input claims even without sufficient literature evidence, indicating they possess specialized biomedical knowledge.

Figure: Figure 4: Llama-3-70B-Instruct’s performance in RAG across different scopes of literature.
Refer to caption: extracted/5699777/figs/restrict_corpus_exp.png

We also conduct an ablation experiment on three scopes of RAG, as shown in Figure 4, where ‘all’ refers to the abstract of 5,664 articles, ‘partial’ denotes the 1,888 abstracts containing ground truth evidence of claims, and ‘match’ corresponds to the abstracts of the ground truth evidence for the claims. Interestingly, we observe an unexpected phenomenon where the model’s performance in the ‘match’ setting only increases in terms of the right quotes metric, while the accuracy metric actually decreases. In the ‘all’ setting, we initially thought that irrelevant literature would introduce interference, but the accuracy metric, on the contrary, increases. This suggests that there is a potential connection among the extensive literature, where large models exhibit a form of “analogical reasoning”. This provides us with insights for conducting extensive literature research in simulating human scientific research.

#### 4.2 Main Results and Analysis: BKGAgent

**Table 6: Process and outcome metrics of BKGAgent on KGCheck task, where bold numbers denote better scores among the two models while green numbers denote 100% acuracy.**
| Model | KG Query Task | Validation Task | Final Result |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Tool Selection | Executability | Tool Selection | Execurability | Exact Match | Executability |  |
|  | Web Database KGCheck |  |  |  |  |  |
| GPT-4 [] | 65.0 | 65.0 | 88.1 | 88.8 | 64.5 | 96.9 |
| Llama-3-70B-Instruct [] | 96.9 | 96.9 | 97.5 | 97.5 | 38.1 | 100.0 |
|  | Publication Database KGCheck |  |  |  |  |  |
| GPT-4 [] | 67.7 | 67.7 | 69.2 | 69.2 | 61.5 | 95.4 |
| Llama-3-70B-Instruct [] | 100.0 | 100.0 | 95.4 | 95.4 | 41.5 | 63.1 |
|  |  |  |  |  |  |  |

The experiment results including both process and final answer are shown in Table 6, where the executability metric of the assistant agent means whether it is correctly activated. The exact match score of the final result agrees with what we observed in the atomic abilities experiments: GPT-4 surpasses Llama-3-70B-Instruct in the final result, which shows the best performance in the atomic abilities experiments. We discovered some interesting phenomena based on the analysis of BKGAgent chat history.

- •
Leader agent performance significantly influences the team behavior. While the behavior of the assistant agents like KG agent can be modified by the leader’s instruction, the leader itself lacks action-related feedback from others, meaning that a bad decision made by the leader may lead to a catastrophe. We found four common error cases induced by the leader, as shown in Figure 5. Among these cases, the leader either fails to give effective instructions to team members, becomes trapped in repeated self-talks, or attempts to perform the tasks that are meant for the assistants.
- •
Assistant agents can always select the right tool. As shown in the results, assistant agents’s executability is almost the same as its right tool selection rate, which means once the agent is correctly woken up, it has a good chance of selecting the right tool.
- •
Good process results do not necessarily mean a good final result. Surprisingly, the agent based on Llama surpasses the one based on GPT in terms of assistant agents’ tool selection. However, the final result is far from satisfactory. We watch for the chat history of the Llama-based agent and find out that while the right answer can be found in the history of the agent in many cases, the team starts chatter instead of sending the result to the user thus leading to a final wrong answer.
- •
Balanced abilities of BKGAgent. We classify our tasks into two categories by the way the agent checks the information from KG, namely the web database (UniProt [] and STRING []) check task and the publication database check task. The performance of them is separately evaluated. Considering the exact match score of the final result, both the GPT-based agent and Llama-based agent show similar performance in both types of tasks, indicating BKGAgent does not show any preference for specific tasks.

Figure: Figure 5: Error analysis. Here, we show a failure case due to a leader’s various mistakes: the hallucination of the leader misleading the later task or using the wrong process, the leader producing unnecessary repeated tasks and misunderstanding leads to the wrong process.
Refer to caption: extracted/5699777/figs/errorcase.jpg

#### 4.3 Capability Analysis

Biomedical agents are not widely employed, as most tasks rely solely on models (refer to Appendix F). Additionally, most agents are primarily focused on QA tasks. For example, MedAgents[] employs a Multi-disciplinary Collaboration (MC) framework and was tested on nine QA datasets. General agents such as the multi-agent framework AutoGen and the single-agent framework AutoGPT are capable of performing web searches and database retrieval enhancements through tool calls. However, since accessing knowledge graphs and biomedical information is beyond daily usage, and the information directly gained from the web being unreliable, it is challenging to achieve the same level of biomedical information retrieval and verification effectiveness using existing general agent frameworks.

As presented in Table 7, our framework encompasses the basic abilities, especially for retrieval in the biomedical domain. The range of our information for KG result validation includes the UniProt database, STRING database, and publications selected from PubMed, which surpasses the limited information from a simple web search.

**Table 7: Comparison of capabilities for BKGAgent and other frameworks.**
| Scenario | Framework | MA | KGq | IR | IF |
| --- | --- | --- | --- | --- | --- |
| Common | AutoGPT(^1^11[https://news.agpt.co/](https://news.agpt.co/)) | ✗ | ✗ | ✗ | ✗ |
| AutoGen [] | ✔ | ✗ | ✗ | ✔ |  |
| HuggingGPT [] | ✔ | ✗ | ✗ | ✔ |  |
| OpenAgents [] | ✔ | ✗ | ✗ | ✔ |  |
| AgentVerse [] | ✔ | ✗ | ✗ | ✔ |  |
| Xagent [] | ✔ | ✗ | ✗ | ✔ |  |
| BabyAGI [] | ✔ | ✗ | ✗ | ✔ |  |
| Domain | MedAgents [] | ✔ | ✗ | ✗ | ✔ |
| gpt-researcher(^2^22[https://gptr.dev/](https://gptr.dev/)) | ✔ | ✗ | ✗ | ✔ |  |
| BKGAgent(ours) | ✔ | ✔ | ✔ | ✔ |  |
| MA=multi-agent; KGq=KG-query; |  |  |  |  |  |
| IR=information retrieval; IF=intervention free |  |  |  |  |  |

### 5 Conclusion

We present BioKGBench, an interactive benchmark that encompasses the KGCheck task with two atomic capabilities for assessment: KGQA and SCV. KGCheck offers agents a valuable scenario for detecting knowledge hallucination within large-scale data, akin to the experience of researchers making discoveries amidst voluminous literature in the real world. We conduct evaluations of these two atomic capabilities across 13 LLMs and select the top-performing open-source model, Llama-3-70B-Instruct, and API-based model, GPT-4, to construct BKGAgent—–a multi-agent system serving as the baseline. Comparisons with existing general and biomedical agents revealed their poor performance due to the absence of certain process capabilities, thereby demonstrating the challenging nature of our benchmark. We expect BioKGBench to serve as a valuable endeavor towards paving the path for biomedical agents to become AI scientists.

Limitations and Future Work.
In KGCheck, we guide agents to identify knowledge-based errors within the KG by providing them with specific instructions. This process involves atomic-level inspections from single nodes to triples, which agents could potentially implement autonomously. Future work will explore how agents can autonomously conduct real-time error detection in large datasets by leveraging logic rules and prior knowledge. Additionally, our case studies reveal that despite agents’ current underperformance in KGCheck tasks (as shown in Figure 5), their occasional correct responses are often due to chance, attributed to the binary nature of answers which inherently allows a 50% success rate from random guesses. More granular metrics instead of exact match score should be introduced for a more accurate evaluation, which will be part of future work.

## Part I Appendix

### Appendix A Dataset and Code

#### A.1 Dataset and Code Access

Following NeurIPS Dataset and Benchmark Track guidelines, our data and code are available at [https://github.com/westlake-autolab/BioKGBench](https://github.com/westlake-autolab/BioKGBench) under MIT license. More details are also documented at this address. The Croissant metadata record is available at [https://huggingface.co/api/datasets/AutoLab-Westlake/BioKGBench-Dataset/croissant](https://huggingface.co/api/datasets/AutoLab-Westlake/BioKGBench-Dataset/croissant).

#### A.2 Responsibility Statement

We declare that we assume full responsibility for any violations of rights. We confirm that the data used in this paper is properly licensed and ensure that its use complies with all relevant laws, regulations, and licensing agreements.

### Appendix B Datasheet

Here, we provide a detailed description of our benchmark dataset, following the guidelines of the “Datasheet for Datasets” [].

#### B.1 Motivation

Our benchmark dataset was created to address the lack of benchmarks for evaluating biomedical agents from the perspective of an “AI scientists”. In [], it is stated that “AI scientists” can be realized as AI agents supported by humans, LLMs, ML models, and other tools like experimental platforms that cooperate to solve complex tasks. However, the current evaluation methods for biomedical agents remain unexplored, limited to simple question-answering tasks, which not only fail to avoid the hallucination problem inherent in solely relying on LLMs but also do not assess agents’ abilities to utilize external tools and knowledge bases. Our proposed benchmark fills this gap by designing tasks ranging from easy to hard, based on two atomic capabilities: tool-query and memory-RAG. These tasks evaluate the agents’ ability to leverage external support, including external knowledge and tools, when handling large-scale and multi-modal data. Moreover, Our data collected for KGCheck, the most challenging task, provides scenarios for agents to comprehend knowledge from heterogeneous sources and make discoveries.

The conception and construction of this dataset were jointly completed by the biomedical experts and AI researchers listed in the author list.

#### B.2 Composition

We provide the necessary data for constructing the knowledge graph, literature for RAG, as well as the development and test data for KGQA, SCV, and KGCheck, where knowledge graph and literature are external knowledge sources provided for agent.

The knowledge graph is derived from a subset of Clinical Knowledge Graph (CKG) []. We specifically retain twelve key node types to ensure there is no loss of generality: Protein, Biological process (BP), Molecular function (MF), Cellular component (CC), Amino acid sequence, Tissue, Protein structure, Pathway, Modified protein, Modification, Disease, and Gene. The statistics of the triples in our knowledge graph are presented in Table 8. Detailed information stored in our knowledge graph is shown in Table 9.

**Table 8: The data statistics of our knowledge graph drawn from CKG.**
| Head Node | Tail Node | Relation | Number |
| --- | --- | --- | --- |
| Protein | Protein_structure | HAS_STRUCTURE | 271,512 |
|  | Amino_acid_sequence | HAS_SEQUENCE | 20,598 |
|  | Cellular_component | ASSOCIATED_WITH | 3,796,383 |
|  | Tissue | ASSOCIATED_WITH | 7,117,321 |
|  | Disease | ASSOCIATED_WITH | 5,882,437 |
|  | Molecular_function | ASSOCIATED_WITH | 85,013 |
|  | Biological_process | ASSOCIATED_WITH | 153,219 |
|  | Protein | ACTS_ON | 985,376 |
|  | Pathway | ANNOTATED_IN_PATHWAY | 357,739 |
|  | Protein | CURATED_INTERACTS_WITH | 3,448 |
|  | Modified_protein | HAS_MODIFIED_SITE | 4,498 |
| Disease | Disease | HAS_PARENT | 16,058 |
| Modified_protein | Protein | IS_SUBSTRATE_OF | 6,633 |
|  | Modification | HAS_MODIFICATION | 4,559 |
| Gene | Protein | TRANSLATED_INTO | 179,854 |
| Biological_process | Biological_process | HAS_PARENT | 49,081 |
| Molecular_function | Molecular_function | HAS_PARENT | 13,659 |

**Table 9: Details of the information stored in the nodes of our knowledge graph.**
| Entity Type | Content | Example |
| --- | --- | --- |
| Protein | name, id, accession, synonyms | {‘name’: ‘PLEKHG6’, ‘id’: ‘Q3KR16’, ‘accession’: ‘PKHG6_HUMAN’, ‘synonyms’: [‘PKHG6_HUMAN’, ‘PLEKHG6’, ‘9606.ENSP00000380185’, ‘ENSG00000008323’], ‘taxid’: ‘9606’} |
| Disease | name, description, id(DOID), type, synonyms | {‘synonyms’: [‘sulfamethoxazole allergy’, ‘SMX allergy’, ‘SMZ allergy’, ‘sulphamethoxazole allergy’], ‘name’: ‘sulfamethoxazole allergy’, ‘description’: ‘A drug allergy that has_allergic_trigger sulfamethoxazole. [url:[https://www.ncbi.nlm.nih.gov/pubmed/7602118](https://www.ncbi.nlm.nih.gov/pubmed/7602118)]’, ‘id’: ‘DOID:0040016’, ‘type’: ‘-26’} |
| Protein structure | link, id, source | {‘link’: [http://www.rcsb.org/structure/6XWD](http://www.rcsb.org/structure/6XWD), ‘id’: ‘6XWD’, ‘source’: ‘Uniprot’} |
| Amino acid sequence | sequence, header, source, id, size | {‘sequence’: ‘LRGAAGRLGGGLLVL’, ‘size’: ‘15’, ‘header’: ‘tr|A0A0A0MTA2|A0A0A0MTA2_HUMAN’, ‘source’: ‘UniProt’, ‘id’: ‘A0A0A0MTA2’} |
| Cellular component | name, description, id, type, synonyms | {‘name’: ‘Golgi membrane’, ‘description’: ‘The lipid bilayer surrounding any of the compartments of the Golgi apparatus. [GOC:mah]’, ‘id’: ‘GO:0000139’, ‘type’: ‘-21’, ‘synonyms’: [‘Golgi membrane’]} |
| Molecular function | name, description, id(GO), type, synonyms | {‘name’: ‘polymeric immunoglobulin binding’, ‘description’: ‘Interacting selectively and non-covalently with a J-chain-containing polymeric immunoglobulin of the IgA or IgM isotypes. [GOC:add, ISBN:0781735149]’, ‘id’: ‘GO:0001790’, ‘type’: ‘-21’, ‘synonyms’: [‘polymeric immunoglobulin binding’]} |
| Biological process | name, description, id(GO), type, synonyms | {‘synonyms’: [‘mitochondrion inheritance’], ‘name’: ‘mitochondrion inheritance’, ‘description’: ‘The distribution of mitochondria, including the mitochondrial genome, into daughter cells after mitosis or meiosis, mediated by interactions between mitochondria and the cytoskeleton. [GOC:mcc, PMID:10873824, PMID:11389764]’, ‘id’: ‘GO:0000001’, ‘type’: ‘-21’} |
| Pathway | name, description, linkout, id, source | {‘name’: ‘Antigen processing: Ubiquitination & Proteasome degradation’, ‘description’: ‘Antigen processing: Ubiquitination & Proteasome degradation’, ‘linkout’: [https://reactome.org/PathwayBrowser/##/R-HSA-983168](https://reactome.org/PathwayBrowser/##/R-HSA-983168), ‘id’: ‘R-HSA-983168’, ‘source’: ‘Reactome’} |
| Tissue | name, description, id, type, synonyms | {‘name’: ‘stratum basale’, ‘description’: ‘The deepest layer, as of the epidermis or the endometrium. In the epidermis it is a single layer of cells. In the endometrium it provides the regenerative tissue after menstrual loss of the functional layer. [Dorlands_Medical_Dictionary:MerckMedicus]’, ‘id’: ‘BTO:0004680’, ‘type’: ‘-25’, ‘synonyms’: [‘stratum basale’, ‘basal layer’]} |
| Modified protein | sequence_window, protein, position, source, id, residue | {‘sequence_window’: ‘MEPAPARsPRPQQDP’, ‘protein’: ‘P29590’, ‘position’: ‘8’, ‘source’: ‘SIGNOR’, ‘id’: ‘P29590_S8-p’, ‘residue’: ‘S’} |
| Modification | synonyms, name, description, id, type | {‘synonyms’: [‘Unimod’, ‘Source: “none”’], ‘name’: ‘Unimod’, ‘description’: ‘Entry from Unimod. [PubMed:18688235]’, ‘id’: ‘MOD:00003’, ‘type’: ‘-41’} |
| Gene | taxid, synonyms, name, id, family | {‘taxid’: ‘9606’, ‘synonyms’: [‘54843’, ‘ENSG00000137501’, ‘OTTHUMG00000166977’, ‘uc010rti.4’, ‘AJ303364’], ‘name’: ‘synaptotagmin like 2’, ‘id’: ‘SYTL2’, ‘family’: “‘Protein phosphatase 1 regulatory subunits|Synaptotagmin like tandem C2 proteins”’} |

Besides the knowledge graph, literature also serves as an external source of knowledge. We provide a corpus of 5,664 abstracts (under ODC-By 1.0) for SCV and 51 full papers for KGCheck. The 5,664 abstracts are sourced from existing datasets SciFact [] (under CC BY 4.0) and PubMedQA [] (under MIT license), while the 51 full papers, all of which are open access, were selected by experts based on entries in the IntAct [] database and CKG. Table 10 summarizes the sources of the abstracts, and Figure 6 describes the literature with more details. Table 11 summarizes the sources of the 51 full papers, and Figure 7 provides more details about the literature.

**Table 10: The 5,664 papers come from 1,484 journals. Due to space limitations, we only list the names of journals with an IF greater than 70 and use ‘others’ to represent journals with an IF less than 70.**
| Journal | Count |
| --- | --- |
| Nature reviews. Microbiology | 3 |
| CA: a cancer journal for clinicians | 3 |
| The Lancet. Infectious diseases | 3 |
| Nature reviews. Drug discovery | 5 |
| Nature reviews. Molecular cell biology | 14 |
| Nature reviews. Immunology | 21 |
| Lancet (London, England) | 46 |
| The New England journal of medicine | 46 |
| BMJ (Clinical research ed.) | 90 |
| JAMA | 113 |
| Nature medicine | 138 |
| others | 5182 |
| Total | 5664 |

**Table 11: Journal distribution of the 51 full papers**
| Journal | Count |
| --- | --- |
| Brain research | 1 |
| Molecular & cellular proteomics : MCP | 1 |
| Aging cell | 1 |
| Cell reports | 1 |
| PloS one | 1 |
| Genes to cells : devoted to molecular & cellular mechanisms | 1 |
| EMBO reports | 1 |
| IUBMB life | 1 |
| The Journal of allergy and clinical immunology | 1 |
| The EMBO journal | 1 |
| Open biology | 1 |
| Nature communications | 1 |
| Pigment cell & melanoma research | 1 |
| Molecular biology of the cell | 1 |
| Mobile DNA | 1 |
| Journal of molecular biology | 1 |
| Nutrients | 1 |
| Biological research for nursing | 1 |
| Genes & development | 1 |
| Developmental cell | 1 |
| Bone | 1 |
| Cancers | 1 |
| Animals : an open access journal from MDPI | 1 |
| Nucleic acids research | 2 |
| Molecular cell | 2 |
| Scientific reports | 3 |
| Proceedings of the National Academy of Sciences of the United States of America | 3 |
| Molecular and cellular biology | 4 |
| Cell | 4 |
| The Journal of biological chemistry | 10 |
| Total | 51 |

Figure: (a) Distribution of literature IF (Impact Factor)
Refer to caption: x2.png

Figure: (a) Distribution of literature IF (Impact Factor)
Refer to caption: x4.png

For evaluation, we carefully collected 698, 1385, and 225 instances for KGQA, SCV, and KGCheck respectively. These datasets are split into development (dev) and test sets at an approximate ratio of 1:10. The dev data is intended for users to debug and fine-tune their evaluation code, while the test data is reserved for the final assessment. Each instance includes both input and output (ground truth answer or label) pairs, with additional information to make the data easier to understand. The dataset for SCV is reconstructed from well-known existing datasets SciFact and PubMedQA, while the rest is self-contained. The dataset represents a carefully selected sample of instances from a larger set, ensuring a comprehensive and representative coverage of the key aspects.

#### B.3 Collection Process

Biomedical experts and AI researchers listed in the author list were invloved in the data collection process. The collection process for different tasks varies:

KGQA. The collection process can be summarized into two steps: manually constructing question templates and automatically generating questions:

- •
Workflow for the Handcrafted Question Templates: The process commenced by selecting specific biomedical research fields and identifying relevant entity types and relationships from our knowledge graph. We defined various types of natural language questions, including one-hop questions, multi-hop questions, and conjunction questions (involving multiple entities). For each question, we created corresponding queries in both human-readable and machine-readable formats. These questions and queries, along with their associated metadata, such as question type and query structure, underwent rigorous peer reviews to ensure syntactic and semantic correctness.
- •
Workflow for the Auto-generated Questions and Answers: In the expansion of our benchmark, we initiated the process with the creation of auto-generated question templates. For instance, we used handcrafted question templates like "Which pathway are the proteins <Protein1> and <Protein2> both annotated in?" and then scoured our knowledge graph for data that fit the criteria to formulate both questions and answers, thereby augmenting the size of our dataset. This comprehensive dataset enables the development of assessing the robustness and accuracy of various LLM agents, providing a comprehensive benchmark that contributes to the advancement of the field with extensive biomedical knowledge.

SCV. We combine two high-quality biomedical datasets, PubMedQA and SciFact, into a single dataset for SCV. This results in a corpus consisting of abstracts from 5,664 We combine two high-quality biomedical datasets, PubMedQA and SciFact, into a single dataset for SCV. This results in a corpus consisting of abstracts from 5,664 scholarly articles and a dataset containing 1,385 biomedical claims.

KGCheck. Considering the characteristics of the knowledge graph, we decompose the approach to checking the knowledge graph into two atomic-level checks: nodes and triples. Further, we subdivide these into whether a node should exist in the knowledge graph, whether the information stored in the node is correct, whether the relationship between two connected nodes in the knowledge graph truly exists, and whether there is a potential relationship between two nodes that are not connected by an edge. To collect this data, we selected well-maintained external knowledge sources such as the UniProt database, the IntAct database, and literature. We cross-verified the information in our knowledge graph with these reliable sources, labeling mutually corroborative data as ‘support’ and data that contradicts the external reliable sources as ‘refute’. Specifically, for the data collection to check nodes, we review some update information from databases, such as entries removed due to errors or entries with updated information. Based on this information, we used Cypher queries to check our knowledge graph and label the data accordingly. For checking triple relationships, we sampled some triples from our knowledge graph where two nodes were either related or unrelated. We then queried the CKG to obtain literature that documents the entities represented by both nodes. We collected the literature annotated in the database and had experts read these documents to label the relationships of the triples in the CKG. As a result, we obtained 225 high-quality annotated data.

#### B.4 Distribution

The dataset is open to the public and has been released via GitHub under the MIT license. More information can be found in Section A.

#### B.5 Maintenance

The dataset will be maintained by Xinna Lin and Siqi Ma in Westlake University. If needed, the owner/curator/manager(s) of the dataset can be contacted through following emails: Xinna Lin ([linxinna@westlake.edu.cn](mailto:linxinna@westlake.edu.cn)), and Siqi Ma ([masiqi@westlake.edu.cn](mailto:masiqi@westlake.edu.cn)). The datasets will be revised as needed to maintain accuracy, and notifications will be provided accordingly. Currently, there is no erratum, but if any errors are discovered in the future, we will create a dedicated issue on GitHub for corrections. Older versions of the dataset will not be maintained.

#### B.6 Uses

The dataset has not been used for any tasks yet. Currently, we have not identified any tasks that are not permitted to use our dataset.

The way we collect question and answer pairs can be referenced to expand more KGQA data, whether on our knowledge graph or new knowledge graphs. Additionally, our approach to collecting data for KGCheck provides insights into identifying errors in these large knowledge graphs, which is very helpful for subsequent error correction and data updates.

### Appendix C Breakdown Results

#### C.1 KGQA

We conducted a more detailed evaluation of LLMs’ performance on the KGQA task based on the question types: one-hop, multi-hop, and conjunction. The evaluation metrics used were F1 and executability, as shown in Table 12. We find that although API-based commercial LLMs and large-scale open-source models generally perform well on overall metrics, when breaking down the KGQA task by question type, some medium-scale models perform better on certain metrics. For instance, Qwen1.5-14B-Chat exhibits higher executability on more challenging multi-hop and conjunction types of questions, although its F1 score is not high. In terms of the executability metric, open-source models seem to outperform API-based commercial LLMs. This may be because API-based LLMs are more cautious in determining whether an answer has been obtained, tending to conclude the interaction and return an answer only after confirming its correctness.

**Table 12: KGQA Test set (standard) results by question type: one-hop, multi-hop, and conjunction. Bold/underline and red/blue indicate the best and second in the subgroup and overall.**
| LLM<br>Type | Models | F1 | executability |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| one-hop | multi-hop | conjunction | one-hop | multi-hop | conjunction |  |  |
| API | GPT-4 | 87.2 | 73.7 | 77.4 | 88.0 | 90.0 | 86.9 |
| GLM-4 | 76.0 | 73.0 | 58.0 | 82.9 | 90.0 | 68.2 |  |
| OSS<br>(Large) | Qwen1.5-72B-Chat | 76.3 | 73.4 | 71.4 | 99.7 | 94.0 | 86.9 |
| Llama-3-70B-Instruct | 83.6 | 72.5 | 85.1 | 95.7 | 98.5 | 99.1 |  |
| DeepSeek-LLM-67B-Chat | 80.6 | 61.8 | 44.1 | 88.5 | 90.5 | 70.1 |  |
| OSS<br>(Medium) | Qwen1.5-32B-Chat | 67.3 | 63.2 | 57.0 | 87.2 | 84.5 | 64.5 |
| Qwen1.5-14B-Chat | 63.7 | 70.5 | 65.7 | 67.5 | 95.5 | 87.9 |  |
| Baichuan2-13B-Chat | 64.9 | 20.4 | 9.8 | 81.8 | 91.5 | 66.4 |  |
| OSS<br>(Small) | Llama-3-8B-Instruct | 59.2 | 66.4 | 16.5 | 90.8 | 66.4 | 68.2 |
| Qwen1.5-7B-chat | 55.7 | 32.1 | 26.4 | 80.3 | 79.0 | 67.3 |  |
| OSS<br>(MoE) | Mixtral-8x7B-Instruct-v0.1 | 80.3 | 68.4 | 35.9 | 90.5 | 91.5 | 50.5 |
| Starling-LM-alpha-8x7B-MoE-GPTQ | 6.2 | 25.0 | 11.7 | 12.0 | 57.5 | 48.6 |  |
| Qwen1.5-MoE-A2.7B-Chat | 38.2 | 20.2 | 9.7 | 94.4 | 45.0 | 40.2 |  |

#### C.2 SCV

As shown in Figure 8, this is the process we followed when performing the SCV task using RAG. In the main text, we observed an interesting phenomenon where expanding the RAG scope improved accuracy. To ensure that this result was not due to the idiosyncratic performance differences of a single model, we conducted the same experiment on another model, as shown in Table 13. It can be observed that the accuracy of both models in the SCV task increased with the expansion of the RAG scope, although the right quotes metric was the lowest across the three settings when performing RAG at the maximum scope. This experimental result further demonstrates that this interesting phenomenon is not due to model-specific characteristics.

Figure: Figure 8: The pipeline of RAG.
Refer to caption: x6.png

**Table 13: Supplementary Experiments on the Scope of RAG, where ‘all’ refers to the abstract of 5,664 articles, ‘partial’ denotes the 1,888 abstracts containing ground truth evidence of claims, and ‘match’ corresponds to the abstracts of the ground truth evidence for the claims. Bold/underline indicate the best and suboptimal.**
| Corpus | Qwen1.5-72B-Chat | Llama-3-70B-Instruct |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| accuracy | right quotes | error | accuracy | right quotes | error |  |
| all | 86.2 | 82.1 | 0.1 | 86.0 | 86.9 | 0.2 |
| partial | 84.4 | 88.1 | 0.1 | 82.2 | 90.4 | 0.1 |
| match | 84.3 | 88.2 | 0.1 | 82.1 | 89.9 | 0.1 |

#### C.3 KGCheck

We exhibit our BKGagent performance on KGCheck tasks based on the data source for verification(i.e. web database KGCheck and publication database KGCheck) in the main body for clarity. However, there is a more detailed category of the task considering the tools used at different stages (see Table  16). The performance based on this category is shown in Table  14.

**Table 14: Task performance categorized on agent tool calling**
|  | KG Query Task | Validation Task | Final Result |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Tool selection | Executability | Tool selection | Executability | Exact Match | Executability | Sample Size |
|  | task type 1 | 32 |  |  |  |  |  |
| GPT-4 | 78.1 | 78.1 | 75.0 | 75.0 | 71.9 | 96.9 |  |
| Llama-3-70B-Instruct | 100.0 | 100.0 | 93.8 | 93.8 | 62.5 | 81.3 |  |
|  | task type 2 | 60 |  |  |  |  |  |
| GPT-4 | 70.0 | 70.0 | 70.0 | 71.7 | 65.0 | 93.3 |  |
| Llama-3-70B-Instruct | 100.0 | 100.0 | 100.0 | 100.0 | 36.7 | 100.0 |  |
|  | task type 3 | 55 |  |  |  |  |  |
| GPT-4 | 32.7 | 32.7 | 98.2 | 98.2 | 60.0 | 98.2 |  |
| Llama-3-70B-Instruct | 90.9 | 90.9 | 92.7 | 92.7 | 36.4 | 100.0 |  |
|  | task type 4 | 45 |  |  |  |  |  |
| GPT-4 | 97.8 | 97.8 | 100.0 | 100.0 | 97.8 | 100.0 |  |
| Llama-3-70B-Instruct | 100.0 | 100.0 | 100.0 | 100.0 | 42.2 | 100.0 |  |
|  | task type 5 | 33 |  |  |  |  |  |
| GPT-4 | 57.6 | 57.6 | 63.6 | 63.6 | 51.5 | 93.9 |  |
| Llama-3-70B-Instruct | 100.0 | 100 | 97.0 | 97.0 | 21.2 | 45.5 |  |
| type 1 description: find the interaction (CURATED) between two specified proteins and verify it using RAG |  |  |  |  |  |  |  |
| type 2 description: find the interaction between two specified proteins and verify it using STRING API |  |  |  |  |  |  |  |
| type 3 description: find the specified attribute of the specified protein and verify it using UniProt API |  |  |  |  |  |  |  |
| type 4 description: check whether a specified protein exists in KG and validate it using UniProt API |  |  |  |  |  |  |  |
| type 5 description: find the relationship between two specified entities (not two proteins) and verify it using RAG |  |  |  |  |  |  |  |

The GPT-based agent shows better performance compared to the Llama-based one when being evaluated on a more detailed task category, which is consistent with our conclusion in the main body. Besides this, there are more details we can delve into:
Possible unfairness in evaluation. It should be pointed out that while the Llama-based agent successfully executed most of the tasks, it reached a comparably low score in final result excitability in the tasks involving RAG(i.e. task type 1 and task type 5). It is induced by an 8000 token limit of the model which means it is unable to process long texts, leading to an underlying unfairness in evaluation.
One-shot prompt may negatively influence GPT-based agent. GPT-based agent shows even better performance with zero-shot compared to the current one-shot strategy in our preliminary experiments. However, since OSS models perform poorly with a zero-shot strategy, we have to make a compromise and several versions of the prompt have been tested to reach a satisfied state but it is hard to thoroughly eliminate the negative influence on the GPT-based agent.
The support/refute result given by the agent is NOT reliable. As shown in Table  15, our instructions only ask the agent to provide a support/refute result as the final answer, which is intended to standardize the evaluation. However, when we read the chat history of the agent solving one specific task randomly selected from the all the records, we find out that right support/refute conclusion can be drawn from wrong analysis process, indicating that the result is not quite reliable. A more comprehensive evaluation system should be explored in future work, say evidence comparison, chatter detection, and hallucination detection.
The GPT-based agent tends to explain. Though we urge the agent to respond with support or refute(see Table  15), the GPT-based agent tends to provide explanations besides the support/refute conclusion which makes it easier for the user to judge whether the answer is derived from a reasonable process. The Llama-based agent, in contrast, strictly obeys the requirement, providing only support/refute answers.

**Table 15: Examples of instructions categorized on agent tool calling**
| Task type | instruction example |
| --- | --- |
| 1 | Please check the relationship in the knowledge graph from the node of type ’Protein’ with id ’Q14790’ to the node of type ’Protein’ with id ’Q13158’. If a relationship exists, verify its existence. Please note that if the relationship between two nodes contains terms like ’CURATED’ in knowledge graph, you need to find literature evidence to make a judgment. If no relationship exists, confirm that it indeed does not exist. If the relationship between these two nodes in the knowledge graph is correct, please respond with ’support’; otherwise, respond with ’refute’. |
| 2 | Please check the relationship in the knowledge graph from the node of type ’Protein’ with id ’P84085’ to the node of type ’Protein’ with id ’P11279’. If a relationship exists, verify its existence. Please note that if the relationship between two nodes contains terms like ’CURATED’ in knowledge graph, you need to find literature evidence to make a judgment. If no relationship exists, confirm that it indeed does not exist. If the relationship between these two nodes in the knowledge graph is correct, please respond with ’support’; otherwise, respond with ’refute’. |
| 3 | Please check if the ’name’ attribute of the node with type Protein and id Q4G0T1 in the knowledge graph is correct. If it’s correct, please respond with ’support’; if not, respond with ’refute’. |
| 4 | Due to certain reasons, some entries were removed from the database. Please check whether the node with the type ’Protein’ and the id ’A2RUG3’ exists in the knowledge graph, and confirm whether it exists in the corresponding database. If its existence is consistent, please respond with ’support’; otherwise, answer ’refute’. |
| 5 | Please check the relationship in the knowledge graph from the node of type ’Protein’ with id ’Q96QP1’ to the node of type ’Tissue’ with id ’BTO:0000007’. If a relationship exists, verify its existence. Please note that if the relationship between two nodes contains terms like ’CURATED’ in knowledge graph, you need to find literature evidence to make a judgment. If no relationship exists, confirm that it indeed does not exist. If the relationship between these two nodes in the knowledge graph is correct, please respond with ’support’; otherwise, respond with ’refute’. |
|  |  |

### Appendix D Experimental Details

#### D.1 Construction of Knowledge Graph

We pulled a Neo4j image from Docker Hub and created a Neo4j Docker on the server to host a knowledge graph. We used the latest data parsed from various databases in April 2024, including UniProt [], TISSUES [], DISEASES [], HGNC [], IntAct [], STRING [], DisGeNet [], Pathway Commons [], Reactome [], SMPDB [], and SIGNOR [], Disease Ontology [], Brenda Tissue Ontology [], Gene Ontology [], Protein Modification Ontology [], Molecular Interactions Ontology []. These databases are under loose license and the data can be used directly. We parsed the information from these databases into TSV files in a specific format and then imported the contents of these TSV files into Neo4j using Cypher statements (Cypher is the declarative graph query language provided by Neo4j) to construct the knowledge graph. This knowledge graph can be queried using Cypher statements.

#### D.2 Deployment of Open-Source LLMs

We deployed open-source LLMs using the vLLM framework and inference is performed using a server with 4 NVIDIA A40 GPUs an Intel(R) Xeon(R) Gold 6330 CPU, with parameters kept constant at startup.

#### D.3 Experimental Setup

Atomic Abilities. To evaluate two atomic abilities, we adopt an interactive evaluation of LLM-as-Agent [] and include in total of 13 models for evaluation. These models can be categorized into API-based Commercial LLMs and Open-Sourced (OSS) LLMs. The latter is further segmented based on model size into three classifications: ‘Large’, ‘Medium’, and ‘Small’. Models utilizing the MoE (Mixture of Experts) framework are distinguished as a separate category. Refer to Appendix D.4 for details about the prompt we designed for the following tasks.

Agent Task. For the construction of our BKGAgent, we selectively employed the best-performing models in atomic capabilities from both API-based and OSS models, specifically GPT-4 and Llama-3-70B-Instruct. To avoid being trapped in an endless loop where agents repeat the same talk or start to chatter, we limit the memory entries of one single agent to 20, which is more than enough to finish any of the tasks. It should be noted that each agent only keeps memory of the conversations related to it, while all chats returned by every agent are stored in the graph state. Since zero-shot setup in various types of tasks shows inferior performances in our preliminary experiments, we provide one-shot prompt for each type of task. We analyze both the process and final result of each task execution, considering the correctness of tool selection and agent executability during the process evaluation, and assessing the exact match of the right answer and framework executability for the final result evaluation, to gain a comprehensive understanding of the agent’s performance.

We detail our implementation of two sub-tasks here:

- •
KGQA: We developed a suite of atomic tools for querying KGs for LLMs. Every LLM is prompted in the same way with a detailed task description, information about provided tools and a one-shot demonstration, which employs the “Thought,” “Action,” “Observation” cognitive trajectory from the ReAct [], with the “Thought” component assisted by Chain of Thought (CoT) [] reasoning. We constrain the LLM to a maximum of fifteen interactive turns, within which it may only take one action per turn. If the LLM can respond within these fifteen turns, executability is assigned a score of 1. Subsequently, we compare the response to the ground truth to calculate the F1 score and the Exact Match score (EM).
- •
SCV: We first convert the entire corpus into semantic vectors using jina[] and store them in a vector database. Claims are similarly transformed into semantic vectors via Jina, with the top 50 scoring vectors being submitted to the LLM with a standardized prompt template. We require the LLM to return results in JSON format, considering any deviation as an error. The outcomes mainly include answers and quotes. For analysis, we adopt a flexible interpretation of answers: “Unsure” and “Unrelated” as “Unsure”; “Supported” and “Supports” as “Supports”; “Unsupported” and “Unsupports” as “Unsupports”, “Refuted”, and “Refutes” as “Refutes”. Any other results are also considered errors. The experiments for each model are repeated three times, with the final performance averaged to ensure the robustness of the evaluation. Notably, beyond the conventional accuracy and the aforementioned error metrics, we introduce a “right quotes” metric, which assesses whether the retrieved quotes match the ground truth evidences of the claim.

#### D.4 Prompt

##### D.4.1 KGQA

We provide a unified prompt for single-agent systems built on different LLMs, ensuring the fairness of the evaluation.

Considering the complexity of the task and the high information density of our KG, LLMs find it challenging to address high-difficulty questions, such as multi-hop and conjunction types, within the constraints of a zero-shot setting and a 15-turn interaction limit. Therefore, we provide a one-shot demonstration.

##### D.4.2 SCV

We provide a unified prompt describing task, where ‘context_docs_str’ represents quotes retrieved by RAG and ‘user_claim’ represents the input scientific claim to be evaluated.

##### D.4.3 KGCheck

BKGAgent is a multi-agent system and each agent of it is equipped with a system prompt which includes role introduction, tool introduction, and tool calling rules.

For team leader:

For KG agent:

For validation agent:

We offer a one-shot prompt for each type of task, categorized by the necessary tools needed to complete it as shown in Table 16. Detailed prompts for every agent in every task are listed below.

**Table 16: Task types categorized by requiring tools**
| Task type | Requiring tools | Description |  |
| --- | --- | --- | --- |
|  | KG agent | Validation agent |  |
| 1 | query relation between nodes | publication RAG | find the interaction (CURATED) between two specified proteins and verify it using RAG |
| 2 | query relation between nodes | check interaction on STRING | find the interaction between two specified proteins and verify it using STRING API |
| 3 | query node attribute | get UniProt protein information | find the specified attribute of the specified protein and verify it using UniProt API |
| 4 | query node existence | get UniProt protein information | check whether a specified protein exists in KG and validate it using UniProt API |
| 5 | query relation between nodes | publication RAG | find the relationship between two specified entities (not two proteins) and verify it using RAG |
|  |  |  |  |

Task type 1

Team leader:

KG agent:

Validation agent:

Task type 2

Team leader:

KG agent:

Validation agent:

Task type 3

Team leader:

KG agent:

Validation agent:

Task type 4

Team leader:

KG agent:

Validation agent:

Task type 5

Team leader:

KG agent:

Validation agent:

### Appendix E Case Study

#### E.1 KGQA

We sampled 6 cases for demonstration, with one correct case and one incorrect case for each question type: one-hop, multi-hop, and conjunction, as shown in Figures 9, 10, 11, 12, 13 and 14.

#### E.2 SCV

We sampled 8 examples for demonstration, including 4 correct answers and 4 incorrect answers. Each case has certain differences and is representative, as shown in Figures 15, 16, 17, 18, 19, 20, 21 and 22.

#### E.3 KGCheck

We select several classic success and failure cases for each type of task as presented in  Figures 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36 and 37 as a supplementary for some common error cases in our main body. There are many interesting cases when the team leader properly corrects the behavior of assistant agents, getting the workflow back on track, and we choose one such case of task type 1 as a representation. As mentioned before, there are also cases where the right final answer is derived from a wrong analysis process. We select this kind of case for every type of the task except type 4 (this case does not exist in this type of task).

As introduced in the main body, our BKGAgent framework is comprised of three agents: the team leader, KG agent, and validation agent. The typical workflow from the agent role perspective of our framework can be simplified as team leader - KG agent - team leader - validation agent- team leader. We present the chat of three agents in table format, omitting the interactions of the assistant agent and tool executor. The columns respectively stand for the agent role, the action they take, the chat content, and the human annotation of this chat round. The green check mark means the process is consistent with our anticipation, while the yellow exclamation mark means the chat content may lead to an unwanted result, and the red cross stands for a wrong action or error chat content. Comments are attached to a negative review for explanation. Error or dangerous contents are underlined and colored red, while contents related to tool usage or evidence consistent with the golden answer are in bold green font. The blue row stands for an expected chat round, in contrast, the yellow row indicates something is wrong in this chat round. We send tool results and behavior correction prompts in the role of a human; this kind of chat is colored grey in our table.

Figure: Figure 9: Performance of Mixtral-8x7B-Instruct-v0.1-based Agent: Incorrect final result when answering the one-hop type questions due to passing incorrect parameters when executing a function. Session highlighted in pink represents the error information.
Refer to caption: x7.png

Figure: Figure 10: Performance of GPT-4-based Agent: Correct final result when answering the one-hope type questions. Session highlighted in light green represents the correct information.
Refer to caption: x8.png

Figure: Figure 11: Performance of Mixtral-8x7B-Instruct-v0.1-based Agent: Incorrect final result when answering the multi-hop type questions due to executing the wrong actions. Session highlighted in pink represents the error information.
Refer to caption: x9.png

Figure: Figure 12: Performance of GPT-4-based Agent: Correct final result when answering multi-hop type questions. Session highlighted in light green represents the correct information.
Refer to caption: x10.png

Figure: Figure 13: Performance of Mixtral-8x7B-Instruct-v0.1-based Agent: Incorrect final result when answering the input conjunction type question in 15-turn limit due to executing the wrong action. Session highlighted in pink represents the error information.
Refer to caption: x11.png

Figure: Figure 14: Performance of GPT-4-based Agent: Correct final result when answering the conjunction-type questions. Session highlighted in light green represents the correct information.
Refer to caption: x12.png

Figure: Figure 15: Performance of Llama-3-70B-Instruct-based Agent: Correct final result with the correct quotes. Session highlighted in light green represents the correct information.
Refer to caption: x13.png

Figure: Figure 16: Performance of Llama-3-70B-Instruct-based Agent: Correct final result but with an incorrect quote. Session highlighted in pink represents the error information.
Refer to caption: x14.png

Figure: Figure 17: Performance of GPT-4-based Agent: Correct final result but without any quotes. Session highlighted in pink represents the error information.
Refer to caption: x15.png

Figure: Figure 18: Performance of Llama-3-70B-Instruct-based Agent: Correct final result with the correct quotes. Session highlighted in light green represents the correct information.
Refer to caption: x16.png

Figure: Figure 19: Performance of Llama-3-70B-Instruct-based Agent: Incorrect final result but with the correct quote. Session highlighted in light green represents the correct information.
Refer to caption: x17.png

Figure: Figure 20: Performance of Qwen1.5-72B-Chat-based Agent: Incorrect final result but with the correct quote.
Refer to caption: x18.png

Figure: Figure 21: Performance of Qwen1.5-72B-Chat-based Agent: Incorrect final result without any quotes. Session highlighted in pink represents the error information.
Refer to caption: x19.png

Figure: Figure 22: Performance of Baichuan2-13B-Chat-based Agent: Parsing error occurring in the final result due to failing to respond in JSON format. Session highlighted in pink represents the error information.
Refer to caption: x20.png

Figure: Figure 23: A sample success case of task type 1 where validation agent performs the task as one-shot prompt instructed without getting affected by the leader’s improper instruction.
Refer to caption: x21.png

Figure: Figure 24: A sample case of task type 1 with right final answer but the wrong analyse process. The validation is deeply influenced by the leader’s command and thus wastes tokens on unnecessary tool calling. While the validation agent is expected to find the evidence in the documents that supports the claim, it draws the right conclusion based on the result by visiting STRING.
Refer to caption: x22.png

Figure: Figure 25: A sample failure case of task type 1. The validation agent fails to understand the meaning of "CURATED INTERACTS WITH relationship" but tries to find an exact match, leading to the wrong conclusion. The team leader fails to extract the useful information from the validation agent’s reply to make a right final conclusion.
Refer to caption: x23.png

Figure: Figure 26: A sample case of the team leader properly correcting the assistant agent’s behavior. The kG agent passes wrong arguments to the tool the first time it performs its subtask but is corrected by the team leader, indicating that the supervisor role of the team leader helps regulate the workflow.
Refer to caption: x24.png

Figure: Figure 27: A sample success case of task type 2 where validation agent performs the task as one-shot prompt instructed without getting affected by the leader’s improper instruction.
Refer to caption: x25.png

Figure: Figure 28: A sample case of task type 2 with the right final answer but the wrong analyse process. Assistant agents are deeply affected by one-shot prompts resulting in hallucinations in the task execution. The team leader misunderstands the instruction for the response but coincidently gives the right answer.
Refer to caption: x26.png

Figure: Figure 29: A sample failure case of task type 2. The whole process including assistant agents’ tool calling and feedback processing is as expected, but the team leader misunderstands the instruction for the response causing a wrong final result.
Refer to caption: x27.png

Figure: Figure 30: A sample success case of task type 3 where validation agent performs the task as one-shot prompt instructed without getting affected by the leader’s improper instruction.
Refer to caption: x28.png

Figure: Figure 31: A sample case of task type 3 with right final answer but the wrong analyse process. The team leader tries to replace the KG agent with itself and generates false KG information which happens to be consistent with the actual name missing condition of the specified protein in KG (whether the name is wrong or missing the answer will be ’refute’).
Refer to caption: x29.png

Figure: Figure 32: A sample failure case of task type 3. The team leader fails to send the command to KG agent due to the message format, triggering an autonomous human prompt to help the leader correct its behavior. However, the team leader tries to replace the KG agent and generate false KG information instead of regenerating the command sent to KG agent, resulting in the wrong final answer.
Refer to caption: x30.png

Figure: Figure 33: A sample success case of task type 4. Both assistant agents and team leader perform their tasks as expected. Core chats are presented.
Refer to caption: x31.png

Figure: Figure 34: A sample failure case of task type 4. The team leader misunderstands the instruction for the response and makes a wrong conclusion though the analyse process is right.
Refer to caption: x32.png

Figure: Figure 35: A sample success case of task type 5. Both assistant agents and team leader perform their tasks as expected. Core chats are presented.
Refer to caption: x33.png

Figure: Figure 36: A sample case of task type 5 with right final answer but the wrong analyse process. The validation agent makes a wrong conclusion and the team leader misunderstands the instruction for response, surprisingly leading to the right answer.
Refer to caption: x34.png

Figure: Figure 37: A sample failure case of task type 5. The team leader fails to send the command to KG agent due to the message format, triggering an autonomous human prompt to help the leader correct its behavior. However, the team leader tries to replace the KG agent and generate false KG information instead of regenerating the command sent to KG agent. The validation agent makes a wrong conclusion worsening the situation.
Refer to caption: x35.png

### Appendix F Other Related Work

Recent research has increasingly focused on the application of LLMs in various scientific domains. These models, initially developed for general purposes, are now being utilized to tackle domain-specific scientific tasks. This involves integrating essential domain-specific context and knowledge into the LLMs, either during training or prior to task inference. A critical challenge in this process is balancing the inclusion of relevant domain knowledge with the model’s reasoning capabilities, especially when domain-specific data is limited.

Various approaches have been explored to utilize LLMs for specific scientific applications, depending on the availability of data and model accessibility [, , , , ]. Common strategies in the scientific domain include training domain-specific LLMs from scratch, fine-tuning general-purpose LLMs, and employing few-shot or zero-shot learning with prompting. Training domain-specific LLMs from scratch offers the highest flexibility and customization, as demonstrated by models like Galactica [], which constructs large scientific corpora and trains LLMs in a self-supervised manner [, ]. Fine-tuning pre-trained LLMs with domain-specific datasets has yielded promising results, as seen in BioMedLM [] and med-PALM [, ]. Fine-tuning can also be performed with smaller amounts of paired data in a supervised fashion, exemplified by DrugChat []. Few-shot or zero-shot learning, also known as in-context learning, is effective for using advanced instruction-tuned LLMs like GPT-4 [] for scientific tasks by incorporating domain knowledge into prompts. This approach has shown success in fields such as Social Science [] and astronomy [], as well as in benchmarking LLMs on chemistry tasks []. Recent studies like CancerGPT [] and SynerGPT [] investigate LLMs for drug synergy prediction and other complex scientific interactions. Furthermore, augmenting LLMs with external tools, such as using Web APIs for genomics questions [], and integrating domain-specific tools into language model prompts to access specialized knowledge [, , ], are promising directions. Efforts are also underway to develop LLM-based agents for scientific discovery by connecting LLMs with experimental tools in fields like Chemistry [] and Machine Learning []. LeanDojo [, ], for example, is an open-source toolkit for theorem proving that integrates retrieval-augmented LLMs to enhance theorem proving capabilities. Despite these advancements, the diverse data modalities across different scientific domains pose significant challenges for the direct application of LLMs in many areas.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the questions in the "Weaknesses" section
soundness: 1
strengths: * The constructed test cases for KG QA and claim verification might be helpful in evaluating clinical-related models and agents
* The designed evaluation mechanism would be used as a good reference for future agent evaluation design
summary: This paper proposes several tasks related to the scientific literature review and evaluates the performance of LLMs and LLM-based agents. The authors also propose an agent to address the task. These tasks include KG QA, scientific claim verification and the combined KG checking tasks. Lots of mainstream LLMs are tested. Several agent designs are also evaluated with access to the tools defined by the authors.
weaknesses: - Novelty as a benchmark work: 
	- When evaluating the LLMs only (Table 7), the LLMs do not have access to the KG or literature database, making the task a straightforward task. Similar tasks/datasets have been proposed a lot
	- When evaluating the agents (Table 8), the assumption is that the tool design is not part of the agent design (as the same set of tools are provided by the authors instead of designed by the designer of the baseline agents). This assumption makes this benchmark not generalizable and limited by the design of tools used to access KG and literature
	- The agent benchmark task is not designed to be result-oriented 
- Comparison with works of the KG-guided QA setting, such as Think-on-graph [1]
	- Both lines of work assume that external knowledge sources can be used to answer questions. What is the uniqueness of this work besides applying KG-enhanced QA to the biomedical domain?
- Soundness of the evaluation metrics
	- Why the potential relationship is refute if the relationship is not on the KG (which could totally due to the sparsity of the KG)? What is the difference between existing relationship and potential relationship?
- Missing details
	- For the claim verification task, when predicting label for a claim, does the model have access to the while corpus with 5664 articles, or it only have access to the particular article that the claim is induced from?
	- If this task is about global claims (instead of context-specific claims per article), how to handle the potential conflicts between claims?
	- What is the additional information the Qwen model used for evaluation has access to (compared with the evaluated LLMs that produce the answers)? 
	- Does the framework support unseen nodes on the KG, or does it assume all entities used for evaluation have to be on the KG already?
	- For the baseline agent without access to the "our tools", how do they access the KG and literature pool?

[1] https://arxiv.org/abs/2307.07697

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. What's the difference between KGQA and STaRK [1]?

2. Clinical Knowledge Graph used in the paper has difference references including [32], [33], and [53] in the paper.

[1] Wu, Shirley, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Zou, and Jure Leskovec. "STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases." arXiv preprint arXiv:2404.13207 (2024).
soundness: 2
strengths: 1. The paper proposes three tasks—Knowledge Graph Question Answering (KGQA), Scientific Claim Verification (SCV), and Knowledge Graph Checking (KGCheck).

2. The study successfully identifies 96 inaccuracies within the knowledge graph, marked as "Refute" annotations. This finding is particularly significant because it challenges the common assumption that knowledge graphs contain precise information.
summary: This paper introduces BioKGBench, a benchmark for evaluating AI agents in biomedical science, designed to tackle the challenges of understanding and verifying scientific knowledge. Unlike traditional benchmarks that focus on narrow tasks like question-answering, BioKGBench emphasizes two critical abilities: Knowledge Graph Question Answering (KGQA) and Scientific Claim Verification (SCV). These tasks help assess agents' abilities to analyze both structured (knowledge graphs) and unstructured (scientific literature) data. 
The authors also introduce BKGAgent, a baseline model that effectively identifies inaccuracies in biomedical knowledge graphs, offering a useful tool for keeping scientific databases accurate and up-to-date.
weaknesses: 1. The proposed method utilizes the Retrieval-Augmented Generation (RAG) approach; however, it lacks comparison with recent RAG-based baselines such as Self-RAG and KG-RAG. This absence makes it challenging to evaluate the effectiveness of the proposed method against state-of-the-art techniques, potentially limiting the credibility of the results.

2. The KGCheck task is primarily focused on verifying knowledge within knowledge graphs. While this is valuable, its narrow application may restrict the overall utility of the task in broader contexts.

3. The approach may face scalability issues when applied to larger and more complex knowledge graphs. 

4. The study relies solely on one KG proposed in 2020, which may not reflect the most current state of knowledge representation. Given that the KG used is relatively old, there is a risk that the findings regarding inaccuracies and errors may not accurately reflect the current capabilities and improvements in more recent KGs. As KG evolve, their accuracy and completeness can change, potentially rendering the results of this study less relevant over time.

5. The results of the study may not be easily generalized to other KGs beyond the one used. Different KGs can vary significantly in structure, content, and accuracy, making it uncertain whether the identified issues and the effectiveness of the proposed tasks will hold true across various knowledge representations.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1.	What is the proportion of different reasoning types (one-hop, multi-hop, conjunction) involved in the KGCheck task?
2.	What is the recall rate of errors detected by BKGAgent on the KGCheck task?
soundness: 2
strengths: 1.	The sub-task design of KGCheck aligns well with the methodology of human scientific research, which consists of database queries and literature review. This benchmark may be useful for future study.
2.	The proposed BKGAgent incorporates various tools and interactions between agents, and achieves better performance than baselines.
summary: This paper introduces BioKGBench, which explores the LLMs’ capability to do “KGCheck” - checking existing knowledge graph and cross-referencing with external literature or databases. They divide this task into two atomic sub-tasks: (1) knowledge graph question answering (KGQA), where the agent needs to answer questions of a knowledge graph; and (2) scientific claim verification (SCV), where the agent verifies a claim by querying the evidence from external datasets or literature. 

To solve the tasks, this paper proposes BKGAent, which is a multi-agent framework containing a team leader, KG agent and validation agent equipped with a set of tools. The proposed framework achieves better performance than existing baselines.
weaknesses: 1.	From Figure 5 and Table 8, the performance of “BKGAgent” is similar to that of “AutoGen w/ our tools”. It seems that the key performance gain is achieved by the tools. Thus, the effectiveness of using three agents in this framework is not clear.
2.	The first sub-task – KGQA is closely related to the body of work on KBQA. However, the relation with KBQA is not discussed in this paper. Also, is it possible to benchmark KBQA methods in the KGQA subtask and use them instead of KG agents in the KGCheck task?
3.	The performance of proposed framework on the sub-tasks are not reported and compared with baselines (Table 7).
4.	The metrics (understanding, reasoning, efficiency, KG process, information retrieval) are not explained and it is not clear what these scores imply.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: a) How the labels are made for SCV task? Any human agreement tests on the labeling?

b) Do the authors have a vision on the difference between proposed agent method and the existing agent methods? How the difference can impact on the performance on BioKGBench?

c) Do the authors have a vision on how the generalizability of proposed pipeline to other domains?

d) In line 363-364, the authors may want to avoid using the word "propose" since the metrics used are common.

e) How the classification is made for SCV task? Is it a self-critique / prompting process or a trainable process? If it's a prompt-based process, what is the scoring criteria?
soundness: 3
strengths: a) A novel benchmark called BioKGBench for evaluating biomedical ai agent is proposed.

b) Extensive evluation on existing ai agents are compared.

c) A baseline called BKGAgent is proposed.
summary: This work investigates the challenge of "ai agent for scientific literature" that focus solely on factual QA, leading to LLM hallucinations. A novel benchmark BioKGBench which integrates a Knowledge Graph Question Answering task and a Scientific Claim Verification task is proposed. Detailed construction (data source, statistics etc.) and a baseline is proposed to validate the quality of BioKGBench.
weaknesses: a) Lack cost analysis of BKGAgent.

b) The generalizability of proposed pipeline to other domains is not discussed.

c) Minor writing issues.

Meta-review: This paper introduces BioKGBench, a novel benchmark designed to evaluate the capability of LLMs and AI-driven agents in biomedical science. The study distinguishes itself by attempting to address the challenges of factual verification and literature grounding in a systematic way. 

However, as pointed out by the reviewers, there are several concerns with the methodology and generalizability of the findings: the simplicity of some tasks, which LLMs can already perform with high accuracy, questions the usefulness of such benchmarks; the use of a single, potentially outdated knowledge graph may not accurately represent current knowledge states, thus limiting the applicability of the results; and the assumption that tools provided by the authors, rather than developed by agent designers, limits the benchmark's generalizability. Even though the authors tried to address these concerns during the rebuttal, the reviewers were not fully satisfied with the responses.

Decision: Reject