BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain
Authors: Yunsoo Kim, Yusuf Abdulle, Honghan Wu
Accepted at: acl25

BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedicine

Yunsoo Kim1   Yusuf Abdulle1,2   Honghan Wu1,3
1UCL   2King’s College London   3University of Glasgow
{yunsoo.kim.23, honghan.wu}@ucl.ac.uk

Abstract
Biomedical reasoning often requires traversing interconnected relationships across entities such as drugs, diseases, and proteins. Despite the increasing prominence of large language models (LLMs), existing benchmarks lack the ability to evaluate multi-hop reasoning in the biomedical domain, particularly for queries involving one-to-many and many-to-many relationships. This gap leaves the critical challenges of biomedical multi-hop reasoning underexplored. To address this, we introduce BioHopR, a novel benchmark designed to evaluate multi-hop, multi-answer reasoning in structured biomedical knowledge graphs. Built from the comprehensive PrimeKG, BioHopR includes 1-hop and 2-hop reasoning tasks that reflect real-world biomedical complexities.

Evaluations of state-of-the-art models reveal that O3-mini, a proprietary reasoning-focused model, achieves \( \mathbf{37.93\%} \) precision on 1-hop tasks and \( \mathbf{14.57\%} \) on 2-hop tasks, outperforming proprietary models such as GPT4O and open-source biomedical models including HuatuoGPT-01-70B and Llama-3.3-70B. However, all models exhibit poor capabilities in the multi-hop reasoning, underscoring the challenges of resolving implicit reasoning steps in the biomedical domain. By addressing the lack of benchmarks for multi-hop reasoning in biomedical domain, BioHopR sets a new standard for evaluating reasoning capabilities and highlights critical gaps between proprietary and open-source models while paving the way for future advancements in biomedical LLMs. BioHopR is available at https://huggingface.co/datasets/knowlab-research/BioHopR.

1 Introduction
Recent advances in large language models (LLMs) and Question Answering (QA) systems have shifted the focus from simple factoid retrieval tasks to more sophisticated reasoning capabilities (Huang and Chang, 2022; Plaat et al., 2024; OpenAI, 2025). Among these, multi-hop reasoning has emerged as a critical area of research, where answering a question requires traversing multiple interconnected reasoning steps (Misra et al., 2023; Yang et al., 2024; Schnitzler et al., 2024). For example, to answer “Who is the wife of the president of the United States?”, a LLM must first identify the president (step 1) and then determine their spouse (step 2). This type of reasoning, referred to as multi-hop reasoning, is especially vital in domains where information is highly interconnected, such as the biomedical field.

In the biomedical domain, knowledge is often structured in ontologies and knowledge graphs (KGs), where entities like drugs, diseases, proteins, and phenotypes are represented as nodes, and their relationships as edges (Himmelstein et al., 2017; Sung et al., 2021; Chandak et al., 2023). Biomedical queries frequently demand multi-step reasoning over these graphs (Sung et al., 2021; Su et al., 2024; Matsumoto et al., 2025). For instance, identifying diseases associated with a drug might require a single-hop relation, while determining proteins targeted by that drug through its associated disease involves two reasoning steps. Furthermore, biomedical reasoning often involves one-to-many or many-to-many relationships. For example, a single question might have several correct answers - like a drug that works on multiple proteins (Liang et al., 2019). This complexity highlights the need for specialized benchmarks that rigorously evaluate models’ ability to reason across multiple steps while generating comprehensive, multi-answer responses.

Existing benchmarks for multi-hop reasoning, such as Hetionet (Himmelstein et al., 2017) and other biomedical QA datasets (Rao et al., 2022), have laid the groundwork for evaluating multi-hop capabilities in the biomedical domain. However, these benchmarks primarily focus on single-
hop tasks or utilize pre-defined templates that fail to fully capture the intricacies of multi-step reasoning. Similarly, general-domain benchmarks like TWOHOPFACT (Yang et al., 2024) test models’ latent multi-hop reasoning ability but lack the domain-specific challenges of biomedical reasoning, such as reasoning over structured relationships and handling multi-answer outputs. As a result, the unique challenges of biomedical multi-hop reasoning remain underexplored.

To address these limitations, we introduce BioHopR, a new benchmark specifically designed to test the multi-hop reasoning capabilities of LLMs in the biomedical domain. Unlike general-domain benchmarks that rely on reasoning across disconnected documents, BioHopR focuses on reasoning within a single, structured biomedical knowledge graph. Our benchmark systematically constructs 1-hop (e.g., Drug–Disease) and 2-hop (e.g., Drug–Disease–Protein) question-answer pairs from the PrimeKG knowledge graph (Chandak et al., 2023). Questions are designed to evaluate models’ abilities to reason step-by-step, explicitly requiring the inference of intermediate entities, and generate multi-answer responses reflective of real-world biomedical complexity.

Contributions. Our main contributions are as follows:
• A New Benchmark for Multi-Hop Reasoning: We propose BioHopR, the first publicly available benchmark explicitly designed to evaluate multi-hop, multi-answer reasoning within structured biomedical knowledge graphs. We will release the dataset and the code for evaluation.
• Multi-hop Knowledge Curated from a Comprehensive Up-To-Date Knowledge Graph: Leveraging the comprehensive and up-to-date PrimeKG knowledge graph, we systematically construct a dataset of 1-hop and 2-hop biomedical questions and their answers, ensuring real-world relevance.
• Evaluation and Analysis of LLMs in Biomedical Multi-hop Reasoning: We evaluate state-of-the-art LLMs on our benchmark, highlighting their strengths and, more importantly, limitations in handling biomedical multi-hop reasoning tasks.

By introducing BioHopR, we aim to fill a critical gap in multi-hop QA research and advance the development of LLMs capable of robust and interpretable reasoning in structured, high-stakes domains like biomedical research and healthcare.

2 Related Works

Biomedical Question Answering. Research in medical LLMs has been facilitated by the development of question-answering (QA) datasets that benchmark models’ understanding of medical domain knowledge (Hendrycks et al., 2020; Jin et al., 2021; Pal et al., 2022). These datasets typically consist of multiple-choice questions (MCQs) focused on single-hop reasoning tasks, providing a straightforward way to evaluate LLMs’ ability to comprehend and respond to diverse medical inquiries. While these benchmarks have driven significant progress, they primarily measure classification accuracy, which is insufficient for capturing the nuanced reasoning required for medical expertise.

Medical QA often involves interconnected concepts where reasoning over multiple steps is crucial. However, current benchmarks rarely go beyond single-hop tasks and do not evaluate models’ ability to provide explanations for their answers or justify their reasoning process. Recently, MedExQA introduced an evaluation framework with detailed explanations for assessing the reasoning capabilities of LLMs (Kim et al., 2024). While this is a step forward, it remains constrained to single-hop reasoning and does not address the need for multi-hop reasoning or the generation of multiple valid answers—a common requirement in biomedical inquiries.

Knowledge Graph Question Answering. Knowledge Graph Question Answering (KGQA) systems leverage structured knowledge graphs to answer questions that require reasoning over graph-based relationships. In the biomedical domain, Hetionet (Himmelstein et al., 2017) introduced a knowledge graph containing entities like genes, drugs, and diseases, enabling structured reasoning. Extensions of Hetionet have been used for multi-hop QA tasks (Rao et al., 2022), but these datasets often rely on fixed templates and predefined reasoning paths, limiting their ability to evaluate the nuanced multi-hop reasoning required in real-world biomedical applications. This work explored techniques such as knowledge graph embeddings and graph neural networks (Kipf and Welling, 2016; Hamilton et al., 2018), and transformer-based models like BioBERT (Lee
<table>
  <tr>
    <th>Dataset</th>
    <th>Domain</th>
    <th>Reasoning</th>
    <th>Answer-Level</th>
  </tr>
  <tr>
    <td>MedQA (Jin et al., 2021)</td>
    <td>Biomedical</td>
    <td>No</td>
    <td>Single Answer</td>
  </tr>
  <tr>
    <td>Hetionet QA (Himmelstein et al., 2017)</td>
    <td>Biomedical</td>
    <td>Graph-based Reasoning (MH)</td>
    <td>Single Answer</td>
  </tr>
  <tr>
    <td>MedExQA (Kim et al., 2024)</td>
    <td>Biomedical</td>
    <td>Explanation Generation</td>
    <td>Single Answer</td>
  </tr>
  <tr>
    <td>TWOHOPFACT (Yang et al., 2024)</td>
    <td>General</td>
    <td>Implicit Reasoning (MH)</td>
    <td>Single Answer</td>
  </tr>
  <tr>
    <td><b>BioHopR (Ours)</b></td>
    <td>Biomedical</td>
    <td>Implicit Reasoning (MH)</td>
    <td>Multi Answers</td>
  </tr>
</table>

Table 1: Comparison of BioHopR with existing datasets. Key differentiators include domain focus, reasoning type (MH is tagged for multi-hop reasoning supported dataset), and answer-level, such as multi-answer capability.

et al., 2020) to extract and utilize graph-based knowledge. However, this dataset tests the model’s performance in a classification task to a single answer. Also, this dataset is not publicly available, limiting its role in facilitating biomedical large language model research.

In domains like biomedical science, many questions inherently involve multiple correct answers. For instance, identifying all drugs that treat a specific disease or all proteins associated with a disease phenotype requires models to retrieve comprehensive sets of answers rather than a single response.

Latent Multi-Hop Reasoning in Large Language Models. Recent work has explored the latent reasoning capabilities of LLMs, focusing on whether models can implicitly infer intermediate entities and use them for multi-step reasoning. The TWOHOPFACT dataset (Yang et al., 2024) evaluates this capability by testing whether LLMs can identify "bridge entities" in two-hop reasoning tasks. While TWOHOPFACT demonstrates that LLMs can perform latent multi-hop reasoning in general domains, it does not address the unique challenges of biomedical reasoning. Biomedical queries often require explicit reasoning over structured data and demand comprehensive answers involving one-to-many or many-to-many relationships.

These gaps highlight the need for a benchmark like BioHopR, which explicitly evaluates models’ ability to perform step-by-step reasoning and generate multi-answer outputs in the biomedical domain.

Multi-Answer Reasoning. Existing QA benchmarks, both in general and biomedical domains, typically assume a one-to-one mapping between questions and answers, which oversimplifies the complexity of real-world reasoning tasks. This assumption is especially problematic in the biomedical domain, where relationships between entities are often one-to-many or many-to-many.

BioHopR BioHopR addresses this limitation by introducing questions that require multi-answer reasoning, ensuring that the benchmark captures the intricate relational structures and knowledge dependencies present in biomedical science. The differences between our dataset and relevant datasets are summarized in Table 1.

3 BioHopR: Multi-hop Reasoning in Biomedicine

![Illustration of the relationships observed in this work using a biomedical knowledge graph. Each subplot visualizes a different type of question–answer relationship, highlighting the complexity and variability inherent in biomedical data.](page_1016_1092_388_246.png)

Figure 1: Illustration of the relationships observed in this work using a biomedical knowledge graph. Each subplot visualizes a different type of question–answer relationship, highlighting the complexity and variability inherent in biomedical data.

BioHopR is a benchmark specifically designed to evaluate the ability of large language models (LLMs) in performing multi-hop reasoning and generating multi-answer outputs in the biomedical domain. Compared to other knowledge graphs
such as Hetionet, PrimeKG provides a broader coverage of biomedical entities, richer relational structures, and up-to-date knowledge in the field (Chandak et al., 2023). This allows for the generation of diverse, clinically relevant up-to-date multi-hop queries. By carefully creating questions over PrimeKG, the dataset is designed to follow a one-to-many-to-many relationship structure, where one question can have several correct answers. This restriction ensures that queries reflect real-world biomedical scenarios where entities like drugs, diseases, and proteins exhibit hierarchical and complex interconnected relationships.

3.1 Multi-hop, Multi-answer Knowledge Formalization

Nodes and Relations. In our dataset, the entities in PrimeKG are represented as nodes, and their relationships are directed edges. For any query, the node from which reasoning starts is defined as the query node, and the node(s) forming the final answers are the target nodes. In the case of 2-hop reasoning, the intermediate node connecting the query and target is defined as the bridge node. We restrict node types to the following: Drug, Proteins, Disease, Phenotype.

Relationship Structure. As seen in Figure 1, the dataset is restricted to follow a one-to-many-to-many relationship structure, where each question has multiple correct answers and each answer may relate to several different entities.

In 1-hop questions, a direct relationship connects the query node to the target nodes. For example:

Query (Drug) \( \xrightarrow{\text{treats}} \) Target (Diseases). (1)

This setup reflects a single reasoning step where a query node is linked to multiple target nodes.

In 2-hop questions, the query node connects to the target nodes via an intermediate bridge node, forming a two-step reasoning chain. For example:

Query (Phenotype) \( \xrightarrow{\text{side_effects_of}} \) Bridge (Drug) \( \xrightarrow{\text{treats}} \) Target (Diseases). (2)

Here, the bridge node (e.g., drug), used to query for 1-hop questions, serves as the intermediate entity linking the query and target.

Answer Definition. The target nodes are the final answers to the query. For 1-hop reasoning, this corresponds to all nodes directly connected to the query node. For 2-hop reasoning, the answers are all nodes that are accessible through the graph traversal via the bridge node, requiring models to infer both the intermediate (bridge) and final (target) nodes.

3.2 Dataset Construction Pipeline

The dataset is constructed using the following systematic process:

1. Entity Sampling: Nodes representing drugs, diseases, proteins, and phenotype entities are extracted from PrimeKG.

2. 2-Hop Path Definition: For 2-hop questions, valid paths are constructed by combining two connected edges, ensuring the query-bridge-target structure follows the one-to-many-to-many relationship:

\[
\text{Query} \xrightarrow{\text{Relation}_1} \text{Bridge} \xrightarrow{\text{Relation}_2} \text{Target}.
\]

(3)

3. 1-Hop Relationship Extraction: For 1-hop questions, all relationships connecting query nodes (e.g., drugs) to their target nodes (e.g., diseases) are extracted. To maintain consistency with 2-hop questions, 1-hop relationships without a corresponding 2-hop path are excluded.

4. Answer Extraction: For each question, all target nodes accessible through the graph traversal are extracted as answers. This ensures that the multi-answer nature of the dataset is preserved.

3.3 1-Hop and 2-Hop Questions

1-Hop Questions. For a 1-hop question, the model is required to directly link the query node to the target node, which is populated using a template "Name a Type({Query}) that is Label({Relationship}) by {Query}". For example,

"Name a disease that is treated by Drug Dr?"

with the answer set defined as:

\[
A = \{D_1, D_2, \ldots, D_n\},
\]

where \( D_i \) represents disease linked to the query drug \( Dr \).

2-Hop Questions. For a 2-hop question, the model must infer both the bridge node and the target node. The question template is "Name
a Type({Query}) that is Label({Relationship2}) by {Bridge} that has a Label({Relationship1}) {Query}". An example query is:

    "Name a disease that is treated by a drug that has a side effect S?" (6)

The model needs to traverse the graph through an intermediate bridge node (drug) before reaching the final target node (disease):

\[
A = \{D_1, D_2, \ldots, D_n\},
\]

where \( D_i \) represents disease linked to the phenotype that is a side effect of drug \( Dr \).

3.4 Dataset Statistics

<table>
  <tr>
    <th>Relation (Query:Target)</th>
    <th>Count</th>
  </tr>
  <tr>
    <td>Protein:Disease</td>
    <td>731</td>
  </tr>
  <tr>
    <td>Protein:Drug</td>
    <td>589</td>
  </tr>
  <tr>
    <td>Disease:Drug</td>
    <td>297</td>
  </tr>
  <tr>
    <td>Drug:Phenotype</td>
    <td>248</td>
  </tr>
  <tr>
    <td>Drug:Disease</td>
    <td>234</td>
  </tr>
  <tr>
    <td>Drug:Protein</td>
    <td>165</td>
  </tr>
  <tr>
    <td>Disease:Protein</td>
    <td>113</td>
  </tr>
  <tr>
    <td>Disease:Phenotype</td>
    <td>79</td>
  </tr>
  <tr>
    <td>Phenotype:Drug</td>
    <td>33</td>
  </tr>
  <tr>
    <td>Phenotype:Disease</td>
    <td>5</td>
  </tr>
</table>

Table 2: Distribution of 1-hop relations in BioHopR.

The BioHopR dataset consists of **2,494** unique 1-hop questions and **7,633** unique 2-hop questions, resulting in a total of **279,738** answers. On average, each question is associated with **36.65** answers, reflecting the dataset’s complexity and the many-to-many relationships inherent in biomedical knowledge. The dataset includes 10 distinct 1-hop relation types and 12 2-hop relation types, and the breakdown of the number of questions for each relation type is summarized in Tables 2 and 3.

The restriction to one-to-many-to-many relationships ensures that the dataset mirrors real-world biomedical reasoning scenarios, where single entities often relate to multiple downstream entities. This design makes the dataset uniquely suited for evaluating large language models (LLMs) on tasks requiring multi-step reasoning and comprehensive answer generation.

3.5 Qualitative Analysis

To better understand the models’ reasoning capabilities, we conducted a qualitative analysis on the questions about Type II Diabetes, as it is one of the widely studied diseases (Skyler et al., 2017).

<table>
  <tr>
    <th>Relation (Query:Bridge:Target)</th>
    <th>Count</th>
  </tr>
  <tr>
    <td>Drug:Protein:Disease</td>
    <td>3029</td>
  </tr>
  <tr>
    <td>Disease:Drug:Phenotype</td>
    <td>949</td>
  </tr>
  <tr>
    <td>Disease:Protein:Drug</td>
    <td>899</td>
  </tr>
  <tr>
    <td>Protein:Disease:Drug</td>
    <td>577</td>
  </tr>
  <tr>
    <td>Phenotype:Disease:Drug</td>
    <td>546</td>
  </tr>
  <tr>
    <td>Protein:Drug:Disease</td>
    <td>462</td>
  </tr>
  <tr>
    <td>Disease:Drug:Protein</td>
    <td>381</td>
  </tr>
  <tr>
    <td>Drug:Disease:Protein</td>
    <td>321</td>
  </tr>
  <tr>
    <td>Phenotype:Drug:Disease</td>
    <td>215</td>
  </tr>
  <tr>
    <td>Drug:Disease:Phenotype</td>
    <td>213</td>
  </tr>
  <tr>
    <td>Disease:Phenotype:Drug</td>
    <td>36</td>
  </tr>
  <tr>
    <td>Drug:Phenotype:Disease</td>
    <td>5</td>
  </tr>
</table>

Table 3: Distribution of 2-hop relations in BioHopR.

3.6 Reasoning Benchmark

BioHopR presents significant reasoning challenges:
• Models must implicitly identify intermediate bridge nodes in 2-hop questions while ensuring the correctness of the final answers.
• The many-to-many nature of biomedical relationships requires models to handle diverse answer sets while preserving reasoning consistency.

4 Experiments

We evaluate a range of LLMs on the BioHopR benchmark to assess their ability to reason over one-to-many-to-many relationships. The evaluation focuses on both single-answer and multi-answer reasoning for 1-hop and 2-hop questions, highlighting the challenges posed by multi-step reasoning and comprehensive answer generation.

4.1 Experimental Setup

**Models Evaluated.** We consider a diverse set of LLMs, categorized into general-purpose proprietary, reasoning proprietary, medical-specific, and open-source models, as detailed in Table 4. General-purpose models include GPT4O and smaller variants such as GPT4O-mini (Hurst et al., 2024). We also added O3-mini as it was most recent cost-effective reasoning proprietary model (OpenAI, 2025). We also evaluate open-source Llama models (Llama3.1 and Llama3.3) with varying parameter scales (8B and 70B) (Dubey
et al., 2024). We selected medical-specific models that are based on the baseline Llama3.1 architectures: UltraMedical-8B, HuatuoGPT-o1-8B, and HuatuoGPT-o1-70B (Zhang et al., 2024; Chen et al., 2024). HuatuoGPT-o1 models are trained for medical complex reasoning for medical problems.

<table>
  <tr>
    <th>Model Name</th>
    <th>Domain</th>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>General</td>
  </tr>
  <tr>
    <td>GPT4O-mini</td>
    <td>General</td>
  </tr>
  <tr>
    <td>O3-mini</td>
    <td>Reasoning</td>
  </tr>
  <tr>
    <td>Llama3.1 8B</td>
    <td>General</td>
  </tr>
  <tr>
    <td>Llama3.1 70B</td>
    <td>General</td>
  </tr>
  <tr>
    <td>Llama3.3 70B</td>
    <td>General</td>
  </tr>
  <tr>
    <td>UltraMedical-8B</td>
    <td>Medical</td>
  </tr>
  <tr>
    <td>HuatuoGPT-o1-8B</td>
    <td>Medical Reasoning</td>
  </tr>
  <tr>
    <td>HuatuoGPT-o1-70B</td>
    <td>Medical Reasoning</td>
  </tr>
</table>

Table 4: Models evaluated in the experiments.

4.2 Evaluation

The proprietary GPT models (GPT4O, GPT4O-mini, and O3-mini) were accessed using OpenAI’s API1. For open-source models, we used four A100 GPUs with 80GB memory per GPU for 70B parameter models and one A6000 GPU for 8B parameter models. The evaluation was conducted in a zero-shot setting, with a batch size of 1 and a temperature set to 0, except O3-mini model which does not support temperature parameter, to ensure deterministic responses. The evaluation code for open-source models were implemented using the HuggingFace Transformers library (Wolf, 2019).

4.3 Evaluation Metrics

Embedding-Based Precision. The precision (Prec) is computed using the cosine similarity between the predicted response and the ground truth answer list, leveraging BioLORD-2023-C embeddings (Remy et al., 2023). Let \( p \) denote the embedding of the predicted response and \( \{a_1, a_2, \ldots, a_n\} \) denote the embeddings of the ground truth answers. The cosine similarity for a prediction \( p \) and an answer \( a_i \) is defined as:

\[
\cos(p, a_i) = \frac{p \cdot a_i}{\|p\|\|a_i\|}.
\]

If the maximum cosine similarity across all ground truth answers satisfies:

\[
\max_{i \in \{1, \ldots, n\}} \cos(p, a_i) > \tau,
\]

then the prediction is considered a true positive. The precision (Prec) is then calculated as:

\[
\text{Prec} = \frac{|\text{True Positives}|}{|\text{Predicted Responses}|}
\]

We use \( \tau = 0.9 \) for BioLORD-2023-C embeddings after a grid search of threshold values from 0.5 to 0.9, which led an optimal setting with 0.9. Please refer to more details in the Appendix Figure 4. This threshold prioritizes precision, ensuring that only highly confident predictions are accepted as correct. By setting a high threshold, we align with the strict requirements of biomedical applications, minimizing false positives while maintaining robust handling of biomedical definition-level similarity and ambiguous synonyms.

5 Results and Discussion

5.1 Proprietary Models Demonstrate Robust Multi-Hop Reasoning

Proprietary models (GPT4O, GPT4O-mini, and O3-mini) demonstrate consistently strong performance across all metrics. For 1-hop tasks (Prec_HOP1), O3-mini achieves the highest precision (37.93%), followed by GPT4O (32.88%) and GPT4O-mini (28.11%). Interestingly, all proprietary models achieve identical performance on 2-hop tasks (Prec_HOP2: 14.57%), suggesting a possible shared capabilities for implicit reasoning or complex reasoning.

These results reflect the impact of the reasoning step before answering. O3-mini’s higher Prec_HOP1 indicates the reasoning capability of the model allowed it to reason well on single-step queries.

5.2 Open-Source Biomedical Models Face Significant Challenges

Open-source biomedical models struggle to match the performance of proprietary models, particularly on multi-hop tasks. HuatuoGPT-o1 models perform the worst, achieving near-zero precision for both 1-hop (Prec_HOP1: 0.20% for HuatuoGPT-o1-8B) and 2-hop (Prec_HOP2: 0.00% for HuatuoGPT-o1-70B). In contrast, UltraMedical-8B performs better (Prec_HOP1: 13.75%, Prec_HOP2: 5.21%).

These results suggest that although HuatuoGPT-1 was trained for medical complex reasoning, it’s generalizability is far less than UltraMedical.
<table>
  <tr>
    <th>Model</th>
    <th>Prec_HOP1 (%)</th>
    <th>Prec_HOP2 (%)</th>
    <th>BOTH_COR (%)</th>
    <th>BOTH_WR (%)↓</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>0.12</td>
    <td>0.05</td>
    <td>0.00</td>
    <td>99.76</td>
  </tr>
  <tr>
    <td>HuatuoGPT-o1-70B</td>
    <td>0.16</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>99.93</td>
  </tr>
  <tr>
    <td>HuatuoGPT-o1-8B</td>
    <td>0.20</td>
    <td>0.04</td>
    <td>0.00</td>
    <td>99.54</td>
  </tr>
  <tr>
    <td>UltraMedical-8B</td>
    <td>13.75</td>
    <td>5.21</td>
    <td>2.28</td>
    <td>82.33</td>
  </tr>
  <tr>
    <td>Llama-3.3-70B</td>
    <td>25.58</td>
    <td>9.58</td>
    <td>4.94</td>
    <td>68.33</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>26.38</td>
    <td>9.47</td>
    <td>4.93</td>
    <td>65.64</td>
  </tr>
  <tr>
    <td>GPT4O-mini</td>
    <td>28.11</td>
    <td>14.57</td>
    <td>6.54</td>
    <td>64.69</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>32.88</td>
    <td>14.57</td>
    <td>7.86</td>
    <td>57.96</td>
  </tr>
  <tr>
    <td>O3-mini</td>
    <td><b>37.93</b></td>
    <td><b>14.57</b></td>
    <td><b>8.93</b></td>
    <td><b>52.14</b></td>
  </tr>
</table>

Table 5: Performance metrics (in percentages) for various models. Prec_HOP1 and Prec_HOP2 represent the precision on 1-hop and 2-hop tasks, respectively. BOTH_COR indicates cases where both hops are correct, and BOTH_WR indicates cases where both hops are incorrect (the lower the better).

The reasoning demands of BioHopR is far different from medical license examination based QA datasets such as MedQA, which HuatuoGPT-o1 used for training. Still UltraMedical-8B’s performance, when compared to a larger general domain open-source models such as Llama3.1-70B and Llama3.3-70B, is far behind, suggesting persistent challenges in resolving bridge nodes for multi-hop queries.

Error Patterns. The BOTH_WR metric reveals systemic challenges in multi-hop reasoning for all models. Open-source models like HuatuoGPT-o1-70B exhibit the highest BOTH_WR rates (>99%), reflecting widespread failure in both reasoning hops. Proprietary models demonstrate significantly lower failure rates, with O3-mini achieving the best performance (<b>BOTH_WR: 52.14%</b>). However, even the best-performing models show substantial error rates in both hops, indicating that multi-step inference remains a bottleneck.

5.3 Multi-Hop Reasoning Remains a Bottleneck

Across all models, performance declines sharply from 1-hop to 2-hop tasks. For example, GPT4O’s precision drops from <b>Prec_HOP1: 32.88%</b> to <b>Prec_HOP2: 14.57%</b>, while open-source models like Llama-3.1-8B exhibit near-complete failure (<b>Prec_HOP2: 0.05%</b>).

This decline highlights the inherent complexity of multi-hop reasoning. Resolving 2-hop queries requires implicit inference of intermediate entities (e.g., bridge nodes) and alignment of reasoning chains across multiple steps.

5.4 Qualitative Analysis - Case Studies

Our qualitative analysis on various diseases, including Type II Diabetes and Schizophrenia aligns well with the evaluation result in Table 5. We highlight the diabetes-related questions in Figure 2.

<table>
  <tr>
    <th colspan="3">Questions</th>
  </tr>
  <tr>
    <th colspan="3">Hop1: "...a side effect of drug Troglitazone."</th>
  </tr>
  <tr>
    <th colspan="3">Hop2: "...a side effect of a drug ... treat type 2 diabetes."</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>Hop1 Prediction</th>
    <th>Hop2 Prediction</th>
  </tr>
  <tr>
    <td>HuatuoGPT-o1-70B</td>
    <td>"Alright, let's think about Troglitazone..."</td>
    <td>"Alright, let's think about this..."</td>
  </tr>
  <tr>
    <td>HuatuoGPT-o1-8B</td>
    <td>"<b>Hepatotoxicity</b>"</td>
    <td>"<b>Hypoglycemia</b>"</td>
  </tr>
  <tr>
    <td>UltraMedical-8B</td>
    <td>"<b>Hepatotoxicity</b>"</td>
    <td>"<b>Lactic acidosis, Hypoglycemia, Hyperkalemia</b>"</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>"<b>Hepatotoxicity</b>"</td>
    <td>"<b>Weight gain</b>"</td>
  </tr>
  <tr>
    <td>O3-mini</td>
    <td>"<b>Hepatotoxicity</b>"</td>
    <td>"<b>Weight gain</b>"</td>
  </tr>
</table>

Figure 2: Qualitative analysis of model responses to diabetes-related questions. Red-colored text shows the wrong answer. Orange-colored text shows the answer that is not in the answer list, but is plausible. Blue-colored text shows the correct answer.

Diabetes-related questions for drug Troglitazone, which has 202 side effects listed from PrimeKG, highlighted mixed performance among models. For instance, HuatuoGPT-o1-8B correctly predicted answers but diverged from the task constraints by elaborating on its reasoning instead of adhering to the prompt. Similarly, UltraMedical produced multiple answers when a single response was requested, with only some of the predictions being correct. In contrast, proprietary models such as GPT-4 reliably adhered to prompted task, consistently including
<table>
  <tr>
    <th rowspan="2">Relation Type</th>
    <th colspan="2">1-Hop</th>
    <th colspan="2">2-Hop</th>
  </tr>
  <tr>
    <th>GPT4O (Single/Multi)</th>
    <th>GPT4O-mini (Single/Multi)</th>
    <th>GPT4O (Single/Multi)</th>
    <th>GPT4O-mini (Single/Multi)</th>
  </tr>
  <tr>
    <th colspan="5">Same Query and Bridge</th>
  </tr>
  <tr>
    <td>Disease:Drug:Phenotype</td>
    <td>47.47 / 36.49</td>
    <td>43.77 / 26.56</td>
    <td>25.08 / 4.71</td>
    <td>25.08 / 4.01</td>
  </tr>
  <tr>
    <td>Disease:Drug:Protein</td>
    <td>47.47 / 36.49</td>
    <td>43.77 / 26.56</td>
    <td>3.67 / 1.13</td>
    <td>3.67 / 1.29</td>
  </tr>
  <tr>
    <td>Drug:Disease:Phenotype</td>
    <td>55.13 / 13.67</td>
    <td>50.85 / 14.75</td>
    <td>22.07 / 6.77</td>
    <td>22.07 / 7.12</td>
  </tr>
  <tr>
    <td>Drug:Disease:Protein</td>
    <td>55.13 / 13.67</td>
    <td>50.85 / 14.75</td>
    <td>4.67 / 0.27</td>
    <td>4.67 / 0.43</td>
  </tr>
  <tr>
    <th colspan="5">Same Query and Target</th>
  </tr>
  <tr>
    <td>Disease:Phenotype:Drug</td>
    <td>20.25 / 8.48</td>
    <td>22.78 / 7.14</td>
    <td>16.67 / 2.38</td>
    <td>16.67 / 1.50</td>
  </tr>
  <tr>
    <td>Disease:Protein:Drug</td>
    <td>35.40 / 7.8</td>
    <td>26.55 / 2.93</td>
    <td>8.12 / 4.60</td>
    <td>8.12 / 2.81</td>
  </tr>
  <tr>
    <td>Drug:Phenotype:Disease</td>
    <td>23.39 / 4.94</td>
    <td>31.05 / 3.70</td>
    <td>0.00 / 0.00</td>
    <td>0.00 / 1.01</td>
  </tr>
  <tr>
    <td>Drug:Protein:Disease</td>
    <td>20.61 / 5.77</td>
    <td>20.00 / 44.00</td>
    <td>20.14 / 1.84</td>
    <td>20.14 / 2.82</td>
  </tr>
  <tr>
    <th colspan="5">Others</th>
  </tr>
  <tr>
    <td>Phenotype:Disease:Drug</td>
    <td>0.00 / <b>8.08</b></td>
    <td>0.00 / <b>3.03</b></td>
    <td>14.47 / <b>17.24</b></td>
    <td>14.47 / 6.86</td>
  </tr>
  <tr>
    <td>Phenotype:Drug:Disease</td>
    <td>15.15 / 10.00</td>
    <td>24.24 / 2.43</td>
    <td>3.72 / 3.26</td>
    <td>3.72 / 2.40</td>
  </tr>
  <tr>
    <td>Protein:Disease:Drug</td>
    <td>35.29 / 7.20</td>
    <td>27.50 / 5.26</td>
    <td>2.95 / 1.89</td>
    <td>2.95 / 0.48</td>
  </tr>
  <tr>
    <td>Protein:Drug:Disease</td>
    <td>23.60 / 20.00</td>
    <td>14.43 / 9.35</td>
    <td>1.08 / <b>1.40</b></td>
    <td>1.08 / <b>1.40</b></td>
  </tr>
  <tr>
    <th>Overall</th>
    <th><b>32.88 / 8.09</b></th>
    <th>28.11 / 6.11</th>
    <th><b>14.57 / 3.46</b></th>
    <th>14.57 / 3.18</th>
  </tr>
</table>

Table 6: Comparison of Single-Answer prompting and Multi-Answer prompting for GPT4O and GPT4O-mini across 1-hop and 2-hop relation types. Precision (Prec) is reported.

relevant responses such as hepatotoxicity, even if these were not explicitly part of the predefined answer set. This behavior suggests that proprietary models may apply broader medical reasoning compared to open-source models. Proprietary models generally outperform open-source models in both task adherence and reasoning accuracy.

5.5 Ablation Study: Prompting Strategy

Prompting Setup. The dataset can also support multi-answer prompting, making two prompting strategies designed to evaluate different aspects of model reasoning:

• Single-Answer Prompting: The model is prompted to provide one correct answer (e.g., “Name a gene associated with Disease X.”). This evaluates the model’s ability to identify the most probable answer using implicit reasoning.

• Multi-Answer Prompting: The model is prompted to provide all correct answers (e.g., “Name all genes associated with Disease X.”). This evaluates the model’s ability to generate exhaustive, comprehensive outputs, which is inherently more challenging.

While both strategies are valuable for understanding model performance, Multi-Answer Prompting poses significant challenges. On average, each question in the dataset has 36.65 correct answers, making it computationally expensive and cognitively demanding for large language models to generate a complete answer set.

5.5.1 Evaluation Metric for Multi-Answer Prompting

Precision for Multi-Answer Prompting. For Multi-Answer Prompting, precision is computed using cosine similarity-based matching. Let \( P = \{p_1, p_2, \ldots, p_m\} \) denote the embeddings of the predicted responses and \( A = \{a_1, a_2, \ldots, a_n\} \) denote the embeddings of the ground truth answers. A predicted response \( p_j \) is considered a true positive if:
\[
\max_{i \in \{1, \ldots, n\}} \cos(p_j, a_i) > \tau,
\]
where we set \( \tau = 0.9 \) for high-confidence matches. Then the precision is computed in the same way as the Single-Answer Prompting.

5.5.2 Analysis and Results.

To analyze the feasibility of Multi-Answer Prompting, we conducted an ablation study using GPT4O and GPT4O-mini, the proprietary models. Table 6 presents the performance metrics for Single-Answer and Multi-Answer prompting across 1-hop and 2-hop tasks.

Single-Answer Prompting Outperforms Multi-Answer Prompting: GPT4O achieves an av-
erage Prec of 32.88% in 1-hop tasks, significantly higher than its Multi-Answer score of 8.09%. The gap is even more pronounced in 2-hop tasks, where GPT4O achieves 14.57% Prec compared to just 3.46%. A similar pattern holds for GPT4O-mini, showing 28.11% vs. 6.11% in 1-hop and 14.57% vs. 3.18% in 2-hop respectively. Relations with abstract or less structured targets (e.g., Disease:Drug:Protein and Drug:Phenotype:Disease) exhibit particularly poor precision scores under Multi-Answer prompting, with both models achieving below 2% in 2-hop tasks. These results highlight the difficulty of generating comprehensive answer sets, especially for complex reasoning paths.

Cases Where Multi-Answer Prompting Performs Competitively. Despite the overall trend, there are a few relation types where Multi-Answer prompting yields surprisingly competitive or even stronger results. For example, in the 2-hop relation Phenotype:Disease:Drug, GPT4O achieves 17.24% precision under Multi-Answer prompting—exceeding its Single-Answer precision of 14.47%. Similarly, in 1-hop tasks such as Phenotype:Disease and Protein:Drug, GPT4O-mini achieves 44.00% with Multi-Answer prompting compared to 20.00% under Single-Answer prompting. These cases suggest that for certain structured relations with high overlap between bridge and target entities, models may benefit from listing multiple answers rather than specifying a specific answer.

Based on these findings, we restricted our evaluation of all other models to Single-Answer Prompting. This decision is motivated by higher robustness and computational overhead of multi-answer prompting. Also in many real-world scenarios, users typically seek the most probable or relevant answer, aligning more closely with Single-Answer prompting.

While Multi-Answer prompting offers valuable insights into a model’s ability to generate exhaustive outputs, it remains a challenging evaluation paradigm. Future work could focus on improving model training and prompting strategies to better support comprehensive answer generation.

6 Conclusion

We introduced BioHopR, a benchmark for evaluating multi-hop, multi-answer reasoning in the biomedical domain. Built on the PrimeKG knowledge graph, BioHopR captures the complexity of real-world biomedical queries through one-to-many and many-to-many relationships, rigorously assessing reasoning over 1-hop and 2-hop tasks.

Evaluation results highlight that O3-mini, a proprietary model with a reasoning step, outperforms open-source models including biomedical models like HuatuoGPT-01. Across all models, the performance drop from 1-hop to 2-hop tasks underscores the difficulty of aligning intermediate reasoning steps, especially in bridging entities.

By addressing the lack of benchmarks for multi-hop reasoning in biomedical domain, BioHopR sets a new standard for evaluating reasoning capabilities and provides a critical step toward more robust and interpretable LLMs for biomedical research and real-world applications. Future directions include expanding the dataset to other knowledge sources and domains, such as chemistry.

Limitation

While BioHopR provides a rigorous benchmark for evaluating multi-hop reasoning in the biomedical domain, several limitations exist. BioHopR is currently focused on 4 major entities only: Protein, Phenotype, Drug, Disease. Also, it relies exclusively on a single knowledge graph, PrimeKG, which, while comprehensive, may not fully capture the diversity of biomedical knowledge or its real-world dynamics. This lack of diversity could bias model evaluation toward the structure and content of 4 major node types and PrimeKG, potentially under-representing a model’s ability to generalize to other knowledge and sources. While human evaluation was not the primary focus of this work, future efforts could include more extensive and diverse human evaluations to validate model-generated outputs.

Broader Impacts and Ethics Statement

Our work raises no major ethical concerns. All evaluations and experiments were conducted strictly for research purposes.

We will release BioHopR. License and copyright information, along with Terms of Use, will be made available upon release of the dataset and associated materials. While BioHopR facilitates advancements in biomedical reasoning tasks, it is not designed for use in real-world clinical applications. Consequently, models evaluated or trained on BioHopR should not be used for clinical decision-
making without rigorous validation and regulatory approvals.

This restriction aims to mitigate potential risks associated with incorrect reasoning or hallucinated outputs, which could lead to harmful clinical outcomes. Additionally, while BioHopR supports research into biomedical reasoning, it is critical that researchers use the benchmark responsibly, with appropriate safeguards in place to ensure the ethical use of derived insights and outputs.
References

Payal Chandak, Kexin Huang, and Marinka Zitnik. 2023. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67.

Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024. Huatuogpt-01, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. 2018. Embedding logical queries on knowledge graphs. Advances in neural information processing systems, 31.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. 2017. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. Elife, 6:e26726.

Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Di Jin, Eileen Pan, Nassim Ouattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.

Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024. Medexqa: Medical question answering benchmark with multiple explanations. arXiv preprint arXiv:2406.06331.

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240.

Xiaomin Liang, Daifeng Li, Min Song, Andrew Madden, Ying Ding, and Yi Bu. 2019. Predicting biomedical relationships using the knowledge and graph embedding cascade model. PLoS One, 14(6):e0218264.

Nicholas Matsumoto, Hyunjun Choi, Jay Moran, Miguel E Hernandez, Mythreye Venkatesan, Xi Li, Jui-Hsuan Chang, Paul Wang, and Jason H Moore. 2025. Escargot: an ai agent leveraging large language models, dynamic graph of thoughts, and biomedical knowledge graphs for enhanced reasoning. Bioinformatics, 41(2):btaf031.

Kanishka Misra, Cicero Nogueira dos Santos, and Siamak Shakeri. 2023. Triggering multi-hop reasoning for question answering in language models using soft prompts and random walks. arXiv preprint arXiv:2306.04009.

OpenAI. 2025. Openai o3-mini. https://openai.com/index/openai-o3-mini/. Accessed: 2025-02-01.

Ankit Pal, Logesh Kumar Umapathi, and Malaikanan Sankarasubbu. 2022. Medmca: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, pages 248–260. PMLR.

Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. 2024. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511.

Dattaraj J Rao, Shraddha S Mane, and Mukta A Palival. 2022. Biomedical multi-hop question answering using knowledge graph embeddings and language models. arXiv preprint arXiv:2211.05351.

François Remy, Kris Demuynck, and Thomas Deemeester. 2023. Biolord-2023: Semantic textual representations fusing llm and clinical knowledge graph insights. arXiv preprint arXiv:2311.16075.

Julian Schnitzler, Xanh Ho, Jiahao Huang, Florian Boudin, Saku Sugawara, and Akiko Aizawa. 2024. Morehopqa: More than multi-hop reasoning. arXiv preprint arXiv:2406.13397.

Jay S Skyler, George L Bakris, Ezio Bonifacio, Tamara Darsow, Robert H Eckel, Leif Groop, Per-Henrik Groop, Yehuda Handelsman, Richard A Insel, Chantal Mathieu, et al. 2017. Differentiation of diabetes by pathophysiology, natural history, and prognosis. Diabetes, 66(2):241–255.

Xiaorui Su, Yibo Wang, Shanghua Gao, Xiaolong Liu, Valentina Giunchiglia, Djork-Arné Clevert, and Marinka Zitnik. 2024. Knowledge graph based agent for complex, knowledge-intensive qa in medicine. arXiv preprint arXiv:2410.04660.

Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. 2021. Can language models be biomedical knowledge bases? arXiv preprint arXiv:2109.07154.

T Wolf. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.16837.

Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, et al. 2024. Ultramedical: Building specialized generalists in biomedicine. arXiv preprint arXiv:2406.03949.
Appendix

The Figure 3 illustrates the frequency distribution of target and bridge entities within the BioHopR dataset, highlighting key patterns. The left panel demonstrates the prevalence of proteins (e.g., CYP3A4), phenotypes (e.g., Nausea), drugs (e.g., Olanzapine), and diseases (e.g., Schizophrenia) as target nodes in multi-hop queries. Meanwhile, the right panel showcases the distribution of bridge entities, which frequently include proteins (e.g., CDK2), phenotypes (e.g., Neoplasm of the skin), drugs (e.g., Fostamatinib), and diseases. These patterns reflect the diversity and real-world complexity of biomedical entities, emphasizing the challenges of reasoning over structured knowledge graphs for multi-hop queries.

Figure 4 illustrates the results of a grid search for determining the optimal cosine similarity threshold for BioLORD-2023-C embeddings. The x-axis represents the threshold values, ranging from 0.1 to 0.9, while the y-axis shows the accuracy for "Both Correct" predictions. A sharp decline in accuracy is observed as the threshold increases, with accuracy plateauing beyond 0.8. The chosen threshold of 0.9 ensures high precision by accepting only highly confident predictions, aligning with the strict requirements of biomedical reasoning tasks.

![Bar charts showing the most common target and bridge entities for each node type in BioHopR](page_158_1047_1136_496.png)

Figure 3: Common target and bridge entities for each node type in BioHopR.

A Detailed Qualitative Analysis

We further included other diseases: Vitamin A Deficiency, Lung Cancer, Alzheimer’s Disease, Schizophrenia. We selected these medical conditions because they represent a range of domains within the biomedical field, which include nutritional deficiencies, metabolic disorders, chronic diseases and neurodegenerative conditions. This selection allows for a more comprehensive assessment of the models’ ability to reason across different medical contexts and complexities. Additionally, for conditions such as Type II diabetes and Vitamin A deficiency, the answers may seem quite straightforward, making them useful for assessing whether the models can correctly identify and reason over well-established medical knowledge. Whereas, for complex conditions such as Lung Cancer and Alzheimer’s Disease, we can evaluate the models ability to reason through more intricate, multi-factorial diseases.

Our qualitative analysis showed several key findings regarding the models’ reasoning capabilities across different diseases. Interestingly, none of the models generated questions for Alzheimer’s Disease, which was unexpected given its significant global impact and strong presence in the Knowledge Graph. In contrast, the models seemed to reason well over diabetes-related questions, although
![Grid search results showing the relationship between cosine similarity threshold and accuracy for "Both Correct" predictions. The chosen threshold of 0.9 is marked, reflecting the strict precision requirements in the biomedical domain.](page_184_120_1080_670.png)

Figure 4: Grid search results showing the relationship between cosine similarity threshold and accuracy for "Both Correct" predictions. The chosen threshold of 0.9 is marked, reflecting the strict precision requirements in the biomedical domain.

it would often provide multiple correct answers, even when prompted for a single response. This could suggest an alignment with well-established medical knowledge in this domain. For cancer-related questions, the models tended to select the most straightforward and common answers, though the Knowledge Graph contained a broader mix of more complex phenotypes. This seems to indicate a preference for simplicity in model-generated reasoning, potentially overlooking more nuanced aspects of the disease.

When comparing open-source models against proprietary models, the qualitative analysis shows that proprietary models generally performed better than open-source models in terms of providing structured and direct responses. proprietary models demonstrated a better adherence to the prompt constraints, whilst the open source models seem to show more explanatory or multi-component answers. For example, the HuatuoGPT-70B open source model, consistently responded with "thinking" before elaborating on its reasoning instead of directly providing a single answer for both 1-hop and 2 hop prediction as prompted. This suggests that the model prioritises explaining its reasoning than strictly following the prompt’s format. In contrast, however, proprietary models such as GPT-4 more reliably adhered to the prompt constraints. When prompted to give a single answer for one hop and two hop questions, GPT-4 consistently did so, and this was present across the closed-source GPT family, suggesting that these proprietary models may be better optimised for tasks requiring direct and efficient responses. Among the open source models tested, LLaMA Ultra Medical, a medical open source LLM, tended to provide multiple answers when prompted for one single answer, and of those multiple answers, apart from Type II diabetes, most answers were incorrect.

Taking an look into responses related to diabetes, responses were quite mixed. For instance, HuatuoGPT-01-8B performed outside the constraints of the task, correctly predicting the answer before proceeding to provide its reasoning. On the other hand, LLaMA 8B Instruct struggled with both Hop 2 and Hop 1 predictions, failing to generate the correct responses. Similarly, LLaMA Ultra Medical did not fully adhere to the prompt’s instructions — when asked to provide a single answer for Hop 2, it instead generated a list of multiple possible answers. While the listed responses were correct, this deviation indicates a challenge in following explicit
task constraints. Moreover, for Hop 1, the model’s response was incorrect, further highlighting inconsistencies in its performance. Interestingly, GPT-4 did not correctly predict the Hop 2 or Hop 1 answers in a strict sense. However, the model consistently included hepatotoxicity as a response—a condition that, while not explicitly listed among the correct answers, is still a relevant and justifiable finding. This pattern was observed across the GPT model family, suggesting that these models might apply broader medical reasoning even when their direct predictions do not align with predefined correct answers.

Schizophrenia, as seen in our figure, appeared frequently in the data. For GPT-4, in one-hop predictions, the model frequently guessed Clozapine as a treatment for schizophrenia. While this answer is medically correct, it was not explicitly part of the datasets predefined answer set. This suggests that the model is leveraging broader clinical knowledge rather than strictly adhering to the dataset’s constraints. This trend was also consistent across the other GPT-family models, GPT-4o mini. However, for the o3 models, the one-hop predictions were correct and within our predefined list of answers.

Reviewer paper summaries: The paper introduces BioHopR, a benchmark specifically designed to evaluate multi-hop, multi-answer reasoning in the biomedical domain using structured knowledge graphs. Built from PrimeKG, BioHopR includes both 1-hop and 2-hop reasoning tasks that reflect real-world biomedical complexity, such as one-to-many and many-to-many relationships.

This study develops a new benchmark dataset called BioHopR for multi-hop, multi-answer reasoning in biomedical domains. It conducts a series of evaluations of the most recent LLMs on the given task. Evaluation result suggests that O3-mini models outperformance other LLMs, and all models show a performance drop in multi-hop performance, which indicates the challenge for implicit reasoning detection in biomedical NLP tasks.
Summary of strengths: The paper addresses an important and underexplored challenge—multi-hop, multi-answer reasoning in the biomedical domain—by introducing a benchmark specifically designed for this purpose.

Overall, the study is well-written with a good justification of the motivation of developing the current benchmark dataset. 

The developed dataset could be of interest to the community and potentially useful for downstream applications and NLP tasks focusing on similar problems.

It also conducts a thorough comparison of the current SOTA LLMs on the developed tasks, showing the limited of the models in biomedical reasoning, especially the implicit ones.
Summary of weaknesses: 1. While the paper introduces the concept of one-to-many relationships in the introduction, the explanations of many-to-many and the one-many-many relationship structure are underdeveloped. A clearer and more detailed description of these key concepts would help readers better understand the dataset's structure and its reasoning challenges.
2. Although the authors mention they explored a range of thresholds from 0.5 to 0.9 and selected 0.9 as “optimal,” the paper does not provide supporting experiments or case studies to justify this choice. Including an ablation study or illustrative examples showing the impact of different thresholds would strengthen the credibility of this evaluation approach.
3. The dataset is constrained by a small set of entity node types, which results in relatively simple and limited relationships. Additionally, the experiments rely solely on straightforward prompting without exploring more advanced prompting strategies or providing the actual prompts used, which reduces reproducibility and depth of analysis.

When looking at the class distribution for some categories, some categories seem to have very data points compared to the others – could the authors elaborate more on some of the potential strategies for the skewed dataset distribution if the benchmark may be used widely in future evaluation?
Comments, suggestions and typos: see above

Please see the previous section for comments and suggestions.
Overall assessment scores: 2

3.5