Learning from Negative Samples in Biomedical Generative Entity Linking
Authors: Chanhwi Kim, Hyunjae Kim, Sihyeon Park, Jiwoo Lee, Mujeen Sung, Jaewoo Kang
Accepted at: acl25

Learning from Negative Samples in Biomedical Generative Entity Linking

Chanhwi Kim1*, Hyunjae Kim1*, Sihyeon Park1, Jiwoo Lee1,
Mujeen Sung2†, Jaewoo Kang1,3‡

1Korea University, 2Kyung Hee University, 3AIGEN Sciences
{chanhwi_kim, hyunjae-kim, sh10, hijiwoo7}@korea.ac.kr
mujeensung@khu.ac.kr, kangj@korea.ac.kr

Abstract

Generative models have become widely used in biomedical entity linking (BioEL) due to their excellent performance and efficient memory usage. However, these models are usually trained only with positive samples, i.e., entities that match the input mention’s identifier, and do not explicitly learn from hard negative samples, which are entities that look similar but have different meanings. To address this limitation, we introduce ANGEL (Learning from Negative Samples in Biomedical Generative Entity Linking), the first framework that trains generative BioEL models using negative samples. Specifically, a generative model is initially trained to generate positive entity names from the knowledge base for given input entities. Subsequently, both correct and incorrect outputs are gathered from the model’s top-k predictions. Finally, the model is updated to prioritize the correct predictions through preference optimization. Our models outperform the previous best baseline models by up to an average top-1 accuracy of 1.4% on five benchmarks. When incorporating our framework into pre-training, the performance improvement increases further to 1.7%, demonstrating its effectiveness in both the pre-training and fine-tuning stages. The code and model weights are available at https://github.com/dmis-lab/ANGEL.

1 Introduction

Biomedical entity linking (BioEL) involves aligning entity mentions in text with standardized concepts from biomedical knowledge bases (KB) such as UMLS (Bodenreider, 2004) or MeSH (Lipscomb, 2000).1 BioEL encounters significant challenges due to the diverse and ambiguous nature of biomedical terminology, including synonyms, abbreviations, and terms that look similar but have different meanings. For instance, ‘ADHD’ (CUI:C1263846, where CUI stands for Concept Unique ID) has synonyms such as hyperkinetic disorder and attention deficit hyperactivity disorder. Additionally, ‘ADA’ can be mapped to either adenosine deaminase (CUI:C1412179) or American Diabetes Association (CUI:C1705019) depending on the context in which the entity appears.

Recent studies have focused on addressing these challenges, broadly categorized into two approaches: similarity-based and generative BioEL. Similarity-based models (Sung et al., 2020; Liu et al., 2021; Lai et al., 2021; Bhowmik et al., 2021) encode input mentions and entities from KBs into the same vector space using embedding models.

1UMLS and MeSH are short for the Unified Medical Language System and Medical Subject Headings, respectively.

![Comparison of training approaches between existing generative BioEL models and our ANGEL method. The main limitation of current generative BioEL methods is that they are trained only on positive samples. This restricts their ability to distinguish between entity names that are similar in surface form but different in meaning depending on the context. Our ANGEL framework addresses this issue by training the model to prefer positive samples over negative ones.](page_1012_320_388_388.png)

Figure 1: Comparison of training approaches between existing generative BioEL models and our ANGEL method. The main limitation of current generative BioEL methods is that they are trained only on positive samples. This restricts their ability to distinguish between entity names that are similar in surface form but different in meaning depending on the context. Our ANGEL framework addresses this issue by training the model to prefer positive samples over negative ones.
They then calculate similarity scores to identify the most similar entities for each input entity. Although these approaches have achieved remarkable improvements, they require significant space to index and load embedding vectors for all candidate entities (De Cao et al., 2020). Furthermore, representing both the input and candidate entities as single vectors using a bi-encoder can limit the quality of their representations, making it difficult to handle challenging cases.

On the other hand, generative models (De Cao et al., 2020; Yuan et al., 2022a,b), built upon an encoder-decoder structure (Lewis et al., 2020; Raffel et al., 2020), directly generate the most likely entity name from the KB for the input entity. The output space is dynamically controlled through a constrained decoding strategy, ensuring that only entities from the target KB are generated. Generative models offer several advantages over similarity-based models, including greater memory efficiency and higher performance. They eliminate the need to index large external embedding vectors, and their auto-regressive formulation effectively cross-encodes the input document and candidate entities.

However, existing generative models are trained solely on positive samples and do not explicitly learn from negative samples. Despite their high performance, they encounter limitations when distinguishing between biomedical entities with similar surface forms but different meanings. Although similarity-based models address this issue by incorporating negative samples through synonym marginalization (Sung et al., 2020) or contrastive learning (Liu et al., 2021), applying these approaches to generative models is not straightforward. Consequently, generative models may overfit to surface-level features, reducing the models’ ability to generalize effectively across varied contexts, as illustrated in Figure 1.

To harness the benefits of generative approaches while overcoming their limitation of not using negative samples, we introduce a novel training framework, ANGEL. Our framework operates in two stages: positive-only training and negative-aware training (see Figure 2). In the first stage, a generative model is trained to generate biomedical terms from the KB that share the same identifier as the given input entity. In the second stage, we gather both correct and incorrect outputs from the model’s top-k predictions. The model is then updated to prioritize the correct predictions using a preference optimization algorithm (Burges, 2010; Rafailov et al., 2024). Models trained on our ANGEL framework significantly outperform the previous best similarity-based and generative BioEL models, achieving an average accuracy improvement of 1.7% across five datasets. Our contributions are as follows:

• We introduce ANGEL, the first-of-its-kind training framework that utilizes negative samples in generative entity linking. ANGEL overcomes the limitations of existing generative approaches by effectively employing negative samples during training.

• ANGEL is a versatile framework, demonstrating its applicability in both the pre-training and fine-tuning phases, leading to performance improvements at each stage. Additionally, our method is model-agnostic, consistently improving results across various backbone language models, with gains ranging from 0.9% to 1.7%.

• Our best model, pre-trained and fine-tuned with our framework, outperforms the previous best baseline model by 1.7% across five benchmark datasets.

2 Related Work

2.1 Biomedical Entity Linking

Biomedical entity linking (BioEL), also known as biomedical entity normalization, is a crucial task because of its application in several downstream tasks in the biomedical domain, such as literature search (Lee et al., 2016), knowledge extraction (Li et al., 2016a; Xiang et al., 2021; Zhang et al., 2023), knowledge graph alignment (Cohen and Hersh, 2005; Lin et al., 2022), and automatic diagnosis (Shi et al., 2021; Yuan and Yu, 2024). Typically, it is assumed that the target mention is already provided, and the task is solely to link this mention to the appropriate entity name from the KB. End-to-end BioEL (Zhou et al., 2021; Ujiie et al., 2021), which also involves identifying mentions within a sentence, is being actively researched, but this is not our focus and will not be discussed in detail.

Traditional classification-based approaches (Limsopatham and Collier, 2016a; Miftahutdinov et al., 2019) employed a softmax layer for classification, treating concepts as categorical variables and thereby losing the detailed information of concept names. Similarity-based (Sung et al.,
Figure 2: Overview of our method ANGEL. The core idea is to enhance both pre-training and fine-tuning by incorporating negative samples, which are obtained either through TF-IDF similarity or the model’s top-k predictions. This approach helps the model distinguish subtle differences between correct and incorrect entities.

2020; Liu et al., 2021; Lai et al., 2021; Zhang et al., 2022) models have significantly improved BioEL performance, which encodes mentions and candidate entity names in the same vector space. They are characterized by high memory consumption due to the need to encode entities into pre-computed embeddings, posing scalability challenges with large datasets (De Cao et al., 2020). Several studies have integrated the concept of clustering into BioEL (Angell et al., 2021; Agarwal et al., 2022).

2.2 Generative Entity Linking

Generative models have become a powerful method for entity linking by overcoming the limitations of similarity-based models. The GENRE framework (De Cao et al., 2020) was the first to demonstrate this approach. To enhance precision and reduce memory usage, GENRE introduced a constrained decoding method (Hokamp and Liu, 2017) using a prefix tree (trie), which restricts the output space to valid entity names. This technique also facilitates easy updates to the set of entities, making the system highly adaptable to changes in the KB. In the biomedical field, notable examples of generative models include GenBioEL (Yuan et al., 2022b) and BioBART (Yuan et al., 2022a). GenBioEL, in particular, is the first model to apply a generative model BART (Lewis et al., 2020) to BioEL, after pre-training it using UMLS. Additionally, several hybrid approaches, known as retrieve-and-generate methods, have been proposed (Xu et al., 2023; Lin et al., 2024). In these methods, a similarity-based model first retrieves the top-k candidates, which are then reranked using a generative model. Although generative approaches have shown high performance, their training has typically been limited to positive samples, as discussed in the introduction section. This absence of explicit negative sample learning often leads to confusion when entities share very similar surface forms but represent different concepts. In this study, we introduce the use of negative samples during training and demonstrate that this approach can significantly enhance
the performance of generative models.

3 Method

3.1 Task Formulation
Let \( \mathcal{D} = \{(\mathbf{x}_n, y_n)\}_{n=1}^N \) be a human-labeled dataset, where \( \mathbf{x}_n \) represents an input text and \( y_n \) is the gold identifier defined in a KB denoted by \( \mathcal{E} \). Each \( \mathbf{x}_n = (c^-_n, m_n, c^+_n) \) contains a target entity mention \( m_n \) along with its surrounding contextual information \( c^-_n \) and \( c^+_n \), which represents the tokens before and after the entity mention \( m_n \), respectively. For simplicity, we will omit the subscript \( n \). Our goal is to map each mention \( m \) to its corresponding gold identifier \( y \) from the set of entity names \( \mathcal{E} \). To achieve this, we define the model’s prediction \( y^* \) as follows:

\[
y^* = \mathcal{F}(\operatorname{argmax}_{e \in \mathcal{E}}\ p_\theta(e|m)),
\]

where e is an entity name defined in the KB, \( \mathcal{F} \) is a function that aligns entities to their identifiers, and \( \theta \) represents the model parameters.

A single gold identifier may have multiple associated entity names that refer to the same concept; we refer to these as synonyms. Previous generative BioEL approaches train the model to generate a textual synonym \( s \in \mathcal{S}_y \), where \( \mathcal{S}_y \subset \mathcal{E} \) denotes the set of entity names associated with the identifier \( y \), in an autoregressive manner as follows:

\[
p_\theta(s \mid \mathbf{x}, \mathbf{v}) = \prod_{t=1}^T p_\theta(s_t \mid s_{<t}, \mathbf{x}, \mathbf{v}),
\]

where \( T \) is the number of tokens of the synonym \( s \), \( s_t \) indicates the \( t \)-th token of the synonym, and \( \mathbf{v} \) is the prompt. In an encoder-decoder model structure (Lewis et al., 2020), the input to the encoder is formatted as follows:

\[
[\text{BOS}] \ c^- \ [\text{ST}] \ m \ [\text{ET}] \ c^+ \ [\text{EOS}],
\]

where the special tokens [ST] and [ET] surround the target mention, and the special tokens [BOS] and [EOS] represent the ‘Begin Of Sentence’ and ‘End Of Sentence’, respectively. The prefix prompt \( \mathbf{v} \) to the decoder, represented as ‘m is’, is concatenated with [BOS] and input to the decoder. The prompt is designed to make the decoder’s output resemble a natural language sentence, which helps to minimize discrepancies between language modeling and fine-tuning on the BioEL task.

As shown in Equation 2, existing models are trained solely to predict synonyms for the input mention (i.e., positive samples), without leveraging negative samples. In contrast, we propose a novel approach using negative samples, which we will describe in detail in the following sections.

3.2 ANGEL Framework
Our framework comprises two main stages: positive-only training, which warms up the model using positive samples to learn morphological similarities among synonyms, and negative-aware training, which progressively refines the model by incorporating negative samples (see Figure 2).

Positive-only training  We initialize the model to generate synonyms, similar to previous methods (see Equation 2). For the input mention, we select the most similar synonyms based on their vector similarity, which is calculated as follows:

\[
\hat{\mathcal{S}}_y = \operatorname{argsort}_{s \in \mathcal{S}_y} (\operatorname{TFIDF}(m, s)),
\]

where \( \operatorname{TFIDF}(\cdot) \) returns the TF-IDF similarity between tri-grams of the mention and its synonyms. We use the top-k subset \( \hat{\mathcal{S}}_y[:k] = \{s^{(1)}, \ldots, s^{(k)}\} \) as training instances for each mention.

Negative-aware training  After obtaining the top-k predictions from the model for each mention in the training set, we construct a dataset of triplets \( (\mathbf{x}, e_w, e_l) \), where \( \mathbf{x} \) denotes the mention along with its context (if available), \( e_w \) is the correct (preferred) entity, and \( e_l \) is an incorrect (dis-preferred) entity. From all possible \( (e_w, e_l) \) pairs, we retain only those for which the model ranks the incorrect entity \( e_l \) above the correct one \( e_w \), thereby reflecting an incorrect model preference. If the top-ranked prediction is already correct, we pair this correct entity \( e_w \) with the highest-ranked incorrect entity \( e_l \) to preserve the model’s original preference structure.

We denote the resulting training set by \( \mathcal{D}' \) and fine-tune the model using a pairwise preference loss, formulated as follows:

\[
\mathcal{L}(\theta) = - \mathbb{E}_{(\mathbf{x}, e_w, e_l) \sim \mathcal{D}'} [
    \log \sigma(\beta (r_\theta(e_w \mid \mathbf{x}) - r_\theta(e_l \mid \mathbf{x})))
],
\]

where \( r_\theta(e \mid \mathbf{x}) \) is a differentiable scoring function (e.g., \( \log p_\theta(e \mid \mathbf{x}) \)), \( \sigma \) is the sigmoid function, and \( \beta \) is a temperature (scaling) hyperparameter. A recent instantiation of this general preference learning framework is Direct Preference Optimization (DPO) (Rafailov et al., 2024), where the scoring
function is defined as a log-ratio with respect to a reference model \( p_{\text{ref}} \) as follows:

\[
r_\theta(\mathbf{e} \mid \mathbf{x}) = \log \frac{p_\theta(\mathbf{e} \mid \mathbf{x})}{p_{\text{ref}}(\mathbf{e} \mid \mathbf{x})},
\]

where \( p_\theta \) is the generative model being trained, \( p_{\text{ref}} \) is a reference generative model trained in a prior stage using positive-only data. We adopt DPO as a practical instantiation within our framework because it offers a principled and empirically effective way to incorporate prior model behavior through a reference distribution.

Applying ANGEL in pre-training Our framework supports not only fine-tuning with labeled datasets but also pre-training with the KB. Specifically, we automatically generate surrounding contextual information for each entity in the KB, using clause templates or definitions, as outlined in GenBioEL (Yuan et al., 2022b). When the entity definition \( d_y \) corresponding to the identifier \( y \) is available, a synonym and its definition are integrated into a pre-defined clause template as follows:1

[BOS] [ST] s [ET] is defined as \( d_y \) [EOS].

When no definitions are available in the KB, we replace \( d_y \) with alternative synonyms as follows:

[BOS] [ST] s_1 [ET] has synonyms such as s_2 [EOS],

where \( s_1 \) and \( s_2 \) are different synonyms. The input for the decoder is "[BOS] s (or s_1) is" and the expected output is another synonym (e.g., s_2) selected from the remaining synonyms.

The pre-training process, like fine-tuning, is divided into two stages: positive-only training and negative-aware training. However, due to the typically large scale of the KB, efficiency considerations are particularly important. In positive-only training, rather than utilizing all possible synonym combinations within the KB, we identify, for each entity, the most similar synonym based on TF-IDF similarity and designate it as the target synonym. For negative-aware training, instead of selecting negatives from the model’s predictions, negative samples are selected from entities exhibiting the highest TF-IDF similarity to the input mentions but possessing distinct identifiers.

1 Refer to Yuan et al. (2022b) for the full set of templates.

4 Experiments

4.1 Datasets

We utilized five popular BioEL benchmark datasets: NCBI-disease (Doğan et al., 2014), BC5CDR (Li et al., 2016b), COMETA (Basaldella et al., 2020), AskAPatient (Limsopatham and Collier, 2016b), and Medmentions (Mohan and Li, 2019), with the ST21pv subset used for Medmentions. Due to the lack of a test set in the AskAPatient dataset, we adhered to the 10-fold evaluation protocol outlined by Limsopatham and Collier (2016b). Also, AskAPatient dataset does not include context for the mentions. In the following tables, NCBI-disease, AskAPatient, and Medmentions are denoted as NCBI, AAP, and MM-ST21pv, respectively. Refer to Appendix A for detailed descriptions and statistics.

4.2 Baseline Models

We used top-performing similarity-based models (Sung et al., 2020; Liu et al., 2021; Lai et al., 2021; Zhang et al., 2022) as our baselines. Notably, Prompt-BioEL (Xu et al., 2023) employs a re-ranking-based approach. In the first stage, a similarity-based model, such as SapBERT, retrieves the top-k candidate entities from the knowledge base. In the second stage, these candidates are reranked using a cross-encoder. Although Prompt-BioEL may not be directly comparable, as it incorporates additional modules on top of existing models, we report its performance alongside for reference. Additionally, we include the previously best-performing generative models for comparison. (1) BART-large (Lewis et al., 2020) is an encoder-decoder language model pre-trained on a general-domain corpus. (2) BioBART-large (Yuan et al., 2022a) is the BART-large model continuously pre-trained on a biomedical-domain corpus. (3) GenBioEL (Yuan et al., 2022b) is initialized with the weights of the BART-large model and then pre-trained specifically for BioEL using UMLS. We excluded several models (Agarwal et al., 2022; Lin et al., 2024) due to the lack of publicly available code or the difficulty in reproducing their reported performance.

4.3 Implementation Details

Our framework was applied to each of these models during fine-tuning, referred to as ANGEL_FT, and during both pre-training and fine-tuning, referred to as ANGEL_PT+FT. For pre-training, we uti-
<table>
  <tr>
    <th>Model</th>
    <th>NCBI</th>
    <th>BC5CDR</th>
    <th>COMETA</th>
    <th>AAP</th>
    <th>MM-ST21pv</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="7">Similarity-based BioEL & Re-ranking</th>
  </tr>
  <tr>
    <td>BioSYN (Sung et al., 2020)</td>
    <td>91.1</td>
    <td>93.3<sup>†</sup></td>
    <td>71.3</td>
    <td>86.5<sup>†</sup></td>
    <td>OOM</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SapBERT (Liu et al., 2021)</td>
    <td>92.3</td>
    <td>88.6<sup>†</sup></td>
    <td>75.1</td>
    <td>89.0</td>
    <td>50.3<sup>†</sup></td>
    <td>79.1</td>
  </tr>
  <tr>
    <td>ResCNN (Lai et al., 2021)</td>
    <td>92.4</td>
    <td>94.0<sup>†</sup></td>
    <td>80.1</td>
    <td>77.4<sup>†</sup></td>
    <td>55.0</td>
    <td>79.3</td>
  </tr>
  <tr>
    <td>KRISSBERT (Zhang et al., 2022)</td>
    <td>91.3</td>
    <td>72.0<sup>†</sup></td>
    <td>80.1<sup>†</sup></td>
    <td>83.1<sup>†</sup></td>
    <td>72.2</td>
    <td>79.7</td>
  </tr>
  <tr>
    <td>Prompt-BioEL (Xu et al., 2023)</td>
    <td>91.9<sup>†</sup></td>
    <td>94.3<sup>†</sup></td>
    <td>82.7<sup>†</sup></td>
    <td>89.7<sup>†</sup></td>
    <td>72.6<sup>†</sup></td>
    <td>86.2</td>
  </tr>
  <tr>
    <th colspan="7">Generative BioEL (reported)</th>
  </tr>
  <tr>
    <td>BART (Lewis et al., 2020)</td>
    <td>90.2</td>
    <td>92.5</td>
    <td>80.7</td>
    <td>88.8</td>
    <td>71.5</td>
    <td>84.7</td>
  </tr>
  <tr>
    <td>BioBART (Yuan et al., 2022a)</td>
    <td>89.9</td>
    <td>93.3</td>
    <td>81.8</td>
    <td>89.4</td>
    <td>71.8</td>
    <td>85.2</td>
  </tr>
  <tr>
    <td>GenBioEL (Yuan et al., 2022b)</td>
    <td>91.9</td>
    <td>93.3</td>
    <td>81.4</td>
    <td>89.3</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <th colspan="7">Generative BioEL (reproduced)</th>
  </tr>
  <tr>
    <td>BART<sup>†</sup> (Lewis et al., 2020) + ANGEL<sub>FT</sub> (Ours)</td>
    <td>90.3</td>
    <td>93.0</td>
    <td>80.4</td>
    <td>88.7</td>
    <td>70.1</td>
    <td>84.5</td>
  </tr>
  <tr>
    <td></td>
    <td>91.4 (+1.1)</td>
    <td>93.6 (+0.6)</td>
    <td>81.3 (+0.9)</td>
    <td>89.5 (+0.8)</td>
    <td>71.2 (+1.1)</td>
    <td>85.4 (+0.9)</td>
  </tr>
  <tr>
    <td>BioBART<sup>†</sup> (Yuan et al., 2022a) + ANGEL<sub>FT</sub> (Ours)</td>
    <td>89.4</td>
    <td>93.5</td>
    <td>81.3</td>
    <td>89.3</td>
    <td>71.3</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td></td>
    <td>91.9 (+2.5)</td>
    <td><b>94.7</b> (+1.2)</td>
    <td>82.2 (+0.9)</td>
    <td><b>89.9</b> (+0.6)</td>
    <td><b>73.4</b> (+2.1)</td>
    <td><b>86.4</b> (+1.4)</td>
  </tr>
  <tr>
    <td>GenBioEL<sup>†</sup> (Yuan et al., 2022b) + ANGEL<sub>FT</sub> (Ours)</td>
    <td>91.0</td>
    <td>93.1</td>
    <td>80.9</td>
    <td>89.3</td>
    <td>70.7</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td></td>
    <td>92.5 (+1.5)</td>
    <td>94.4 (+1.3)</td>
    <td>82.4 (+1.5)</td>
    <td>89.9 (+0.6)</td>
    <td>71.9 (+1.2)</td>
    <td>86.2 (+1.2)</td>
  </tr>
  <tr>
    <td>+ ANGEL<sub>PT+FT</sub> (Ours)</td>
    <td><b>92.8</b> (+1.8)</td>
    <td><b>94.5</b> (+1.4)</td>
    <td><b>82.8</b> (+1.9)</td>
    <td><b>90.2</b> (+0.9)</td>
    <td><b>73.3</b> (+2.6)</td>
    <td><b>86.7</b> (+1.7)</td>
  </tr>
</table>

Table 1: The top-1 accuracy of the models across the five BioEL datasets. Our ANGEL framework is applied to generative BioEL models during fine-tuning (ANGEL<sub>FT</sub>) and both pre-training and fine-tuning (ANGEL<sub>PT+FT</sub>). '†': the results have been reproduced. 'OOM': an out-of-memory error occurred when using a single 80G A00 GPU.

lized the 2020AA version of the UMLS database,2 which comprises 3.09M entities, of which 199K concepts contain definitions. During pre-training, we saved checkpoints every 500 steps over 5 epochs and selected the best one based on the validation sets. We used top-3 synonyms as positive samples in positive-only training. The other hyperparameter configurations are detailed in Appendix B. In pre-processing, following Yuan et al. (2022b), we expanded abbreviations using AB3P (Sohn et al., 2008), lowercase texts, mark mention boundaries with special tokens [ST] and [ET], and discard mentions that overlap or are missing from the target KB. During pre-training, our models were trained using eight 80G A100 GPUs for 12 hours. During fine-tuning, a single A100 GPU was used.

4.4 Results

Consistent with previous studies (Sung et al., 2020; Liu et al., 2021), we used accuracy at top-1 (Acc@1) as our evaluation metric, which quantifies the percentage of mentions where the model correctly ranks the gold standard identifier as the top choice. To assess statistical significance, we employed bootstrapping with the same sample size as the original datasets, repeating the process 100 times, followed by a paired t-test. Table 1 shows that our framework consistently outperformed the performance of generative models (\( p \leq 8.2e^{-22} \) for all comparisons). Specifically, our fine-tuning method (i.e., ANGEL<sub>FT</sub>) improved the Acc@1 scores of BART, BioBART, and GenBioEL by 0.9%, 1.4%, and 1.2%, respectively. When pre-training is also applied (i.e., ANGEL<sub>PT+FT</sub>) to GenBioEL, the improvement increases to 1.7%, further highlighting the effectiveness of both pre-training and fine-tuning in ANGEL.

Similarity-based models often exhibit limited robustness, with performance varying significantly across datasets. In contrast, generative models tend to deliver more consistent results, highlighting a key strength. Among the baseline methods, the re-ranking-based model Prompt-BioEL achieves strong performance, substantially outperforming its underlying retriever, SapBERT, though at the cost of increased inference time. Notably, our ANGEL<sub>PT+FT</sub> model surpasses Prompt-BioEL across all datasets without relying on any re-ranking component, achieving an average improvement of 0.5%. Given this strong baseline, incorporating a re-ranking component into our model in future work may further enhance performance.

2https://www.nlm.nih.gov/research/umls/licensedcontent/umlsarchives04.html
<table>
  <tr>
    <th>Model</th>
    <th>NCBI</th>
    <th>BC5CDR</th>
    <th>COMETA</th>
    <th>AAP</th>
    <th>MM-ST21pv</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="7">Models with Negative-aware Training</th>
  </tr>
  <tr>
    <td>ANGEL (Ours)</td>
    <td>92.8</td>
    <td><b>94.5</b></td>
    <td><b>82.8</b></td>
    <td><b>90.2</b></td>
    <td>73.3</td>
    <td><b>86.7</b></td>
  </tr>
  <tr>
    <td>Prediction-based \( e_t \Rightarrow \) TF-IDF-based \( e_t \)</td>
    <td>91.8</td>
    <td>94.4</td>
    <td>81.6</td>
    <td>90.0</td>
    <td>71.5</td>
    <td>85.9</td>
  </tr>
  <tr>
    <td>\( p_\theta(e_t) > p_\theta(e_w) \) Pairs \( \Rightarrow \) All Possible Pairs</td>
    <td><b>92.9</b></td>
    <td>94.0</td>
    <td>81.9</td>
    <td>90.0</td>
    <td>72.0</td>
    <td>86.2</td>
  </tr>
  <tr>
    <td>\( e_t \) within Top-5 \( \Rightarrow \) Top-10 Predictions</td>
    <td>92.5</td>
    <td>94.0</td>
    <td>82.1</td>
    <td>89.6</td>
    <td>72.6</td>
    <td>86.2</td>
  </tr>
  <tr>
    <th colspan="7">Models without Negative-aware Training</th>
  </tr>
  <tr>
    <td>GenBioEL (Yuan et al., 2022b)</td>
    <td>91.0</td>
    <td>93.1</td>
    <td>80.9</td>
    <td>89.3</td>
    <td>70.7</td>
    <td>85.0</td>
  </tr>
</table>

Table 2: The ablation study on positive (\( e_w \)) and negative (\( e_t \)) pair selection during negative-aware fine-tuning. ‘⇒’ indicates a modification in our method.

<table>
  <tr>
    <th>Model</th>
    <th>NCBI</th>
    <th>BC5CDR</th>
  </tr>
  <tr>
    <td>TF-IDF (trigram-based)</td>
    <td><b>91.0</b></td>
    <td><b>92.6</b></td>
  </tr>
  <tr>
    <td>BioBERT-NLI</td>
    <td>90.1</td>
    <td>71.9</td>
  </tr>
  <tr>
    <td>SapBERT</td>
    <td>90.2</td>
    <td>84.1</td>
  </tr>
</table>

Table 3: Comparison of similarity models for retrieving positive and negative samples from KBs.

<table>
  <tr>
    <th>Model</th>
    <th>FT</th>
    <th>BC5CDR</th>
    <th>AAP</th>
  </tr>
  <tr>
    <td>BART</td>
    <td>x</td>
    <td>0.8</td>
    <td>15.6</td>
  </tr>
  <tr>
    <td>GenBioEL</td>
    <td>x</td>
    <td>33.1</td>
    <td>50.6</td>
  </tr>
  <tr>
    <td>+ ANGEL (Ours)</td>
    <td>x</td>
    <td><b>49.7</b></td>
    <td><b>61.5</b></td>
  </tr>
  <tr>
    <td>BART</td>
    <td>✓</td>
    <td>93.0</td>
    <td>88.7</td>
  </tr>
  <tr>
    <td>GenBioEL</td>
    <td>✓</td>
    <td>93.1</td>
    <td>89.3</td>
  </tr>
  <tr>
    <td>+ ANGEL (Ours)</td>
    <td>✓</td>
    <td><b>94.5</b></td>
    <td><b>90.2</b></td>
  </tr>
</table>

Table 4: The top-1 accuracy of models with different pre-training strategies, along with the fine-tuned scores. ‘FT’ denotes fine-tuning, with x representing pre-trained models without fine-tuning, and ✓ indicating models fine-tuned on human-annotated training sets.

5 Analysis

5.1 Ablation Study

We conducted in-depth analyses on the selection of positive and negative pairs, the effect of similarity models on synonym retrieval, and the effect of pre-training. Additional analyses and results—including the number of synonyms used in positive-only training and the effect of optimization functions—can be found in Appendix C.

Selection of positive and negative pairs Analyzing the impact of how positive-negative pairs are constructed during negative-aware training is crucial for determining the optimal strategy for selecting hard negatives and the appropriate number of pairs. We investigated the effects of three factors: (1) negative sampling techniques (i.e., whether to use the model’s incorrect predictions as negatives or rely on TF-IDF-based sampling), (2) the relative ranking of positive and negative samples, and (3) top-k selection (i.e., the number of negatives to include). Detailed results can be found in Table 2. Ultimately, selecting negatives from the model’s incorrect predictions proved to be the most important factor, with an average score difference of 0.8%, while the other factors showed smaller differences of 0.5%. More importantly, regardless of the specific negative-aware training approach, its application leads to significant performance improvements compared to models like GenBioEL, which do not incorporate such training. All models applying negative-aware training, including our ANGEL model, outperformed GenBioEL by 0.9% to 1.7% (\( p \leq 1.7e^{-5} \) for all comparisons).

Effect of similarity models Similarity models play a critical role in retrieving synonymous terms from the KB, and their choice can have a substantial impact on overall system performance. To assess their effectiveness, we evaluated three models: (1) TF-IDF, (2) BioBERT-NLI,\footnote{https://huggingface.co/gsarti/biobert-nli} a sentence embedding model fine-tuned on natural language inference datasets, and (3) SapBERT. These models were incorporated into the positive-aware training framework of GenBioEL on the NCBI-Disease and BC5CDR datasets. As presented in Table 3, the TF-IDF-based approach outperforms the two embedding-based models. Although the strong performance of trigram-based similarity highlights the utility of surface-level matching in BioEL, this does not imply that the task is inherently simple. While many synonyms exhibit similar surface forms, a substantial portion do not—posing challenging edge cases that demand more nuanced semantic understanding.
Figure 3: Analysis of the effect of negative-aware training. The x-axis represents the TF-IDF similarity between the input mentions and negative entities, while the y-axis depicts the difference in log probabilities between the top-1 positive prediction and negative entities for a given input mention. The NCBI-disease dataset was used.

Effect of pre-training Table 4 highlights the effectiveness of ANGEL’s pre-training by comparing other pre-training methods. BART, pre-trained using a standard language modeling objective but not specifically tailored for BioEL tasks, shows the lowest performance. In contrast, GenBioEL, pre-trained using synonyms from UMLS in a similar manner to our positive-only training, initially demonstrates a substantial performance advantage over BART. However, this gap narrows considerably after fine-tuning, to the point where it is no longer statistically significant. When ANGEL’s negative-aware training is applied to GenBioEL, its performance improves significantly, achieving gains of 16.6% on BC5CDR and 10.9% on AAP. Even after fine-tuning, the performance gap remains noticeable, with a difference of 1.4% on BC5CDR and 0.9% on AAP.

5.2 Understanding the Effectiveness of Negative-aware Training

Figure 3 provide an interpretation of how negative-aware training leads to performance improvements. While positive-only training increases the probability of identifying synonyms, it also raises the risk of incorrectly boosting the probability of negative samples that are morphologically similar to the input mention. In contrast, negative-aware training improves the identification of synonyms while simultaneously reducing the probability of incorrect negatives, making it particularly effective when these negatives share morphological similarity with the input mention. To verify this, we divided input mention-negative pairs from the NCBI-disease dataset into 10 bins based on their tri-gram TF-IDF similarity. We then computed the log probabilities of negatives within each bin for the corresponding input mentions, comparing them to the log probabilities of the top-1 positive predictions (i.e., the synonym assigned the highest probability by the model). As the similarity between the input mention and the negative entities increased, the probabilities assigned by GenBioEL to positive and negative samples became more similar, eventually leading to higher probabilities for the negative samples. In contrast, our model demonstrated a clear distinction in behavior, consistently prioritizing positive samples over negatives.

Figure 4: In-depth evaluation of GenBioEL and our ANGEL models based on the TF-IDF similarity between the input mentions and gold-standard entities. The NCBI-disease dataset was used.

5.3 Error Analysis

We conducted an in-depth evaluation of the models based on the similarity between the input mentions and the gold-standard entities. Similarity was calculated using tri-gram TF-IDF, with the gold-standard entity determined as the candidate synonym with the highest similarity score to the input mention. The similarity scores, ranging from 0 to 1.0, were divided into five bins, and accuracy was measured for each bin. As shown in Figure 4, errors predominantly occurred in the 0-0.2 and 0.2–0.4 bins, as indicated by the height of the hatched bars, which represent the number of errors. This suggests that models tend to struggle when the surface forms of the input mentions are not closely aligned with those of the gold-standard entities. Our method improves the generalizability of the model, leading to an overall reduction in GenBioEL’s errors across all bins, with particularly notable improvements in
<table>
  <tr>
    <th>Rank</th>
    <th>SapBERT</th>
    <th>GenBioEL</th>
    <th>ANGEL (Ours)</th>
  </tr>
  <tr>
    <td colspan="4">... aggressive the same way someone with [ST] ASPD [ET] would be, except teenagers ...</td>
  </tr>
  <tr>
    <td>1</td>
    <td>ASP</td>
    <td>Anankastic personality disorder</td>
    <td><b>Antisocial personality disorder (disorder)*</b></td>
  </tr>
  <tr>
    <td>2</td>
    <td>Acquired immune deficiency syndrome (disorder)</td>
    <td>Borderline personality disorder</td>
    <td><b>Antisocial personality disorder*</b></td>
  </tr>
  <tr>
    <td>3</td>
    <td>Acquired immune deficiency syndrome</td>
    <td>Oppositional defiant disorder</td>
    <td>Borderline personality disorder (disorder)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Mesalazine</td>
    <td><b>Antisocial personality disorder*</b></td>
    <td>Obsessive compulsive disorder (disorder)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Cryopyrin associated periodic syndrome (disorder)</td>
    <td>Oppositional defiant disorder (disorder)</td>
    <td><b>Dissocial personality disorder*</b></td>
  </tr>
  <tr>
    <td colspan="4">... I switched from lantus to [ST] basaglar [ET] in january and ...</td>
  </tr>
  <tr>
    <td>1</td>
    <td>Beagle</td>
    <td>Linagliptin substance</td>
    <td><b>Insulin glargine substance*</b></td>
  </tr>
  <tr>
    <td>2</td>
    <td>Basiliximab sodium</td>
    <td>Benzodiazepine substance</td>
    <td>Insulin glargine*</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Basiliximab substance</td>
    <td>Carisoprodol substance</td>
    <td>Insulin glulisine substance</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Albiglutide</td>
    <td>Cariprazine</td>
    <td>Ultripristal substance</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Albiglutide substance</td>
    <td>Benzocaine containing product</td>
    <td>Lansoprazole</td>
  </tr>
  <tr>
    <td colspan="4">... effects on amino acid (r-aminobutyric acid (GABA), [ST] glutamine [ET], aspartate and glutathione) levels ...</td>
  </tr>
  <tr>
    <td>1</td>
    <td>Glutamine</td>
    <td>Glutamine</td>
    <td>L-glutamine</td>
  </tr>
  <tr>
    <td>2</td>
    <td><b>Glutamic acid*</b></td>
    <td><b>Glutamic acid*</b></td>
    <td>Glutamine</td>
  </tr>
  <tr>
    <td>3</td>
    <td>L-glutamine</td>
    <td>Glutamylmethionine</td>
    <td>D-glutamine</td>
  </tr>
  <tr>
    <td>4</td>
    <td><b>L-glutamic acid*</b></td>
    <td>Glutamylalanine</td>
    <td>Glutamic acids</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Glutamic acids</td>
    <td>Glutaminic acids</td>
    <td><b>Glutamic acid*</b></td>
  </tr>
</table>

Figure 5: Top-5 predictions from different BioEL models are presented. Entity names with correct identifiers are highlighted in boldface with an asterisk. The first and second examples highlight the strengths of our model, while the final example illustrates its limitations. For a detailed explanation, please refer to the main text.

cases of low similarity. However, significant challenges remain, as the accuracy of our model is only 34.2% in the 0–0.2 bin, highlighting the need for further improvement.

5.4 Case Study

Figure 5 illustrates the predictions of SapBERT, GenBioEL, and ANGEL. In the first example, the mention ‘ASPD’ is an abbreviation for ‘antisocial personality disorder’ (also known as ‘dissocial personality disorder’). SapBERT incorrectly predicts ‘ASP’ due to the similarity in surface form. GenBioEL struggles to distinguish between correct entity names and those containing the words ‘personality disorder’. In contrast, our model successfully identifies the correct entities, without being misled by false entity names that contain overlapping terms. The second example involves the mention ‘basaglar,’ a medication that contains insulin glargine, a long-acting insulin. The challenge here arises from the fact that product names can differ significantly from the biomedical terms used to describe their active ingredients. This discrepancy leads to failures in both SapBERT and GenBioEL, as they struggle to connect the brand name to its corresponding biomedical entity. Nevertheless, our model successfully identifies the correct entity, showcasing its ability to handle such complex cases effectively. In the final example, our method was less effective. For the mention of ‘glutamine,’ neither SapBERT nor GenBioEL identified the correct answer, but they did rank ‘Glutamic acid,’ the correct entity, within the top 5 candidates. Our model, however, ranked the correct answer slightly lower. Consequently, while our model shows a notable improvement in top-1 accuracy, the increase in top-5 accuracy is relatively modest in some datasets. The effectiveness of our method also varies across different datasets. We discuss this limitation in more detail in Appendix E, noting that such cases are an area for further exploration.

6 Conclusions

In this study, we discussed the importance of negative samples in training generative BioEL models and introduced ANGEL, the first framework in this field to effectively incorporate negative-aware training into a generative model. Our models demonstrated the ability to learn subtle distinctions between entities with similar surface forms and contexts. Experimental results showed that ANGEL outperformed existing similarity-based and generative models, with notable performance improvements of 0.9%, 1.4%, and 1.7% for BART, BioBART, and GenBioEL, respectively, achieving the best performance across five public BioEL datasets.
Limitations

Our method is versatile and applicable to any generative model, but it has only been tested on encoder-decoder models and not on decoder-only models such as BioGPT (Luo et al., 2022). We plan to further investigate the effect of our method on these models. Additionally, it has not been tested on recent open-source large language models (LLMs) (Touvron et al., 2023; Chen et al., 2023). While we acknowledge that incorporating comparisons with LLMs and further assessing the effectiveness of our approach would be an interesting direction, using LLMs for entity linking presents new challenges. The primary concern with larger models is their inefficiency, particularly regarding slower inference speeds and higher memory requirements, which may render them unsuitable for most real-world applications. This issue becomes particularly problematic in biomedical information extraction, where processing millions of publications to extract meaningful insights is essential.

Our negative-aware training method may not be limited to a specific domain, yet we have only evaluated it on biomedical-domain datasets, which restricts the demonstration of its broad applicability. Nevertheless, we would like to emphasize the reasons for focusing on the biomedical domain. Biomedical entity linking has unique characteristics that differentiate it from other domains, making this problem both challenging and interesting. In general domains, ambiguity typically arises between different types of entities (e.g., whether “Liverpool” refers to a city or a sports club). Similarly, in the biomedical domain, ambiguity exists between different types, such as whether “Ebola” in Figure 1 refers to a disease or a virus. Additionally, biomedical entities often exhibit significant variations in their surface forms, even when they share the same identifier, i.e., they refer to the same entity. As shown in Figure 5, “Basaglar” can be expressed as other variations such as “insulin glargine substance” or “insulin glargine.” Furthermore, terms like “substance” in the entity “insulin glargine substance” overlap with many other entities (e.g., “Basiliximab substance,” “Linagliptin substance,” “Benzodiazepine substance”), making the task even more complex. Therefore, distinguishing between numerous candidates with similar surface forms is especially crucial in biomedical entity linking. We believe that our method, which trains the model using negative samples with similar structures, is particularly well-suited to tackle this challenge. However, exploring the application of our approach in other domains would be a valuable direction for future research.

Ethical Considerations

This study complies with ethical standards, ensuring that all datasets and models adhere to their respective licenses and usage terms. While our method was evaluated on five widely used datasets, these serve primarily as benchmarks and may not fully capture real-world complexities. Although the model demonstrates significant improvements, its limitations in handling low-similarity cases highlight the need for thorough validation before deployment, particularly in sensitive applications.

Acknowledgements

This research was supported by (1) the National Research Foundation of Korea (NRF-2023R1A2C3004176, RS-2023-00262002), (2) the Ministry of Health & Welfare, Republic of Korea (HR20C002103), (3) ICT Creative Consilience Program through the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (IITP-2025-RS-2020-II201819). M.S. was supported by (1) No. RS-2022-00155911: Artificial Intelligence Convergence Innovation Human Resources Development(Kyung Hee University), (2) No. RS-2024-00509257: Global AI Frontier Lab, and (3) the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program(IITP-2024-RS-2024- 00438239, 15%).

References

Dhruv Agarwal, Rico Angell, Nicholas Monath, and Andrew McCallum. 2022. Entity linking via explicit mention-mention coreference modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4644–4658.

Rico Angell, Nicholas Monath, Sunil Mohan, Nishant Yadav, and Andrew McCallum. 2021. Clustering-based inference for biomedical entity linking. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2598–2608.
Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020. COMETA: A corpus for medical entity linking in the social media. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3122–3137. Association for Computational Linguistics.

Rajarshi Bhowmik, Karl Stratos, and Gerard De Melo. 2021. Fast and effective biomedical entity linking using a dual encoder. In Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis, pages 28–37.

Olivier Bodenreider. 2004. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research, 32(suppl_1):D267–D270.

Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning, 11(23-581):81.

Eunsuk Chang and Javed Mostafa. 2021. The use of snomed ct, 2013-2020: a literature review. Journal of the American Medical Informatics Association, 28(9):2017–2026.

Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. 2023. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079.

Aaron M Cohen and William R Hersh. 2005. A survey of current work in biomedical text mining. Briefings in bioinformatics, 6(1):57–71.

Allan Peter Davis, Thomas C Wiegers, Michael C Rosenstein, and Carolyn J Mattingly. 2012. Medic: a practical disease vocabulary used at the comparative toxicogenomics database. Database, 2012:bar065.

N De Cao, G Izacard, S Riedel, and F Petroni. 2020. Autoregressive entity retrieval. In ICLR 2021-9th International Conference on Learning Representations, volume 2021. ICLR.

Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10.

Ada Hamosh, Alan F. Scott, Joanna S. Amberger, Carol A. Bocchini, and Victor A. McKusick. 2004. Online mendelian inheritance in man (omim), a knowledgebase of human genes and genetic disorders. Nucleic Acids Research, 33:D514 – D517.

Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535–1546.

Tuan Lai, Heng Ji, and ChengXiang Zhai. 2021. Bert might be overkill: A tiny but effective biomedical entity linker based on residual convolutional neural networks. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1631–1639.

Sunwon Lee, Donghyeon Kim, Kyubum Lee, Jaehoon Choi, Seongsoon Kim, Minji Jeon, Sangrak Lim, Donghee Choi, Sunkyu Kim, Aik-Choon Tan, et al. 2016. Best: next-generation biomedical entity search tool for knowledge discovery from biomedical literature. PloS one, 11(10):e0164680.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880.

Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016a. Biorecreative v cdr task corpus: a resource for chemical disease relation extraction. Database, 2016.

Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016b. Biorecreative v cdr task corpus: a resource for chemical disease relation extraction. Database, 2016.

Nut Limspacharn and Nigel Collier. 2016a. Normalising medical concepts in social media texts by learning semantic representation. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1014–1023.

Nut Limspacharn and Nigel Collier. 2016b. Normalising medical concepts in social media texts by learning semantic representation. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1014–1023.

Zhenxi Lin, Ziheng Zhang, Meng Wang, Yinghui Shi, Xian Wu, and Yefeng Zheng. 2022. Multi-modal contrastive representation learning for entity alignment. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2572–2584.

Zhenxi Lin, Ziheng Zhang, Xian Wu, and Yefeng Zheng. 2024. Biomedical entity linking as multiple choice question answering. In International Conference on Language Resources and Evaluation.

Carolyn E Lipscomb. 2000. Medical subject headings (mesh). Bulletin of the Medical Library Association, 88(3):265.
Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021. Self-alignment pretraining for biomedical entity representations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4228–4238.

Renqian Luo, Lai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409.

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235.

Zulfat Miftahutdinov, Elena Tutubalina, Samsung-PDMI Joint AI Center, and PDMI RAS. 2019. Deep neural models for medical concept normalization in user-generated texts. ACL 2019, page 393.

Sunil Mohan and Donghui Li. 2019. Medmentions: A large biomedical corpus annotated with umls concepts. arXiv preprint arXiv:1902.09476.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67.

Xiaoming Shi, Sendong Zhao, Yuxuan Wang, Xi Chen, Ziheng Zhang, Yefeng Zheng, and Wanxiang Che. 2021. Understanding patient query with weak supervision from doctor response. IEEE Journal of Biomedical and Health Informatics, 26(6):2770–2777.

Sunghwan Sohn, Donald C Comeau, Won Kim, and W John Wilbur. 2008. Abbreviation definition identification based on automatic precision estimates. BMC bioinformatics, 9:1–10.

Mujeen Sung, Hwisang Jeon, Jinhyuk Lee, and Jaewoo Kang. 2020. Biomedical entity representations with synonym marginalization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3641–3650.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944.

Shogo Ujiiie, Hayate Iso, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2021. End-to-end biomedical entity linking with span-based dictionary matching. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 162–167.

Yuejia Xiang, Ziheng Zhang, Jiaoyan Chen, Xi Chen, Zhenxi Lin, and Yefeng Zheng. 2021. Ontoea: Ontology-guided entity alignment via joint knowledge graph embedding. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1117–1128.

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: pushing the boundaries of llm performance in machine translation. In Proceedings of the 41st International Conference on Machine Learning, pages 55204–55224.

Zhenran Xu, Yulin Chen, and Baotian Hu. 2023. Improving biomedical entity linking with cross-entity interaction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13869–13877.

Hongyi Yuan and Sheng Yu. 2024. Efficient symptom inquiring and diagnosis via adaptive alignment of reinforcement learning and classification. Artificial Intelligence in Medicine, 148:102748.

Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. 2022a. Biobart: Pretraining and evaluation of a biomedical generative language model. BioNLP 2022@ ACL 2022, page 97.

Hongyi Yuan, Zheng Yuan, and Sheng Yu. 2022b. Generative biomedical entity linking via knowledge base-guided pre-training and synonyms-aware fine-tuning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4038–4048.

Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Nau mann, Jianfeng Gao, and Hoifung Poon. 2022. Knowledge-rich self-supervision for biomedical entity linking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 868–880.

Yongqi Zhang, Quanming Yao, Ling Yue, Xian Wu, Ziheng Zhang, Zhenxi Lin, and Yefeng Zheng. 2023. Emerging drug interaction prediction enabled by a flow-based graph neural network with biomedical network. Nature Computational Science, 3(12):1023–1033.
Baohang Zhou, Xiangrui Cai, Ying Zhang, and Xiaojie Yuan. 2021. An end-to-end progressive multi-task learning framework for medical named entity recognition and normalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6214–6224.
A Datasets

Table A presents the statistics of the five datasets used, along with their corresponding target knowledge bases.

NCBI-disease The NCBI-disease dataset (Doğan et al., 2014) contains 793 PubMed abstracts annotated with 6,892 disease mentions that are mapped to 790 unique disease concepts using the MEDIC ontology (Davis et al., 2012). MEDIC is a medical dictionary that integrates disease concepts, synonyms, and definitions from both MeSH (Lipscomb, 2000) and OMIM (Hamosh et al., 2004), encompassing a total of 9,700 unique disease entities. This dataset is primarily used for disease recognition and concept normalization tasks.

BC5CDR The BC5CDR dataset (Li et al., 2016b) includes 1,500 PubMed abstracts with 4,409 chemical entities, 5,818 disease entities, and 3,116 chemical-disease interactions. All annotated entities are mapped to the MeSH ontology (Lipscomb, 2000), which is a subset of UMLS (Bodenreider, 2004). This dataset is widely used for biomedical entity recognition and interaction studies. To fit the purpose of our study, we use only the chemical and disease annotations and discard the interaction annotations.

COMETA COMETA (Basaldella et al., 2020) focuses on layman medical terminology, compiled from four years of content across 68 health-related subreddits. This dataset consists of 20K biomedical entity mentions annotated with concepts from SNOMED CT (Chang and Mostafa, 2021). It is utilized for the normalization of consumer health expressions into standardized terminologies.

AskAPatient (AAP) The AskAPatient dataset (Limsopatham and Collier, 2016b) contains 8,662 phrases from social media language, each mapped to medical concepts from SNOMED CT (Chang and Mostafa, 2021). This dataset does not include contextual information, meaning that mentions are disambiguated solely based on the phrases themselves. Since the AskAPatient dataset lacks a test set, we employed a 10-fold cross-validation approach as outlined in the original paper by Limsopatham and Collier (2016a). The statistics reported are the averages across these folds.

MM-ST21pv The Medmentions dataset (Mohan and Li, 2019) is a large-scale resource for biomedical entity recognition. The ST21pv subset includes 4,392 PubMed abstracts with over 200,000 entity mentions linked to 21 selected UMLS semantic types. This dataset provides a comprehensive resource for training and evaluating biomedical entity recognition systems. Unlike the original dataset, we use the 2020AA version of UMLS as the KBs because the 2017AA version of UMLS is not directly accessible. This leads to some differences after preprocessing due to variations between versions. Specifically, our dataset deviates from the original Medmentions dataset by 741 training samples (0.6%), 284 validation samples (0.7%), and 235 test samples (0.6%).

B Hyperparameter Configurations

Table B details the hyperparameters used for positive-only training and negative-aware training across the BioEL benchmark datasets. We searched for the optimal hyperparameter settings using the validation sets. We refer to the study of Yuan et al. (2022b) to determine the range of the hyperparameters. During pre-training, we used the same hyperparameters as in GenBioEL. For positive-only training, we explored a range of training steps between 20K and 40K, a learning rate between 2e-5 and 3e-7, and batch sizes from 8 to 16, except during pre-training. During negative-aware training, we fixed the \( \beta \) at 0.1, in accordance with the basic configuration of DPO, and searched the hyperparameter space using a learning rate between 2e-5 and 1e-6 and batch sizes ranging from 8 to 64. We used the source codes provided by Yuan et al. (2022b)\footnote{https://github.com/Yuanhy1997/GenBioEL} and alignment handbook (Tunstall et al., 2023)\footnote{https://github.com/huggingface/alignment-handbook}.

C Ablation Study

Effect of pre-training In addition to Table 4 which shows the effect of pre-training on BC5CDR and AAP, Table C demonstrates that ANGEL’s pre-training improves top-1 accuracy on NCBI-disease, COMETA, and MM-ST21pv, both before fine-tuning (\( \times \)) and after fine-tuning (\( \checkmark \)).

Effect of optimization functions Our negative-aware framework is compatible with various optimization methods. To demonstrate this flexibility, we evaluated three additional loss functions during fine-tuning: (i) a simple pairwise loss, where
<table>
  <tr>
    <th>Dataset</th>
    <th>NCBI</th>
    <th>BC5CDR</th>
    <th>COMETA</th>
    <th>AAP</th>
    <th>MM-ST21pv</th>
  </tr>
  <tr>
    <th>Entity types</th>
    <th>Disease</th>
    <th>Disease/chemical</th>
    <th>Medical concepts</th>
    <th>Medical concepts</th>
    <th>21 UMLS types</th>
  </tr>
  <tr>
    <th colspan="6"># Examples</th>
  </tr>
  <tr>
    <td>Training</td>
    <td>5,784</td>
    <td>9,285</td>
    <td>13,489</td>
    <td>15,665</td>
    <td>121,498</td>
  </tr>
  <tr>
    <td>Validation</td>
    <td>787</td>
    <td>9,515</td>
    <td>2,176</td>
    <td>793</td>
    <td>40,600</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>960</td>
    <td>9,654</td>
    <td>4,350</td>
    <td>866</td>
    <td>39,922</td>
  </tr>
  <tr>
    <th colspan="6">KB statistics</th>
  </tr>
  <tr>
    <td>Entity names</td>
    <td>108,092</td>
    <td>809,929</td>
    <td>904,798</td>
    <td>3,398</td>
    <td>6,051,091</td>
  </tr>
  <tr>
    <td>Identifiers</td>
    <td>14,944</td>
    <td>268,162</td>
    <td>350,830</td>
    <td>1,036</td>
    <td>3,092,324</td>
  </tr>
</table>

Table A: The statistics of the benchmark datasets and their corresponding KBs.

<table>
  <tr>
    <th rowspan="2">Hyperparameter</th>
    <th rowspan="2">Pre-training</th>
    <th colspan="5">Fine-tuning</th>
  </tr>
  <tr>
    <th>NCBI</th>
    <th>BC5CDR</th>
    <th>COMETA</th>
    <th>AAP</th>
    <th>MM-ST21pv</th>
  </tr>
  <tr>
    <th colspan="7">Positive-only Training</th>
  </tr>
  <tr>
    <td>Training Steps</td>
    <td>80K</td>
    <td>20K</td>
    <td>30K</td>
    <td>40K</td>
    <td>30K</td>
    <td>40K</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>4e-5</td>
    <td>3e-7</td>
    <td>5e-6</td>
    <td>2e-5</td>
    <td>5e-6</td>
    <td>3e-5</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>384</td>
    <td>16</td>
    <td>16</td>
    <td>16</td>
    <td>16</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Adam \( \epsilon \)</td>
    <td>1e-8</td>
    <td>1e-8</td>
    <td>1e-8</td>
    <td>1e-8</td>
    <td>1e-8</td>
    <td>1e-8</td>
  </tr>
  <tr>
    <td>Adam \( \beta \)</td>
    <td>(0.9,0.999)</td>
    <td>(0.9,0.999)</td>
    <td>(0.9,0.999)</td>
    <td>(0.9,0.999)</td>
    <td>(0.9,0.999)</td>
    <td>(0.9,0.999)</td>
  </tr>
  <tr>
    <td>Warmup Steps</td>
    <td>1,600</td>
    <td>0</td>
    <td>500</td>
    <td>1000</td>
    <td>0</td>
    <td>1,000</td>
  </tr>
  <tr>
    <td>Attention Dropout</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Clipping Grad</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Label Smoothing</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
  </tr>
  <tr>
    <th colspan="7">Negative-aware Training</th>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>5</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>1e-5</td>
    <td>1e-5</td>
    <td>1e-6</td>
    <td>5e-6</td>
    <td>5e-6</td>
    <td>5e-6</td>
  </tr>
  <tr>
    <td>\( \beta \) (DPO)</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>64</td>
    <td>16</td>
    <td>16</td>
    <td>32</td>
    <td>8</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Warmup Steps</td>
    <td>1000</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table B: Hyperparameters for positive-only training and negative-aware training.

<table>
  <tr>
    <th>Model</th>
    <th>FT</th>
    <th>NCBI</th>
    <th>COMETA</th>
    <th>MM</th>
  </tr>
  <tr>
    <td>BART</td>
    <td>x</td>
    <td>10.7</td>
    <td>8.4</td>
    <td>0.9</td>
  </tr>
  <tr>
    <td>GenBioEL</td>
    <td>x</td>
    <td>58.2</td>
    <td>42.4</td>
    <td>10.4</td>
  </tr>
  <tr>
    <td>+ ANGEL (Ours)</td>
    <td>✓</td>
    <td><b>64.6</b></td>
    <td><b>49.8</b></td>
    <td><b>18.2</b></td>
  </tr>
  <tr>
    <td>BART</td>
    <td>✓</td>
    <td>90.3</td>
    <td>80.4</td>
    <td>70.1</td>
  </tr>
  <tr>
    <td>GenBioEL</td>
    <td>✓</td>
    <td>91.0</td>
    <td>80.9</td>
    <td>70.7</td>
  </tr>
  <tr>
    <td>+ ANGEL (Ours)</td>
    <td>✓</td>
    <td><b>92.8</b></td>
    <td><b>82.8</b></td>
    <td><b>73.3</b></td>
  </tr>
</table>

Table C: The top-1 accuracy of models with different pre-training strategies, along with the fine-tuned scores. 'FT' denotes fine-tuning, with x representing pre-trained models without fine-tuning, and ✓ indicating models fine-tuned on human-annotated training sets. 'MM' represents the MM-ST21pv dataset.

<table>
  <tr>
    <th>Model</th>
    <th>Acc@1</th>
  </tr>
  <tr>
    <td>GenBioEL</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td>ANGEL<sub>FT</sub> (Pairwise)</td>
    <td>85.9</td>
  </tr>
  <tr>
    <td>ANGEL<sub>FT</sub> (CPO) (Xu et al., 2024)</td>
    <td>85.9</td>
  </tr>
  <tr>
    <td>ANGEL<sub>FT</sub> (SimPO) (Meng et al., 2024)</td>
    <td>86.1</td>
  </tr>
  <tr>
    <td>ANGEL<sub>FT</sub> (DPO) (Rafailov et al., 2024)</td>
    <td>86.2</td>
  </tr>
</table>

Table D: Performance with different optimization functions. Average top-1 accuracy across the five benchmarks is reported.

\( r_\theta(e \mid x) \) is defined as \( \log p_\theta(e; \cdot) \); and two preference optimization methods that build upon and improve DPO: (ii) Contrastive Preference Optimization (CPO) (Xu et al., 2024), and (iii) Similarity Preference Optimization (SimPO) (Meng et al., 2024). Note that Equation 4 in the Method section presents a simplified version for clarity of explanation. In practice, CPO and SimPO introduce additional terms and require slight extensions to this formulation. For a detailed comparison of the exact equations, we refer readers to Table 7 in Meng et al. (2024), which provides a clear summary of the differences.

As shown in Table D, while DPO achieves the highest accuracy, the performance differences among the optimizers are relatively small. Regard-
Figure A: The ablation study to determine the optimal number of synonyms. GenBioEL with ANGEL_{PT} was fine-tuned in this experiment. The scores are generally the highest when \( k = 3 \).

Figure B: In-depth evaluation of GenBioEL and ANGEL using TF-IDF similarity between input mentions and gold entities on the COMETA dataset.

less of the specific method used, all optimizers consistently enhance performance and outperform the GenBioEL baseline.

The number of synonyms To evaluate the impact of incorporating multiple synonyms during fine-tuning (Equation 3), we conducted experiments by varying the number of synonyms associated with each mention, testing with 1, 3, and 5 synonyms. As a result, using 3 synonyms proved to be optimal, outperforming the approach that used only a single top-1 synonym in the study of Yuan et al. (2022b).

D Error Analysis

Consistent with the analysis on the NCBI-disease dataset (Figure 4), Figure B reveals that models on the COMETA dataset most frequently made errors in the 0.0–0.2 bin, where input mentions have low similarity to gold-standard entities. Across all similarity bins, our ANGEL framework consistently improved upon GenBioEL’s performance, leading to overall gains. A similar trend is observed in Figure C for the MedMentions dataset, where ANGEL again outperforms GenBioEL across all bins. These results highlight the need for future research focused on reducing errors in low-similarity scenarios.

Figure C: In-depth evaluation of GenBioEL and ANGEL using TF-IDF similarity between input mentions and gold entities on the Medmentions dataset.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">BC5CDR</th>
    <th colspan="2">AAP</th>
  </tr>
  <tr>
    <th>Acc@1</th>
    <th>Acc@5</th>
    <th>Acc@1</th>
    <th>Acc@5</th>
  </tr>
  <tr>
    <td>GenBioEL</td>
    <td>93.1</td>
    <td>95.7</td>
    <td>89.3</td>
    <td><b>95.4</b></td>
  </tr>
  <tr>
    <td>+ ANGEL_{PT}</td>
    <td>94.4</td>
    <td>96.5</td>
    <td>89.5</td>
    <td>94.7</td>
  </tr>
  <tr>
    <td>+ ANGEL_{PT + FT}</td>
    <td><b>94.5</b></td>
    <td><b>96.8</b></td>
    <td><b>90.2</b></td>
    <td>95.2</td>
  </tr>
</table>

Table E: Comparison of top-1 and top-5 accuracy between the baseline model and models trained with ANGEL method after fine-tuning and pre-training on the BC5CDR and AAP datasets.

E Top-5 Accuracy

Table E presents our model’s top-1 and top-5 accuracy on the BC5CDR and AAP datasets. It compares the performance of our model in its baseline form (GenBioEL) and after fine-tuning (ANGEL_{PT}) and combined pre-training and fine-tuning (ANGEL_{PT + FT}). Our approach consistently boosts top-1 accuracy across all datasets, though the trends in top-5 accuracy are less uniform. In BC5CDR, both top-1 and top-5 accuracy show significant improvements: top-1 accuracy rises by 1.4 percentage points (from 93.1% to 94.5%), and top-5 accuracy increases by 1.1 percentage points (from 95.7% to 96.8%). However,
the AAP dataset exhibits a different pattern. While top-1 accuracy improves by 0.9 percentage points (from 89.3% to 90.2%), top-5 accuracy slightly declines: there is a 0.7 percentage points drop (from 95.4% to 94.7%) after fine-tuning and a 0.2 percentage points decrease (from 95.4% to 95.2%) after combined pre-training and fine-tuning. This decline in top-5 accuracy may be due to the AAP dataset’s limited contextual information, forcing the model to rely predominantly on the mention form, making it more challenging to maintain high accuracy across multiple predictions. Additionally, the negative sampling strategy could unintentionally bias the model toward optimizing top-1 accuracy, thereby impacting top-5 performance. In conclusion, while our method consistently improves top-1 accuracy, the occasional slight decreases in top-5 accuracy, as observed in the AAP dataset, underscore the need for further refinement to maintain balanced accuracy across different ranking levels. Future work should focus on training strategies that preserve or enhance top-5 accuracy alongside top-1 improvements.

Reviewer paper summaries: This paper focuses on the biomedical entity linking and try to introduce negative samples to generative models.
Summary of strengths: 1.Introducing negative samples to generative biomedical entity linking.
Summary of weaknesses: 1.The novelty is relatively low. In fact, negative samples in entity linking is common and this paper only utilizes a method of other paper to introduce negative samples to generative biomedical entity linking.

2.Lack of a large number of biomedical entity linking baselines. The methods compared in this paper are all pre-2022. Biomedical entity linking has published a large number of papers in recent years, and there is a lack of discussion and comparison with these papers. e.g. BioPRO: Context-Infused Prompt Learning for Biomedical Entity Linking and BioFEG: Generate Latent Features for Biomedical Entity Linking.

3.Paper writing and motivation need to be further improved.
Comments, suggestions and typos: N/A
Overall assessment scores: 1.5