A Comprehensive Evaluation of Biomedical Entity Linking Models
Authors: David Kartchner
Jennifer Deng
Shubham Lohiya
Tejasri Kopparthi
Prasanth Bathala
Daniel Domingo-Fernández
Cassie S. Mitchell
Venue: EMNLP 2023 Main (2023)
Keywords: Entity Linking
Entity Normalization
Candidate Generation
Biomedical Natural Language Processing
TL;DR: An evaluation on the robustness and errors of biomedical entity linking models across a broad range of data and entity types.

Abstract: Biomedical entity linking (BioEL) is the process of connecting entities referenced in documents to entries in biomedical databases such as the Unified Medical Language System (UMLS) or Medical Subject Headings (MeSH).  The study objective was to comprehensively evaluate nine recent state-of-the-art biomedical entity linking models under a unified framework.  We compare these models along axes of (1) accuracy, (2) speed, (3) ease of use, (4) generalization, and (5) adaptability to new ontologies and datasets.  We additionally quantify the impact of various preprocessing choices such as abbreviation detection. Systematic evaluation reveals several notable gaps in current methods. In particular, current methods struggle to correctly link genes and proteins and often have difficulty effectively incorporating context into linking decisions. To expedite future development and baseline testing, we release our unified evaluation framework and all included models on GitHub at https://github.com/davidkartchner/biomedical-entity-linking

A Comprehensive Evaluation of Biomedical Entity Linking Models
David Kartchner
Jennifer Deng*
Shubham Lohiya²
Tejasri Kopparthi²
Prasanth Bathala²
Daniel Domingo-Fernández"
Cassie S. Mitchell
'Enveda Biosciences
GGorgia Institute of Technology
david. www.thheeeegggtccc.ccc..o. edu dani@envedab com messiemiitcin @bme gatech. edu
Abstract
While entity linking a critical task for text mining, BioEL remains an unsolved problem with diverse challenges. First, biomedical literature has complex, specialized jargon that may differ between biomedical subspecialties. This leads to large, varied sets of synonyms that can be used to reference the same entity. For example, the entity ncbigene:37970 can be referred to by the aliases "ORC", "ORC4", "origin recognition complex sub- unit 4" "CG2917", "rDmORC", "dmOre4", etc. Moreover, the entity referenced by particular sur- face form is context-dependent and may require specialized domain expertise to disambiguate. For instance, within the Unified Medical Language Sys- tem (UMLS), "AD" could refer Alzheimer's Dis- ease, Atopic Dermatitis, Actinomycin D, or Admit- ting Diagnosis.
Biomedical entity linking (BioEL) the process of connecting entities referenced in documents to entries in biomedical databases such as the Unified Medical Language System (UMLS) or Medical Subject Headings (MeSH). The study objective was to comprehensively evaluate nine recent state-of-the-art biomedical entity linking models under a unified frame- work. We compare these models along axes of (1) accuracy. (2) speed, (3) ease of use, (4) generalization, and (5) adaptability to new on- tologies and datasets. We additionally quantify the impact of various preprocessing choices such abbreviation detection. Systematic evaluation reveals several notable gaps in current methods. In particular, current methods struggle correctly link genes and proteins and often have difficulty effectively incorporat- ing context into linking decisions. To expedite future development and baseline testing, release our unified evaluation framework and all included models on GitHub https: / //gi thub. com/ dwvidkaviiakrrddr wwweeikktttteer/// bi omedi caw-llleeeic cectalleenttyy-e inkinng
Second, annotating a biomedical corpus is time- consuming task that requires specialized domain expertise, which have limited availability to label data. Concretely, the largest labeled BioEL dataset, MedMentions (Mohan and Li, 2019), covers ap- proximately 1% of the candidate entities its ref- erence ontology while annotating 0.17% of the ab- stracts in PubMed.
1 Introduction
Biomedical entity linking (BioEL) is the process of identifying biomedical concepts (e.g. diseases, chemicals, cell types, etc.) in text and connecting them to unique identifier in a knowledge base (KB). Entity linking (EL) is critical in text min- ing, as it allows concepts to be connected across disparate literature. This "harmonization" enables quick access to connected information in the knowl- edge base and allows for unified reasoning regard- ing diverse surface forms and mentions.
Third, though dozens of ontologies and termi- nologies have been curated in recent years, cepts are often not cross-referenced, leading to a lack of interoperability. Furthermore, even care- fully unified collections such as UMLS lack syn- onyms and definitions for the vast majority of con- cepts.
Most biomedical concepts are not labeled in any gold-standard EL corpus. Thus, robust zero-shot performance critical for effectively performing EL at scale. However, lack of labelled data by spe- cialized domain experts simultaneously makes it
These authors contributed equally to this work. Correspondence: david kartchnerrhhee @ gat

---

Model Comparison
$n0091
UMLS
SapBERT
ClsttrEL
AtboEL
MedMeniire
GNomPlus
+
Sci
Error Analysis
ScSpacy
MedLinker
Krissbeit
NLM Gene
NLM Chem NGBI
Robustness Reproducibility
Dall
BioGenEL
BioBART
MetaMap
Datasets
Models
Results
Figure 1: Overview of BioEL evaluation framework.
difficult accurately assess the capacity of current models to generalize unseen data.
Symbol
Definition
D
Corpus of documents
While some BioEL surveys have been published (French and McInnes, 2022), they do not evaluate models in consistent way or on a uniform collec- tion of datasets. Rather than traditional survey, we contend systematic evaluation of current BioEL models is needed to: I) accurately compare current models; 2) identify strengths and weaknesses; 3) prioritize directions for future research; 4) provide framework expedite future BioEL. development. To address these needs, this paper contributes the following:
d,
Individual document in corpus An entity mention in document i
my (+)
Ci
Left (right) context of entity mention ma
M
Collection of all entities in context
E
Database of entities
Individual entity
Table I: Notation used throughout paper
pus (c.g. clinical note, biomedical research ab- stract, etc.). Each document is annotated with mentions spans mij d,, where every mention span maj 11 ,.. is sequence of tokens corresponding
We release a synthesized collection of cur- rent BioEL models, which can be uniformly evaluated on a large collection of biomedical datasets.
single entity. Lvvrry mention given with surrounding contextual information Cjj and ctj- which correspond token spans be- fore and after the entity mention myj- Define the collection of contextual mentions for a document M, m, ) Subsequently, we discuss mentions within the context of single document and thus drop the document subscript i from men- tion and context annotations.
We present systematic framework to evalu- ate entity linking models along axes of scala- bility, adaptability, and zero-shot robustness (Section 5).
We conduct, knowledge, the largest and most comprehensive comparative evaluation of BioEI. models date.
We assume that database of entities is pro- vided & {ex ) Each entity is identified by unique identifier and may also contain informa- tional metadata such as entity type(s), definition, aliases, etc. Most entity-linkers assume access ground truth entity mention spans. However, these can be determined programmatically via named entity recognition algorithm.
We highlight strengths and pitfalls of current BioEL modeling techniques and suggest di- rections for future improvement (Section 7).
We provide our unified framework as open source repo to expedite future BioEL method development and baseline testing.
The task of entity linking is to learn a function f M + that maps each mention m, to the correct entity e, E.
2 Problem Definition
We assume that we are given a corpus D [ of text, where each d; is a document in the cor-
Most entity linkers use a two-stage approach to find the correct entity link for a given mention span.

---

The first stage Candidate Generation (CG), which defines a function fcG M - Ed that filters& down to a set of high-quality candidate entities. Once a set of entity candidates have been generated, they are passed into a Named Entity Disambiguation (NED) module fNED E" x M + E, which chooses the best candidate for final entity link. In practice, fcG is chosen to be computationally inexpensive algorithm with high recall, while FNED is more costly and precise. The final entity linker is defined as f = -NN ED fca-
used for entity linking. For methods able pro- cess additional metadata (ArboEL, ClusterEL), we add species information for each gene the entity description. For alias matching methods, we added the species name of each gene to its canonical name when the canonical name was not unique. We did not augment other aliases with species information.
3.2 Excluded Datasets
This evaluation focuses on entity linking in biomed- ical scientific research articles (BioEL). Therefore, this systematic evaluation excludes EL in non- scientific texts. Additionally, text extracted from electronic health records (EHR), such as notes or discharge summaries, are also excluded. EL for EHR is distinct from BioEL in its scope, purpose, and accessibility. Previous EHR EL efforts for informal, patient-generater text include CADEC (Karimi et al., 2015), AskAPatient (Limsopatham and Collier, 2016), and PsyTAR (Zolnoori et al., 2019). These EHR EL platforms link diseases, symptoms, and adverse drug reaction mentions a variety of relevant ontologies. Similarly, COMETA (Basaldella et al., 2020) links a diverse array of entities in Reddit posts to SNOMED-CT.
3 Datasets
We evaluate included BioEL methods on a variety of biomedical datasets (Table 2), with detailed de- scriptions of each in the Appendix A. All datasets used were taken from BigBio (Fries et al., 2022). Additionally, Table 10 in the appendix describes the extent which entities and mentions overlap between the training and testing data. Entity over- lap is defined as the proportion of entities in the testing data that are in the training data. Mention overlap represents the proportion of mentions in the testing data whose entity is present in the training data (e.g. if an entity is mentioned more than once in the test set).
Models
3.1 Data Preprocessing
A wide variety of methods have been used for BioEL. Here we describe families of models used for BioEL and list included models from each cate- gory. More detailed descriptions of each individual model are found in Appendix B. We summarize the different models evaluated in Table 3.
In order to simplify data processing, we pulled all included datasets from BigBio (Fries et al., 2022), recent effort to unify the format of biomedical text datasets for improved consistency and ease of use. Any bug and error fixes for included datasets were contributed directly to BigBio. For KBs, we down- loaded the KBs to which each database linked, namely UMLS (Bodenreider, 2004), McSH (Lip- scomb, 2000), Entrez Gene (Maglott et al., 2005). and the MEDIC dictionary (Davis et al., 2019), which contains disease entities from MeSH and OMIM (Hamosh et al., 2005). The KBs used for each dataset are listed in Table 2.
Models evaluated were those with near state-of- the-art performance at time of publication when evaluated on least one included BioEL entity linking dataset. From this pool, we excluded mod- els with no open-source implementation or whose implementation was rendered unusable due to lack of documentation or software updates. With the exception of MetaMap, all models were published in the past 5 years.
We removed any entity mentions whose Concept Unique Identifiers (CUIs) were no longer available in the corresponding ontology or remapped them to the updated CUIs when possible. We used Ab3P (Sohn et al., 2008) to identify and (optionally) re- solve abbreviations train/inference time.
4.1 Alias Matching EL
Alias based entity linking seeks to link entities by matching entity mention with correct entity alias in KB. The simplest form of this is exact string matching, but can be extended using any model that produces similarity scores between a mention and a set of candidate aliases. Evaluated alias matching methods include MetaMap (Aron-
In Entrez gene, we additionally dropped "IRNA" and "hypothetical protein" gene types that were not
"Note that some works focus only on NED and assume that candidates are given by an existing model.

---

Dataset
Num Docs
Mentions
Unique Ents
Ent Types
Doc Type
Ontology
MedMentions Full
4,392
352,496
34,724
127
PubMed Abstracts
UMLS
MedMentions ST2IPV
4.392
203,282
25,419
PubMed Abstracts
UMLS
BC5CDR
1,500
29,044
2,348
2
PubMed Abstracts
McSH
GNormPlus
533
6,252
1,353
2
PubMed A Abstracts
Entrez
NCBI Disease
792
6,881
789
PubMed Abstracts
MEDIC
NLM Chem
150
37,999
1,787
1
PMC Full-Text
McSH
NLM Gene
550
15,553
3,348
5
PMC Full-Text
Entrez
Table 2: Summary of datasets used for evaluation.
Characteristies
Requirements
Reproducibllity Code
Usubility
Model
Supervised
Type
Names
Definitions
Aliase
Pregronssiig
Model Source
Protrained Model
Docamentatiin
New Datawet
Medl
Yes
Contettraized
Yes
Yes
Yos
No
Yes
No
Fair
No
SelSpacy
Alia March
Yes
Optional
Yes
Excellent
Yes
ClusteEEL
Yes
Contexmaaizzd
Yes
Optional
Optional
Yes
Yes
No
Good
No
ArboFI
Contestsalieed
Yes
Optional
Optionl
Yes
KRISSBERT
Distant
Contestralizdd
Yes
Optional
Optional
No
Puttil
Yes
Good
No
BioSyn
Distant
Alias Masch
Yes
No
Yes
Yes
Yes
Yes
Good
No
SupRKRT
Distant
Match
You
No
Yes
Partial
BioBART
Yes
Autoregressive
Yes
No
Yes
No
Yes
Yes
Pooe
No
BioGenEL
Autenogressive
Yes
Table 3: Comparison of model characteristics, reproccciitttt....uciiiilltttttt and usability
and Lang, 2010), SciSpacy (Neumann al., 2019), BioSyn (Sung et al., 2020), and SapBERT (Liu et al., 2021). Note that BioSyn included via SapBERT since the latter is a higher-performing edition of BioSyn.
4.3 Autoregressive EL
First proposed by (Cao et al., 2021), autoregres- sive EL uses a generative language model to map the text of each mention to its canonical entity name, rather than identifying the index of the cor- rect database entity. It claims the potential to better accommodate additions to a database because an existing model can easily normalize to new entity names without needing to re-train final output layer. Autoregressive EL can also preform alias matching by training on alias table potentially reducing the need for hand-labeled training data. Our survey includes BioGenEL (Yuan al., 2022b) and BioBART (Yuan et al., 2022a).
4.2
Contextualized EL
Much of the work in transformer-based EL has built upon seminal works in zero-shot EL using semantic similarity between contextualized men- tions and entity descriptions (Logeswaran et al., 2019; Wu et al., 2020). These methods use entity description metadata to generate and disambiguate entity candidates without the use of alias tables or large-scale supervised mentions, making it easier generalize EL beyond the scope of training data. Wu et al. (2020) in particular uses pretrained BERT bi-encoder (Devlin et al., 2019) model to generate candidates by encoding similarity between mentions and descriptions. It then uses more ex- pensive cross-encoder model to disambiguate can- didates for the final entity link. Our evaluation includes MedLinker (Loureiro and Jorge, 2020), ClusterEL (Angell al., 2021), ArboEL (Agarwal al., 2022), and KRISSBERT (Zhang et al., 2021). We also note that Bootleg (Varma al., 2021: Orr et al., 2021) has been used for biomedical entity linking but do not include it due to lack code for configuring/running their published BioEL models.
5 Evaluation Strategy
As noted in (Zhang al., 2021), evaluation strate- gies between different entity linking papers are inconsistent, leading to wide disparities in reported results. Differences primarily revolve around how to score predictions where multiple normalizations are given for named entity, c.g. because all pre- dicted entities share the same alias. We identified three main strategies for this in the literature.
Basic resolves ties by randomly ordering all equally ranked entities.
Relaxed counts an entity link as correct any of the predicted normalizations match any of

---

the ground-truth normalizations for given entity.
Results
Our main result in Table 4 shows the recall @ (accuracy) and recall of each model across all of the datasets. This estimates how well models perform both on candidate ranking and overall can- didate generation. Here ArboEL. outperforms most models across the majority of datasets. An addi- tional visualization of how recall @k changes for for l .,...,, 10 is shown in Figure 2.
3. Striet counts normalization correct only if all predicted normalizations match ground- truth normalizations for a given entity. Same as basic if no equally ranked normalizations.
For each dataset, generate ranked entity can- didates from each model in Sec. 4. For models that only natively link to UMLS, links to other KBs are computed by predicting entities in UMLS (Bo- denreider, 2004) and mapping these predictions to other KBs using cross references provided by UMLS and OBOFoundary (Smith et al., 2007) Predictions are ranked and evaluated using recall @ k for k E (1,2.... ..... 10} (note that recall @ is equivalent accuracy). We perform main eval- uations using the basic evaluation strategy unless otherwise specified.
6.1 Performance on specific entity types
While most of the datasets evaluated contain only 1-2 entity types, MedMentions contains 127 dis- tinct entity types split into 10 semantic groups. Similarly, both NLM-Gene and GNormPlus link gene mentions from many different species. We compared whether models perform better on spe- cific semantic groups (MedMentions) or genes from specific species (NLM-Genee) The results are shown in Tables and 12 (Appendix) respectively.
5.1 Error Analysis
For errors in the dataset, we analyze the following:
6.2 Performance on entities with limited metadata
Stage of EL failure: For incorrectly linked men- tions, did the failure occur in CG or NED phase? For failures that occur in candidate generation phase, what proportion of generated candidates have the correct semantic type/semantic group?
We analyzed the models' performance on differ- ent data slices, as described in section 5.1. Linked entities are biased towards more commonly seen entities, which enables more robust extrapolation of model zero-shot performance and performance on entities with limited metadata (e.g. aliases, def- initions, etc). Results for MedMentions ST21PV are shown in Table
Failure subgroups: When model fails, can identify slices with high/low chances of failure? Inspired by Orr et al. (2021) and Chen al. (2021), we investigate possible failure modes including:
Discussion
Entity type. Are entities of particular types frequently linked incorrectly? Are generated candidates in the correct semantic type/group?
the models evaluated, there was no model that clearly performed "best" for all datasets or evalu- ation metrics. However, ArboËL showed consis- tently high performance and was always among the highest-performing models each dataset. SapBERT arguably the best-performing alias matching method, sometimes surpassing ArboEL in recall @ for various datasets.
Popularity. How often are incorrectly linked entities present in the training data?
Available metadata. Do incorrectly linked sur- face forms match aliases in the KB? Are KB entities with few aliases and/or no definition more likely to be incorrectly linked?
One noteworthy result is the relatively poor per- formance of all models in Table 4 on gene recogni- tion. For alias matching models we see significant increases in recall@k as k increases on both NLM- Gene and GNormPlus than we do for any other datasets. We hypothesize this due gene aliases being poorly differentiated between species. This is supported by the steeply increasing recall @k performance of autoregressive and diass-matching
Common Misunderstandingss There some where all models in our comparison find the incorrect entity link in our data. We manually ex- amined cases where all BioEI. models provided an incorrect entity link and describe common mistakes made by current BioEL models.

---

MM-ST21PV
GNu
NLM-Gene
1.0
recall@k
0.6
0.4
0.2
0.0
NLM-Chem
BC5CDR
NCBI-Disease
1.0
recall@k
0.6
0.4
0.2
2
8
10
6
8
10
4
6
10
k
K
SciSpacy MetaMap
MedLinker SapBERT
KRISSBERT
BioBART ClusterEL
ArboEL
BioGenEL
Figure Recall @ K for all models using basic evaluation.
BCSCDR
MM-Full
MM-ST21PV
GNormPlus
NLM-Chem
NLM-Gene
NCBI-Disease
5
5
5
5
SupBERT
0.883
0.934
0.611
0.786
0.637
0.788
0.234
0.614
0.812
0.889
0.075
0.348
0.753
0.896
MetaMap
0.828
0.588
0.731
0.624
0.680
0.261
0.263
0.712
KRISSBERT
0.735
0.755
0.701
0.079
0.560
0.279
0.482
SeiSpacy
0.780
0.830
0.582
0.759
0.572
0.741
0.471
0.772
0.467
0.503
0.163
0.349
0.680
0.780
MedLinker
0.720
0.767
0.568
0.662
0.521
0.627
0.178
0.469
0.514
0.542
0.084
0.255
0.545
0.768
ClusterEI.
0.876
0.938
0.696
0.851
0.692
0.849
0.302
0.448
0.758
0.868
0.490
0.676
0.748
0.801
ArboEL
0.921
0.958
NR
NR
0.747
0.890
0.441
0.524
0,828
0.882
0,543
0.734
0.774
0.832
BioBART
0.572
0.733
0.548
0.764
0.496
0.700
0.175
0.499
0.5
0.650
0.051
0.229
0.423
0.608
BioGenEL
0.909
0.953
0.567
0.763
0.520
0.691
0.081
0.281
0.786
0.879
0.043
0.233
0.518
0.692
Table 4: Recall@ (accuracy) and recall @ 5 of all models. NR=Not reproducible
Semuntic Group
SupBERT
MetaMap
KRISSBERT
SciSpacy
ClusterFi.
ArboEI.
BioBART
RioGenFi
0.083:
0.065:
0.026
0.0711
0.038!
0.0331
0.051
0.073!
0.202
Chemicals & Drugs
-0.027
-0.011
-0.103
0.007
-0,045
-0.034
-0,101
0.000
0,185
Procedures
.0.097
-0.133
0.018
-0.127
-0.019
-0.009
-0.099
-0.066
0.165
Living Beings
0.063
0.031
0.045!
0.043!
0.043
0.047!
0.100
0.053
0.099
Physiology
-0.044
0.060
0.0461
40.001
0.040
0.016
0.068
0.024
0.095
Concepts
0.060
-0.019
-0.014
-0.029F
-0.018
Anatomy
0.058
025
0.047!
0.073!
0.035!
0.0311
0.014
0.0591
0.082
Genes & Molecular Sequenses
-0.144
-0.098
-0.192!
-0.14
-0.152
-0.1291
00.133
-0.499
0.028
Other
.0.030*
0.027
.0.0391
0.008
.0.39
.0.032
.0.00
.0.112
0.055
Table 5: Performance on different semantic groups within MedMentions. Values represent absolute difference in
slice accuracy vs. overall accuracy for each model. "p<0.05; 1<00.01; /p<0.001 after Bonferroni correction.

---

Slice
SapBERT
MetaMap
KRISSBERT
SeiSpacy
ClusterEL.
ArboEL
BioBART
BioGenEL
Prevalence
is abbree
0.037
0.080F
-0,062
0.076
-0.023*
0.003
-0,038
0.023
0.091
bas match
0.280
0.289!
0.114
0.298
0,2051
0.194
0,064
0.161
0.157
no dlins_match
-0.052
-0.054
-0,021
-0.055
-0,038l
-0.036
-0.012
-0.030
0.843
wrong alias match
-02599
-0.213
-0129t
-0.175
-0.156
-0.150
-0.156*
-0.213+
0.081
train text march
0.094
0.082
0.2301
0.077
0.124!
0.0991
0.094
0,077
0.556
train train_catity catity_match match
0.0151
0.023|
0,163l
0.011
0.058!
0.046
0.037!
0.0171
0.774
single_aie alias
-00755
-0.117
-0.041
-0.148
0.005
-0.031
-0.116*
-0.133
0.096
five alias or Jess
-0.074
-0.085l
-0.055l
-0.085
-0.04l
-0.051
-0.056
-0,0791
0.448
no defdefinitinn
-0.101
-0.157!
-0.262
-0.126
-0.158!
-0.144|
-0.152
-0.1131
0.196
zero zero_shot shot
-0.0514
-0.08F
-0.559
-0.038F
-0,2004
-0.157
-0.128
-0.059t
0.226
Table 6: Performance differential of models on various slices of data, micro-averaged over all datasets. Values represent absolute difference in slice accuracy vs. overall accuracy for each model. "p<0.05; p<00011; $p<0.001 after Bonferroni correction.
Model
BCSCDR CG
MM-Full CG
MM-ST2IPV CG NED
GNormPlus CG
NLM-Chem CG
NLM-Gene
NCBI-Disease
NED
NED
NED
NED
CG
NED
CG
NED
SapBERT
0.552
0.448
0.462
0.538
0.546
0.454
0.058
0.942
0.511
0.489
0.141
0.853
0.257
0.743
MetaMap
0.836
0.164
0.640
0.360
0.682
0.318
0.976
0.024
0.914
0.086
0.996
0.004
0.868
0.132
KRISSBERT
0.860
0.140
0.541
0.459
0.628
0.372
0.991
0.009
0.894
0.106
0.668
0.332
0.744
0.256
SciSpacy
0.613
0,383
0,430
0.566
0,441
0.555
0.331
0.669
0.819
0.181
0.729
0.267
0.590
0.407
MedLinker
0.783
0.217
0.689
0.311
0.689
0,311
0.323
0.677
0.919
0.081
0.499
0,501
0.410
0.590
ClusterEL
0,310
0.688
0.297
0.698
0.292
0.703
0.669
0.324
0.399
0.599
0.475
0.519
0.620
0.380
ArboEL
0.403
0,597
NR
NR
0.275
0,722
0.780
0.219
0.536
0.464
0.477
0,521
0.677
0.323
BioBART
0.291
0.709
0.306
0.691
0.325
0.672
0.202
0.795
0.320
0.680
0.375
0.619
0.242
0.747
BioGenEL
0.308
0.692
0.353
0.644
0.417
0.582
0.510
0.481
0.324
0.676
0.358
0.639
0.449
0.544
Table Stage of model (CG or NED) at which entity linking failed. Values represent the proportion of errors that occurred in each stage. NR=Not reproducible
models, which cannot differentiate between multi- ple entities containing the same alias. Comparison to the recall@K k curves under a relaxed evaluation (Figure 11, Appendix) reveals that these models are excellent at finding the correct alias but lack the capacity to choose the correct entity from among them.
biguation. Other models tend have more evenly distributed errors, with failure stage being highly dataset dependent. Overall, these results indicate that substantial gains can be made to EL through work on both CG and NED.
7.2 Impact of Abbreviation Resolution
For datasets focusing on chemicals and diseases (BC5CDR, NCBI-Disease, NLM-Chem), curves comparing recall from 10 flatten out quickly; this result indicates that when the correct candidate is retrieved, it is generally ranked highly.
Abbreviation resolution (AR) is commonly used a means to potentially improve the performance of EL models. We investigated to what extent this is true by running each of the models with and with- out AR. The results, shown in Table 8, indicate that AR has a positive, statistically significant effect overall EL performance: AR improved perfor- mance by up to 69.5% on abbreviated entities in some datasets. However, this was not the case for gene normalization where AR showed negative or insignificant effect. We hypothesize this is because genes are more commonly referred to by their ab- breviations than by their longer full names, which limits the usefulness of AR.
7.1 Failure Stage
Most entity linking models consist of two stages, CG and NED. Therefore, it is useful see which stage each model failed. If a model is not choosing a set of candidates with the correct entity in the CG stage, the NED stage will never be able to choose the correct one. Table shows how errors are split between candidate generation and reranking for each model.
7.3 Robustness on Slices + Zero Shot
Failure stage varies widely by dataset and model. MetaMap and KRISSBERT tend to struggle most with candidate generation while BioBART and Bio- GenEL make most of their errors in entity disam-
In addition AR, we evaluated how models per- formed on different subsets of the data. Some com- mon entity characteristics, along with their perfor-

---

Overal
0.75
MODEL
0.7
dlle
0.70
CusteELL
02
0.65
GCRERT
0.0
MetetAap
eeGenEL
0:55
KRISSBERT
HHART
WORLL
Bacenti
Scpaacy
10"
Traning Tima
Figure 3: Performance on zero-shot, few alias, and unmatcheed/issmathhed test set instances, evaluated on MedMentions ST21PV.
Figure 4: Comparison of training time (s) vs. top-1 entity linking accuracy for BioEL, models. All experi- ments were performed on single NVIDIA A40 GPU.
mance, are shown in Table 6. A plot of performance in low-data slices (no/wrong alias match in training data; few aliases in KB; zero-shot performance) for MedMentions are shown in Figure 3. Unsur- prisingly, we see that the models have significantly improved performance on entities that match an alias in the target ontology: are in the training set; or have definitions. The models performed worse when the mention matches the alias of differ- ent entity; when the ground-truth entity does not have definition; and when only few aliases present for an entity in the ontology. We also see that performance degrades in zero-shot settings, but this degradation proportion seems lowest in alias matching models. Overall zero-shot performance is highest on ArboEL, followed by SapBERT.
most available research models for EL lacked some or all important elements of reproducibility. For example, surprising number of models lacked instructions on how to test their method dif- ferent dataset and many models had poor/outdated usage documentation. Some were missing critical details needed to reproduce reported experiments or to simply to run the baseline model. At the time of our evaluation, SciSpacy had the best documen- tation and use instructions. MedLinker, BioGenEL, and ArboEL were the most difficult to adapt and reproduce.
Future Work and Conclusion
8.1 Future Directions
Taken whole, these results indicate that "in the wild" entity linking performance will suffer for entities outside of the training distribution, but these effects can be mitigated by model choice.
Large language models (LLMs), such as GPT 3.5 (Ouyang et al., 2022), PaLM (Chowdhery et al., 2022), and BLOOM (Scao et al., 2022) have shown powerful few and zero-shot performance riety of tasks. However, these models are known to hallucinate and produce factually incorrect in- formation. To our knowledge, little work has been done to analyze how well these models can cor- rectly link entities, especially biomedical entities that may not be well represented within their train- ing distributions. An evaluation of --sed EL stands to improve the performance of BioEL mod- els while also improving the quality and accuracy of --nnerrte text.
va-
7.4 Scalability
Scalability is critical for deploying models in prac- tice. To measure the scalability of the models, we compared training and evaluation time on Med- Mentions. We compared training time in Figure 4 and evaluation time in Figure 5 (Appendix). When model came pretrained, we include the loading and/or dictionary embedding time as part of its training time. We generally found that simpler alias matching models tended to be faster than au- toregressive and contextualized models.
8.2 Conclusion
Entity linking an essential task for knowledge- intensive natural language processing and is partic- ularly in scientific and biomedical domains. This paper presents a systematic evaluation of BioEL
7.5 Usability, Adaptability, Reproducibility
We compared the usability and reprodueibility of models in Table 3. At the time of our evaluation,

---

Dataset
SapBERT
MetaMap
KrissBERT
SeiSpacy
ClusterEL
ArboEL
BCSCDR
0.59g1
0,588!
0.136!
0.6951
0.3291
0.263
MM-Fuull
0.426
0.4721
0.1421
0,408
0.181
NLA
MM-ST2IPV
0.398!
0,4541
0.131
0.403!
0.1871
0.198'
GNormPlus
0.039
0.004
0.019
-0.169t
-0.039
0.004
NLM-Chem
0.644
0.6021
0.0581
0.548
0.331
0.375
NLM-Gene
0.058
0.018
-0.003
0.003
-0.063
-0.087
NCBI-Dis
0.1391
0.468F
0.035
0,381
0.221
0.091
Overall
0,447f
0.464
0.095t
0.426
0.221
0,2271
Table 8: Absolute difference in accuracy for abbreviated entities after abbreviation resolution of abbreviation resolution. "p<0.05; p<0.01:; 1p<0.001 after Bonferroni correction.
models along axes of performance, scalability, us- ability, and robustness, enabling more principled, rigorous development and evaluation of future EL work.
National Institute of Health grant U19-AG056169 sub-award to C.M., the Morningside Center for Innovative and Affordable Medicine at Emory Uni- versity via the Brown Innovation to Market Fund to C.M., and by the Chan Zuckerberg Initiative grant 253558 to C.M.
Limitations
One limitation of our paper is a lack of extensive hy- perparameter tuning due to computing constraints. While we did perform early stopping on multiple methods to find the optimal amount of model train- ing, we did not perform an exhaustive hyperparam- eter search for the models listed. For most models, we followed the parameter choices listed by the authors in their respective papers.
References
Dhruv Agarwal, Rico Angell, Nicholas Monath, and Andrew McCallum. 2022. Entity linking via explicit coreference modeling. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Associa- tion for Computational Linguistics.
In addition to the general, multi purliip BioEL models included in this work, there are other mod- els designed to address specific entity types (e.g. genes, chemicals). Such models may be better able to deal with nuances of certain data types, such as species selection for gene/protein BioEL datasets. While these models could offer potential improvements on certain datasets and/or data slices, evaluating them is beyond the scope of this work.
Rico Angell, Nicholas Monath, Sunil Mohan, Nishant Yadav, and Andrew McCallum. 2021. Clustering- based inference for biomedical entity linking. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 2598-26686 2608, Online, Association for Computa- tional Linguistics.
Alan R Aronson and Francois-Michel Lang. 2010. An overview of metamap: historical perspective and re- cent advances. Journal of the American Medical Informatics Association.
KBs evolve over time with new discoveries and additional curation. While we performed signifi- cant manual efforts to identify and cither update or remove deprecated entity links within the datasets used, additional curation would be required to en- sure that every entity identifier properly aligns with the newer KB versions used when the original was unavailable.
Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020. Cometa: A corpus for medi- cal entity linking
social media. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNIP), pages 3122 3137, Online. Association for Computational Linguistics.
Finally, while there could be benefits from per- forming multi-task entity linking on combination of multiple datasets, exploring this option and the challenges associated with aligning multiple KBs is beyond the scope of this work.
Olivier Bodenreider. 2004. The unified medical lan- guage system (umls): integrating biomedical termi- nology. Nucleic acids research, D270.
D):D2
9 Acknowledgements
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representa- lions.
This research was funded by the National Science Foundation CAREER grant 1944247 to C.M. the

---

Anthony Chen, Pallavi Gudipati, Shayne Longpre, Xiao Ling, and Sameer Singh. 2021. Evaluating entity disambiguation and the role of popularity in retrieval- based NLP. In Proceedings of the 59th Annual Meet- ing of the Association for Compulational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Popers), pages 4472 4485, Online. Association for Computa- tional Linguistics.
Peng, David Cissel, Cathleen Coss, Carol Fisher, et al. 2021a. Nim-chem, a new resource for chemical en- tity recognition in pubmed full text literature. Scien- rific Data, 8-1):1112.
Rezarta Islamaj, Chih-Hsuan Wei, David Cissel, Nicholas Miliaras, Olga Printseva, Oleg Rodionov. Keiko Sekiya, Janice Ward, and Zhiyong Lu. 2021b. Nlm-gene, a richly annotated gold standard dataset for gene entities that addresses ambiguity and multi- species gene recognition. Journal of Biomedical In- formatics, 1 118: 18:103779.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311
Sarvnaz Karimi, Alejandro Metke-Jimenez, Madonna Kemp, and Chen Wang. 2015. Cadec: A corpus of ad- verse drug event annotations. Journal of biomedical informatics, 55:73-81.
Allan Peter Davis, Cynthia J Grondin, Robin J John- son, Daniela Sciaky, Roy McMorran, Jolene Wiegers, Thomas C Wiegers, and Carolyn J Mattingly. 2019. The comparative toxicogenomics database: update 2019. Nucleic acids research, (7(DI)::9488DD)
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-seqqeenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Dina Bemner-Fushmenn Willie J Rogers, and Alan R Aronson. 2017. Metamap lite: an evaluation of' a new java implementation of metamap. Journal of the American Medical Informatics Association, 24(4):841-84)
Jiao Li, Yueping Sun. Robin J Johnson, Daniela Sci- aky. Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas Wiegers, and Zhiyong Lu. 2016. Biocreative edr task corpus: resource for chemical disease relation extraction. Database, 2016.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186, Minneapolis, Minnesota. Association for Com- putational Linguistics.
Nut I Limsopatham and Nigel Collier. 2016. Normalis- ing medical concepts in social media texts by learn- ing semantic representation.
Proceedings of the 54th annual meeting of the association for compu- tational linguistics (volume 1: long papers), pages 1014-1023.
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Nebi disease corpus: a resource for dis- case name recognition and concept normalization. Journal of biomedical informatics, 47:1-10.
Carolyn E Lipscomb. 2000. Medical subject headings (mesh). Bulletin of the Medical Library Association, 88(3):265.
Evan French and Bridget T Mclnnes. 2022. An overview of biomedical entity linking throughout the years. Journal of Biomedical Informatics, page 104252.
Fangyu Liu, Ehsan Shareghi. Zaiqiao Meng. Marco Basaldella, and Nigel Collier. 2021. Self-alignment pretraining for biomedical entity representations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 4228-4238.
Jason Fries, Leon Weber, Natasha Seelam, Gabriel Al- tay. Debajyoti Datta, Samuele Garda, Sunny Kang, Rosaline Su, Wojeiech Kusa, Samuel Cahyawijaya, et al. 2022. Bigbio: A framework for data-centric biomedical natural language processing. Advances in Neural Information Processing Systems, 35:25792- 25806.
Lajanugen Logeswaran, Ming Wei Chang. Kenton Lee. Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity de- scriptions. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 3449-3460, Florence, Italy. Association for Computational Linguistics.
Ada Hamosh, Alan F Scott, Joanna S Amberger, Carol A Bocchini, and Victor A McKusick. 2005. Online mendelian inheritance in man (omim), a knowledge- base of human genes and genetic disorders. Nucleic acids research, 33(suppl_ [[ssupp_])::514444 1:D514-D517.
Daniel Loureiro and Alípio Mário Jorge. 2020. Medlinker: Medical entity linking with neural rep- resentations and dictionary matching. In European
Rezarta Islamaj. Robert Leaman, Sun Kim, Dongseop Kwon, Chih-Hsuan Wei, Donald C Comeau, Yifan

---

Conference on Information Retrieval, pages 230-237. Springer.
Chih- Wei, Hung- Yu Kao, and Zhiyong Lu. 2015. Gnormplus: an integrative approach for tagging genes, gene families, and protein domains. BioMed research international, 2015.
Donna Maglott, Jim Ostell, Kim D Pruitt, and Ta- tiana Tatusova. 2005, Entrez gene: gene-centered information at ncbi. Nucleic acids research,
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 6397-6407, Online. Association for Computa- tional Linguistics.
336suppl_):D544DD0
Sunil Mohan and Donghui Li. 2019, Medmentions: A large biomedical corpus annotated with UMLS con- cepts. In Ist Conference on Automated Knowledge Base Construction, AKBC 2019, Amherst, MA, USA, May 20-22, 2019.
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. ceedings of the 18th BioNLP Workshop and Shared Task. pages 319-327. Florence, Italy. Association for Computational I. Linguistics.
Hongyi Yuan, Zheng Yuan, Ruyi Gan. Jiaxing Zhang. Yutao Xie, and Sheng Yu. 2022a. BioBART: Pre- training and evaluation of a biomedical generative language model. In Proceedings of the 21st Work- shop on Biomedical Language Processing, pages 97- 109, Dublin, Ireland. Association for Computational Linguistics.
Pro-
Laurel J. OII, Megan Leszezynski, Neel Guha. Sen Wu. Simran Arora, Xiao Ling, and Christopher Ré. 2021. Bootleg: Chasing the tail with self-supervised named entity disambiguation. In 11th Conference on In- novative Data Systems Research, CIDR 2021, Vir- tual Event, January 11-15, 2021, Online Proceedings. www.cidrdb.org.
Hongyi Yuan, Zheng Yuan, and Sheng Yu. 2022b. Gen- erative biomedical entity linking via knowledge base- guided pre-training and synonyms-awar aware fine-tuning, In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 4038 4048, Seattle, United States. Association for Computational Linguistics.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Nau- mann, Jianfeng Gao, and Hoifung Poon. 2021. Knowledge-rich self-supervised entity linking. arXiv preprint arXiv: 2tt::2222..780 12.07887.
Teven Le Scao, Angela Fan, Christopher Akiki, Ell lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 1rxiv2221..01000
Maryam Zolnoori. Kin Wah Fung, Timothy B Patrick. Paul Fontelo, Hadi Kharrazi, Anthony Faiola, Ni- lay D Shah, Yi Shuan Shirley Wu, Christina E El- dredge. Jake Luo. al. 2019. The psytar dataset: From patients generated narratives to a corpus of adverse drug events and effectiveness of psychiatric medications. Data brief. 24:103838.
Barry Smith, Michael Ashburner, Cornelius Rosse, Jonathan Bard, William Bug, Werner Ceusters, Louis J Goldberg, Karen Eilbeck, Amelia Ireland, Christopher Mungall, et al. 2007. The obo foundry: coordinated evolution of ontologies to sup- port biomedical data integration. Nature biotechnol- ogy, (251
A Datasets
A.1 Additional Dataset Statisties
Table 9 presents key statistes about our datasets, particularly about the variety of mentions and ab- breviations seen in the datasets
Sunghwan Sohn, Donald C Comeau, Won Kim, and W John Wilbur: 2008. Abbreviation definition iden- tification based on automatic precision estimates. BMC bioinformatics, 91):1-10.
A.2 Dataset Descriptions
Mujeen Sung. Hwisang Jeon, Jinhyuk Lee, and Jaewoo Kang. 2020. Biomedical entity representations with synonym marginalization. In ACL.
Detailed descriptions of datasets included in our paper are as follows. Table 10 describes overlap of entities and mentions between the train and test sets.
Maya Varma, Laurel Oir, Sen Wu, Megan Leszezyn- ski, Xiao Ling, and Christopher Ré. 2021. Cross- domain data integration for named entity disambigua- tion in biomedical text. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2021, pages 4566 4575, Punta Cana, Dominican Republic. Association for Computational Linguistics.
MedMentions (MM) (Mohan and Li, 2019) is a collection of 4,392 randomly selected PubMed abstracts linked to the Unified Medical Language System (UMLS). Each abstract is comprehensively

---

Dataset
Total tions
Men-
Unique Mentions
Total Abbrevi-
Unique Abbre-
ations
viations
BC5CDR
29,018
5,915
2,811
388
GNormPlus
6,252
2,180
991
196
MM-Full
352,312
90,842
22,399
3,906
MM-ST21PV
203.185
65,947
18,701
3,398
NCBI Disease
6,881
2,136
1,611
143
NLM Gene
15,553
5,298
2,356
462
NLM Chem
37,999
4,706
8,684
372
Table Metadata for each dataset
annotated with all terms from UMLS, making Med- Mentions the largest and most comprehensive EL dataset containing span-level annotations. Due to the diversity of UMLS entity types, some cate- gories not particularly relevant to the majority of biomedical research (c.g. "Professional Group"). Accordingly, MM is most commonly evaluated on the ST21PV subset, which filters candidate entities to come from 18 high-quality ontologies and to fall under 21 semantic type groups.
Dataset
Ent. Overlap
Ment. Overlap
MedMentions Full
0.6199
0.8221
MedMentions ST21PV
0.5755
0.7741
BCSCDR
0,5300
0.7733
GNormPlus
0.0789
0,0838
NCBI Disease
0.6700
0,8156
NLM Chem
0.4747
0,6229
NLM Gene
0.4819
0.5408
Table 10: Overlap between entities train and test sets. Mention overlap refers to the proportion of mentions in the test set whose entities are in training set mentions.
Biocreative V CDR (BC5CDR) (Li et al., 2016) is subset of 1,500 abstracts with chemical and dis- ease annotations from the Comparative Toxicoge- nomies Database. Tagged diseases and chemicals are linked to the MeSH ontology.
A wide variety of methods have been used for BioEL. Here describe families of models used for BioEL and list included models from each cate- gory. Models evaluated were those with near state- of-the-art performance at time of publication when evaluated on at least one included BioEL entity linking dataset. From this pool, we excluded mod- els with open-soure implementation whose implementation rendered unusable due lack of documentation or software updates. With the exception of MetaMap, all models were published in the past 5 years. We summarize the different models evaluated Table 3.
GNormPlus (Wei et al., 2015) is a benchmark of 694 PubMed abstracts annotated with gene men- tions linked to the Entrez ontology of genes. It con- tains the BioCreative II gene mention (BC2BM) task subset and additional set of 151 anno- tated abstracts.
NLM Chem Corpus (Islamaj et al., 2021a) rep- resents the most diverse gold-standard chemical entity linking corpus. Chemical mentions in 150
PMC full-text articles are normalized to MeSH.
B.1 Alias Matching EL
NLM Gene Corpus (Islamaj al., 2021b) is a corpus of over 500 full-text articles with gene mentions linked Entrez gene.
SciSpacy (Neumann et al., 2019) SciSpacy is widely used, off-the-shelf library which offers a diversity of pipelines and models for identifying and linking entities in biomedical documents. SciS- pacy jointly performs named entity recognition and abbreviation detection for end-to-end EL. EL is performed using TF-IDF matching on character 3-grams of entity mentions.
NCBI Disease Corpus (Dogan et al., 2014) links disease mentions in PubMed abstracts to the NCBI disease ontology.
B Additional details included models
Here we provide additional details about the al- gorithms used by included models to supplement section
MetaiMap (Aronson and Lang, 2010) MetaMap is a tool developed by the National Library Medicine (NLM), first used 1994. uses nat-

---

ural language processing to map biomedical enti- ties to concepts in the Unified Medical Language System (UMLS) Metathesaurus. Input undergoes syntactic/lexical analysis, where candidate con- cepts and mappings are generated from phrases found. MetaMap's ss usage is highly configurable, both in processing and display options. Output can be shown excluding restricting semantic types, specific vocabularies, concept unique identifiers (CUIs), etc. Its generation of word variants thor- ough, and it is domain independent. On the other hand, MetaMap is limited to the English language. Computational speed is relatively slow, especially in the case where complex phrases are present.
as the graph- reranking of the candidates re- trieved by ArboEL's candidate retrieval biencoder (described below).
ArboEL (Agarwal et al., 2022) ArboLiL extends the work in ClusterEL by improving the scalability and training regimen of ClusterEL. While ArboEL uses bi-encoder similar (Wu et al., 2020), it also incorporates a training scheme based on a mention- mention similarity graph to identify hard negatives, which ultimately lead to better model precision.
KRISSBERT (Zhang et al., 2021) KRISSBERT presents a self-supervised framework for EL using contrastive learning on distantly supervised entity mentions. After distantly labeling a large number of potential entity links with the UMLS metathe- saurus, KRISSBERT learns a set of "prototypes" for each entity by training the model to separate mentions of different entities. They show that this can be extended additional fine-tuning by simply swapping noisy prototypes for supervised ones, which achieves per- formance on-par with the best supervised EL mod- els.
BioSyn (Sung et al., 2020) BioSyn performs EL by normalizing each mention surface form to the best alias seen at training time. does this via a combination of character-level sparse mention features and learned dense vector representations of each mention and entity, which are trained via an alias table such as the UMLS metathesaurus.
supervised setting without
SapBERT (Liu et al., 2021) SapBERT (for "self-alignment pretraining BERT") fine-tunes a BioBERT O) model to treat each alias of an en- tity equivalently and to map entity mentions to an alias contained in UMLS. Zhang et al. (2021) point out that SapBERT is unable to distinguish between aliases shared by multiple entities and returns all entities with an alias matching the normalized sur- face form.
B.2 Autoregressive EL
BioGenEL and BioBART (Yuan et al., 2022b,a) BioGenEL adapts BART (Lewis et al., 2020) to perform entity linking via sequence-to-sequence modeling. It is trained to generate the correct sur- face form for an entity mention. BioBART uses the same procedure to generate text but addition- ally provides BART model with a biomedical vocabulary and pre-trained on biomedical text.
Contextualized EL
MedLinker (Loureiro and Jorge, 2020) MedLinker was one of the first EL works evaluated on MedMentions. combines a BiLSTM model pre-trained on biomedical literature with approx- imate string matching from UMLS to conduct zero-shot EL (Mohan and Li, 2019).
C Framework
Our evaluation framework seeks to uniformly eval- uate biomedical entity linking datasets by using uniform protocols for 1) dataset processing, 2) on- tology processing, and 3) evaluation. All packages are implemented in python. We describe each com- ponent of our evaluation framework below.
ClusterEL (Angell et al., 2021) ClusterEL takes a unique approach to EL by treating linking as a supervised clustering problem. ClusterFL begins by creating an similarity graph of mentions within each document, which is then refined via edge re- moval until each cluster contains a maximum of one entity. This strategy has the dual benefit of jointly modeling EL with co-reference, enabling the NED model to compensate for failures that may occur in the candidate generation phase of EL. Since original implementation of ClusterEL has been merged into ArboEL, we evaluate ClusterEL
Our framework's 's dataset module builds on the BigBio framework (https: / /hugg i face. / /bigbio) by adding additional preprocessing to prepare entity linking datasets for effective mod- eling. It provides APIs for stitching passages into whole documents, deduplicating entity mentions, resolving abbreviations, removing deprecated enti- ties, and contextualizing mentions for modeling.

---

The ontology processing module of our frame- work enables different biomedical ontologies such as UMLS, Entrez, and others to be standardized to share common attributes. These attributes in- clude database identifier, semantic type(s), canoni- cal name, aliases, alternate IDs, descriptions, and other metadata such as species. Some of these on- tologies are very large with elements distributed across multiple files. Accordingly, we provide APIs for extracting relevant subsets, particularly from UMLS.
the repository; adding the flag to the MetaMap command would suffice to compare the results, For all datasets, using the relaxed model produced the best results. Four methods of evaluation were tested from toggling two options: 1) ranking mappings first, and/or 2) resolving abbreviations. In addition to candidate concepts, MetaMap generates mappings, which are groups of the most promising candidates. A key point of interest when evaluating MetaMap was seeing whether ranking mappings first would improve evaluation metrics over ranking candidates first. Another salient point was examining the impact of expanding abbre- viations. For example, the abbreviation for the chemical OCT can be expanded to 22-0xacalcitriol, which may improve MetaMap performance. The abbreviations within the datasets expanded from mappings for each PMID, and the expanded forms are added to the original text in each dataset. For each method, we selected the configuration of parameters that maximized recall at 1, which varied between ranking mappings first but almost always resolved abbreviations.
The evaluation portion of our framework enables straightforward evaluation of multiple entity link- ing models across multiple metrics. It creates a standardized format for model outputs well as evaluation pipeline that can compute different metrics across the various evaluation strategies de- scribed in the paper.
D Model Evaluation Details
D.1 MetaMap
A single-line delimited input text file was generated with the unique text mentions from each dataset. The metadata shown in Table 6. MetaMap's highly customizable nature means that many pa- rameters can be altered to see the impact on model performance. Six parameters were adjusted for each dataset: model year, semantic types, vocab- ularies, strict or relaxed model, and term process- ing ODemner-Fushman et al., 2017). Term process- ing was added with relaxed model runs, there significant difference between strict and re- laxed model performance otherwise. For each run, the NLM data version was used, which includes the full UMLS other than a select number of vo- cabularies (Demner-Fushman et al., 2017). The 2022AA version was used for all datasets except for MedMentions, as those were originally anno- tated with the 2017AA UMLS. MetaMap does not handle non-ASCII characters, input through Java file that replaces/removes non- ASCII characters. mapping was generated that keeps track of the terms that are altered, so evalua- tion can be done correctly.
E
Additional Results, Discussion, and Analysis
E.1
Runtime Comparison
In addition to training time, we also measured the evaluation time of each included model. The results comparing eval time and accuracy pictured in Figure 5.
0.75
AbDEL
Uslee
Accumay
0.85
EeRERT
pre-processed
3.90
MataMap
BeGnFL
KRSSRERT Spacy
GOBART
Evaluation Time
Figure 5: Comparison of evaluation time (s) vs. top-1 entity linking accuracy for
D.2 Evaluation Strategy for MetaMap
We performed grid search over multiple different MetaMap settings, including strict vs relaxed model, term processing, and with/without WSD. WSD did not provide significant improvements in model performance and is not included in
E.2 Relaxed Evaluation
We provide full results for the models evaluated under a relaxed evaluation strategy. A table of

---

BCSCDR
MM-Full
MM-ST2IPV
GNormPlus
NLM-Chem
NLM-Gene
NCBI-Discase
5
5
SapBERT
0.883
0.934
0.725
0.814
0.695
0.794
0.795
0.944
0.812
0.889
0.716
0.867
0.833
0.929
MetaMap
0.828
0.856
0.588
0.731
0.568
0.699
0.624
0.633
0.680
0.707
0.261
0.263
0.669
0.712
KRISSBERT
0.736
0.766
0.591
0.755
0.559
0.701
0.081
0.087
0.562
0.596
0.286
0.494
0.754
0.803
SeiSpacy
0.772
0.797
0.799
0.807
0.778
0,789
0.836
0.854
0.426
0.484
0.396
0.399
0.752
0.752
MedLinker
0.720
0.767
0.568
0.662
0.521
0.627
0.178
0.469
0.514
0.542
0.084
0.255
0.545
0.768
ClusterEL
0.876
0,938
0.851
0.302
0.448
0.758
0.868
0.490
0.676
0.748
0.823
ArboEL.
0.958
0.000
0.441
0.524
0.828
0.543
0.774
0.832
BioBART
0.572
0.733
0.662
0.800
0.544
0.711
0.696
0.847
0.512
0.650
0.521
0.714
0.457
0.689
BioGenEL
0,909
0.953
0.686
0.793
0.562
0.698
0.350
0.527
0.786
0.879
0,504
0.698
0.582
0.733
Table 11: Top-1 and top-5 accuracy of all models using relaxed evaluation.
MM-ST21PV
GNormPlus
NLM Gene
1.0
0.8
recall@k
0.6
0.4
0.2
NLM-Chem
BC5CDR
NCBI-Disease
1.0
0.8
recall@k
0.6
0.4
8
10
k
k
SciSpacy MetaMap
MedLinker SapBERT
KRISSBERT
BioBART ClusterEL
ArboEL
BioGenEL
Figure 6: Recall @ K for all models using relaxed evaluation.

---

results is given in Table 11 with corresponding plot of @ecall@k in Figure
E.3 Slice-specific Model Performance
Here we include additional data on the performance models on various data slices and entity types. Table 12 presents data on performance differentials for different species included in NLM-Gene.
E.4 Prediction Correlation
It is useful to know to what extent models make similar predictions to know how well they could be ensembled to improve overall results. We accord- ingly plot the correlation of whether the top-1 pre- dictions match each model. The results, pictured in Figure 7, indicate that models are generally some- what closely correlated, but differ substantially on gene datasets.

---

Taxonomy
SapBERT
MetaMap
KRISSBERT
SciSpacy
ClusterEL
ArboEL
BioBART
BioGenEL
Prevalence
Homo sapiens
-0.021
0.307
0.064
0.201
0.125t
0.107F
-0.099
-0.014
0.447
Mus musculos
-0.048
-0.246!
0.029
-0.162
-0.010
0.016
-0.40l
-0.031
0.351
Rattss norvegicus
0.075
-0.244
0.160
-0.163
-0.499
-03688
-0.06
000431
0.090
Saccharomyces verevisiae
0.046
-0.261#
-0.204
-0.163F
-0.256
-0.216
0.0691
0.039
Danio
0.490
-0.261
-0.316
-0.2251
0,573
0,551
Arahidopsis
0.601
-0.2611
0,361
Ovis aries
-0.038
-0.261"
-0.279
-0.163
-0.045
0.086
-0.014
-0.006
0.010
Caenorhabditis elegans
0.675
-0,261
0.021
-0.163
-0.190
0.157
0,549f
0,507
0.007
other
0.365
-0.261
-0.179
-0.163"
-0.410F
00.333
0.309
0.017
0.018
Table 12: Performance difference on genes of different species within NLM-Gene compared to overall performance. "p<0.05; 1p<0.01; 1pc0.001 after Bonferroni correction.
MM ST21PV
GhormPlus
NLM
1.000.55
0.55
K0 SSBERT
0:230.31
1.00
0.230
09000.05
0.07
0.16
0.1000
Sufowty
0.580.54
0.24
1.000.37
1.00
0.37
380
0.33
0.53
0.27 0.52
0.02
1.00
0.
0.07
00007.0.000
0.08 0.077.
0.11
0:73
0.16
1.00
0...8...99.000
0.101.1
McdLinkrr
0.36
0.32
1.00
0.30
0.29
0.35
0.17 0.21
0.040.14
0.40.143.00
0.04..044
09199......00
0.33
0.02
0.18
1.00
0.001.00.110110001
DkssteEL
0.37 32
0.200..0000000
0.29
1.00
0.97
0.09
0:03
0.25
0.000.01
0.070.04
0.340.00
0.23
16
0.38
0.190.17
1.000.90.22
0:202
Arbe
0.39 0.30
0.31
0.300
0.307.97
1.00
0.36
0.09 0.04
2.25
0.07
0.
0.97
0.000.15..0
0.28
0.20
0.94
1.00
0.30.23
BioBART
0.39 0.4
0.33
0.35
0.20
0.32
0.33
0.54
0.21
0.09
0.09
88
0.140.15
1:011.0000000
0.25
0.38
0.08
0.03
0.11
0.15
0.23
0.400.45
3.Gen.
1,58 0.300.5
0.28
1.00
0.13
0.08
0.07 0 0.08
0.09
1.00
0.03
0.0
0.
0.10
0.1
0.450.00
NLMI-Chon
BCSCOR
NCB Disease
SapBERT
0.00
0.140.4..2220
26..0.2.0000
0.510
0.52
1.00 0.50
0.02
0.32
0
0.26
0.260.220...
1.00
0.06
0.25
0.20
0.04
0.05
0.
MeeaMap
0.431.00..0
0.030
0.30
0.255.2..00
0.29
09
0.00
0.52
1.000.06
0.68
0.10
0.000......6.0
0.25
1.000.17
0:23
0.230.00
0.23
0.330.55000
0.87
0.13
005SBERT
0.140.08..00
0.000.33.000
0.30
0.30
0.
0.23
0.02
0.06
1.00
0.28
0
0.18
130.12
0.06
0.17
1.00
0.02
0.390.331
0.31
2.200.16
-02
Scieacy
0.260.38
0.330.0000
0.33
0.08
0.25
26
0.22
0.32 0..20.444
0.28
1.00
0.20
0.23
0.25
0.02
1.000
0.200.03300033
0.260.34
Medi Incer
0.200.22
0.09
0.00
0.000.000
0.15
0.11
0.09
0.05
0.10
0.10
11.00.100000
0.22
230
0900
1.00
0.000200.0000
260.29
Clsttr
0.15
0.30
0.08
0.33
0.20
0.200..44
0.5.0.5.. 0.33
0.35
0.26
0.266.25
1.0000
0.35
0.24 0.000.003 1.40
(0153881
Mod Hot
Cluterts
FRUSSSERI
Med =bos
Clutora
NUAEL
DIENRTT
BirCerEL
BeDART
BoGarEL
Figure 7: Correlation top-1 accuracy across datasets. Low and negative correlations indicate that models able
to correctly link distinct subsets of data.


Reviews:
[{"review_id": "sOoW3x7QRZ", "reviewer": "Reviewer_iWzH", "summary": "", "questions": "It would be important to described how the evaluation framework is structured. Were high level APIs and good practices from popular training/evaluation frameworks followed? Examples that come to mind are Hugging Face and Pykg2vec.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback on our paper.  We appreciate your recommendation to improve our paper by describing the presented framework.  We have provided an explanation of our framework below, which will be included as an appendix of our paper:\n\n\n*“Our evaluation framework seeks to uniformly evaluate biomedical entity linking datasets by using uniform protocols for 1) dataset processing, 2) ontology processing, and 3) evaluation.   All packages are implemented in python.  We describe each component of our evaluation framework below.*\n\n*Our framework’s dataset module builds on the BigBio framework (https://huggingface.co/bigbio) by adding additional preprocessing to prepare entity linking datasets for effective modeling.  It provides APIs for stitching passages into whole documents, deduplicating entity mentions, resolving abbreviations, removing deprecated entities, and contextualizing mentions for modeling.*\n\n*The ontology processing module of our framework enables different biomedical ontologies such as UMLS, Entrez, and others to be standardized to share common attributes.  These attributes include database identifier, semantic type(s), canonical name, aliases, alternate IDs, descriptions, and other metadata such as species.  Some of these ontologies are very large with elements distributed across multiple files.  Accordingly, we provide APIs for extracting relevant subsets, particularly from UMLS.*\n\n\n*The evaluation portion of our framework enables straightforward evaluation of multiple entity linking models across multiple metrics.  It creates a standardized format for model outputs as well as an evaluation pipeline that can compute different metrics across the various evaluation strategies described in the paper.”*\n\n\n**Additional comments:**\n\n***The work is focused on an important but specific topic (within a specific domain), which limits its audience (a biomedical NLP conference could be a more suited forum).***\n\n\nWhile biomedical entity linking is a specialized subdomain of NLP, entity linking is a knowledge-intensive task that is important to the usability and safety of more mainstream NLP technologies, e.g. the correct interpretation and response to medical questions posed to LLMs.  It is our hope that this will enable more entity linking papers to consider biomedical tasks alongside general domain tasks that all build on wikipedia.  \n\n\n\n***Figure 3 could be better presented as a radar chart.***\n\nWe believe that a bar plot better represents this data because of sensitivity of radar charts to ordering of variables.  A good example is shown here: https://twitter.com/Koen_VdE/status/1693610007177150721?t=smlerLuqcONEjaEO07NWsg&s=19 \n\n\n\nThank you again for your time to review our paper and provide valuable feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The evaluation framework could be a valuable tool to benchmark new models and/or extend the evaluation.\nThe paper presents a good summary of the state-of-the-art results in biomedical entity linking.", "reasons_to_reject": "The framework, which is potentially the most relevant contribution depending on how structured and friendly it is, is not described.\nThe work is focused on an important but specific topic (within a specific domain), which limits its audience (a biomedical NLP conference could be a more suited forum).", "questions_to_authors": "It would be important to described how the evaluation framework is structured. Were high level APIs and good practices from popular training/evaluation frameworks followed? Examples that come to mind are Hugging Face and Pykg2vec.\n", "paper_topic_and_main_contributions": "This paper presents a systematic assessment of several entity linking models in the biomedical domain. The work is supported by a \"unified evaluation framework\", as referred to by the authors, which is not described but which will be made publicly available. \nThe paper is well organised and the comprehensive evaluation is of interest to the topic.\nThe evaluation framework will \"expedite future BioEL method development and baseline testing\"; although no details are given on this framework, this could be a relevant contribution to researcher working on this topic.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 3 could be better presented as a radar chart."}, "extra_text": {}}, {"review_id": "B05dpkrrVh", "reviewer": "Reviewer_LFDp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a systematic framework to evaluate biomedical entity linking models along axes of scalability, adaptability, and zero-shot robustness. The authors analyze the strengths and pitfalls of these BioEL models and suggest directions for future improvement. The comprehensive evaluation for best practices is valuable in BioEL applications.", "reasons_to_reject": "The authors have addressed my major comments and added more details in this version. This version is improved.", "questions_to_authors": "", "paper_topic_and_main_contributions": "I have reviewed this paper in the ACL ARR 2023 April round. The authors have addressed my major comments and added more details in this version.\nThis paper presents a comprehensive evaluation of the existing models for the biomedical entity linking (BioEL) task. The authors classify nine popular BioEL models into three categories and evaluated them on the seven BioEL datasets. They also conduct an in-depth analysis of the experimental results.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aeeAwB9DjV", "reviewer": "Reviewer_wM6u", "summary": "", "questions": "A) What is the previous reported SOTA performance for each of the dataset used in the paper?  For instance, GNormPlus (https://www.hindawi.com/journals/bmri/2015/918710/tab2/) paper reports over 80% Recall, while your paper reports the highest recall@1 as 0.624 in Table 4?\n\nB) How did you choose the models to experiment ? Are they based on previous SOTA performance?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your feedback on our paper.  Below are comments to clarify your questions and concerns.\n\n\n_**The paper claims in the line 16-18, \"current methods struggle to correctly link genes and proteins and often have difficulty effectively incorporating context into linking decisions\", but the paper does not report the previously published SOTA results (much higher at least for gnormplus dataset https://www.hindawi.com/journals/bmri/2015/918710/tab2/) for each of the dataset. Hence, it is difficult to understand the context of \"current methods struggle to correctly link genes\" . Can you please clarify what you mean by this claim? For instance, is it that a single model / approach does not perform well across all datasets?**_\n\n\nThank you for mentioning this.  The table you referenced from the GNormPlus paper reports the performance of _“of **human species** gene normalization”_ **only** as per the figure caption.   Restricting to the subset of human genes is not a fair comparison, as models tend to perform substantially better on this subset (see Table 12 in the Appendix or our paper).  This is not a good representation of the performance of gene entity linking in general.  \n\nThe authors of the referenced GNormPlus mention this same point: “GNormPlus compares favorably to other state-of-the-art methods when evaluated on two widely used public benchmarking datasets, achieving 86.7% F1-score on the BioCreative II Gene Normalization task dataset and **50.1% F1-score** on the BioCreative III Gene Normalization task dataset”\n\nIn light of the above, the referenced claim our the abstract specifically means that current methods struggle to link gene mentions to the correct species.  Species information is generally not given in a gene mention and must be determined by the surrounding context.  \n\nWe have updated this sentence in the abstract to read \"most current methods struggle to link gene mentions to the correct species, often due to failure to incorporate mention context into linking decisions\"\n\n\n\n_**A) What is the previous reported SOTA performance for each of the dataset used in the paper? For instance, GNormPlus (https://www.hindawi.com/journals/bmri/2015/918710/tab2/) paper reports over 80% Recall, while your paper reports the highest recall@1 as 0.624 in Table 4?**_\n\nComparing SOTA across papers is not always straightforward due to differences in evaluation strategies, as described in Section 5 of our paper and in Zhang et al, 2021 (https://arxiv.org/pdf/2112.07887.pdf).  This can be easily seen by examining the difference between Figure 2 and Figure 6 (appendix), which show the same set of results evaluated under different strategies.  Changing the evaluation strategy can result in a performance difference of over 50 percentage points and dramatically shift the ordering of SOTA. This further underscores the importance of this paper to provide a clear comparison of different methods across a uniformly evaluated set of benchmarks.\n\nThus, one of the key purposes of our paper is to challenge the evaluations presented in previous papers and present a fair comparison of different methods.\n\nAs far as we can tell, there are no reported results for all species on the full GNormPlus dataset.  Previous methods have evaluated on the BioCreative II Gene Normalization (BC2GN) dataset, which is a subset of the GNormPlus dataset.  The full GNormPlus dataset includes BC2GN + 151 additional annotated abstracts.  As stated above, the paper you linked only reports recall on human genes.  \n\nTo clarify this point, we will add a table with the previously reported SOTA + our evaluated performance to the paper, especially highlighting when these differ due to differing evaluation strategies.\n\n_**B) How did you choose the models to experiment ? Are they based on previous SOTA performance?**_\n\nWith the exception of MetaMap, chosen were those with near SOTA performance on at least one BioEL dataset at time of publication, published in the past 5 years.  To clarify, we will add the following paragraph to Appendix B:\n\n_“Models evaluated were those with near state-of-the-art performance at time of publication when evaluated on at least one included BioEL entity linking dataset.  From this pool, we excluded models with no open-source implementation or whose implementation was rendered unusable due to lack of documentation or software updates.  With the exception of MetaMap, all models were published in the past 5 years.”_\n\n\nAdditionally we chose not to include the GNormPlus and other type-specific models for two reasons:  \n\n1. GNormPlus is only applicable to genes and is not relevant to other biomedical entity linking tasks\n2. The only version currently available for use is an integrated named entity recognition + entity linking pipeline. This is available via the PubTator API. \n\nFor the sake of completeness, we will add a table with the results of GNormPlus and other type-specific models included in the PubTator API.  \n\n_**Missing References:\nGNormPlus: An Integrative Approach for Tagging Genes, Gene Families, and Protein Domains**_\n\nThe reference you mentioned was already in the paper.  You can find it on lines 718-721.\n\n\nThank you again for the time taken to review our paper and provide feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is a survey  paper on bio entity linking, the experiments and error analysis are quite thorough.", "reasons_to_reject": "- The paper claims in the line 16-18, *\"current methods struggle to correctly link genes and proteins and often have difficulty effectively incorporating context into linking decisions\"*, but the paper does not report the previously published SOTA results (much higher at least for gnormplus dataset https://www.hindawi.com/journals/bmri/2015/918710/tab2/) for each of the dataset. Hence, it is difficult to understand the context of  *\"current methods struggle to correctly link genes** . Can you please clarify what you mean by this claim? For instance, is it that a single model / approach does not perform well across all datasets?", "questions_to_authors": "A) What is the previous reported SOTA performance for each of the dataset used in the paper?  For instance, GNormPlus (https://www.hindawi.com/journals/bmri/2015/918710/tab2/) paper reports over 80% Recall, while your paper reports the highest recall@1 as 0.624 in Table 4?\n\nB) How did you choose the models to experiment ? Are they based on previous SOTA performance?", "paper_topic_and_main_contributions": "- This paper comprehensively evaluates / analyses a set of 8-10 existing methods for Bio entity linking across multiple datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. GNormPlus: An Integrative Approach for Tagging Genes, Gene Families, and Protein Domains"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper's comprehensive evaluation of existing Biomedical Entity Linking models is agreed upon by all reviewers, that the framework is a potentially valuable tool benefiting the development of future BioEL methods.

Despite these strengths, the reviewers simultaneously acknowledged certain shortcomings. These include a lack of a precise explanation of the evaluation framework and the audience limitation due to the specific focus of the paper.

In summary, the paper is sound and offers significant insights regarding the BioEL task. However, it is recommended that the authors undertake a revision to address the issues and suggestions raised by the reviewers. Specifically, there is a need for additional clarity regarding some of their claims about current methods' struggles, as well as an accurate reporting of previously published SOTA results for every dataset. Further, a detailed outline of the evaluation framework is necessary.
Decision: Accept-Main