Efficient Biomedical Entity Linking: Clinical Text Standardization with Low-Resource Techniques (2024)

Abstract: Clinical text is rich in information, with mentions of treatment, medication and anatomy among many other clinical terms. Multiple terms can refer to the same core concepts which can be referred as a clinical entity. Ontologies like the Unified Medical Language System (UMLS) are developed and maintained to store millions of clinical entities including the definitions, relations and other corresponding information. These ontologies are used for standardization of clinical text by normalizing varying surface forms of a clinical term through Biomedical entity linking. With the introduction of transformer-based language models, there has been significant progress in Biomedical entity linking. In this work, we focus on learning through synonym pairs associated with the entities. As compared to the existing approaches, our approach significantly reduces the training data and resource consumption. Moreover, we propose a suite of contextbased and context-less reranking techniques for performing the entity disambiguation. Overall, we achieve similar performance to the state-ofthe-art zero-shot and distant supervised entity linking techniques on the Medmentions dataset, the largest annotated dataset on UMLS, without any domain-based training. Finally, we show that retrieval performance alone might not be sufficient as an evaluation metric and introduce an article level quantitative and qualitative analysis to reveal further insights on the performance of entity linking methods.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 493–505 August 16, 2024. ©2024 Association for Computational Linguistics
493

heading: 1 Introduction and Related Work
text: Medical text consists of a diverse vocabulary derived from various nomenclatures including varying surface forms corresponding to terms like diagnosis, treatment, medications, etc. This diversity poses a challenge for effective communication
*akshit.achara@gehealthcare.com †sanand.sasidharan@gehealthcare.com ‡gagan.n@gehealthcare.com
across medical institutions and organizations. One of the techniques to mitigate this inherent diversity present in multiple references to the same term is entity linking. Entity linking is used to map these references to standardized codes. These codes are curated and maintained by medical organizations for standardization of medical nomenclature.
Given a corpus, entity linking includes the mapping of a mention m which is a span of k words, to an entity ϵ, where the entity belongs to a knowledge base such as Wikipedia. In the biomedical domain, the textual phrases are linked with the corresponding concepts from a knowledge base constructed using the medical ontologies like UMLS (Bodenreider, 2004), SNOMED (ElSappagh et al., 2018), etc. The UMLS ontology comprises of a broad range of clinical entities along with rich information for each entity like synonyms, definitions, etc. Traditional approaches for entity linking, such as Support Vector Machines (Cristianini and Shawe-Taylor, 2000) and Random Forests (Breiman, 2001), rely heavily on hand-crafted features, thereby restricting generalization to diverse data. Neural networks have emerged as a prominent technique for entity linking due to their ability to learn semantic representations from textual data.
Alias matching based techniques like (Aronson, 2001; Neumann et al., 2019; Liu et al., 2020) have been proposed where an input mention is mapped to an alias associated with an entity in the knowledge-base. However, these techniques require large amount of training data. Contextualized entity linking approaches (Zhang et al., 2021) utilize the semantic similarity between contextualized mentions. This approach requires a list of entities in advance and includes distant-supervision on articles containing examples of these entities. Generating medical codes using large language models can be error prone (Soroush et al., 2024). In (Yuan et al., 2022b), the authors use a seq2seq model to map a
mention to its canonical entity name. This method is resource intensive and requires generation of synthetic examples for pretraining, utilizing entity definitions and synonyms. In (Kong et al., 2021), the authors propose a zero-shot entity linking approach by leveraging synonym and graph based tasks. However, the approaches require training samples from UMLS for both these tasks. Moreover, entity disambiguation has not been explored in the work.
Efficient student models like MiniLM (Wang et al., 2020) can be used to perform contrastive learning on synonyms of entities. This results in a significantly less embedding size (384) as compared to the approaches like SAPBERT (Liu et al., 2020) with an embedding size of 768. The predicted candidates in alias based techniques are ranked based on the cosine similarity score. However, there are ambiguous cases where multiple entities have similar scores for a common mention. Therefore, there is a requirement to disambiguate these candidates through reranking. CrossAttention based reranking approaches utilize supervised training on the concatenated mention and candidate representations as inputs (Zhang et al., 2021). More recent approaches utilize homonym disambiguation (Garda and Leser, 2024) and have shown to improve the performance of autoregressive approaches like GenBioEL.
In comparison to the discussed techniques, we propose an efficient and low resource zero-shot biomedical entity linking approach along with a suite of disambiguation techniques. Furthermore, we introduce an article level similarity analysis to obtain further insights. This also allows us to conduct a qualitative analysis without manually going through all the articles manually.
Our contributions are as follows:
• Data: We show that the impact of training is negligible on a finetuned MiniLM model1 as compared to the pretrained MiniLM model. Moreover, the pretrained MiniLM model when finetuned on all UMLS synonym pairs has worse performance than the allMiniLM model.
• Disambiguation We show that reranking on entity-level semantic information provided in UMLS can be highly effective for entity disambiguation. We further propose a parametric
1https://huggingface.co/ sentence-transformers/all-MiniLM-L6-v2
reranking technique that is beneficial for aliasbased entity linking solutions.
• Evaluation We propose a comprehensive evaluation of entity linking which utilizes the semantic representation of articles coupled with the strict matching and related matching of predicted and gold standard entities. This evaluation is used to highlight issues related to the annotation granularity, missing context and surface form bias (for abbreviations) without the need of going through all the articles.

heading: 2 Datasets
text: In this work, we explore entity linking on the Medmentions (Mohan and Li, 2019) dataset which consists of titles and abstracts from 4392 English biomedical articles. These articles comprise of textual spans annotated with mentions of UMLS 2017AA entities. The dataset provides two versions: a full version containing 34724 unique entities and an st21pv version with 25419 unique entities, the latter being recommended by the authors for information retrieval. Further details about the dataset versions are discussed in Table 9 in (Kartchner et al., 2023).

heading: 2.1 Preprocessing
text: We replace the abbreviations with their corresponding full forms using Ab3p (Sohn et al., 2008). The abbreviation expansion using Ab3p has shown to significantly improve the entity linking performance across different approaches (Kartchner et al., 2023). Prior to creating synonym pairs for training, we remove all the suppressed entities, deleted entities and deprecated entities. Some deprecated entities have also been merged with other entities having a synonymous relation. We map these deprecated entities to the corresponding active entities with a synonymous relation.
Some annotations in Medmentions (prepared with UMLS 2017AA) are deprecated in the UMLS
2023AB (see details in table 1). Therefore, approaches utilizing UMLS 2023AB version may want to use an updated version of Medmentions. Furthermore, the prototype space (feature vector space) consisting of UMLS entities will have to be updated to the remove deprecated entities. This would help in avoiding deprecated entities to be predicted as candidates.

heading: 3 Methodology
text: In this work, we create a prototype vector space comprising of the encodings (feature vectors) associated with the canonical name of each entity in the UMLS ontology. To obtain meaningful encodings for constructing this prototype space, we train an encoder-based transformer (Vaswani et al., 2017) model on pairs of canonical names of entity synonyms. This is similar to the training approaches utilized in (Kong et al., 2021) and (Liu et al., 2020). The prototype space constructed using this trained model is used for performing semantic search, where the query encoding is obtained by passing the mention through the same model. This step is known as candidate generation. The candidate generation may lead to ambiguous results where multiple predicted entities have equal similarity scores. This is addressed through the reranking approaches discussed in section 3.3. Finally, we utilize both semantic similarity and retrieval performance for our quantitaive and qualitative evaluation. The comprehensive structure of our proposed approaches is depicted in the figure 1.
The following sub-sections discuss the individual components used in our work:

heading: 3.1 Training
text: We construct a training dataset by taking all the canonical names for each entity from UMLS and create pairs of canonical names corresponding to the same entity. Each pair is of the form (ϵi, ϵ∗i ), where ϵ∗i represents the canonical name of a synonym of entity ϵi. The preprocessing steps are discussed in the section 2.1. We use this dataset to finetune a sentence-transformer (Reimers and Gurevych, 2019) model using Multiple Negatives Ranking loss (Henderson et al., 2017). We use MiniLM (Wang et al., 2020) which is a distilled version of BERTBASE model obtained using an effective knowledge distillation approach outperforming other lightweight models like TinyBERT and DistillBERT. We also utilize a finetuned all-
MiniLM2 model for training/finetuning on this dataset. The all-MiniLM model is obtained by training the MiniLM model on a 1B sentence pairs dataset using a contrastive learning objective. The corresponding MiniLM and all-MiniLM models trained/finetuned on k examples are hereafter referred as MiniEL∗k and MiniELk respectively. For example, the all-MiniLM model finetuned on 10 pairs/examples is referred as MiniEL10.
The Multiple Negatives Ranking loss function is defined as:
L(x, y, θ) = 1
B B∑ j=1 logP (yj |xj) (1)
Here, θ represents the network parameters, (x, y) represents a pair of phrases and B represents the batch size. The parameters details for training are provided in section in Appendix in the section A.1.

heading: 3.2 Candidate Generation
text: A prototype space is prepared for the UMLS 2017AA version comprising of the encodings of canonical names of each entity and its synonyms. These encodings are computed using the MiniEL* and MiniEL models. The prototype space is used for performing semantic search where the queries are formed using the labeled mentions from the Medmentions dataset. The top-k concepts are retrieved based on the cosine similarity of the query and entity encodings. These candidates are referred as generated candidates.

heading: 3.3 Disambiguation
text: The candidate generation solely relies on the cosine similarity score between the mention and prototype space candidate encodings. However, there may be cases where multiple candidates have similar scores or the scores alone may not be sufficient to rank the candidates. Therefore, there is a need to rerank the candidates. We propose the following reranking approaches that to perform the entity disambiguation:

heading: 3.3.1 Parametric Reranking
text: In this section, we propose a parametric approach to rerank the generated candidates. We consider three parameters based on the prototype space and our training framework for disambiguation namely, cosine similarity score (CSS), representative alias
2https://huggingface.co/ sentence-transformers/all-MiniLM-L6-v2
score (RAS) and the candidate entity frequency score (CEFS) as our parameters. The parameters have the corresponding coefficients a, b and c respectively. These parameters are used to compute a new ranking score for each candidate. The equation below shows the updated score (δ∗(., .)) computation for reranking each of the generated candidates.
δ∗(q, v) = a ∗ δ(q, v) + b ∗ 1 n n∑ j=1 δ(q, vj) + c ∗ n (2)
Here, q is the query encoding, v is a generated candidate encoding and n is the number of aliases of v in the generated candidates.
The optimal selection of coefficients a, b and c corresponding to each of these parameters is performed through a grid search on a subset of manually defined bounds. Further details on the grid search and the impact of the these coefficients are discussed in appendix in the section A.2.

heading: 3.3.2 With UMLS Semantic Information
text: UMLS comprises of additional classification associated with individual entities, grouping them based on their semantic types and semantic groups. Each semantic type and semantic group has a canonical name. In this section, we calculate the cosine similarity between the mention’s semantic type or semantic group canonical name encoding and the corresponding canonical names of the top-k candidates. This similarity score is added to the initial candidate generation score to rerank the top-k candidates.
1. Assuming Availability of Gold Standard Information: In this case, we assume that the gold standard semantic type and semantic
group information is available for each mention. We rerank the candidates by utilizing the following methods:
(a) Semantic Type Based Disambiguation: In this method, calculate the cosine similarity between canonical name encodings of semantic types of a mention and each of its top-k candidates. The updated score is computed as follows:
δ∗(q, v) = δ(q, v) + δ(TUI(q), TUI(v)) (3)
Here, TUI(.) maps the input mention/entity to the encoding of corresponding semantic type canonical names.
(b) Semantic Group Based Disambiguation: In this method, calculate the cosine similarity between canonical name encodings of semantic groups of a mention and each of its top-k candidates. The updated score is computed as follows:
δ∗(q, v) = δ(q, v) + δ(SG(q), SG(v)) (4)
Here, SG(.) maps the input mention/entity to the encoding of the corresponding semantic group canonical names.
2. Semantic Type/Group Prediction: In scenarios where the semantic type/group information of the mentions is not available, the methods proposed in (Le et al., 2022) and (Mao et al., 2023) can be used to predict the semantic type or group based on the input mentions. This can be followed by the computational methods discussed in the section 3.3.2.

heading: 4 Results and Discussion
text: We obtain the retrieval performance for the discussed approaches by considering the top-k closest candidates (that include aliases) from the prototype space. We observe that the retrieval performance (considering top-128 candidates) of all the miniEL and miniEL1000 approaches is around 87% for the st21pv version and 88% for the full version of the Medmentions dataset.

heading: 4.1 Quantitative Analysis
text: In this section, we present the quantitive analysis associated with candidate generation (see section 4.1.1 and tables 2, 3) and reranking (see section 4.1.2, figure 2 and tables 4). Furthermore, the intricate analysis on the distribution of exact, related and missed candidate matches are discussed in the section 4.1.3.
4.1.1 How much data do we need? In this section, we discuss the candidate generation performance of our approaches trained using varying number of examples. It can be seen that the performance of miniEL has a negligible training impact and the performance is stable across different number of examples (see tables 2 and 3). However, the miniEL∗ approach improves consistently with increasing number of training examples.The miniEL approach without any finetuning still outperforms the miniEL∗ approach trained on all the training examples.
In comparison, our approach outperforms generative methods like BioBART (Yuan et al., 2022a) and BioGenEL (Yuan et al., 2022b) that are resource intensive. Since these approaches use the Medmentions training set to finetune the models, we only compare the test set performance. The R@1 candidate generation performance of MiniEL
is 0.552 as compared to the overall performance of 0.496 and 0.520 of BioBART and BioGenEL respectively (the results are taken from (Kartchner et al., 2023)).

heading: 4.1.2 Reranking Performance
text: In the following subsections, we discuss the candidate reranking results. The results corresponding to the parametric approach and those corresponding to the semantic disambiguation approaches are discussed in the following subsections.
1. Parametric Reranking: The top-k candidates selected based on the parametric approach discussed in section 3.3.1 and the corresponding results are shown in figure 2 and
table 4. It can be seen that the retrieval performance improves by from 0.553 to 0.614 for the st21pv version and from 0.567 to 0.630 for the full version of Medmentions. The a, b and c values used to obtain these results have the proportion a : b : c ∝ 50 : 2 : 1 (see section A.2 for more details).
2. With UMLS Semantic Information: In this section, we discuss the retrieval performance improvements after the reranking using the semantic type and group information. The details of these methods are discussed in section 3.3.2.
Figure 2 and table 4 show the R@1 performance of the MiniEL0 model after applying these reranking strategies. The performance improves from 0.553 to 0.649 for semantic group and to 0.661 for semantic type reranking for the st21pv version of Medmentions. Similar observations can be made for the full version of Medmentions. Moreover, the retrieval performance does not deviate significantly with the increase in the top-k candidates used for reranking (see figure 2 for details).
The improvement in candidate ranking is approached in two ways. Firstly, to maximize the R@1 performance by reranking the generated candidates (see details in section 3.3) and secondly, to include context for addressing the context based ambiguity (see details in Appendix in section A.3).

heading: 4.1.3 How should the performance be evaluated?
text: In the retrieval-based evaluation strategy, we compute the retrieval performance on gold standard and predicted entity matches. However, there are cases where the most similar candidate is related to the gold standard entity. It can be seen in the table 5
that about 77% entities are exacting matching or are related to the gold standard entity. The details of each type of relation we have considered are provided by UMLS.3
Figure 3 shows that the effect of parametric reranking is directed primarily towards converting related matches to exact matches, coverting 36% of related matches into exact matches. The semantic group and semantic type based reranking approaches convert both missed and related matches into exact matches.
The following analysis is focused on the further evaluation of related and missed matches. In this article level analysis, we replace a mention with the closest generated candidate’s canonical name for each mention in the article where the closest candidate is a related match or a missed match respectively. This results in an article AP . We compute the cosine similarity between the original article A and the modified aricle AP called SP using a
3https://www.nlm.nih.gov/research/ umls/knowledge_sources/metathesaurus/ release/abbreviations.html#mrdoc_REL
PubmedBERT-base (Gu et al., 2020) model4 finetuned using sentence transformers (Reimers and Gurevych, 2019) on biomedical data. Similarly, we also replace the mentions with the gold standard canonical names to create an article AG. This is followed by computation of cosine similarity between A and AG called SG. We focus on scenarios where SP and SG deviate significantly as compared to the mean deviation of the articles. These are highlighed in the figure 4. This forms a base for our qualitative analysis where we use this deviation to provide insights on the granularity of gold standard predictions as well as highlight current issues in the approach.

heading: 4.2 Qualitative Analysis
text: We perform a qualitative analysis on the entity linking predictions to highlight the difference in the granularity of the gold standard and predicted entities.
In this section, we qualitatively evaluate the articles displayed in the regions A and B of figure 4. The region A consists of articles where the predicted article AP is semantically more similar to the original article A as compared to the gold standard articleAG. Whereas, the region B consists of articles where AG is more similar to A as compared to AP .
4https://huggingface.co/NeuML/ pubmedbert-base-embeddings
Table 6 shows the qualitiative examples from region A where it can be observed that our approach is penalized for granular or highly related predictions. For example, The mention gene expression classifier has a gold standard entity Research Activities as compared to the more granular prediction Gene Expression Profiling. Similarly, the mention cytoplasmic tails has a gold standard enttity CytoPlasmic as compared to the more granular prediction Cytoplasmic Domain.
Table 7 shows the qualitatuve examples corresponding to the region B where it can be seen that the gold standard annotation is based on the context of mention in the article. More specifically, the mention mice has a gold standard entity: Laboratory mice based on the article context. However, this context is missing in the mention surface form. Therefore, to address these kind of cases, we need to provide the necessary context in the query. We utilize three different disambiguation techniques and show examples of the corresponding predictions. We observe that additional context from the articles may result in granular predictions.
However, the results are highly sensitive to the context and overall retrieval performance drops significantly (see section A.3 for more details).
We also observe an inconsistency in the granularity of gold standard entities in these examples. The mention experimental mice has a gold standard entity Animals, Laboratory as compared to the more granular prediction Laboratory mice.

heading: 5 Conclusion
text: Biomedical entity linking has been an active area of research with various approaches being proposed to improve medical text standardization (see details in section 1). We propose a multi-stage approach where the first stage retrieves candidates with a high recall (∼ 87% for top-128 candidates). This is followed by application of the proposed reranking approaches focused on improving the R@1 retrieval performance. The reranking improves the performance by more than 10% (see figure 2 and table 4). We investigate the misses in R@1 and segregate the candidates into related and missed matches. Following this, we compute the article level semantic similarity together with the article level retrieval performance. This analysis highlights qualitative examples that can be used to obtain further insights about the framework. The semantic analysis is used to select the following types of qualitative examples: a) low retrieval performance and high similarity and, b) low retrieval performance and low similarity. The former can be highlight issues pertaining to granularity of gold standard entities and the latter can be used to highlight issues pertaining to the retrieval performance. Overall, the proposed techniques are highly effective in entity linking and have negligible training, prototype-space creation and inference costs (see
table 9 for more details).

heading: 5.1 Future Scope
text: We believe that there is a significant scope for future developments in biomedical entity linking across different components of existing deep learning solutions. Firstly, there can be multiple biomedical normalizations for a mention or surface form. However, there is no method to determine the "closeness" of a prediction to a surface form as opposed to the binary matching. We believe that there should be a partial scoring instead of a binarized computation in order to accomodate the quality of predictions in the evaluation. Moreover, semantic similarities can also determined by experts to provide a ranking that could be used across biomedical entity linking for disambiguation.

heading: 5.2 Limitations
text: We observe that while an abbreviation preprocessing module is utilized in the proposed approaches, it doesn’t convert all the abbreviations into their full forms. This causes a high amount of ambiguity in the results and often times the retrieval candidates do not consist of the correct entity. This drawback in positive pairs based learning has also been highlighted in (Zhang et al., 2021). Research addressed towards improving abbreviation expansion can help improve the recall of our candidate generation. Moreover, the region B in figure 4 highlights the examples where missing context in the surface form causes our framework to predict broader entities as the closest candidates. We utilize various approaches to include additional implicit and explicit context into our queries and analyze the corresponding retrieval performance (see details in Appendix section A.3).

heading: A Appendices
text: A.1 Terminology and Parameters
This section includes the terminology details and training, inference or other parameters used in this work.
Table 9 shows the memory consumption and carbon emissions associated with the MiniEL0 approach. It can be seen that our proposed techniques is low resource and results in very low amount of carbon emissions.
A.2 Ablation Studies
In this section, we discuss the influence of parameters used in the parametric disambiguation approach discussed in the section 3.3. Specifically, we consider the candidate generation results obtained by using the MiniEL0 model and perform the reranking by removing b and c parameters respectively. To highlight the impact of changing the a, b and c values, we perform a grid search on a manually selected range of values.
Furthermore, considering the top-10 candidates for reranking, removal of the parameter b results in an R@1 of 0.611, removal of c results in 0.481. This can be compared to the baseline R@1 0.553 and the R@1 of 0.614 obtained using optimal a,b
and c. The performance is computed on the st21pv version of Medmentions. Overall, the impact of parameter c is highly significant in the performance improvement.
A.3 Contextualized Queries
In our framework, the encoded representations of mentions are queried on the prototype space to get relevant candidates from UMLS. However, the mention spans alone may lack the necessary context to map the mention to their corresponding UMLS entities. In this section, we evaluate multiple techniques for incorporating context in the queries. Specifically, we use a running span based context addition, an implicit context addition and an attention based span context addition.
A.3.1 Neighboring Context In this approach, we select a few words before and after the mention span to update the mention m and encode the updated mention to form a query.
Firstly, we add 5 neighbouring words before and after the mention and observe that the retrieval performance drops drastically (R@1 ∼ 10%). Therefore, we the number of words to 2 on both sides of the mention which results in a drastic drop in retrieval performance (R@1 ∼ 22%).
Overall, this context addition approach results in a significant drop in our retrieval performance and may not be suitable for contextual disambiguation.
A.3.2 Attention-Based Context In this section, we perform experiments to identify the most influential words from the articles that
attend to the span in consideration. We modify the original mentions by adding these words as additional context. This is done by utilizing the attention mechanism of encoder based transformer models namely BioBERT (Lee et al., 2020). Firstly, the entire title and abstract text is tokenized and passed to these models. The corresponding attention outputs are obtained and passed to the mention enrichment algorithm.
Let k be the number of word-piece tokens obtained from the encoder model, for each head H of Layer L, the attention matrix can be mentioned as:
A =  a11 a12 ... a1k a21 a22 ... a2k ... ... .... ... ... ... .... ...
ak1 ak2 .... akk
 (5)
The mention spans lie in the range [c, d] where 0 ≤ c < d ≤ k. Therefore, the matrix A can be shortened to a submatrix of interest B mentioned as:
B =  acc ac(c+1) ... acd a(c+1)c a(c+1)(c+1) ... a(c+1)d ... ... .... ... ... ... .... ...
akc ak(c+1) .... akd
 (6)
Equivalently, B = [ bc b(c+1) ... bd ] (7)
where bi represents a column of B. Next, the token corresponding to the maximum attention value of each column is obtained as T (max(bi)) where T (j) represents the token at index j ∈ {1, 2, ..., k}in the text spanning from 1st to the kth token. The resulting token vector from the attention head Hm and Layer Ln is represented as: Rnm = [ T (max(bc) T (max(b(c+1)) ... T (max(bd)
] (8)
The ENRICH function discussed in the algorithm 1 return the enriched context for a given mention m, which is then modified as shown below:
m∗ = m : Rmn[0], Rmn[1] (9)
Finally, stop words are removed from Rmn[0] and Rmn[1]. An example mention cold can be modified as cold: severe,recent where, ’severe, recent’ is the added context.
Algorithm 1 Enrichment Context Selection procedure SORTmcbl(V : 1D vector) {most common by length in descending order}
C = {x | count(x) = max(count(T )) ∀ T ∈ V } C∗ = {x | x ∈ C and len(x) >= len(y) ∀ y ∈ C} return C∗
end procedure {Rn denotes the representative token from all attention heads in Layer n} {Ln denotes the representative token(s) from Layer n} {M denotes the representative token(s) for the tokenk in mention M} {E denotes the representative token context (E) for mention M} procedure ENRICH(Rn : 1D vector) {enrich mention with context}
C∗ = SORTmcbl(Rnm) Rn = C ∗ 1 or Rn = C∗(1) Ln = {R1, R2, ..., Rz} C∗ = SORTmcbl(Ln) Mt = {C∗1 , C∗2} M = {M1,M2, ...,Mk} C∗ = SORTmcbl(M) E = {C∗1 , C∗2}
end procedure
A.3.3 Implicit Context In this approach, we utilize mean-pooled embedding of the mention encodings taken from the entire article as an input. Firstly, the entire text is used as an input to obtain the tokenwise encodings from the model.
f(text, θ) = {ET1 , ET2 , ..., ETn} (10)
Here, ET is encoding of token T and n are the number of tokens in the input text.
Given a span s, consisting of l tokens and tokens in the span {Tk, ...., Tk+l}, we take the corresponding encodings from the model outputs {ETk , ..., ETk+l}. We perform a mean pooling on these encodings to obtain the updated query representation Q = 1l ∑k+l k {ETk , ..., ETk+l}. The prototype space consists of the sentence encodings of the canonical names of all the entities in UMLS.
The R@1 candidate generation performance drops drastically in this setup where a drop of more than 30% is observed. Overall, we observe that these implicit contextual queries are not helpful in improvement of retrieval performance.
A.3.4 Evaluation In this section, we perform the quantitative and qualitative analysis of our context based approaches on the Medmentions st21pv version. The qualitative examples shown below highlight the predictions provided by the proposed context based
0 approaches
on examples from Medmentions.
approaches. As discussed in the qualitative analysis of region B (see section 4.2), the surface forms have missing context resulting in an inaccurate prediction.
It can be observed in table 10 that the mention mice is correctly predicted as the entity Laboratory mice using the MiniELAC0 and MiniELNC reranking approaches. We also highlight the effect semantic type reranking approach though the example mentions kindlin-2 and kindlin-3 where the prediction semantic type changed from ’Gene’ to the correct type ’Protein’. Here, the MiniELNC0 , MiniELAC0 and MiniEL IC 0 methods correspond to the results obtained using the Neighboring Context, Attention-based Context and Implicit Context approaches, respectively, utilizing MiniEL0 as the base model.
It can be observed that the AC approach provides meaningful outputs as it includes the necessary context in the surface form. Similar outputs are provided by the NC approach. However, the neighbouring words may not necessarily contain the context and this can be seen in the following qualitative example listed in the table 11.
We observe that the attention span based context enrichment approach is sensitive to the context addition as it induces bias the surface form and the resulting candidates may be more similar to the bias term as compared to the base form. Therefore, to understand the impact of bias on the surface form, we observe the retrieval performance based on the number of words in the mention. The figure 6 shows that the performance of MiniELAC0 is better
0 approaches on examples
from Medmentions.
on mentions with higher length as compared to the mentions with lower lengths. A similar trend is observed for the MiniELNC0 approach. This trend is not seen for the MiniELIC0 approach where the performance drops with the increase in number of words in the mentions. However, the attention span based approach has better performance as compared to the neighboring context approach. For each specific mention word count, we select mentions with at least about a 100 examples for this
analysis. To summarize, the quantitative and qualitative context enrichment analysis shows that the MiniELAC0 approach outperforms the other approaches and is effective in context addition. However, the sensitivity in the encodings results in large deviations in the candidate generation (see table 12). Therefore, the robustness of this contextual approach needs to be improved.

heading: Future Work
text: We believe that there is a significant scope for future developments in biomedical entity linking across different components of existing deep learning solutions. Firstly, there can be multiple biomedical normalizations for a mention or surface form. However, there is no method to determine the "closeness" of a prediction to a surface form as opposed to the binary matching. We believe that there should be a partial scoring instead of a binarized computation in order to accomodate the quality of predictions in the evaluation. Moreover, semantic similarities can also determined by experts to provide a ranking that could be used across biomedical entity linking for disambiguation