Venue: NAACL 2021

# Self-Alignment Pretraining for Biomedical Entity Representations

Fangyu Liu\({}^{\clubsuit}\), Ehsan Shareghi\({}^{\diamondsuit,\spadesuit}\), Zaiqiao Meng\({}^{\clubsuit}\), Marco Basaldella\({}^{\lozenge}\), Nigel Collier\({}^{\clubsuit}\)

\({}^{\clubsuit}\)Language Technology Lab, TAL, University of Cambridge

\({}^{\diamondsuit}\)Department of Data Science & AI, Monash University \({}^{\heartsuit}\)Amazon Alexa

\({}^{\clubsuit}\){f1399, zm324, nhc30}@cam.ac.uk

\({}^{\diamondsuit}\)ehsan.shareghi@monash.edu\({}^{\heartsuit}\)mbbasald@amazon.co.uk

###### Abstract

Despite the widespread success of self-supervised learning via masked language models (MLM), accurately capturing fine-grained semantic relationships in the biomedical domain remains a challenge. This is of paramount importance for entity-level tasks such as entity linking where the ability to model entity relations (especially synonymy) is pivotal. To address this challenge, we propose SapBert, a pretraining scheme that self-aligning the representation space of biomedical entities. We design a scalable metric learning framework that can leverage UMLS, a massive collection of biomedical ontologies with +M+ concepts. In contrast with previous pipeline-based hybrid systems, SapBert offers an elegant one-model-for-all solution to the problem of medical entity linking (MEL), achieving a new state-of-the-art (SOTA) on six MEL benchmarking datasets. In the scientific domain, we achieve SOTA even without task-specific supervision. With substantial improvement over various domain-specific pretrained MLMs such as BioBert, SciBert and PubMedBert, our pretraining scheme proves to be both effective and robust.1

Footnote 1: For code and pretrained models, please visit: [https://github.com/cambridgeltl/saapbert](https://github.com/cambridgeltl/saapbert).

+
Footnote †: _Correspondedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4228–4238 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Biomedical entity2 representation is the foundation for a plethora of text mining systems in the medical domain, facilitating applications such as literature search Lee et al. (2016), clinical decision making Roberts et al. (2015) and relational knowledge discovery (e.g. chemical-disease, drug-drug and protein-protein relations, Wang et al. 2018). The heterogeneous naming of biomedical concepts poses a major challenge to representation learning. For instance, the medication _Hydroxychloroquine_ is often referred to as _Oxichlorochine_ (alternative name), _HCQ_ (in social media) and _Plaquenil_ (brand name).

Footnote 2: In this work, _biomechanical entity_ refers to the surface forms of biomedical concepts, which can be a single word (e.g. _fewer_), a compound (e.g. _sars-cov-2_) or a short phrase (e.g. _abnormal retinal vascular development_).

MEL addresses this problem by framing it as a task of mapping entity mentions to unified concepts in a medical knowledge graph.3 The main bottleneck of MEL is the quality of the entity representations Basaldella et al. (2020). Prior works in this domain have adopted very sophisticated text pre-processing heuristics D'Souza and Ng (2015); Kim et al. (2019); Ji et al. (2020); Sung et al. (2020) which can hardly cover all the variations of biomedical names. In parallel, self-supervised learning has shown tremendous success in NLP via leveraging the masked lan

Figure 1: The t-SNE Maaten and Hinton (2008) visualisation of UMLS entities under PubMedBert Bert pretrained on PubMed papers) & PubMedBert+SapBert (PubMedBert further pretrained on UMLS synonyms). The biomedical names of different concepts are hard to separate in the heterogeneous embedding space (left). After the self-alignment pretraining, the same concept’s entity names are drawn closer to form compact clusters (right).



objective to learn semantics from distributional representations Devlin et al. (2019); Liu et al. (2019). Domain-specific pretraining on biomedical corpora (e.g. BioBert, Lee et al.2020 and BioMegatron, Shin et al.2020) have made much progress in biomedical text mining tasks. Nonetheless, representing medical entities with the existing SOTA pretrained MLMs (e.g. PubMedBert, Gu et al.2020) as suggested in Fig. 1 (left) does not lead to a well-separated representation space.

To address the aforementioned issue, we propose to pretrain a Transformer-based language model on the biomedical knowledge graph of UMLS Bodenreider (2004), the largest interlingua of biomedical ontologies. UMLS contains a comprehensive collection of biomedical synonyms in various forms (UMLS 2020AA has 4M+ concepts and 10M+ synonyms which stem from over 150 controlled vocabularies including MeSH, SNOMED CT, RxNorm, Gene Ontology and OMIM).4 We design a self-alignment objective that clusters synonyms of the same concept. To cope with the immense size of UMLS, we sample hard training pairs from the knowledge base and use a scalable metric learning loss. We name our model as Self-aligning pretrained Bert (SupBert).

Footnote 4: [https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/release/statistics.html](https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/release/statistics.html)

Being both simple and powerful, SapBert obtains new SOTA performances across all six MEL benchmark datasets. In contrast with the current systems which adopt complex pipelines and hybrid components Xu et al. (2020); Ji et al. (2020); Sung et al. (2020), SapBert applies a much simpler training procedure without requiring any pre- or post-processing steps. At test time, a simple nearest neighbour's search is sufficient for making a prediction. When compared with other domain-specific pretrained language models (e.g. BioBert and SciBert), SapBert also brings substantial improvement by up to 20% on accuracy across all tasks. The effectiveness of the pretraining in SapBert is especially highlighted in the scientific language domain where SapBert outperforms previous SOTA even without fine-tuning on any MEL datasets. We also provide insights on pretraining's impact across domains and explore pretraining with fewer model parameters by using a recently introduced Adapter module in our training scheme.

## 2 Method: Self-Alignment Pretraining

We design a metric learning framework that learns to self-align synonymous biomedical entities. The framework can be used as both pretraining on UMLS, and fine-tuning on task-specific datasets. We use an existing Bert model as our starting point. In the following, we introduce the key components of our framework.

Formal Definition.Let \((x,y)\in{\cal X}\times{\cal Y}\) denote a tuple of a name and its categorical label. For the self-alignment pretraining step, \({\cal X}\times{\cal Y}\) is the set of all (name, CUI5) pairs in UMLS, e.g. (_Remdesivir_, C4726677); while for the fine-tuning step, it is formed as an entity mention and its corresponding mapping from the ontology, e.g. (_scratchy throat_, 102618009). Given any pair of tuples \((x_{i},y_{i}),(x_{j},y_{j})\in{\cal X}\times{\cal Y}\), the goal of the self-alignment is to learn a function \(f(\cdot;\theta):{\cal X}\rightarrow\mathbb{R}^{d}\) parameterised by \(\theta\). Then, the similarity \(\langle f(x_{i}),f(x_{j})\rangle\) (in this work we use cosine similarity) can be used to estimate the resemblance of \(x_{i}\) and \(x_{j}\) (i.e., high if \(x_{i},x_{j}\) are synonyms and low otherwise). We model \(f\) by a Bert model with its output [CLS] token regarded as the representation of the input.6 During the learning, a sampling procedure selects the informative pairs of training samples and uses them in the pairwise metric learning loss function (introduced shortly).

Footnote 5: In UMLS, CUI is the Concept Unique Identifier.

Footnote 6: We tried multiple strategies including first-token, mean-pooling, (CLS) and also nospec (recommended by Vulic et al.2020) but found no consistent best strategy (optimal strategy varies on different *Berts).

Online Hard Pairs Mining.We use an online hard triplet mining condition to find the most

Figure 2: The distribution of similarity scores for all sampled PubMedBert representations in a mini-batch. The left graph shows the distribution of **+** and **-** pairs which are easy and already well-separated. The right graph illustrates larger overlap between the two groups generated by the online mining step, making them harder and more informative for learning.

 

informative training examples (i.e. hard positive/negative pairs) within a mini-batch for efficient training, Fig. 2. For biomedical entities, this step can be particularly useful as most examples can be easily classified while a small set of very hard ones cause the most challenge to representation learning.7 We start from constructing all possible triplets for all names within the mini-batch where each triplet is in the form of \((x_{a},x_{p},x_{n})\). Here \(x_{a}\) is called _anchor_, an arbitrary name in the mini-batch; \(x_{p}\) a positive match of \(x_{a}\) (i.e. \(y_{a}=y_{p}\)) and \(x_{n}\) a negative match of \(x_{a}\) (i.e. \(y_{a}\neq y_{n}\)). Among the constructed triplets, we select out all triplets that violate the following condition:

Footnote 6: [https://github.com/jml/datasets/tree-](https://github.com/jml/datasets/tree-) 

figurations specified in App. SSB4). For other hyperparameters used, please view App. SSC.2.

Evaluation Data and Protocol.We experiment on 6 different English MEL datasets: 4 in the scientific domain NCBI, Dogan et al. (2014); BC5CDR-c and BC5CDR-d, Li et al. (2016); MedMentions, Mohan and Li (2018) and 2 in the social media domain COMETA, Basaldella et al. (2020) and AskAPatient, Limsopatham and Collier (2016). Descriptions of the datasets and their statistics are provided in App. SSA. We report Acc\({}_{@1}\) and Acc\({}_{@5}\) (denoted as \(@1\) and \(@5\)) for evaluating performance. In all experiments, SapBert denotes further pretraining with our self-alignment method on UMLS. At the test phase, for all SapBert models we use nearest neighbour search without further fine-tuning on task data (unless stated otherwise). Except for numbers reported in previous papers, all results are the average of five runs with different random seeds.

Fine-Tuning on Task Data.The red rows in Tab. 1 are results of models (further) fine-tuned on the training sets of the six MEL datasets. Similar to pretraining, a positive pair list is generated through traversing the combinations of mention and all ground truth synonyms where mentions are from the training set and ground truth synonyms are from the reference ontology. We use the same optimiser and learning rates but train with a batch size of 256 (to accommodate the memory of 1 GPU). On scientific language datasets, we train for 3 epochs while on AskAPatient and COMETA we train for 15 and 10 epochs respectively. For BioSyn on social media language datasets, we empirically found that 10 epochs work the best. Other configurations are the same as the original BioSyn paper.

### Main Results and Analysis

*Bert + SapBert (Tab. 1, top).We illustrate the impact of SapBert pretraining over 7 existing Bert-based models (*Bert = {BioBert, PubMedBert, ...}). SapBert obtains consistent improvement over all *Bert models across all datasets, with larger gains (by up to 31.0% absolute Acc\({}_{@1}\) increase) observed in the social media domain. While SciBert is the leading model before applying SapBert, PubMedBert+AspBert performs the best afterwards.

**SupBert vs. SOTA (Tab. 1, bottom).** We take PubMedBert+SpBert (w/wo fine-tuning) and compare against various published SOTA results (see App. SSC.1 for a full listing of 10 baselines)

\begin{table}
\begin{tabular}{l c c c c c c c c|c c c c} \hline \multirow{2}{*}{model} & \multicolumn{9}{c}{scientific language} & \multicolumn{4}{c}{social media language} \\ \cline{2-13}  & NCBI & BCSCDR-d & BC5CDR-c & MedMentions & AskPatient & \multicolumn{2}{c}{COMETA} \\ \cline{2-13}  & @1 & @5 & @1 & @5 & @1 & @5 & @1 & @5 & @1 & @5 & @1 & @5 \\ \hline vanilla BERT Devlin et al. (2019) & 67.6 & 77.0 & 81.4 & 89.1 & 79.8 & 91.2 & 39.6 & 60.2 & 38.2 & 43.3 & 40.4 & 47.7 \\  + SapBert & **916** & **95.2** & 92.7 & 95.4 & **96.1** & 98.0 & 52.5 & 72.6 & 68.4 & 57.6 & 59.5 & 76.8 \\ BioBERT Lee et al. (2020) & 71.3 & 84.1 & 79.8 & 92.3 & 74.0 & 90.0 & 24.2 & 38.5 & 41.4 & 51.5 & 35.9 & 46.1 \\  + SapBert & 910.4 & 94.7 & 93.3 & 95.5 & **96.6** & 97.6 & **83.0** & **73.7** & 72.4 & 89.1 & 63.3 & 77.0 \\  + SimpleBert Peng et al. (2019) & 75.7 & 87.2 & 83.2 & 91.0 & 87.7 & 94.1 & 41.6 & 61.9 & 41.5 & 48.5 & 42.9 & 52.9 \\  + SapBert & 90.9 & 84.0 & 93.4 & 96.0 & 86.7 & 89.2 & 48.9 & 73.1 & 72.4 & 89.1 & 66.0 & 78.8 \\  + Simbert Alsenter et al. (2019) & 72.1 & 84.5 & 82.7 & 91.6 & 75.9 & 88.5 & 43.9 & 54.3 & 43.1 & 51.8 & 40.6 & 61.8 \\  + SapBert & 91.1 & 95.1 & 90.9 & 95.7 & 106.6 & 97.7 & 51.5 & 73.0 & 71.1 & 88.8 & 64.3 & 77.3 \\  + SetBert & 91.7 & 95.2 & 93.3 & 95.7 & 96.6 & 98.0 & 50.1 & **73.9** & 72.1 & 88.7 & 64.5 & 77.5 \\  + UnifierNet Mistelopoulos et al. (2020) & 77.0 & 85.4 & 85.5 & 92.3 & 88.9 & 94.1 & 56.1 & 55.8 & 44.4 & 54.5 & 44.6 & 53.0 \\  + SapBert & 91.2 & 95.2 & 92.8 & 95.5 & 96.6 & 97.7 & 52.1 & 73.2 & 72.6 & 89.1 & 63.4 & 76.9 \\  + SupBert (Ga et al., 2020) & 77.8 & 86.9 & 93.8 & 93.0 & 94.6 & 43.9 & 64.7 & 42.5 & 49.6 & 46.8 & 53.2 \\  + SapBert & **92.0** & 95.6 & 93.5 & 96.0 & 96.5 & 98.2 & 50.8 & 74.4 & 70.5 & 88.9 & 65.9 & 77.9 \\ \hline supervised SOTA & 91.1 & 93.9 & 93.2 & 96.0 & 96.6 & 97.2 & OOM & OOM & 87.5 & - & 79.0 & - \\ PointDider & 77.8 & 86.9 & 89.0 & 93.8 & 93.0 & 94.6 & 43.9 & 64.7 & 42.5 & 49.6 & 46.8 & 53.2 \\  + SapBert & 92.0 & 95.6 & 93.5 & 96.0 & 96.5 & 98.2 & 50.8 & 74.4 & 70.5 & 88.9 & 65.9 & 77.9 \\  + SapBert & 91.5 & 95.8 & 93.6 & 96.3 & 96.3 & 96.5 & 98.0 & 50.7 & **75.0** & 67.5 & 87.1 & 64.5 & 74.9 \\  + SapBert (Adapter13\({}^{\star}\)) & 90.9 & 95.4 & **93.4** & **96.5** & 96.5 & 97.9 & **52.1** & 74.8 & 65.7 & 84.0 & 63.5 & 74.2 \\  + SapBert (Fun-tunned) & 92.3 & 95.5 & 93.2 & 95.4 & 96.5 & 97.9 & 50.4 & 73.9 & **89.1** & **96.2** & 75.1 (**81.1**) & 85.5 (**86.1**) \\ BioSyn & 91.1 & 93.9 & 93.2 & 96.0 & 96.6 & 97.2 & OOM & OOM & 82.6 & 87.0 & 71.3 & 77.8 \\  + Gitt. w/ SapBert & **92.5** & **96.2*** & 93.6 & 96.2 & **96.8 

which all require task supervision. For the scientific language domain, the SOTA is BioSyn(Sung et al., 2020). For the social media domain, the SOTA are Basaldella et al. (2020) and GenRank(Xu et al., 2020) on COMETA and AskAPatient respectively. All these SOTA methods combine Bert with heuristic modules such as tf-idf, string matching and information retrieval system (i.e. Apache Lucene) in a multi-stage manner.

Measured by Acc\({}_{@1}\), SapBert achieves new SOTA with statistical significance on 5 of the 6 datasets and for the dataset (BC5CDR-c) where SapBert is not significantly better, it performs on par with SOTA (96.5 vs. 96.6). Interestingly, on scientific language datasets, SapBert outperforms SOTA without any task supervision (fine-tuning mostly leads to overfitting and performance drops). On social media language datasets, unsupervised SapBert lags behind supervised SOTA by large margins, highlighting the well-documented complex nature of social media language (Baldwin et al., 2013; Limsopatham and Collier, 2015, 2016; Basaldella et al., 2020; Tutubalina et al., 2020). However, after fine-tuning on the social media datasets (using the MS loss introduced earlier), SapBert outperforms SOTA significantly, indicating that knowledge acquired during the self-aligning pretraining can be adapted to a shifted domain without much effort.

The Adapter Variant.As an option for parameter efficient pretraining, we explore a variant of SapBert using a recently introduced training module named Adapter(Houlsby et al., 2019). While maintaining the same pretraining scheme with the same SapBert online mining + MS loss, instead of training from the full model of PubMedBert, we insert new Adapter layers between Transformer layers of the fixed PubMedBert, and only train the weights of these Adapter layers. In our experiments, we use the enhanced Adapter configuration by Pfeiffer et al. (2020). We include two variants where trained parameters are 13.22% and 1.09% of the full SapBert variant. The Adapter variant of SapBert achieves comparable performance to full-model-tuning in scientific datasets but lags behind in social media datasets, Tab. 1. The results indicate that more parameters are needed in pretraining for knowledge transfer to a shifted domain, in our case, the social media datasets.

#### The Impact of Online Mining (Eq. (1)).

As suggested in Tab. 2, switching off the online hard pairs mining procedure causes a large performance drop in \(@1\) and a smaller but still significant drop in \(@5\). This is due to the presence of many easy and already well-separated samples in the mini-batches. These uninformative training examples dominated the gradients and harmed the learning process.

#### Integrating SapBert in Existing Systems.

SapBert can be easily inserted into existing Bert-based MEL systems by initialising the systems with SapBert pretrained weights. We use the SOTA scientific language system, BioSyn (originally initialised with BioBert weights), as an example and show the performance is boosted across all datasets (last two rows, Tab. 1).

## 4 Conclusion

We present SapBert, a self-alignment pretraining scheme for learning biomedical entity representations. We highlight the consistent performance boost achieved by SapBert, obtaining new SOTA in all six widely used MEL benchmarking datasets. Strikingly, without any fine-tuning on task-specific labelled data, SapBert already outperforms the previous supervised SOTA (sophisticated hybrid entity linking systems) on multiple datasets in the scientific language domain. Our work opens new avenues to explore for general domain self-alignment (e.g. by leveraging knowledge graphs such as DBpedia). We plan to incorporate other types of relations (i.e., hypernymy and hyponymy) and extend our model to sentence-level representation learning. In particular, our ongoing work using a combination of SapBert and Adapter is a promising direction for tackling sentence-level tasks.

## Acknowledgements

We thank the three reviewers and the Area Chair for their insightful comments and suggestions. FL is supported by Grace & Thomas C.H. Chan Cambridge Scholarship. NC and MB would like to

\begin{table}
\begin{tabular}{l l l} \hline \hline configuration & \(@1\) & \(@5\) \\ \hline Mining switched-on & **67.2** & **80.3** \\ Mining switched-off & 52.3\({}_{14.9}\) & 76.1\({}_{14.2}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: This table compares PubMedBert+SapBert’s performance with and without online hard mining on COMETA (zeroshot general).

 

acknowledge funding from Health Data Research UK as part of the National Text Analytics project.

## References

* Asentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
* Baldwin et al. (2013) Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013. How noisy social media text, how diffrnt social media sources? In _Proceedings of the Sixth International Joint Conference on Natural Language Processing (IJCNLP)_, pages 356-364, Nagoya, Japan. Asian Federation of Natural Language Processing.
* Basaldella et al. (2020) Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020. COMETA: A corpus for medical entity linking in the social media. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3122-3137, Online. Association for Computational Linguistics.
* Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3615-3620, Hong Kong, China. Association for Computational Linguistics.
* Bodenreider (2004) Olivier Bodenreider. 2004. The unified medical language system (UMLS): integrating biomedical terminology. _Nucleic Acids Research_, 32:D267-D270.
* Davis et al. (2019) Allan Peter Davis, Cynthia J Grondin, Robin J Johnson, Daniela Sciaky, Roy McMorran, Jolene Wiegers, Thomas C Wiegers, and Carolyn J Mattingly. 2019. The comparative toxicogenomics database: update 2019. _Nucleic Acids Research_, 47:D948-D954.
* Davis et al. (2012) Allan Peter Davis, Thomas C Wiegers, Michael C Rosenstein, and Carolyn J Mattingly. 2012. MEDIC: a practical disease vocabulary used at the comparative toxicogenomics database. _Database_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dogan et al. (2014) Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: a resource for disease name recognition and concept normalization. _Journal of Biomedical Informatics_, 47:1-10.
* Donnelly (2006) Kevin Donnelly. 2006. SNOMED-CT: The advanced terminology and coding system for eHealth. _Studies in health technology and informatics_, 121:279.
* D'Souza and Ng (2015) Jennifer D'Souza and Vincent Ng. 2015. Sieve-based entity linking for the biomedical domain. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP) (Volume 2: Short Papers)_, pages 297-302, Beijing, China. Association for Computational Linguistics.
* Goldberger et al. (2005) Jacob Goldberger, Geoffrey E Hinton, Sam T Roweis, and Russ R Salakhutdinov. 2005. Neighbourhood components analysis. In _Advances in Neural Information Processing Systems_, pages 513-520.
* Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing. _arXiv:2007.15779_.
* He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 9729-9738.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_, volume 97 of _Proceedings of Machine Learning Research_, pages 2790-2799. PMLR.
* Ji et al. (2020) Zongcheng Ji, Qiang Wei, and Hua Xu. 2020. BERT-based ranking for biomedical entity normalization. _AMIA Summits on Translational Science Proceedings_, 2020:269.
* Kim et al. (2019) Donghyeon Kim, Jinhyuk Lee, Chan Ho So, Hwisang Jeon, Minbyl Jeong, Yonghwa Choi, Wonjin Yoon, Mujeen Sung, , and Jaewoo Kang. 2019. A neural named entity recognition and multi-type normalization tool for biomedical text mining. _IEEE Access_, 7:73729-73740.
* Leaman and Lu (2016) Robert Leaman and Zhiyong Lu. 2016. TaggerOne: joint named entity recognition and normalization with semi-markov models. _Bioinformatics_, 32:2839-2846.
* Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyun Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model 

for biomedical text mining. _Bioinformatics_, 36(4):1234-1240.
* Lee et al. [2016] Sunwon Lee, Donghyeon Kim, Kyubum Lee, Jaehoon Choi, Seongsoon Kim, Minji Jeon, Sangrak Lim, Donghee Choi, Sunkyu Kim, Aik-Choon Tan, et al. 2016. BEST: next-generation biomedical entity search tool for knowledge discovery from biomedical literature. _PloS one_, 11:e0164680.
* Li et al. [2016] Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. _Database_, 2016.
* Limsopatham and Collier [2015] Nut Limsopatham and Nigel Collier. 2015. Adapting phrase-based machine translation to normalise medical terms in social media messages. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1675-1680, Lisbon, Portugal. Association for Computational Linguistics.
* Limsopatham and Collier [2016] Nut Limsopatham and Nigel Collier. 2016. Normalising medical concepts in social media texts by learning semantic representation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 1014-1023.
* Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Loshchilov and Hutter [2018] Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* van der Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. _Journal of machine learning research_, 9(Nov):2579-2605.
* Michalopoulos et al. [2020] George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen Chen, and Alex Wong. 2020. Umlsbert: Clinical domain knowledge augmentation of contextual embeddings using the unified medical language system metathesaurus. _arXiv preprint arXiv:2010.10391_.
* Mohan and Li [2018] Sunil Mohan and Donghui Li. 2018. MedMentions: A large biomedical corpus annotated with UMLS concepts. In _Automated Knowledge Base Construction_.
* Song et al. [2016] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. 2016. Deep metric learning via lifted structured feature embedding. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 4004-4012.
* von den Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _arXiv preprint arXiv:1807.03748_.
* Peng et al. [2019] Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In _Proceedings of the 2019 Workshop on Biomedical Natural Language Processing_, pages 58-65.
* Pfeiffer et al. [2020] Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7654-7673, Online. Association for Computational Linguistics.
* Phan et al. [2019] Minh C Phan, Aixin Sun, and Yi Tay. 2019. Robust representation learning of biomedical names. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3275-3285.
* Roberts et al. [2015] Kirk Roberts, Matthew S Simpson, Ellen M Voorhees, and William R Hersh. 2015. Overview of the trec 2015 clinical decision support track. In _TREC_.
* Schroff et al. [2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 815-823.
* Schumacher et al. [2020] Elliot Schumacher, Andriy Mulayr, and Mark Dredze. 2020. Clinical concept linking with contextualized neural representations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8585-8592.
* Shin et al. [2020] Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani. 2020. BioMegatron: Larger biomedical domain language model. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4700-4706, Online. Association for Computational Linguistics.
* Sun et al. [2020] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. 2020. Circle loss: A unified perspective of pair similarity optimization. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 6398-6407.
* Sung et al. [2020] Mujeen Sung, Hwisang Jeon, Jinhyuk Lee, and Jaewoo Kang. 2020. Biomedical entity representations with synonym marginalization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 3641-3650, Online. Association for Computational Linguistics.
* Tutubalkina et al. [2020] Elena Tutubalkina, Artur Kadurin, and Zulfat Miftathutdinov. 2020. Fair evaluation in concept normalization: a large-scale comparative analysis for berts-based models. In _Proceedings of the 28th International Conference on Computational Linguistics (COLING)_ 

* Tutubbalina et al. (2018) Elena Tutubalina, Zulfat Miftahutdinov, Sergey Nikolenko, and Valentin Malykh. 2018. Medical concept normalization in social media posts with recurrent neural networks. _Journal of Biomedical Informatics_, 84:93-102.
* Vulic et al. (2020) Ivan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavas, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7222-7240, Online. Association for Computational Linguistics.
* Wang et al. (2019) Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. 2019. Multi-similarity loss with general pair weighting for deep metric learning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 5022-5030.
* Wang et al. (2018) Yanshan Wang, Sijia Liu, Naveed Afzal, Majid Rastegar-Mojarad, Liwei Wang, Feichen Shen, Paul Kingsbury, and Hongfang Liu. 2018. A comparison of word embeddings for the biomedical natural language processing. _Journal of Biomedical Informatics_, 87:12-20.
* Wright et al. (2019) Dustin Wright, Yannis Katsis, Raghav Mehta, and Chun-Nan Hsu. 2019. Normco: Deep disease normalization for biomedical knowledge base construction. In _Automated Knowledge Base Construction_.
* Xu et al. (2020) Dongfang Xu, Zeyu Zhang, and Steven Bethard. 2020. A generate-and-rank framework with semantic type regularization for biomedical concept normalization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8452-8464.

## Appendix A Evaluation Datasets Details

We divide our experimental datasets into two categories (1) scientific language datasets where the data is extracted from scientific papers and (2) social media language datasets where the data is coming from social media forums like Reddit.com. For an overview of the key statistics, see Tab. 3.

### Scientific Language Datasets

NCBI disease Dogan et al. (2014)isa a corpus containing 793 fully annotated PubMed abstracts and 6,881 mentions. The mentions are mapped into the MEDIC dictionary Davis et al. (2012). We denote this dataset as "NCBI" in our experiments.

BC5CDR Li et al. (2016)consists of 1,500 PubMed articles with 4,409 annotated chemicals, 5,818 diseases and 3,116 chemical-disease interactions. The disease mentions are mapped into the MEDIC dictionary like the NCBI disease corpus. The chemical mentions are mapped into the Comparative Toxicogenomics Database (CTD) Davis et al. (2019) chemical dictionary. We denote the disease and chemical mention sets as "BC5CDRd" and "BC5CDRc-ci" respectively. For NCBI and BC5CDR we use the same data and evaluation protocol by Sung et al. (2020).11

Footnote 11: [https://github.com/dmis-lab/BioSyn](https://github.com/dmis-lab/BioSyn)

MedMentions Mohan and Li (2018)is a very-large-scale entity linking dataset containing over 4,000 abstracts and over 350,000 mentions linked to UMLS 2017AA. According to Mohan and Li (2018), training TaggerOne Leaman and Lu (2016), a very popular MEL system, on a subset of MedMentions require >900 GB of RAM. Its massive number of mentions and more importantly the used reference ontology (UMLS 2017AA has 3M+ concepts) make the application of most MEL systems infeasible. However, through our metric learning formulation, SapBert can be applied on MedMentions with minimal effort.

### Social-Media Language Datasets

AskAPatient Limsopatham and Collier (2016)includes 17,324 adverse drug reaction (ADR) annotations collected from askapatient.com blog posts. The mentions are mapped to 1,036 medical concepts grounded onto SNOMED-CT Donnelly (2006) and AMT (the Australian Medicines Terminology). For this dataset, we follow the 10-fold evaluation protocol stated in the original paper.12

Footnote 12: [https://zenodo.org/record/55013](https://zenodo.org/record/55013)

COMETA Basaldella et al. (2020)is a recently released large-scale MEL dataset that specifically focuses on MEL in the social media domain, containing around 20k medical mentions extracted from health-related discussions on reddit.com. Mentions are mapped to SNOMED-CT. We use the "stratified (general)" split and follow the evaluation protocol of the original paper.13

Footnote 13: [https://www.siphas.org/corpus](https://www.siphas.org/corpus)

## Appendix B Model & Training Details

### The Choice of Base Models

We list all the versions of Bert models used in this study, linking to the specific versions in Tab. 5. Note that we exhaustively tried all official variants of the selected models and the best performing ones are chosen. All Bert models refer to the BertBase architecture in this paper.

 



### Comparing Loss Functions

We use COMETA (zeroshot general) as a benchmark for selecting learning objectives. Note that this split of COMETA is different from the stratified-general split used in Tab. 4. It is very challenging (so easy to see the difference of the performance) and also does not directly affect the model's performance on other datasets. The results are listed in Tab. 6. Note that online mining is switched on for all models here.

The cosine loss was used by Phan et al. (2019) for learning UMLS synonyms for LSTM models. The max-margin triplet loss was used by Basaldella et al. (2020) for training MEL models. A very similar (though not identical) hinge-loss was used by Schumacher et al. (2020) for clinical concept linking. InfoNCE has been very popular in self-supervised learning and contrastive learning (Oord et al., 2018; He et al., 2020). Lifted-Structure loss (Oh Song et al., 2016) and NCA loss (Goldberger et al., 2005) are two very classic metric learning objectives. Multi-Similarity loss (Wang et al., 2019) and Circle loss (Sun et al., 2020) are two recently proposed metric learning objectives and have been considered as SOTA on large-scale visual recognition benchmarks.

### Details of Adapters

In Tab. 7 we list number of parameters trained in the three Adapter variants along with full-model-tuning for easy comparison.

\begin{table}
\begin{tabular}{l c} \hline \hline loss & @\(1\) & @\(5\) \\ \hline cosine loss (Phan et al., 2019) & 55.1 64.6 \\ max-margin triplet loss (Basaldella et al., 2020) & 64.6 74.6 \\ NCA loss (Goldberger et al., 2005) & 65.2 77.0 \\ Lifted-Structure loss (Oh Song et al., 2016) & 62.0 72.1 \\ InfoNCE (Oord et al., 2018; He et al., 2020) & 63.3 74.2 \\ Circle loss (Sun et al., 2020) & 66.7 78.7 \\ \hline \multicolumn{2}{c}{Multi-Similarity loss (Wang et al., 2019)} & **67.2 80.3** \\ \end{tabular}
\end{table}
Table 6: This table compares loss functions used for SapBert pretraining. Numbers reported are on COMETA (zeroshot general).

\begin{table}
\begin{tabular}{l r r r r r r r r r r r} \hline \hline  & NCBI & BC5CDR-d & BC5CDR-c & MedMentions & AskAPatient & COMETA (z.g.) & COMETA \\ \hline \multirow{2}{*}{
\begin{tabular}{l} Ontology \\ \(C_{\text{searched}}\subset\mathcal{C}_{\text{ontology}}\)? \\ |\(\mathcal{C}_{\text{swatched}}\) \\ |\(\mathcal{C}_{\text{swatched}}\) \\ |\(\mathcal{C}_{\text{swatched}}\) \\ |\(\mathcal{C}_{\text{swatched}}\) \\ |\(\mathcal{M}_{\text{train}}\) | & 5,134 & 4,182 & 5,203 & 282,091 & 15,665.2 & 13,489 & 14,062 \\ \(|\mathcal{M}_{\text{validation}}|\) & 787 & 4,244 & 5,347 & 71,062 & 792.6 & 2,176 & 1,958 \\ \(|\mathcal{M}_{\text{test}}|\) & 960 & 4,424 & 5,385 & 70,405 & 866.2 & 4,350 & 3,995 \\ \hline \hline \end{tabular}
\end{table}
Table 3: This table contains basic statistics of the MEL datasets used in the study. \(\mathcal{C}\) denotes the set of concepts; \(\mathcal{S}\) denotes the set of all surface forms / synonyms of all concepts in \(\mathcal{C}\); \(\mathcal{M}\) denotes the set of mentions / queries. COMETA (s.g.) and (z.g.) are the stratified (general) and zeroshot (general) split respectively.

\begin{table}
\begin{tabular}{l l l l l l l l l l l l} \hline \hline \multirow{2}{*}{model} & \multicolumn{2}{c}{NCBI} & \multicolumn{2}{c}{BC5CDR-d} & \multicolumn{2}{c}{BC5CDR-c} & MedMentions & AskAPatient & COMETA \\ \cline{2-13}  & \(@1\) & \(@5\) & \(@1\) & \(@5\) & \(@1\) & \(@5\) & \(@1\) & \(@5\) & \(@1\) & \(@5\) \\ \hline Siieve-BASED (D’Souza and Ng, 2015) & 84.7 & - & 84.1 & - & 90.7 & - & - & - & - & - & - & - \\ WordCNN (Linasgowhamt and Collier, 2016) & - & - & - & - & - & - & - & - & 81.4 & - & - & - \\ WordGRU+TF-IDF (Trushalina et al., 2018) & - & - & - & - & - & - & - & 85.7 & - & - & - \\ TaggerONE (Leaman and Lu, 2016) & 87.7 & - & 88.9 & - & 94.1 & - & OOM & OOM & - & - & - & - \\ NorMCO (Wrighten et al., 2019) & 87.8 & - & 88.0 & - & - & - & - & - & - & - & - \\ BNE (Phan et al., 2019) & 87.7 & - & 90.6 & - & 95.8 & - & - & - & - & - & - & - \\ BertRank (Ji et al., 2020) & 89.1 & - & - & - & - & - & - & - & - & - & - \\ Gen-RAN (Xu et al., 2020) & - & - & - & - & - & - & - & **87.5** & - & - & - \\ BioSYn (Suya et al., 2020) & **91.1** & **93.9** & **93.2** & **96.0** & **96.6** & **97.2** & OOM & OOM & 82.6” & 87.0” & 71.3” & 77.8” \\ Dict+Soilos+Neural (Basaldella et al., 2020) & - & - & - & - & - & - & - & - & - & - & **79.0** & - \\ supervised SOTA & 91.1 & 93.9 & 93.2 & 96.0 & 96.6 & 97.2 & OOM & OOM & 87.5 & - & 79.0 & - \\ \hline \hline \end{tabular}
\end{table}
Table 4: A list of baselines on the 6 different MEL datasets, including both scientific and social media language ones. The last row collects reported numbers from the best performing models. “\(*\)” denotes results produced using official released code. “-” denotes results not reported in the cited paper. “OOM” means out-of-memorey.





### Hardware Configurations

All our experiments are conducted on a server with specifications listed in Tab. 8.

## Appendix C Other Details

### The Full Table of Supervised Baseline Models

The full table of supervised baseline models is provided in Tab. 4.

### Hyper-Parameters Search Scope

Tab. 9 lists hyper-parameter search space for obtaining the set of used numbers. Note that the chosen hyper-parameters yield the overall best performance but might be sub-optimal on any single dataset. Also, we balanced the memory limit and model performance.

### A High-Resolution Version of Fig. 1

We show a clearer version of t-SNE embedding visualisation in Fig. 3.

\begin{table}
\begin{tabular}{l c c c} \hline \hline method & reduction rate & \#params & \multicolumn{2}{c}{\#params} \\ \cline{1-2}  & & \#params in BEXT \\ \hline Adapter\({}_{13\%}\) & 1 & 14.47M & 13.22\% \\ Adapter\({}_{1\%}\) & 16 & 0.60M & 1.09\% \\ \hline full-model-tuning & - & 109.48M & 100\% \\ \hline \hline \end{tabular}
\end{table}
Table 7: This table compares number of parameters trained in Adapter variants and also full-modeling tuning.

\begin{table}
\begin{tabular}{l l} \hline \hline model & URL \\ \hline vanilla Bert Devlin et al. (2019) & [https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased) \\ BioBert Lee et al. (2020) & [https://huggingface.co/dmi-lab/blobert-v1.1](https://huggingface.co/dmi-lab/blobert-v1.1) \\ BlueBert Peng et al. (2019) & [https://huggingface.co/bmiomlp/blobert_pubmed_mimic_mucased_L-12_N-768_A-12](https://huggingface.co/bmiomlp/blobert_pubmed_mimic_mucased_L-12_N-768_A-12) \\ ClinicalBert Alsentzer et al. (2019) & [https://huggingface.co/emilsilsentzer/Big_ClinicalBERT](https://huggingface.co/emilsilsentzer/Big_ClinicalBERT) Beltagy et al. (2019) \\ SciBert Beltagy et al. (2019) & [https://huggingface.co/alsent/scibet_evisco_unanced](https://huggingface.co/alsent/scibet_evisco_unanced) \\ UmlSbert Michalopoulos et al. (2020) & [https://www.dropbox.com/a/aqaqjgfeng5doc/unlanbert_tar.xz7dl-0](https://www.dropbox.com/a/aqaqjgfeng5doc/unlanbert_tar.xz7dl-0) \\ PubMedBert Gu et al. (2020) & [https://huggingface.co/microsoft/BieceMDF-PubMedSEM=bese-uncased-abstract-fulltex](https://huggingface.co/microsoft/BieceMDF-PubMedSEM=bese-uncased-abstract-fulltex) \\ \hline \hline \end{tabular}
\end{table}
Table 5: This table lists the URL of models used in this study.





\begin{table}
\begin{tabular}{l l} \hline hyper-parameters & search space \\ \hline learning rate for pretraining \& fine-tuning SapBert & \{1e-4,2e-5’,5e-5,1e-5,1e-6\} \\ pretraining batch size & \{128, 256, 512’, 1024\} \\ pretraining training iterations & \{10k, 20k, 30k, 40k, 50k (1 epoch)*, 100k (2 epochs)\} \\ fine-tuning epochs on scientific language datasets & \{1, 2, 3’, 5’\} \\ fine-training epochs on AskAPatient & \{5, 10, 15’, 20\} \\ fine-training epochs on COMETA & \{5, 10\({}^{*}\), 15, 20\} \\ max\_seq\_length of Bert tokenizer & \{15, 20, 25’, 30\} \\ \(\lambda\) in Online Mining & \{-0.05, -0.1, -0.2’, -0.3\} \\ \(\alpha\) in MS loss & \{1, 2 (Wang et al., 2019)*, 3’\} \\ \(\beta\) in MS loss & \{40, 50 (Wang et al., 2019)*, 60\} \\ \(\epsilon\) in MS loss & \{0.5”, 1 (Wang et al., 2019)\} \\ \(\alpha\) in max-margin triplet loss & \{0.05, 0.1, 0.2 (Basaldella et al., 2020)*, 0.3\} \\ softmax scale in NCA loss & \{1 (Goldberger et al., 2005), 5, 10, 20\({}^{*}\), 30\} \\ \(\alpha\) in Lifted-Structured loss & \{0.75’, 1 (Oh Song et al., 2016)\} \\ \(\tau\) (temperature) in InfoNCE & \{0.07 (He et al., 2020)*, 0.5 (Oord et al., 2018)\} \\ \(m\) in Circle loss & \{0.25 (Sun et al., 2020)*, 0.4 (Sun et al., 2020)\} \\ \(\gamma\) in Circle loss & \{80 (Sun et al., 2020), 256 (Sun et al., 2020)*\} \\ \end{tabular}
\end{table}
Table 9: This table lists the search space for hyper-parameters used. \(*\) means the used ones for reporting results.

Figure 3: Same as Fig. 1 in the main text, but generated with a higher resolution.