SlowGuess/NAACL_2021_OCR huggingface dataset:
Emily Alsentzer, John Murphy, William Boag, WeiHung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013. How noisy social media text, how diffrnt social media sources? In Proceedings of the Sixth International Joint Conference on Natural Language Processing (IJCNLP), pages 356-364, Nagoya, Japan. Asian Federation of Natural Language Processing.
Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020. COMETA: A corpus for medical entity linking in the social media. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3122-3137, Online. Association for Computational Linguistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620, Hong Kong, China. Association for Computational Linguistics.
Olivier Bodenreider. 2004. The unified medical language system (UMLS): integrating biomedical terminology. *Nucleic Acids Research*, 32:D267-D270.
Allan Peter Davis, Cynthia J Grondin, Robin J Johnson, Daniela Sciaky, Roy McMorran, Jolene Wiegers, Thomas C Wiegers, and Carolyn J Mattingly. 2019. The comparative toxicogenomics database: update 2019. Nucleic Acids Research, 47:D948-D954.
Allan Peter Davis, Thomas C Wiegers, Michael C Rosenstein, and Carolyn J Mattingly. 2012. MEDIC: a practical disease vocabulary used at the comparative toxicogenomics database. Database.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: a resource for disease name recognition and concept normalization. Journal of Biomedical Informatics, 47:1-10.
Kevin Donnelly. 2006. SNOMED-CT: The advanced terminology and coding system for eHealth. Studies in health technology and informatics, 121:279.
Jennifer D'Souza and Vincent Ng. 2015. Sieve-based entity linking for the biomedical domain. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP) (Volume 2: Short Papers), pages 297-302, Beijing, China. Association for Computational Linguistics.
Jacob Goldberger, Geoffrey E Hinton, Sam T Roweis, and Russ R Salakhutdinov. 2005. Neighbourhood components analysis. In Advances in Neural Information Processing Systems, pages 513-520.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing. arXiv:2007.15779.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Zongcheng Ji, Qiang Wei, and Hua Xu. 2020. BERT-based ranking for biomedical entity normalization. AMIA Summits on Translational Science Proceedings, 2020:269.
Donghyeon Kim, Jinhyuk Lee, Chan Ho So, Hwisang Jeon, Minbyul Jeong, Yonghwa Choi, Wonjin Yoon, Mujeen Sung, and Jaewoo Kang. 2019. A neural named entity recognition and multi-type normalization tool for biomedical text mining. IEEE Access, 7:73729-73740.
Robert Leaman and Zhiyong Lu. 2016. TaggerOne: joint named entity recognition and normalization with semi-markov models. Bioinformatics, 32:2839-2846.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pretrained biomedical language representation model
for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Sunwon Lee, Donghyeon Kim, Kyubum Lee, Jaehoon Choi, Seongsoon Kim, Minji Jeon, Sangrak Lim, Donghee Choi, Sunkyu Kim, Aik-Choon Tan, et al. 2016. BEST: next-generation biomedical entity search tool for knowledge discovery from biomedical literature. PloS one, 11:e0164680.
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016.
Nut Limsopatham and Nigel Collier. 2015. Adapting phrase-based machine translation to normalise medical terms in social media messages. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1675-1680, Lisbon, Portugal. Association for Computational Linguistics.
Nut Limsopatham and Nigel Collier. 2016. Normalising medical concepts in social media texts by learning semantic representation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1014-1023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov):2579-2605.
George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen Chen, and Alex Wong. 2020. Umlsbert: Clinical domain knowledge augmentation of contextual embeddings using the unified medical language system metathesaurus. arXiv preprint arXiv:2010.10391.
Sunil Mohan and Donghui Li. 2018. MedMentions: A large biomedical corpus annotated with UMLS concepts. In *Automated Knowledge Base Construction*.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. 2016. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4004-4012.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing, pages 58-65.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
Minh C Phan, Aixin Sun, and Yi Tay. 2019. Robust representation learning of biomedical names. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3275-3285.
Kirk Roberts, Matthew S Simpson, Ellen M Voorhees, and William R Hersh. 2015. Overview of the trec 2015 clinical decision support track. In TREC.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815-823.
Elliot Schumacher, Andriy Mulyar, and Mark Dredze. 2020. Clinical concept linking with contextualized neural representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8585-8592.
Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani. 2020. BioMegatron: Larger biomedical domain language model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4700-4706, Online. Association for Computational Linguistics.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. 2020. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6398-6407.
Mujeen Sung, Hwisang Jeon, Jinhyuk Lee, and Jaewoo Kang. 2020. Biomedical entity representations with synonym marginalization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3641-3650, Online. Association for Computational Linguistics.
Elena Tutubalina, Artur Kadurin, and Zulfat Miftahutdinov. 2020. Fair evaluation in concept normalization: a large-scale comparative analysis for bert-based models. In Proceedings of the 28th International Conference on Computational Linguistics (COLING).
Elena Tutubalina, Zulfat Miftahutdinov, Sergey Nikolko, and Valentin Malykh. 2018. Medical concept normalization in social media posts with recurrent neural networks. Journal of Biomedical Informatics, 84:93-102.
Ivan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222-7240, Online. Association for Computational Linguistics.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. 2019. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5022-5030.
Yanshan Wang, Sijia Liu, Naveed Afzal, Majid Rastegar-Mojarad, Liwei Wang, Feichen Shen, Paul Kingsbury, and Hongfang Liu. 2018. A comparison of word embeddings for the biomedical natural language processing. Journal of Biomedical Informatics, 87:12-20.
Dustin Wright, Yannis Katsis, Raghav Mehta, and Chun-Nan Hsu. 2019. Normco: Deep disease normalization for biomedical knowledge base construction. In *Automated Knowledge Base Construction*.
Dongfang Xu, Zeyu Zhang, and Steven Bethard. 2020. A generate-and-rank framework with semantic type regularization for biomedical concept normalization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8452-8464.
A Evaluation Datasets Details
We divide our experimental datasets into two categories (1) scientific language datasets where the data is extracted from scientific papers and (2) social media language datasets where the data is coming from social media forums like Reddit.com. For an overview of the key statistics, see Tab. 3.
A.1 Scientific Language Datasets
NCBI disease (Doğan et al., 2014) is a corpus containing 793 fully annotated PubMed abstracts and 6,881 mentions. The mentions are mapped into the MEDIC dictionary (Davis et al., 2012). We denote this dataset as "NCBI" in our experiments.
BC5CDR (Li et al., 2016) consists of 1,500 PubMed articles with 4,409 annotated chemicals, 5,818 diseases and 3,116 chemical-disease interactions. The disease mentions are mapped into the MEDIC dictionary like the NCBI disease corpus.
The chemical mentions are mapped into the Comparative Toxicogenomics Database (CTD) (Davis et al., 2019) chemical dictionary. We denote the disease and chemical mention sets as "BC5CDR-d" and "BC5CDR-c" respectively. For NCBI and BC5CDR we use the same data and evaluation protocol by Sung et al. (2020).
MedMentions (Mohan and Li, 2018) is a very-large-scale entity linking dataset containing over 4,000 abstracts and over 350,000 mentions linked to UMLS 2017AA. According to Mohan and Li (2018), training TAGGERONE (Leaman and Lu, 2016), a very popular MEL system, on a subset of MedMentions require  $>900$  GB of RAM. Its massive number of mentions and more importantly the used reference ontology (UMLS 2017AA has  $3\mathrm{M}+$  concepts) make the application of most MEL systems infeasible. However, through our metric learning formulation, SAPBERT can be applied on MedMentions with minimal effort.
A.2 Social-Media Language Datasets
AskAPatient (Limsopatham and Collier, 2016) includes 17,324 adverse drug reaction (ADR) annotations collected from askapatient.com blog posts. The mentions are mapped to 1,036 medical concepts grounded onto SNOMED-CT (Donnelly, 2006) and AMT (the Australian Medicines Terminology). For this dataset, we follow the 10-fold evaluation protocol stated in the original paper.[12]
COMETA (Basaldella et al., 2020) is a recently released large-scale MEL dataset that specifically focuses on MEL in the social media domain, containing around 20k medical mentions extracted from health-related discussions on reddit.com. Mentions are mapped to SNOMED-CT. We use the "stratified (general)" split and follow the evaluation protocol of the original paper.[13]
B Model & Training Details
B.1 The Choice of Base Models
We list all the versions of BERT models used in this study, linking to the specific versions in Tab. 5. Note that we exhaustively tried all official variants of the selected models and the best performing ones are chosen. All BERT models refer to the BERT<sub>Base</sub> architecture in this paper.
<sup>11</sup>https://github.com/dmis-lab/BioSyn
12https://zenodo.org/record/55013
13https://www.siphs.org/corpus
dataset	NCBI	BC5CDR-d	BC5CDR-c	MedMentions	AskAPAtient	COMETA (s.g.)	COMETA (z.g.)
Ontology	MEDIC	MEDIC	CTD	UMLS 2017AA	SNOMED & AMT	SNOMED	SNOMED
\( \mathcal{C}_{\text{searched}} \subsetneq \mathcal{C}_{\text{ontology?}} \)	✘	✘	✘	✘	✓	✘	✘
\( |_{\mathcal{C}_{\text{searched}}} | \)	11,915	11,915	171,203	3,415,665	1,036	350,830	350,830
\( |_{\mathcal{S}_{\text{searched}}} | \)	71,923	71,923	407,247	14,815,318	1,036	910,823	910,823
\( |_{\mathcal{M}_{\text{train}}} | \)	5,134	4,182	5,203	282,091	15,665.2	13,489	14,062
\( |_{\mathcal{M}_{\text{validation}}} | \)	787	4,244	5,347	71,062	792.6	2,176	1,958
\( |_{\mathcal{M}_{\text{test}}} | \)	960	4,424	5,385	70,405	866.2	4,350	3,995
Table 3: This table contains basic statistics of the MEL datasets used in the study.  $\mathcal{C}$  denotes the set of concepts;  $S$  denotes the set of all surface forms / synonyms of all concepts in  $\mathcal{C}$ ;  $\mathcal{M}$  denotes the set of mentions / queries. COMETA (s.g.) and (z.g.) are the stratified (general) and zeroshot (general) split respectively.
model	NCBI	BC5CDR-d	BC5CDR-c	MedMentions	AskAPatient	COMETA
@1	@5	@1	@5	@1	@5	@1	@5	@1	@5	@1	@5
SIEVE-BASED (D'Souza and Ng, 2015)	84.7	-	84.1	-	90.7	-	-	-
WORDCNN (Limsopatham and Collier, 2016)	-	-	-	-	-	-	-	-	81.4	-	-	-
WORDGRU+TF-IDF (Tutubalina et al., 2018)	-	-	-	-	-	-	-	-	85.7	-	-	-
TAGGERONE (Leaman and Lu, 2016)	87.7	-	88.9	-	94.1	-	OOM	OOM	-	-	-	-
NORMCO (Wright et al., 2019)	87.8	-	88.0	-	-	-	-	-	-	-	-	-
BNE (Phan et al., 2019)	87.7	-	90.6	-	95.8	-	-	-	-	-	-	-
BERTRANK (Ji et al., 2020)	89.1	-	-	-	-	-	-	-	-	-	-	-
GEN-RANK (Xu et al., 2020)	-	-	-	-	-	-	-	-	87.5	-	-	-
BIOSYN (Sung et al., 2020)	91.1	93.9	93.2	96.0	96.6	97.2	OOM	OOM	82.6*	87.0*	71.3*	77.8*
DICT+SOILS+NEURAL (Basaldella et al., 2020)	-	-	-	-	-	-	-	-	-	-	79.0	-
supervised SOTA	91.1	93.9	93.2	96.0	96.6	97.2	OOM	OOM	87.5	-	79.0	-
B.2 Comparing Loss Functions
We use COMETA (zeroshot general) as a benchmark for selecting learning objectives. Note that this split of COMETA is different from the stratified-general split used in Tab. 4. It is very challenging (so easy to see the difference of the performance) and also does not directly affect the model's performance on other datasets. The results are listed in Tab. 6. Note that online mining is switched on for all models here.
Table 4: A list of baselines on the 6 different MEL datasets, including both scientific and social media language ones. The last row collects reported numbers from the best performing models. “*” denotes results produced using official released code. “-” denotes results not reported in the cited paper. “OOM” means out-of-memory.
loss	@1 @5
cosine loss (Phan et al., 2019)	55.1 64.6
max-margin triplet loss (Basaldella et al., 2020)	64.6 74.6
NCA loss (Goldberger et al., 2005)	65.2 77.0
Lifted-Structure loss (Oh Song et al., 2016)	62.0 72.1
InfoNCE (Oord et al., 2018; He et al., 2020)	63.3 74.2
Circle loss (Sun et al., 2020)	66.7 78.7
Multi-Similarity loss (Wang et al., 2019)	67.2 80.3
Table 6: This table compares loss functions used for SAPBERT pretraining. Numbers reported are on COMETA (zereshot general).
The cosine loss was used by Phan et al. (2019) for learning UMLS synonyms for LSTM models. The max-margin triplet loss was used by Basaldella
et al. (2020) for training MEL models. A very similar (though not identical) hinge-loss was used by Schumacher et al. (2020) for clinical concept linking. InfoNCE has been very popular in self-supervised learning and contrastive learning (Oord et al., 2018; He et al., 2020). Lifted-Structure loss (Oh Song et al., 2016) and NCA loss (Goldberger et al., 2005) are two very classic metric learning objectives. Multi-Similarity loss (Wang et al., 2019) and Circle loss (Sun et al., 2020) are two recently proposed metric learning objectives and have been considered as SOTA on large-scale visual recognition benchmarks.
B.3 Details of ADAPTERS
In Tab. 7 we list number of parameters trained in the three ADAPTER variants along with full-model-tuning for easy comparison.
model	URL
vanilla BERT (Devlin et al., 2019)	https://huggingface.co/bert-base-uncased
BIOBERT (Lee et al., 2020)	https://huggingface.co/dmis-lab/biobert-v1.1
BLUEBERT (Peng et al., 2019)	https://huggingface.co/bionlp/bluebert_pubmed_mimic_uncased_L-12_H-768_A-12
CLINICALBERT (Alsentzer et al., 2019)	https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT
SCIBERT (Beltagy et al., 2019)	https://huggingface.co/allenai/scibert_scivocab_uncased
UMLSBERT (Michalopoulos et al., 2020)	https://www.dropbox.com/s/qaoq5gfen69xdcc/umlsbert.tar.xz?dl=0
PUBMEDBERT (Gu et al., 2020)	https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext
Table 5: This table lists the URL of models used in this study.
method	reduction rate	#params	#params
#params in BERT
ADAPTER13%	1	14.47M	13.22%
ADAPTER1%	16	0.60M	1.09%
full-model-tuning	-	109.48M	100%
B.4 Hardware Configurations
All our experiments are conducted on a server with specifications listed in Tab. 8.
Table 7: This table compares number of parameters trained in ADAPTER variants and also full-modeltuning.
hardware	specification
RAM	192 GB
CPU	Intel Xeon W-2255 @3.70GHz, 10-core 20-threads
GPU	NVIDIA GeForce RTX 2080 Ti (11 GB) × 4
Table 8: Hardware specifications of the used machine.
C Other Details
C.1 The Full Table of Supervised Baseline Models
The full table of supervised baseline models is provided in Tab. 4.
C.2 Hyper-Parameters Search Scope
Tab. 9 lists hyper-parameter search space for obtaining the set of used numbers. Note that the chosen hyper-parameters yield the overall best performance but might be sub-optimal on any single dataset. Also, we balanced the memory limit and model performance.
C.3 A High-Resolution Version of Fig. 1
We show a clearer version of t-SNE embedding visualisation in Fig. 3.
hyper-parameters	search space
learning rate for pretraining & fine-tuning SAPBERT	{1e-4, 2e-5*, 5e-5, 1e-5, 1e-6}
pretraining batch size	{128, 256, 512*, 1024}
pretraining training iterations	{10k, 20k, 30k, 40k, 50k (1 epoch)*, 100k (2 epochs)}
fine-tuning epochs on scientific language datasets	{1, 2, 3*, 5}
fine-training epochs on AskAPatient	{5, 10, 15*, 20}
fine-training epochs on COMETA	{5, 10*, 15, 20}
max_seq_length of BERT tokenizer	{15, 20, 25*, 30}
λ in Online Mining	{-0.05, -0.1, -0.2*, -0.3}
α in MS loss	{1, 2 (Wang et al., 2019)*, 3}
β in MS loss	{40, 50 (Wang et al., 2019)*, 60}
ε in MS loss	{0.5*, 1 (Wang et al., 2019)}
α in max-margin triplet loss	{0.05, 0.1, 0.2 (Basaldella et al., 2020)*, 0.3}
softmax scale in NCA loss	{1 (Goldberger et al., 2005), 5, 10, 20*, 30}
α in Lifted-Structured loss	{0.5*, 1 (Oh Song et al., 2016)}
τ (temperature) in InfoNCE	{0.07 (He et al., 2020)*, 0.5 (Oord et al., 2018)}
m in Circle loss	{0.25 (Sun et al., 2020)*, 0.4 (Sun et al., 2020)}
γ in Circle loss	{80 (Sun et al., 2020), 256 (Sun et al., 2020)*}
Table 9: This table lists the search space for hyper-parameters used. * means the used ones for reporting results.
Figure 3: Same as Fig. 1 in the main text, but generated with a higher resolution.