SlowGuess/NAACL_2022_OCR huggingface dataset:
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826.
Shogo Ujiie, Hayate Iso, and Eiji Aramaki. 2021. Biomedical entity linking via contrastive context matching. arXiv preprint arXiv:2106.07583.
Maya Varma, Laurel Orr, Sen Wu, Megan Leszczynski, Xiao Ling, and Christopher Ré. 2021. Cross-domain data integration for named entity disambiguation in biomedical text. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4566-4575, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Zero-shot entity linking with dense entity retrieval. In EMNLP.
Sheng Yu, Katherine P Liao, Stanley Y Shaw, Vivian S Gainer, Susanne E Churchill, Peter Szolovits, Shawn N Murphy, Isaac S Kohane, and Tianxi Cai. 2015. Toward high-throughput phenotyping: unbiased automated feature extraction and selection from knowledge sources. Journal of the American Medical Informatics Association, 22(5):993-1000.
Hongyi Yuan and Sheng Yu. 2021. Efficient symptom inquiring and diagnosis via adaptive alignment of reinforcement learning and classification. arXiv preprint arXiv:2112.00733.
Zheng Yuan, Zhengyun Zhao, Haixia Sun, Jiao Li, Fei Wang, and Sheng Yu. 2022. Coder: Knowledge-infused cross-lingual medical term embedding for term normalization. Journal of Biomedical Informatics, page 103983.
A Dataset Summary and Statistics
We pre-process the datasets by the following procedures: (1) the abbreviations in the texts are expanded using AB3P (Sohn et al., 2008); (2) the texts are lower-cased, and the beginning and ending of a mention are marked by two special tokens [ST] and [ET]; (3) the overlapping mentions and mentions absent from the target KB are discarded.
BC5CDR (Li et al., 2016) is a benchmark for biomedical entity recognition and disambiguation. The dataset annotates 1500 PubMed article abstracts with 4409 chemicals, 5818 diseases entities, and 3116 chemical-disease interactions. All the annotated entities are mapped to MeSH ontology, which is a smaller medical vocabulary that comprises a subset of UMLS (Bodenreider, 2004). In this work, in consideration of fairness, we follow two most recent works (Angell et al., 2021; Varma et al., 2021) that use MeSH contained in UMLS 2017 AA release to construct the target knowledge base.
NCBI (Doogan et al., 2014) contains a corpus of 793 PubMed abstracts. It consists of 6892 annotated disease mentions of 790 unique disease concepts. All the mentions are labeled with concepts in MELIC ontology (Davis et al., 2012). MELIC is a medical dictionary that merges the diseases concepts, synonyms, and definitions in MeSH and OMIM and is composed of 9700 unique diseases. In our work, we used the processed data and the target ontology provided by BioSyn (Sung et al., 2020) and ResCNN (Lai et al., 2021). We followed their works to construct our training, developing, and testing data.
COMETA (Basaldella et al., 2020) consists of 20k English biomedical entity mentions from publicly available and anonymous health discussions on Reddit. All the mentions are expert-annotated with concepts from SNOMEL CT. We use the "stratified (general)" split and follow the evaluation protocol of SapBert (Liu et al., 2021a) and ResCNN (Lai et al., 2021).
AskAPatient (Limsopatham and Collier, 2016) is a dataset containing 8,662 phrases of social media language. Each phrase can be mapped to one of the 1,036 medical concepts from SNOMEL-CT and AMT (the Australian Medicines Terminology). The samples in AskAPatient do not include contextual information and mentions can only be disambiguated by phrases per se. We follow the experimental settings from works of Sung et al. (2020) and Limsopatham and Collier (2016) and apply the 10-fold evaluation protocol.
Statistics of above-mentioned datasets are listed in Table 6.
NCBI	BC5CDR	COMETA	AskAPatient
Target names ||S||	108,071	809,929	904,798	3,381
Target concepts||E||	14,967	268,162	350,830	1,036
Train samples	5784	9,285	13,489	16,826
Dev samples	787	9,515	2,176	1,663
Test samples	960	9,654	4,350	1,712
Table 6: Basic statistics of NCBI-disease, BC5CDR, COMETA, AskAPatient datasets and the corresponding target knowledge bases.
Dataset	NCBI	BC5CDR	COMETA	AAP
FT	95.6±0.1	95.3±0.1	88.7±0.2	95.6±0.1
PT+FT	96.3±0.1	95.8±0.2	88.2±0.1	96.0±0.1
Table 7: The Recall@5 results of our proposed method on different benchmarks.
B License and Availability of Resources
BC5CDR, NCBI, COMETA, and AskAPatients are all publicly available datasets on the Internet. Their target KBs MeSH, MELIC, and SNOMEL CT are covered by UMLS Metathesaurus License. One can require such a license by signing up for a UMLS terminology services account to access KBs mentioned above.
C.1 Recall@5 Results
We show the Recall@5 result of our method with and without pre-training in Table 7.
C.2 Sub-population Analysis
Following Varma et al. (2021), we split test samples into different sub-populations. The details of different sub-population categories we use is shown in Table 8.
We provide case studies on the NCBI-disease benchmark to give an insight and justification of the performance of our method. For the case of mention colorectal adenomas which is annotated with D018256 adenomatous polyp, the mention exists for 6 times in the test set (account for 0.63 score of Recall@1). Our model fails to correctly disambiguate all such mentions while linking the mention to other concepts D000236 colorectal adenomas or D003123 hereditary nonpolyposis colorectal cancer. In the training set, the mention colorectal adenomas exists for two times and is annotated with D003132 and D000236 respectively. Thus, through this case, we can see (1) our model learns
the information contained in the training set; (2) such inconsistent test samples are hard to disambiguate correctly.
In this section, we provide more details of our experiments.
E.1 Knowledge Base Pre-processing
Given different knowledge bases, we pre-process their content by the following procedure:
1. For each concept, we include all its synonyms from the original target KB. We also expand the synonym set for each concept using UMLS. We use the 2017 AA Active Release of UMLS.
2. For synonyms in the expanded name set  $S$ , we lowercase them and remove the symbols (e.g., dash line - or comma,).
3. There may exist a name as a synonym for multiple concepts. We de-duplicate these overlapped synonyms by removing the synonym from the concept with more other synonyms to avoid the unbalanced number of synonyms in each concept.
It is worth noticing that we do not de-duplicate the synonyms in the target KB of NCBI in consideration of comparison fairness. In the previous works, a mention link to multiple concepts, and correct disambiguation is claimed if the target concept is hit by one of the predicted concepts in NCBI.
E.2 Pre-training Clause Templates
We list pre-training clause templates we used in Table 10. For those concepts containing only 2 synonyms,  $s_e^a$  and  $s_e^b$  are the two synonyms respectively and  $c_e$  is the same as  $s_e^b$ . For those concepts containing only 1 sole synonym,  $s_e^a$ ,  $s_e^b$  and  $c_e$  are the same.
Subset	Definition
Overall	Full set of the data
Single Word Mentions	Mentions that have one sole word.
Multi-Word Mentions	Mentions that have multiple words(Separated by blank spaces).
Unseen Mentions	Mentions not existing in fine-tuning set.
Unseen Concepts	Concepts not existing in fine-tuning set.
Not Direct Match	Mentions that are not a synonym of the target concept in KB.
Top 100	Mentions that mapped to the top 100 concepts in existing frequency in fine-tuning set.
Table 8: Accuracy over sub-populations on BC5CDR of our proposed methods and Varma et al. (2021).
Subset	Varma et al. (2021)	Ours	Sample Size
Baseline	Full	FT	PT + FT
Overall	89.5	91.3	92.5	93.4	9.65k
Single Word Mentions	91.4	92.9	95.8	96.6	7.04k
Multi-Word Mentions	84.5	86.8	85.3	84.8	2.62k
Unseen Mentions	75.3	79.6	81.8	83.3	3.28k
Unseen Concepts	69.7	77.5	86.3	86.9	2.16k
Not Direct Match	89.4	91.9	83.0	84.3	3.83k
Top 100	97.3	97.2	97.9	98.1	3.31k
Table 9: Accuracy over sub-populations on BC5CDR of our proposed methods and Varma et al. (2021). The best results are presented in bold letters and the second best results are highlighted by underlines.
Concepts	Templates
Definition	>2 synonyms	Encoder Side	Decoder Side
√	√/×	se<is defined as>ce.sae<is described as>cε.c<are the definitions of>sæ.c<describe>se.c<define>sæ.	seaisb.e.
×	√	ce<are the synonyms of>sæ.c<indicate the same concept as>sæ.se<has synonyms such as>cε.sae<refers to the same concepts as>cε.	seaisb.e.
×	×	ce<is>s.a.c<is the same as>s.a.se<is>cε.sae<is the same as>cε.	seaisb.e.
Table 10: The templates used for constructing pre-training samples for different kinds of concepts.  $s_e^a$  is the selected synonym as the input mention,  $s_e^b$  is the synonym selected as the decoding target.  $c_e$  is the contextual information comprised by definition, other synonyms, or mention itself. The template words are between  $\Leftrightarrow$ , and we omit the special tokens for marking mentions for conciseness.
E.3 Experiment Parameters
Our model contains 406M parameters with 12-layer transformer encoders and 12-layer transformer decoders. We list the hyper-parameters of our model for KB-guided pre-training and synonyms-aware fine-tuning on different benchmarks in Table 11. For pre-training, we heuristically select our parameters. For fine-tuning, we tune training steps among  $\{20000,30000,40000\}$  on development set. For BC5CDR and COMETA, we use learning rate as  $1e - 5$  and warmup steps as 500. For AskAPatient and NCBI, we search learning rate among  $\{5e - 6,8e - 7,3e - 7\}$ , and do not use warmup. We only evaluate our model at the end of training. For each benchmark, we run three times to calculate means and standard deviations.
E.4 Computational Resource
For our KB-guided pre-training, we implement our model on 6 A100 GPU with 40 GB memory with the help of DeepSpeed ZeRO 2 (Rajbhandari et al., 2020) and train for 1 day. For fine-tuning on different benchmarks, we implement our model on 1 A100 GPU.
Parameters	Pre-train	BC5CDR	COMETA	AskAPatient	NCBI
Training Steps	80,000	20,000	40,000	30,000	20,000
Learning Rate	4e-5	1e-5	1e-5	5e-6	3e-7
Weight Decay	0.01	0.01	0.01	0.01	0.01
Batch Size	384	8	8	8	8
Adam ε	1e-8	1e-8	1e-8	1e-8	1e-8
Adam β	(0.9,0.999)	(0.9,0.999)	(0.9,0.999)	(0.9,0.999)	(0.9,0.999)
Warmup Steps	1,600	500	500	0	0
Attention Dropout	0.1	0.1	0.1	0.1	0.1
Clipping Grad	0.1	0.1	0.1	0.1	0.1
Label Smoothing	0.1	0.1	0.1	0.1	0.1
Table 11: Hyper-parameters for KB-guided pre-training and synonyms-aware fine-tuning.