SlowGuess/ACL_2020_OCR huggingface dataset:
Our synonym marginalization method uses marginal maximum likelihood (MML) as the objective function. To verify the effectiveness of our proposed method, we compare our method with two different strategies: hard EM (Liang et al., 2018) and the standard pair-wise training (Leaman et al., 2013). The difference between hard EM and MML is that hard EM maximizes the probability of a single positive candidate having the highest probability. In contrast, MML maximizes marginalized probabilities of all synonyms in the top candidates. For hard EM, we first obtain a target  $\tilde{n}$  as follows:
$$
\tilde {n} = \operatorname {a r g m a x} _ {n \in N _ {1: k}} P (n | m; \theta) \tag {9}
$$
where most notations are the same as Equation 1. The loss function of hard EM is computed as follows:
$$
\mathcal {L} = - \frac {1}{M} \sum_ {i = 1} ^ {M} \log P (\tilde {n} | m _ {i}; \theta). \tag {10}
$$
The pair-wise training requires a binary classification model. For the pair-wise training, we minimize the binary cross-entropy loss using samples created by pairing each positive and negative candidate in the top candidates with the input mention. Table 3 shows the results of applying three different loss functions on BC5CDR Disease and BC5CDR Chemical. The results show that MML used in our framework learns better semantic representations than other methods.
6.1 Iterative Candidate Samples
In Table 4, we list top candidates of BIOSYN from the NCBI Disease development set. Although the
Methods	BC5CDR D.	BC5CDR C.
Acc@1	Acc@5	Acc@1	Acc@5
MML	91.1	95.4	96.7	97.7
Hard EM	91.0	95.8	96.5	97.5
Pair-Wise Training	90.7	94.4	96.3	97.2
Table 3: Comparison of two different training methods on the development sets of BC5CDR Disease, BC5CDR Chemical
initial candidates did not have positive samples due to the limitation of sparse representations, candidates at epoch 1 begin to include more positive candidates. Candidates at epoch 5 include many positive samples, while negative samples are also closely related to each mention.
In Table 5, we analyze the error cases of our model on the test set of NCBI Disease. We manually inspected all failure cases and defined the following error cases in the biomedical entity normalization task: Incomplete Synset, Contextual Entity, Overlapped Entity, Abbreviation, Hypernym, and Hyponym. Remaining failures that are difficult to categorize are grouped as Others.
Incomplete Synset is the case when the surface form of an input mention is very different from the provided synonyms of a gold CUI and requires the external knowledge for the normalization. Contextual Entity denotes an error case where an input mention and the predicted synonym are exactly the same but have different CUIs. This type of error could be due to an annotation error or happen when the same mention can be interpreted differently depending on its context. Overlapped Entity is an error where there is an overlap between the words of input mention and the predicted candidate. This includes nested entities. Abbreviation is an error where an input mention is in an abbreviated form but the resolution has failed even with the external module Ab3P. Hybernym and Hyponym are the cases when an input mention is a hybernym or a hyponym of the annotated entity.
Based on our analyses, errors are mostly due to ambiguous annotations (Contextual Entity, Overlapped Entity, Hypernym, Hyponym) or failure of pre-processings (Abbreviation). Incomplete Synset can be resolved with a better dictionary having richer synonym sets. Given the limitations in annotations, we conclude that the performance of BIOSYN has almost reached the upper bound.
Rank	tfidf	Epoch 0	Epoch 1	Epoch 5
prostate carcinomas (MeSH:D011471)
1	carcinomas	prostatic cancers*	prostate cancers*	prostate cancers*
2	teratocarcinomas	prostate cancers*	prostatic cancers*	prostate cancer*
3	pancreatic carcinomas ...	glioblastomas	prostate neoplasms*	prostatic cancers*
4	carcinomatoses	carcinomas	prostate cancer*	prostate neoplasms*
5	carcinomatosis	renal cell cancers	prostate neoplasm*	prostatic cancer*
6	breast carcinomas	renal cancers	prostatic cancer*	cancers prostate*
7	teratocarcinoma	retinoblastomas	prostatic neoplasms*	prostate neoplasm*
8	carcinoma	cholangiocarcinomas	advanced prostate cancers*	cancer of prostate*
9	breast carcinoma	pulmonary cancers	prostatic neoplasm*	cancer of the prostate*
10	carcinosarcomas	gonadoblastomas	prostatic adenomas	cancer prostate*
brain abnormalities (MeSH:D001927)
1	nail abnormalities	brain dysfunction minimal	brain pathology*	brain disorders*
2	abnormalities nail	brain pathology*	brain disorders*	brain disorder*
3	facial abnormalities	deficits memory	white matter abnormalities	brain diseases*
4	torsion abnormalities	memory deficits	brain disease*	brain disease*
5	spinal abnormalities	neurobehavioral manifestations	brain diseases*	abnormalities in brain dev...
6	skin abnormalities	white matter diseases	brain disorder*	nervous system abnormalities
7	genital abnormalities	brain disease metabolic	neuropathological abnormalities	white matter abnormalities
8	nail abnormality	neuropathological abnormalities	brain dysfunction minimal	metabolic brain disorders
9	clinical abnormalities	neurobehavioral manifestation	white matter lesions	brain metabolic disorders
10	abnormalities in brain dev...	brain disease*	brain injuries	brain pathology*
type ii deficiency (OMIM:217000)
1	mat i iii deficiency	deficiency disease	type ii c2 deficient*	factor ii deficiency
2	naga deficiency type iii ...	type 1 citrullinemia	deficiency disease	type ii c2 deficient*
3	properdin deficiency type iii ...	cmo ii deficiency	deficiency diseases	factor ii deficiencies
4	properdin deficiency type i ...	mitochondrial trifunctional ...	type ii c2d deficiency*	type ii c2d deficiency*
5	naga deficiency type iii	type ii c2 deficient*	factor ii deficiency	diabetes mellitus type ii
6	naga deficiency type ii	deficiency aga	deficiency protein	deficiency factor ii
7	properdin deficiency type iii	sodium channel myotonia	deficiency vitamin	c2 deficiency*
8	properdin deficiency type ii	deficiency diseases	deficiency factor ii	t2 deficiency
9	tc ii deficiency	tuftsin deficiency	deficiency arsa	tc ii deficiency
10	si deficiency	triosephosphate isomerase ...	class ii angle	mitochondrial complex ii ...
Table 4: Changes in the top 10 candidates given the two input mentions from the NCBI Disease development set. Synonyms having correct CUIs are indicated in boldface with an asterisk.
Error Type	Input	Predicted	Annotated	Statistics
Incomplete Synset	hypomania	hypodermyiasis	mood disorders	25 (29.4%)
Contextual Entity	colorectal adenomas	colorectal adenomas	polyps adenomatous	3 (3.5%)
Overlapped Entity	desmoid tumors	desmoid tumor	desmoids	11 (12.9%)
Abbreviation	scal	ocal	spinocerebellar ataxia 1	10 (11.8%)
Hypernym	campomelia	campomelic syndrome	campomelia cumming type	10 (11.8%)
Hyponym	eye movement abnormalities	eye movement disorder	eye abnormalities	23 (27.1%)
Others	hamartoma syndromes	hamartomas	multiple hamartoma syndromes	3 (3.5%)
Table 5: Examples and statistics of error cases on the NCBI Disease test set
In this study, we introduce BIOSYN that utilizes the synonym marginalization technique and the iterative candidate retrieval for learning biomedical entity representations. On four biomedical entity normalization datasets, our experiment shows that our model achieves state-of-the-art performance on all datasets, improving previous scores up to  $2.6\%$ . Although the datasets used in our experiments are in English, we expect that our methodology would work in any language as long as there is a synonym
dictionary for the language. For future work, an extrinsic evaluation of our methods is needed to prove the effectiveness of learned biomedical entity representations and to prove the quality of the entity normalization in downstream tasks.
This research was supported by National Research Foundation of Korea (NRF-2016M3A9A7916996, NRF-2014M3C9A3063541). We thank the members of Korea University, and the anonymous reviewers for their insightful comments.
Emily Alsentzer, John R Murphy, Willie Boag, WeiHung Weng, Di Jin, Tristan Naumann, WA Redmond, and Matthew BA McDermott. 2019. Publicly available clinical bert embeddings. In NAACL-HLT.
Elliot G Brown, Louise Wood, and Sue Wood. 1999. The medical dictionary for regulatory activities (meddra). Drug safety, 20(2).
Allan Peter Davis, Cynthia J Grondin, Robin J Johnson, Daniela Sciaky, Roy McMorran, Jolene Wiegers, Thomas C Wiegers, and Carolyn J Mattingly. 2018. The comparative toxicogenomics database: update 2019. NAR, 47(D1):D948-D954.
Allan Peter Davis, Thomas C Wiegers, Michael C Rosenstein, and Carolyn J Mattingly. 2012. Medic: a practical disease vocabulary used at the comparative toxicogenomics database. Database.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. *NAACL-HLT*.
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1-10.
Jennifer D'Souza and Vincent Ng. 2015. Sieve-based entity linking for the biomedical domain. In ACL.
Shobeir Fakhraei, Joel Mathew, and Jose-Luis Ambite. 2019. Nseen: Neural semantic embedding for entity normalization. In ECML-PKDD.
John M Giorgi and Gary D Bader. 2018. Transfer learning for biomedical named entity recognition with neural networks. Bioinformatics, 34(23):4087-4094.
Maryam Habibi, Leon Weber, Mariana Neves, David Luis Wiegandt, and Ulf Leser. 2017. Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics, 33(14):i37-i48.
Zongcheng Ji, Qiang Wei, and Hua Xu. 2019. Bert-based ranking for biomedical entity normalization. arXiv preprint arXiv:1908.03548.
Qiao Jin, Bhuwan Dhingra, William Cohen, and Xinghua Lu. 2019. Probing biomedical embeddings from language models. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In *ICLR*.
Robert Leaman, Rezarta Islamaj Dogan, and Zhiyong Lu. 2013. Dnorm: disease name normalization with pairwise learning to rank. Bioinformatics, 29(22):2909-2917.
Robert Leaman and Zhiyong Lu. 2016. Taggerone: joint named entity recognition and normalization with semi-markov models. Bioinformatics, 32(18):2839-2846.
Robert Leaman, Chih-Hsuan Wei, and Zhiyong Lu. 2015. tmchem: a high performance approach for chemical named entity recognition and normalization. Journal of cheminformatics, 7(Suppl 1).
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Sunwon Lee, Donghyeon Kim, Kyubum Lee, Jaehoon Choi, Seongsoon Kim, Minji Jeon, Sangrak Lim, Donghee Choi, Sunkyu Kim, Aik-Choon Tan, et al. 2016. Best: next-generation biomedical entity search tool for knowledge discovery from biomedical literature. PloS one, 11(10).
Fei Li, Meishan Zhang, Guohong Fu, and Donghong Ji. 2017a. A neural joint model for entity and relation extraction from biomedical text. BMC bioinformatics, 18(1):198.
Haodi Li, Qingcai Chen, Buzhou Tang, Xiaolong Wang, Hua Xu, Baohua Wang, and Dong Huang. 2017b. Cnn-based ranking for biomedical entity normalization. BMC bioinformatics, 18(Suppl 11):385.
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and semantic parsing. In NIPS.
Chen Lin, Timothy Miller, Dmitriy Dligach, Steven Bethard, and Guergana Savova. 2019. A bert-based universal model for both within-and cross-sentence clinical temporal relation extraction. In Proceedings of the 2nd Clinical Natural Language Processing Workshop.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS.
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017. Learning to match using local and distributed representations of text for web search. In WWW.
Ishani Mondal, Sukannya Purkayastha, Sudeshna Sarkar, Pawan Goyal, Jitesh Pillai, Amitava Bhattacharyya, and Mahanandeeshwar Gattu. 2019. Medical entity linking using triplet network. In Proceedings of the 2nd Clinical Natural Language Processing Workshop.
Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. 2016. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. TASLP, 24(4).
Minh C Phan, Aixin Sun, and Yi Tay. 2019. Robust representation learning of biomedical names. In ACL.
Sampo Pyysalo, F Ginter, Hans Moen, T Salakoski, and Sophia Ananiadou. 2013. Distributional semantics resources for biomedical text processing. Proceedings of Languages in Biology and Medicine.
Juan Ramos et al. 2003. Using tfidf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning.
Kirk Roberts, Dina Demner-Fushman, and Joseph M Tonning. 2017. Overview of the tac 2017 adverse reaction extraction from drug labels track. In TAC.
Sunil Sahu and Ashish Anand. 2016. Recurrent neural network models for disease name recognition using domain invariant features. In ACL.
Sunghwan Sohn, Donald C Comeau, Won Kim, and W John Wilbur. 2008. Abbreviation definition identification based on automatic precision estimates. BMC bioinformatics, 9:402.
Diana Sousa, André Lamúrias, and Francisco M Couto. 2019. A silver standard corpus of human phenotype-gene relations. In NAACL-HLT.
Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. 2018. Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics, 35(10):1745-1752.
Chih-Hsuan Wei, Hung-Yu Kao, and Zhiyong Lu. 2015. Gnormplus: an integrative approach for tagging genes, gene families, and protein domains. BioMed research international.
Dustin Wright. 2019. NormCo: Deep disease normalization for biomedical knowledge base construction. Ph.D. thesis, UC San Diego.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR.
Jun Xu, Yonghui Wu, Yaoyun Zhang, Jingqi Wang, Hee-Jin Lee, and Hua Xu. 2016. Cd-rest: a system for extracting chemical-induced disease relation in literature. Database.