SlowGuess/ACL_2020_OCR huggingface dataset:
CharCNN+Attend+MT Niu et al. (2019) use a multi-task attentional character-level convolution neural network. They first convert the mention into a character embedding matrix. The auxiliary task network takes the embedding matrix as input for a CNN to learn to generate character-level domain-related importance weights. Such learned importance weights are concatenated with the character embedding matrix and fed as input to another CNN model with a softmax layer for multi-class classification.
CharLSTM+WordLSTM Han et al. (2017) first use a forward LSTM over each character of the mention and its corresponding character class such as lowercase or uppercase to generate a character-level vector representation, then use another bi-directional LSTM over each word of the mention to generate a word-level representation. They concatenate character-level and word-level representations and feed them as input to a softmax layer for multi-class classification.
LR+MeanEmbedding Belousov et al. (2017) calculate the mean of three different weighted word embeddings pre-trained on GoogleNews, Twitter and DrugTwitter as vector representations for
https://github.com/huggingface/transformers
8457
Approach	TwADR-L	AskAPatient	SMM4H-17
Dev	Test	Dev	Test	Dev	Test
WordCNN (Limsopatham and Collier, 2016)	-	44.78	-	81.41	-	-
WordGRU+Attend+TF-IDF (Tutubalina et al., 2018)	-	-	-	85.71	-	-
BERT+TF-IDF (Miftahutdinov and Tutubalina, 2019)	-	-	-	-	-	89.64
CharCNN+Attend+MT (Niu et al., 2019)	-	46.46	-	84.65	-	-
CharLSTM+WordLSTM (Han et al., 2017)	-	-	-	-	-	87.20
LR+MeanEmbedding (Belousov et al., 2017)	-	-	-	-	-	87.70
BERT	47.08	44.05	88.63	87.52	84.74	87.36
BERT + BERT-rank	48.07	46.32	88.14	87.10	84.44	87.66
BERT + BERT-rank + ST-reg	47.98	47.02	88.26	87.46	84.66	88.24
BERT + gold + BERT-rank	52.70	49.69	89.06	87.92	88.57	90.16
BERT + gold + BERT-rank + ST-reg	52.84	50.81	89.68	88.51	88.87	91.08
the mention, where word weights are calculated as inverse document frequency. Such vector representations are fed as input to a multinomial logistic regression (LR) model for multi-class classification.
Sieve-based Luo et al. (2019b) build a sieve-based normalization model which contains exact-match and MetaMap (Aronson, 2001) modules. Given a mention as input, the exact-match module first looks for mentions in the training data that exactly match the input, and then looks for concepts from the ontology whose synonyms exactly match the input. If no concepts are found, the mention is fed into MetaMap. They run this sieve-based normalization model twice. In the first round, the model lower-cases the mentions and includes acronym/abbreviation tokens during dictionary lookup. In the second round, the model lower-cases the mentions spans and also removes special tokens such as "&apos;s", "&quot;," etc.
Since our focus is individual systems, not ensembles, we compare only to other non-ensembles<sup>6</sup>.
4.6 Models
We separate out the different contributions from the following components of our architecture.
BERT The BERT-based multi-class classifier. When used alone, we select the most probable concept as the prediction.
Table 2: Comparisons of our proposed concept normalization architecture against the current state-of-the-art performances on TwADR-L, AskAPatient, and SMM4H-17 datasets.
Approach	MCN
Dev	Test
Sieve-based (Luo et al., 2019b)	-	76.35
Lucene	79.25
Lucene+BERT-rank	83.56	82.75
Lucene+BERT-rank+ST-reg	84.44	83.56
Lucene+gold+BERT-rank	86.89	84.77
Lucene+gold+BERT-rank+ST-reg	88.59	86.56
Table 3: Accuracy of our proposed concept normalization architecture on MCN dataset.
Lucene The Lucene-based dictionary look-up. When used alone, we take the top-ranked candidate concept as the prediction.
+BERT-rank The BERT-based list-wise classifier, always used in combination with either BERT or Lucene as a candidate generator
+ST-reg The semantic type regularizer, always used in combination with BERT-ranker.
We also consider the case  $(+\mathbf{gold})$  where we artificially inject the correct concept into the candidate generator's list if it was not already there.
Table 2 shows that our complete model, BERT + BERT-rank + ST-reg, achieves a new state-of-the-art on two of the social media test sets, and Table 3 shows that Lucene + BERT-rank + ST-reg achieves a new state-of-the-art on the clinical MCN test set. The TwADR-L dataset is the most difficult, with our complete model achieving  $47.02\%$  accuracy. In the other datasets, performance of our complete
An ensemble of three systems (including CharLSTM+WordLSTM and LR+MeanEmbedding) achieved  $88.7\%$  accuracy on the SMM4H-17 dataset (Sarker et al., 2018).
8458
model is much higher:  $87.46\%$  for AskAPatient,  $88.24\%$  for SMM4H-17 $^{7}$ .
On the TwADR-L, SMM4H-17, and MCN test sets, adding the BERT-based ranker improves performance over the candidate generator alone, and adding the semantic type regularization further improves performance. For example, Lucene alone achieves  $79.25\%$  accuracy on the MCN data, adding the BERT ranker increases this to  $82.75\%$ , and adding the semantic type regularizer increases this to  $83.56\%$ . On AskAPatient, performance of the full model is similar to just the BERT multiclass classifier, perhaps because in this case BERT alone already successfully improves the state-of-the-art from  $85.71\%$  to  $87.52\%$ . The +gold setting allows us to answer how well our ranker would perform if our candidate generator made no mistakes. First, we can see that if the correct concept is always in the candidate list, our list-based ranker (+BERT-rank) outperforms the multi-class classifier (BERT) on all test sets. We also see in this setting that the benefits of the semantic type regularizer are amplified, with test sets of TwADR-L and MCN showing more than  $1.00\%$  gain in accuracy from using the regularizer. These findings suggest that improving the quality of the candidate generator should be a fruitful future direction.
Overall, we see the biggest performance gains from our proposed generate-and-rank architecture in the MCN dataset. This is the most realistic setting, where the number of candidate concepts is large and many test concepts were never seen during training. In such cases, we cannot use a multiclass classifier as a candidate generator since it would never generate unseen concepts. Thus, our ranker shines in its ability to sort through the long list of possible concepts.
6 Qualitative analysis
Table 4 shows an example that is impossible for the multi-class classifier approach to concept normalization. The concept mention "an abdominal wall hernia" in the clinical MCN dataset needs to be mapped to the concept with the preferred name "Hernia of abdominal wall", but that concept never appeared in the training data. The Lucene-based candidate generator finds this concept, but only
Candidates	L	BR
Repair of abdominal wall hernia	1	3
Repair of anterior abdominal wall hernia	2	4
Obstructed hernia of anterior abdominal wall	3	5
Hernia of abdominal wall	4	1
Abdominal wall hernia procedure	5	2
Table 4: Predicted candidate concepts for mention An abdominal wall hernia and their rankings among the outputs of Lucene (L) and BERT-Ranker (BR). Gold concept is Hernia of abdominal wall.
Candidates	BR	STR	ST
Influenza-like illness	1	2	DS
Influenza	2	4	DS
Influenza-like symptoms	3	1	SS
Feeling tired	4	5	F
Muscle cramps in feet	5	3	SS
Table 5: Predicted candidate concepts for mention felt like I was coming down with flu and their rankings among the outputs of BERT-Ranker (BR) and BERT-Ranker + semantic type regularizer (STR). Gold concept is flu-like symptoms. Semantic types (ST) of the candidates include: disease or syndrome (DS), sign or symptom (SS), finding (F)
through character overlap (step d.) and several other concepts have high overlap as well. Thus Lucene ranks the correct concept 4th in its list. The BERT ranker is able to compare "an abdominal wall hernia" to "Hernia of abdominal wall" and recognize that as a better match than the other options, re-assigning it to rank 1.
Table 5 shows an example that illustrates why the semantic type regularizer helps. The mention "felt like I was coming down with flu" in the social media AskAPatient dataset needs to be mapped to the concept with the preferred name "influenza-like symptoms", which has the semantic type of a sign or symptom. The BERT ranker ranks two disease or syndromes higher, placing the correct concept at rank 3. After the semantic type regularizer is added, the system recognizes that the mention should be mapped to a sign or symptom, and correctly ranks it above the disease or syndromes. Note that this happens even though the ranker does not get to see the semantic type of the input mention at prediction time.
7 Limitations and future research
The available concept normalization datasets are somewhat limited. Lee et al. (2017) notes that AskAPatient and TwADR-L have issues including
<sup>7</sup>Miftahutdinov and Tutubalina (2019) use the same architecture as our BERT-based multi-class classifier (row 7), but they achieve  $89.28\%$  of accuracy on SMM4H-17. We were unable to replicate this result as their code and parameter settings were unavailable.
8459
Duplicate instances, which can lead to bias in the system; many phrases have multiple valid mappings to concepts but the context necessary to disambiguate is not part of the dataset; and the 10-fold cross-validation makes training complex models unnecessarily expensive. These datasets are also unrealistic in that all concepts in the test data are seen during training. Future research should focus on more realistic datasets that follow the approach of MCN in annotating mentions of concepts from a large ontology and including the full context.
Our ability to explore the size of the candidate list was limited by our available computational resources. As the size of the candidate list increases, the true concept is more likely to be included, but the number of training instances also increases, making the computational cost larger, especially for the datasets using 10-fold cross-validation. We chose candidate list sizes as large as we could afford, but there are likely further gains possible with larger candidate lists.
Our semantic type regularizer is limited to exact matching: it checks only whether the semantic type of a candidate exactly matches the semantic type of the true concept. The UMLS ontology includes many other relations, such as is-a and part-of relations, and extending our regularizer to encode such rich semantic knowledge may yield further improvements in the BERT-based ranker.
We propose a concept normalization framework consisting of a candidate generator and a list-wise classifier based on BERT.
Because the candidate ranker makes predictions over pairs of concept mentions and candidate concepts, it is able to predict concepts never seen during training. Our proposed semantic type regularizer allows the ranker to incorporate semantic type information into its predictions without requiring semantic types at prediction time. This generate-and-rank framework achieves state-of-the-art performance on multiple concept normalization datasets.
We thank the anonymous reviewers for their insightful comments on an earlier draft of this paper. This work was supported in part by National Institutes of Health grant R01LM012918 from the National Library of Medicine (NLM) and grant
R01GM114355 from the National Institute of General Medical Sciences (NIGMS). The computations were done in systems supported by the National Science Foundation under Grant No. 1228509. This research was supported in part by an appointment to the Oak Ridge National Laboratory Advanced Short-Term Research Opportunity (ASTRO) Program, sponsored by the U.S. Department of Energy and administered by the Oak Ridge Institute for Science and Education. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, National Science Foundation, or Department of Energy.
Alan R. Aronson. 2001. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. In Proceedings of the AMIA Symposium, pages 17-21. American Medical Informatics Association.
Maksim Belousov, William Dixon, and Goran Nenadic. 2017. Using an Ensemble of Generalised Linear and Deep Learning Models in the SMM4H 2017 Medical Concept Normalisation Task. In CEUR Workshop Proceedings, volume 1996, pages 54-58.
Olivier Bodenreider. 2004. The Unified Medical Language System (UMLS): integrating biomedical terminology. *Nucleic Acids Research*, 32(suppl_1):D267-D270.
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to Rank: From Pairwise Approach to Listwise Approach. In Proceedings of the 24th International Conference on Machine Learning, pages 129-136. Association for Computing Machinery.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jennifer D'Souza and Vincent Ng. 2015. Sieve-based entity linking for the biomedical domain. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 297-302, Beijing, China. Association for Computational Linguistics.
Sven Festag and Cord Spreckelsen. 2017. Word Sense Disambiguation of Medical Terms via Recurrent Convolutional Neural Networks. In Health Informatics Meets EHealth: Digital Insight Information-Driven Health & Care. Proceedings of the 11th EHealth2017 Conference, volume 236, pages 8-15.
Wilco W.M. Fleuren and Wynand Alkema. 2015. Application of text mining in the biomedical domain. Methods, 74:97-106.
Graciela H. Gonzalez, Tasnia Tahsin, Britton C. Goodale, Anna C. Greene, and Casey S. Greene. 2015. Recent Advances and Emerging Applications in Text and Data Mining for Biomedical Discovery. Briefings in Bioinformatics, 17(1):33-42.
Sifei Han, Tung Tran, Anthony Rios, and Ramakanth Kavuluru. 2017. Team UKNLP: Detecting ADRs, Classifying Medication Intake Messages, and Normalizing ADR Mentions on Twitter. In CEUR Workshop Proceedings, volume 1996, pages 49-53.
Zongcheng Ji, Qiang Wei, and Hua Xu. 2019. Bert-based ranking for biomedical entity normalization. arXiv preprint arXiv:1908.03548.
Jitendra Jonnagaddala, Toni Rose Jue, Nai-Wen Chang, and Hong-Jie Dai. 2016. Improving the dictionary lookup approach for disease normalization using enhanced dictionary and query expansion. Database, 2016:baw112.
Rohit J. Kate. 2016. Normalizing clinical terms using learned edit distance patterns. Journal of the American Medical Informatics Association, 23(2):380-386.
Andre Leal, Bruno Martins, and Francisco Couto. 2015. *ULisboa: Recognition and normalization of medical concepts*. In *Proceedings of the 9th International Workshop on Semantic Evaluation* (SemEval 2015), pages 406-411, Denver, Colorado. Association for Computational Linguistics.
Robert Leaman, Rezarta Islamaj Dogan, and Zhiyong Lu. 2013. DNorm: disease name normalization with pairwise learning to rank. Bioinformatics, 29(22):2909-2917.
Hsin-Chun Lee, Yi-Yu Hsu, and Hung-Yu Kao. 2016. AuDis: an automatic CRF-enhanced disease normalization in biomedical text. Database, 2016. Baw091.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. Btz682.
Kathy Lee, Sadid A. Hasan, Oladimeji Farri, Alok Choudhary, and Ankit Agrawal. 2017. Medical Concept Normalization for Online User-Generated Texts. In 2017 IEEE International Conference on Healthcare Informatics (ICHI), pages 462-469. IEEE.
Fei Li, Yonghao Jin, Weisong Liu, Bhanu Pratap Singh Rawat, Pengshan Cai, and Hong Yu. 2019. FineTuning Bidirectional Encoder Representations From Transformers (BERT)-Based Models on LargeScale Electronic Health Record Notes: An Empirical Study. JMIR Med Inform, 7(3):e14830.
Haodi Li, Qingcai Chen, Buzhou Tang, Xiaolong Wang, Hua Xu, Baohua Wang, and Dong Huang. 2017. CNN-based ranking for biomedical entity normalization. BMC Bioinformatics, 18(11):385.
Nut Limsopatham and Nigel Collier. 2016. Normalising medical concepts in social media texts by learning semantic representation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1014-1023, Berlin, Germany. Association for Computational Linguistics.
Feifan Liu, Abhyuday Jagannatha, and Hong Yu. 2019. Towards drug safety surveillance and pharmacovigilance: Current progress in detecting medication and adverse drug events from electronic health records. Drug Saf, 42:95-97.
Hongwei Liu and Yun Xu. 2017. A Deep Learning Way for Disease Name Representation and Normalization. In *Natural Language Processing and Chinese Computing*, pages 151-157. Springer International Publishing.
Yen-Fu Luo, Weiyi Sun, and Anna Rumshisky. 2019a. A Hybrid Normalization Method for Medical Concepts in Clinical Narrative using Semantic Matching. In AMIA Joint Summits on Translational Science proceedings, volume 2019, pages 732-740. American Medical Informatics Association.
Yen-Fu Luo, Weiyi Sun, and Anna Rumshisky. 2019b. MCN: A comprehensive corpus for medical concept normalization. Journal of Biomedical Informatics, pages 103-132.
Zulfat Miftahutdinov and Elena Tutubalina. 2019. Deep neural models for medical concept normalization in user-generated texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 393-399, Florence, Italy. Association for Computational Linguistics.
Shikhar Murty, Patrick Verga, Luke Vilnis, Irena Radovanovic, and Andrew McCallum. 2018. Hierarchical losses and new resources for fine-grained entity typing and linking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 97-109, Melbourne, Australia. Association for Computational Linguistics.
Thanh Ngan Nguyen, Minh Trang Nguyen, and Thanh Hai Dang. 2018. Disease Named Entity Normalization Using Pairwise Learning To Rank and Deep Learning. Technical report, VNU University of Engineering and Technology.
8461
Jinghao Niu, Yehui Yang, Siheng Zhang, Zhengya Sun, and Wensheng Zhang. 2019. Multi-task Character-Level Attentional Networks for Medical Concept Normalization. *Neural Process Lett*, 49(3):1239-1256.
Sameer Pradhan, Noémie Elhadad, Wendy Chapman, Suresh Manandhar, and Guergana Savova. 2014. SemEval-2014 task 7: Analysis of clinical text. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 54–62, Dublin, Ireland. Association for Computational Linguistics.
Anna Rumshisky, Marzyeh Ghassemi, Tristan Naumann, Peter Szolovits, V.M. Castro, T.H. McCoy, and R.H. Perlis. 2016. Predicting early psychiatric readmission with natural language processing of narrative discharge summaries. Transl Psychiatry, 6(10):e921-e921.
Abeed Sarker, Maksim Belousov, Jasper Friedrichs, Kai Hakala, Svetlana Kiritchenko, Farrokh Mehryary, Sifei Han, Tung Tran, Anthony Rios, Ramakanth Kavuluru, Berry de Bruijn, Filip Ginter, Debanjan Mahata, Saif M. Mohammad, Goran Nenadic, and Graciela Gonzalez-Hernandez. 2018. Data and systems for medication-related text classification and concept normalization from Twitter: insights from the Social Media Mining for Health (SMM4H)-2017 shared task. Journal of the American Medical Informatics Association, 25(10):1274-1283.
Guergana K. Savova, Anni R. Coden, Igor L. Sominsky, Rie Johnson, Philip V. Ogren, Piet C. De Groen, and Christopher G. Chute. 2008. Word sense disambiguation across two domains: Biomedical literature and clinical notes. Journal of Biomedical Informatics, 41(6):1088-1100.
Guergana K. Savova, James J. Masanz, Philip V. Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C. Kipper-Schuler, and Christopher G. Chute. 2010. Mayoclinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informatics Association, 17(5):507-513.
Mark Stevenson, Yikun Guo, Abdulaziz Alamri, and Robert Gaizauskas. 2009. Disambiguation of biomedical abbreviations. In Proceedings of the BioNLP 2009 Workshop, pages 71-79, Boulder, Colorado. Association for Computational Linguistics.
Hanna Suominen, Sanna Salantera, Sumithra Velupillai, Wendy W. Chapman, Guergana Savova, Noemie Elhadad, Sameer Pradhan, Brett R. South, Danielle L. Mowery, Gareth J.F. Jones, Johannes Leveling, Liadh Kelly, Lorraine Goeuriot, David Martinez, and Guido Zuccon. 2013. Overview of the ShARE/CLEF eHealth Evaluation Lab 2013. In Information Access Evaluation. Multilinguality, Multimodality, and Visualization, pages 212-231. Springer Berlin Heidelberg.
Elena Tutubalina, Zulfat Miftahutdinov, Sergey Nikolenko, and Valentin Malykh. 2018. Medical concept normalization in social media posts with recurrent neural networks. Journal of Biomedical Informatics, 84:93-102.
Özlem Uzuner, Brett R. South, Shuying Shen, and Scott L. DuVall. 2011. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552-556.
Davy Weissenbacher, Abeed Sarker, Arjun Magge, Ashlynn Daughton, Karen O'Connor, Michael J. Paul, and Graciela Gonzalez-Hernandez. 2019. Overview of the fourth social media mining for health (SMM4H) shared tasks at ACL 2019. In Proceedings of the Fourth Social Media Mining for Health Applications (#SMM4H) Workshop & Shared Task, pages 21-30, Florence, Italy. Association for Computational Linguistics.
Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th International Conference on Machine Learning, pages 1192-1199. Association for Computing Machinery.
Antonio Jimeno Yepes. 2017. Word embeddings and recurrent neural networks based on Long-Short Term Memory nodes in supervised biomedical word sense disambiguation. Journal of Biomedical Informatics, 73:137-147.
8462
A.1 Lucene-based dictionary look-up system
The lucene-based dictionary look-up system consists of the following components:
(a) Lucene index over the training data finds all CUI-less mentions that exactly match mention  $m$ .
(b) Lucene index over the training data finds CUIs of all training mentions that exactly match mention  $m$ .
(c) Lucene index over UMLS finds CUIs whose preferred name exactly matches mention  $m$ .
(d) Lucene index over UMLS finds CUIs where at least one synonym of the CUI exactly matches mention  $m$ .
(e) Lucene index over UMLS finds CUIs where at least one synonym of the CUI has high character overlap with mention  $m$ . To check the character overlap, we run the following three rules sequentially: token-level matching, fuzzy string matching with a maximum edit distance of 2, and character 3-gram matching..
along with the concept mention, to the BERT-based reranker (f).
During training, we used component (e) alone instead of the combination of components (b)-(e) to generate training instances for the BERT-based reranker (f) as it generated many more training examples and resulted in better performance on the dev set. During evaluation, we used the whole pipeline.
See Figure A1 for the flow of execution across the components. Whenever there are multiple CUIs generated from a component (a) to (e), they are fed,
Figure A1: Architecture of the lucene-based dictionary look-up system. The edges out of a search process indicate the number of matches necessary to follow the edge. Outlined nodes are terminal states that represent the predictions of the system.
8463