SlowGuess/NAACL_2021_OCR huggingface dataset:
Because our model is initialized with Bio_ClinicalBERT model and pre-trained on the MIMIC-III dataset, it is not surprising that it does not outperform the BERT model on i2b2 2014 (The  $\mathrm{BERT}_{\mathrm{base}}$  model achieved  $95.2\%$  on i2b2 2014). This is probably due to the nature of the de-ID challenges which is described in detail in (Alsentzer et al., 2019). In summary, protected health information (PHI) are replaced with a sentinel 'PHI' marker in the MIMIC dataset, but in the de-ID challenge dataset (i2b2 2014), the PHI is replaced with different synthetic masks, and thus, the sentence structure that appears in BERT's training is not present at the down-stream task (Alsentzer et al., 2019). However, even in this task, UmlsBERT achieves a better performance than the other biomedical BERT models.
These results confirm that augmenting contextual embedding through domain (biomedical) knowledge is indeed beneficial for the model's performance in a variety of biomedical down-stream tasks.
5.1.3 Effect of semantic type embeddings
In order to understand the effect that semantic type embeddings have on the model performance, we conduct an ablation test where the performance of
two variations of the UmlsBERT model are compared, where in one model the semantic type embeddings are available to it, and in the other, they are not. The results of this comparison are listed in Table 5. We observe that for every dataset, Umls-Bert achieves its best performance when semantic type embeddings are available. This experiment further confirms the positive effect of the semantic type embeddings on the performance of the Umls-BERT model.
Table 4: The two nearest neighbors for six words in three semantic categories (two clinical and one generic). Note that only UmsBERT finds word associations based on the CUIs of the UMLS Metathesaurus that have clinical meaning whereas in the generic category, there is no discernible discrepancies between the models.
Dataset	UmlsBERT_ST	UmlsBERT
MedNLI	Ac.	82.3 ± 0.2	83.0 ± 0.1
i2b2 2006	F1	93.3 ± 0.7	93.6 ± 0.5
i2b2 2010	F1	88.3 ± 0.3	88.6 ± 0.1
i2b2 2012	F1	79.1 ± 0.2	79.4 ± 0.1
i2b2 2014	F1	94.7 ± 0.1	94.9 ± 0.1
Table 5: Results of mean  $\pm$  standard deviation of five runs for both variations of UmlsBERT on the test sets of all the datasets; In UmlsBERT  ${}_{-{ST}}$  ,the semantic type embeddings are not available.
5.2 Qualitative Embedding Comparisons
Table 4 shows the nearest neighbors for 6 words from 3 semantic categories using UmlsBERT, Bio_ClinicalBERT, BioBERT and BERT. The first two categories ('ANATOMY' and 'DISORDER') are chosen to demonstrate the ability of the models to identify similar words in a clinical context, and the third category ('GENERIC') is used to validate that the medical-focus BERT models can find meaningful associations between words in a general domain even if they are trained on medical-domain text datasets.
This analysis demonstrates that augmenting the contextual embedding of UmlsBERT with Clinical Metathesaurus (UMLS) information is indeed beneficial for discovering associations between words with similar meanings in a clinical context. For instance, only UmlsBERT discovers the connection between 'kidney' and 'ren' (from the latin word 'renes', which means kidneys), between 'mass' and 'lump', between 'bleeding' and 'hem' (a commonly used term to refer to blood) and between 'feet' and 'pedal'(a term pertaining to the foot or feet in a medical context).
These associations are the result of changing the nature of the Masked LM training phase of UmlsBERT to a multi-label scenario by connecting different words which share a common CUI in UMLS. In the previously mentioned examples, 'kidney' and 'ren' have CUI:C0022646; 'mass' and 'lump' have CUI:C0577559; 'bleeding' and 'hem' have CUI:C0019080 and 'feet' and 'pedal' have CUI:C0016504.
Finally, the results in the generic list of words indicate that the medical-focused BERT models did not trade their ability to find meaningful associations in a general domain in order to be more precise in a clinical context as there is no meaningful difference observed in the list of neighbour words that the four models identified.
5.3 Semantic Type Embedding Visualization
(b)
Figure 3: UMAP visualization of the clustering (a) of the Bio_ClinicalBert input embedding (word embedding) (b) of the UmlsBert input embedding (word embedding + semantic type embedding).
In order to demonstrate the effect of the semantic types on the input embeddings, we present in Figure 3, a UMAP dimensionality reduction (McInnes and Healy, 2018) mapping comparison between Bio_ClinicalBERT and UmlsBERT. We compare the input embedding of Bio_ClinicalBERT with the input embedding of UmlsBERT for all the clinical terms that UMLS identified in the standard
BERT vocabulary. It should be noted that in the graph, we group the medical terms by their semantic groups, which are clusters that consist of different semantic types. For example, the semantic types 'Cell' and 'Body System' are grouped in the semantic group 'ANATOMY'. It is evident that the clustering according to the semantic group that exists in the UmlsBERT embeddings (Figure 3b) cannot be found in the Bio_ClinicalBERT embeddings (Figure 3a). Thus, we can conclude that more meaningful input embeddings can be provided to the model, by augmenting the input layer of the BERT architecture with the semantic type vectors, as they force the embeddings of the words of the same semantic type to become more similar.
This paper presents UmlsBERT, a novel BERT-based architecture that incorporates domain (biomedical) knowledge in the pre-training process of a contextual word embeddings model. We demonstrate that UmlsBERT can learn the association of different clinical terms with similar meaning in the UMLS Metathesaurus. UmlsBERT can also create more meaningful input embeddings by leveraging information from the semantic type of each (biomedical) word. Finally, we confirm that these modifications can improve the model's performance as our UmlsBERT model outperforms other biomedical BERT models in various downstream tasks.
As for future work, we plan to address the limitations of this study including: (i) Examining the effect of augmenting contextual embeddings with medical knowledge when more complicated layers are used atop of the output embedding of UmlsbERT. (ii) Exploring the UMLS hierarchical associations between words that extend the concept connection that we investigated in this paper. (iii) Testing our model in other datasets and biomedical tasks (e.g. relation extraction task (Krallinger et al., 2017)) to investigate further the strengths and weaknesses of our model.
We acknowledge the generous support from Microsoft AI for Health Program, MITACS Accelerate grant (#IT19239), Semantic Health Inc., NSERC and Canada Research Chairs Program.
Contextual word embeddings models have achieved state-of-the-art results in many (clinical) NLP tasks such as NER or relation extraction (Devlin et al., 2019; Lee et al., 2019). These results suggest that medical-based contextual word embeddings models, such as our model (UmlsBERT), can be a valuable tool for better processing and understanding the vast volume of health data that is amassed at a rapid speed in health and biomedical domain.
However, one of obstacles for adopting such a model in any system lies in the computing cost of pre-training. For example, our UmlsBERT model was trained for 10 days using 2 nVidia V100 16GB GPU's with 224 GB of system RAM running Ubuntu 18.04.3 LTS, and we acknowledge that investing these types of computational resources or even time is not a viable option for many research groups, let alone regular healthcare providers. This is the reason for making the UmlsBERT model publicly available, as we hope that the clinical NLP community can benefit from using our model. In addition, UmlsBERT is the first contextual word embedding model, to the best of our knowledge, that integrated structured medical-domain knowledge into its pre-training phase. Although this study demonstrates the beneficial effect of incorporating structured biomedical domain knowledge in the pre-training phase of a contextual embedding model on the performance of the model, it is not a far-fetched hypothesis that similar pretraining strategy can be applied to incorporate structured domain-knowledge in different disciplines (e.g. environment, sciences, etc) to improve the performance of the model in the respective domainspecific down-stream tasks.
Finally, we believe that many research groups in the clinical NLP field could benefit from the use of our models by either using the contextual embeddings of our model or fine-tuning our model in specific down-stream tasks, for example, automatic encoding of diseases and procedures in electronic medical records. This automatic encoding model can significantly reduce time and cost in data extraction and reporting. Success in such task will have huge impact in clinical practices and research since assigning correct codes for diseases and clinical procedures are important for making care or operational decisions in healthcare.
Emily Alsentzer, John Murphy, William Boag, WeiHung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
O. Bodenreider. 2004. The Unified Medical Language System (UMLS): integrating biomedical terminology. *Nucleic Acids Res.*, 32(Database issue):D267-270.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Ish Kumar Dhammi and Sudhir Kumar. 2014. Medical subject headings (mesh) terms. Indian journal of orthopaedics vol., 48,5.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of EMNLP.
Maryam Habibi, Leon Weber, Mariana Neves, David Luis Wiegandt, and Ulf Leser. 2017. Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics, 33(14):i37-i48.
Luyao Huang, Sun Chi, Xipeng Qiu, and Xuanjing Huang. 2019. Glossbert: Bert for word sense disambiguation with gloss knowledge. pages 3500-3505.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimiciii, a freely accessible critical care database. Scientific data, 3:160035.
Martin Krallinger, O. Rabal, S. A. Akhondi, M. Pérez, Jesús Santamaría, Gael Pérez Rodríguez, G. Tsatsaronis, Ander Intxaurrondo, J. A. López, Umesh Nandal, E. V. Buel, A. Chandrasekhar, Marleen Rodenburg, A. Lægreid, Marius A. Doornenbal, J. Oyarzábal, Anália Lourenço, and A. Valencia. 2017. Overview of the biocreative vi chemicalprotein interaction track.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pretrained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. Sensebert: Driving some sense into bert. ArXiv, abs/1908.05646.
Alexa McCray, Anita Burgun, and Olivier Bodenreider. 2001. Aggregating umls semantic types for reducing conceptual complexity. Studies in health technology and informatics, 84:216-20.
L. McInnes and J. Healy. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. *ArXiv*, abs/1802.03426.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
World Health Organization. 2004. Icd-10: international statistical classification of diseases and related health problems: tenth revision.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora.
Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clinical domain. pages 1586-1596.
Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. 2010. Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications. Journal of the American Medical Informatics Association, 17(5):507-513.
Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts. 2019. Enhancing clinical concept extraction with contextual embeddings. Journal of the American Medical Informatics Association : JAMIA, 26.
Amber Stubbs, Christopher Kotfila, and Ozlem Uzuner. 2015. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/uthealth shared task track 1. Journal of biomedical informatics, 58 Suppl:S11-9.
Ozlem Uzuner, Yuan Luo, and Peter Szolovits. 2007. Evaluating the state-of-the-art in automatic de-identification. Journal of the American Medical Informatics Association: JAMIA, 14:550-63.
Ozlem Uzuner, Brett South, Shuying Shen, and Scott DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association: JAMIA, 18:552-6.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv, abs/1706.03762.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyls, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv, abs/1609.08144.