SlowGuess/ACL_2020_OCR huggingface dataset:
Dongfang Xu and Zeyu Zhang and Steven Bethard
School of Information
{dongfangxu9, zeyuzhang, bethard}  $@$  email.arizona.edu
Concept normalization, the task of linking textual mentions of concepts to concepts in an ontology, is challenging because ontologies are large. In most cases, annotated datasets cover only a small sample of the concepts, yet concept normalizers are expected to predict all concepts in the ontology. In this paper, we propose an architecture consisting of a candidate generator and a list-wise ranker based on BERT. The ranker considers pairings of concept mentions and candidate concepts, allowing it to make predictions for any concept, not just those seen during training. We further enhance this list-wise approach with a semantic type regularizer that allows the model to incorporate semantic type information from the ontology during training. Our proposed concept normalization framework achieves state-of-the-art performance on multiple datasets.
Mining and analyzing the constantly-growing unstructured text in the bio-medical domain offers great opportunities to advance scientific discovery (Gonzalez et al., 2015; Fleuren and Alkema, 2015) and improve the clinical care (Rumshisky et al., 2016; Liu et al., 2019). However, lexical and grammatical variations are pervasive in such text, posing key challenges for data interoperability and the development of natural language processing (NLP) techniques. For instance, heart attack, MI, myocardial infarction, and cardiovascular stroke all refer to the same concept. It is critical to disambiguate these terms by linking them with their corresponding concepts in an ontology or knowledge base. Such linking allows downstream tasks (relation extraction, information retrieval, text classification, etc.) to access the ontology's rich knowledge about biomedical entities, their synonyms, semantic types and mutual relationships.
Concept normalization is a task that maps concept mentions, the in-text natural-language mentions of ontological concepts, to concept entries in a standardized ontology or knowledge base. Techniques for concept normalization have been advancing, thanks in part to recent shared tasks including clinical disorder normalization in 2013 ShARE/CLEF (Suominen et al., 2013) and 2014 SemEval Task 7 Analysis of Clinical Text (Pradhan et al., 2014), and adverse drug event normalization in Social Media Mining for Health (SMM4H) (Sarker et al., 2018; Weissenbacher et al., 2019). Most existing systems use a string-matching or dictionary look-up approach (Leal et al., 2015; D'Souza and Ng, 2015; Lee et al., 2016), which are limited to matching morphologically similar terms, or supervised multi-class classifiers (Belousov et al., 2017; Tutubalina et al., 2018; Niu et al., 2019; Luo et al., 2019a), which may not generalize well when there are many concepts in the ontology and the concept types that must be predicted do not all appear in the training data.
We propose an architecture (shown in Figure 1) that is able to consider both morphological and semantic information. We first apply a candidate generator to generate a list of candidate concepts, and then use a BERT-based list-wise classifier to rank the candidate concepts. This two-step architecture allows unlikely concept candidates to be filtered out prior to the final classification, a necessary step when dealing with ontologies with millions of concepts. In contrast to previous list-wise classifiers (Murty et al., 2018) which only take the concept mention as input, our BERT-based list-wise classifier takes both the concept mention and the candidate concept name as input, and is thus able to handle concepts that never appear in the training data. We further enhance this list-wise approach with a semantic type regularizer that allows our ranker to leverage semantic type information from
8452
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8452-8464 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Proposed architecture for concept normalization: candidate generation and ranking.
the ontology during training.
Our work makes the following contributions:
- Our proposed concept normalization framework achieves state-of-the-art performance on multiple datasets.
- We propose a concept normalization framework consisting of a candidate generator and a list-wise classifier. Our framework is easier to train and the list-wise classifier is able to predict concepts never seen during training.
- We introduce a semantic type regularizer which encourages the model to consider the semantic type information of the candidate concepts. This semantic type regularizer improves performance over the BERT-based listwise classifier on multiple datasets.
The code for our proposed generate-and-rank framework is available at https://github.com/ dongfang91/Generate-and-Rank-ConNorm.
Traditional approaches for concept normalization involve string match and dictionary look-up. These approaches differ in how they construct dictionaries, such as collecting concept mentions from the labeled data as extra synonyms (Leal et al., 2015; Lee et al., 2016), and in different string matching techniques, such as string overlap and edit distance (Kate, 2016). Two of the most commonly used knowledge-intensive concept normalization tools, MetaMap (Aronson, 2001) and cTAKES (Savova et al., 2010) both employ rules to first generate lexical variants for each noun phrase and then conduct dictionary look-up for each variant. Several systems (D'Souza and Ng, 2015; Jonnagaddala et al., 2016) have demonstrated that rule-based concept normalization systems achieve performance competitive with other approaches in a sieve-based approach that carefully selects combinations and orders of dictionaries, exact and partial matching,
and heuristic rules. However, such rule-based approaches struggle when there are great variations between concept mention and concept, which is common, for example, when comparing social media text to medical ontologies.
Due to the availability of shared tasks and annotated data, the field has shifted toward machine learning techniques. We divide the machine learning approaches into two categories, classification (Savova et al., 2008; Stevenson et al., 2009; Limsopatham and Collier, 2016; Yepes, 2017; Festag and Spreckelsen, 2017; Lee et al., 2017; Tutubalina et al., 2018; Niu et al., 2019) and learning to rank (Leaman et al., 2013; Liu and Xu, 2017; Li et al., 2017; Nguyen et al., 2018; Murty et al., 2018).
Most classification-based approaches using deep neural networks have shown strong performance. They differ in using different architectures, such as Gated Recurrent Units (GRU) with attention mechanisms (Tutubalina et al., 2018), multi-task learning with auxiliary tasks to generate attention weights (Niu et al., 2019), or pre-trained transformer networks (Li et al., 2019; Miftahutdinov and Tutubalina, 2019); different sources for training word embeddings, such as Google News (Limsopatham and Collier, 2016) or concept definitions from the Unified Medical Language System (UMLS) Metathesaurus (Festag and Speckelsen, 2017); and different input representations, such as using character embeddings (Niu et al., 2019). All classification approaches share the disadvantage that the output space must be the same size as the number of concepts to be predicted, and thus the output space tends to be small such as 2,200 concepts in (Limsopatham and Collier, 2016) and around 22,500 concepts in (Weissenbacher et al., 2019). Classification approaches also struggle with concepts that have only a few example mentions in the training data.
Researchers have applied point-wise learning to rank (Liu and Xu, 2017; Li et al., 2017), pairwise learning to rank (Leaman et al., 2013; Nguyen
8453
et al., 2018), and list-wise learning to rank (Murty et al., 2018; Ji et al., 2019) on concept normalization. Generally, the learning-to-rank approach has the advantage of reducing the output space by first obtaining a smaller list of possible candidate concepts via a candidate generator and then ranking them. DNorm (Leaman et al., 2013), based on a pair-wise learning-to-rank model where both mentions and concept names were represented as TF-IDF vectors, was the first to use learning-to-rank for concept normalization and achieved the best performance in the ShARE/CLEF eHealth 2013 shared task. List-wise learning-to-rank approaches are both computationally more efficient than pairwise learning-to-rank (Cao et al., 2007) and empirically outperform both point-wise and pair-wise approaches (Xia et al., 2008). There are two implementations of list-wise classifiers using neural networks for concept normalization: Murty et al. (2018) treat the selection of the best candidate concept as a flat classification problem, losing the ability to handle concepts not seen during training; Ji et al. (2019) take a generate-and-rank approach similar to ours, but they do not leverage resources such as synonyms or semantic type information from UMLS in their BERT-based ranker.
3.1 Concept normalization framework
We define a concept mention  $m$  as an abbreviation such as "MI", a noun phrase such as "heart attack", or even a short text such as "an obstruction of the blood supply to the heart". The goal is then to assign  $m$  with a concept  $c$ . Formally, given a list of pre-identified concept mentions  $M = \{m_1, m_2, \dots, m_n\}$  in the text and an ontology or knowledge base with a set of concepts  $C = \{c_1, c_2, \dots, c_t\}$ , the goal of concept normalization is to find a mapping function  $c_j = f(m_i)$  that maps each textual mention to its correct concept.
We approach concept normalization in two steps: we first use a candidate generator  $G(m, C) \to C_m$  to generate a list of candidate concepts  $C_m$  for each mention  $m$ , where  $C_m \subseteq C$  and  $|C_m| \ll |C|$ . We then use a candidate ranker  $R(m, C_m) \to \hat{C_m}$  where  $\hat{C_m}$  is a re-ranked list of candidate concepts sorted by their relevance, preference, or importance. But unlike information retrieval tasks where the order of candidate concepts in the sorted list  $\hat{C_m}$  is crucial, in concept normalization we care only that the one true concept is at the top of the list.
The main idea of the two-step approach is that we first use a simple and fast system with high recall to generate candidates, and then a more precise system with more discriminative input to rank the candidates.
3.2 Candidate generator
We implement two kinds of candidate generators: a BERT-based multi-class classifier when the number of concepts in the ontology is small, and a Lucene-based dictionary look-up when there are hundreds of thousands of concepts in the ontology.
3.2.1 BERT-based multi-class classifier
BERT (Devlin et al., 2019) is a contextualized word representation model that has shown great performance in many NLP tasks. Here, we use BERT in a multi-class text-classification configuration as our candidate concept generator. We use the final hidden vector  $V_{m} \in \mathbb{R}^{H}$  corresponding to the first input token ([CLS]) generated from  $BERT(m)$  and a classification layer with weights  $W \in \mathbb{R}^{|C| \times H}$ , and train the model using a standard classification loss:
$$
L _ {G} = y * \log (\operatorname {s o f t m a x} \left(V _ {m} W ^ {T}\right)) \tag {1}
$$
where  $y$  is a one-hot vector, and  $|y| = |C|$ . The score for all concepts is calculated as:
$$
p (C) = \operatorname {s o f t m a x} \left(V _ {m} W ^ {T}\right) \tag {2}
$$
We select the top  $k$  most probable concepts in  $p(C)$  and feed that list  $C_m$  to the ranker.
3.2.2 Lucene-based dictionary look-up system
Multi-pass sieve rule based systems (D'Souza and Ng, 2015; Jonnagaddala et al., 2016; Luo et al., 2019b) achieve competitive performance when used with the right combinations and orders of different dictionaries, exact and partial matching, and heuristic rules. Such systems relying on basic lexical matching algorithms are simple and fast to implement, but they are only able to generate candidate concepts which are morphologically similar to a given mention.
Inspired by the work of Luo et al. (2019b), we implement a Lucene-based sieve normalization system which consists of the following components (see Appendix A.1 for details):
'https://lucene.apache.org/
8454
a. Lucene index over the training data finds all mentions that exactly match  $m$ .
b. Lucene index over ontology finds concepts whose preferred name exactly matches  $m$ .
c. Lucene index over ontology finds concepts where at least one synonym of the concept exactly matches  $m$ .
d. Lucene index over ontology finds concepts where at least one synonym of the concept has high character overlap with  $m$ .
The ranked list  $C_m$  generated by this system is fed as input to the candidate ranker.
3.3 Candidate ranker
After the candidate generator produces a list of concepts, we use a BERT-based list-wise classifier to select the most likely candidate. BERT allows us to match morphologically dissimilar (but semantically similar) mentions and concepts, and the list-wise classifier takes both mention and candidate concepts as input, allowing us to handle concepts that appear infrequently (or never) in the training data.
Here, we use BERT similar to a question answering configuration, where given a concept mention  $m$ , the task is to choose the most likely candidate concept  $c_{m}$  from all candidate concepts  $C_m$ . As shown in Figure 1, our classifier input includes the text of the mention  $m$  and all synonyms of the candidate concept  $c_{m}$ , and takes the form [CLS]  $m$  [SEP]  $\mathrm{syn}_1(c_m)$  [SEP] ... [SEP]  $\mathrm{syn}_s(c_m)$  [SEP], where  $\mathrm{syn}_i(c_m)$  is the  $i^{\text{th}}$  synonym of concept  $c_m^2$ . We calculate the final hidden vector  $V_{(m,c_m)} \in \mathbb{R}^H$  corresponding to the first input token ([CLS]) generated from BERT for each such input, and then concatenate the hidden vectors of all candidate concepts to form a matrix  $V_{(m,C_m)} \in \mathbb{R}^{|C_m| \times H}$ . We use this matrix and classification layer weights  $W \in \mathbb{R}^H$ , and compute a standard classification loss:
$$
L _ {R} = y * \log (\operatorname {s o f t m a x} (V _ {(m, C _ {m})} W ^ {T})). \quad (3)
$$
where  $y$  is a one-hot vector, and  $|y| = |C_m|$ .
3.4 Semantic type regularizer
To encourage the list-wise classifier towards a more informative ranking than just getting the correct
concept at the top of the list, we propose a semantic type regularizer that is optimized when candidate concepts with the correct semantic type are ranked above candidate concepts with incorrect types. The semantic type of the candidate concept is assumed correct only if it exactly matches the semantic type of the gold truth concept. If the concept has multiple semantic types, all must match. Our semantic type regularizer consists of two components:
$$
R _ {p} \left(\hat {y} _ {t}, \hat {y} _ {p}\right) = \sum_ {p \in P (y)} \left(m _ {1} + \hat {y} _ {p} - \hat {y} _ {t}\right) \tag {4}
$$
$$
R _ {n} \left(\hat {y} _ {p}, \hat {y} _ {n}\right) = \sum_ {p \in P (y)} \max  _ {n \in N (y)} \left(m _ {2} + \hat {y} _ {n} - \hat {y} _ {p}\right) \tag {5}
$$
where  $\hat{y} = V_{(m,c_m)}W^T$ ,  $N(y)$  is the set of indexes of candidate concepts with incorrect semantic types (negative candidates),  $P(y)$  (positive candidates) is the complement of  $N(y)$ ,  $\hat{y}_t$  is the score of the gold truth candidate concept, and thus  $t \in P(y)$ . The margins  $m_1$  and  $m_2$  are hyper-parameters for controlling the minimal distances between  $\hat{y}_t$  and  $\hat{y}_p$  and between  $\hat{y}_p$  and  $\hat{y}_n$ , respectively. Intuitively,  $R_p$  tries to push the score of the gold truth concept above all positive candidates at least by  $m_1$ , and  $R_n$  tries to push the best scored negative candidate below all positive candidates by  $m_2$ .
The final loss function we optimize for the BERT-based list-wise classifier is:
$$
L = L _ {R} + \lambda R _ {p} \left(\hat {y} _ {t}, \hat {y} _ {p}\right) + \mu R _ {n} \left(\hat {y} _ {p}, \hat {y} _ {n}\right) \tag {6}
$$
where  $\lambda$  and  $\mu$  are hyper-parameters to control the tradeoff between standard classification loss and the semantic type regularizer.
Our experiments are conducted on three social media datasets, AskAPatient (Limsopatham and Collier, 2016), TwADR-L (Limsopatham and Collier, 2016), and SMM4H-17 (Sarker et al., 2018), and one clinical notes dataset, MCN (Luo et al., 2019b). We summarize dataset characteristics in Table 1.
AskAPatient The AskAPatient dataset<sup>3</sup> contains 17,324 adverse drug reaction (ADR) annotations collected from blog posts. The mentions are mapped to 1,036 medical concepts with
2 In preliminary experiments, we tried only the concept's preferred term and several other ways of separating synonyms, but none of these resulted in better performance.
3http://dx.doi.org/10.5281/zenodo. 55013
8455
Dataset	AskAPatient	TwADR-L	SMM4H-17	MCN
Ontology	SNOMED-CT & AMT	MedDRA	MedDRA (PT)	SNOMED-CT & RxNorm
Subset	Y	Y	N	N
|Contology|	1,036	2,220	22,500	434,056
|STontology|	22	18	61	125
|Cdataset|	1,036	2,220	513	3,792
|M|	17,324	5,074	9,149	13,609
|Mtrain|	15665.2	4805.7	5,319	5,334
|Mtest|	866.2	142.7	2,500	6,925
|M|/|Cdataset|	16.72	2.29	17.83	3.59
|Ctest - Ctrain|	0	0	43	2,256
|Mtest - Mtrain|/Mtest	39.7%	39.5%	34.7%	53.9%
|Mambiguous|/|M|	1.2%	12.8%	0.8%	4.5%
Table 1: Dataset statistics, where  $C$  is a set of concepts,  ${ST}$  is a set of semantic types,and  $M$  is a set of mentions.
22 semantic types from the subset of Systematized Nomenclature Of Medicine-Clinical Term (SNOMED-CT) and the Australian Medicines Terminology (AMT). We follow the 10-fold cross validation (CV) configuration in Lim-sopatham and Collier (2016) which provides 10 sets of train/dev/test splits.
TwADR-L The TwADR-L dataset contains 5,074 ADR expressions from social media. The mentions are mapped to 2,220 Medical Dictionary for Regulatory Activities (MedDRA) concepts with 18 semantic types. We again follow the 10-fold cross validation configuration defined by Limsopatham and Collier (2016).
SMM4H-17 The SMM4H-17 dataset consists of 9,149 manually curated ADR expressions from tweets. The mentions are mapped to 22,500 concepts with 61 semantic types from MedDRA Preferred Terms (PTs). We use the 5,319 mentions from the released set as our training data, and keep the 2,500 mentions from the original test set as evaluation.
MCN The MCN dataset consists of 13,609 concept mentions drawn from 100 discharge summaries from the fourth i2b2/VA shared task (Uzuner et al., 2011). The mentions are mapped to 3792 unique concepts out of 434,056 possible concepts with 125 semantic types in SNOMED-CT and RxNorm. We take 40 clinical notes from the released data as training, consisting of 5,334 mentions, and the standard evaluation data with 6,925 mentions as our test set. Around  $2.7\%$  of mentions in MCN could not be mapped to any
concepts in the terminology, and are assigned to the CUI-less label.
A major difference between the datasets is the space of concepts that systems must consider. For AskAPatient and TwADR-L, all concepts in the test data are also in the training data, and in both cases only a couple thousand concepts have to be considered. Both SMM4H-17 and MCN define a much larger concept space: SMM4H-17 considers 22,500 concepts (though only 513 appear in the data) and MCN considers 434,056 (though only 3,792 appear in the data). AskAPatient and TwADR-L have no unseen concepts in their test data, SMM4H-17 has a few (43), while MCN has a huge number (2,256). Even a classifier that perfectly learned all concepts in the training data could achieve only  $70.15\%$  accuracy on MCN. MCN also has more unseen mentions:  $53.9\%$ , where the other datasets have less than  $40\%$ . The MCN dataset is thus harder to memorize, as systems must consider many mentions and concepts never seen in training.
Unlike the clinical MCN dataset, in the three social media datasets - AskAPatient, TwADR-L, and SMM4H-17 - it is common for the ADR expressions to share no words with their target medical concepts. For instance, the ADR expression "makes me like a zombie" is assigned the concept "C1443060" with preferred term "feeling abnormal". The social media datasets do not include context, only the mentions themselves, while the MCN dataset provides the entire note surrounding each mention. Since only  $4.5\%$  of mentions in the MCN dataset are ambiguous, for the current experiments we ignore this additional context information.
4.2 Unified Medical Language System
The UMLS Metathesaurus (Bodenreider, 2004) links similar names for the same concept
http://dx.doi.org/10.17632/rxwfb3tysd.1
from nearly 200 different vocabularies such as SNOMED-CT, MedDRA, RxNorm, etc. There are over 3.5 million concepts in UMLS, and for each concept, UMLS also provides the definition, preferred term, synonyms, semantic type, relationships with other concepts, etc.
In our experiments, we make use of synonyms and semantic type information from UMLS. We restrict our concepts to the three vocabularies, MedDRA, SNOMED-CT, and RxNorm in the UMLS version 2017AB. For each concept in the ontologies of the four datasets, we first find its concept unique identifier (CUI) in UMLS. We then extract synonyms and semantic type information according to the CUI. Synonyms (English only) are collected from level 0 terminologies containing vocabulary sources for which no additional license agreements are necessary.
4.3 Evaluation metrics
For all four datasets, the standard evaluation of concept normalization systems is accuracy. For the AskAPatient and TwADR-L datasets, which use 10-fold cross validation, the accuracy metrics are averaged over 10 folds.
We use the BERT-based multi-class classifier as the candidate generator on the three social media datasets AskAPatient, TwADR-L, and SMM4H-17, and the Lucene-based candidate generator for the MCN dataset. In the social media datasets, the number of concepts in the data is small, few test concepts are unseen in the training data, and there is a greater need to match expressions that are morphologically dissimilar from medical concepts. In the clinical MCN dataset, the opposites are true.
For all experiments, we use BioBERT-base (Lee et al., 2019), which further pre-trains BERT on PubMed abstracts (PubMed) and PubMed Central full-text articles (PMC). We use huggingface's pytorch implementation of BERT<sup>5</sup>. We select the best hyper-parameters based on the performance on dev set. See Appendix A.2 for hyperparameter settings.
4.5 Comparisons with related methods
We compare our proposed architecture with the following state-of-the-art systems.
WordCNN Limsopatham and Collier (2016) use convolutional neural networks over pre-trained word embeddings to generate a vector representation for each mention, and then feed these into a softmax layer for multi-class classification.
WordGRU+Attend+TF-IDF Tutubalina et al. (2018) use a bidirectional GRU with attention over pre-trained word embeddings to generate a vector representation for each mention, concatenate such vector representations with the cosine similarities of the TF-IDF vectors between the mention and all other concept names, and then feed the concatenated vector to a softmax layer for multi-class classification.
BERT+TF-IDF Miftahutdinov and Tutubalina (2019) take similar approach as Tutubalina et al. (2018), but use BERT to generate a vector representation for each mention. They concatenate the vector representations with the cosine similarities of the TF-IDF vectors between the mention and all other concept names, and then feed the concatenated vector to a softmax layer for multi-class classification.