Using QuickUMLS for biomedical named entity recognition | Towards Data Science

At the outset of any new R&D project, I look to match the most appropriate solution to the problem at hand. While there are some very exciting large models available, one of my biggest concerns is bringing the solution into production without compromising on the outcomes I’d like to support. A question I had recently was if a simple string-matching method — coupled with some simple optimizations — could compete with a supervised model on a biomedical Named Entity Recognition (NER) task. I have put these two methods in a head-to-head test.
On the string-matching system side, I leveraged the QuickUMLS classifier. QuickUMLS [1] — as a string-matching system — takes a string as input (e.g., a paper or abstract of a paper containing medical concepts), and outputs all spans from the document that match Unified Medical Language System (UMLS) concepts. These concepts can then be re-used in other settings, or as input to other machine learning systems. For this reason, QuickUMLS can be seen as a handy pre-processing tool to get relevant concepts from clinical and biomedical text. In this blog post, however, we will focus on using QuickUMLS as a classifier on the challenging MedMentions dataset. [2]
Before we dive into the problem we’re trying to solve, it is useful to describe some idiosyncrasies of biomedical NER. In general, the problem of NER is to find named entities (e.g., notable locations, persons, organizations, etc.) in text. As you can probably surmise, many of these entities can be found through context. For example, in a sentence such as: “Sam and Drew went to the Colosseum”, we can infer that the Colosseum is a location because you generally go to locations. Similarly, we can also surmise that “Sam” is a proper name because words that are in the subject position of “to go” that aren’t commonplace words tend to be names.
In contrast, biomedical NER is about finding and disambiguating biomedical terms of interest from text, such as diseases, drug names, but also generic terms such as “hospital”, “ICU ward”, or “alcohol”. This is an important distinction, since there is very little contextual information that determines whether a given word is of medical importance. To give a bit of a loaded example, consider the word “alcohol” in the sentence “the patient drank a lot of alcohol”. The severity of this finding depends on whether it refers to alcohol such as beer or wine, or pure alcohol, such as rubbing alcohol. For a more complete overview of the state of the art of biomedical NER, see this blog post by my colleague at Slimmer AI, Sybren Jansen.
Knowing which concepts are of medical importance is difficult to learn without a large amount of training data, which is not typically readily available. Many systems, therefore, use the Unified Medical Language System (UMLS), which is a large ontology containing many different concepts together with their string representations and other information. Please note that a concept is distinct from a string because many strings can refer to more than one concept. The string “alcohol” for example, can refer to rubbing alcohol or to alcoholic drinks.
In UMLS, each concept is described by a Concept Unique Identifier (CUI), which is a symbolic ID for any given unique concept, and a Semantic Type (STY), which is a family identifier that groups concepts with similar characteristics. One reason the UMLS is useful, but also challenging to work with, is its sheer size; the 2020AB version of the UMLS, which is the one we will use in what follows, has over 3 million unique English concepts. It is unlikely that a sizable portion of these concepts will show up, even in large, annotated datasets.
One such dataset is MedMentions. It consists of 4,392 Pubmed papers (titles and abstracts) from 2016; annotated with 352K concepts (CUI IDs) and semantic types from the UMLS. The papers have about 34K annotated unique concepts, which is still only about 1% of the total number of concepts in the UMLS. This shows that annotating UMLS mentions is a challenging task that cannot necessarily be solved using supervised machine learning.
Of particular interest in this regard is that the MedMentions corpus includes CUIs in the test set that do not occur in training. In general, however, this task is still approached as a supervised machine learning task by using the semantic types of the UMLS concepts as labels. As UMLS has 127 semantic types, this still leads to a large label space. The MedMentions dataset also has a smaller version, the st21pv dataset, which consists of the same documents as the regular dataset, but with only the 21 most frequent semantic types annotated.
A semi-Markov baseline gets about 45.3 F-score on the entity level. [2] Other approaches, including BlueBERT [3] and BioBERT [4] were tested, and improved the score to 56.3 using exact matching on the entity level. [5] Note that all these approaches are supervised, and thus rely on a certain amount of overlap between the train and test set in terms of concepts. If a concept or label has never been seen during training, a supervised machine learning approach will have a challenging time classifying it correctly. In what follows, we will use the semantic types of the MedMentions dataset as labels.
In contrast to BERT, QuickUMLS is inherently an unsupervised method, which means that it doesn’t rely on training data. More accurately, QuickUMLS is a knowledge-based method. This means that the model, instead of having parameters that tell it what to predict, relies on an external knowledge base to predict labels. This implies two things:
The quality of the model is limited by the quality of the knowledge base. The model cannot predict things that the knowledge base does not contain.The model can generalize beyond annotated data. A supervised model that has not seen a specific label during training cannot predict those things accurately, in general. An exception to this rule is zero-shot learning methods.
The quality of the model is limited by the quality of the knowledge base. The model cannot predict things that the knowledge base does not contain.
The model can generalize beyond annotated data. A supervised model that has not seen a specific label during training cannot predict those things accurately, in general. An exception to this rule is zero-shot learning methods.
From these two facts, we argue that knowledge-based methods are a good fit for the MedMentions dataset. Regarding the first point, the MedMentions database was annotated using UMLS concepts, so the mapping between the knowledge base and the dataset is an exact mapping. Regarding the second point, the MedMentions dataset contains concepts in test that are not present in the training set.
QuickUMLS, as a model, is simple. It first parses the text using a parser, spacy. The model then selects word ngrams, i.e., sequential sequences of words, based on part of speech tag patterns and stopword lists. In brief, this means that the model discards certain word ngrams if they contain unwanted tokens and punctuation marks. Details of these rules can be found in the original paper. [1] After selecting all word ngram candidates, the whole UMLS database is queried for concepts that partially match the word ngrams. Because exact matching on such a huge database is inefficient and difficult, the authors perform approximate string matching using simstring. [6] When given a text, QuickUMLS thus returns a list of concepts in the UMLS, together with their similarity to the query string and other associated information. For example, the text: “The patient had a hemorrhage” returns the following candidates, using a (default) string similarity threshold of 0.7:
For the word patient:
{‘term’: ‘Inpatient’, ‘cui’: ‘C1548438’, ‘similarity’: 0.71, ‘semtypes’: {‘T078’}, ‘preferred’: 1},{‘term’: ‘Inpatient’, ‘cui’: ‘C1549404’, ‘similarity’: 0.71, ‘semtypes’: {‘T078’}, ‘preferred’: 1},{‘term’: ‘Inpatient’, ‘cui’: ‘C1555324’, ‘similarity’: 0.71, ‘semtypes’: {‘T058’}, ‘preferred’: 1},{‘term’: ‘*^patient’, ‘cui’: ‘C0030705’, ‘similarity’: 0.71, ‘semtypes’: {‘T101’}, ‘preferred’: 1},{‘term’: ‘patient’, ‘cui’: ‘C0030705’, ‘similarity’: 1.0, ‘semtypes’: {‘T101’}, ‘preferred’: 0},{‘term’: ‘inpatient’, ‘cui’: ‘C0021562’, ‘similarity’: 0.71, ‘semtypes’: {‘T101’}, ‘preferred’: 0}
For the word hemorrhage:
{‘term’: ‘No hemorrhage’, ‘cui’: ‘C1861265’, ‘similarity’: 0.72, ‘semtypes’: {‘T033’}, ‘preferred’: 1},{‘term’: ‘hemorrhagin’, ‘cui’: ‘C0121419’, ‘similarity’: 0.7, ‘semtypes’: {‘T116’, ‘T126’}, ‘preferred’: 1},{‘term’: ‘hemorrhagic’, ‘cui’: ‘C0333275’, ‘similarity’: 0.7, ‘semtypes’: {‘T080’}, ‘preferred’: 1},{‘term’: ‘hemorrhage’, ‘cui’: ‘C0019080’, ‘similarity’: 1.0, ‘semtypes’: {‘T046’}, ‘preferred’: 0},{‘term’: ‘GI hemorrhage’, ‘cui’: ‘C0017181’, ‘similarity’: 0.72, ‘semtypes’: {‘T046’}, ‘preferred’: 0},{‘term’: ‘Hemorrhages’, ‘cui’: ‘C0019080’, ‘similarity’: 0.7, ‘semtypes’: {‘T046’}, ‘preferred’: 0}
As you can see, the word “patient” has three matches with the correct semantic type (T101), and two matches with the correct concept (C0030705). The word hemorrhage also has superfluous matches, including the concept for “No hemorrhage”. Nevertheless, the highest ranked candidate, if we go by similarity, is the correct one in both cases.
In a default application of QuickUMLS, we only keep preferred terms, i.e., terms for which preferred is 1, and then sort by similarity. We then take the semantic type (i.e., semtype) of the highest ranked candidate as the prediction- we call this the baseline model. We used seqeval with a strict matching paradigm which is comparable to previous work. [5]
╔═══╦══════╦═══════╗║   ║ BERT ║ QUMLS ║╠═══╬══════╬═══════╣║ P ║  .53 ║   .27 ║║ R ║  .58 ║   .36 ║║ F ║  .56 ║   .31 ║╚═══╩══════╩═══════╝Table 1: baseline performance
Not too impressive, right? Unfortunately, the baseline suffers from a bad case of not being optimized for specific task. As such, let’s optimize the baseline using simple heuristics.
There are a couple of ways to improve QuickUMLS beyond its initial performance. First, we note that the standard parser used by QuickUMLS is the default spacy model, i.e., en_core_web_sm. Given that we are dealing with biomedical text, we are better served by using a biomedical language model. In our case, we swapped out the model to a scispacy [7] model, en_core_sci_sm. This already improves performance a little bit, at no cost.
╔═══╦══════╦═══════╦═════════╗║   ║ BERT ║ QUMLS ║ + Spacy ║╠═══╬══════╬═══════╬═════════╣║ P ║  .53 ║   .27 ║     .29 ║║ R ║  .58 ║   .36 ║     .37 ║║ F ║  .56 ║   .31 ║     .32 ║╚═══╩══════╩═══════╩═════════╝Table 2: adding scispacy
Other improvements can be gained by using some information from the training corpus. While this does turn QuickUMLS from a purely unsupervised method into a supervised one, there is still no dependence on a large body of specific annotations. In other words, there is no explicit “fit” step on a specific corpus: the improvements we are going to make can also be estimated using a small set of annotations or prior knowledge by a physician.
The default settings for QuickUMLS, include a threshold of 0.7, and a set of metrics. The metric determines how the string similarity is counted, and can be set to “Jaccard”, “cosine”, “overlap” and “dice”. We perform a grid search over the metric and different thresholds. The best results ended up being a threshold of .99, meaning we only perform exact matches using SimString, and the “Jaccard” metric, which outperforms all other options in terms of speed and score. As you can see, we are getting closer and closer to the performance of BERT.
╔═══╦══════╦═══════╦═════════╦════════╗║   ║ BERT ║ QUMLS ║ + Spacy ║ + Grid ║╠═══╬══════╬═══════╬═════════╬════════╣║ P ║  .53 ║   .27 ║     .29 ║    .37 ║║ R ║  .58 ║   .36 ║     .37 ║    .37 ║║ F ║  .56 ║   .31 ║     .32 ║    .37 ║╚═══╩══════╩═══════╩═════════╩════════╝Table 3: Grid searching over settings
Recall that above, we simply selected the best matching candidate based on whether it was a preferred string and their similarity. However, in many cases different concepts will have the same string representation, such as in the aforementioned “alcohol” example. This makes it difficult to choose an optimal candidate without having a disambiguation model, which requires context, and again turns the learning problem into a supervised one, or at least one that requires examples of contexts in which terms occur. One easy way out of this conundrum is to consider that, all else being equal, some semantic types are simply more likely, and therefore more likely in the given corpus. This is also called a prior.
In our case, we estimate the class prior through the training set, as you would do in the well-known naive bayes classifier. Then, for each semantic type we extract for each set of candidates, we take the maximum similarity, and then multiply it with the prior. In neural network terms, you can think of this of max pooling on the class level. This also means we ignore the preferred term status of any candidates.
╔═══╦══════╦═══════╦═════════╦════════╦══════════╗║   ║ BERT ║ QUMLS ║ + Spacy ║ + Grid ║ + Priors ║╠═══╬══════╬═══════╬═════════╬════════╬══════════╣║ P ║  .53 ║   .27 ║     .29 ║    .37 ║      .39 ║║ R ║  .58 ║   .36 ║     .37 ║    .37 ║      .39 ║║ F ║  .56 ║   .31 ║     .32 ║    .37 ║      .39 ║╚═══╩══════╩═══════╩═════════╩════════╩══════════╝Table 4: adding priors
Unfortunately, this is as far as we could get using the simple system in QuickUMLS. Given that we ended up using a threshold of .99 means that we do not use the approximate matching functionality of QuickUMLS at all. Removing the approximate matching would also speed up the entire system immensely, since most of the time of the algorithm is now spent on the matching in QuickUMLS.
As we are doing a Named Entity Recognition task, there are a couple of classes of errors we can make. First off, we can extract the right span, but the wrong class. This happens when we find a correct term, but give it the wrong label, e.g., we think “alcohol” refers to the drink when it refers to the disinfectant. Second, we can also extract the span partially, but still match the right label. In this case, we can think of the match as a “partial match”. In our scoring, we only counted exact matches as correct. An example of this is extracting “mild anesthetic” when the gold standard is “anesthetic”. We can also miss spans completely, e.g., because UMLS does not contain the term, or extract spans that do not correspond to gold standard mentions. The figure below shows what kinds of these errors our system makes:
This shows that the errors QuickUMLS makes do not belong to one specific category. It extracts too many items, but, when it extracts items, it also often assigns the wrong label to those items. This shows that QuickUMLS could be employed as a pre-extraction system, after which a disambiguation system could be used to assign the correct label.
As you can see from the results, an off-the-shelf terminology extraction system can be used as an efficient and effective NER system without training. Getting training data for specific use cases can often be a time-consuming process, hampering R&D speed. The QuickUMLS classifier we built shows that we can go a long way with very few training examples. And by being smart about the way we apply our resources, we have saved lot of time in the R&D process for biomedical NER. Our modified QuickUMLS classifier can be tried out here on github. The benefit of this approach can mean a solution robust enough for the outcome, simple to develop and test, and small enough for easy implementation into product development.
[1] L. Soldaini, and N. Goharian. Quickumls: a fast, unsupervised approach for medical concept extraction, (2016), MedIR workshop, SIGIR
[2] S. Mohan, and D. Li, Medmentions: a large biomedical corpus annotated with UMLS concepts, (2019), arXiv preprint arXiv:1902.09476
[3] Y. Peng, Q. Chen, and Z. Lu, An empirical study of multi-task learning on BERT for biomedical text mining, (2020), arXiv preprint arXiv:2005.02799
[4] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C.H. So, and J. Kang, BioBERT: a pre-trained biomedical language representation model for biomedical text mining, (2020), Bioinformatics, 36(4)
[5] K.C. Fraser, I. Nejadgholi, B. De Bruijn, M. Li, A. LaPlante and K.Z.E. Abidine, Extracting UMLS concepts from medical text using general and domain-specific deep learning models, (2019), arXiv preprint arXiv:1910.01274.
[6] N. Okazaki, and J.I. Tsujii, Simple and efficient algorithm for approximate dictionary matching, (2010, August), In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)
[7] M. Neumann, D. King, I. Beltagy, and W. Ammar, Scispacy: Fast and robust models for biomedical natural language processing, (2019), arXiv preprint arXiv:1902.07669.