Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

Abstract: Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We make the datasets, pre-trained models, and codes publicly available at https://github.com/ncbi-nlp/BLUE_Benchmark.

With the growing amount of biomedical information available in textual form, there have been significant advances in the development of pre-training language representations that can be applied to a range of different tasks in the biomedical domain, such as pre-trained word embeddings, sentence embeddings, and contextual representations BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 .
In the general domain, we have recently observed that the General Language Understanding Evaluation (GLUE) benchmark BIBREF5 has been successfully promoting the development of language representations of general purpose BIBREF2 , BIBREF6 , BIBREF7 . To the best of our knowledge, however, there is no publicly available benchmarking in the biomedicine domain.
To facilitate research on language representations in the biomedicine domain, we present the Biomedical Language Understanding Evaluation (BLUE) benchmark, which consists of five different biomedicine text-mining tasks with ten corpora. Here, we rely on preexisting datasets because they have been widely used by the BioNLP community as shared tasks BIBREF8 . These tasks cover a diverse range of text genres (biomedical literature and clinical notes), dataset sizes, and degrees of difficulty and, more importantly, highlight common biomedicine text-mining challenges. We expect that the models that perform better on all or most tasks in BLUE will address other biomedicine tasks more robustly.
To better understand the challenge posed by BLUE, we conduct experiments with two baselines: One makes use of the BERT model BIBREF7 and one makes use of ELMo BIBREF2 . Both are state-of-the-art language representation models and demonstrate promising results in NLP tasks of general purpose. We find that the BERT model pre-trained on PubMed abstracts BIBREF9 and MIMIC-III clinical notes BIBREF10 achieves the best results, and is significantly superior to other models in the clinical domain. This demonstrates the importance of pre-training among different text genres.
In summary, we offer: (i) five tasks with ten biomedical and clinical text-mining corpora with different sizes and levels of difficulty, (ii) codes for data construction and model evaluation for fair comparisons, (iii) pretrained BERT models on PubMed abstracts and MIMIC-III, and (iv) baseline results.
There is a long history of using shared language representations to capture text semantics in biomedical text and data mining research. Such research utilizes a technique, termed transfer learning, whereby the language representations are pre-trained on large corpora and fine-tuned in a variety of downstream tasks, such as named entity recognition and relation extraction.
One established trend is a form of word embeddings that represent the semantic, using high dimensional vectors BIBREF0 , BIBREF11 , BIBREF12 . Similar methods also have been derived to improve embeddings of word sequences by introducing sentence embeddings BIBREF1 . They always, however, require complicated neural networks to be effectively used in downstream applications.
Another popular trend, especially in recent years, is the context-dependent representation. Different from word embeddings, it allows the meaning of a word to change according to the context in which it is used BIBREF13 , BIBREF2 , BIBREF7 , BIBREF14 . In the scientific domain, BIBREF15 released SciBERT which is trained on scientific text. In the biomedical domain, BioBERT BIBREF3 and BioELMo BIBREF16 were pre-trained and applied to several specific tasks. In the clinical domain, BIBREF17 released a clinical BERT base model trained on the MIMIC-III database. Most of these works, however, were evaluated on either different datasets or the same dataset with slightly different sizes of examples. This makes it challenging to fairly compare various language models.
Based on these reasons, a standard benchmarking is urgently required. Parallel to our work, BIBREF3 introduced three tasks: named entity recognition, relation extraction, and QA, while BIBREF16 introduced NLI in addition to named entity recognition. To this end, we deem that BLUE is different in three ways. First, BLUE is selected to cover a diverse range of text genres, including both biomedical and clinical domains. Second, BLUE goes beyond sentence or sentence pairs by including document classification tasks. Third, BLUE provides a comprehensive suite of codes to reconstruct dataset from scratch without removing any instances.
BLUE contains five tasks with ten corpora that cover a broad range of data quantities and difficulties (Table 1 ). Here, we rely on preexisting datasets because they have been widely used by the BioNLP community as shared tasks.
The sentence similarity task is to predict similarity scores based on sentence pairs. Following common practice, we evaluate similarity by using Pearson correlation coefficients.
BIOSSES is a corpus of sentence pairs selected from the Biomedical Summarization Track Training Dataset in the biomedical domain BIBREF18 . To develop BIOSSES, five curators judged their similarity, using scores that ranged from 0 (no relation) to 4 (equivalent). Here, we randomly select 80% for training and 20% for testing because there is no standard splits in the released data.
MedSTS is a corpus of sentence pairs selected from Mayo Clinic’s clinical data warehouse BIBREF19 . To develop MedSTS, two medical experts graded the sentence's semantic similarity scores from 0 to 5 (low to high similarity). We use the standard training and testing sets in the shared task.
The aim of the named entity recognition task is to predict mention spans given in the text BIBREF20 . The results are evaluated through a comparison of the set of mention spans annotated within the document with the set of mention spans predicted by the model. We evaluate the results by using the strict version of precision, recall, and F1-score. For disjoint mentions, all spans also must be strictly correct. To construct the dataset, we used spaCy to split the text into a sequence of tokens when the original datasets do not provide such information.
BC5CDR is a collection of 1,500 PubMed titles and abstracts selected from the CTD-Pfizer corpus and was used in the BioCreative V chemical-disease relation task BIBREF21 . The diseases and chemicals mentioned in the articles were annotated independently by two human experts with medical training and curation experience. We use the standard training and test set in the BC5CDR shared task BIBREF22 .
ShARe/CLEF eHealth Task 1 Corpus is a collection of 299 deidentified clinical free-text notes from the MIMIC II database BIBREF23 . The disorders mentioned in the clinical notes were annotated by two professionally trained annotators, followed by an adjudication step, resulting in high inter-annotator agreement. We use the standard training and test set in the ShARe/CLEF eHealth Tasks 1.
The aim of the relation extraction task is to predict relations and their types between the two entities mentioned in the sentences. The relations with types were compared to annotated data. We use the standard micro-average precision, recall, and F1-score metrics.
DDI extraction 2013 corpus is a collection of 792 texts selected from the DrugBank database and other 233 Medline abstracts BIBREF24 . The drug-drug interactions, including both pharmacokinetic and pharmacodynamic interactions, were annotated by two expert pharmacists with a substantial background in pharmacovigilance. In our benchmark, we use 624 train files and 191 test files to evaluate the performance and report the micro-average F1-score of the four DDI types.
ChemProt consists of 1,820 PubMed abstracts with chemical-protein interactions annotated by domain experts and was used in the BioCreative VI text mining chemical-protein interactions shared task BIBREF25 . We use the standard training and test sets in the ChemProt shared task and evaluate the same five classes: CPR:3, CPR:4, CPR:5, CPR:6, and CPR:9.
i2b2 2010 shared task collection consists of 170 documents for training and 256 documents for testing, which is the subset of the original dataset BIBREF26 . The dataset was collected from three different hospitals and was annotated by medical practitioners for eight types of relations between problems and treatments.
The multilabel classification task predicts multiple labels from the texts.
HoC (the Hallmarks of Cancers corpus) consists of 1,580 PubMed abstracts annotated with ten currently known hallmarks of cancer BIBREF27 . Annotation was performed at sentence level by an expert with 15+ years of experience in cancer research. We use 315 ( $\sim $ 20%) abstracts for testing and the remaining abstracts for training. For the HoC task, we followed the common practice and reported the example-based F1-score on the abstract level BIBREF28 , BIBREF29 .
The aim of the inference task is to predict whether the premise sentence entails or contradicts the hypothesis sentence. We use the standard overall accuracy to evaluate the performance.
MedNLI is a collection of sentence pairs selected from MIMIC-III BIBREF30 . Given a premise sentence and a hypothesis sentence, two board-certified radiologists graded whether the task predicted whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). We use the same training, development, and test sets in Romanov and Shivade BIBREF30 .
Following the practice in BIBREF5 and BIBREF3 , we use a macro-average of F1-scores and Pearson scores to determine a system's position.
For baselines, we evaluate several pre-training models as described below. The original code for the baselines is available at https://github.com/ncbi-nlp/NCBI_BERT.
BERT BIBREF7 is a contextualized word representation model that is pre-trained based on a masked language model, using bidirectional Transformers BIBREF31 .
In this paper, we pre-trained our own model BERT on PubMed abstracts and clinical notes (MIMIC-III). The statistics of the text corpora on which BERT was pre-trained are shown in Table 2 .
We initialized BERT with pre-trained BERT provided by BIBREF7 . We then continue to pre-train the model, using the listed corpora.
We released our BERT-Base and BERT-Large models, using the same vocabulary, sequence length, and other configurations provided by BIBREF7 . Both models were trained with 5M steps on the PubMed corpus and 0.2M steps on the MIMIC-III corpus.
BERT is applied to various downstream text-mining tasks while requiring only minimal architecture modification.
For sentence similarity tasks, we packed the sentence pairs together into a single sequence, as suggested in BIBREF7 .
For named entity recognition, we used the BIO tags for each token in the sentence. We considered the tasks similar to machine translation, as predicting the sequence of BIO tags from the input sentence.
We treated the relation extraction task as a sentence classification by replacing two named entity mentions of interest in the sentence with pre-defined tags (e.g., @GENE$, @DRUG$) BIBREF3 . For example, we used “@CHEMICAL$ protected against the RTI-76-induced inhibition of @GENE$ binding.” to replace the original sentence “Citalopram protected against the RTI-76-induced inhibition of SERT binding.” in which “citalopram” and “SERT” has a chemical-gene relation.
For multi-label tasks, we fine-tuned the model to predict multi-labels for each sentence in the document. We then combine the labels in one document and compare them with the gold-standard.
Like BERT, we provided sources code for fine-tuning, prediction, and evaluation to make it straightforward to follow those examples to use our BERT pre-trained models for all tasks.

Figure and table captions:
Table 1: BLUE tasks
Table 2: Corpora
Table 3: Baseline performance on the BLUE task test sets.

Questions about this paper:
Could you tell me more about the metrics used for performance evaluation?

Answers:
BLUE utilizes different metrics for each of the tasks: Pearson correlation coefficient, F-1 scores, micro-averaging, and accuracy