Abstract: Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We make the datasets, pre-trained models, and codes publicly available at https://github.com/ncbi-nlp/BLUE_Benchmark.

Introduction: With the growing amount of biomedical information available in textual form, there have been significant advances in the development of pre-training language representations that can be applied to a range of different tasks in the biomedical domain, such as pre-trained word embeddings, sentence embeddings, and contextual representations BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . In the general domain, we have recently observed that the General Language Understanding Evaluation (GLUE) benchmark BIBREF5 has been successfully promoting the development of language representations of general purpose BIBREF2 , BIBREF6 , BIBREF7 . To the best of our knowledge, however, there is no publicly available benchmarking in the biomedicine domain. To facilitate research on language representations in the biomedicine domain, we present the Biomedical Language Understanding Evaluation (BLUE) benchmark, which consists of five different biomedicine text-mining tasks with ten corpora. Here, we rely on preexisting datasets because they have been widely used by the BioNLP community as shared tasks BIBREF8 . These tasks cover a diverse range of text genres (biomedical literature and clinical notes), dataset sizes, and degrees of difficulty and, more importantly, highlight common biomedicine text-mining challenges. We expect that the models that perform better on all or most tasks in BLUE will address other biomedicine tasks more robustly. To better understand the challenge posed by BLUE, we conduct experiments with two baselines: One makes use of the BERT model BIBREF7 and one makes use of ELMo BIBREF2 . Both are state-of-the-art language representation models and demonstrate promising results in NLP tasks of general purpose. We find that the BERT model pre-trained on PubMed abstracts BIBREF9 and MIMIC-III clinical notes BIBREF10 achieves the best results, and is significantly superior to other models in the clinical domain. This demonstrates the importance of pre-training among different text genres. In summary, we offer: (i) five tasks with ten biomedical and clinical text-mining corpora with different sizes and levels of difficulty, (ii) codes for data construction and model evaluation for fair comparisons, (iii) pretrained BERT models on PubMed abstracts and MIMIC-III, and (iv) baseline results.

Question: Why does the model improve in monolingual spaces as well? 
Answer: because word pair similarity increases if the two words translate to similar parts of the cross-lingual embedding space