Before revision:  We introduce the method to train a BERT model on a small medical corpus both in English and in Japanese , and we present the evaluation of each model in terms of the biomedical language understanding evaluation (BLUE) benchmark and the medical document classification task in Japanese, respectively. After confirming their satisfactory performances, we applied our method to develop a modelcomparable to the publicly available models. OuBioBERT, short for Bidirectional Encoder Representations from Transformers for Biomedical Text Mining by Osaka University, achieved the best score in terms of the BLUE benchmark .
After revision:  We introduce the method to train a BERT model on a small medical corpus both in English and in Japanese , and we present the evaluation of each model in terms of the biomedical language understanding evaluation (BLUE) benchmark . Moreover, our enhanced biomedical BERT model, in which clinical notes were not used during pre-training, showed that both the clinical and biomedical scores of the BLUE benchmark .