license: cc
Bio-ELECTRA Base 1m (cased)
Pretrained (from scratch for one million steps) ELECTRA discriminator model on 2021 Base PubMed abstracts with a domain specific word piece vocabulary generated using SentencePiece byte-pair-encoding (BPE) model from PubMed abstract texts. This model is case-sensitive: it makes a difference between english and English.
Intended uses & limitations
This model is mostly intended to be fine-tuned on a downstream biomedical domain task.
Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence to make decisions, such as classification, information retrieval, relation extraction or question answering.
Training data
The pretraining corpus was built using 21.2 million PubMed abstracts from the January 2021 baseline distribution. To build the corpus, title and abstract text sentences were extracted resulting in a corpus of 3.6 billion words.
Training procedure
The training procedure follows the original ELECTRA training.
Preprocessing
A domain specific vocabulary of size 31,620 is generated using SentencePiece byte-pair-encoding (BPE) model from PubMed abstract texts. The title and abstract text sentences were extracted using an in-house sentence segmenter trained on biomedical text. The sentences are pre-tokenized using an in-house biomedical tokenizer for proper tokenization of biomedical entities such as gene/protein names, organisms, antibodies, cell lines. The SentencePiece BPE vocabulary of word pieces are applied during pre-training to the properly tokenized and segmented sentences.
Pretraining
The model is pretrained on a single 8 core version 3 tensor processing unit (TPU) with 128 GB of RAM for 1,000,000 steps with a batch size of 256. The training paprameters were the same as the original ELECTRA base model. The model has 110M parameters, 12 transformers layers with hidden layer size of 768 and 12 attention heads.
BibTeX entry and citation info
@inproceedings{ozyurt-etal-2021-detecting,
title = "Detecting Anatomical and Functional Connectivity Relations in Biomedical Literature via Language Representation Models",
author = "Ozyurt, Ibrahim Burak and
Menke, Joseph and
Bandrowski, Anita and
Martone, Maryann",
editor = "Beltagy, Iz and
Cohan, Arman and
Feigenblat, Guy and
Freitag, Dayne and
Ghosal, Tirthankar and
Hall, Keith and
Herrmannova, Drahomira and
Knoth, Petr and
Lo, Kyle and
Mayr, Philipp and
Patton, Robert M. and
Shmueli-Scheuer, Michal and
de Waard, Anita and
Wang, Kuansan and
Wang, Lucy Lu",
booktitle = "Proceedings of the Second Workshop on Scholarly Document Processing",
month = jun,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.sdp-1.4",
doi = "10.18653/v1/2021.sdp-1.4",
pages = "27--35",
abstract = "Understanding of nerve-organ interactions is crucial to facilitate the development of effective bioelectronic treatments. Towards the end of developing a systematized and computable wiring diagram of the autonomic nervous system (ANS), we introduce a curated ANS connectivity corpus together with several neural language representation model based connectivity relation extraction systems. We also show that active learning guided curation for labeled corpus expansion significantly outperforms randomly selecting connectivity relation candidates minimizing curation effort. Our final relation extraction system achieves $F_1$ = 72.8{\%} on anatomical connectivity and $F_1$ = 74.6{\%} on functional connectivity relation extraction.",
}