|
--- |
|
license: cc |
|
--- |
|
# Bio-ELECTRA Base 1.2m (cased) |
|
|
|
Pretrained (from scratch for 1.2 million steps) ELECTRA discriminator model on 2021 Base PubMed abstracts and PMC open access papers with a domain specific word piece vocabulary generated using SentencePiece |
|
byte-pair-encoding (BPE) model from PubMed abstract texts. This model is case-sensitive: it makes a difference between english and English. |
|
|
|
|
|
# Intended uses & limitations |
|
This model is mostly intended to be fine-tuned on a downstream biomedical domain task. |
|
|
|
Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence to |
|
make decisions, such as classification, information retrieval, relation extraction or question answering. |
|
|
|
# Training data |
|
|
|
The pretraining corpus was built using 21.2 million PubMed abstracts from the January 2021 baseline distribution. To build the corpus, |
|
title and abstract text sentences were extracted resulting in a corpus of 3.6 billion words. The PMC open access corpus (January 2021) is |
|
a 12.3 billion words corpus built using the sentences extracted from the sections of PMC open access papers |
|
excluding the references sections. |
|
|
|
# Training procedure |
|
|
|
The training procedure follows the original ELECTRA training. |
|
|
|
## Preprocessing |
|
|
|
A domain specific vocabulary of size 31,620 is generated using SentencePiece byte-pair-encoding (BPE) model from PubMed abstract texts. |
|
The title and abstract text sentences were extracted using an in-house sentence segmenter trained on biomedical text. The sentences are |
|
pre-tokenized using an in-house biomedical tokenizer for proper tokenization of biomedical entities such as gene/protein names, |
|
organisms, antibodies, cell lines. The SentencePiece BPE vocabulary of word pieces are applied during pre-training |
|
to the properly tokenized and segmented sentences. For the PMC open access corpus, JATS XML files for the full text papers are parsed |
|
to extract sections excluding the reference section and section title and section body is processed in the same fashion |
|
as the PubMed abstracts corpus. |
|
|
|
## Pretraining |
|
|
|
The model is pretrained on a single 8 core version 3 tensor processing unit (TPU) with 128 GB of RAM for 1,200,000 steps |
|
with a batch size of 256. The first 1,000,000 steps are pre-trained on PubMed abstracts. |
|
After that, the model is pre-trained for another 200,000 steps on PMC open access papers. |
|
The training parameters were the same as the original ELECTRA base model. The model has 110M parameters, |
|
12 transformers layers with hidden layer size of 768 and 12 attention heads. |
|
|
|
|
|
# BibTeX entry and citation info |
|
|
|
``` |
|
@inproceedings{ozyurt-etal-2021-detecting, |
|
title = "Detecting Anatomical and Functional Connectivity Relations in Biomedical Literature via Language Representation Models", |
|
author = "Ozyurt, Ibrahim Burak and |
|
Menke, Joseph and |
|
Bandrowski, Anita and |
|
Martone, Maryann", |
|
editor = "Beltagy, Iz and |
|
Cohan, Arman and |
|
Feigenblat, Guy and |
|
Freitag, Dayne and |
|
Ghosal, Tirthankar and |
|
Hall, Keith and |
|
Herrmannova, Drahomira and |
|
Knoth, Petr and |
|
Lo, Kyle and |
|
Mayr, Philipp and |
|
Patton, Robert M. and |
|
Shmueli-Scheuer, Michal and |
|
de Waard, Anita and |
|
Wang, Kuansan and |
|
Wang, Lucy Lu", |
|
booktitle = "Proceedings of the Second Workshop on Scholarly Document Processing", |
|
month = jun, |
|
year = "2021", |
|
address = "Online", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2021.sdp-1.4", |
|
doi = "10.18653/v1/2021.sdp-1.4", |
|
pages = "27--35", |
|
abstract = "Understanding of nerve-organ interactions is crucial to facilitate the development of effective bioelectronic treatments. Towards the end of developing a systematized and computable wiring diagram of the autonomic nervous system (ANS), we introduce a curated ANS connectivity corpus together with several neural language representation model based connectivity relation extraction systems. We also show that active learning guided curation for labeled corpus expansion significantly outperforms randomly selecting connectivity relation candidates minimizing curation effort. Our final relation extraction system achieves $F_1$ = 72.8{\%} on anatomical connectivity and $F_1$ = 74.6{\%} on functional connectivity relation extraction.", |
|
} |
|
``` |
