Update README.md

README.md

@@ -1,3 +1,76 @@
 ---
 license: cc
 ---
+# Bio-ELECTRA Base 1.2m (cased)
+
+Pretrained (from scratch for 1.2 million steps) ELECTRA discriminator model on 2021 Base PubMed abstracts and PMC open access papers with a domain specific word piece vocabulary generated using SentencePiece
+byte-pair-encoding (BPE) model from PubMed abstract texts. This model is case-sensitive: it makes a difference between english and English.
+
+
+# Intended uses & limitations
+This model is mostly intended to be fine-tuned on a downstream biomedical domain task. 
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence to 
+make decisions, such as classification, information retrieval, relation extraction or question answering. 
+
+# Training data
+
+The pretraining corpus was built using 21.2 million PubMed abstracts from the January 2021 baseline distribution. To build the corpus, 
+title and abstract text sentences were extracted resulting in a corpus of 3.6 billion words.
+
+# Training procedure
+
+The training procedure follows the original ELECTRA training.
+
+## Preprocessing
+
+A domain specific vocabulary of size 31,620 is generated using SentencePiece byte-pair-encoding (BPE) model from PubMed abstract texts. 
+The title and abstract text sentences were extracted using an in-house sentence segmenter trained on biomedical text. The sentences are 
+pre-tokenized using an in-house biomedical tokenizer for proper tokenization of biomedical entities such as gene/protein names, 
+organisms, antibodies, cell lines. The SentencePiece BPE vocabulary of word pieces are applied during pre-training 
+to the properly tokenized and segmented sentences. 
+
+## Pretraining
+
+ The model is pretrained on a single 8 core version 3 tensor processing unit (TPU) with 128 GB of RAM for 1,200,000 steps 
+ with a batch size of 256. The first 1,000,000 steps are pre-trained on PubMed abstracts. 
+ After that the model is pre-trained for another 200,000 steps on PMC open access papers. 
+ The training paprameters were the same as the original ELECTRA base model. The model has 110M parameters, 
+ 12 transformers layers with hidden layer size of 768 and 12 attention heads.
+
+
+ # BibTeX entry and citation info
+
+```
+ @inproceedings{ozyurt-etal-2021-detecting,
+    title = "Detecting Anatomical and Functional Connectivity Relations in Biomedical Literature via Language Representation Models",
+    author = "Ozyurt, Ibrahim Burak  and
+      Menke, Joseph  and
+      Bandrowski, Anita  and
+      Martone, Maryann",
+    editor = "Beltagy, Iz  and
+      Cohan, Arman  and
+      Feigenblat, Guy  and
+      Freitag, Dayne  and
+      Ghosal, Tirthankar  and
+      Hall, Keith  and
+      Herrmannova, Drahomira  and
+      Knoth, Petr  and
+      Lo, Kyle  and
+      Mayr, Philipp  and
+      Patton, Robert M.  and
+      Shmueli-Scheuer, Michal  and
+      de Waard, Anita  and
+      Wang, Kuansan  and
+      Wang, Lucy Lu",
+    booktitle = "Proceedings of the Second Workshop on Scholarly Document Processing",
+    month = jun,
+    year = "2021",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.sdp-1.4",
+    doi = "10.18653/v1/2021.sdp-1.4",
+    pages = "27--35",
+    abstract = "Understanding of nerve-organ interactions is crucial to facilitate the development of effective bioelectronic treatments. Towards the end of developing a systematized and computable wiring diagram of the autonomic nervous system (ANS), we introduce a curated ANS connectivity corpus together with several neural language representation model based connectivity relation extraction systems. We also show that active learning guided curation for labeled corpus expansion significantly outperforms randomly selecting connectivity relation candidates minimizing curation effort. Our final relation extraction system achieves $F_1$ = 72.8{\%} on anatomical connectivity and $F_1$ = 74.6{\%} on functional connectivity relation extraction.",
+}
+```