|
--- |
|
language: |
|
- english |
|
thumbnail: |
|
tags: |
|
- language model |
|
license: |
|
datasets: |
|
- EMBO/biolang |
|
metrics: |
|
- |
|
--- |
|
|
|
# bio-lm |
|
|
|
## Model description |
|
|
|
This model is a [RoBERTa base pre-trained model](https://huggingface.co/roberta-base) that was further trained using a masked language modeling task on a compendium of english scientific textual examples from the life sciences using the [BioLang dataset](https://huggingface.co/datasets/EMBO/biolang). |
|
|
|
## Intended uses & limitations |
|
|
|
#### How to use |
|
|
|
The intended use of this model is to be fine-tuned for downstream tasks, token classification in particular. |
|
|
|
To have a quick check of the model as-is in a fill-mask task: |
|
|
|
```python |
|
from transformers import pipeline, RobertaTokenizerFast |
|
tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base', max_len=512) |
|
text = "Let us try this model to see if it <mask>." |
|
fill_mask = pipeline( |
|
"fill-mask", |
|
model='EMBO/bio-lm', |
|
tokenizer=tokenizer |
|
) |
|
fill_mask(text) |
|
``` |
|
|
|
#### Limitations and bias |
|
|
|
This model should be fine-tuned on a specifi task like token classification. |
|
The model must be used with the `roberta-base` tokenizer. |
|
|
|
## Training data |
|
|
|
The model was trained with a masked language modeling taskon the [BioLang dataset](https://huggingface.co/datasets/EMBO/biolang) wich includes 12Mio examples from abstracts and figure legends extracted from papers published in life sciences. |
|
|
|
## Training procedure |
|
|
|
The training was run on a NVIDIA DGX Station with 4XTesla V100 GPUs. |
|
|
|
Training code is available at https://github.com/source-data/soda-roberta |
|
|
|
- Command: `python -m lm.train /data/json/oapmc_abstracts_figs/ MLM` |
|
- Tokenizer vocab size: 50265 |
|
- Training data: EMBO/biolang MLM |
|
- Training with: 12005390 examples |
|
- Evaluating on: 36713 examples |
|
- Epochs: 3.0 |
|
- `per_device_train_batch_size`: 16 |
|
- `per_device_eval_batch_size`: 16 |
|
- `learning_rate`: 5e-05 |
|
- `weight_decay`: 0.0 |
|
- `adam_beta1`: 0.9 |
|
- `adam_beta2`: 0.999 |
|
- `adam_epsilon`: 1e-08 |
|
- `max_grad_norm`: 1.0 |
|
- tensorboard run: lm-MLM-2021-01-27T15-17-43.113766 |
|
|
|
End of training: |
|
``` |
|
trainset: 'loss': 0.8653350830078125 |
|
validation set: 'eval_loss': 0.8192330598831177, 'eval_recall': 0.8154601116513597 |
|
``` |
|
|
|
## Eval results |
|
|
|
Eval on test set: |
|
``` |
|
recall: 0.814471959728645 |
|
``` |
|
|
