sd-ner

Model description

This model is a RoBERTa base model that was further trained using a masked language modeling task on a compendium of english scientific textual examples from the life sciences using the BioLang dataset. It as then fine-tuned for token classification on the SourceData sd-nlp dataset with the ROLES task to perform pure context-dependent semantic role classification of bioentities.

Intended uses & limitations

How to use

The intended use of this model is to infer the semantic role of gene products (genes and proteins) with regard to the causal hypotheses tested in experiments reported in scientific papers.

To have a quick check of the model:

from transformers import pipeline, RobertaTokenizerFast, RobertaForTokenClassification
example = """<s>The <mask> overexpression in cells caused an increase in <mask> expression.</s>"""
tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base', max_len=512)
model = RobertaForTokenClassification.from_pretrained('EMBO/sd-roles')
ner = pipeline('ner', model, tokenizer=tokenizer)
res = ner(example)
for r in res:
    print(r['word'], r['entity'])

Limitations and bias

The model must be used with the roberta-base tokenizer.

Training data

The model was trained for token classification using the EMBO/sd-panels dataset wich includes manually annotated examples.

Training procedure

The training was run on a NVIDIA DGX Station with 4XTesla V100 GPUs.

Training code is available at https://github.com/source-data/soda-roberta

  • Command: python -m tokcl.train /data/json/sd_panels NER --num_train_epochs=3.5
  • Tokenizer vocab size: 50265
  • Training data: EMBO/biolang MLM
  • Training with 31410 examples.
  • Evaluating on 8861 examples.
  • Training on 15 features: O, I-SMALL_MOLECULE, B-SMALL_MOLECULE, I-GENEPROD, B-GENEPROD, I-SUBCELLULAR, B-SUBCELLULAR, I-CELL, B-CELL, I-TISSUE, B-TISSUE, I-ORGANISM, B-ORGANISM, I-EXP_ASSAY, B-EXP_ASSAY
  • Epochs: 3.5
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • learning_rate: 0.0001
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0

Eval results

On test set with sklearn.metrics:

                precision    recall  f1-score   support

          CELL       0.77      0.81      0.79      3477
     EXP_ASSAY       0.71      0.70      0.71      7049
      GENEPROD       0.86      0.90      0.88     16140
      ORGANISM       0.80      0.82      0.81      2759
SMALL_MOLECULE       0.78      0.82      0.80      4446
   SUBCELLULAR       0.71      0.75      0.73      2125
        TISSUE       0.70      0.75      0.73      1971

     micro avg       0.79      0.82      0.81     37967
     macro avg       0.76      0.79      0.78     37967
  weighted avg       0.79      0.82      0.81     37967
New: fine-tune this model in a few clicks by selecting AutoNLP in the "Train" menu!
Downloads last month
13
Hosted inference API
Token Classification
This model can be loaded on the Inference API on-demand.