---
language:
- en
license: apache-2.0
---

[`bioformer-8L`](https://huggingface.co/bioformers/bioformer-8L) pretrained on 164,179 COVID-19 abstracts (from [LitCovid website](https://www.ncbi.nlm.nih.gov/research/coronavirus/)) for 100 epochs. 

In our evaluation, this pretraining process leads to improved performance on the multi-label COVID-19 topic classification task (BioCreative VII track 5).