|
--- |
|
license: cc-by-nc-2.0 |
|
library_name: transformers |
|
datasets: |
|
- CCDS |
|
- Ensembl |
|
pipeline_tag: fill-mask |
|
tags: |
|
- protein language model |
|
- biology |
|
widget: |
|
- text: >- |
|
( Z [MASK] V L P Y G D E K L S P Y G D G G D V G Q I F s C B L Q D T N N F F G A |
|
g Q N K % O P K L G Q I G % S K % u u i e d d R i d D V L k n ( T D K @ p p |
|
^ v |
|
example_title: Fill mask (E) |
|
--- |
|
|
|
# cdsBERT |
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/62f2bd3bdb7cbd214b658c48/yA-f7tnvNNV52DK2QYNq_.png" width="350"> |
|
|
|
## Model description |
|
|
|
[cdsBERT](https://doi.org/10.1101/2023.09.15.558027) is a pLM with a codon vocabulary that was seeded with [ProtBERT](https://huggingface.co/Rostlab/prot_bert_bfd) and trained with a novel vocabulary extension pipeline called MELD. cdsBERT offers a highly biologically relevant latent space with excellent EC number prediction. |
|
Specifically, this is the full-precision checkpoint after the MLM objective on 4 million CDS examples. |
|
|
|
## How to use |
|
|
|
```python |
|
# Imports |
|
import torch |
|
from transformers import BertForMaskedLM, BertTokenizer, pipeline |
|
|
|
model = BertForMaskedLM.from_pretrained('lhallee/cdsBERT') # load model |
|
tokenizer = BertTokenizer.from_pretrained('lhallee/cdsBERT') # load tokenizer |
|
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') # gather device |
|
model.to(device) # move to device |
|
model.eval() # put in eval mode |
|
|
|
sequence = '( Z [MASK] V L P Y G D E K L S P Y G D G G D V G Q I F s C # L Q D T N N F F G A g Q N K % O P K L G Q I G % S K % u u i e d d R i d D V L k n ( T D K @ p p ^ v ]' # CCDS207.1|Hs110|chr1 |
|
|
|
# Create a fill-mask prediction pipeline |
|
unmasker = pipeline('fill-mask', model=model, tokenizer=tokenizer) |
|
# Predict the masked token |
|
prediction = unmasker(sequence) |
|
print(prediction) |
|
``` |
|
|
|
## Intended use and limitations |
|
cdsBERT serves as a general-purpose protein language model with a codon vocabulary. Fine-tuning with Huggingface transformers models like BertForSequenceClassification enables downstream classification and regression tasks. Currently, the base capability enables feature extraction. This checkpoint after MLM can conduct mask-filling, while the cdsBERT+ checkpoint has a more biochemically relevant latent space. |
|
|
|
## Our lab |
|
The [Gleghorn lab](https://www.gleghornlab.com/) is an interdisciplinary research group at the University of Delaware that focuses on solving translational problems with our expertise in engineering, biology, and chemistry. We develop inexpensive and reliable tools to study organ development, maternal-fetal health, and drug delivery. Recently we have begun exploration into protein language models and strive to make protein design and annotation accessible. |
|
|
|
## Please cite |
|
@article {Hallee_cds_2023, |
|
author = {Logan Hallee, Nikolaos Rafailidis, and Jason P. Gleghorn}, |
|
title = {cdsBERT - Extending Protein Language Models with Codon Awareness}, |
|
year = {2023}, |
|
doi = {10.1101/2023.09.15.558027}, |
|
publisher = {Cold Spring Harbor Laboratory}, |
|
journal = {bioRxiv} |
|
} |
