File size: 1,068 Bytes
fcf5eb5 cd5e192 fcd05f8 86ae9fd 28bad80 86ae9fd |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
---
license: mit
---
# ESM-2 QLoRA for Predicting Binding Sites
This model is the ESM-2 model [esm2_t12_35M_UR50D](https://huggingface.co/facebook/esm2_t12_35M_UR50D) finetuned with QLoRA on
[this dataset](https://huggingface.co/datasets/AmelieSchreiber/2600K_binding_sites) of 2.6M protein sequences with binding and active
site annotations from UniProt. The model and dataset size were scaled in a one-to-one way (following the Chinchilla paper) up from the smaller
QLoRA adaptations of the `esm2_t6_8M_UR50D` models which were trained on 600K proteins. Since this model is 4.375 times larger, a dataset
approximately 4.375 times larger is needed if Chinchilla scaling laws hold for QLoRA finetuning of protein language models. Determining if
such scaling laws also hold is part of this project, so checking for improvements in performance metrics over a period of 3 epochs, as well
as checking for signs of overfitting for each epoch are underway.
## QLoRA Info
```
trainable params: 71046 || all params: 17246053 || trainable%: 0.41195512967517844
```
|