license: mit
ESM-2 QLoRA for Predicting Binding Sites
This model is the ESM-2 model esm2_t12_35M_UR50D finetuned with QLoRA on
this dataset of 2.6M protein sequences with binding and active
site annotations from UniProt. The model and dataset size were scaled in a one-to-one way (following the Chinchilla paper) up from the smaller
QLoRA adaptations of the esm2_t6_8M_UR50D models which were trained on 600K proteins. Since this model is 4.375 times larger, a dataset
approximately 4.375 times larger is needed if Chinchilla scaling laws hold for QLoRA finetuning of protein language models. Determining if
such scaling laws also hold is part of this project, so checking for improvements in performance metrics over a period of 3 epochs, as well
as checking for signs of overfitting for each epoch are underway.
QLoRA Info
trainable params: 71046 || all params: 17246053 || trainable%: 0.41195512967517844