---
license: mit
language:
- en
library_name: peft
tags:
- ESM-2
- QLoRA
- Binding Sites
- biology
---

# ESM-2 QLoRA

These are the checkpoints for the first ever QLoRA for ESM-2! They haven't been checked for overfitting yet, so use with caution!
You can load and use them similarly to the LoRA models. This is the smallest `esm2_t6_8M_UR50D` model, so the metrics aren't great. 
Scaling to larger models for better metrics is in progress. These checkpoints were trained using [the 600K dataset](https://huggingface.co/datasets/AmelieSchreiber/600K_data). 

## QLoRA Info

Note, we are only training 0.58% of the parameters, using only the query, key, and value weight matrices. 

```
trainable params: 23682 || all params: 4075265 || trainable%: 0.5811155838945443
```

## Testing for Overfitting

### Checkpoint 1

### Checkpoint 2

### Checkpoint 3

### Checkpoint 4

```python
Train metrics:
{'eval_loss': 0.24070295691490173,
'eval_accuracy': 0.9018779246397052,
'eval_precision': 0.16624103834249204,
'eval_recall': 0.8651772818812425,
'eval_f1': 0.27889357183237473,
'eval_auc': 0.8839390799308487,
'eval_mcc': 0.3536803490333407}

Test metrics:
{'eval_loss': 0.26776671409606934,
'eval_accuracy': 0.8902711124906878,
'eval_precision': 0.13008662855482372,
'eval_recall': 0.7084623832213568,
'eval_f1': 0.219811797752809,
'eval_auc': 0.8013943890942485,
'eval_mcc': 0.2721459410994918}
```