AmelieSchreiber
commited on
Commit
•
32ec1db
1
Parent(s):
27d5c3c
Update README.md
Browse files
README.md
CHANGED
@@ -5,9 +5,10 @@ license: mit
|
|
5 |
# ESM-2 QLoRA for Binding Site Prediction
|
6 |
|
7 |
In this model, we wanted to see how the performance metrics were effected by adapting additional weight matrices with QLoRA. This was
|
8 |
-
shown to be the most important hyperparameter for improvement in performance metrics by far
|
|
|
9 |
simply using the query, key, and value weight matrix adapters to using adapters for all possible weight matrices. The comparison for the
|
10 |
-
first epoch can be seen below. Note the minor performance improvements for the model using every possible weight matrix
|
11 |
|
12 |
### This model
|
13 |
|
|
|
5 |
# ESM-2 QLoRA for Binding Site Prediction
|
6 |
|
7 |
In this model, we wanted to see how the performance metrics were effected by adapting additional weight matrices with QLoRA. This was
|
8 |
+
shown to be the most important hyperparameter for improvement in performance metrics by far, whereas hyperparameters such as rank and scaling
|
9 |
+
factor were shown to be negligible in importance, with lower rank being just as good as higher rank. So, we decided to test the difference between
|
10 |
simply using the query, key, and value weight matrix adapters to using adapters for all possible weight matrices. The comparison for the
|
11 |
+
first epoch can be seen below. Note the minor performance improvements for the model using every possible weight matrix.
|
12 |
|
13 |
### This model
|
14 |
|