Update README.md
Browse files
README.md
CHANGED
@@ -29,3 +29,22 @@ Other training configuration:
|
|
29 |
| ADAM_BETA2 | 0.98 |
|
30 |
| ADAM_WEIGHT_DECAY | 0.01 |
|
31 |
| PEAK_LR | 1e-3 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
29 |
| ADAM_BETA2 | 0.98 |
|
30 |
| ADAM_WEIGHT_DECAY | 0.01 |
|
31 |
| PEAK_LR | 1e-3 |
|
32 |
+
|
33 |
+
Evaluation Results on GLUE after pretraining (30k steps):
|
34 |
+
|
35 |
+
| Task | Score |
|
36 |
+
|----------------------|-----------|
|
37 |
+
| CoLA-MattCorr | 20.0 |
|
38 |
+
| MRPC-F1 | 83.6 |
|
39 |
+
| STS-B-Pearson | 82.7 |
|
40 |
+
| SST-2-Acc | 89.7 |
|
41 |
+
| QQP-F1 | 89.0 |
|
42 |
+
| MNLI-m | 79.7 |
|
43 |
+
| MNLI-mm | 80.1 |
|
44 |
+
| QNLI-Acc | 89.3 |
|
45 |
+
| WNLI-Acc | 31.0 |
|
46 |
+
| RTE-Acc | 57.4 |
|
47 |
+
|
48 |
+
Evaluation Scores Curve (AVG of scores) during pretraining:
|
49 |
+
|
50 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/62927c2e56fedc76e396b3ca/4mVztXjyyTAXd-AdBuCnt.png)
|