metadata
library_name: transformers
tags: []
Mula 4x160m
Step | Evaluation Loss | Perplexity | Total Energy Consumption |
---|---|---|---|
50000 | 3.03 | 20.73 | 0.041 |
100000 | 2.84 | 17.14 | 0.041 |
150000 | 2.73 | 15.35 | 0.041 |
200000 | 2.64 | 14.05 | 0.041 |
250000 | 2.56 | 12.95 | 0.041 |
300000 | 2.49 | 12.14 | 0.041 |
350000 | 2.46 | 11.75 | 0.041 |
400000 | 2.46 | 11.72 | 0.040 |
ARC | HellaSwag | MMLU | TruthfulQA | Average | |
---|---|---|---|---|---|
TTL-460m | 29.40 | 33.00 | 28.55 | 41.10 | 33.01 |
TTL-160m | 26.15 | 29.29 | 28.11 | 41.12 | 31.16 |
Mula-4x160m | 27.09 | 31.41 | 28.15 | 39.81 | 31.61 |
ASSIN2 RTE | ASSIN2 STS | BLUEX | ENEM | FAQUAD NLI | HateBR | OAB Exams | Average | |
---|---|---|---|---|---|---|---|---|
TTL-460m | 53.93 | 12.66 | 22.81 | 19.87 | 49.01 | 33.59 | 27.06 | 31.27 |
TTL-160m | 53.36 | 2.58 | 21.84 | 18.75 | 43.97 | 36.88 | 22.60 | 28.56 |
Mula-4x160m | 33.55 | 8.88 | 20.58 | 20.08 | 43.97 | 33.65 | 22.92 | 26.23 |