---
library_name: transformers
tags: []
---

# Mula 4x160m

| Step   | Evaluation Loss | Perplexity |
|--------|-----------------|------------|
| 50000  | 3.03            | 20.73      |
| 100000 | 2.84            | 17.14      |
| 150000 | 2.73            | 15.35      |
| 200000 | 2.64            | 14.05      |
| 250000 | 2.56            | 12.95      | 
| 300000 | 2.49            | 12.14      |
| 350000 | 2.46            | 11.75      | 
| 400000 | 2.46            | 11.72      |

|                  | **ARC**   | **HellaSwag** | **MMLU**  | **TruthfulQA** | **Average** |
|------------------|-----------|---------------|-----------|----------------|-------------|
| **TTL-460m**     | 29.40     | 33.00         | 28.55     | 41.10          | 33.01       |
| **TTL-160m**     | 26.15     | 29.29         | 28.11     | 41.12          | 31.16       |
| **Mula-4x160m**  | 27.09     | 31.41         | 28.15     | 39.81          | 31.61       |


|                   | **ASSIN2 RTE** | **ASSIN2 STS** | **BLUEX** | **ENEM** | **FAQUAD NLI** | **HateBR** | **OAB Exams** | **Average** |
|-------------------|----------------|----------------|-----------|----------|----------------|------------|---------------|-------------|
| **TTL-460m**      | 53.93          | 12.66          | 22.81     | 19.87    | 49.01          | 33.59      | 27.06         | 31.27       |
| **TTL-160m**      | 53.36          | 2.58           | 21.84     | 18.75    | 43.97          | 36.88      | 22.60         | 28.56       |
| **Mula-4x160m**   | 33.55          | 8.88           | 20.58     | 20.08    | 43.97          | 33.65      | 22.92         | 26.23       |