I cut my TinyLlama 1.1B cinder v 2 down from 22 layers to 14. At 14 there was no coherent text but there were emerging ideas of a response. 1000 steps on step-by-step dataset. 6000 on Reason-with-cinder. The loss was still over 1 and the learning rate was still over 4. This model needs significat training. I am putting it up as a base model that needs work. If you continue training please let me know on the tinyllama discord, I have some interesting plans for this model.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 29.55 |
| AI2 Reasoning Challenge (25-Shot) | 25.09 |
| HellaSwag (10-Shot) | 33.82 |
| MMLU (5-Shot) | 24.43 |
| TruthfulQA (0-shot) | 42.90 |
| Winogrande (5-shot) | 51.07 |
| GSM8k (5-shot) | 0.00 |
- Downloads last month
- 2,464
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard25.090
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard33.820
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard24.430
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard42.900
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard51.070
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard0.000