File size: 1,820 Bytes
5af5d78 f754474 5af5d78 ae96f79 5af5d78 d9acebe 5af5d78 d9acebe da911da ae96f79 d9acebe ae96f79 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
---
license: llama3
datasets:
- JeanKaddour/minipile
language:
- en
tags:
- axolotl
- mergekit
- llama
---
> 🚨 THIS IS A BASE MODEL 🚨
>
> This model is pruned from the base Llama 3 70B, which has no instruction tuning and randomly initialized special tokens.
>
> Using this with the Llama 3 instruction format is injecting random noise into latent space and will give you deranged results. (It's pretty funny actually.)
> Treat this as the untrained foundation model this is and use appropriate prompts.
Meta's Llama 3 70B pruned to 42B parameters using the methodology described in [The Unreasonable Ineffectiveness of the Deeper Layers](https://arxiv.org/abs/2403.17887). Post-pruning trained using QLoRA for ~100M tokens from [JeanKaddour/minipile](https://huggingface.co/datasets/JeanKaddour/minipile).
Layers to prune selected using [PruneMe](https://github.com/arcee-ai/PruneMe).
Still evaluating, don't get too excited! Might be incredibly dumb. Check out these numbers though:
| Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu |N/A |none | 0|acc |0.7669|± |0.0034|
| - humanities |N/A |none | 5|acc |0.7296|± |0.0062|
| - other |N/A |none | 5|acc |0.8101|± |0.0067|
| - social_sciences|N/A |none | 5|acc |0.8668|± |0.0060|
| - stem |N/A |none | 5|acc |0.6825|± |0.0079|
|winogrande| 1|none | 5|acc |0.8027|± |0.0112|
|hellaswag| 1|none | 10|acc_norm|0.8025|± |0.0040|
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl) |