File size: 1,393 Bytes
6024e15 206899b 3909b06 706a33e 206899b 5b05bb0 d7c6bda 5b05bb0 206899b 3909b06 206899b e917fa2 206899b 3909b06 206899b 60561a7 afc1c63 60561a7 206899b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
---
license: apache-2.0
language:
- en
---
# Model Card for TurboSparse-Mixtral
The [TurboSparse-Mixtral](https://arxiv.org/abs/2406.05955) Large Language Model (LLM) is an sparsified version of the Mixtral.
<img src="takeaway.png" alt="avatar" width="300" height="200"/>
The average performance is evaluated using benchmarks from the OpenLLM Leaderboard.
## Inference
Our code for accelerating TurboSparse-Mixtral is currently being refined. Stay tuned! Now you can run this model like dense model.
## Chat-Template
During sparsification, we also utilize some SFT datasets.
We take ChatML as our chat template:
```
<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n
```
## Allow Finetuning
As we merged the predictors for FFN neurons in models, you can finetune TurboSparse-Mixtral with any framework and algorithm.
## Limitations
* TurboSparse, having just undergone training with 150B tokens, may still exhibit performance gaps in certain tasks.
* The TurboSparse model has only been trained on English-language datasets, hence its capabilities in other languages are still lacking.
* The model may produce unexpected outputs due to its small size, limited training tokens and probabilistic generation paradigm.
## License
The model is licensed under Apache-2.0, while model weights are fully open for academic research and also allow **free** commercial usage. |