|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
--- |
|
|
|
# Model Card for SuperSparse-Mixtral |
|
The SuperSparse-Mixtral Large Language Model (LLM) is an sparsified version of the Mixtral. |
|
|
|
<img src="takeaway.png" alt="avatar" width="300" height="200"/> |
|
|
|
The average performance is evaluated using benchmarks from the OpenLLM Leaderboard. |
|
|
|
## Inference |
|
|
|
Our code for accelerating SuperSparse-Mixtral is currently being refined. Stay tuned! Now you can run this model like dense model. |
|
|
|
## Chat-Template |
|
|
|
During sparsification, we also utilize some SFT datasets. |
|
We take ChatML as our chat template: |
|
``` |
|
<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n |
|
``` |
|
|
|
## Allow Finetuning |
|
|
|
As we merged the predictors for FFN neurons in models, you can finetune SuperSparse-Mixtral with any framework and algorithm. |
|
|
|
## License |
|
|
|
The model is licensed under Apache-2.0, while model weights are fully open for academic research and also allow **free** commercial usage. |
