| license: mit | |
| library_name: transformers | |
| tags: | |
| - mergekit | |
| - moe | |
| - mixture of experts | |
| # Custom MOE with Mergekit | |
| Base model: `mistralai/Mistral-7B-Instruct-v0.2`. Fused with experts: | |
| - HuggingFaceH4/zephyr-7b-beta | |
| - mistralai/Mistral-7B-Instruct-v0.2 | |
| - teknium/OpenHermes-2.5-Mistral-7B | |
| - meta-math/MetaMath-Mistral-7B | |
| This can be downloaded to improve the base mistrals model to reason across mathematical and other objectives. |