metadata
license: mit
library_name: transformers
tags:
- mergekit
- moe
- mixture of experts
Custom MOE with Mergekit
Base model: mistralai/Mistral-7B-Instruct-v0.2. Fused with experts:
- HuggingFaceH4/zephyr-7b-beta
- mistralai/Mistral-7B-Instruct-v0.2
- teknium/OpenHermes-2.5-Mistral-7B
- meta-math/MetaMath-Mistral-7B
This can be downloaded to improve the base mistrals model to reason across mathematical and other objectives.