---
license: mit
library_name: transformers
tags:
- mergekit
- moe
- mixture of experts
---

# Custom MOE with Mergekit

Base model: `mistralai/Mistral-7B-Instruct-v0.2`. Fused with experts: 

- HuggingFaceH4/zephyr-7b-beta
- mistralai/Mistral-7B-Instruct-v0.2
- teknium/OpenHermes-2.5-Mistral-7B
- meta-math/MetaMath-Mistral-7B

This can be downloaded to improve the base mistrals model to reason across mathematical and other objectives.