Hugging Face's logo

cloudyu
/
Mixtral_34Bx2_MoE_60B

Text Generation
Transformers
Safetensors
mixtral
yi
Mixture of Experts
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
16

How does the MoE work?

#5
by PacmanIncarnate - opened
Discussion
PacmanIncarnate

Can you explain how the gate picks between the two included models? Was there any additional merge system involved as well, or simple MoE?

SamuelAzran

Also, did you fine-tuned after the merge?

dillfrescott

Most likely mergekit was used https://github.com/cg123/mergekit/blob/mixtral/moe.md

cloudyu
 Owner

https://arxiv.org/abs/2401.04088

Sign up or log in to comment