SteelStorage

/

Umbra-v3-MoE-4x11b

like 4

Setting experts to four just means you now have four "eyes" on every token, in-fact it has shown to help improve quality of the output on Mixtral 8x7, altho there is a drop off after so many extra experts as it does increase processing time slightly.

also it is still very different to a regular merge, a single model is just a jack of all trades and master of none it just merges all knowledge, while MoEs or multiple experts all have there own specialties or domains they are experts in, hence "Mixture if experts".

here is an experiment performed by a member of the local llama subreddit:

https://www.reddit.com/r/LocalLLaMA/comments/18la6ao/comment/kdx4ada/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Even if it's not as good, could you still make an alt version with 2 experts per token? For me, this version with 4 is just too slow :\

Sure! I'll see if I can get one posed soon

here is the two expert version!

Steelskull/Umbra-v3-MoE-4x11b-2ex

I would be interested in seeing beckmarks between the two versions, just to verify objectively how much impact the extra experts make.

i actually though that as well it has been submitted to the leaderboard for a bench