Quantization made by Richard Erkhov.

Frankenstein-MoE-en-10.7Bx6 - GGUF

Model creator: https://huggingface.co/MoEMoEKKung/
Original model: https://huggingface.co/MoEMoEKKung/Frankenstein-MoE-en-10.7Bx6/

Name	Quant method	Size
Frankenstein-MoE-en-10.7Bx6.Q2_K.gguf	Q2_K	17.99GB
Frankenstein-MoE-en-10.7Bx6.IQ3_XS.gguf	IQ3_XS	20.14GB
Frankenstein-MoE-en-10.7Bx6.IQ3_S.gguf	IQ3_S	21.29GB
Frankenstein-MoE-en-10.7Bx6.Q3_K_S.gguf	Q3_K_S	21.27GB
Frankenstein-MoE-en-10.7Bx6.IQ3_M.gguf	IQ3_M	21.65GB
Frankenstein-MoE-en-10.7Bx6.Q3_K.gguf	Q3_K	23.61GB
Frankenstein-MoE-en-10.7Bx6.Q3_K_M.gguf	Q3_K_M	23.61GB
Frankenstein-MoE-en-10.7Bx6.Q3_K_L.gguf	Q3_K_L	25.57GB
Frankenstein-MoE-en-10.7Bx6.IQ4_XS.gguf	IQ4_XS	26.61GB
Frankenstein-MoE-en-10.7Bx6.Q4_0.gguf	Q4_0	27.81GB
Frankenstein-MoE-en-10.7Bx6.IQ4_NL.gguf	IQ4_NL	28.08GB
Frankenstein-MoE-en-10.7Bx6.Q4_K_S.gguf	Q4_K_S	28.06GB
Frankenstein-MoE-en-10.7Bx6.Q4_K.gguf	Q4_K	29.86GB
Frankenstein-MoE-en-10.7Bx6.Q4_K_M.gguf	Q4_K_M	29.86GB
Frankenstein-MoE-en-10.7Bx6.Q4_1.gguf	Q4_1	30.89GB
Frankenstein-MoE-en-10.7Bx6.Q5_0.gguf	Q5_0	33.96GB
Frankenstein-MoE-en-10.7Bx6.Q5_K_S.gguf	Q5_K_S	33.96GB
Frankenstein-MoE-en-10.7Bx6.Q5_K.gguf	Q5_K	35.02GB
Frankenstein-MoE-en-10.7Bx6.Q5_K_M.gguf	Q5_K_M	35.02GB
Frankenstein-MoE-en-10.7Bx6.Q5_1.gguf	Q5_1	37.04GB
Frankenstein-MoE-en-10.7Bx6.Q6_K.gguf	Q6_K	40.5GB
Frankenstein-MoE-en-10.7Bx6.Q8_0.gguf	Q8_0	52.46GB

Original model description:

language: - en license: cc-by-nc-sa-4.0

Frankenstein-MoE

Method

To initialize the gate projection weight of the MoE layer, the H6 trainset was sampled and used. We sampled 400 and selected the final 30 with low PPL.

trufulqa used gpt4 to generate data.

Evals

in progress