Quantization made by Richard Erkhov.

Frankenstein-MoE-en-10.7Bx4 - GGUF

Model creator: https://huggingface.co/MoEMoEKKung/
Original model: https://huggingface.co/MoEMoEKKung/Frankenstein-MoE-en-10.7Bx4/

Name	Quant method	Size
Frankenstein-MoE-en-10.7Bx4.Q2_K.gguf	Q2_K	12.28GB
Frankenstein-MoE-en-10.7Bx4.IQ3_XS.gguf	IQ3_XS	13.74GB
Frankenstein-MoE-en-10.7Bx4.IQ3_S.gguf	IQ3_S	14.52GB
Frankenstein-MoE-en-10.7Bx4.Q3_K_S.gguf	Q3_K_S	14.5GB
Frankenstein-MoE-en-10.7Bx4.IQ3_M.gguf	IQ3_M	14.8GB
Frankenstein-MoE-en-10.7Bx4.Q3_K.gguf	Q3_K	16.1GB
Frankenstein-MoE-en-10.7Bx4.Q3_K_M.gguf	Q3_K_M	16.1GB
Frankenstein-MoE-en-10.7Bx4.Q3_K_L.gguf	Q3_K_L	17.45GB
Frankenstein-MoE-en-10.7Bx4.IQ4_XS.gguf	IQ4_XS	18.13GB
Frankenstein-MoE-en-10.7Bx4.Q4_0.gguf	Q4_0	18.95GB
Frankenstein-MoE-en-10.7Bx4.IQ4_NL.gguf	IQ4_NL	19.13GB
Frankenstein-MoE-en-10.7Bx4.Q4_K_S.gguf	Q4_K_S	19.11GB
Frankenstein-MoE-en-10.7Bx4.Q4_K.gguf	Q4_K	20.33GB
Frankenstein-MoE-en-10.7Bx4.Q4_K_M.gguf	Q4_K_M	20.33GB
Frankenstein-MoE-en-10.7Bx4.Q4_1.gguf	Q4_1	21.04GB
Frankenstein-MoE-en-10.7Bx4.Q5_0.gguf	Q5_0	23.13GB
Frankenstein-MoE-en-10.7Bx4.Q5_K_S.gguf	Q5_K_S	23.13GB
Frankenstein-MoE-en-10.7Bx4.Q5_K.gguf	Q5_K	23.84GB
Frankenstein-MoE-en-10.7Bx4.Q5_K_M.gguf	Q5_K_M	23.84GB
Frankenstein-MoE-en-10.7Bx4.Q5_1.gguf	Q5_1	25.23GB
Frankenstein-MoE-en-10.7Bx4.Q6_K.gguf	Q6_K	27.58GB
Frankenstein-MoE-en-10.7Bx4.Q8_0.gguf	Q8_0	35.73GB

Original model description:

language: - en license: cc-by-nc-sa-4.0

Frankenstein-MoE

Method

To initialize the gate projection weight of the MoE layer, the H6 trainset was sampled and used. We sampled 400 and selected the final 30 with low PPL.

trufulqa used gpt4 to generate data.

Evals

in progress