Quants for the following model : https://huggingface.co/cloudyu/Mixtral_34Bx2_MoE_60B
I'm not satisfied with them, though. Their size is weird.
For now, prefer the quants of The Bloke : https://huggingface.co/TheBloke/Mixtral_34Bx2_MoE_60B-GGUF
Bench of a Q3_K_M from TheBloke :
- mixtral_34bx2_moe_60b.Q3_K_M.gguf,-,Hellaswag,84.5,84.25,400,2024-01-27 00:00:00,,70b,Llama_2,4096,,,GGUF,,,
- mixtral_34bx2_moe_60b.Q3_K_M.gguf,-,Hellaswag,,,1000,2024-01-27 00:00:00,,70b,Llama_2,4096,,,GGUF,,,
- mixtral_34bx2_moe_60b.Q3_K_M.gguf,-,Arc-Challenge,61.53846154,,299,2024-01-27 05:40:00,,01.3b,Llama_2,4096,,,GGUF,,,
- mixtral_34bx2_moe_60b.Q3_K_M.gguf,-,Arc-Easy,82.28070175,,570,2024-01-27 05:40:00,,01.3b,Llama_2,4096,,,GGUF,,,
- mixtral_34bx2_moe_60b.Q3_K_M.gguf,-,MMLU,40.89456869,,313,2024-01-27 05:40:00,,01.3b,Llama_2,4096,,,GGUF,,,
- mixtral_34bx2_moe_60b.Q3_K_M.gguf,-,Thruthful-QA,42.35006120,,817,2024-01-27 05:40:00,,01.3b,Llama_2,4096,,,GGUF,,,
- mixtral_34bx2_moe_60b.Q3_K_M.gguf,-,Winogrande,79.0845,,1267,2024-01-27 05:40:00,,01.3b,Llama_2,4096,,,GGUF,,,
- mixtral_34bx2_moe_60b.Q3_K_M.gguf,-,wikitext,5.3715,512,512,2024-01-27 00:00:00,,70b,Llama_2,4096,,,GGUF,,,
- mixtral_34bx2_moe_60b.Q3_K_M.gguf,-,wikitext,5.1792,4096,4096,2024-01-27 00:00:00,,70b,Llama_2,4096,,,GGUF,,,24