Other quant types.
Hello. I just wan't to make sure I understood everything correctly when reading the GitHub PR comments. The Q5_K_M is not implemented yet, right? If so that's unfortunate, because I have all my GGUF models in Q5_K_M, but with how efficient Mixtral is supposed to be I'll manage with Q6_K.
On a sidenote. Thank you very much for your work! I love playing with new technologies and with my hardware limitations it wouldn't be possible without your quick and consistent work. I wish you all the best!
Now that I've re-uploaded the quants, I've used the correct names rather than trying to re-name them.
GGeorganov said earlier that for Mixtral the _S and _L sizes are currently identical to the _M size. And the Q4_K_M is the same size (but not the same content) as Q4_0 and likewise Q5_K_M is same size (but different content) to Q5_0.
So for now, my files for Mixtral will be:
- Q2_K
- Q3_K_M
- Q4_K_M
- Q4_0
- Q5_K_M
- Q5_0
- Q6_K
- Q8_0