Thanks. iQ1_S Actually Works.
#28
by
deleted
- opened
Since your Mixtral-8x22B-v0.1.IQ1_S version just barely fit in my 32 GB of RAM I decided to give it a shot simply for the sake of curiosity and expecting nonsensical outputs, but it was surprisingly coherent and correctly answered some simple questions. How a 1-bit quantization functions at all is beyond me.
Thank you for confirming my own experience with IQ-1 models! This new way of quantizing by using imatix.data really improves the quality of the quantized models!
PS: this is a fine-tuned model by HuggingFace based on this mode, you might want to check the IQ-1 models I just pushed there as well. That should be much easier to instruct: https://huggingface.co/MaziyarPanahi/zephyr-orpo-141b-A35b-v0.1-GGUF