General discussion / GGUF-Imatrix quants.

#1
by Lewdiculous - opened

I should be uploading my set of GGUF-IQ-Imatrix quants of TheSpice-7b-v0.1.1 until early morning at:
https://huggingface.co/Lewdiculous/TheSpice-7b-v0.1.1-GGUF-IQ-Imatrix

The default list was requested in #16, if anyone also feels like any quant outside of these should be added, feel free to chime in.

    quantization_options = [
        "Q4_K_M", "Q4_K_S", "IQ4_XS", "Q5_K_M", "Q5_K_S",
        "Q6_K", "Q8_0", "IQ3_M", "IQ3_S", "IQ3_XXS"
    ]

Keep up the good work, @cgato !

I will say that I, just as a user personally, would much prefer if you released model weights in .safetensors instead.

It is a potential security risk as pickle files - and the same applies to derived binaries - can allow for remote code execution and it's just something we can avoid, .safetensors are already best practice in that sense.

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment