Quants
Thanks @TheBloke for doing his thing :)
I'll keep this list updated if GGUFs come along (See https://huggingface.co/augmxnt/shisa-7b-v1/discussions/1 to follow along on that story, basically llama.cpp bugged atm for most BPE tokenizers so no point in quantizing).
- AWQ: https://huggingface.co/TheBloke/shisa-7B-v1-AWQ
- GPTQ: https://huggingface.co/TheBloke/shisa-7B-v1-GPTQ
It looks like mmnga was able to get GGUF conversion working with a custom_shisa.py conversion script that combines the extra BPE characters into the spm tokenizer. Seems to run great, thanks!
If anyone does their own (EXLs, etc) feel free to post it in here.
I noted while clicking around that @LoneStriker made some EXL2 quants (thanks!):
- https://huggingface.co/LoneStriker/shisa-7b-v1-8.0bpw-h8-exl2
- https://huggingface.co/LoneStriker/shisa-7b-v1-6.0bpw-h6-exl2
- https://huggingface.co/LoneStriker/shisa-7b-v1-5.0bpw-h6-exl2
- https://huggingface.co/LoneStriker/shisa-7b-v1-4.0bpw-h6-exl2
- https://huggingface.co/LoneStriker/shisa-7b-v1-3.0bpw-h6-exl2
Note, also of interest, while I was doing inference benchmarking, I also created an H6 4.63BPW quant to match the BPW of @mmnga 's q4_K_M GGUF: https://huggingface.co/augmxnt/shisa-7b-v1-exl2-h6-4.63bpw