Which calibration set is chose?
#4
by
frankxyy
- opened
I used to quantize qwen-14b using awq, but the accuracy on real dataset is not as good as that of qwen-14b-int4 with gptq.
The calibration dataset is linked in the README. I made this one with wikitext, but in recent times I use VMware Open Instruct, which is a higher quality dataset.
And if there will be a gguf version ? https://huggingface.co/TheBloke/Qwen-7B-Chat-GPTQ/discussions/1
@TheBloke
Hi, how many samples are used here for calibration?
https://github.com/ggerganov/llama.cpp/pull/4593#awq-py/README.md
Looking forward for llama.cpp version of AWQ