Hugging Face's logo

TheBloke
/
Llama-2-13B-chat-GPTQ

Text Generation
Transformers
Safetensors
PyTorch
English
llama
facebook
meta
llama-2
text-generation-inference
4-bit precision
gptq
Model card Files Community
54

Dataset used for quantisation

#31
by CarlosAndrea - opened
Discussion
CarlosAndrea

Hello,
what dataset did you use for quantisation and with how many samples?
In my experience, the number of samples is the key to achieve good inference quality

CarlosAndrea changed discussion title from Dataset used for quantization to Dataset used for quantisation
TheBloke
 Owner

wikitext, 128 samples of 4096 tokens each.

In my more recent models I am listing this info in the README but I hadn't done that at the time I made Llama 2.

I will soon be updating all my GPTQ READMEs to the later format so it will be recorded here soon.

RonanMcGovern

I notice that the paper uses the c4 dataset (which is somewhat broken on HuggingFace). May be a bit more diverse than wikitext? I'm not sure.

I'm trying to understand as well to what extent the choice of quantization dataset affects perplexity outside of that dataset. I've reached out to the paper's authors.

Sign up or log in to comment