README.md

starcoderbase-3b-GPTQ

Quantized starcoderbase-3b model to GPTQ format (4-bit precision) using Auto-GPTQ.

Quantization script

Benchmark

Benchmarking script

Baseline starcoderbase-3b model (non-quantized)

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
codexglue_code2text	N/A	none	None	smoothed_bleu_4	1.3519	±	0.3067
- code2text_go	1	none	None	smoothed_bleu_4	1.5781	±	0.3734
- code2text_java	1	none	None	smoothed_bleu_4	1.2778	±	0.1991
- code2text_javascript	1	none	None	smoothed_bleu_4	1.1443	±	0.1181
- code2text_php	1	none	None	smoothed_bleu_4	0.5171	±	0.5171
- code2text_python	1	none	None	smoothed_bleu_4	2.8338	±	1.5323
- code2text_ruby	3	none	None	smoothed_bleu_4	0.7601	±	0.7601

Groups	Version	Filter	n-shot	Metric	Value		Stderr
codexglue_code2text	N/A	none	None	smoothed_bleu_4	1.3519	±	0.3067

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
bigbench_code_line_description_generate_until	1	none	None	exact_match	0	±	0

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
bigbench_code_line_description_multiple_choice	0	none	None	acc	0.25	±	0.0564

Quantized starcoderbase-3b model to GPTQ format

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
codexglue_code2text	N/A	none	None	smoothed_bleu_4	0.9254	±	0.2109
- code2text_go	1	none	None	smoothed_bleu_4	1.4702	±	0.4813
- code2text_java	1	none	None	smoothed_bleu_4	0.6907	±	0.6907
- code2text_javascript	1	none	None	smoothed_bleu_4	0.9469	±	0.0339
- code2text_php	1	none	None	smoothed_bleu_4	0.5171	±	0.5171
- code2text_python	1	none	None	smoothed_bleu_4	1.1676	±	0.2156
- code2text_ruby	3	none	None	smoothed_bleu_4	0.7601	±	0.7601

Groups	Version	Filter	n-shot	Metric	Value		Stderr
codexglue_code2text	N/A	none	None	smoothed_bleu_4	0.9254	±	0.2109

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
bigbench_code_line_description_generate_until	1	none	None	exact_match	0	±	0

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
bigbench_code_line_description_multiple_choice	0	none	None	acc	0.1	±	0.1