ISTA-DASLab/Mistral-Nemo-Instruct-2407-AQLM-PV-2Bit-1x16-hf

Official AQLM quantization of mistralai/Mistral-Nemo-Instruct-2407 finetuned with PV-Tuning.

For this quantization, we used 1 codebook of 16 bits and groupsize of 8.

Results:

Model	Quantization	MMLU (5-shot)	ArcC	ArcE	Hellaswag	PiQA	Winogrande	Model size, Gb
mistralai/Mistral-Nemo-Instruct-2407	None	0.6819	0.5606	0.8241	0.6332	0.8090	0.7498	24.5
	1x16g8	0.6071	0.5017	0.7942	0.5930	0.7987	0.7356	5.8

Note

We used lm-eval=0.4.0 for evaluation.