Qwen2.5-7B-Instruct-nf4

A 4-bit NF4 quantization of Qwen/Qwen2.5-7B-Instruct, produced by quantize_nf4.py for MS628 Assignment 2.1.

Quantization settings

Setting	Value
Method	bitsandbytes, load_in_4bit=True
Quant type	nf4
Compute dtype	bfloat16
Double quantization	True

Size

Measurement	Value
Source weights on disk	14.19 GB
Saved 4-bit folder	5.18 GB
Reduction	63.5%
Reported GPU footprint	5.07 GB

How it was produced

python quantize_nf4.py --model Qwen/Qwen2.5-7B-Instruct --outdir Qwen2.5-7B-Instruct-nf4

Produced on Windows 10, PyTorch 2.11.0+cu128, bitsandbytes 0.50.2, GPU NVIDIA GeForce RTX 5090 Laptop GPU.

Loading it again

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen2.5-7B-Instruct-nf4", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-7B-Instruct-nf4")

The 4-bit weights are already baked into the saved files, so no quantization config is needed on the way back in. A CUDA GPU and bitsandbytes are still required to run it.