🏗️ Building on HF

nikki PRO

b4ph

AI & ML interests

Composer, Audio Director, Musician Mathematician working in Digital Forensics You Know What You Have To Do.

Recent Activity

repliedto their post about 9 hours ago

For those who wanna run Qwen3-4B on CPU alone, I tested the whole ladder: Same machine for everything: CPU only, 16 threads, no GPU offload. Perplexity is WikiText-2 raw, which llama.cpp also uses in CI. ```text Quant Size PPL vs F16 tg t/s F16 8.05 GB 13.4304 — 2.52 Q8_0 4.28 GB 13.4409 +0.01 5.04 Q6_K 3.31 GB 13.4404 +0.01 5.93 Q5_K_M 2.89 GB 13.5362 +0.11 5.64 Q4_K_M 2.50 GB 13.7304 +0.30 6.50 Q4_K_M + imatrix 2.50 GB 13.6760 +0.25 6.41 Q3_K_M 2.08 GB 15.6641 +2.23 6.56 Q3_K_M + imatrix 2.08 GB 14.8237 +1.39 5.89 ``` The winner is Q4_K_M - it gets the model down from 8.05 GB to 2.5 GB in size/load, and perplexity only moves by 0.30. Q3 is the dropoff point where inmatrix makes a difference. Inmatrix drops Q3 from 15.66 to 14.82 PPL; Q4 it barely changed anything. My now informed recommendation is: Q4 if you have the memory, imatrix Q3 if you don’t. I uploaded all eight weights, the harness, and the full results here: huggingface.co/b4ph/qwen3-4b-lowram-bench I also made a tiny picker because apparently I needed to turn this into a whole project: huggingface.co/spaces/b4ph/qwen3-4b-quant-picker

updated a model about 9 hours ago

b4ph/qwen3-4b-lowram-bench

liked a dataset 1 day ago

wikimedia/wikipedia

View all activity

Organizations

replied to their post about 9 hours ago

Hey, thanks for the comment! I updated the repo with all the bench_*v2.md files with the deviation and perp*_.log.

The short answer to your question is ± 0.27 (from bench_Q3_K_M_v2.md: 6.56 ± 0.27 t/s) - which is well within the noise floor. You nailed it. I'll do more rigorous checking in the future.

posted an update 1 day ago

Post

Quant              Size      PPL       vs F16   tg t/s
F16                8.05 GB   13.4304   —        2.52
Q8_0               4.28 GB   13.4409   +0.01    5.04
Q6_K               3.31 GB   13.4404   +0.01    5.93
Q5_K_M             2.89 GB   13.5362   +0.11    5.64
Q4_K_M             2.50 GB   13.7304   +0.30    6.50
Q4_K_M + imatrix   2.50 GB   13.6760   +0.25    6.41
Q3_K_M             2.08 GB   15.6641   +2.23    6.56
Q3_K_M + imatrix   2.08 GB   14.8237   +1.39    5.89

The winner is Q4_K_M - it gets the model down from 8.05 GB to 2.5 GB in size/load, and perplexity only moves by 0.30.

Q3 is the dropoff point where inmatrix makes a difference. Inmatrix drops Q3 from 15.66 to 14.82 PPL; Q4 it barely changed anything.

My now informed recommendation is: Q4 if you have the memory, imatrix Q3 if you don’t.

I uploaded all eight weights, the harness, and the full results here:

huggingface.co/b4ph/qwen3-4b-lowram-bench

I also made a tiny picker because apparently I needed to turn this into a whole project:

huggingface.co/spaces/b4ph/qwen3-4b-quant-picker

3 replies

nikki PRO

AI & ML interests

Recent Activity

Organizations

b4ph's activity