slm-lab-gguf

GGUF conversions of this project's two models — the fine-tuned 1B at four quantization levels and the 12.6M from-scratch model at two — so both can be run on a CPU and the quantization measurements can be reproduced.

What it is for. Quantization is usually reported as a single perplexity number per bit width. This repo exists so the same files can be scored a second way: KL divergence from the fp16 model, position by position, which answers "how far from the original model is this?" rather than "what is its average loss?" Those two questions have different answers here, and the table below is where they disagree.

The quantized files, so the quantization results can be reproduced and both models actually run without a GPU.

A research artifact for learning, not a product: these files exist to measure quantization. Every one is strictly worse than the fp16 model, and the question the results answer is by how much, and where.

How they were made

llama.cpp's converter and quantizer, run on the machine that trained the model, then loaded back through llama-cpp-python and scored on the same held-out sequences as every other row in the comparison, under one protocol version. Two details that decide whether the numbers mean anything:

• Tokenization is verified, not assumed. The chat-template text is tokenized with special=True and checked against the reference tokenizer before any scoring — 16 sequences checked, 0 mismatches. Without that flag the template's control tokens are split into ordinary text and every number below would be measuring a different input.
• KL is measured at response positions, four in each of 64 sequences, not at the last prompt position. At the last prompt position the model emits { with near-certainty, so KL there has no resolution and reads ~0 for every method.

Results

file	bits	size	perplexity	KL mean	KL median
fp16 (the reference)	16	—	22.0899	0	0
json1b-Q8_0.gguf	8	1.23 GB	22.1814	0.000458	2.5e-08
json1b-Q5_K_M.gguf	5	0.85 GB	20.9572	0.003421	3.6e-07
json1b-Q4_K_M.gguf	4	0.75 GB	19.4149	0.016311	2.3e-06
json1b-Q3_K_M.gguf	3	0.64 GB	20.8142	0.052735	4.7e-06

The 12.6M from-scratch model is converted too, so it can be run on a CPU with no GPU anywhere in the loop:

file	size
phase03-Q4_K_M.gguf	9.6 MB
phase03-Q8_0.gguf	13.0 MB

Perplexity and KL are deliberately not reported for these two. The comparison table's reference is the 1B model in fp16; scoring a different model with a different vocabulary against it would produce numbers that look like a comparison and are not one. The record on the Hub carries "measured": false for exactly that reason, and it is a choice rather than a gap.

Both ran under Ollama on the runtime that built them:

model	file	stop behaviour	throughput
ollama run slm-lab	9.6 MB, template slm	2/3 stopped on their own	946 tok/s
ollama run slm-lab-json1b	770.3 MB, template llama32	2/2 stopped on their own	160 tok/s

3 of the 4 quantizations score a lower perplexity than fp16 — Q4_K_M by 2.68. That is normally an evaluation bug; it is not one here, because token counts are identical across rows and a 16-bit null-quantization control reproduces the fp16 baseline exactly. A small weight perturbation moves a scalar average in a random direction, and with four rows some of them move down.

KL cannot do this, because it is a distance. It rises monotonically as the bit width falls, and it places Q4_K_M about 36× further from fp16 than Q8_0 — the opposite ordering to perplexity's. Lower perplexity does not mean closer to the reference model, and this table is the cleanest demonstration of it the project found.

Quantization damage is concentrated, not spread. Mean KL runs 10,000–35,000× the median. Most positions are untouched — the JSON braces and field names the model was already certain of — and the mean is carried by a few positions where it was uncertain: the computed date and amount fields. The weakest thing the model does is the first thing quantization takes away, and a single averaged perplexity says none of this.

Running one

hf download Dhevenddra/slm-lab-gguf json1b-Q4_K_M.gguf Modelfile-json1b --local-dir .
ollama create slm-lab-json1b -f Modelfile-json1b
ollama run slm-lab-json1b

Modelfile-json1b includes the SYSTEM line the adapter was trained with. Without it the template's system slot is empty and the model is asked at inference for something it was never asked for in training — it still generates, so the omission is easy to miss. Modelfile-phase03 does the same for the 12.6M model, whose template is this project's own.

Limitations

Specific, and written out in full — this is the section a knowledgeable reader checks first.

• Everything the adapter is limited by, inherited unchanged: one narrow synthetic task, an unknown base-model pretraining corpus, and no safety training of this project's own.
• Quantization damage lands on the computed fields. Not spread evenly — the date and amount fields, already the model's weakest, absorb nearly all of it. If those matter for your use, the quantized files are worse than the table's averages suggest.
• Perplexity alone will mislead you here, and this repo's own numbers are the demonstration. Compare on KL, or on task accuracy, as well.
• The 12.6M GGUFs are not comparable to the 1B rows. Different model, different vocabulary, different tokenizer; they are published to be run, not to be scored against the table above.

Where this came from

Part of slm-lab — a language model built end to end on free compute: a hand-written BPE tokenizer, a transformer from scratch, a controlled ablation study with a measured noise floor, post-training, QLoRA on a 1B model, DPO, and quantization written by hand and benchmarked against every production method.

• Start here: the SLM Lab collection lists every repo in the order it was built, with a one-line finding on each.
• Code, guide and the full failure log: https://github.com/Dhevenddra/slm-lab
• Every result file, dataset and tokenizer: https://huggingface.co/datasets/Dhevenddra/slm-lab-data

This repo is the project's quantization (phase 08) stage; the published write-up for it is under results/ in the dataset repo, named for that phase.

The project's own failure log (FAILURES.md, 26 entries) is the part most worth reading. The recurring shape: the instrument measured something correlated with the question rather than equal to it.