slm-lab-sft-15m

A 12,587,904-parameter decoder-only transformer, trained from scratch on TinyStories on a free Colab T4 — this repo is the instruction-tuned model.

It exists to separate two things that are usually reported as one: learning the chat protocol and being useful. This model learns the protocol almost completely and stays nearly useless, and that dissociation is the finding.

Supervised fine-tuning on top of the mid-trained model, with the loss masked to the assistant turns only.

A research artifact for learning, not a product: built to understand and measure the training stack, and of no practical use as anything else.

How it was built

Parameters	12,587,904
d_model / layers / heads	384 / 6 / 6 (2 KV heads, GQA)
Vocabulary	8192, byte-level BPE trained in this project
Context length	512
Architecture	RMSNorm, RoPE, SwiGLU, tied embeddings
Data	TinyStories, full train split (2,119,719 documents)
Optimizer	AdamW, lr 0.0004 → 4e-05, cosine, 120 warmup steps
Precision	float16 with GradScaler (the T4 has no bf16)

RMSNorm, RoPE, grouped-query attention and SwiGLU are each implemented twice in the repository — once slow and readable, once fast — with tests asserting the two agree to floating-point tolerance.

The run is safe to relaunch after any session kill: it resumes from its last checkpoint, and the resume is verified bit-identical — weights, Adam moments and data-loader position all agreeing to 0.000e+00 across a process boundary.

Results

Validation loss, response tokens only	4.0842 → 1.9476 (-2.1366)
Well-formed assistant turns	72.0% (0.0% before fine-tuning)
Stopped at the end-of-turn token	72.0%
Leaked a role token	0.0%
Ran to the token ceiling instead of stopping	28.0%
Mean new tokens per answer	128
Training examples	2,488
Epochs	3
Benchmark accuracy above chance	+0.0062 mean, over 4 multiple-choice tasks

The dissociation is the finding. Format compliance goes from 0% to 72.0% in 3 epochs over 2,488 examples: nothing but supervised fine-tuning produces the protocol, and it produces it almost immediately.

Whether the answers are any good is a separate question, and it moved far less. Usefulness was scored by hand — a rate computed over automatic checks would only have re-measured the format — and it is reported with its rubric in results/phase05.md, alongside the detail that decides how to read it: the base model is already non-zero on it, because a model that answers nothing still lands some prompts by accident. That accidental floor is what any responsiveness claim has to beat, and a phase scoring this model against an assumed floor of zero would have reported a large effect where there is a small one.

Those hand scores live only in that write-up's prose: the published phase05_eval.json and phase05_scores.jsonl carry responsive: null, so the rate is not reproducible from the records here. It is cited rather than restated for that reason.

Every number above is read from this repo's own training record, and the full write-ups are published under results/ at https://huggingface.co/datasets/Dhevenddra/slm-lab-data.

Limitations

Specific, and written out in full — this is the section a knowledgeable reader checks first.

• It only knows children's stories. TinyStories is synthetic text in a deliberately small vocabulary of simple English. The model has seen nothing else and will not handle anything else.
• It scores at chance on every standard benchmark. That is what the scaling curve predicts at this size, not a training defect — and it is measured above rather than asserted.
• 512-token context, with RoPE. It degrades past that length rather than failing outright, but it has never been trained there.
• No safety training of any kind. No filtering, no alignment, no refusal behaviour. It is a next-token predictor over children's stories.
• Not evaluated for bias. The corpus is synthetic and narrow, which constrains the harm surface but is not the same as having checked.

Contamination: ruled out, with confidence. The entire corpus was controlled within this project, so benchmark test data cannot have appeared in pretraining. That is an unusual thing to be able to say and it is why these models are useful as a floor measurement.

Reproducing it

git clone https://github.com/Dhevenddra/slm-lab && cd slm-lab/slm-lab
pip install -e ".[dev,data]"
python scripts/phase03_pretrain.py \
    --tokenizer artifacts/tokenizers/tinystories_8192.json

Where this came from

Part of slm-lab — a language model built end to end on free compute: a hand-written BPE tokenizer, a transformer from scratch, a controlled ablation study with a measured noise floor, post-training, QLoRA on a 1B model, DPO, and quantization written by hand and benchmarked against every production method.

• Start here: the SLM Lab collection lists every repo in the order it was built, with a one-line finding on each.
• Code, guide and the full failure log: https://github.com/Dhevenddra/slm-lab
• Every result file, dataset and tokenizer: https://huggingface.co/datasets/Dhevenddra/slm-lab-data

This repo is the project's supervised fine-tuning (phase 05) stage; the published write-up for it is under results/ in the dataset repo, named for that phase.

The project's own failure log (FAILURES.md, 26 entries) is the part most worth reading. The recurring shape: the instrument measured something correlated with the question rather than equal to it.