slm-lab-base-15m

A 12,587,904-parameter decoder-only transformer, trained from scratch on TinyStories on a free Colab T4 — this repo is the pretrained base model.

It predicts the next token of a children's story, and that is all it does. Its purpose is to make the pretraining stack measurable: every component — RMSNorm, RoPE, grouped-query attention, SwiGLU, the tokenizer, the packing, the resume — was written in this project and is checked against this model's own numbers.

The pretrained base. Continues text; it does not follow instructions, because nothing has post-trained it yet.

A research artifact for learning, not a product: built to understand and measure the training stack, and of no practical use as anything else.

How it was built

Parameters	12,587,904
d_model / layers / heads	384 / 6 / 6 (2 KV heads, GQA)
Vocabulary	8192, byte-level BPE trained in this project
Context length	512
Architecture	RMSNorm, RoPE, SwiGLU, tied embeddings
Data	TinyStories, full train split (2,119,719 documents)
Optimizer	AdamW, lr 0.0004 → 4e-05, cosine, 120 warmup steps
Precision	float16 with GradScaler (the T4 has no bf16)

RMSNorm, RoPE, grouped-query attention and SwiGLU are each implemented twice in the repository — once slow and readable, once fast — with tests asserting the two agree to floating-point tolerance.

The run is safe to relaunch after any session kill: it resumes from its last checkpoint, and the resume is verified bit-identical — weights, Adam moments and data-loader position all agreeing to 0.000e+00 across a process boundary.

Results

Validation loss	1.5581
Perplexity	4.75
Bits per byte	0.1437
Tokens seen	294,912,000
Training throughput	81,602 tok/s (11.3% MFU) on one Tesla T4
Peak memory	2,873 MB
Chat-format compliance (never trained for it)	0.0%
Benchmark accuracy above chance	+0.0022 mean, over 4 multiple-choice tasks

Validation is held out by document, not by offset. A random-offset split on a packed token stream leaks the tail of a training document into validation and makes validation loss come out below training loss — which is how a data-handling bug disguises itself as a good result.

The benchmark row is the point of publishing a 12.6M model at all: it scores at chance, deliberately measured rather than omitted, and the same four tasks put a 1.24B model well above chance. That brackets the capability floor across a factor of 98.

Every number above is read from this repo's own training record, and the full write-ups are published under results/ at https://huggingface.co/datasets/Dhevenddra/slm-lab-data.

Limitations

Specific, and written out in full — this is the section a knowledgeable reader checks first.

• It only knows children's stories. TinyStories is synthetic text in a deliberately small vocabulary of simple English. The model has seen nothing else and will not handle anything else.
• It scores at chance on every standard benchmark. That is what the scaling curve predicts at this size, not a training defect — and it is measured above rather than asserted.
• 512-token context, with RoPE. It degrades past that length rather than failing outright, but it has never been trained there.
• No safety training of any kind. No filtering, no alignment, no refusal behaviour. It is a next-token predictor over children's stories.
• Not evaluated for bias. The corpus is synthetic and narrow, which constrains the harm surface but is not the same as having checked.

Contamination: ruled out, with confidence. The entire corpus was controlled within this project, so benchmark test data cannot have appeared in pretraining. That is an unusual thing to be able to say and it is why these models are useful as a floor measurement.

Reproducing it

git clone https://github.com/Dhevenddra/slm-lab && cd slm-lab/slm-lab
pip install -e ".[dev,data]"
python scripts/phase03_pretrain.py \
    --tokenizer artifacts/tokenizers/tinystories_8192.json

Where this came from

Part of slm-lab — a language model built end to end on free compute: a hand-written BPE tokenizer, a transformer from scratch, a controlled ablation study with a measured noise floor, post-training, QLoRA on a 1B model, DPO, and quantization written by hand and benchmarked against every production method.

• Start here: the SLM Lab collection lists every repo in the order it was built, with a one-line finding on each.
• Code, guide and the full failure log: https://github.com/Dhevenddra/slm-lab
• Every result file, dataset and tokenizer: https://huggingface.co/datasets/Dhevenddra/slm-lab-data

This repo is the project's pretraining (phase 03) stage; the published write-up for it is under results/ in the dataset repo, named for that phase.

The project's own failure log (FAILURES.md, 26 entries) is the part most worth reading. The recurring shape: the instrument measured something correlated with the question rather than equal to it.