slm-lab-chat-15m

A 12,587,904-parameter decoder-only transformer, trained from scratch on TinyStories on a free Colab T4 — this repo is the mid-trained model.

It exists to isolate one question: how much of chat behaviour comes from continued pretraining on chat-shaped text, before any instruction tuning at all? The answer this model gives is almost none of the format, and a measurable amount of forgetting — which is the useful result, and the table below is it.

The base model continued on a chat-formatted corpus, so it has seen the template before it is asked to produce it. Mid-training is the step that is usually skipped and usually missed.

A research artifact for learning, not a product: built to understand and measure the training stack, and of no practical use as anything else.

How it was built

Parameters	12,587,904
d_model / layers / heads	384 / 6 / 6 (2 KV heads, GQA)
Vocabulary	8192, byte-level BPE trained in this project
Context length	512
Architecture	RMSNorm, RoPE, SwiGLU, tied embeddings
Data	TinyStories, full train split (2,119,719 documents)
Optimizer	AdamW, lr 0.0004 → 4e-05, cosine, 120 warmup steps
Precision	float16 with GradScaler (the T4 has no bf16)

RMSNorm, RoPE, grouped-query attention and SwiGLU are each implemented twice in the repository — once slow and readable, once fast — with tests asserting the two agree to floating-point tolerance.

The run is safe to relaunch after any session kill: it resumes from its last checkpoint, and the resume is verified bit-identical — weights, Adam moments and data-loader position all agreeing to 0.000e+00 across a process boundary.

Results

	before	after	delta
Chat-corpus loss	5.4059	2.8836	-2.5222
TinyStories loss	1.5732	2.1276	+0.5544

Tokens seen	24,576,000
Epochs over the chat corpus	1.22
Chat corpus	HuggingFaceTB/smoltalk, 20,137,800 tokens
Chat-format compliance after mid-training	0.0%

Read the two rows together. Mid-training gained 2.5222 nats on conversational text and lost 0.5544 nats on the original corpus — against a seed noise floor this project measured at 0.0146 nats, so both movements are real, and the forgetting is roughly 38× the noise rather than a rounding artifact.

And it bought no format at all: well-formed assistant turns stay at 0.0%. Seeing the template is not learning to emit it. That takes supervised fine-tuning, which is the next repo.

Every number above is read from this repo's own training record, and the full write-ups are published under results/ at https://huggingface.co/datasets/Dhevenddra/slm-lab-data.

Limitations

Specific, and written out in full — this is the section a knowledgeable reader checks first.

• It only knows children's stories. TinyStories is synthetic text in a deliberately small vocabulary of simple English. The model has seen nothing else and will not handle anything else.
• It scores at chance on every standard benchmark. That is what the scaling curve predicts at this size, not a training defect — and it is measured above rather than asserted.
• 512-token context, with RoPE. It degrades past that length rather than failing outright, but it has never been trained there.
• No safety training of any kind. No filtering, no alignment, no refusal behaviour. It is a next-token predictor over children's stories.
• Not evaluated for bias. The corpus is synthetic and narrow, which constrains the harm surface but is not the same as having checked.

Contamination: ruled out, with confidence. The entire corpus was controlled within this project, so benchmark test data cannot have appeared in pretraining. That is an unusual thing to be able to say and it is why these models are useful as a floor measurement.

Reproducing it

git clone https://github.com/Dhevenddra/slm-lab && cd slm-lab/slm-lab
pip install -e ".[dev,data]"
python scripts/phase03_pretrain.py \
    --tokenizer artifacts/tokenizers/tinystories_8192.json

Where this came from

Part of slm-lab — a language model built end to end on free compute: a hand-written BPE tokenizer, a transformer from scratch, a controlled ablation study with a measured noise floor, post-training, QLoRA on a 1B model, DPO, and quantization written by hand and benchmarked against every production method.

• Start here: the SLM Lab collection lists every repo in the order it was built, with a one-line finding on each.
• Code, guide and the full failure log: https://github.com/Dhevenddra/slm-lab
• Every result file, dataset and tokenizer: https://huggingface.co/datasets/Dhevenddra/slm-lab-data

This repo is the project's mid-training (phase 05) stage; the published write-up for it is under results/ in the dataset repo, named for that phase.

The project's own failure log (FAILURES.md, 26 entries) is the part most worth reading. The recurring shape: the instrument measured something correlated with the question rather than equal to it.