miniLLM — 100M decoder-only transformer trained from scratch

A Llama-3-shaped decoder-only language model trained from scratch on FineWeb-Edu, on a single RTX 4060 Laptop (8GB). No distillation, no pretrained weights, no fine-tuning off someone else's base model — tokenizer, data pipeline, architecture, and training loop are all in the repo.

Its sibling is chessLLM, the same architecture retrained on chess notation to test whether next-token prediction alone builds a world model.

Architecture

Parameters	100.7M non-embedding (125.8M total)
Layers	12
Attention	12 query heads / 4 KV heads (GQA), head_dim 64
d_model	768
FFN	SwiGLU, hidden 2048
Position	RoPE, θ=10000
Context	1024 tokens
Vocab	32,768 (byte-level BPE, trained on this corpus)
Norm	RMSNorm, QK-norm
Embeddings	Tied
Optimizer	Muon (2-D hidden matrices) + AdamW (rest)
Precision	bf16
Final val loss	3.2840 (ppl 26.7) at step 9,144/9,144

Training

12.44 hours on one 8GB laptop GPU, base preset, Muon lr=0.02, WSD schedule.

Steps × tokens/step	9,144 × 131,072 = 1.20B tokens
Tokens per param	11.9 (under Chinchilla's 20 — a wall-clock budget, not a compute-optimal one)
Throughput	26.8k tok/s sustained, 5.6 GB of 7.6 GB VRAM

Val loss by step: 3.83 @1.7k → 3.69 @3.4k → 3.63 @5.4k → 3.60 @7.4k → 3.28 @9.1k. The last leg is the WSD decay phase, and most of the final drop happens there — stopping before the decay completes leaves roughly 0.3 nats on the table.

Muon is the single biggest win. In a controlled A/B at a matched 200-step / 13.1M-token budget, Muon (lr=0.02) reached val 5.0917 against AdamW (lr=1.5e-3) at 6.0712 — −0.98 nats, 2.7× lower perplexity — for ~2% less throughput. That is an early-training comparison, not a full-run one. Details in the repo README.

Decoding matters more than the loss curve suggests

This is the most useful practical finding, and the defaults below are not the obvious ones.

The model ranks true facts above plausible false ones 95% of the time (40 common-knowledge cloze pairs, chance = 50%). But at temperature 0.8, free generation only stated the correct fact 30% of the time — a 65-point gap between what the model knows and what it says. That is sampling noise, not missing knowledge.

temperature	rep. penalty	fact%	worst repeated-4gram%
0.8	1.0	30–50%	~74%	naive defaults
0.3	1.25	67%	5%	recommended
0.0	1.35	80%	2%	max factuality
0.0	1.50	80%	0%	over-penalised; starts padding with enumerations

Lower the temperature for facts, raise it for variety. The repetition penalty (CTRL-style, windowed to the last 256 tokens) is what removes the "the National Park of Pakistan became the National Park of Pakistan" failure mode. Below ~1.25 this model loops badly; above ~1.4 it starts avoiding words it needs.

Factual accuracy climbed monotonically through the entire run — 77.5% at step 1,713 to 95.0% at step 9,144, still rising at the end. Val loss hides this: the constant-LR phase looks flat (−0.020 nats/1k steps) while knowledge accumulates.

Usage

Download ckpt.pt and tokenizer_v2.json — the tokenizer is not bundled in the checkpoint and generation needs both.

import torch
from model import MiniLLM, ModelConfig      # from github.com/amanmprojects/llm
from tokenizers import Tokenizer

ck  = torch.load("ckpt.pt", map_location="cpu", weights_only=False)
cfg = ModelConfig(**{k: v for k, v in ck["cfg"].items()
                     if k in ModelConfig.__dataclass_fields__})
model = MiniLLM(cfg)
model.load_state_dict(ck["model"])
model.eval()

tok = Tokenizer.from_file("tokenizer_v2.json")
ids = torch.tensor([tok.encode("The causes of the French Revolution were").ids])
out = model.generate(ids, 120, temperature=0.3, repetition_penalty=1.25)
print(tok.decode(out[0].tolist()))

Or from a clone of the repo, which fetches both files for you:

python scripts/download_weights.py text
python sample.py --prompt "The causes of the French Revolution were"

sample.py already applies the tuned defaults above (temperature 0.3, repetition penalty 1.25).

Limitations

• 100M parameters trained on 1.2B tokens. It writes fluent, mostly-grammatical encyclopedic prose and knows a lot of common facts, but it confabulates specifics freely — dates, names, and numbers should be assumed wrong unless checked.
• Base model, not a chat model. It completes text; it does not follow instructions. A second phase adding continued pretraining plus a chat mixture was designed and is documented in the repo, but it crashed at step 1,750 of 7,535 (laptop suspend killed the CUDA context) and its partial checkpoint is worse than this one (val 3.469 vs 3.284). This upload is the completed run 1, which is the best checkpoint that exists.
• 1024-token context. No long-context capability.
• Sensitive to decoding settings, as the table above shows. At high temperature it will state falsehoods it internally ranks as false.
• English, educational-web register, inherited from FineWeb-Edu. No code or math specialization, no multilingual capability, and it carries whatever biases that corpus contains.
• Not safety-tuned in any way. There is no RLHF, no refusal behaviour, no filtering beyond the corpus itself.

Data

FineWeb-Edu (CC-MAIN shards), tokenized with a byte-level BPE trained on the same corpus. Weights are MIT-licensed; the underlying text belongs to its original authors under FineWeb-Edu's terms.

Full code, training pipeline, and evaluation harness: https://github.com/amanmprojects/llm

This checkpoint is inference-only — bf16, no optimizer state, so it cannot resume training.