Tiny GPT (5.8M parameters)
A small GPT-2-style decoder-only transformer, written from scratch in PyTorch (no transformers classes), trained on a
laptop CPU on a subset of TinyStories. It continues short children's stories.
Code, training logs and the full write-up are in the GitHub repository.
Model details
| Architecture | pre-norm GPT, 6 layers, d_model 256, 8 heads, context 256, learned positions, tied embeddings |
| Parameters | 5,853,184 |
| Tokenizer | byte-level BPE, vocabulary 4096, trained on the training split only (tokenizer.json) |
| Files | model.safetensors, model_config.json (our ModelConfig), tokenizer.json, model_info.json |
Training
- Data: first 300,000 stories (60.8M tokens) of TinyStories V2 (GPT-4 generated); validation on 5,000 held-out stories.
- Run: 9,500 steps x 8,192 tokens = 77.8M tokens (about 1.28 passes over the data), AdamW, peak lr 1e-3 with warmup and cosine decay, 12 CPU threads, 7.9 hours of running time on one laptop (Intel Core Ultra 9 185H).
Evaluation
| Model | Validation loss | Perplexity |
|---|---|---|
| Uniform guess | 8.32 | 4096 |
| Unigram counts | 5.88 | 357 |
| Bigram counts | 3.65 | 38.4 |
| This model | 1.688 | 5.41 |
Perplexity is per BPE token with this 4096-token tokenizer, on TinyStories text, so it is not comparable to numbers from other tokenizers or datasets.
Intended use
Learning, demos and experiments with small language models; generating simple children's-story-style text.
Limitations and risks
- Narrow domain. It only knows the simple vocabulary and plots of TinyStories. It cannot follow instructions, answer questions or hold a conversation.
- Quality. Stories are fluent at sentence level but repeat phrases, lose objects and plot threads, and sometimes contradict themselves. Samples in the repository's
docs/RESULTS.mdinclude failures, not just good examples. - Synthetic data. The training stories were written by a large language model, so the model inherits that model's stylistic and social biases (names, gender roles, settings). It has not been evaluated for bias or safety.
- Not for factual content, advice, or anything where wrong or biased text could cause harm.
Data and licence
Code and weights in this repository are released under the MIT licence. The training data, TinyStories by Ronen Eldan and Yuanzhi Li, is licensed CDLA-Sharing-1.0 on the dataset card; please credit it and check the dataset card for current terms.
Usage
git clone <github repo> && cd tiny-gpt && make setup
TINYGPT_HF_REPO=<this repo id> TINYGPT_MODEL_DIR=./model uv run uvicorn tinygpt.serving.app:create_app --factory