Sprout 🌱 — 32.5M parameters, seven hours on an RTX 4060

A small English story-completion transformer trained from random initialization on a prefix of TinyStories. Not a chatbot or instruction-following model.

Code and training artifacts: https://github.com/amanmprojects/sprout

Results

• 32,514,560 parameters; 8 layers, width 512, 8 heads, context 512.
• RoPE, RMSNorm, SwiGLU, untied embeddings, bf16 flash attention.
• Custom byte-level BPE: 8,192 tokens.
• Training corpus: 378,032 complete stories / 82,162,213 tokens.
• Seven-hour run: 89,990 updates / 1,658,695,680 tokens processed, with repeated corpus sampling.
• Released checkpoint: step 67,861; best monitored validation loss 1.4100.
• Final checkpoint had worse monitored validation loss (1.4268); it is not the inference release.

Validation uses 24,576 tokens in fixed sampled windows from the official validation split, not the entire validation corpus. The same sample selected the best checkpoint. No independent test benchmark was run. See FINDINGS.md for the recipe, samples, and limitations.

Run locally

python -m pip install torch==2.8.0 tokenizers==0.23.2 huggingface_hub
python -c "from huggingface_hub import snapshot_download; snapshot_download('amanm10000/sprout', local_dir='sprout')"
cd sprout
python sample.py --checkpoint best.pt --tokenizer tokenizer.json --prompt "Once upon a time, there was a little fox who lived in the forest."

CPU is the default. Add --device cuda for a bf16-capable CUDA GPU. This checkpoint uses the included custom PyTorch implementation; it is not directly loadable via Transformers AutoModel or a hosted Inference API. Review downloaded Python code before executing it.

best.pt contains float32 model weights, architecture configuration, tokenizer hash, and checkpoint metadata. Optimizer state was removed; this release is for inference, not exact training resume.

Limitations and provenance

Learns simple grammar and story structure, but invents props, changes pronouns and speaker roles, and loses plot continuity. May generate biased, unsafe, or incorrect text. It has not been safety-aligned and should not be used as an unsupervised children's application or a factual assistant.

Source dataset: TinyStories, with upstream CDLA-Sharing-1.0 terms. Raw training data is not redistributed here. No separate blanket license grant is asserted for the model in this release; review upstream terms and contact the repository owner for licensing questions.