mpu-30m-base

mpu-30m is a 30.3M-parameter GPT-style language model trained from scratch on 400M tokens of TinyStories. It is the first model in the mpu series and the proof-of-concept run for an elastic, checkpoint-driven training platform that migrates a single training job across free and disposable GPU sessions (Colab/Kaggle) without losing progress.

It writes short, simple children's stories. It is small on purpose: the goal of this run was to validate the training infrastructure end to end, not to compete on quality.

Model details

Architecture	Decoder-only transformer (GPT-2 style, pre-LN, tied embeddings)
Parameters	30.3M total (10.6M non-embedding)
Layers / heads / width	6 / 6 / 384
Context length	1024
Vocabulary	GPT-2 BPE, 50,304 (padded)
Precision (training)	fp16 + loss scaling (Tesla T4)
Format	safetensors

Training

Data	TinyStories, ~400M tokens (GPT-2 tokenizer, uint16 memmap shards)
Steps	6,104 (65,536 tokens/step: batch 8 × 1024 ctx × grad-accum 8)
Optimizer	AdamW (β=0.9/0.95, wd 0.1), lr 6e-4, cosine to 10%, 300 warmup steps
Hardware	1× NVIDIA T4 (free Colab), ~2 hours, ~49k tokens/sec
Checkpointing	Atomic checkpoints pushed to this repo every 300 steps

The interesting part: how it was trained

This model was trained by an elastic training system in which compute is disposable and checkpoints are persistent. Every ~20 minutes the trainer writes a full checkpoint (weights, optimizer, scaler, RNG state) and promotes it atomically to this repo. A session can be killed at any time; any other GPU session resumes from the latest promoted checkpoint and — because data batches are a pure function of (seed, step) — continues bit-identically, verified by test. This run survived a mid-training kill-and-resume across sessions.

Usage

The checkpoint uses a custom (nanoGPT-style) architecture and is not loadable via transformers.AutoModel. Weights are standard safetensors (see config.json in the checkpoint folder for the architecture: 6 layers, 6 heads, width 384, GPT-2 BPE tokenizer). The training and inference code is not yet public; it will be released alongside a later model in the series.

Sample output

Once upon a time there was a little robot. He was very happy and liked to roll with his friends. But one day, he rolled too fast and fell into a big puddle. He tried to roll out of his wet puddle, but he couldn't. He was stuck and couldn't get out. Luckily, a kind little girl saw the robot and knew just what to do. [...] From then on, the robot was extra careful.

Limitations

• Trained only on synthetic children's stories: tiny vocabulary in practice, simple grammar, no factual knowledge, English only.
• At this scale the model loses track of characters and pronouns, and occasionally substitutes a wrong noun mid-story.
• No instruction tuning, no safety tuning, no formal evaluation. Not for any production use — this is an educational artifact.