MaxGPT-1 (23M)
The first of four language models built from scratch. Written entirely by hand: no line of its code was written by an AI.
What it is
- 23M parameters: 6 transformer blocks, hidden size 384, 6 heads, a 256-token context.
- Its own byte-pair tokenizer (16,000 tokens), trained on a sample of the data.
- Trained in about 30 minutes on one RTX 5070 (12 GB), 15,000 steps, on TinyStories plus OpenAssistant, the same small mix as MaxGPT-2 so the only thing that changes between them is size.
What to expect
Word salad with English shape, which is the point of the smallest model in the series: you can watch coherence appear as the models grow.
Run it
pip install torch safetensors
python run.py "USER: How do I make brownies?\nASSISTANT:"
run.py loads model.safetensors with the model's own code in this folder (config.py, model.py, tokenizer.py, sample.py) and samples 120 tokens at temperature 0.8, top-k 50, on CUDA, Apple MPS or CPU.
The MaxGPT family
Built from scratch by Maxwell Brohm, a high school senior, in PyTorch with no pretrained weights and no transformers library. Code and full write-up: https://github.com/MaxwellBrohm/MaxGPT.
| Model | Parameters | Layers x width | Context | What it shows |
|---|---|---|---|---|
| MaxGPT-1 | 23M | 6 x 384 | 256 | coherent English emerges |
| MaxGPT-2 | 110M | 12 x 768 | 512 | learns the USER/ASSISTANT chat format |
| MaxGPT-3 | 235M | 16 x 1024 | 1024 | a real-looking chatbot |
| MaxGPT-3.5 | 235M | MaxGPT-3 plus chat fine-tuning | 1024 | replies like an assistant |
| MaxGPT-Mini (Neo series) | 113M | 12 x 768, grouped-query attention, RoPE, SwiGLU | 1024 | the modern design, chat-tuned with SFT and DPO |
MaxGPT-Ultra, a 1.09B model on a modern decoder design, is in progress.