MaxGPT-2 (110M)
The second of four language models built from scratch: 2x deeper and 2x wider than MaxGPT-1.
What it is
- 110M parameters: 12 transformer blocks, hidden size 768, 12 heads, a 512-token context.
- The same 16,000-token byte-pair tokenizer as MaxGPT-1, written from scratch.
- Trained for about 10 hours on one RTX 5070 (12 GB) on TinyStories plus OpenAssistant, the same mix as MaxGPT-1, so the comparison is about size alone.
What to expect
Grammatical and recipe-shaped, and it has learned the USER/ASSISTANT format, but the content wanders (ask it for brownies and it may reach for cereal).
Run it
pip install torch safetensors
python run.py "USER: How do I make brownies?\nASSISTANT:"
run.py loads model.safetensors with the model's own code in this folder (config.py, model.py, tokenizer.py, sample.py) and samples 120 tokens at temperature 0.8, top-k 50, on CUDA, Apple MPS or CPU.
The MaxGPT family
Built from scratch by Maxwell Brohm, a high school senior, in PyTorch with no pretrained weights and no transformers library. Code and full write-up: https://github.com/MaxwellBrohm/MaxGPT.
| Model | Parameters | Layers x width | Context | What it shows |
|---|---|---|---|---|
| MaxGPT-1 | 23M | 6 x 384 | 256 | coherent English emerges |
| MaxGPT-2 | 110M | 12 x 768 | 512 | learns the USER/ASSISTANT chat format |
| MaxGPT-3 | 235M | 16 x 1024 | 1024 | a real-looking chatbot |
| MaxGPT-3.5 | 235M | MaxGPT-3 plus chat fine-tuning | 1024 | replies like an assistant |
| MaxGPT-Mini (Neo series) | 113M | 12 x 768, grouped-query attention, RoPE, SwiGLU | 1024 | the modern design, chat-tuned with SFT and DPO |
MaxGPT-Ultra, a 1.09B model on a modern decoder design, is in progress.