MaxGPT-3.5 (235M, chat fine-tuned)
MaxGPT-3 after supervised chat fine-tuning: it answers like an assistant.
What it is
- MaxGPT-3 (235M parameters, 16 blocks x 1024, 1,024-token context) fine-tuned on chat-only data for about one epoch: 300K steps at learning rate 5e-5, about 16 hours on one RTX 5070.
- Loss masking: the loss counts only the assistant's tokens, found at the byte level, so the model learns to reply rather than to imitate the user.
- Finished at about 1.87 train and 1.95 validation loss, the two tracking each other closely.
Results
Scored with several samples per prompt by independent raters, chat fine-tuning took the model from 25% to 92% on resolving a reference back across turns, while single-turn quality moved from about 2.5 to 2.3 out of 5. Neither MaxGPT-3 nor 3.5 can recall a name stated earlier in a conversation, the gap Max's MaxGPT-Planck research studies.
What to expect
Clean assistant formatting that stays on topic more often than MaxGPT-3, with content that is still often wrong: it is a 235M model.
Run it
pip install torch safetensors
python run.py "USER: How do I make brownies?\nASSISTANT:"
run.py loads model.safetensors with the model's own code in this folder (config.py, model.py, tokenizer.py, sample.py) and samples 120 tokens at temperature 0.8, top-k 50, on CUDA, Apple MPS or CPU.
The MaxGPT family
Built from scratch by Maxwell Brohm, a high school senior, in PyTorch with no pretrained weights and no transformers library. Code and full write-up: https://github.com/MaxwellBrohm/MaxGPT.
| Model | Parameters | Layers x width | Context | What it shows |
|---|---|---|---|---|
| MaxGPT-1 | 23M | 6 x 384 | 256 | coherent English emerges |
| MaxGPT-2 | 110M | 12 x 768 | 512 | learns the USER/ASSISTANT chat format |
| MaxGPT-3 | 235M | 16 x 1024 | 1024 | a real-looking chatbot |
| MaxGPT-3.5 | 235M | MaxGPT-3 plus chat fine-tuning | 1024 | replies like an assistant |
| MaxGPT-Mini (Neo series) | 113M | 12 x 768, grouped-query attention, RoPE, SwiGLU | 1024 | the modern design, chat-tuned with SFT and DPO |
MaxGPT-Ultra, a 1.09B model on a modern decoder design, is in progress.