MaxGPT-Mini (113M)

The first model of the Neo series: a modern decoder design at 113M parameters, pretrained from scratch, then chat-tuned with supervised fine-tuning and DPO. It was built to prove out the whole pipeline before the 1.09B MaxGPT-Ultra, and Max judged it better than the 235M MaxGPT-3 in side-by-side chats.

What it is

• 113M parameters: 12 blocks, width 768, 12 query heads sharing 4 key/value heads (grouped-query attention), a SwiGLU feed-forward (2,048 wide), RMSNorm, rotary position embeddings, QK-norm, and tied input and output embeddings.
• A 1,024-token context and its own 49,152-token byte-level BPE tokenizer (digits split, byte fallback), with ChatML turns (<|im_start|>user ... <|im_end|>).
• Trained from scratch on one RTX 5070 (12 GB): pretraining, then supervised fine-tuning, then DPO. This checkpoint is the DPO stage's.

Run it

pip install -r requirements.txt
python run.py "How do I make brownies?"

run.py builds the model from config.yaml with the code in this folder (model/, tokenizer/, cfg.py), loads model.safetensors, wraps your question as a chat turn and samples up to 200 tokens (temperature 0.8, top-k 50), stopping at the end of the reply. Runs on CUDA, Apple MPS or CPU.

What to expect

Well-formed, assistant-style answers (numbered steps, on topic) whose content is often wrong: it is a 113M model. MaxGPT-Ultra adds gated attention, a normalized value residual, depth-scaled norms and the NorMuon optimizer, the recipe that won an A/B test of four variants at this model's size.

The MaxGPT family

Built from scratch by Maxwell Brohm, a high school senior, in PyTorch with no pretrained weights and no transformers library. Code, training curves and write-up: github.com/MaxwellBrohm/MaxGPT.

Model	Parameters	Layers x width	Context	What it shows
MaxGPT-1	23M	6 x 384	256	coherent English emerges
MaxGPT-2	110M	12 x 768	512	learns the USER/ASSISTANT chat format
MaxGPT-3	235M	16 x 1024	1024	a real-looking chatbot
MaxGPT-3.5	235M	MaxGPT-3 plus chat fine-tuning	1024	replies like an assistant
MaxGPT-Mini (Neo series)	113M	12 x 768, grouped-query attention, RoPE, SwiGLU	1024	the modern design, chat-tuned with SFT and DPO

MaxGPT-Ultra, a 1.09B model on a modern decoder design, is in progress.