mini-gpt-shakespeare

A decoder-only Transformer trained from scratch (no nn.Transformer, no scaled_dot_product_attention — attention is hand-written Q/K/V, scaling, causal mask, softmax, weighted values).

Corpus	tiny Shakespeare (char-level, 65-token vocab)
Architecture	4 layers, 4 heads, 128-dim, context 128, pre-norm
Training	5,000 steps, AdamW, RTX 5060
Validation loss	4.24 → 1.61

Checkpoint format: torch.save dict with format_version, config, tokenizer, model_state_dict, step, metrics — loads with checkpoint.load_checkpoint(path, device) from nkthienai-lucky/mini-gpt.

Demo prompt (greedy): "ROMEO:" → Shakespeare-like speaker headings and verse.