TyrkerS/handmadellm-30m

A Llama-style decoder-only LLM built entirely from scratch — own byte-level BPE tokenizer, RoPE + RMSNorm + grouped-query attention + SwiGLU, hand-written training loop. Trained on TinyStories on a single RTX 5070 Ti.

• Params: 28M
• Architecture: dim 512, 8 layers, 8 heads (4 KV heads, GQA), context 512, vocab 8192
• Val bits-per-byte: 0.4556

Code, training scripts, evals, quantization, a fused Triton kernel, and the full writeup: https://github.com/TyrkerS/HandmadeLLM

Usage

import torch
from llm.config import ModelConfig, _build
from llm.model import Transformer
from llm.bpe import BPETokenizer

ckpt = torch.load("best.pt", map_location="cpu", weights_only=False)
model = Transformer(_build(ModelConfig, ckpt["config"]["model"]))
model.load_state_dict(ckpt["model"]); model.eval()
tok = BPETokenizer.load("tokenizer.json")

ids = torch.tensor([tok.encode("Once upon a time")])
out = model.generate(ids, max_new_tokens=200, temperature=0.8, top_k=50, eos_id=tok.eot_id)
print(tok.decode(out[0].tolist()))

This model is a research/portfolio artifact trained on TinyStories; it writes simple children's-story English and is not an instruction-following assistant (unless you loaded the SFT/DPO variant).