SLM-30M-Base

A 30M-parameter decoder-only GPT model, built and pretrained entirely from scratch as a personal learning project to understand how language models work under the hood — not intended for production use.

Architecture

• Decoder-only GPT, 8 layers, 8 attention heads
• 512 embedding dimension, 512 token context window
• Weight-tied input/output embeddings
• PyTorch scaled_dot_product_attention (FlashAttention where available)

Training

• Data: English Wikipedia (wikimedia/wikipedia, 20231101.en)
• Tokenizer: custom byte-level BPE, 16,384 vocabulary
• Tokens seen: [fill in your total token count if you have it]
• Optimizer: AdamW (β1=0.9, β2=0.95, weight decay 0.1)
• LR schedule: cosine annealing with warmup (peak 1e-3, min 1e-4)
• Stability: gradient accumulation + gradient clipping (1.0)

Purpose

This is a from-scratch implementation built purely for learning — to understand tokenization, attention, and the GPT training loop in depth rather than just calling a library. Not fine-tuned for any downstream task (see slm-30m-qa for the fine-tuned Q&A version).

Usage

import torch
model = torch.load("latest.pt", map_location="cpu")
# add your model class definition before loading if using state_dict rather than full pickle

Limitations

Trained on a single Wikipedia dump with a reduced vocabulary and small parameter count — not comparable to production LLMs, and may produce incoherent or repetitive text on out-of-distribution prompts.