Aashw1n/openwebtext-103M

OpenWebText 103M

A 102.7M parameter decoder-only transformer trained from scratch on an OpenWebText sample, with a custom byte-level BPE tokenizer. No transformers, no pretrained components — every layer written from primitives.

Results

Validation loss	3.516 (perplexity 33.6)
Parameters	102.7M total, 56.6M non-embedding
Training tokens	819.2M
Hardware	1x A100 80GB, ~3.5 hours

Comparable in size to GPT-2 small (124M), which reaches 2.85 on OpenWebText with substantially more training. This model saw 819M tokens against 102.7M parameters (8:1, or ~14.5:1 excluding embeddings) — well under Chinchilla-optimal, a deliberate tradeoff against a fixed GPU-hour budget.

Architecture

Pre-norm decoder. RMSNorm, RoPE (theta=10000), SwiGLU FFN, causal multi-head attention. 8 layers, 12 heads, d_model 768, d_ff 2048, context 512, vocab 30000. Embeddings are untied and account for 45% of parameters.

Trained with hand-written AdamW, cosine LR schedule with 700-step warmup, peak LR 6e-4, global-norm gradient clipping at 1.0, TF32 matmuls.

Usage

Not a transformers architecture. Requires torch, einops, safetensors.

import json, torch
from safetensors.torch import load_file
from model import TransformerLM
from tokenizer import Tokenizer

cfg = json.load(open("config.json"))
model = TransformerLM(
    cfg["vocab_size"], cfg["context_length"], cfg["num_layers"],
    cfg["d_model"], cfg["num_heads"], cfg["d_ff"], cfg["theta"],
)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

tok = Tokenizer.from_file("owt_vocab.json", "owt_merges.txt", ["<|endoftext|>"])

Limitations

Learns register convincingly — encyclopedic prose, news style, quoted dialogue — with no factual grounding. Generates confident historical nonsense. Prone to repetition loops under low-entropy sampling.

Prompt: The history of the Roman Empire

The history of the Roman Empire, however, is difficult to overstate. The Medieval days in the Middle Ages and early in the Middle Ages, under the pagan order, were filled with slaves who took advantage of the culture of the Christian religion. [...] In 1791, an important Jewish uprising against the Christian order was initiated.

Structurally sound encyclopedia prose; every factual claim is invented.