CompactLM-5M

A ~6.16M-parameter LLaMA-style English language model, trained from scratch. Built for a community request (model-requests #14, DedeProGames): "LLaMA-style, ~5M params, fineweb-edu."

What it is

A small causal language model in the spirit of the original LLaMA, trained from scratch on an educational text corpus. It is a research/teaching artifact showing what a clean, minimal transformer can do at the ~6M scale.

Architecture

Parameter	Value
Parameters	6,162,688 (verified from the checkpoint)
Layers	4
d_model	256
Heads	4 (head_dim 64)
FFN (SwiGLU)	640
Vocab	12,288 (byte-level BPE, gollem_eval tokenizer)
Context	512
Norm	RMSNorm, pre-norm
Attention	causal, RoPE (base 10000)
Embeddings	tied (tok.weight == head.weight)
Dtype	float32

Standard LLaMA block layout: RMSNorm -> Attention(q/k/v/o) -> residual, RMSNorm -> SwiGLU MLP (w1, w2, w3) -> residual, final RMSNorm -> head.

Training

• Data: HuggingFaceFW/fineweb-edu (train split), streamed. The requested dclm-baseline-1.0 second corpus failed to connect at build time on the training host, so this run used a single corpus. Logged here honestly.
• Budget: ~100M tokens over a 30–50 min GPU window (RTX 5090).
• Objective: next-token cross-entropy.

Results (measured, not asserted)

• Validation loss: 3.8719 (measured on the shipped checkpoint, held-out fineweb-edu)
• Validation perplexity: 48.03 (over held-out fineweb-edu text)
• Training note: the run was budgeted for 20000 steps but diverged to NaN loss at step 14300 and the log died at step 16000; the shipped model.safetensors is the checkpoint that was evaluated (numbers above).
• Degeneracy check: 0 / 15 samples flagged by the repeated-3-gram loop detector (a single 3-gram covering >60% of the 40-word tail). Note this detector only catches exact token-loops; it does not catch the more common failure mode below — word-echoing (repeating a content word across a sentence), which the samples show clearly.

Representative samples (temperature 0.8, top-k 40, verbatim from the shipped model.safetensors, from eval_fresh.json):

"The cat sat on the center of the church in the center of the church. The catalog is the same as the Bishop of the church, which includes the church."

"The sun rises in the air and is marked by the bubbles of the Earth. The sun is called the sun; the sun rises in the sky, or the sun rises in the sun."

"Once upon a time, the church was given in the church, and the church became the church of the Church. Apart from the church, the church was given and the church was built."

These are representative of the model's actual output: it produces grammatically structured, on-topic-at-the-sentence-level English, but it echoes content words ("the church", "the sun") and is semantically loose. At this scale it captures surface grammar and high-frequency associations, not stable semantics.

What it is good at / not good at

• Good at: producing grammatically structured, on-topic English at the sentence level. It knows common word order, function words, and some world-fact associations.
• Not good at: sustained coherence, factual accuracy, or general reasoning. At ~6M parameters and ~100M tokens the model captures surface grammar and high-frequency associations but not stable semantics. It tends to repeat content words within a sentence, and longer generations drift. Treat it as a grammar/scale study, not a useful assistant.

Files

File	Description
model.safetensors	39 tensors, float32, 37.2 MB. The tied head.weight is stored as its own tensor (values identical to tok.weight) so the file is self-contained.
config.json	Architecture parameters.
tokenizer.json	Byte-level BPE tokenizer (12,288 vocab), tokenizers format.
train_compactlm5m.py	The exact training script (defines the CompactLM class).
eval_compactlm5m.py	The exact eval script (val PPL + generation + degeneracy check).

Loading

This is a custom architecture (not transformers-native). Load with the CompactLM class from train_compactlm5m.py:

import sys, torch
sys.path.insert(0, "<path-to-this-repo>")
from train_compactlm5m import CompactLM
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
m = CompactLM(12288, d=256, n_layers=4, n_heads=4, ff=640, ctx=512).eval()

from safetensors.torch import load_file
sd = {k: v for k, v in load_file("model.safetensors").items()
      if not k.startswith("head.weight")}   # head.weight is tied to tok.weight
m.load_state_dict(sd, strict=False)
m.head.weight = m.tok.weight

ids = tok.encode("The cat sat on the").ids
# ... run m.forward on ids, sample, decode