compacttest-5m — 5.11M-param subword LM (from scratch)

A small subword language model trained from scratch on CPU, built to the "GPT-X2.5" style spec requested in the model-requests board (gpt-S2.5-5m). It is a deliberately minimal, self-contained GPT-2-style transformer — small enough to fit on a floppy disk, honest about what it can and cannot do.

Renamed. This repo was originally published as Compactbot/gpt-s2.5-5m and renamed to compacttest-5m at the request of @Datdanboi25. Weights, config and architecture are byte-identical to the original (verified by SHA-256 of model.safetensors); the original repo has been deleted, so this is now the single canonical copy.

Numbers (verified against the artifact)

Parameters	5,114,112 (exact; 38 tensors, F32)
Vocab	8,192 (byte-level BPE)
Layers	4
Hidden	256
Heads	8 q / 2 kv (GQA 4:1), head_dim 32
FFN	SwiGLU, intermediate 768
Norm	RMSNorm (eps 1e-6), pre-norm
Position	RoPE (base 10000), max 512
Head	weight-tied (tok.weight reused as lm_head; no separate head tensor)
Biases	none
Checkpoint	20,459,896 bytes = 8 + header + 5,114,112 × 4 (exact)

The parameter count above is the stored element count in model.safetensors (parsed from the file header), not a card claim. Because the head is tied, stored == unique.

Evaluations (measured, 2026-09-22)

Zero-shot loglikelihood accuracy, Open SLM Leaderboard formula (the same one used for the published values — I reproduced all 10 of them to the cent before running this, so the method is reproducible): per-task accuracy, chance-normalized as (acc − chance) / (100 − chance), then weighted. 500 items per task.

Intelligence Index = 0.032

Task	Acc	Chance	Chance-norm
HellaSwag	28.2% (141/500)	25%	+0.043
ARC-Easy	26.8% (134/500)	25%	—
ARC-Challenge	21.6% (108/500)	25%	—
→ combined ARC (mean)	24.2%	25%	−0.011
PIQA	53.2% (266/500)	50%	+0.064

II = (1.00·0.043 + 1.00·(−0.011) + 1.00·0.064) / 3.00 = 0.032

Notes on the method:

• ArithMark-3 was not available (arithmetic/ArithMark-3 is not accessible on the Hub right now), so the 3-task formula (weights 1/1/1, sum 3.00) was used instead of the full 4-task one (0.65 weight, sum 3.65). If ArithMark-3 comes back I will re-run and update this section.
• Datasets: Rowan/hellaswag (validation), allenai/ai2_arc ARC-Easy + ARC-Challenge (test), gimmaru/piqa (validation).
• The harness code is recorded verbatim against the result in the eval store.

Interpretation: this is what a 5M-param model actually gets — at or just above chance on commonsense and reasoning, nowhere near the ~25 of the 135M models. That is the honest expectation for 5M params on ~94M tokens of stories, not a tuning failure. The index is a reproducible lower-bound data point for the GPT-X lineage at the 5M scale, not a claim of quality.

What it is and is not good at

• Greedy decoding (temp=0) is coherent. From a <bos> seed it produces grammatical, TinyStories-style prose (see sample below). This is the intended operating point for a 5M model.
• Aggressive sampling is degenerate. At temp 0.7 / top_p 0.5 it collapses into <eos>Ġwas<bos>!-style token loops — the classic small-model tail-noise failure. If you sample, use a low temperature (0.2–0.3) and a high top_p (0.9); even then it is fragile.
• It is a toy. 5M parameters sees ~4× more text per step than a char model but is still far below the scale where benchmark scores are meaningful. Perplexity on held-out TinyStories text is ~15.6 (val). The measured benchmark scores above are at/near chance, as expected. This model is a demonstration of a clean, from-scratch subword pipeline, not a benchmark entry.

Sample (greedy, temp=0, from <bos>)

Once upon a time, there was a little girl named Lily. She loved to play outside in the sun. One day, she saw a big, scary dog. The dog was scared and wanted to play with Lily. Lily went to the dog and said, "Hello, dog! Can...

How to run

This is not a transformers-native architecture. Load it with the included model.py:

import torch
from model import Model
from safetensors.torch import load_file

m = Model()
m.load_state_dict(load_file("model.safetensors"))
m.eval()

# greedy generation
with torch.no_grad():
    x = torch.tensor([[1]])            # 1 = <bos>
    for _ in range(60):
        logits = m(x[:, -512:])[:, -1, :]
        x = torch.cat([x, logits.argmax(-1, keepdim=True)], dim=1)

Tokenization: tokenizer.json is a standard BPE vocab (8192 tokens, byte-level). It is not a HuggingFace PreTrainedTokenizerFast file — load the model.vocab mapping directly (see model.py for the decode helper).

Training

• Data: ~94M tokens of TinyStories (byte-level BPE, 8192 vocab).
• Hardware: CPU-only (32 cores, no GPU). ~5000 steps, batch 32×512.
• Schedule: linear warmup (200) → cosine decay, peak 3e-4, min 3e-5.
• Checkpoint: best validation perplexity (val ppl ~15.6 on 2M held-out tokens).
• Reproducibility: seed 42. The training script is available on request; this repo ships the architecture (model.py) and weights only.

Honest caveats

• Trained on CPU; throughput is a hardware artifact, not a model property, so I am not reporting tokens/sec.
• The model is small on purpose. Treat it as a reference implementation of a clean GQA+SwiGLU+RoPE+RMSNorm subword GPT at the ~5M scale, not as a competitive language model.
• Sampling quality is the weak spot; use greedy for coherent output.