SupraNeo-4M

A 4.07M-parameter decoder-only language model built to test a single question: how much of a tiny model should actually compute anything?

Most models at this scale spend the majority of their parameters on a vocabulary lookup table. SupraNeo-4M spends 82% on the transformer stack. The design starts from a custom 4,096-token BPE tokenizer, which drops the embedding matrix to 524k parameters and frees the rest for a deep, narrow stack. The aspect ratio (d/L ≈ 13) is deliberately below what the 100M class uses — following the MobileLLM finding that optimal depth-to-width shifts toward depth as models shrink.

Architecture

Parameters	4,070,240 (3.41M non-embedding, 83.9%)
Architecture	Qwen3 (Qwen3ForCausalLM)
Hidden size	160
Layers	12
Attention heads	4 (head_dim 40)
KV heads	2 (GQA 2:1)
MLP intermediate	432 (SwiGLU)
Vocabulary	4,096 (custom BPE, ~3.2 chars/token)
Context length	1,024
Normalization	RMSNorm + QK-Norm
Embeddings	tied
Precision	float32

No custom modeling code — trust_remote_code is not required.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DedeProGames/SupraNeo-4M")
model = AutoModelForCausalLM.from_pretrained("DedeProGames/SupraNeo-4M")

ids = tok("The main reason that", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=40, temperature=0.8, top_p=0.9, do_sample=True)
print(tok.decode(out[0]))

This is a base model with no chat template. apply_chat_template will fail by design.

Training

Pre-trained from scratch on a single NVIDIA L4 over 5B tokens (~1,230 tokens per parameter).

Data	84% HuggingFaceFW/fineweb-edu filtered to int_score ≥ 4, 16% mlfoundations/dclm-baseline-1.0
Schedule	WSD, peak LR 4e-3, 1,500-step warmup, 1-sqrt decay over the final 20%
Anneal	last 20% on the FineWeb-Edu subset only, context extended 512 → 1,024
Batch	65,536 tokens/step
Optimizer	AdamW (β 0.9/0.95, wd 0.1 on 2D params, grad clip 1.0), z-loss 1e-4
Init	residual branches scaled by 1/√(2L)

The heavy FineWeb-Edu weighting is deliberate. DCLM-baseline was tuned to win reasoning benchmarks in the 1–7B range, and none of that transfers at 4M — what a model this size can learn is register and local fluency, and FineWeb-Edu's uniform expository prose is far easier to model.

Evaluation — BananaMind Base Bench 1.1

350 items, 4-way continuation-likelihood, add_special_tokens=False, no BOS, selection by highest conditional mean log-prob, Elo by weighted MLE with a prior of 4 games at 1000. Official run: dataset checksum verified, schema verified, 0 truncated contexts, 0 truncated continuations.

Category	Accuracy	z vs chance	Elo	Sig.
Language Completion	56.0%	+5.06	963	✅
Logical Reasoning	38.0%	+2.12	978	✅
World Knowledge	34.0%	+1.47	820
Context Tracking	34.0%	+1.47	856
Commonsense	26.0%	+0.16	770
Quantitative	24.0%	−0.16	834
Code Completion	22.0%	−0.49	883

Overall Elo	868
Chance floor	805
Above floor	+63
Raw accuracy	33.4% (95% CI 28.5–38.4%)
z vs chance	+3.64 — significant

By difficulty: easy 39.3%, medium 29.1%, hard 31.9%.

Reading these numbers honestly

The aggregate is significantly above chance, but the signal is concentrated in one place. Language Completion at 56% (+5.06σ) is the only strongly separated category, and that is exactly what a 4M model should be able to do: local grammatical and register plausibility. Logical Reasoning clears the bar marginally. The remaining five categories sit within noise, and Quantitative and Code Completion land at or slightly below chance — this model has no arithmetic or code capability, and the card should not be read as claiming otherwise.

The medium/hard inversion (29.1% vs 31.9%) is noise at n≈117, not evidence that harder items are easier.

BananaMind Base Bench was calibrated for the 65M–100M+ range. At 4M, with 350 four-way items, the detection floor at 1.96σ is roughly 29.5% accuracy — most of this benchmark simply lacks resolution here. For tracking progress at this scale, bits-per-byte on held-out text and BLiMP are the metrics with actual sensitivity. HellaSwag, PIQA and ARC are not reported because they sit at chance and measure nothing.

Limitations

SupraNeo-4M produces grammatical, register-consistent English with coherence over one to two sentences and topical drift beyond that. It has no factual reliability, no arithmetic, no code ability, and no instruction following. The 4,096-token vocabulary means it compresses text ~35% less efficiently than a standard 32k tokenizer, and its outputs are not comparable to other models by raw cross-entropy — use bits-per-byte.

This is a research artifact for studying small-scale pretraining, vocabulary budgets, and data mixtures. It is not intended for deployment.

Carbon footprint

Training was estimated to emit 0.46 kg CO₂ eq. — a single L4 for the duration of the run. For reference, that is roughly the footprint of driving a passenger car about 2.5 kilometers.

---

by DedeProGames