Supra2.5-Pico

2.98M parameters · pretrained from scratch · 12.96 on the Open SLM Leaderboard Intelligence Index

Supra2.5-Pico is the smallest model in the Supra2.5 line: a decoder-only language model with 2,982,480 parameters, trained from scratch on 12B tokens of English text with a custom 4,104-token tokenizer. It is a base model for text completion. It has not been instruction-tuned, chat-tuned or aligned in any way.

At release it scores 12.96 on the Open SLM Leaderboard metric. That is the highest score of any model under 30M parameters on the leaderboard as of 22 October 2026 - 0.80 points above Supra2-Medium-Instruct (12.16, 25.4M) and 3.35 points clear of GPT-S3-3M (9.61), the best sub-3M entry.

It is also the most parameter-efficient model in its class that we measured: 4.35 Intelligence Index points per million parameters, against 3.23 for GPT-S3-3M and 1.78 for the nearest 10M-class entry listed below.

---

Performance

Zero-shot, acc_norm, full test sets, measured with lm-evaluation-harness and the leaderboard's own ArithMark-3 script on the float32 weights in this repository.

Benchmark	Score
HellaSwag	27.66
ARC-Easy	31.65
ARC-Challenge	23.55
PIQA	56.58
ArithMark-3	56.30
Intelligence Index	12.96

The Index is (N(HS,25) + N(ARCavg,25) + N(PIQA,50) + 0.65·N(AM3,25)) / 3.65, where N(v,c) = 100·(v−c)/(100−c) and ARCavg is the mean of ARC-Easy and ARC-Challenge.

Against the sub-3M class

Model	Params	Index	HellaSwag	ARC-Easy	ARC-Challenge	PIQA	ArithMark-3
Supra2.5-Pico	2.98M	12.96	27.66	31.65	23.55	56.58	56.30
GPT-S3-3M	2.97M	9.61	27.68	34.76	23.12	57.02	39.10
Tokle-3M	2.91M	8.91	27.20	34.85	23.98	55.01	40.80
pulvis-v2	2.96M	8.62	27.93	31.86	22.78	56.86	37.40
Purrence-3M	2.99M	8.49	27.81	34.55	23.63	56.64	34.80
ZeroS-Micro-v1.0	2.87M	6.49	28.46	31.27	22.01	53.86	35.60

Figures for the other models are from the Open SLM Leaderboard or the respective model cards. They are shown for context only and were not re-measured here.

Against the sub-30M board

Best entries under 30M parameters, recomputed from the leaderboard's published figures on 22 October 2026:

Model	Params	Index
Supra2.5-Pico	2.98M	12.96
Supra2-Medium-Instruct	25.38M	12.16
ForgePlex-M2.2-10M	9.99M	12.12
Supra2-Medium-Base	25.37M	12.09
BananaMind-2-Mini	25.18M	12.01

Efficiency

Model	Params	Index	Index per million parameters
Supra2.5-Pico	2.98M	12.96	4.35
GPT-S3-3M	2.97M	9.61	3.23
Tokle-3M	2.91M	8.91	3.06
TinyBrainBot-10M-Base	9.90M	17.61	1.78
void.1	90.2M	23.92	0.27

---

Model details

Developed by	SupraLabs
Model type	Causal decoder-only transformer (custom SLMForCausalLM)
Language	English
Parameters	2,982,480
Context length	1,024 (trained at that length)
Precision	float32 weights, bf16 training
License	Apache 2.0

Architecture

Hyperparameter	Value
Hidden size	144
Layers	10
Attention	GQA, 3 query heads / 1 KV head, head_dim 48
QK normalization	RMSNorm per head
Positional encoding	partial RoPE on 24 of 48 dimensions, θ = 10,000
Attention extras	XSA + per-head sigmoid output gate
Residual routing	softmax attention residuals over all previous block outputs
Intermediate size (SwiGLU)	376
Normalization	RMSNorm, ε = 1e-6, fp32 upcast
Tied embeddings	Yes
Vocab size	4,104
Attention implementation	SDPA

Tokenizer

A 4,104-token byte-level BPE: 4,096 merges plus eight length tokens <|len1|>…<|len8|>. Numbers are written with their length token and their digits in units-first order, on both the encode and the decode side; 62 becomes <|len2|> 2 6. Loading the tokenizer with trust_remote_code=True reproduces this behaviour automatically.

---

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SupraLabs/Supra2.5-Pico"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.float32
).eval()

prompt = "The water cycle begins when"
ids = tok(prompt, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**ids, max_new_tokens=64, do_sample=False,
                         repetition_penalty=1.1, no_repeat_ngram_size=4)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required: SLMForCausalLM and the tokenizer class are defined in modeling_slm.py and tokenization_pascaline.py inside this repository.

Evaluate in float32. bf16 rounding moves logits enough to flip borderline multiple-choice answers.

---

Reproducing the evaluation

Everything needed is in eval/:

eval/reproduce_evaluation.py   runs the harness, ArithMark, and recomputes the index
eval/bencharithmark-3.py       official ArithMark-3 scorer (1,000 items, acc_norm)
eval/arithmark-3.jsonl         the 1,000 ArithMark-3 items

pip install -U lm-eval accelerate torch transformers datasets
cd eval
python reproduce_evaluation.py --model . --output results

Or by hand, which is what the numbers above come from:

python -m lm_eval --model hf \
  --model_args pretrained=SupraLabs/Supra2.5-Pico,trust_remote_code=True,dtype=float32 \
  --tasks hellaswag,arc_easy,arc_challenge,piqa \
  --num_fewshot 0 --batch_size 8 --device cuda

python bencharithmark-3.py --model . --data-path arithmark-3.jsonl \
  --dtype float32 --device cuda --batch-size 8

batch_size 8 is deliberate: at 1,024 context, larger batches exhaust a 24GB card when the evaluation runs alongside anything else.

---

Intended use

Intended: research on small-scale pretraining, scaling behaviour and tokenizer design; a cheap starting point for fine-tuning; education; fast iteration on a single GPU.

Not intended: production or user-facing deployment, factual question answering, decision support, non-English text. At three million parameters the model confabulates freely, repeats itself on long generations, and its arithmetic is limited to short grade-school items.

Limitations

• English only, 1,024-token context, no instruction tuning, no safety tuning, no RLHF.
• Weak on general language modelling: it loses roughly 3 points of LAMBADA accuracy and 0.09 bits/byte on WikiText-2 to GPT-S3-3M, which saw more than twice as many tokens.
• Arithmetic ability is narrow. The ArithMark-3 score reflects the benchmark's task shape at this model size and should not be read as general mathematical reasoning.
• No chat template is provided. Applying one will not produce sensible output.

Citation

@misc{supra25pico,
  title  = {Supra2.5-Pico},
  author = {SupraLabs},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/SupraLabs/Supra2.5-Pico}}
}

License

Apache 2.0. See LICENSE.