SupraLabs/Supra2.6-Pico

Supra2.6 Pico

2,972,496 parameters · 1,024 context · 4,096-token vocabulary · hybrid GDN-2 / gated-GQA

Apology for the accidental benchmaxing issues with 2.5 Pico.

---

Specifications

Parameters (unique)	2,972,496
Parameters (as stored)	3,562,320 tensors — the 589,824-parameter embedding is tied to the LM head and counted twice on disk
Non-embedding parameters	2,382,672
Layers	8
Layer pattern	[GDN-2, GDN-2, GDN-2, gated GQA] × 2
d_model	144
FFN hidden	352 (SwiGLU)
GDN-2 heads	3 per layer, conv kernel 4, channel-wise erase/write + per-channel log-decay
GQA	3 query heads / 1 KV head (head_dim 48), RoPE on 24 dims, q/k RMSNorm, sigmoid output gate
Attention residuals	learned softmax over all preceding layer outputs, one query per layer
Context length	1,024 tokens
Vocabulary	4,096 (BPE, trained on the training mixture)
Embeddings	tied input/output
Norm	pre-RMSNorm
Dtype	fp32 weights, trained in bf16 autocast
Weight file	model.pt — 11.9 MB

Architecture

Eight blocks in two identical groups of four. Each group is three gated-delta-net (GDN-2) blocks followed by one gated grouped-query attention block, so 6 of 8 layers are linear-time recurrence and 2 carry global attention. GDN-2 uses the channel-wise erase/write recurrence

S_t = (I − k_t (b_t ⊙ k_t)ᵀ) Diag(exp(g_t)) S_{t−1} + k_t (w_t ⊙ v_t)ᵀ

implemented with the fla chunked kernel. Attention-residual mixing lets every layer read a learned softmax over the outputs of all previous layers.

Training recipe

Tokens trained	9,998,172,160
Tokens per step	262,144 (batch 256 × 1,024)
Steps	38,140
Optimizer	Muon (2-D matrices, lr 0.02, momentum 0.95, 5 Newton–Schulz steps) + AdamW (embeddings/norms, lr 6e-3, β 0.9/0.95)
LR schedule	linear warmup 10,000 steps, peak lr 0.02 held thereafter
Gradient clipping	1.0
Epochs per source	≤ 1 (each source packed once, sampled proportionally)
Precision	bf16 autocast, fp32 master weights
Seed	0
Hardware	1× RTX Pro 6000 Blackwell, 20 vCPU
Stack	PyTorch 2.x, fla-core 0.6.0 (chunked GDN-2 kernel), tokenizers

Mixture (sampling shares; tokens sampled over the 9.998B run):

source	share	tokens
ClimbMix	36.3%	3.63B
Cosmopedia v2	19.5%	1.95B
Ultra-FineWeb L2 HQ	19.5%	1.95B
TXT360-v2 web-high-medium	13.6%	1.36B
open-web-math	5.1%	0.51B
OpenMathInstruct-2	4.8%	0.48B
NuminaMath-CoT	0.53%	0.053B
TinyStories	0.48%	0.048B
OpenStax	0.10%	0.010B
orca-math-word-problems	0.10%	0.010B
GSM8K	0.004%	0.0004B

Benchmarks

Full evaluation sets, log-likelihood scoring with length normalisation (acc_norm):

benchmark	score	items
HellaSwag	26.40	10,042
ARC-Easy	33.54	2,376
ARC-Challenge	23.21	1,172
PIQA	56.09	1,838
Arithmark-3	38.10	1,000
Index	8.192	—

Index = ( N(HellaSwag, 25) + N(ARC mean, 25) + N(PIQA, 50) + 0.65 · N(Arithmark-3, 25) ) / 3.65, with N(v, c) = 100 · (v − c) / (100 − c).

Raw acc for the same checkpoint: HellaSwag 26.49 · ARC-Easy 33.50 · ARC-Challenge 17.92 · PIQA 56.91 · Arithmark-3 38.10.

Usage

model_h.py uses the Triton fla kernel; model_h_local.py (+ gdn2_ref.py) is the same model in pure PyTorch for CPU-only machines.

import torch
from tokenizers import Tokenizer
import model_h            # or: import model_h_local as model_h

net = model_h.ModelH(vocab=4096, d=144, inter=352, n_gdn=6, n_gqa=2, ctx=1024)
net.load_state_dict(torch.load("model.pt", map_location="cpu", weights_only=False)["model"])
net.eval()
tok = Tokenizer.from_file("tokenizer.json")

ids = torch.tensor([tok.encode("The capital of France is").ids])
logits = net(ids)[0]

model.safetensors stores the tied LM head once (155 tensors, 2,972,496 parameters); load it with strict=False after building the model, or use model.pt which carries both tied entries.

At this scale, it is not usable as a general assistant, a chatbot, or for any production task.