SupraLabs/Supra2-Medium-Base

Supra2-Medium Base

Ultra-efficient base model • 25M Parameters • 1K Context

Supra2-Medium Base is a 25M-parameter decoder-only language model pretrained from scratch by SupraLabs on 20B tokens of English web text. It uses the Qwen3 architecture with a custom 16,384-token tokenizer.

This is a base model. It has not been instruction-tuned, chat-tuned, or aligned in any way.

At only 25 million parameters, Supra2-Medium demonstrates that meaningful language modeling can be achieved with extreme parameter efficiency—trained at ~800 tokens per parameter, which is significantly higher than typical pretraining ratios. This makes it ideal for research into data-efficient scaling and ultra-lightweight deployments.

---

Let the model speak - a sample

Prompt: "Artificial intelligence (AI) is " Completion:

 Artificial intelligence (AI) is espoused by the AI community.
The AI community is a group of people who are interested in AI and are interested in the use of AI in the field of AI.
The goal of AI is to improve the quality of life of people in the field.
The aim of AI is the development of AI and the application of AI in a society.
The purpose of AI is that it can be used to improve the performance of the society.
It is a technology that is used to improve human intelligence.
The technology is used to make the human intelligence.

Evaluation & Benchmarks

All benchmarks were evaluated using the EleutherAI LM-Eval Harness.

Model	PIQA (acc_norm)	HellaSwag (acc_norm)	ARC-Easy (acc_norm)	ARC-Challenge (acc_norm)
Supra-50M-Base (50M)	0.62	0.32	0.46	0.25
Supra2-Medium-Base (25M)	59.14	29.29	41.84	23.72
Supra2-100M-Base (100M)	0.65	0.36	0.48	0.25

Final Train Loss: 3.2469 (no eval loss available)

Note: The model shows strong performance relative to its size, particularly given the high token-per-parameter ratio. It even strongly competes with our previous 50M model!

---

Model Details

Developed by	SupraLabs
Model type	Causal decoder-only transformer (Qwen3)
Language	English
Parameters	25.37M total / ~20M non-embedding
Training tokens	20B (~800 tokens per parameter)
Context length	1,024
Precision	bfloat16
License	Apache 2.0

Architecture

Hyperparameter	Value
Hidden size	512
Layers	7
Attention heads	8 (MHA)
Head dim	64
Intermediate size (SwiGLU)	896
Vocab size	16,384
Positional encoding	RoPE θ=10,000
Normalization	RMSNorm, ε=10⁻⁶
Tied embeddings	Yes
Attention implementation	SDPA
Architecture ID	qwen3-d07-h512-i0896

---

Training Data

Source	Share	Approx. tokens
HuggingFaceFW/fineweb-edu (sample-100BT)	100%	20B

Documents were tokenized with the custom tokenizer_16k, concatenated into a flat uint16 token stream, and packed into contiguous 1,024-token chunks (no padding, no document masking — sequences may cross document boundaries).

---

Training Procedure

Setting	Value
Optimizer	AdamW (fused), β₁=0.9, β₂=0.95, ε=10⁻⁸
Peak learning rate	3×10⁻³
LR schedule	Cosine with min LR (10% of peak)
Warmup steps	1,000
Micro batch size	32
Gradient accumulation	4
Effective batch	256 sequences = 262,144 tokens/step
Weight decay	0.1
Gradient clipping	1.0
Mixed precision	BF16 + TF32
Hardware	2× GPU (DDP): RTX 5060 Ti 16GB + RTX 5060 8GB
Training optimizations	Liger Kernel, DDP bucketing (10MB), gradient_as_bucket_view

Key Design Decisions

• High token-per-parameter ratio: At 800 tokens/param, this model pushes the boundaries of data efficiency for small models
• Compact vocabulary: 16K vocab reduces embedding overhead while maintaining coverage
• Tied embeddings: Reduces parameter count and improves training stability
• Multi-head attention (MHA): Unlike larger Supra2 models using GQA, this architecture uses standard MHA for simplicity at this scale
• No sliding window: Full attention within the 1K context

---

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SupraLabs/Supra2-Medium"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "The future of artificial intelligence"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.2,
        top_p=0.85,
        top_k=25,
        no_repeat_ngram_size=3,
        repetition_penalty=1.2,
    )

print(tokenizer.decode(out[0], skip_special_tokens=True))

Tokenizer notes

The model uses a custom 16,384-token vocabulary optimized for English web text. The tokenizer is shared across the Supra2 family's smaller models for consistency and efficient multi-task fine-tuning.

---

Intended Use

Intended:

• Research on extreme parameter efficiency and data-efficient pretraining
• Ultra-lightweight edge deployments where memory is severely constrained
• Educational use for understanding transformer architectures at minimal scale
• Starting point for domain-specific fine-tuning when compute resources are limited
• Ablation studies on small model behavior and scaling laws

Not intended:

• Production deployment for critical applications
• Factual question answering or knowledge-intensive tasks
• Long-form coherent generation beyond a few paragraphs
• Non-English text (trained exclusively on English)
• Any application requiring safety guarantees or alignment

---

Limitations and Bias

• Very small. At 25M parameters, the model has extremely limited capacity. Expect frequent hallucinations, factual errors, repetitive outputs, and poor reasoning.
• Base model. No RLHF, no safety tuning, no refusal behavior. It will continue whatever text you give it, including harmful or offensive prompts.
• Web-derived data. FineWeb-Edu is a filtered CommonCrawl derivative and carries the biases, stereotypes, and factual errors of the open web.
• Short context. Trained exclusively at 1,024 tokens. Extrapolation beyond this length is untested and likely degraded.
• No document masking. Attention could cross document boundaries within a packed chunk, which slightly blurs document independence.
• English only. Performance on other languages will be poor to non-existent.
• High token-per-parameter ratio. While efficient, training at 800 tok/param means the model may be under-trained compared to models trained at lower ratios (e.g., 300 tok/param).

---

Performance Characteristics

Despite its tiny size, Supra2-Medium achieves surprisingly coherent short-form generation. Key observations:

• Strengths: Basic grammar, simple factoids, short completions, pattern matching
• Weaknesses: Complex reasoning, arithmetic, long-range coherence, factual accuracy, nuanced understanding
• Best use case: Generating 1-3 sentence completions, simple text transformations, educational demonstrations

---

Future work includes instruction-tuned variants and exploration of even more efficient architectures at this scale.

© SupraLabs 2026