Blaze (48.3M)

Blaze is a 48.3M parameter causal language model developed by SurjoLabs. It scores 15.45 on the Intelligence Index, placing #1 in the sub-50M parameter category on the Open SLM Leaderboard.

The model uses XSA (orthogonal value-subtraction) attention with recurrent layer sharing, achieving an effective computational depth of 26 layers while storing only 14 physical layers.

---

Architecture Specifications

Parameter	Value
Total Parameters	48,251,136
Physical Layers	14 (1 prelude + 12 recurrent + 1 coda)
Recurrent Passes	2 (effective depth: 26 layers)
Hidden Size	512
Intermediate Size	1536
Attention Heads	8 Query, 4 Key-Value (2:1 GQA)
Head Dimension	64
Vocabulary Size	8,192 (tied embeddings)
Context Length	1,024 tokens

---

Training & Checkpoint Selection

• Total Tokens: ~20.97B tokens (20,000 steps at 2^20 = 1,048,576 tokens/step)
• Schedule: WSD (Warmup-Stable-Decay) learning rate scheduler
• Selected Checkpoint: Checkpoint 19,500 achieved peak performance across benchmarks and is the official set of weights released in this repository.

---

Benchmark Results

Evaluated 0-shot using normalized accuracy (acc_norm):

Benchmark	Score
PIQA	62.51%
ARC-Easy	41.84%
ArithMark-3.0	37.80%
HellaSwag	31.84%
ARC-Challenge	24.91%
Intelligence Index	15.45

---

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SurjoLabs/Blaze"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).cuda()

prompt = "The speed of light is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=32)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

License

MIT