Petal-2-50M

Small language model (53.4M parameters), Qwen3.5 hybrid (Gated DeltaNet + Gated Attention) architecture, trained from scratch.

Architecture

Property	Value
Layers	14
Hidden size	512
Intermediate size	1408
Attention heads	8 (GQA kv=4, head_dim=64)
Full / Linear (DeltaNet) layers	14/0
DeltaNet conv kernel	4
DeltaNet kv heads	8/16
Partial rotary factor	0.25
Max sequence length	1024
Vocab size	16384
Tied embeddings	True
Total parameters	53.363M

Training

• Tokens seen: 576,153,600
• Val loss: 2.8450
• Val PPL: 17.20

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("CyanMonkey/Petal-2-50M")
model = AutoModelForCausalLM.from_pretrained("CyanMonkey/Petal-2-50M")
inputs = tokenizer("Hello", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))