BLU-135M

BLU is a 135M-parameter base language models trained from scratch. One 750B-token pre-training run was finished with two alternative 150B-token learning-rate decays, and the second decay was then extended to a 32k-token context. We release four checkpoints, each in its own folder of this repository:

Checkpoint	What it is	Context	Training tokens	9-task average
BLU-135M-Decay1	Pre-training with the first 150B-token learning-rate decay	4k	0.90T	34.39
BLU-135M-Decay2	Pre-training with the second, re-mixed 150B-token decay	4k	0.90T	34.30
BLU-135M-32k	Decay 2 extended to a 32k context	32k	0.91T	33.82
BLU-135M-Souped	Weight average of 32k (40%), Decay 2 (30%) and Decay 1 (30%), run with YaRN ×2	32k	1.06T¹	34.47

¹ The souped model averages checkpoints from both decays, so it includes 750B shared tokens, two 150B decays and the 12B long-context stage.

The 9-task average is the SmolLM2 model-card benchmark suite in fp32 (details under Evaluation). On it, Decay 1, Decay 2 and the souped model beat SmolLM2-135M (34.2), which was trained on 2T tokens: 2.2× more than Decay 1 and Decay 2, and 1.9× more than the souped model.

These are base models: they are not instruction-tuned or aligned, and they continue text rather than follow instructions.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo, ckpt = "BluBridgeAI/BLU-135M", "BLU-135M-Souped"   # or BLU-135M-Decay1, BLU-135M-Decay2, BLU-135M-32k
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=ckpt)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=ckpt, torch_dtype=torch.bfloat16)

inputs = tokenizer("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

• Requires transformers ≥ 4.51 (tested with 4.57.6 and 5.20).
• The tokenizer adds a start-of-text token (<|endoftext|>, id 0) to every prompt, as in training. Keep it: the models were trained with it.
• BLU-135M-Souped ships its RoPE scaling (YaRN ×2 over 32,768 tokens) in config.json. It is part of the evaluated model, and loading the checkpoint as shipped applies it. The configuration accepts up to 65,536 positions, but the model was evaluated only up to 32,768 tokens.
• Decay 1 and Decay 2 are stored in fp32, the 32k and souped checkpoints in bf16. All four run in bf16 or fp32.

Model architecture

A deep-narrow, decoder-only dense transformer with 134,396,672 parameters (109.17M outside the embedding).

Component	Setting
Layers × width	37 × 512
Attention	Grouped-query: 4 query heads, 2 key/value heads, head dimension 128, causal
QK-norm	RMSNorm on queries and keys per head, applied before RoPE
Position encoding	RoPE (NeoX half-split layout): θ = 50,000 for Decay 1 and Decay 2, θ = 800,000 for 32k, θ = 800,000 with YaRN ×2 over 32,768 tokens for Souped
Feed-forward	SwiGLU, inner size 1,408
Normalisation	Pre-norm RMSNorm (eps 1e-5) plus a final RMSNorm
Biases	None
Embedding	49,280 × 512, tied with the output head; the input embedding is scaled by √512
Context	4,096 tokens (Decay 1, Decay 2); 32,768 tokens (32k, Souped)

Part	Parameters
Token embedding (tied)	25,231,360
Attention per layer (Q 512×512, K and V 512×256, O 512×512)	786,432
SwiGLU per layer (gate + up 512×2,816, down 1,408×512)	2,162,688
Norm scales per layer	1,280
37 layers	109,164,800
Final norm	512
Total	134,396,672

Design notes.

• Depth over width. SmolLM2-135M is 30 × 576; BLU is deeper and narrower. Its KV cache is 37 KB per token (37 layers × 2 KV heads × 128 × keys and values × bf16), against 23 KB for SmolLM2-135M, and its decode-heavy throughput in vLLM is 0.63× SmolLM2-135M's.
• √d input scale. The scale applies to the embedding output only; the tied output head reads the unscaled matrix. The released checkpoints store the scaled embedding, tie the output head to it and fold 1/√512 into the final norm, so they load as a standard tied model.

Tokenizer

Property	Value
Type	Byte-level BPE, 48,879 merges, no normaliser, no byte fallback
Pre-tokenisation	Digits split individually, then GPT-2-style byte-level regex
Vocabulary	49,152 tokens, including 17 special tokens
Embedding rows	49,280 (padded to a multiple of 128)
Document boundary	Every training document starts with <|endoftext|> (id 0)

Evaluation

Base-model benchmarks

Zero-shot except GSM8K (5-shot); fp32 inference; the same harness, prompts and items for every model, except as noted below the table. ARC is the mean of ARC-Easy and ARC-Challenge. MMLU-Pro is not part of the 9-task average. The best score in each column is in bold.

Model	Params	Tokens	HellaSwag	ARC	PIQA	MMLU	CSQA	TriviaQA	Winogrande	OBQA	GSM8K	Avg (9)	MMLU-Pro
BLU Decay 1 (900B)	134.4M	0.90T	42.3	44.2	68.1	31.0	32.1	4.4	51.1	34.4	1.7	34.4	11.5
BLU Decay 2 (900B)	134.4M	0.90T	42.1	43.5	68.6	30.6	31.5	5.3	52.2	33.6	1.2	34.3	11.7
BLU 32k (long context)	134.4M	0.91T	41.7	43.5	68.1	30.0	31.4	2.8	51.8	33.6	1.4	33.8	11.4
BLU Souped	134.4M	1.06T¹	42.3	44.4	68.5	30.4	32.5	4.7	52.0	33.2	2.1	34.5	11.1
SmolLM2-135M	134.5M	2T	42.1	44.1	68.5	31.1	33.8	4.2	51.2	34.4	1.4	34.5	11.2
SmolLM2-135M, before long context	134.5M	2T	41.8	43.9	66.5	30.8	33.7	4.4	51.6	33.6	1.1	34.2	11.3
Pythia-160M	162.3M	0.3T	29.5	31.6	62.0	26.8	28.9	0.4	50.1	28.2	0.2	28.6	11.7
OPT-125M	125.2M	0.18T	31.0	31.2	62.0	26.0	32.6	1.9	50.4	29.0	0.2	29.4	11.5
MobileLLM-R1-140M-base	140.2M	4.2T	33.8	37.1	62.9	28.5	30.5	4.2	51.5	29.6	20.1	33.1	10.8
Gemma-3-270M	268.1M	6T	40.5	42.5	68.8	29.9	34.6	8.8	52.2	31.8	1.7	34.5	11.0
LFM2.5-230M-Base	229.7M	28T	38.1	47.3	63.6	32.8	30.3	11.0	52.5	33.4	20.0	36.6	20.4

Gemma 3 and LFM2.5 are scored on TriviaQA and GSM8K one prompt at a time, without padding: their batched generation under transformers 4.57 changes their answers and lowers their scores.

Long context: RULER at 8k, 16k and 32k

Mean over all 13 RULER synthetic tasks, 500 samples each. A dash (-) means the length is beyond the context the model was trained for, so it is not run.

Model	Trained context	8k	16k	32k
BLU Decay 1 (900B)	4k	-	-	-
BLU Decay 2 (900B)	4k	-	-	-
BLU 32k (long context)	32k	31.3	32.8	19.5
BLU Souped	32k	36.2	20.4	17.3
SmolLM2-135M	8k	30.2	-	-
SmolLM2-135M, before long context	2k	-	-	-
Pythia-160M	2k	-	-	-
OPT-125M	2k	-	-	-
MobileLLM-R1-140M²	32k	15.7	5.7	0.1
Gemma-3-270M	32k	33.5	30.0	27.4
LFM2.5-230M-Base	32k	38.8	26.8	10.2

² MobileLLM-R1-140M is a reasoning fine-tune, not a base model. Its pre-training ran at 2k–4k tokens; only its final reasoning fine-tuning stage ran at 32k. It is run with the same plain-text prompt as every other model.

How we evaluated

Tools

	Base-model benchmarks	RULER
Harness	lighteval 0.13.0 with the task definitions used for the SmolLM2 model card, Hugging Face transformers backend	NVIDIA RULER synthetic suite, served with vLLM 0.32
Libraries	transformers 4.57.6, torch 2.7.1 (CUDA 12.8)	transformers 5.19
Precision	fp32	bf16
Decoding	Log-likelihood scoring; greedy generation for TriviaQA and GSM8K	Greedy
Prompt	Plain text, no chat template	RULER's base template (no chat template)
Hardware	8× NVIDIA H200, one model per GPU	8× NVIDIA H200, one vLLM server per GPU

Benchmarks

• Tasks and scoring. HellaSwag, ARC-Easy, ARC-Challenge, PIQA, MMLU, CommonsenseQA, Winogrande and OpenBookQA are 0-shot cloze tasks, scored by log-likelihood normalised by answer length in characters. TriviaQA (0-shot, up to 20 generated tokens) and GSM8K (5-shot, up to 256 tokens) use greedy generation scored by quasi-exact match. MMLU-Pro is 0-shot lettered multiple choice with 10 options (A–J); chance is 10%. Every model is scored on the full test or validation split: 63,753 items.
• Harness changes. lighteval 0.13.0 with four fixes: log-softmax computed in fp32, a sequence-slicing fix, stripping of padding tokens, and a fix to a bug that left-truncated some GSM8K few-shot prompts.

RULER

• Tasks. All 13 tasks: 8 needle-in-a-haystack variants, variable tracking, common and frequent word extraction, single-hop QA (qa_1, SQuAD) and multi-hop QA (qa_2, HotpotQA). 500 samples per task and length.
• Trained context only. Each model is run only at lengths within the context it was trained for, with its released weights and RoPE configuration. The souped model uses its shipped configuration, YaRN ×2 over 32k.

Limitations

• BLU models are small base models. They make factual errors, can produce repetitive or incoherent text, and have not been aligned or safety-tuned.

License

Apache-2.0.

Citation

@misc{blu135m,
  title  = {BLU-135M},
  author = {{BluBridge}},
  year   = {2026},
  url    = {https://huggingface.co/BluBridgeAI/BLU-135M}
}