d0rj/t5-moe-55M-base

T5 MoE 55M Base (UL2)

A 54,858,240-parameter English base model in the Tiny llm ablation experiment. Trained from scratch on exactly 3,932,160,000 source tokens over 15,000 optimizer steps. The token count measures processed input blocks, not unique text or supervised target tokens.

Architecture and references

6 encoder + 6 decoder layers, width 512; encoder 8-head attention, decoder 8 query / 2 KV heads; 8 experts per MoE layer, top-2 routing, expert width 160; RoPE, RMSNorm, FP32 residuals and tied shared embeddings. Maximum encoder length 2050 including controls; raw training blocks 2048.

Architecture inspiration: yandex/AliceAI-T5-35B-A0.6B. Tokenizer foundation: q-project/Q-50M-Base, preserving all 32,768 original IDs and adding 3 mode tokens + 512 sentinels (33,283 entries). Weights were initialized randomly. This small adaptation does not reproduce Alice’s corpus, optimizer or routing recipe.

Training

• Data: FineWeb-Edu, sample-10BT, streamed from local Parquet shards; shuffle buffer 100,000.
• Objective: UL2 with seven equally likely denoisers: R(15%, mean span 3/8), S(suffix), X(50%,3), X(50%,8), X(15%,64), X(50%,64). S masks a uniformly sampled suffix of length 1..L/2. Targets contain corrupted spans and control tokens. Training adds router auxiliary loss with coefficient 0.01. These sampler choices are explicit local choices; see UL2.
• Batch: 16 sequences × 8 accumulation × 2048 tokens = 262,144 source tokens per step.
• Fused AdamW; peak LR 0.001, betas (0.9, 0.95), weight decay 0.1 (no decay for bias/norm/1D parameters), gradient clipping 1.0. Linear warmup for 150 steps, then cosine decay to 10% of peak LR.
• BF16 compute on one RTX 5070 Ti (16 GB), seed 2026; checkpoints retain FP32 weights. Exact training configuration.

Evaluation

Full official task splits, zero-shot, lm-eval 0.4.12, no chat template, BF16 on RTX 5070 Ti, maximum context 2048. Accuracy is a percentage. ± is one standard error; the separate bracketed column is the 95% Wilson confidence interval. Intervals describe finite evaluation-sample uncertainty, not variation across training seeds; no multiple-comparison correction is applied.

Dataset	Split	Examples	Metric	Score ± SE (%)	95% CI (%)
HellaSwag	validation	10,042	acc_norm	27.79 ± 0.45	[26.93, 28.68]
ARC-Easy	test	2,376	acc_norm	38.93 ± 1.00	[36.99, 40.91]
ARC-Challenge	test	1,172	acc_norm	23.81 ± 1.24	[21.46, 26.33]
PIQA	validation	1,838	acc_norm	55.60 ± 1.16	[53.32, 57.86]
WinoGrande	validation	1,267	acc	48.46 ± 1.40	[45.72, 51.21]
OpenBookQA	test	500	acc_norm	26.20 ± 1.97	[22.54, 30.23]
BoolQ	validation	3,270	acc	49.94 ± 0.87	[48.23, 51.65]
LAMBADA OpenAI	test	5,153	acc	14.83 ± 0.50	[13.88, 15.82]

T5 uses UL2 S-mode: encoder S + prefix + sentinel + EOS; decoder BOS + sentinel + shifted answer. Only answer text is scored; control tokens and router loss are excluded, with the full vocabulary retained in the softmax. Its encoder sees at most 2047 text-prefix tokens after reserving controls. LAMBADA accuracy requires the complete final-word token sequence. acc_norm is harness length-normalized option scoring; raw accuracy is also stored in results.json.

WikiText-2 raw test, conditional continuation: CPU FP32 re-evaluation on 291 nonoverlapping blocks (512 prefix + 512 scored suffix tokens), 148,992 scored tokens; 335 tail tokens excluded. NLL 3.920357, 95% CI [3.879299, 3.960742]; token PPL 50.418, 95% CI [48.390, 52.496]. Percentile block bootstrap, 10,000 resamples, seed 2026; exponentiate NLL endpoints for PPL. Blocks are the resampling unit; this does not model all within-document dependence. This is not standard rolling AR or word PPL. The earlier BF16 point is retained separately in TensorBoard, with no borrowed FP32 interval.

The metadata contains author-reported model-index scores. The evaluated dataset repositories had no registered eval.yaml on 2026-09-20, so no .eval_results leaderboard entry or verified badge is claimed. Machine-readable results and provenance.

Usage

Install requirements.txt (tested with Transformers 5.17.0 / PyTorch 2.11.0). Custom model code is included; trust_remote_code=True is required. This example runs on CPU.

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
repo = "d0rj/t5-moe-55M-base"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True).eval()
c = model.config.ul2
prefix = tokenizer.encode("The capital of France is", add_special_tokens=False)
inputs = torch.tensor([[c["mode_ids"]["S"], *prefix, c["sentinel_ids"][0], c["eos_id"]]])
decoder = torch.tensor([[model.config.decoder_start_token_id, c["sentinel_ids"][0]]])
output = model.generate(input_ids=inputs, decoder_input_ids=decoder,
                        max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0, decoder.shape[1]:], skip_special_tokens=True))

To reproduce the core evaluation from a downloaded repository, install evaluation/requirements.txt and run:

python evaluation/run_core.py --device cuda:0 --dtype bfloat16 --batch-size 16 --output evaluation-rerun

TensorBoard and limitations

TensorBoard event files include training telemetry and eval/<task>/<metric> at step 15,000, plus separate CI bounds. Training telemetry covers steps 20–15,000 (750 loss points), including token CE, router loss, gradient norm, throughput, memory, padding and denoiser fractions.

These are small English continuation models, not instruction-tuned assistants. Equal source-token budgets do not imply equal target-token supervision or FLOPs. Benchmark contamination was not audited; results are from one training seed. Reference-model scores from different prompts, tokenizers or corpora are not directly interchangeable.