Puddings22/MIRAI-GlassBox-LLM

MIRAI — a 1.08B glass-box language model built on signal theory

A from-scratch 1.08B-parameter language model built to be a glass box: every write to the residual stream decomposes exactly — to 5×10⁻⁷ relative error — into 129 named spectral atoms whose time constants were fixed before training. Nothing is reverse-engineered after the fact. You read the computation off the forward pass the way you read a voltage off a circuit node, and the shipped instrument panel shows all of it, live, for every generated token.

The temporal layer is a fixed-pole linear filter bank, not learned attention over time: each of the 129 atoms is a damped complex exponential B_f·e^(a·k)·(cos ωk + i sin ωk) — the impulse response of a first-order resonator — applied by causal convolution. The poles are identical in all 10 layers, strictly inside the unit circle (|p| ≤ 0.995, so stable by construction), spanning time constants from 200.5 down to 13.0 tokens. Because those coordinates are chosen in advance rather than discovered, every named part has a meaning that does not depend on what training happened to do with it.

Where this stops being enough is stated plainly below and is the subject of the paper: seeing the computation exactly is not the same as knowing which parts matter.

Released with both checkpoints, the training code, the interpretability tooling, and a hash-pinned reproduction package for every number quoted below.

⚠️ This model is UNDERTRAINED — read this first

Do not expect good text out of it. MIRAI saw 25B tokens (23.1 per parameter) in a single compute-limited run, and the loss was still descending when the budget ran out — training was stopped by cost, not by convergence. A 1.08B model needs far more than this to saturate.

Concretely, expect:

• weak world knowledge — it will state wrong facts confidently, and it has no knowledge cutoff worth quoting because it never learned enough to have one
• short coherence — the attention window is 256 tokens; long-range consistency degrades quickly
• repetition and drift in longer generations
• an instruction-tuned checkpoint that follows the chat format well while still being limited by the base model's knowledge — the format is learnable at this scale; the knowledge is not
• no safety tuning of any kind — no alignment, no RLHF, no content filtering

What it is for. MIRAI is a research instrument for studying interpretability. Its value is that every step of its computation can be read exactly, and that its own transparency claims can be tested and falsified — not that it writes well. Judge it on the measurements below, not on its prose.

If you want a model that produces good text, this is not it. If you want a billion-parameter model whose every internal step you can read and verify, that is exactly what this is.

Model summary

Parameters	1,081.4M (1.08B)
Layers · heads	10 · 12
Model width C	1536
Gated width G	384
Spectral atoms Fr	129 fixed poles/layer, identical across layers, frozen
Time constants τ	200.5 → 13.0 tokens (pole magnitude ≤ 0.995, stable by construction)
Readout filter bank	129 poles, learned (τ 61.4 → 0.29 tokens)
Attention window	256
Pointwise feed-forward	none
Tokenizer	TokenMonster englishcode-32000-consistent-v1, 32,000 tokens — included in tokenizer/
Output head	hybrid — 15,268 learned + 16,732 rare tokens composed from 8,192 atoms
Training	25.006B tokens · 2,035,000 steps · 23.1 tokens/param
Weights	fp32 (all 230 tensors)
Training compute	bf16 autocast, fp32 master weights; fp32 island for the sparse code composition
Optimizer	Muon (hidden 2-D matrices) + AdamW (all else)
Validation	0.8204 BPB (fixed 2M-token prefix)
License	Apache-2.0

Loss was still descending at the compute budget. This is a research model for studying interpretability, not a state-of-the-art Transformer replacement and not trained to saturation.

Checkpoints

file	what	key numbers
checkpoints/m5_25B_ddp1/final.pt	pretrained base	0.8204 BPB · sha 98d7709b1d1a
checkpoints/m5_ft_25b4k_surgical/final.pt	instruction-tuned chat	chat NLL 2.7543 · sha 448aca59d0f6

⚠ Read this before loading

comp_map and rare_mask in cache/ are registered persistent=False, so they are not inside the checkpoints. Loading without them succeeds silently and yields a model with no rare-token head — wrong outputs, no error. Always assert missing_keys == [].

Quickstart

Requires torch and tokenmonster (pip install torch tokenmonster). The tokenizer vocabulary ships in this repo, so nothing is fetched at runtime:

import tokenmonster
vocab = tokenmonster.load("tokenizer/englishcode-32000-consistent-v1.vocab")
ids = vocab.tokenize("The capital of France is")

import sys, torch; sys.path.insert(0, "REBUILD")
from mirai5_clean import Mirai5, PRESETS

cm = torch.load("cache/compcode_map_englishcode-32000-consistent-v1_8192.pt",
                map_location="cpu", weights_only=False)
rm = torch.load("cache/rare_mask_englishcode-32000-consistent-v1_q60.pt",
                map_location="cpu", weights_only=False)
if isinstance(rm, dict): rm = rm.get("rare_mask", rm.get("mask"))

model = Mirai5(32000, **PRESETS["b1"], comp_map=cm, rare_mask=rm.view(-1).bool())
ck = torch.load("checkpoints/m5_25B_ddp1/final.pt", map_location="cpu", weights_only=False)
sd = {k.removeprefix("_orig_mod."): v for k, v in ck["model"].items()}
missing, unexpected = model.load_state_dict(sd, strict=False)
assert not missing, missing          # the trap above
model.eval()

Interactive chat with the full instrument panel:

python bin/mirai_glassbox.py --ckpt checkpoints/m5_ft_25b4k_surgical

For every generated token it prints which positions attention read, the spectral atom activations and their exact contributions, the router gate, both branch writes, the readout geometry against the 32,000 token codes, and a per-atom causal ablation. Each quantity carries an evidence tag (DIRECT / DERIVED-EXACT / DERIVED / PROBE / HEURISTIC / INTERVENTION), and the panel self-verifies bitwise against the executed arithmetic on every run.

Interpretability

For a fixed forward pass the selective gate is constant, so each layer's spectral write is exactly

Δwave  =  Σ_f c_f  +  b          (f = 1..129 named atoms)

an algebraic identity rather than a fitted approximation.

Measurement	Value
Residual-stream writes decomposing exactly into named contributors	100%
Max relative error of that decomposition (2,000 context×layer cells)	5.05×10⁻⁷
Instrument fidelity — replayed vs live forward pass	bitwise (0.00×10⁰)
Predictive gain retained under a structured restriction	94.7% (94.1–95.3%, six windows)
…against a stronger interpolated-bigram reference	92.6%
Fine-tuning damage recovered by restoring 33,792 LayerNorms (0.016% of what moved)	53.0%
Write magnitude split	wave 19.8% / attention 80.2%
Parameters with an a-priori specified role	0.00024% (2,580 of 1,081.4M)
Exact contribution → causal effect (median ρ)	0.26–0.42
True top-10 causal atoms recovered by ranking on it	3.9 / 10 (chance 0.78)

The last two rows are the point, not a disclaimer: the computation is exactly decomposable, yet exact contribution predicts causal importance only modestly. Observability and causal faithfulness are separate properties, and this model lets both be measured. The paper develops that argument — paper/MIRAI_paper.pdf.

The second instrument: how compressible is the computation?

bin/mirai_glassbox.py shows one forward pass in full detail. bin/skeleton_joint_matched_v2.py asks a different question — how much of the model's learned ability survives when its internals are forced into a restricted, describable form? Three intermediate computations are restricted simultaneously, and the loss is measured against the unigram floor: attention pruned to the top-K edges per query per head, values projected onto a rank-R basis, and the 129 filters snapped to a coarse (decay, frequency) taxonomy.

Restriction	Retained
pole bank → 18 classes	13.9%
pole bank → 35 classes	75.2%
pole bank → slow poles exact + 32 classes	99.2%
joint, K=8 · R=192	74.3%
joint, K=16 · R=384	91.9%
joint, K=32 · R=384	94.7%

The 13.9% row is the informative one. Compressing the pole bank to 18 classes nearly destroys the model, but keeping the slow poles exact and coarsening everything else recovers 99.2% — so the structure that matters is concentrated in the long-time-constant filters, while the fast ones are largely interchangeable. That is a measured statement about where the model's capacity lives.

This is diagnostic, not a reconstruction: the full quantities are computed by the intact trained weights before being restricted, so no inference speedup is demonstrated.

What you can actually inspect

bin/mirai_glassbox.py opens an interactive chat and, for every generated token, prints the panels below. Each is tagged with its evidence class so measurement is never confused with inference.

Panel	What it shows	Class
Spectral pipeline	each atom's state through gain, mixing, gating, projection	DIRECT
Atom output energy	per-atom output energy (squared magnitude) for all 129 atoms × 10 layers	DIRECT
Router gate	selective gate 1+tanh(sel), and where it saturates	DIRECT
Attention routing	which positions each head read, per layer	DIRECT
Branch writes	wave vs attention contribution to the residual stream	DIRECT
Named scalars	every learned scalar in the model, by name	DIRECT
Readout compass	prediction geometry against all 32,000 token codes	DIRECT
Code neighbourhood	crowding around the predicted token	DIRECT
Head census	all 120 heads (10×12) named by behaviour on this input: induction / local / anchor / diffuse / mixed	DERIVED
Residual stream	layer-by-layer state evolution	DERIVED
Window truncation	what fell outside the 256-token window	DERIVED
Logit lens	each layer decoded through the real readout	PROBE
Lexical routing support	which prompt tokens support the prediction	HEURISTIC
Position-local attribution (why)	exact contribution of one position to one prediction	INTERVENTION
Organ ablation (ablate)	zero a named atom or head, measure the change	INTERVENTION

The head census is the concrete answer to "which head is doing what?" — it classifies every head on the current input by where its attention mass went (induction heads by attending just after an earlier copy of the current token, local heads by the last four positions, anchor heads by position 0). It is a per-input classification, not a permanent label on the head.

The panel self-verifies bitwise against the executed arithmetic on every run: if the instrument and the model ever disagree, it says so instead of printing a plausible number.

Repository layout

path	contents
checkpoints/	base and instruction-tuned weights
cache/	comp_map + rare_mask — required to load correctly
tokenizer/	TokenMonster vocabulary (840 KB)
REBUILD/	mirai5_clean.py (architecture) · mirai5_ddp_clean.py (trainer)
bin/	glass-box chat, inspection, fine-tuning, restriction and attribution tools
experiments/	verification scripts, frozen data segments, result JSONs
paper/	the paper
verify_repo.py	hashes every file against its pin

Reproduce

experiments/frozen_segments.npz (15 MB) carries the exact evaluation, calibration and reference-fit token segments, so the headline experiments run without the original corpus.

python verify_repo.py                       # check every file hash first
python bin/skeleton_joint_matched_v2.py     # structured restriction    -> 94.7% retained
python experiments/bigram_floor.py          # stronger reference        -> 92.6% retained
python experiments/instrument_fidelity.py   # instrument self-checks    -> bitwise
python bin/ft_damage_attrib.py              # fine-tuning localisation  -> 53% from 33,792 params

Every script refuses to run on a checkpoint, source, artifact or data-segment hash mismatch.

Limitations

Undertrained: compute-limited single run, stopped by budget rather than convergence, with validation loss still descending. Output quality is well below what this parameter count could reach. Decay and frequency are confounded by construction. The poles are tied to the sampling step, so the model is not sample-rate invariant. Modality breadth is not established — one 148M audio pilot only. English only. No safety tuning, alignment or content filtering was applied; treat outputs accordingly.

Citation

@misc{dinca2026mirai,
  title  = {{MIRAI}: A Billion-Parameter Glass-Box Model Where Exact Observability
            Is Not Causal Faithfulness},
  author = {Dinca, Marius},
  year   = {2026},
  note   = {Model weights, interpretability tooling and reproduction package},
  url    = {https://huggingface.co/Puddings22/MIRAI-GlassBox-LLM}
}

Apache-2.0: commercial use, modification and redistribution are permitted provided the copyright notice is preserved. Academic citation is requested, not legally required.