mini-beatrix-2s

The first full-splat craft to complete a designed life. 237M parameters, byte-level (vocab 256), with a governed multi-constellation CausalSplatHUB — signed-address linear attention over learned codebook blackboards — in every one of its 20 blocks. No softmax-over-positions attention anywhere. Trained 16.101B tokens (61,422 steps, ~78h on a single RTX 6000 Pro Blackwell) through a staged curriculum, completed 2026-08-31.

Architecture

• d_model 1024 · 20 layers · ctx 4096 · byte-trigram embedding (raw UTF-8 bytes; input ids are byte values 0–255)
• Hubs (all 20 blocks): 4 constellations × 64 anchors @ D=128 per block, min-separation governed from birth (45°), budget-composed (numerators and agreement masses sum before one divide — reconstructive, never comparative: no argmax, no top-k, inhibition first-class). Constant-size prefix state: each layer encodes the sequence onto a fixed-width addressed blackboard rather than caching it.
• Anchored banks: 3 full-width experts per block, signed dispatch.
• Dual head: linear readout + a signed aleph read (256 anchors @ 256), revived mid-mission via a deterministic boundary-write after the born-null path self-buried — the first functioning aleph head in the lineage.
• Final causal contributions (toggle ledger, held-out): hubs +3.36 bpb · banks +3.31 bpb · head +0.012 bpb.

The special-token control plane

Thirteen ids that valid UTF-8 can never produce (proven by exhaustion) carry structure and are trained:

id	token	meaning
0xFF	DOC	document boundary (taught from step 0)
0xFE / 0xFD	USER / MODEL	turn openers (taught in the final chat phase)
0xFC	END	universal block close
0xFB	SYS	system block opener
0xF7+b	MODE	register tag (+1 ASCII byte)
0xF5+b	ESC	254 extended slots
0xFA 0xF9 0xF8 0xF6 0xC0 0xC1	THINK DATA SEP CUE RES	reserved/instrument

Chat format: [SYS] text [END] [USER] text [END] [MODEL] text [END] — the frame is unforgeable (encoded text cannot contain a special).

Usage

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "AbstractPhil/mini-beatrix-2s", trust_remote_code=True)

prompt = "The history of astronomy begins"
ids = torch.tensor([list(prompt.encode("utf-8"))])
out = model.generate(ids, max_new_tokens=200, do_sample=True, top_p=0.95)
print(bytes(int(i) for i in out[0]).decode("utf-8", errors="replace"))

Chat-framed prompting (the frame was annealed in the final 1B tokens — treat it as a young capability):

SYS, USER, MODEL, END = 0xFB, 0xFE, 0xFD, 0xFC
turns = ([SYS] + list(b"You are Beatrix, a small byte-level language model.")
         + [END] + [USER] + list(b"Who are you?") + [END] + [MODEL])
out = model.generate(torch.tensor([turns]), max_new_tokens=100,
                     do_sample=True, top_p=0.95)

Training

16.101B tokens on one card: wikitext warmup (0.3B) → fineweb-edu (5B) → a nine-stage early-life curriculum, s0–s8 (8.8B: narrative, perspective, concepts, rule-chains, arithmetic, causal, try-fail, mixed, register) → a two-phase anneal (1B distribution shift without the chat frame, then 1B with it — the frame's effect bracketed by boundary reports). Muon + pure Adam split, flat LR, bf16, zero loss spikes across the entire run. Every boundary shipped a report (toggle ledger, probe suite, address census): 17 reports and all checkpoints live in alephllm-mini-beatrix-training under mini-beatrix-2s/.

Final validation: 1.1097 bpb on the fineweb-edu holdout — the run's best reading, set during the chat phase.

Lineage

Code: AbstractEyes/alephllm (this repo vendors the model files verbatim; alephllm 0.8.6). Sibling: mini-beatrix-1 (112M, 3-hub hybrid — the previous rung). The research memory behind every design decision is maintained privately; laws referenced in the code docstrings (supply ≤ 2D, min-sep governance, budget composition, born-null weight-zero) are measured, not aesthetic.