Chat Preview

Byrne-100M-Ultra-MC

114M looped transformer (SpikeWhale v2 / Byrne) with a parallel Memory Cache branch. Trained from scratch on a web/code/math mix, then SFT, then DPO. All three stages are in this repo.

It's a small model. Fluent English, simple chat formatting, not a knowledge base, weak at code and multi-step reasoning. Numbers in Evaluation. morpho/ is a side project: the same block grown as an int8 gate circuit. It is not on the generate path.

Checkpoints

file	stage	use
checkpoints/base_62k.pt	pretrained, Dolma-blend, step 62k	lowest PPL. Continues text, does not answer questions.
checkpoints/sft_7100.pt	SFT on UltraChat, step 7.1k	chat / instruction following. --chat
checkpoints/dpo_3200.pt	DPO on the SFT model, step 3.2k	preference-tuned chat (default). --chat

Same architecture, same loader. Weights and context length differ: SFT was trained at 4096, base and DPO at 1024.

Architecture

Looped decoder (model_v2.py, SpikeWhale v2 / Byrne). Dense — use_moe is off; FFN width is still moe_intermediate_size 2048. Vocab 16,512 via a byte-level length-max tokenizer (spike_tokenizer.py, tokenizer.json). Text → UTF-8 → latin-1 bytes, then greedy longest-match against the vocab. Not a HuggingFace tokenizers file; the wrapper is what AutoTokenizer talks to.

Parameters	113.9M
Hidden size	768
Layers	16
Attention heads	12 (2 KV heads)
Head dim	64
FFN	2048, SwiGLU, dense
Loop count	3 (same stack, three times, shared weights)
Tie embeddings	yes
Context	1024 (base/DPO), 4096 (SFT)

Config lives in config.py + byrne_100m_ultrax_mc.yaml. Released .pt files carry the config they were trained with. Engine and tools read that, not the yaml, because the stages disagree about context length.

Loop

The 16-layer stack runs three times with the same weights. Effective depth 48, parameter cost of 16 layers plus a tiny per-pass embedding (loop_pass_embed, zero-init). loop_count=1 is the dense baseline.

loop_layer_plan walks (layer, pass) pairs. Cache slots are the plan index, not the layer index. Same weights, different activations each pass, so pass 1's K/V cannot share a slot with pass 0. Depth-attention state is reset at the start of every pass so pass 3 does not attend to a stale pass-1 value of the same layer.

loop_mode can be "full" (this model) or "middle_split" (unlooped prefix/suffix, only the middle repeats). Not used here. The engine still sizes the KV cache from the plan, so that mode would not silently break.

MLA, XSA, QK-norm

Multi-head latent attention: Q through q_lora_rank 128, output through o_lora_rank 128, RoPE only on qk_rope_head_dim 16 of each 64-dim head. 2 KV heads (GQA). Per-head RMSNorm on Q and K before RoPE (use_qk_norm). z-loss 1e-4 on the LM head.

XSA (exclusive self-attention) is on: an orthogonality correction that pulls the self-echo out of the attention output. DERF and Elo attention exist in config.py and are off.

Depth attention

Cross-layer, not cross-token. At layer L the current V is mixed with V from earlier checkpointed layers (stride 4), weights from a softmax over the depth axis. Learned skip along depth. One gate parameter per layer, zero-init, no-op at step 0. The attention forward takes depth_kv and returns depth_entry. A loader written for the Mark2 signature misses that and falls back to torch.cat cache growth.

Hyper-Connections, Engram, HRM, MTP

• Hyper-Connections (hc_mult 2): two residual streams with learned routing between layers. Final mix is learned (hc_out_mix, init = mean).
• Engram: hash-table n-gram memory into the embeddings. Compress dim 32, 2 heads, table 4096, max n-gram 3. This tree fixed lookup internally (EngramModule.lookup(prefix)); there is no engram_context_ids kwarg. The engine threads the prefix through a trailing cache slot.
• HRM refinement: one small block after the stack, 1 inner step, dim 128, deep supervision on during training. Inference reads the last step only. Gate init is nonzero so the zero-init up still gets a gradient.
• MTP: one extra head, loss weight 0.3. At inference the engine uses it as a speculative draft. Identity-checked against the main head.

Memory Cache

Adapted from Behrouz, Li, Deng, Zhong, Razaviyayn & Mirrokni, Memory Caching: RNNs with Growing Memory (Google Research, arXiv:2602.24281, 27 Feb 2026). Runs next to MLA as a gated residual.

The paper's setup, short version: transformers keep every past token addressable and pay O(L²) for it. RNNs squash history into a fixed state at O(L) and forget. They split the sequence into segments, stash a checkpoint of the recurrent memory at each boundary, and let a query read the live state plus those checkpoints. Segment count slides you between the two ends.

Here it sits per layer beside softmax attention: mc_num_heads 4, mc_head_dim 32, mc_gate_dim 64, mc_segment_len 256.

1. Cut the sequence into segments of 256.
2. Per segment i, Katharopoulos map φ(x) = elu(x) + 1, accumulate M⁽ⁱ⁾ = Σₘ φ(kₘ) vₘᵀ and z⁽ⁱ⁾ = Σₘ φ(kₘ). Fixed-size summary of the segment, however many tokens went in.
3. Token t in segment s reads φ(q_t) M⁽ⁱ⁾ from every i ≤ s. Own segment is a causal running memory (tokens up to t only). Earlier segments are finished, so their full memory is used as-is.
4. Gate: γ_{t,i} = softmax_i ⟨u_t, meanpool(x over segment i)⟩, masked to i ≤ s. Per token, per layer, pick which segment to look at.
5. One global normaliser, numerators left unnormalised: y = Σᵢ γ·num / (Σᵢ γ·den + ε).
6. Mix in through a zero-init gate: attn_out = attn_out + tanh(mc_gate) · mc_out.

Two bugs that had to stay fixed:

• Diagonal gate needs a causal running mean. Mean-pooling the whole current segment leaks future tokens into token t. Implementation is a cumsum running mean on the diagonal.
• Per-segment normalisation kills it. You lose query–key match magnitude and the only thing left distinguishing segments is the coarse mean-pool gate. Global normaliser is what keeps match strength in the signal.

Runs in fp32, autocast off. The linear-attention denominator underflows in bf16 and training blows up.

This is not an RNN, so the paper's complexity story does not transfer cleanly. Cross-segment readout here is O(N²L). It is not a long-context speed trick. The thing that transferred is a second retrieval path with learned segment-level routing next to token-level attention, on a 114M model that does not have much attention to spare. Recall is the usual failure mode at this size. Zero-init gate means at step 0 the model is identical to the no-MC baseline; the branch only starts mattering if training uses it. Pretrain seq_len was 1024 = 4 segments, so it sees cross-segment memory from the start.

It did get used. DPO 3.2k checkpoint:

metric	value
gate strength (mean |tanh(gate)| over layers)	0.599 (range −0.75 … +0.90)
UltraX PPL, MC on	9.87
UltraX PPL, MC off	11.86
MC contribution	−16.7% perplexity

Gates are open. Knocking the branch out costs ~17% PPL (it was ~9% earlier, before they opened). That is an inference knock-out on this checkpoint, not a model trained without MC. A matched no-MC run is in progress (see below).

If you write a loader: this branch is a full-sequence op. It uses the whole x given to forward. Naive KV cache feeds it one token, it rebuilds segments from a 1-token window, and you get a different function than training. Still fluent. Gate here is ~0.56–0.60 (Mark2 siblings sit around 0.19), so the damage is worse: max |Δlogit| 7.24e-01 vs a full recompute, against a ~1e-5 float32 floor. Shipped code keeps incremental state per segment and per loop pass (loop_count=3, same branch three times on the same positions). That is what makes cached decode match a full recompute. python verify.py (generate.py / model_v2) or python verify_cache.py (spike_infer).

Fractal RoPE

Cantor-spectrum RoPE (gamma=1) instead of the usual geometric schedule. Same endpoints and band count, exponents on the middle-thirds Cantor set. See fractal.py. Swap the trained model to standard geometric RoPE at inference (same weights, different frequencies) and it falls over:

Source	fractal (trained)	standard RoPE (swapped)	Δ
UltraX-web	7.95	77.86	+879%
DCLM	8.21	70.74	+762%
FineWeb-Edu	10.53	81.57	+675%
WikiText-2	9.08	71.06	+683%

The 7–10× PPL spike means the attention is tuned to these frequencies. Any RoPE variant would do this if you swapped the schedule at inference. It shows the fractal schedule is part of this model. It does not tell you fractal beats standard RoPE. That comparison is a matched gamma=0 train, in progress (see below).

Matched runs in progress

Two retrains, same recipe as this base:

• no Memory Cache
• standard geometric RoPE (gamma=0) instead of fractal

Apples to apples against the tables above. The −16.7% MC number and the RoPE swap blowup are both "this checkpoint, then change one thing at load." The new runs train that way from step 0, so they actually answer whether MC or fractal helped. Numbers, the comparison checkpoints, and the training scripts in full should land around 26–28 Aug 2026. I'll drop them in this card when they're done.

Training

• Pretrain: FineWeb-Edu, Wikipedia, DCLM, Cosmopedia-v2, FineMath-4+, Python-Edu, then a Dolma-mix continuation (allenai/dolma3_dolmino_mix-100B-1125). Muon on matrices, AdamW on norms/embeddings. Released at step 62k.
• SFT: HuggingFaceH4/ultrachat_200k, multi-turn, assistant-only loss. ChatML: <|im_start|>{role}\n…<|im_end|>. Step 7.1k.
• DPO: preference tuning on the SFT model. Step 3.2k.

Full hyperparams further down.

Evaluation

PPL is the language model. Benchmarks are task behaviour. SFT/DPO spend PPL to buy the latter.

Perplexity (per-token CE, lower better)

Streamed 10×1024-token windows per source (WikiText-2: 40 windows). Same byte-level tokenizer everywhere, so the columns are comparable.

Domain	base-62k	sft-7100	dpo-3200
Python-Edu (code)	4.52	4.54	4.54
Cosmopedia-v2	5.51	5.38	5.35
FineMath-4+	6.32	6.84	6.86
Dolma	7.11	7.55	7.57
Wikipedia	7.26	7.83	7.84
UltraX-web	7.95	8.58	8.60
DCLM	8.21	9.10	9.12
WikiText-2	9.08	10.11	10.13
FineWeb-Edu	10.53	11.69	11.70
mean (5-src)	6.50	6.89	6.89

mean (5-src) = Python-Edu, Cosmopedia, FineMath, UltraX, DCLM.

Base wins free-text PPL (it's the untuned LM). SFT/DPO raise PPL on web text and improve on synthetic instructional text (Cosmopedia). DPO ≈ SFT on PPL.

Benchmarks

lm-eval-harness style. Capped 200/task, BLiMP 150/paradigm (12 paradigms), ArithMark 500. Same tokenizer on all three stages. Numbers from 2026-08-18.

n: ARC-Easy 200, ARC-Challenge 200, HellaSwag 200, Winogrande 200, PIQA 200, OpenBookQA 200, BoolQ 200, ArithMark 500.

Metric	base-62k	sft-7100	dpo-3200	chance
WikiText-2 byte_ppl ↓	2.308	2.383	2.385	—
BLiMP acc ↑	0.811	0.780	0.779	0.50
ARC-Easy acc	0.410	0.395	0.400	0.25
ARC-Easy acc_norm	0.390	0.400	0.400	0.25
ARC-Challenge acc	0.205	0.265	0.255	0.25
ARC-Challenge acc_norm	0.270	0.260	0.260	0.25
HellaSwag acc	0.370	0.375	0.370	0.25
HellaSwag acc_norm	0.435	0.420	0.415	0.25
Winogrande acc	0.515	0.500	0.505	0.50
PIQA acc	0.565	0.590	0.590	0.50
PIQA acc_norm	0.555	0.575	0.575	0.50
OpenBookQA acc	0.105	0.100	0.100	0.25
OpenBookQA acc_norm	0.290	0.285	0.295	0.25
BoolQ acc	0.355	0.430	0.425	0.50
ArithMark-3.0 acc_norm	0.354	0.380	0.378	0.25

Base wins the LM metrics (byte_ppl, BLiMP). SFT/DPO win BoolQ (+0.07), ARC-Challenge acc (+0.06), ArithMark, PIQA. DPO vs SFT is noise on these scores; DPO's job was response preference, not multiple-choice. OpenBookQA acc is under chance on all three. A lot of the MC tasks sit near chance. 114M.

Benchmarks (full)

Same harness, no 200-example cap. WikiText-2 byte_ppl and BLiMP are the same numbers as above (those were already full). MC tasks and ArithMark are the whole set.

n: ARC-Easy 2,376, ARC-Challenge 1,172, HellaSwag 10,042, Winogrande 1,267, PIQA 1,838, OpenBookQA 500, BoolQ 3,270, ArithMark 1,000.

Metric	base-62k	sft-7100	dpo-3200	chance
WikiText-2 byte_ppl ↓	2.308	2.383	2.385	—
BLiMP acc ↑	0.811	0.780	0.779	0.50
ARC-Easy acc	0.429	0.411	0.411	0.25
ARC-Easy acc_norm	0.394	0.388	0.388	0.25
ARC-Challenge acc	0.190	0.208	0.206	0.25
ARC-Challenge acc_norm	0.220	0.227	0.230	0.25
HellaSwag acc	0.278	0.281	0.281	0.25
HellaSwag acc_norm	0.293	0.291	0.290	0.25
Winogrande acc	0.515	0.521	0.518	0.50
PIQA acc	0.583	0.584	0.583	0.50
PIQA acc_norm	0.583	0.584	0.583	0.50
OpenBookQA acc	0.118	0.120	0.120	0.25
OpenBookQA acc_norm	0.242	0.240	0.244	0.25
BoolQ acc	0.381	0.418	0.416	0.50
ArithMark-3.0 acc_norm	0.358	0.372	0.373	0.25

HellaSwag acc_norm drops from the capped 0.435 / 0.420 / 0.415 to ~0.29 once you take all 10k items. ARC-Challenge SFT bump shrinks too (+0.06 capped, +0.018 full). BoolQ still moves (0.381 → 0.418). DPO and SFT stay within noise. OpenBookQA acc is still under chance.

Escarda-86M-Base (same family, different everything else) hit 2.2228 WikiText-2 byte PPL after ~20B tokens. This base is 2.3085 after ~2B. Different size, architecture, objectives, data. That is a sample- efficiency note, not an MC ablation. The no-MC twin is the run for that.

Vocab (16k on purpose)

Tokenizer is 16,512 (16,384 byte-level length-max plus specials). That was a choice. I wanted to see the architecture, not hide it behind a fat embedding table.

At hidden 768, tied embeddings are already ~12.7M params at 16k. 32k would be ~25M. 64k would be ~50M — almost half this 114M model sitting in the lookup table. 16k keeps the stack (loop, MLA, Memory Cache, Engram, HRM) as the thing you're looking at.

Per-token PPL in the tables above is for this tokenizer. Raise vocab to 32k or 64k and per-token PPL goes up on average even if the language model is the same or better: tokens get longer, fewer of them per document, more bits per token, exp(mean NLL per token) rises. That is a tokenizer effect. WikiText-2 byte_ppl (2.308 on the base) is the number that would still be comparable.

I did not train a 32k/64k twin. If I did, I would expect higher token PPL, similar or slightly better byte PPL if the extra merge rules earned their keep, and a lot less of the param budget left for the block. 16k is the research vocab. Someone SFT'ing the base who wants a "normal" tokenizer should rebuild it, not read 6.50 mean PPL as what this stack would score at 32k.

Usage

Plain PyTorch. No transformers model class. CPU is fine; --device cuda if you have one.

# base (text continuation)
python generate.py --ckpt checkpoints/base_62k.pt -p "The capital of France is" --temp 0.7 --top-k 40 --rp 1.3

# DPO chat (default checkpoint)
python generate.py --chat -p "Explain why the sky is blue in one sentence." --temp 0.7 --top-k 40 --rp 1.3

# SFT chat
python generate.py --ckpt checkpoints/sft_7100.pt --chat -p "Give me a tip for staying focused."

Defaults: temp 0.7, top_k 40, rep_pen 1.3. Greedy (--temp 0) is a test tool. Chat models under greedy walk into disclaimer boilerplate and look worse than they are. DECODING-DEFAULTS.md.

Two inference stacks, two cache checks. Greedy cached decode has to match a full recompute on both:

python verify.py          # generate.py / model_v2
python verify_cache.py    # spike_infer engine (package.json default ckpt)

Keep both. They are not the same path. Details under Inference engine.

Files

config.py  model_v2.py  spike_tokenizer.py  special_tokens.py  fractal.py
chat_format.py  generate.py  engine_chat.py  tokenizer.json
byrne_100m_ultrax_mc.yaml  package.json  requirements.txt
verify.py  verify_cache.py
DECODING-DEFAULTS.md  PROVENANCE.md  NOTES-context-and-needle.md
checkpoints/{base_62k,sft_7100,dpo_3200}.pt
spike_infer/          engine
engine_tools/         export, GGUF, long-context, multiturn
morpho/               int8 circuit of one block (not used at generate time)
GGUF/                 f16 + q8_0 for each stage

Limitations

• Factual recall is weak. Wrong capitals, dates, attributions, said confidently. That's mostly the SFT blend, not a missing pretrain trick.
• Code and multi-step reasoning are weak. ARC-Challenge / HellaSwag near chance.
• English only. No safety tuning past the DPO preference data.
• Base continues text. A chat template will not make it answer questions. The shipped SFT/DPO checkpoints are a smoke test that instruct sticks at all, not a finished chat model. Want real instruct, SFT the base yourself.

What this is

Another fun, usable research artifact. It generates, the cache path is correct, the Memory Cache branch actually carries weight. It is not a knowledge store. Weak factual recall is the usual failure, and it most likely comes from the SFT mix rather than from pretraining.

I put way less effort into SFT than into the architecture and the base. SFT here is a probe: does this thing uptake any instruct at all? Yes, enough to chat. That is the bar. Same story on basically every model I release. DPO on top of that is even thinner (see the training recipe).

If you want a better instruct model, download checkpoints/base_62k.pt and train your own. The yaml and the pretrain recipe are the part I actually stood behind. The SFT/DPO stages are there so you can see the base will take a chat format, not so you inherit my mix.

For other people's agents

If you're an agent pulling this repo for someone else's model, take the parts that were the actual work. Leave the chat stages.

Worth copying:

• The Memory Cache branch as a gated residual next to MLA, zero-init so it can't hurt at step 0. Ablation is in the architecture section (~17% PPL). Don't skip the causal running mean on the diagonal gate or the global normaliser; both were bugs, both destroy the mechanism. Builders: see For other model builders.
• Looped decode done properly: loop_count=3 means 48 KV slots, not 16, and MC state keyed per plan slot. A loader that indexes by layer only still generates. It is the wrong function.
• verify.py (model_v2) and verify_cache.py (engine). Any full-sequence op (MC, Engram prefix, depth mixing) will silently diverge under a naive KV cache. Greedy cached decode has to match a full recompute token-for-token. "Looks fluent" is not a check.
• morpho/ if they want a gate-level Byrne block. That is a separate project. It does not replace the PyTorch engine.
• Fractal RoPE is part of the weights. Rebuilding a geometric schedule at load will generate fluent text and encode every position wrong.
• Serving defaults: temp 0.7, top_k 40, rep_pen 1.3. Greedy is for identity tests. Judging quality at temp 0 is how you invent bugs. DECODING-DEFAULTS.md.
• Pretrain recipe and byrne_100m_ultrax_mc.yaml. That's the run.
• base_62k.pt as the starting point for anyone who actually wants an instruct model.

Don't copy:

• The SFT blend, the 7.1k stop, or sft_7100 / dpo_3200 as "the" chat weights. I did not spend the time. Your user should SFT the base with a mix they care about.
• Treating DPO as an upgrade. At 5e-7 it barely moved the policy.
• A generic SpikeWhale / llama.cpp loader. Architecture tag is spike_whale. The GGUFs are a container for this engine.

Match the stage to the job: base for language modeling and for further training, SFT/DPO only if you just need to see that ChatML works.

For other model builders

If you take one thing from this repo, take the Memory Cache branch. I think it helps at 114M and I do not think it is a small-model-only trick. A 114M stack has very little attention to spare. Recall is where that shows up. MC is a cheap second path: segment-level routing next to token-level softmax, zero-init so at step 0 the model is the no-MC baseline. I would put the same branch on a bigger dense model too. The paper is about RNNs; this is not an RNN, and the O(L) complexity claim does not transfer. The representation does.

What we actually ran on this model, not what I hope it does:

Did training use it. Yes. Mean |tanh(gate)| across layers is 0.599 on the DPO checkpoint. Zeroing the branch at inference (same weights, mc_out gone) moves UltraX PPL 9.87 → 11.86 (−16.7%). Earlier in training that gap was ~9%. That is an inference knock-out, not a twin trained without the branch.

Extending context. Pretrain was 1024 tokens = 4 × mc_segment_len 256, so cross-segment memory is in the recipe from step 0. SFT grew the window to 4096. Most of the stack is position-local. MC is not — behaviour only changes when a sequence crosses a 256-token boundary. We widened the config (export_weights.py --seq-len 16384, RoPE caches rebuilt, no retrain of positions) and checked that cached decode still is the trained function:

check	result
prefill 252 (just under a segment)	6/6 identical, max |Δlogit| 3.05e-05
prefill 260 (just over)	6/6, 3.82e-05
prefill 519 (two segments)	6/6, 4.58e-05
decode stepping over a boundary (prefill 253)	8/8, 3.24e-05
prefill 4,096 / 6,000 / 8,192 after the widen	6/6 each, ~3–4e-05

~3e-05 is the float32 floor at this depth. So: you can run this checkpoint past 4k, including across MC segment edges, without the cache silently computing a different model. 48-slot KV at 16k is 0.40 GB bf16.

That is engine correctness. Positions past 1024/4096 got no gradient. It does not say the model writes well out there. It also does not say MC retrieves facts from depth — the first needle test was the model continuing the start of the prompt; see that section.

What is still missing. A matched train with use_memory_cache: false from step 0. That run is in progress, same recipe as this base, with the training scripts. Should land 26–28 Aug 2026. Until then the −16.7% is "this checkpoint needs the branch," not "MC beat a no-MC model." If you are going to copy it anyway, copy the two bugfixes or you will not have the mechanism: causal running mean on the diagonal gate, and one global normaliser (not per-segment). fp32, autocast off. State keyed per segment and per loop pass if you loop. Set seq_len to a multiple of mc_segment_len so the branch sees more than one segment from the first step.

Provenance

Weights re-saved without optimizer state. Inference code is the cache-correct ("repair2") build: Memory Cache is a full-sequence op, so a naive KV cache recomputes it from one token and you get a different function. This build keys per-loop MC state so cached decode matches a full recompute. verify.py / verify_cache.py. See PROVENANCE.md.

safetensors

Same three stages, safetensors/ instead of .pt. For eval harnesses.

directory	stage	context
safetensors/base_62k/	pretrained base	1,024
safetensors/sft_7100/	SFT	4,096
safetensors/dpo_3200/	DPO	1,024

config.json + model.safetensors + tokenizer.json. Float32. Matches the .pt bit for bit (502 tensors, 0 mismatches).

python safetensors/load.py sft_7100

Details in safetensors/README.md. Short version:

• No lm_head.weight in the file. Tied embeddings; safetensors drops shared storage. Call model.tie_weights() or you get a random head and garbage logits, no exception.
• AutoModelForCausalLM is a no. spike_whale is not in transformers. Use model_v2.py / config.py (load.py already does).
• Context is 1024 / 4096 / 1024. Read the config.json in that directory. Past the end raises, does not wrap.
• Put <bos> on every sequence. Scoring: use_cache=False forwards. Generation: the engine, not a naive KV cache.

GGUF builds

GGUF/ has every stage in two precisions. Tensors keep their original state-dict names. Full config is spike_whale.config_json, so a loader can rebuild the state dict 1:1.

file	stage	size
Byrne-100M-Ultra-MC-base_62k-f16.gguf	pretrained base	228 MB
Byrne-100M-Ultra-MC-base_62k-q8_0.gguf	pretrained base	122 MB
Byrne-100M-Ultra-MC-sft_7100-f16.gguf	SFT	228 MB
Byrne-100M-Ultra-MC-sft_7100-q8_0.gguf	SFT	122 MB
Byrne-100M-Ultra-MC-dpo_3200-f16.gguf	DPO (default)	228 MB
Byrne-100M-Ultra-MC-dpo_3200-q8_0.gguf	DPO (default)	122 MB

Round-trip (re-read GGUF, compare every tensor to source): f16 max |Δ| ≤ 0.008, q8_0 ≤ 0.079. Tokenizer is byte-faithful across all 16,512 tokens. f16 GGUF through the engine is byte-identical greedy output to the .pt.

Rebuild:

python engine_tools/export_weights.py --ckpt checkpoints/sft_7100.pt --out /tmp/exp
cp -r /tmp/exp ./_tmp_model
python engine_tools/convert_gguf.py . --model-subdir _tmp_model --name Byrne-100M-Ultra-MC-sft_7100 --quant f16

Not llama.cpp-compatible. Architecture is spike_whale: looped stack, MLA, fractal RoPE, depth attention, Memory Cache. Nothing upstream implements that. These GGUFs are a weight container for this codebase, not a llama-cli drop-in.

RoPE caches are not stored (recomputed at load). Fractal schedule, so a loader has to rebuild it via fractal.py. If you rebuild standard RoPE instead it loads, generates fluent text, and encodes every position differently from training (inverse frequencies off by 2.2e-01). spike_infer does this; anything else reading these files has to as well.

Inference engine

Two stacks. Both ship. Both have to stay cache-correct.

	generate.py	engine_chat.py / spike_infer/
What	thin wrapper over model_v2.forward	full engine for this checkpoint
Default ckpt	package.json (dpo_3200)	same, unless --ckpt
Sampling	temp 0.7 / top_k 40 / rp 1.3 from the manifest	same, hardcoded to match
Chat	--chat wraps ChatML	ChatML by default, --raw for the base
MTP draft	no	yes, unless --no-mtp
Static KV	model's own past_key_values	in-place StaticKV, 48 slots
Check	verify.py	verify_cache.py

spike_infer/ is written for this model, not a generic SpikeWhale loader. Every forward runs what the checkpoint trained with: MLA + QK-norm + fractal RoPE + XSA, depth attention, the looped stack with per-pass embedding, Hyper-Connections, Engram prefix slot, Memory Cache (fp32, autocast off, state per plan slot), HRM refinement, MTP as a speculative draft.

Load path: package.json → .pt + the config blob inside that .pt (not the yaml). Optional GGUF via from_gguf. Config from the checkpoint matters because SFT is 4096-context and base/DPO are 1024.

Decode path: prefill the prompt from position 0, then one new token at a time. Speculative MTP only proposes; the main head verifies. Mid-stream multi-token prefill is outside what MC incremental state is checked for. Each new user turn re-prefills the window from 0 so RoPE positions, HC streams, MC segments, and the Engram window match a fresh run.

Three things a generic loader gets wrong, all silently:

1. 48 KV cache slots, not 16. loop_count=3 means the same weights see different activations each pass, so every (pass, layer) pair owns its own K/V. Slot index comes from loop_plan. Index by layer alone and pass 1 keys get fed to pass 0. Still generates. Wrong function.
2. Attention carries depth_kv and returns depth_entry. A static-KV patch written for the Mark2 signature does not match, and silently falls back to torch.cat cache growth. Depth mixing happens on the fresh K/V before the cache is touched. Mixed V is what gets cached.
3. No engram_context_ids kwarg. This tree fixed Engram internally: EngramModule.lookup takes a prefix through a trailing cache slot. The engine supplies that slot and reads it back each forward. Mechanism is picked by inspecting the model signature, not by tree name.

Memory-Cache state is keyed per plan slot, since the looped stack calls the same branch once per pass on the same positions.

Do not delete the verify scripts. They are the only proof the cache path equals a full recompute. verify.py hits generate.py. verify_cache.py hits the engine (slot count, token identity, per-step |Δlogit|). Cheap on CPU. A release without them is a release nobody can re-check.

Verified

verify_cache.py. Greedy is deterministic, so cached decode has to produce the same token ids as a full recompute, not "similar text":

stage	tokens identical	max abs Δlogit
sft_7100	8/8	1.72e-05
dpo_3200	12/12	1.91e-05

~2e-05 is the float32 floor at this depth. Incremental path computes the same function, it does not approximate it.

Prompt tokenization is checked id-for-id against the training tokenizer. <bos> is prepended (training starts every sequence with it). Display decode keeps <think> markers that skip_special_tokens=True would drop.

Long context

Two different measurements. A context-test fail is an engine bug. A needle fail is a 114M-model property. Fixing the first does not improve the second. Numbers below are sft_7100 unless noted. Method writeup: NOTES-context-and-needle.md.

Most of the stack is position-local. Memory Cache is not — it is segmented (mc_segment_len 256) — and Engram carries an n-gram prefix across the cache boundary. A short verify.py run never leaves segment 0, so it cannot see a cross-segment bug. check_context_ready.py sits either side of 256 on purpose: RoPE buffers rebuild exactly on a widen, StaticKV footprint at the target length (48 slots × 16384, bf16 = 0.40 GB), cache exactness just under / just over / two segments in, decode itself stepping over a boundary, multi-turn window slide, and position_ids == ctx raising instead of wrapping to 0.

11/11 on sft_7100:

rope cache recomputes from scratch exactly     max |Δ| 0.000e+00 over 4096 positions (fractal=True)
rope cache at 16384 agrees on first 4096       max |Δ| 0.000e+00
StaticKV, 48 slots × 16384 tok, bf16           0.40 GB
prefill 252 tok  (just under a segment)        6/6 identical, max |Δlogit| 3.05e-05
prefill 260 tok  (just over)                   6/6 identical, max |Δlogit| 3.82e-05
prefill 519 tok  (two segments)                6/6 identical, max |Δlogit| 4.58e-05
decode crossing a boundary (prefill 253)       8/8 identical, max |Δlogit| 3.24e-05
window slide                                   3655 ≤ 4032 tok, history dropped
position == ctx                                raises (index out of bounds)

~3e-05 is the float32 floor at this depth. Same token ids as a full recompute, not "similar text".

Widen with export_weights.py --seq-len 16384 and the engine stays exact past the trained length. Those positions got no gradient; this is mechanical soundness, not quality.

prefill	tokens identical	max abs Δlogit
4,096	6/6	3.34e-05
6,000	6/6	2.96e-05
8,192	6/6	4.20e-05

python engine_tools/check_context_ready.py --device cuda \
    --ckpt checkpoints/sft_7100.pt --target-ctx 16384

Needle-in-a-haystack

First pass: The secret access code is BLUEBERRY-7291. as sentence 1, one filler sentence repeated to length, What is the secret access code? at the end. Greedy was the primary arm. keyword = BLUEBERRY came back. exact = full string, digits included.

prompt tokens	greedy: keyword	greedy: exact	temp 0.7: keyword
505	yes	yes	yes
1,011	yes	yes	yes
2,046	yes	yes	no
3,587	yes	yes	—

Greedy copied the digits out to ~3.6k. dpo_3200 is exact at 505; its context is 1,024 so the longer rows do not apply. Temp 0.7 often mashed the suffix (BLUEBERRY-dica, BLUEBERRY-Definition1) and missed the keyword at 2k on that seed. Sampled column is one draw.

That table is what was measured. It is not retrieval. Needle was the first sentence, filler was one sentence on a loop, code was a fixed string. Greedy wrote The secret access code is BLUEBERRY-7291. The archive records r... — the start of the prompt, copied. You pass this test by continuing from position 0. Nothing has to be looked up.

Re-ran with that shortcut closed: varied filler, a topic-specific question, a new randomly generated code every trial.

setting	exact
fixed code, needle at depth 0, no distractors	0.000
random code, depths 0/0.5/1.0, 3 distractors, 512–2048 tok	0.056

Typical miss: the frame is right, the digits are invented — BLUEBERRY-?3, BLUEBERRY-Sch1, BLUEBERRY-tioxy, BLUEBERRY-ggregate. It put out a code 33% of the time and it was the wrong one. Nothing code-shaped 61% of the time.

It will echo a fixed, familiar string sitting at the top of the window. It does not reliably pull an arbitrary 4-digit code out of depth. This data does not split "can't retrieve" from "can't copy four random digits." Both fit. The window is not the only limit.

Engine files

spike_infer/__init__.py       the engine
engine_chat.py                chat / completion CLI (sampling defaults)
verify.py                     generate.py cache == full recompute
verify_cache.py               engine cache == full recompute
engine_tools/
  export_weights.py           .pt  ->  safetensors + config.json
  convert_gguf.py             safetensors  ->  GGUF
  check_context_ready.py      long-context / segment-boundary checks
  test_multiturn.py           multi-turn engine correctness
  compare_checkpoints.py      two stages, same prompts and seeds

morpho/ — the block as a circuit

morpho/ is a separate project in this repo. It does not run at generate time. It grows Byrne's decoder block as combinational int8 logic in MorphoHDL (Mordvintsev, Paradigms of Intelligence), folds real Ultra-MC weights in as constants, and emits synthesizable Verilog.

A transformer on a GPU is weights × data on a processor. This expands the arithmetic into AND/OR/XOR/NOT. No clock, no instruction stream. Put activation bits on the input wires; the graph settling is the forward pass. Same netlist simulates in tiny_morpho.py or goes to Yosys/Vivado/Quartus. Once a weight is a constant, constant-propagation deletes the gates that depended on it, so the circuit is specialized to these trained values.

Numeric contract is int8 Q1.7 activations/weights, int32 accumulators, Q4.4 attention scores, Q0.8 softmax. Checked against integer NumPy at the same fixed-point, not against float.

What it covers: RMSNorm → attention → residual → RMSNorm → GELU MLP → residual. That is the spine. Not grown as a datapath: MLA's low-rank path, Memory Cache mixing, the loop, fractal RoPE. MC projection weights do fold (mc.q_proj in byrne_fold.py).

Scale: a full d=4 block is 419k gates generic, 299k folded (29%), emitted as byrne_block.v. A flat d=768 block would be ~1e9 gates and will not compile. The 768-wide unit that does emit is one projection neuron, byrne_proj768.v (208k gates, real k_proj row folded). A chip would instantiate one of those and clock it, which is also how the looped stack wants to be mapped: one circuit, run three times.

Two ways to run it:

• Grow every gate (byrne_morpho.py, byrne_block_folded.py, byrne_emit_verilog.py). Synthesizable. Does not produce language.
• Run the same int8 MAC inside the real 114M model (byrne_circuit_infer.py). Produces coherent text. A slice is bit-exact against the grown dot.

Reads ../checkpoints/ read-only. Docs: morpho/README.md (what to run, numbers) and morpho/morpho-transformer.md (language notes and the original stage plan).

cd morpho
python byrne_morpho.py
python byrne_fold.py
python byrne_block_folded.py
python byrne_circuit_infer.py -p "The capital of France is"
python byrne_emit_verilog.py

Running it

Temperature sampling: temp 0.7, top_k 40, rep_pen 1.3, random seed per turn. Greedy is for determinism proofs. Judging quality by it gives the wrong answer. A repetition study on a sibling under greedy measured self-repetition 0.364; at the serving defaults it was 0.000.

# chat, DPO stage (default), multi-turn REPL
python engine_chat.py

# one-shot
python engine_chat.py -p "Explain why the sky is blue in one sentence."

# a different stage
python engine_chat.py --ckpt checkpoints/sft_7100.pt -p "Give me a focus tip."

# raw continuation, no chat framing — this is what the base wants
python engine_chat.py --ckpt checkpoints/base_62k.pt --raw -p "The capital of France is"

• Match the stage to the mode. base_62k continues text; use --raw. sft_7100 / dpo_3200 expect ChatML.
• Keep max_new modest (160 default). These checkpoints often don't emit a stop token on open-ended prompts, so a bigger cap just means a longer answer, not a better one.
• <bos> is required. Training prepends it. The engine does this; a hand-rolled prompt has to as well or the whole sequence sits one position off.
• Multi-turn re-prefills from position 0 each turn. Deliberate: RoPE positions, Hyper-Connection streams, Memory-Cache segment state, and the Engram window stay consistent with a fresh run.

Known behaviour

Measured at serving defaults, not guessed:

• Cross-turn collapse, about 1 conversation in 10. Latches onto a template from an earlier reply and repeats it. Three more in ten show partial collapse. New conversation clears it.
• Doom loops / near-repetition: 0/20 on a held-out prompt bank, serving defaults and greedy. distinct-3 = 0.999.
• Topic switching is fine when sampling (adherence 1.00), bad at greedy (0.47, stuck on the old topic 3/6). Another reason not to serve greedy.
• World knowledge is the real hole. France in Spain, water boils at 1.5 °C. A familiar string parked at the top of the prompt gets copied. An arbitrary code sitting in depth does not (see needle). Facts expected from the weights also do not.

Training recipe

Scripts are not in this repo yet (inference-only release). Configs and hyperparameters below are the spec; script names are for reference. The full training scripts ship with the comparison models around 26–28 Aug 2026 (see matched runs above).

Three stages in the order they ran: pretrain → SFT → DPO. dpo_3200 comes from sft_7100.

1. Pretrain → base_62k

train_blend_muon-50m.py --config byrne_100m_ultrax_mc.yaml --muon (yaml is in this repo).

steps	61,000, plus a Dolma-blend continuation; released at 62k
tokens	~2.0B (Chinchilla-ish for ~114M dense)
seq_len	1,024 (= 4 × mc_segment_len, so MC sees cross-segment memory)
batch	6 × grad_accum 4
optimizer	Muon (muon_lr_mult 10) + AdamW for aux params
lr	3e-4, warmup 2,000, cosine to min_lr_frac 0.1
weight decay	0.01, grad clip 1.0
grad checkpointing	on

Data blend (web_source takes leftover weight):

source	weight
FineWeb-Edu (sample-100BT)	0.40
Wikipedia (20231101.en)	0.15
DCLM-baseline-1.0	0.15
Cosmopedia-v2	0.10
FineMath-4+	0.10
NPset-2 Python-Edu	0.10

MC enters through a zero-init gate, so at init the model is the no-MC baseline. Trains to tanh(mc_gate) ≈ 0.561 across all 16 layers, which is why a broken cached path wrecks this one. PROVENANCE.md.

2. SFT → sft_7100

sft_multiturn.py --config byrne_100m_sft_local.yaml --tokenizer ./tokenizer.json

steps	12,000 planned, released at 7,100
seq_len	4,096 — this is where context grows from 1,024
batch	2 × grad_accum 4
lr	1e-5 (Muon ×10), warmup 100, min_lr_frac 0.1
grad checkpointing	on

Conversation-level mix (not token-level):

source	weight
UltraChat-200k (train_sft)	0.32
Tulu-3 personas instruction-following	0.20
SmolTalk2 everyday conversations (no_think)	0.13
SmolTalk2 systemchats-30k (no_think)	0.10
SmolTalk2 smol-rewrite (no_think)	0.07
SmolTalk2 smol-summarize (no_think)	0.06
SmolTalk2 explore-instruct-rewriting (no_think)	0.06
reasoning-corpus-4K	0.06

Easy to get wrong, silent when you do:

• max_position_embeddings is set to seq_len. Training at 4,096 is why the SFT checkpoint is a 4,096-context model. Base and DPO are 1,024 for the same reason.
• Conversations are packed to fill the window. Every sequence starts with <bos> then ChatML (<|im_start|>{role}\n{content}<|im_end|>\n). Packing is how the full position range gets exercised even though individual conversations are short.

3. DPO → dpo_3200

train_dpo.py --config byrne_100m_dpo_local.yaml --resume <sft> --ref-ckpt <sft>

dataset	UltraFeedback-binarized (train_prefs)
beta	0.1
lr	5e-7, warmup 50
steps	4,000 planned, released at 3,200
seq_len	1,024 (context drops back from SFT's 4,096)
batch	2 × grad_accum 8
reference	frozen copy of the SFT checkpoint

DPO at this lr barely moves the policy. Step 2,000 → 3,200, relative L2 of 2.8e-05. Greedy output identical on every prompt tested. UltraFeedback win-rate against the reference stuck at 0.438 → 0.438 even though implicit-reward accuracy hit 0.75. It separates preference pairs without visibly changing what the model writes. Benchmarks agree: DPO and SFT are within noise.

Resuming from a release

Released .pt files are weights + config, no optimizer, so a resume starts with a fresh optimizer. Each file's config is the one it was trained with. Engine and tools read it from the checkpoint, not from the yaml — the stages disagree about context length.

References

Memory Cache branch adapts:

@misc{behrouz2026memorycaching,
  title         = {Memory Caching: RNNs with Growing Memory},
  author        = {Ali Behrouz and Zeman Li and Yuan Deng and Peilin Zhong
                   and Meisam Razaviyayn and Vahab Mirrokni},
  year          = {2026},
  eprint        = {2602.24281},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  note          = {Google Research},
  url           = {https://arxiv.org/abs/2602.24281}
}

Citation

If you use this model, please cite:

@misc{byrne100multramc,
  title        = {Byrne-100M-Ultra-MC: A ~114M-parameter looped SpikeWhaleLM
                  with a Memory Cache branch},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-100M-Ultra-MC}},
  note         = {Quazim0t0/Byrne-100M-Ultra-MC}
}