allenai/tmax-27b Brain Atlas — Deep Hybrid, Absurdly Editable

Cross-post: I somehow atlas'd a 27B hybrid SSM-Mamba-transformer. Required custom kernels and a lot of swearing. But the numbers are wild.

model: allenai/tmax-27b
atlas type: activation census + Sub-Zero brain atlas + OV-circuit SVD
corpus: 8,965 prompts
layers: 64
attention layers: 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55, 59, 63
hybrid layers: literally 48 of them
sacred layers: all 64 flagged; densest Sub-Zero coverage in 16–32
datasets: juiceb0xc0de/tmax-27b-atlas

What this is

TMax-27B is the biggest member of the family, and the one that actually challenged the atlas pipeline. Doubling the depth to 64 layers while keeping the same 25%-attention cadence means 48 hybrid SSM/Mamba layers and only 16 full attention blocks. I had to custom-kernel parts of the backward-pass compute buildup just to fit this thing into the probe. So yeah, this one is special.

What was run

• Activation census over 8,965 prompts.
• Feature taxonomy, per-head analysis, OV-circuit SVD.
• Logit-lens pass.
• Coactivation and code-analysis.
• Sub-Zero surgery + capability fence across the usual five domains.

The deepest Sub-Zero pass is sparser than the first half of the network. I will flag where that matters.

The shape of the thing

What the numbers suggest

Feature taxonomy: more nuanced, not more hyperspecific

The 27B uses more partially and broadly responsive directions, fewer dead directions, and fewer universally-active directions than the smaller family members. It does not build a ton of hyperspecific detectors — the specific_* tail is still under 0.03%, with 1,162 rows out of 4,784,128. It makes the broad directions more nuanced. That is a clean scale signal.

Per-head structure gets more distributed

v head strength is the cleanest signal, but lower than you might expect from a smaller model. That suggests value storage is more spread across 64 layers of depth. q and k F-stats are flatter, meaning the routing machinery is consistent across many layers. Classic redundancy effect.

Attention geometry stays distributed

OV spectral concentration is 0.048 with effective rank ~91. The extra depth did not make attention sparse or concentrated; it kept the same distributed-computation regime. QK and FC concentration sit around 0.15–0.17, which is more distributed than the smaller sizes.

Logit-lens peaks are huge and spread deep

Top gate features now reach F-stats over 860:

• layer 55 gate 5244 — F=863.3
• layer 52 gate 2694 — F=860.2
• layer 55 gate 17355 — F=835.8
• layer 53 gate 3530 — F=817.5
• layer 36 gate 5655 — F=815.1

The output-vocabulary routing is distributed across layers 36–60, not crammed into the final few layers. That is a very different late-stage organization than a dense transformer.

Surgical headroom is absurd

• Classifier accuracy: 0.938–1.000, average 0.979.
• Capability fence: 100% of 207 tested axes pass (1,035 domain rows).
• Average damage: 0.0041.
• Worst damage: layer 63 gate_proj axis 0, 0.031 damage to multilingual.

That is an order of magnitude lower worst-case damage than the 2B and 9B. The 27B has so much redundancy that removing a top singular value barely harms any capability.

What Sub-Zero is actually measuring here

The Sub-Zero pass is not a generic "find all important directions" sweep. It specifically looks for directions that separate corporate style from authentic style, then uses DAS rotation and the capability fence to check whether removing those directions damages code, math, reasoning, factual, or multilingual ability. So the 207 tested axes are compliance/behavior candidate axes, not a census of every load-bearing direction in the model. The near-perfect pass rate means the 27B has plenty of room to edit compliance/behavior style without collateral damage, not that every direction in the model is harmless.

Sub-Zero SV coverage is front-loaded

SV counts are present across layers 0–31, densest around layers 16–32, and then almost entirely absent from layers 33–52 except for a handful of isolated points. This could mean the deepest layers are too entangled for clean top-SV directions, or the run was resource-capped on the tail, or the deepest layers use fine-grained distributed corrections. I lean toward a mix of the last two.

What is novel vs a dense transformer

1. 64 layers, only 16 attention blocks. A dense 27B transformer would usually be ~40 layers. This goes deeper with fewer attention blocks, pushing computation into the hybrid path.
2. Perfect capability fence. 100% of 207 compliance/behavior axes pass. The 27B is ridiculously editable for style/behavior edits.
3. gate vocabulary routing scales with depth. Late gate features remain the strongest logit-lens signal from layer 36 to layer 60. Output-vocabulary computation is spread across many late hybrid gates.
4. Sub-Zero coverage is concentrated in the first half of depth. Densest SV counts are around layers 16–32, and coverage is nearly absent from layers 33–52. In dense transformers the most sensitive directions are usually in the final 10–20%, so this is a different depth organization.

The stuff I deliberately skipped

Same family caveat: hybrid SSM layers that do not tokenize language were not probed because the activations would be noise. I am working on a way to measure whatever those hybrid layers are actually doing, but it is not in this atlas yet.

Caveats

• Tail Sub-Zero is sparse. Claims about layers 33–52 are based on census + per-head + logit-lens, not full SV surgery.
• Custom kernel work required. The atlas existing at all is a technical achievement; some artifacts may reflect pipeline adaptation rather than pure model behavior.
• No Qwen3.5-27B base comparison here.

Bottom line

TMax-27B is the most editable and most redundant model in the family. It keeps the same distributed-attention signature, scales gate-driven output routing across a much deeper late region, and has so much surgical headroom that almost no tested direction fails the capability fence. The only real caveat is that the very deepest Sub-Zero pass is incomplete, so the tail is better described by census and logit-lens behavior than by full surgery.