Qwen3.5-9B brain atlas activation census, OV-circuits, and capability fence (32layers
Qwen/Qwen3.5-9B Brain Atlas — The Editable One
This is not a benchmark. I ran the same GWIQ-style activation census on the 9B Qwen3.5. At this scale the model is a real editing target.
model: Qwen/Qwen3.5-9B
atlas type: activation census + Sub-Zero brain atlas + OV-circuit SVD + logit lens + compliance/behavior contrast
corpus: 8,965 prompts
layers: 32
attention layers: 3, 7, 11, 15, 19, 23, 27, 31
linear-attention / MLP layers: literally everything else
sacred (fully probed) layers: all 32
datasets: juiceb0xc0de/qwen3.5-9b-atlas
What this is
Qwen3.5-9B is a sparse-attention hybrid: full multi-head attention on one layer in four, linear-attention / MLP blocks everywhere else. It has 32 layers and eight full attention layers. I wanted to see whether the extra width at this depth would make the model more editable, or whether it would just make any landmines bigger.
Short answer: it is much safer. There is no catastrophic early direction. The worst surgical damage is small. And the late MLP gates form the clearest vocabulary-routing subspace I have seen at this scale.
What was run
- Activation census over 8,965 prompts.
- Per-layer feature taxonomy for every component.
- Per-head analysis on the 8 full attention layers.
- OV-circuit SVD on every head.
- Logit-lens pass to see which internal directions predict output tokens.
- Coactivation and code-analysis passes.
- Compliance/behavior contrast pass.
- Sub-Zero surgery pass with capability fence across
code,math,reasoning,factual, andmultilingual.
The shape of the thing
| Property | Value |
|---|---|
| Layers | 32 |
| Full attention layers | 8 |
| Heads per attention layer | 16 |
| KV heads | 4 |
| Head dim | 256 |
| d_model | 4,096 implied |
| Hybrid components | gate, up, mlp, linattn_qkv, linattn_z, linattn_out |
The attention cadence is one full multi-head block every four layers. The extra parameters go into width.
What the numbers suggest
Features are mostly partial-shared and broadly-shared
The taxonomy is weighted toward context-dependent directions rather than always-on or dead ones.
| Class | Count | Share |
|---|---|---|
partial_shared |
600,922 | 34.93% |
broadly_shared |
453,573 | 26.37% |
non_activated |
422,865 | 24.58% |
all_shared |
242,492 | 14.10% |
specific_* |
408 | <0.03% |
The specific_* tail is tiny as a fraction, but the raw counts are real. creative_writing (104), ml_ai (79), introspection (61), core_technical (60), and tool_use (52) are all present. The model is carving out domain-flavored directions without going fully hyperspecific.
Attention stays distributed and induction-heavy
OV-circuit averages:
| Metric | Value |
|---|---|
| OV spectral concentration | 0.040 |
| OV effective rank | 94.2 |
| QK spectral concentration | 0.152 |
| FC spectral concentration | 0.161 |
The heads are doing weighted high-dimensional computation, not sparse token lookup. Top induction scores are very high, with layer 27 head 13 at 1.12 and layer 31 head 13 at 1.08.
Layer 31 compresses its OV path: spectral conc 0.071, effective rank 64.2. The final attention layer is a distilled copy-paste engine parked at the exit.
The logit lens peaks at the very last layer
The strongest output-vocabulary predictors are late gate features, and the absolute peak is at the final layer:
| Rank | Layer | Component | Feature | F-stat |
|---|---|---|---|---|
| 1 | 31 | gate |
9995 | 721.2 |
| 2 | 23 | gate |
5067 | 677.6 |
| 3 | 28 | gate |
4316 | 673.5 |
| 4 | 23 | gate |
7732 | 664.4 |
| 5 | 28 | gate |
5085 | 658.9 |
The strongest vocabulary-routing direction is at the very last layer. That is a late-stage organization: the model waits until the exit to commit its strongest output-token predictor.
MLP gates are extremely selective
Code-analysis labels:
| Component | Selective | Entangled |
|---|---|---|
gate |
95.7% | 4.3% |
up |
92.6% | 7.4% |
mlp |
91.6% | 8.4% |
linattn_out |
84.6% | 15.4% |
linattn_qkv |
84.3% | 15.7% |
attn |
84.2% | 15.8% |
heads |
77.9% | 22.1% |
The gate path is the cleanest component in the whole model. Whatever the MLP gating is doing, it has focused jobs.
ml_ai and core_technical rise in coactivation
| Dominant bucket | Count | Share |
|---|---|---|
| uncategorized | 4,834 | 26.71% |
business |
2,684 | 14.83% |
ml_ai |
1,727 | 9.54% |
introspection |
1,612 | 8.91% |
humor |
1,420 | 7.85% |
core_technical |
1,368 | 7.56% |
Business is the top labeled bucket, but ml_ai and core_technical together make up 17.1% of all coactivating pairs. The model's internal feature space is organizing around technical/code content.
Compliance/behavior separation is absurdly sparse
Top compliance/behavior features:
| Layer | Component | Feature | F-stat | Delta |
|---|---|---|---|---|
| 7 | mlp |
11570 | 1,984.8 | 0.034 |
| 11 | q |
6706 | 1,975.9 | 0.335 |
| 21 | linattn_z |
2299 | 1,932.3 | -0.705 |
| 20 | linattn_qkv |
6284 | 1,837.2 | -0.715 |
| 13 | linattn_z |
2504 | 1,835.1 | -0.774 |
F-stats near 2,000 mean almost perfect linear separation between authentic and corporate-style prompts. These features are mostly quiet — high discriminative power, low base activation rate.
Surgical headroom is 95.8%
120 Sub-Zero tests per domain, 600 rows total. 115 pass per domain, a 95.8% pass rate. Average damage is tiny:
| Domain | Mean damage | Max damage |
|---|---|---|
code |
0.023 | 0.451 |
factual |
0.020 | 0.458 |
math |
0.017 | 0.330 |
multilingual |
0.019 | 0.429 |
reasoning |
0.016 | 0.284 |
The worst damage is 0.458 nats/token on layer 6 linattn_in_proj_qkv axis 0 in the factual domain. That is a real rejection, but the magnitude is small. The 9B has enough spare capacity that removing a compliance/behavior axis barely hurts capabilities.
The next worst direction is layer 22 gate_proj axis 0 on code at 0.323. After that, everything is under 0.30. The model is genuinely editable.
v heads carry the cleanest signal
Per-head F-stats:
| Component | Mean best F-stat | Max best F-stat |
|---|---|---|
v |
252.7 | 573.9 |
heads |
206.2 | 484.5 |
q |
175.7 | 459.5 |
k |
157.9 | 281.4 |
Value vectors store task-relevant content, query/key machinery routes.
Classifiers are near-perfect everywhere
Sub-Zero classifier accuracy is 0.9688–1.000 across all 32 layers. The style boundary is separable at every depth.
The stuff I deliberately skipped
- SSM recurrent-state internals and SAE features were not extracted for this run.
- Some layers in the Sub-Zero pass produced very few compliance SVs (1–10 per layer), so the tested axes are a sparse but principled sample rather than an exhaustive sweep.
Caveats
- Some patterns may be width effects rather than architecture effects.
- Sub-Zero damage numbers are for compliance/behavior candidate axes, not every load-bearing direction.
- I do not have a base-vs-instruct split. If this is the instruct checkpoint, some style findings are post-training artifacts.
- The very few compliance SVs per layer means the worst-axis discovery is sampling-based, not guaranteed to find the absolute worst direction.
Bottom line
Qwen3.5-9B is a practical editing target. Surgical headroom is 95.8%, and the late MLP gates form a clean vocabulary-routing subspace that peaks at the final layer. The attention is mostly induction, code is walled off, and the coactivation map leans toward ml_ai and core_technical.
At 9B the model is big enough to be redundant and small enough to be interpretable.