Internal-mechanics atlas for the 1.5B parameter VibeThinker model — all 36 layers tested for activation structure, OV-circuit geometry, and Sub-Zero surgical headroom.

model: WeiboAI/VibeThinker-1.5B
atlas_type: activation census + Sub-Zero brain atlas
corpus: 9,523 diverse prompts
layers: 28
sacred_layers: 18-27

VibeThinker-1.5B Brain Atlas

This is an internal-mechanics atlas for the 1.5B parameter VibeThinker model. The goal was not to benchmark end-task accuracy, but to map what the network is actually doing with its parameters: where it computes, where it stores behaviorally relevant structure, and which late-layer directions are safe to touch.

What was run

• Activation census over 9,523 prompts spanning compliance, reasoning, code, math, multilingual, and refusal-style questions.
• Per-layer feature taxonomy for mlp, gate, up, and attention heads.
• OV-circuit spectral analysis per head (W_V @ W_O).
• Sub-Zero surgery pass on every layer, with a capability fence across code, math, reasoning, factual, and multilingual domains.
• Pipeline was run on a CPU-only environment.

Key geometry

What the numbers suggest

The model is not a lookup table

OV-circuit spectral concentration averages 0.049, with effective rank around 55. That is a distributed signature, not a sparse “copy-paste” attention pattern. Attention heads appear to be doing weighted computation across many directions, not memorizing specific token-to-token jumps.

Feature activation is broad

The feature taxonomy is dominated by partial_shared and broadly_shared classes, with a smaller non_activated tail and very few all_shared features. Most dimensions responded to many prompts rather than one hyper-specific trigger.

Late layers are load-bearing

Sub-Zero finds structured singular-value subspace only in layers 18–27, which is 36% of the network depth. The first half of the model looks like wide preprocessing; the second half does the structured transformation.

Surgical fragility is the main caveat

The capability fence keeps about 74.5% of tested axes, but the rejected ones hit hard:

• Layer 18 up_proj axis 0 does 0.81 damage to code generation.
• Layer 18 up_proj axis 0 also scores the highest math, reasoning, and multilingual damage.
• Several down_proj and gate_proj axes in the early sacred layers fail the fence.

Interpretation: the 1.5B late-layer subspace is doing a lot of work per direction. It has less redundancy than the larger variant, so removing a top singular value tends to break more than one capability at once.

Classifier stability dips in the middle

Sub-Zero classifier accuracy drops to 0.75–0.83 around layers 13–17, then recovers in the late sacred layers. That mid-network region is messier or more entangled than the clean late-layer representation.

Bottom line

VibeThinker-1.5B behaves like a compact reasoning model: distributed attention, broad-feature MLPs, and a deep-but-narrow sacred region where a small number of directions carry most of the task load. It is interpretable, but not easy to edit safely because its late layers are not highly redundant.

Cross-post from a small atlas project I ran this weekend on both VibeThinker sizes. The aim was to see what is actually happening inside the tensors, not just what comes out at the end.

VibeThinker 1.5B vs 3B: a brain-atlas comparison

I ran a full GWIQ-style atlas on both WeiboAI/VibeThinker-1.5B and WeiboAI/VibeThinker-3B: activation census, per-component feature taxonomy, OV-circuit SVD, and a Sub-Zero surgery pass with a capability fence across code / math / reasoning / factual / multilingual. Same 9,523-prompt corpus on both.

TL;DR

• Both models show the same distributed-computation signature: low spectral concentration, high effective rank, broad-feature MLPs.
• The sacred region starts at the same proportional depth on both: layer 18/28 on 1.5B, layer 23/36 on 3B. Both are about 36% of total depth.
• The 3B is more factorized and more surgical: 81.6% of Sub-Zero axes pass the capability fence, worst single-axis damage ~0.30.
• The 1.5B is more load-bearing per direction: only 74.5% pass the fence, worst single-axis damage ~0.81 (layer 18 up_proj).
• The 3B also has a much cleaner compliance/behavior subspace (88% peak CB-SV fraction vs 33% on 1.5B).

Headline numbers

What the atlas says about the architecture

The OV-circuit numbers are the strongest signal. Spectral concentration around 0.05 and effective rank around 55 means the attention heads are not memorized one-shot copy circuits. They are doing weighted, high-dimensional computation. Feature taxonomies confirm it: most dimensions are partial_shared or broadly_shared, not hyper-specific token detectors.

So the “VibeThinker is a small reasoning model” vibe from the chat logs matches the geometry: it looks like a model that reasons in the network rather than indexing a compressed knowledge store.

What the atlas says about scale

The sacred region scales with depth proportionally, not by absolute layer index. That is a nice consistency check — it appears the transition from preprocessing to structured transformation is a relative depth event, not a hand-tuned layer number.

Inside that sacred region, the 3B is much cleaner:

• higher classifier accuracy,
• more axes survive the capability fence,
• lower per-axis damage,
• more isolated style/behavior directions.

The 1.5B is not bad; it is just doing more work with fewer directions, so editing it is riskier.

The most interesting single directions

• 1.5B layer 18 up_proj axis 0 — rejects across all five capability domains, worst damage to code (0.81). This is the very first layer of the sacred region; the model leans hard on it right after the preprocessing stack.
• 3B layer 26 gate_proj axis 0 — rejects across all five domains, worst damage to factual (0.30). A universal late-layer direction, but with ~0.995 explained variance, so the damage is concentrated and interpretable.
• 3B down_proj axes — mostly pass the fence with ~0.97 explained variance intact, making them the safest surgical targets in the larger model.

Pipeline notes for anyone who wants to reproduce

• 1.5B run: CPU-only container, ~1k tok/s forward-pass throughput.
• 3B run: remote rented NVIDIA RTX 6000 Ada Generation with 48 GB VRAM, 128 CPU cores, 503 GB system RAM, CUDA 12.1, PyTorch 2.4.0. Measured device-to-device copy throughput was ~400 GB/s.
• The 3B atlas forward pass peaked around 12–14k tok/s depending on batch size and sequence length.
• Both runs used the same vendored Space code; the only GPU-side change was enabling the pre-compiled FlashAttention wheel.
• The real bottleneck at this corpus size was finalization and compression, not forward passes. If you scale the corpus, optimize the chunk finalizer before chasing bigger GPU batch sizes.
• FlashAttention was the main dependency win. A pre-compiled wheel for the target CUDA/GPU combo removes the “compile for an hour and pray” step.

Caveats

• This is an internal-mechanics atlas, not a downstream benchmark. We are not claiming 3B is “better at reasoning” than 1.5B on every task; we are claiming the 3B has more redundant, more editable late-layer structure.
• The compliance-behaviour axis used a separate pos/neg corpus that did not make it into these SQLite artifacts, so the style-axis numbers here come from the Sub-Zero SV decomposition, not a direct authentic-vs-corporate comparison.
• Both models were very new at the time of the run; numbers may shift if the weights or tokenizer are updated.

Bottom line

VibeThinker scales cleanly: same attention geometry, same proportional sacred depth, but the 3B has enough extra capacity to factor its late layers into more redundant, more isolable directions. If you are fine-tuning, quantizing, or merging these models, the 3B gives you more headroom to do it without breaking the model, and the atlas gives you a map of which directions are safe vs load-bearing.

https://huggingface.co/datasets/juiceb0xc0de/vibethinker-1.5b-atlas
https://huggingface.co/datasets/juiceb0xc0de/vibethinker-3b-atlas