ceselder/glm-5.3-olens-reconstructor

Activation reconstructors for GLM-5.3 -- text span -> block-51 activation direction (LoRA r512 + affine maps)

The text-to-activation half of the GLM-5.3 oracle lens: given a short span of tokens, predict the (mean-centred) block-51 residual activation of the frozen nvidia/GLM-5.3-NVFP4 at the position just before the span. Three LoRA reconstructors ("AR", the olens h2hpfx recipe ported from Qwen3.6-27B) and four closed-form affine maps. lora_ar_h51_sink16/ is the reward model of the lens RL in ceselder/glm-5.3-olens-lens (reward = clamp(cos(AR(bullet), h51 - mu), 0)^2). Held-out centred cosine 0.391 on 16-token spans (Qwen3.6-27B layer-42 AR: 0.400), 0.340 at the 4-16-token bullet positions. Part of the collection glm5.3-olens together with the lens repo. Card regenerated 2026-10-24 by scripts/hf_model_cards.py (ceselder/glm-olens).

Figures

Training loss and held-out centred cosine / FVE of lora_ar_h51_* (block 51, 1.19M rows) vs the Qwen3.6-27B layer-42 reconstructor (2.41M rows), both still rising at the end of one pass.

Per prefix length (b), lora_ar_h51_sink16 (blue) sits below the Kimi-K3 reconstructors and the Qwen3.6-27B reconstructor; (a) is the Kimi-K3 reconstructors' training curve.

The arm comparison figures of this repo (figures/ar_sink_compare.png, figures/affar_glm.png) are shown in section 4.

Vector PDFs sit next to the PNGs under figures/.

1. What is in this repo

folder	kind	input convention	held-out centred cos, bullet spans (last token)	centred cos, 16-token spans	FVE (16-token)	1-token read	8-token read	16-token read (171-row subset)
affine_k12_bare/	affine map, exactly 12 tokens	span alone	-	0.231	0.061	-	-	0.230 (k=12 rows)
affine_k12_sink/	affine map, exactly 12 tokens	[gMASK] + span	-	0.235	0.063	-	-	0.234 (k=12 rows)
affine_k16_bare/	affine map, exactly 16 tokens	span alone	-	0.237	0.064	-	-	0.238 (k=16 rows)
affine_k16_sink/	affine map, exactly 16 tokens	[gMASK] + span	-	0.241	0.066	-	-	0.247 (k=16 rows)
lora_ar_h51_bare/	LoRA r512 reconstructor	span = the bullet (4-16 tokens), no prefix	0.334	0.383	0.165	0.041	0.324	0.401
lora_ar_h51_sink/	LoRA r512 reconstructor	span = the bullet, prefix [gMASK] (ids 154822, 154824)	0.337	0.385	0.167	0.139	0.327	0.401
lora_ar_h51_sink16/	LoRA r512 reconstructor	span = the full 16-token sampled continuation (cut at EOS), prefix [gMASK]; all prefixes 1..16 regressed	0.340	0.391	0.172	0.138	0.330	0.408

LoRA folders: adapter_config.json + adapter_model.safetensors (11.5 GB, fp32 r512 adapter on blocks 0..51), value_head.pt (fp32 Linear(6144, 6144) state), centre_mean.pt ({"mu": ...}, the centre), ar_config.json (prefix ids, span source, read layer -- absent on the bare AR, which predates it: no prefix). Affine folders: value_head.pt (Linear(6144, 6144) state computing W^T (x - mu_x) + mu_y in the centred space), affine.pt (W, mu_x, mu_y, MU), mu.pt (the centre), meta.json (k, prefix ids, read position, ridge, full eval). All held-out numbers are on the first 2,048 rows (LoRA) / all 6,265 rows (affine) of the sft1b held-out file (news, never trained on); "16-token spans" = the full sampled continuation read at its 16th token (the Qwen AR's metric); "bullet spans" = the 4-16-token bullets read at their last token; "k-token read" = centred cosine at content position k over rows with >= k tokens (the 16-token read rests on 171 rows).

2. How they are wired

• Input = prefix ids + span token ids, no chat template, read at the span's LAST token. The sink folders prepend GLM-5.3's sequence start [gMASK]<sop> (ids 154822, 154824) so the first content token does not sit on position 0 -- GLM-5.3's attention-sink slot, where ||h|| is ~1,200 vs ~50 elsewhere; prefix positions are never read or scored. The bare AR takes the span alone. Feed exact token ids (for lens bullets: the sampled content ids after *, whitespace included).
• Model = frozen blocks 0..51 of nvidia/GLM-5.3-NVFP4 (NVFP4 routed experts dequantized per chunk inside grouped GEMMs, bf16 attention/shared experts) + for the LoRA folders an rsLoRA r 512 / alpha 45 adapter (scale alpha/sqrt(r) = 1.99) on the MLA projections (q_a, q_b, kv_a_with_mqa, kv_b, o) and the shared experts (gate/up/down) of those 52 blocks = 407 modules, ~2.9B fp32 parameters; the routed experts stay frozen.
• Output = value_head(h51_AR[last]), a vector in the centred space: compare it with h51_real - mu by cosine (mu = mean h51 over 20,000 training rows, shipped as centre_mean.pt / mu.pt). Only the direction is trained (cosine loss); FVE below uses the best per-row non-negative scale.
• Affine maps: AR_k(span) = W_k^T (x - mu_x) + mu_y with x = the frozen base's block-51 output at the last of exactly k sampled tokens (k = 12 or 16), bare or sink-prefixed -- a linear read of what the base already carries about the activation k tokens back. They need spans of exactly k tokens.

3. How they were trained and how much data they saw

3.1 LoRA reconstructors (lora_ar_h51_*)

• 1,193,984 rows, one pass, no repeats (1,166 steps x 1,024): 21 parquet files: sft1b (598,629 rows) + sft1b_more (648,487 rows, FineFineWeb indices 3417..3432); the AR stream used 1,193,984 of them, each once. Each row = (sampled continuation, h51 at the position before it) from the same top-half-entropy FineFineWeb positions as the lens SFT (see the lens card, section 3.1).
• Arms (everything else identical, same rows in the same order): bare -- span = the bullet (4-16 tokens), no prefix; sink -- span = the bullet, prefix [gMASK] (ids 154822, 154824); sink16 -- span = the full 16-token sampled continuation (cut at EOS), prefix [gMASK]; all prefixes 1..16 regressed.
• Loss: 1 - cos(value_head(h51_AR[pos]), h51 - mu), mean over the span's positions (every prefix 1..n regressed to the same target), then over the step's rows. Value head: Linear(6144, 6144) fp32, identity init, applied at every span position. lr 1e-4, warm-up 50, linear to 5% at step 1166; AdamW (0.9, 0.95), weight decay 0, no clipping, TF32 for the fp32 adapter GEMMs. From scratch (never warm-started), no whitening.
• 10.4 (bare) / 11.3-11.4 (sink arms) s/step on 4 x B200 (manual 4-GPU layer pipeline, gradient checkpointing, peak 100-118 GB/GPU); 3.9 h of 4 x B200 per arm (16 B200-GPU-hours each), plus 0.3 h for the cross-eval.
• Curve (bare arm): 0.296 centred cosine at 60k rows, 0.316 at 150k, 0.334 at 1.19M -- flat-ish from ~150k rows; the 16-token read was still rising (0.384 at 600k -> 0.391 at 1.19M, sink16).

3.2 Affine maps (affine_k{12,16}_{bare,sink})

• closed-form ridge from fp64 shift-centred sufficient statistics accumulated on GPU, one pass, no training loop; fitted on 1,242,978 (k = 12) / 1,242,958 (k = 16) rows = every qualifying training row of the same 21 files except the VAL slice (first 4,096 rows of sft1b/train_000 (never in the fit or the test)).
• Ridge lambda = 0.3 x mean diag(Cxx) (VAL ccos flat from 1e-5 to 0.3, falls above 1); target unit(h51 - MU), MU = mean h51 over 20,000 training rows (raw-centred target identical to 3 decimals). One bare + one prefixed forward per batch covers both k (causal read at positions 11 / 15, +2 with the prefix).
• ~1.2 h of 8 x B200 in total (forward passes only).

4. How good

4.1 LoRA reconstructors (held-out, 2,048 rows, same rows and positions for every arm)

checkpoint	centred cos, bullet spans	centred cos, 16-token spans	FVE (16-token)	raw-space cos (16-token)	other row (far)	self-cos
lora_ar_h51_bare	0.334	0.383	0.165	0.553	0.008	0.027
lora_ar_h51_sink	0.337	0.385	0.167	0.555	0.008	0.027
lora_ar_h51_sink16	0.340	0.391	0.172	0.558	0.008	0.026
Qwen3.6-27B layer-42 AR (anchor; 2.4M row-steps)	-	0.400	0.186	-	0.001	0.10

Centred cosine by content position k (bullet spans; the k-th read uses rows with >= k tokens: 2,048 ... 171 rows):

checkpoint	p1	p2	p3	p4	p5	p6	p7	p8	p9	p10	p11	p12	p13	p14	p15	p16
lora_ar_h51_bare	0.041	0.189	0.229	0.256	0.277	0.296	0.312	0.324	0.336	0.345	0.355	0.358	0.371	0.368	0.374	0.401
lora_ar_h51_sink	0.139	0.191	0.230	0.258	0.280	0.298	0.315	0.327	0.340	0.348	0.359	0.361	0.374	0.371	0.377	0.401
lora_ar_h51_sink16	0.138	0.190	0.229	0.257	0.281	0.300	0.316	0.330	0.343	0.352	0.362	0.366	0.379	0.375	0.382	0.408
Qwen3.6-27B AR	0.240	0.305	0.343	0.370	0.390	0.407	0.420	0.430	0.437	0.444	0.450	0.455	0.460	0.465	0.468	0.472

By bullet length (bare / sink / sink16): 4-7 tokens 0.285 / 0.287 / 0.288; 8-11 tokens 0.342 / 0.345 / 0.349; 12-16 tokens 0.371 / 0.374 / 0.379. The sink prefix only changes the first token (1-token read 0.041 -> 0.139); training on full 16-token spans adds +0.006 at 16 tokens. Of the apparent 0.334-vs-0.400 gap to Qwen, ~0.05 was the span length and the rest (0.391 vs 0.400) remains at half the rows with a still-rising curve.

4.2 Affine maps (held-out, all 6,265 rows)

map	centred cos	other row: neighbour / random	FVE	raw cos	LoRA-comparable rows	LoRA AR at the same k
affine_k12_bare	0.231	0.123 / 0.012	0.061	0.432	0.230 (n=755)	0.358
affine_k12_sink	0.235	0.125 / 0.013	0.063	0.432	0.234 (n=755)	0.358
affine_k16_bare	0.237	0.135 / 0.014	0.064	0.432	0.238 (n=171)	0.401
affine_k16_sink	0.241	0.138 / 0.014	0.066	0.432	0.247 (n=171)	0.401

The affine maps trail the LoRA reconstructor by 0.12-0.16 centred cosine at the same span length and reach half its FVE; the sink prefix adds +0.004. Qwen3.6-27B's span-alone affine map scores 0.264-0.270 (FVE 0.087 at k = 16), i.e. the frozen GLM base is almost as linearly readable as Qwen's, and the larger GLM-vs-Qwen gap sits in what the trained adapter extracts (frozen FP4 routed experts, span lengths, rows).

4.3 As an RL reward

Under the sink16 reconstructor the base model's own 16-token continuation scores 0.155-0.17 (reward) on the RL prompt pool; the SFT lens 0.124 at RL step 1, the RL lens 0.262 at step 100 (centred cosine 0.32 -> 0.50). The reward correlates with word overlap with the true next 16 tokens (+0.21) and with no surface feature (length -0.04, distinct-token ratio +0.03, digits +0.05). See the lens card.

5. How to use

Both repos ship olens_glm53_usage.py (self-contained reference implementation), glm53_stack/ (the five trainer-stack files it imports for path (b)) and the vLLM fork tarball.

What you need. The frozen base nvidia/GLM-5.3-NVFP4 (464 GB, 50 shards; copy it to local NVMe -- vLLM's loader is tensor-count bound and reads it at 0.3-0.5 GB/s from a network mount), B200-class GPUs (4 x B200 for the TP4 engine at ~160 GB/GPU, or 8 x B200; the 52-block reconstructor stack also fits 4 x B200), CUDA 13, torch 2.13, transformers 5.17 (>= 5.16.1, < 5.18), peft 0.21, and for path (a) **vllm==0.31.0** plus the **vllm-metamodels fork, branch glm53** (commit b26f02d: TP-partial layer-output all-reduce for the DeepSeek-V3.2/GLM-5.3 layout, embedding-overwrite fix-up for layer 0's fused embed+norm input, merge_lora with MLA absorbed-weight resync, logprobs_mode="processed_logprobs" support). Plain vLLM or upstream vllm-lens silently captures nothing on this architecture. The fork's source (branch glm53 @ b26f02d, tests and benchmarks stripped, MIT) ships here as **vllm_metamodels_glm53_b26f02d.tar.gz**: pip install --no-deps vllm_metamodels_glm53_b26f02d.tar.gz after pip install vllm==0.31.0 (it registers itself as a vllm.general_plugins entry point, so every TP worker loads the hooks); pip install git+https://github.com/ceselder/vllm-metamodels@glm53 once that branch is public. The engine runs the NVFP4 experts as **W4A16 (Marlin)** -- quant_override_w4a16() rewrites hf_quant_config.json in your local checkpoint copy -- because that is the weight path the trainer used (W4A4 clips the [gMASK]/<sop> sink tokens with static scales). Boot: ~11 min at TP4 from local NVMe, ~34 min at TP8.

Path (a): the lens with vLLM (activation -> bullet):

from olens_glm53_usage import GlmOlensEngine, punctuation_positions
eng = GlmOlensEngine("/local/nvidia__GLM-5.3-NVFP4", tp=4)           # boots the hooked W4A16 engine; asserts the fork's capabilities
ids, pos = punctuation_positions(eng.tok, "The Raiders, who have lost four straight, host the Chiefs on Sunday.")
[h51] = eng.capture_h51([ids], [pos])                                # block-51 activations of the UNMERGED base at the punctuation tokens, [n_pos, 6144]
eng.set_lens("./rl1b_rl1b_2_step_0100")                             # merge the lens LoRA into the served weights (~4 s; set_lens(None) restores the base)
for p, bullets in zip(pos, eng.verbalize(h51, n=4)):                 # 4 one-bullet samples per activation = the bench protocol
    print(repr(eng.tok.decode(ids[: p + 1])[-60:]), bullets)

Under the hood: SteeringVector(mode="replace", layer_indices=[EMBED_LAYER_INDEX], activations=54.1 * v/||v||) at the marker of the one-bullet prompt, SamplingParams(n=4, temperature=1, max_tokens=18, extra_args={"apply_steering_vectors": [...], "bullet_grammar": {"cap": 16, "fixed": False}}) with the grammar as a vLLM AdapterLogitsProcessor; verbalize asserts the fork's rows_replaced counter on every worker (the hook silently no-ops on vLLM API drift otherwise). python olens_glm53_usage.py vllm --model-dir ... --lens-dir ... --tp 4 runs a demo (three probe texts at their last token + a paragraph at every punctuation token).

Path (b): the reconstructor with the HF stack (text -> block-51 direction; no vLLM engine needed, vllm only for its Triton NVFP4 dequant kernel -- a torch fallback exists):

from olens_glm53_usage import HfReconstructor, span_tokens
rc = HfReconstructor("/local/nvidia__GLM-5.3-NVFP4", "./lora_ar_h51_sink16")   # frozen blocks 0..51 (NVFP4 loader, grouped-GEMM experts) + r512 LoRA + value head
[h51] = rc.capture_h51([rc.tok.encode("The capital of France is Paris. Paris", add_special_tokens=False)])   # base activation, adapter disabled
spans = [rc.tok.encode(s, add_special_tokens=False) for s in (" Paris, the capital of France", " a recipe for chocolate cake")]
cos, reward = rc.score(spans, h51.expand(2, -1))                       # centred cosine with h51 - mu, and the RL reward clamp(cos, 0)^2

reconstruct() prepends the folder's prefix ids (ar_config.json / meta.json; [gMASK]<sop> for the sink folders), reads the value head at the span's last token and returns the prediction in the centred space; compare directions only. Affine folders need spans of exactly k tokens. python olens_glm53_usage.py hf --model-dir ... --reconstructor-dir ... prints a small cosine table.

6. Caveats

• Trained on web text. Every lens/reconstructor row is a FineFineWeb window read at a top-half-entropy position (next-token entropy >= 1.3032 nats), plain token ids with no chat template. Chat and agent transcripts, code, and low-entropy positions are out of distribution; the WorkspaceBench numbers are the only measurement we have there.
• Research checkpoints. No safety evaluation; the readouts are sampled continuations and can be wrong or invented (hallucination rate 0.40-0.44 on the bench's hallucination family).
• Direction only. The cosine loss leaves the output norm untrained; FVE is reported with the best per-row non-negative scale.
• Position-0 sink. Bare spans start on GLM-5.3's attention-sink position (||h|| ~1,200 vs ~50): the bare AR reads a 1-token span at 0.04 (Qwen 0.24); the [gMASK]<sop> prefix lifts it to 0.14 and changes nothing beyond 2 tokens (+0.003 overall). Use a sink folder whenever spans can be short.
• Span length is the metric. The 'GLM 0.334 vs Qwen 0.400' gap was mostly 9.85-token bullets vs 16-token spans: at 16 tokens GLM reads 0.391 vs Qwen 0.400, at half the rows and still rising slowly (0.384 at 600k rows -> 0.391 at 1.19M). The curve flattens from ~150k rows: treat that as a capacity/data question (frozen FP4 routed experts, span lengths), not a limit.
• Affine maps read the window-shared topic component: their prediction correlates with a neighbouring position's target (0.12-0.14) as much as that target itself does (0.137); the LoRA AR's prediction is dominated by the row-specific component (0.025 on the same control).
• Neighbour-row control. ccos_other by rolling one row pairs rows of the same document window in these files (0.12 for 16-token spans in every arm); the window-safe control (ccos_other_far, roll by half the set) is 0.008-0.012 everywhere.
• Engine/trainer gap and engine non-repeatability as on the lens card: h51 captured by vLLM and by the HF stack differ at the 3-6% level.

7. Provenance

• Code: ceselder/glm-olens -- src/glm53_ar.py (LoRA AR trainer), src/glm53_ar_eval.py (cross-eval), src/glm53_affar.py (affine maps), src/glm53_hf.py + src/glm53_experts.py (trainer stack); NOTES.md is the full log.
• Lens checkpoints RL'd against lora_ar_h51_sink16: ceselder/glm-5.3-olens-lens. Qwen3.6-27B set for comparison: ceselder/olens-qwen3.6-27b-l42.
• Base model: nvidia/GLM-5.3-NVFP4.