bsf_071226 — Load-Balanced Block-Sparse Featurizer, Qwen3-8B layer 24

BSF with dead-zone usage-bias selection (G=2048 blocks x b=64 dims, 131k code dims = 32x expansion at d=4096; per-token top-k=16 blocks by ||z_g|| + tau_g). The per-block selection bias tau is trained by a dead-zone controller (boost blocks firing < 1e-4, handicap > 50%, decay between; step/clamp calibrated in k-th-place-norm units) and is STORED IN THE CHECKPOINT (selection_bias buffer) — the same biased ranking runs at inference, exactly 16 blocks/token. Signed codes, pure MSE, denoising snr=0.1, ~100M tokens (FineFineWeb).

Results

• reconstruction FVE 0.848 (train-final; rank-1 SAE d65k k80 = 0.803)
• population: 1,927/2,048 blocks deployed-alive; 5 generalists >50% / 48 mid-band (5-50%) / ~270 specialists / ~1,700 rare tail held at ~1e-4; top block 9.3% of energy
• autointerp (delphi, Opus judge; n=1,436 = whole population): detection 0.809 / fuzzing 0.824 per-block mean; 841 blocks >= 0.8 detection
• usage-weighted (per-token) view: detection 0.665 / fuzzing 0.601 — per-block means favor rare crisp blocks; both views reported for honesty

Loading

Requires the nla repo's SAE/bsf_sae.py (the selection_bias buffer is new): BlockSparseSAE.from_pretrained("ae.pt") — group_sizes/k/act_scale/tau load from the state dict. Trained 2026-07-11, run bsf_stream_L24_G2048_b64_k16_biasdz (wandb nla-sae k1abto0r).