Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Paper • 2606.25234 • Published
YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
bsf_070926 — Block-Sparse Featurizer, Qwen3-8B layer 24
Vanilla BSF (Fel et al. 2026, arXiv:2606.25234): G=4096 blocks x b=64 dims (262,144 code dims, 64x expansion at d=4096), per-token top-k=16 blocks by code L2 norm, signed codes, pure MSE, denoising snr=0.1, ~100M tokens (FineFineWeb).
- FVE 0.865 (held-out, rl_shuf 20k acts)
- autointerp (delphi, Opus judge): detection 0.717 / fuzz 0.708, ~22% of scored blocks >= 0.8
- 362 live blocks; firing rate of live blocks p10/p50/p90 = 0.16% / 1.16% / 6.7% (~8 always-on generalists carry ~half the energy)
Load with SAE/bsf_sae.py::BlockSparseSAE.from_pretrained("ae.pt") from the nla repo.
Trained 2026-07-09, run bsf_stream_L24_G4096_b64_k16 (wandb nla-sae).
- Downloads last month
- 2
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support