Qwen3.8-27B-Scope (code-domain SAE suite)
Self-trained Top-K sparse autoencoders for Qwen/Qwen3.8-27B, all 64 decoder layers, coding-domain corpus. Built as the feature-selection teacher for DFlash feature-interface draft training.
- v2/ (main): 64 SAEs, 110M unique coding tokens each (50M base + 60M continuation)
- stage1/: 50M-token checkpoints (reference; better thinking-text recon: −8.5% vs −16% recon-τ)
- interface/: selected-feature lists + compact encoders (1×/2.5×/5× of 5120-dim budgets)
- scores/: per-feature S (code-specificity), A (acceptance-length contrast), U (causal draft utility)
Spec: width 40960 (8×5120), Top-k k=50, AuxK 512 @1/32, input scale √d/mean-norm
(stored in each checkpoint), decoder rows unit-norm. Checkpoint = {state: {b_pre, W_enc, b_enc, W_dec} (bf16), scale, k, width, tokens}.
Recon: x̂ = TopK50(ReLU((x·scale − b_pre)·W_enc + b_enc))·W_dec + b_pre) / scale... see
interface/ encoders for the selected-feature read-out path.
Key measurements: FVU 0.06–0.23 across layers; masking the 5-tap top-1000 code features costs 18.3% draft acceptance (random control ≈0); feature-level task specificity reliability 0.78–0.86.
Corpus: on-policy Qwen3.8 code generations + Magicoder-OSS/Evol + CodeAlpaca (single-pass).