EleutherAI/sae-SmolLM2-1.7B-layer17-32x

SAE for SmolLM2-1.7B, layer 17

A standard (TopK) sparse autoencoder trained with sparsify on the residual stream of HuggingFaceTB/SmolLM2-1.7B at layer 17 (0-indexed, out of 24 layers — roughly 3/4 of the way through the model).

Training details

• Dataset: EleutherAI/SmolLM2-135M-10B, one full epoch (19,044 steps)
• Expansion factor: 32x (65,536 latents, d_in=2048)
• k: 32
• Optimizer: Adam, lr=1e-4 (see train_config.json for the full training config)
• Final FVU: 0.0172, dead latents: 0%

See train_config.json for the complete sparsify.TrainConfig used.

Usage

from sparsify import Sae

sae = Sae.load_from_hub("EleutherAI/sae-SmolLM2-1.7B-layer17-32x")

Automated interpretability

Explanations were generated and scored with delphi automated interpretability over a growing sample of latents (currently the first 900 latent indices, of which 883 cleared the ≥200 activating-example threshold and were scored).

Protocol

• Activations cached over 200M tokens of the training distribution (EleutherAI/SmolLM2-135M-10B, BOS-filtered). Whole-SAE at this budget: 0 dead latents, and 98.4% of latents fire ≥200× (64,456 / 65,536).
• Explainer & scorer: Llama-3.1-70B-Instruct (AWQ-INT4), run locally with vLLM.
• Sampling: 40 train / 50 test examples per latent, example context length 32, 10 activation quantiles.
• Scorers: detection and fuzzing. 95% confidence intervals are bootstrapped by resampling latents (2,000 resamples).

Metric	Detection	Fuzzing
Balanced accuracy	0.553 [0.545, 0.561]	0.587 [0.579, 0.596]
F1	0.615 [0.609, 0.620]	0.517 [0.506, 0.529]
Frequency-weighted F1	0.668 [0.657, 0.690]	0.535 [0.485, 0.708]
Precision	0.540 [0.534, 0.547]	0.624 [0.613, 0.635]
Recall	0.713 [0.703, 0.723]	0.441 [0.429, 0.454]

Scores are computed over a representative latent sample that includes rare features. Balanced accuracy sits near chance for the rarer latents, which are much harder to explain than high-frequency latents; frequency-weighted F1 upweights frequently-firing latents.

Companion variant: EleutherAI/sae-SmolLM2-1.7B-layer17-32x-embedskip. Both SAEs are evaluated on an identical token stream, so the comparison reflects the SAE, not sampling noise.