anonboltzinterp/Boltz1-SAEs-L2-rec1

Boltz-1 SAEs — Pairformer trunk, recycle 1

TopK sparse autoencoders trained on Pairformer trunk activations from Boltz-1, taken at recycle iteration 1. Published for anonymous double-blind review; no authorship or affiliation is attached to this repository.

Architecture

SAE type	TopK, k = 256
Latent width	2048
Input width	384
Weight L2	3e-3
Preprocessing	training-set mean subtracted before encoding
Decoder	unit-normalised
Training steps	500,000
Seeds per layer	3
Base model	Boltz-1

Every run carries its full hyperparameter set in config.json.

Contents

75 runs: 25 layers x 3 seeds. The layers are the even indices 0-46, plus layer 47.

layer<L>/
  pairformer<L>_topk256_lat2048_demean_longtrain500000_l2_3e-3_alive_cross_seed.json
  pairformer<L>_topk256_lat2048_demean_longtrain500000_l2_3e-3_seed<S>/
      checkpoint_step_500000.pt
      config.json
      mean_vector.npy
      eval_step_500000.json
      stats.jsonl

<S> is 1, 2 or 3. The per-layer alive_cross_seed.json reports the latents alive across all three seeds.

Loading

from huggingface_hub import hf_hub_download

repo = "anonboltzinterp/Boltz1-SAEs-L2-rec1"
run = "layer24/pairformer24_topk256_lat2048_demean_longtrain500000_l2_3e-3_seed1"
for name in ("config.json", "mean_vector.npy", "checkpoint_step_500000.pt"):
    hf_hub_download(repo_id=repo, filename=f"{run}/{name}", local_dir="sae")

Subtract mean_vector.npy from the activations before encoding — these SAEs were trained on demeaned inputs, so skipping it silently degrades reconstruction.

Companion repositories

• anonboltzinterp/Boltz1-SAEs-L2-rec0 — the same trunk layers at recycle 0
• anonboltzinterp/Boltz1-SAEs-L2-Diffusion — diffusion-module SAEs