reviews: 
Comments: This work introduces a novel sparse autoencoder (SAE) architecture utilizing TopK filtering in place of the conventional L1 penalty approach (https://transformer-circuits.pub/2023/monosemantic-features), enabling direct specification of activation sparsity levels. The proposed TopK SAE is benchmarked against competing SAE architectures along the reconstruction-sparsity continuum, with latent interpretability assessed through multiple newly developed evaluation metrics. Key contributions include: TopK filtering addresses the shrinkage effects inherent in L1 penalties, yielding superior reconstruction quality at equivalent sparsity levels; the training procedure incorporates an effective initialization strategy and auxiliary loss term to mitigate dead latent formation by encouraging error reconstruction; comprehensive scaling law analysis is conducted across latent counts, k values, and upstream model sizes; and multiple interpretability assessments extend beyond standard MSE/sparsity metrics, including downstream loss evaluation via patching, latent-based probing for binary classification, autointerpretability techniques leveraging Neuron to Graph, and measurement of ablation sparsity effects on model logits.
Strengths: The paper presents an elegant and straightforward SAE architecture with accompanying training procedures that effectively extends to state-of-the-art models and substantial latent populations. Primary advantages relative to concurrent approaches include: direct hyperparameter control over sparsity without indirect tuning requirements; minimal training recipe complexity with few adjustable parameters; competitive or superior performance on the MSE/sparsity frontier with minimal dead latent emergence; extensive empirical validation through detailed comparisons with alternative SAE formulations; thoughtful treatment of autointerpretability method limitations in Section 4.3; and introduction of novel interpretability assessments, particularly regarding latent-based classification probing and downstream effect sparsity quantification.
Weaknesses: The application of individual latents as classifiers for coarse-grained binary tasks (such as sentiment classification) may be overly restrictive given recognized phenomena like feature splitting, potentially reducing the practical relevance of these findings. The TopK mechanism's requirement for exactly k active features per activation can problematically result in spurious feature engagement; this issue is addressed in concurrent work at https://www.alignmentforum.org/posts/Nkx6yWZNbAsfvic98/batchtopk-a-simple-improvement-for-topk-saes. The interpretability evaluation occasionally depends excessively on aggregated high-level statistics without sufficient guidance for interpretation—for example, the meaning and implications of reported 'downstream effect sparsity' values remain somewhat opaque. The scaling laws presentation would benefit from explicit comparison against Anthropic's concurrent scaling work available at https://transformer-circuits.pub/2024/scaling-monosemanticity/#scaling-scaling-laws. The paper exhibits minor inconsistencies in mathematical notation (e.g., alternating use of N and n for latent dimensionality) and contains occasional missing symbol definitions.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 3