reviews: 
Comments: The paper develops training methodology and scaling laws for a new sparse autoencoder (SAE) variant - TopK SAEs, which replaces the L1 penalty in the standard SAE (https://transformer-circuits.pub/2023/monosemantic-features) with a TopK filter, so that the desired sparsity of activations can be set directly. The new architecture is evaluated against other popular SAE variants along the reconstruction-sparsity frontier. The interpretability of the TopK SAE latents is also measured via several novel metrics.

In more detail:
- the TopK filter overcomes the shrinkage problem associated with the L1 penalty, allowing better reconstruction with the same sparsity;
- to minimize dead latents, the TopK training method uses a simple initialization strategy and an auxiliary loss that incentivizes the dead latents to reconstruct the error of the SAE.
- scaling laws for TopK are studied w.r.t. various parameters, such as the number of latents, the $k$ value, and the size of the model whose activations SAEs are trained on. 
- several evaluations beyond MSE/sparsity tradeoffs are performed:
	- downstream loss when SAE reconstructions are patched in
	- using individual SAE latents as probes for binary classification tasks 
	- simple autointerpretability based on Neuron to Graph (N2G)
	- evaluating how sparse the effect of ablating SAE latents is on logits of the model
Strengths: - the paper presents a new, simpler SAE architecture and training stack that demonstrably scales to frontier models and many SAE latents. The main advantages over concurrent work are:
	- the sparsity can be set directly, making it easier to do hyperparameter tuning;
	- the training recipe is also simple and with few knobs to tune;
	- yet results on the MSE/sparsity frontier are state of the art, and there are very few dead latents
- many of the findings are quite well supported by extensive comparisons with other SAE variants
- the discussion on the limitations of autointerpretability methods in 4.3. is illuminating
- the paper proposes some new and interesting interpretability evaluations, especially probing and sparsity of downstream effects
Weaknesses: - as the authors are already aware, using individual SAE latents as probes for very high-level binary tasks (such as sentiment) may be too restrictive, especially in light of phenomena such as feature splitting. This limits the significance of the results
- another practical limitation of the TopK architecture is that it forces every activation to use exactly $k$ active features. In practice (at least in my experience) this may lead to many unimportant features to activate. See concurrent work https://www.alignmentforum.org/posts/Nkx6yWZNbAsfvic98/batchtopk-a-simple-improvement-for-topk-saes
- the interpretability evaluation is at times too reliant on high-level aggregate metrics. For instance, how should we interpret the "sparsity of downstream effects" numbers?
- the presentation on scaling laws would benefit from a comparison with concurrent work by Anthropic: https://transformer-circuits.pub/2024/scaling-monosemanticity/#scaling-scaling-laws
- there is some inconsistent notation (e.g., using $N$ and $n$ for the number of latents) and some missing symbols from the text
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 3