Train Sparse Autoencoders Efficiently by Utilizing Features Correlation
Authors: name: Vadim Kurochkin; name: Yaroslav Aksenov; name: Daniil Laptev; name: Daniil Gavrilov; name: Nikita Balagansky
Published: 2025-05-28 11:41:11

Abstract: Sparse Autoencoders (SAEs) have demonstrated significant promise in
interpreting the hidden states of language models by decomposing them into
interpretable latent directions. However, training SAEs at scale remains
challenging, especially when large dictionary sizes are used. While decoders
can leverage sparse-aware kernels for efficiency, encoders still require
computationally intensive linear operations with large output dimensions. To
address this, we propose KronSAE, a novel architecture that factorizes the
latent representation via Kronecker product decomposition, drastically reducing
memory and computational overhead. Furthermore, we introduce mAND, a
differentiable activation function approximating the binary AND operation,
which improves interpretability and performance in our factorized framework.