FaithfulSAE: Towards Capturing Faithful Features with Sparse
  Autoencoders without External Dataset Dependencies
Authors: name: Seonglae Cho; name: Harryn Oh; name: Donghyun Lee; name: Luis Eduardo Rodrigues Vieira; name: Andrew Bermingham; name: Ziad El Sayed
Published: 2025-06-21 10:18:25

Abstract: Sparse Autoencoders (SAEs) have emerged as a promising solution for
decomposing large language model representations into interpretable features.
However, Paulo and Belrose (2025) have highlighted instability across different
initialization seeds, and Heap et al. (2025) have pointed out that SAEs may not
capture model-internal features. These problems likely stem from training SAEs
on external datasets - either collected from the Web or generated by another
model - which may contain out-of-distribution (OOD) data beyond the model's
generalisation capabilities. This can result in hallucinated SAE features,
which we term "Fake Features", that misrepresent the model's internal
activations. To address these issues, we propose FaithfulSAE, a method that
trains SAEs on the model's own synthetic dataset. Using FaithfulSAEs, we
demonstrate that training SAEs on less-OOD instruction datasets results in SAEs
being more stable across seeds. Notably, FaithfulSAEs outperform SAEs trained
on web-based datasets in the SAE probing task and exhibit a lower Fake Feature
Ratio in 5 out of 7 models. Overall, our approach eliminates the dependency on
external datasets, advancing interpretability by better capturing
model-internal features while highlighting the often neglected importance of
SAE training datasets.