Sparse Autoencoders Learn Monosemantic Features in Vision-Language
  Models
Authors: name: Mateusz Pach, name: Shyamgopal Karthik, name: Quentin Bouniot, name: Serge Belongie, name: Zeynep Akata

Sparse Autoencoders (SAEs) have recently been shown to enhance
interpretability and steerability in Large Language Models (LLMs). In this
work, we extend the application of SAEs to Vision-Language Models (VLMs), such
as CLIP, and introduce a comprehensive framework for evaluating monosemanticity
in vision representations. Our experimental results reveal that SAEs trained on
VLMs significantly enhance the monosemanticity of individual neurons while also
exhibiting hierarchical representations that align well with expert-defined
structures (e.g., iNaturalist taxonomy). Most notably, we demonstrate that
applying SAEs to intervene on a CLIP vision encoder, directly steer output from
multimodal LLMs (e.g., LLaVA) without any modifications to the underlying
model. These findings emphasize the practicality and efficacy of SAEs as an
unsupervised approach for enhancing both the interpretability and control of
VLMs.