Sparse autoencoders reveal selective remapping of visual concepts during adaptation
Authors: Hyesu Lim
Jinho Choi
Jaegul Choo
Steffen Schneider
Venue: ICLR 2025 Poster (2025)
Keywords: interpretability
vision-language models
sparse autoencoder
adaptation
Primary area: interpretability and explainable AI

Abstract: Adapting foundation models for specific purposes has become a standard approach to build machine learning systems for downstream applications. Yet, it is an open question which mechanisms take place during adaptation. Here we develop a new Sparse Autoencoder (SAE) for the CLIP vision transformer, named PatchSAE, to extract interpretable concepts at granular levels (e.g., shape, color, or semantics of an object) and their patch-wise spatial attributions. We explore how these concepts influence the model output in downstream image classification tasks and investigate how recent state-of-the-art prompt-based adaptation techniques change the association of model inputs to these concepts. While activations of concepts slightly change between adapted and non-adapted models, we find that the majority of gains on common adaptation tasks can be explained with the existing concepts already present in the non-adapted foundation model. This work provides a concrete framework to train and use SAEs for Vision Transformers and provides insights into explaining adaptation mechanisms.

Published as a conference paper at ICLR 2025
SPARSE AUTOENCODERS REVEAL SELECTIVE REMAP- PING OF VISUAL CONCEPTS DURING ADAPTATION
Hyesu Lim 1.2+
Jinho Choi
Jaegul Choo?
Steffen Schneider
Institute of Computational Biology, Computational Health Center, Helmholtz Munich *Munich Center for Machine Learning (MCML)
KKAIST AI
ABSTRACT
Adapting foundation models for specific purposes has become standard ap- proach to build machine learning systems for downstream applications. Yet, it is open question which mechanisms take place during adaptation. Here we de- velop a new Sparse Autoencoder (SAE) for the CLIP vision transformer, named PatchSAE, to extract interpretable concepts at granular levels (e.g., shape, color, or semantics of an object) and their patch-wise spatial attributions. We explore how these concepts influence the model output in downstream image classifi- cation tasks and investigate how recent state-of-the-aar prompt-based adaptation techniques change the association of model inputs these concepts. While acti- vations of concepts slightly change between adapted and non-adapted models, we find that the majority of gains on common adaptation tasks can explained with the existing concepts already present in the odnaadeped foundation model. This work provides a concrete framework to train and use SAEs for Vision Transform- ers and provides insights into explaining adaptation mechanisms.
Code and Demo: gi rithub... com/ dynaminnmmmmommmmm cal wwwiinnnneeeaaateeeeeeeeeeeeeeee
INTRODUCTION
Token
CL.S
patch
S4E recorstructed dje frazen tran
Patch lovel
CLIP
g eeen
Vision Encoder
GT dioce
(VIT) *
porch Commin
S: Christmas
Natta
SAE Iutern activation on LT
7: socks
CUP
dre-hetmasdin
IN
Image Kvvel
Prediction
CLP
(a) PatchSAE on CLIP ViT
(b) Patch- & image-level SAE latents
(c) Understanding adaptation mechanism
Figure 1: Overview. (a) We train PatchSAE frozen CLIP VIT with MSE loss and an LI sparsity regularizer using ImageNet (IN) ($3.1). (b) We analyze trained PatchSAE by interpreting patch- and image- level concepts of activated SAE latents ($3.2 & 3.3). (c) We then investigate the influence of SAE latents on the model behavior in classification tasks ($4.1) and explain how adaptation methods improve the downstream task performance ($4.2).
Foundation models excel at fast adaptation to new tasks and domains with limited extra training data (Radford et al., 2021; Caron et al., 2021; Touvron al., 2023). In the space of vision-language models, CLIP (Radford et al., 2021) became an important backbone for numerous applications (Liu al., 2024; Li al., 2023; Rombach al., 2022). The CLIP model consists of two transformer networks to encode text and image inputs. Various parameter-efficient adaptation techniques, such as adopting learnable tokens, have been proposed targeting either of the systems. While early works targeted the text encoder (Zhou et al., 2022b,a), more recently it was shown that joint adaptation of both the text and image encoders can further improve classification performance (K hatttt al., 2023a.b). Despite these advances in adaptation methods, it remains an open question of how repre- sentations a foundation model change during adaptation.
"This work was done during a research visit at Helmholtz Munich. "Correppnndecc: steffen. wwwffennschniiderdee meuiche

---

Published as a conference paper at ICLR 2025
Recently, Sparse Autoencoders (SAEs; Bricken et al., 2023; Cunningham et al., 2023) have gained emerging attention as a tool of mechanistic interpretability (Templeton, 2024; Yun et al., 2021) after showing its effectiivness in widely used LLMs (Bricken et al., 2023; Lieberum et al., 2024). SAEs map dense model representations, which are difficult to interpret because multiple unrelated concepts entangled (polysemantic), sparse and interpretable (monosemantic) concepts.
In this work, we develop PatchSAE, a new SAE model for the CLIP Vision Transformer (ViT) (Fig. (a)). PatchSAE extracts interpretable concepts and their patch-wise spatial attributions, pro- viding localized understanding of multiple visual concepts that can be simultaneously captured from different regions of a single image (Fig. I(b)). Before analyzing model behaviors through the SAE, we first validate PatchSAE as an interpretability tool. Our PatchSAE identifies diverse interpretable concepts, provides localized interpretation, and performs well across multiple datasets ($3.3). We then explore the influence of interpretable concepts on the final output of the CLIP model through classification tasks.
We use our PatchSAE to shed light the internal mechanisms of foundation models during adapta- tion tasks. Recent state-of-the-ard adaptation methods for CLIP (Zhou et al., 2022a,b; Khattak et al., 2023a,b) add trainable, dataset specific tokens for adaptation, akin to system prompt in LLMs. Through extensive analysis, we reveal a wide range of interpretable concepts of CLIP, including simple visual patterns to high-level semantics, employing (Fig. (b)). We also localize the recognized concepts through token-wiss wise inspections of SAE la- tent activations, while extending it to image- class-, and task-wise understandings. Furthermore, we demonstrate that the SAE latents have a crucial impact on the model prediction in classification tasks through ablation studies. Lastly, we show evidence that prompt-based adaptation gains the performance improvement by tailoring the mapping between recognized concepts and the learned task classes (Fig. (c)).
PatchSAE as an interpretability tool
Our paper proceeds as follows: First, we introduce PatchSAE, sparse autoencoder which allows to discover concepts for each token within a triis-ttrnsformer with spatial attributions ($3). We train this model on CLIP, and show the interpretability and generalizability of ImageNet-scale trained SAE across domain-shifted and finer-grained benchmark datasets. We explore the CLIP behavior in classification tasks and how adding learnable prompts changes the model behavior using PatchSAE ($4). In the end, we discuss conclusions and broader implications ($5).
2 RELATED WORK
Sparse autoencoders for mechanistic interpretability. Mechanistic interpretability (Elhage et al., 2021) aims to interpret how neural networks infer their outputs. To achieve this, it is nat- ural to seek a deeper understanding of which feature (concept) is recognized by each neuron in the neural network (Olah et al., 2020). For instance, the logit lens (Nostalgebraist, 2020) approach attempts to understand the intermediate layer output by mapping it into the final classification or decoding layer. However, understanding neurons in human interpretable form is challenging due to the polysemantic (Elhage et al., 2022) nature of neurons, where each neuron activates for multiple unrelated concepts. This property is attributed to superposition (Elhage al., 2022), where neural networks represent more features than the number of dimensions.
To overcome the superposition phenomenon in neural network interpretation, sparse autoencoders (SAEs) (Sharkey et al., 2022; Bricken et al., 2023) have recently gained significant attention. SAEs decompose model activations into sparse latent space, representing them as dictionary of high dimensional vectors. Several studies (Yun et al., 2021; Cunningham et al., 2023) have applied S to language models. Using SAEs, Templeton (2024) discovered bias- and safety-related features in large language models (L..Ms). demonstrating that these features can be steered to alter the behavior of L.LMs. Recent research extended the application of SAES to vision-language models, such as CLIP (Radford et al., 2021). Fry (2024) and Daujotas (2024a) extracted interpretable concepts from the vision encoder of CLIP and Daujotas (2024b) utilized these features to edit image generation in diffusion model. Rao et al. (2024) named the SAE concepts using word embeddings from the CLIP text encoder and used them for concept bottleneck model.
Distinet from previous works, we propose use patch-level image tokens for SAEs which allows intuitive and localized understanding of SAE latents and easily transformable to higher (image- / class- / dataset-) level of analysis. Furthermore, we adopt SAE latents masking method examine
2

---

Published as a conference paper at ICLR 2025
the relationship between interpretable concepts and downstream task-solving ability. For the first time, this allows precise investigation of how foundation models behave during adaptation, and how concepts re-used across datasets.
Prompt-based adaptation. Adapting vision-language foundation models like CLIP through fine- tuning requires large datasets and significant computation resources. In addition, the generaliza- tion ability of the model may be compromised after fine-tuning. As an alternative, prompt-based adaptation has recently emerged, training only few learnable tokens while keeping the weight of pre-trained models fixed. CoOp (Zhou et al., 2022b) proposed adapting CLIP in few-shot learn- ing setting by optimizing learnable tokens in the language branch, while Bahng et al. (2022) applied prompt adaptation to the vision branch. MaPI.e ( (Khattak et al., 2023a) improved few-shot adaptation performance by jointly adding learnable tokens to both the vision and language branches and con- sidered as a base structure for more recent multimodal prompt adaptation methods (Khattak et al., 2023b). Although these studies demonstrate that prompt learning effective for adapting CLIP, there is still a lack of research focusing on how and why prompt learning enables such adaptation.
Our work focuses on exploring the CLIP image encoder using an SAE on a vision transformer. We choose MaPLe (Khattak et al., 2023a) as a representative structure of multimodal prompt adaptation and investigate the internal work of adaptation methods.
3 PATCHSAE: SPATIAL ATTRIBUTION OF CONCEPTS IN VLMS
In this section, we revisit the basic concept of sparse SAE and introduce our new PatchSAE model in $3.1. We then discuss how we discover interpretable SAE latent directions in $3.2. Our analysis includes computing summary statistics of SAE latents, that indicate how often and how strongly one latent is activated, detecting model-recognized concepts by inspecting reference images with high SAE latent activations, and spatially localizing SAE concepts in the image space (Fig. 2).
3.1 PATCHSAE ARCHITECTURE AND TRAINING OBJECTIVES
SAEs typically consist of single linear layer encoder, followed by a ReLU non-linear activation function, and a single linear layer decoder (Bricken et al., 2023; Cunningham et al., 2023). To train an SAE on CLIP Vision Transformer (ViT), we hook intermediate layer outputs from the pre- trained CLIP ViT and use them as self-supervised training data. We leverage all tokens including class (CLS) and image tokens from the residual stream output (Fig. {C)) of an attention block and feed them to the SAE. Formally, we take ViT hook layer output as an SAE input z, multiply it with the encoder layer weight WE E decoder layer weight WD E X The column (or row) vectors of the encoder (or decoder), dSAE vectors size of ddd,, correspond to the candidate concepts, i.e., SAE latent directions. We call the output vector of the activation layer (size of (sAE ) as SAE latent activations. For simplicity, we use for the encoder and g for the decoder:
pass to the ReLU activation é, then multiply with the
(1) To train the SAF, we minimize the mean squared error (MSE) as a reconstruction objective, and use I.1 regularization on the SAE latent activations to learn sparse concept vectors (Fig. 1(a)):
z ViT(x) lhook layer),
SAE(Z) (g o do f)(z) = W (.....
LSAF = [[SAB(z) 212 + Ar @ ((()))
An ideal SAE encoder maps the dense model representations into multiple monosemantic concepts and an ideal decoder reconstructs the original vector by linearly combining these distinct concepts.
Training details. We use CLIP model with an image encoder of ViT-B/16, which results in 14 14x/ 14 image tokens and CLS token as input. It has 12 attention layers with model dimension dvæ of 768. For PatchSAE, we set the expansion factor to 64 that multiplies with dyir. which results in SAE latent dimension ASAF of 49,152. We take the ViT output from the residual stream of the second last attention layer (i.e., 11-th layer output). Note that we use all image tokens, so the input and output of SAE have a size of (number of samples, token length, model dimension dvrr). We average the training loss across all individual tokens. We evaluate the trained SAE for reconstruction ability and the sparsity. We report the training performance of different configurations ($A.1) and show variations for training the SAE different layers (SA.2) in the Appendix.
'The residual stream is the sum of the attention block's 's output and its input, see Fig. 9c). We use hias for linear layers and centralize z, Le. SAE(z bac))
3

---

Published as a conference paper at ICLR 2025
licokes CLS
lickked Image tcben
CLS
SA2 image tolion houss
m/ openn
IN
FALL of cal Surredy
Sare Baline
Mean poulie SAE acoeation
Swifg
whir brew
Dng bock
colira
Irage-vlte
(a) Raference images of SAE latant directions
Versatile inspections of SAE latant activations
(c) Localize SAE latent activations
Figure 2: Analyzing SAE latents. (a) We take an average over patch-level activations for image and keep top-k images having the highest mean activation the reference images for each SAE latent. (b) From patch-level latent activations, we investigate localized concepts. Furthermore, we represent image-, class-, and datasct-wixg wise concepts by aggregating the patch-level activations. (c) For a certain concept, we provide the spatial attribution of the concept by visualizing the patch-level activations as segmentation mask.
3.2
ANALYSIS METHOD AND EVALUATION SETUP
After training, validate PatchSAE model by interpreting the activated SAE latents from input examples. We first discover the candidate concepts the SAE latent directions - by collecting reference images that maximally activate each SAE latent and compute their summary statisties. Then, we investigate the SAE latent activation to see how much the given input aligns with the corresponding SAE latent directions. The token-wise (i.e., patch-wise) investigation allows spatially localized understandings of an image. To derive global interpretation of the image for an SAE latent, we aggregate the patch-level activations into an image-level activation by counting the number of patches activating the corresponding latent. Similarly, we extend it to class- and dataset-level analysis (Eq. 5).
Reference images for SAE latents. As a first step of discovering the interpretable concepts that SAE represents, we consider set of images that maximally activate each SAE latent as reference images. Given trained SAE and dataset, we keep the top-k images having the highest SAE latent activation value for each latent dimension, i.e. we have & x dSAE reference images in total. Here, image- activations to select top-k images. Fig. 2(a) illustrates the procedure.
Summary statisties SAE latents. To inform the general trend of an SAE latent, we compute sum- mary statistics of the activation distribution (Bricken et al., 2023). We use the activated frequency and the mean activation values over a subset of training images. Using the class label informa- tion from a classification benchmark dataset such ImageNet, compute the label entropy (Fry, 2024) and standard deviation from the reference images. Specifically, we compute and interpret the statisties follows:
Sparsity (activated frequency) represents how frequently this latent is activated. We count the number of images having positive SAE latent activations and divide by the total number of seen images. An SAE latent with high frequency either represents a common concept or is an uninterpretable (noisy) latent.
Mean activation value is computed by averaging the positive activation value among the ac- tivated samples. The mean activation value implies the SAE model's confidence. A latent direction is more likely to represent meaningful concept if it has high mean activation value.
Label entropy measures how many unique labels activate the latent. Precisely, we compute the probability of label based on its activation value and compute the entropy as sum,
prob Eeec sum,
entropy = Σ
(prob, probrob.). cEC
where sum, is the summed activation values for label E C. The entropy being equal to zero indicates that all reference images have exactly the same label. Higher entropy indicates that more labels contribute to the latent's activation.
Label standard deviation. In ImageNet, class labels are organized a hierarchical structure based on WordNet's s semantic relationships (Deng et al., 2009; Miller, 1995). We leverage this label structure and use the label standard deviation of reference images as a clue for the semantic granularity besides the label entropy when exploring the latents. We discuss more in 1.
4

---

Published as a conference paper at ICLR 2025
Label Entroey
(a)
(b)
Log teean Actiration
1.5
[a] uninte uninte-pretabe pretable
(b) bext of "MILK"
(c)
Log Actisated Frequency
000K style
(4) Granny apple
Figure 3: SAE latents statistics and reference images. Left: Seatter plot of SAE latent statistics (æ-axis: 10g10 of activated frequency, y-axis: log10 of mean activation) colored by label entropy. Right: Reference images from Imagenet of four SAE latents different regions.
Discovering active SAE latents in diverse levels. Using the reference images as an interpretable proxy for SAE latent directions and the SAE latent activation an input as the similarity with the corresponding latent, we examine which concepts are how strongly active for the input. As depicted in Fig. 2(b) and (c), the patch-level SAE latent activations inform the recognized concepts from each patch. For example, for the input patch containing the "dog's nose", the SAE latent having the reference images of "dog's 's nose" is active.
To obtain global concept from the image, transform the patch-level activations into image- level activation. In specific terms, we first binarize the SAE latent activation of the i-th image at j-th token for the s-th SAE latent [ = ((f())< a threshold. We call the SAE latent above the threshold as an active latent, otherwise an inactive latent. Then we count the number of patches that activate the latent and consider it the image- level activation as[s]. Similarly, we obtain the class- (a.(s) and dataset-level (ap activation by incorporating [s] for the images with the same class or dataset, respectively. Formally, we represent this as below:
E using small positive number 7 as
[s] = 1(barls) > T), where 1 3 S dSAE
(4)
n
Σ [s]
[+]
Caiss) SEI.
ap(s]
Σ [s(s)-
(5)
iED
From the class- or dataset-level activations, we discover the shared concept within the group. We use these group-wise active SAE latents to analyze the relationship between the interpretable concepts and the model behavior in $4.1.
Localizing patch-level SAE latent activations. PatchSAE AE allows localizing an active latent in the image space. We treat the patch-level latent activation as soft segmentation mask. Precisely, given an image x; and an SAE latent index 8, we multiply each patch Xi,i with the corresponding latent activation value [s] for visualization. For example in Fig. 2(c), we highlight "yellow flowers" or "dogs with black, white, and brown colors" concepts from the input image. Separating the patches relevant to the targeting latent from the input and reference images shows a clearer view of the concept
3.3 PATCHSAE DISCOVERS SPATIALLY DISTINCT CONCEPTS IN CLIP
More examples in the following sections are provided in the interactive demo (Fig. 12).
PatchSAF identifies diverse interpretable concepts. As depicted in Fig. 3, we explore the SAE latents guided by the statisties. We observe two big clusters of rarely activated with low activation value (bottom left) and frequently activated with high activation (top right), and near the center. Although the statistics do not ensure the interpretability of the latents, we find several interesting patterns. Many latents from the bottom left region with high label entropy
small cluster
We recommend trying the on/off segmentation mask option in the interactive demo.
5

---

Published as a conference paper at ICLR 2025
(a)
(a)
cm
(7)
(b)
002
(b)
(c)
Input image
Image-level SAE latent activations
Reference images
Segmentation mask
Figure 4: Localizing SAE latent activations under covariate shift. Given two input images of class hen, we show image-level aggregated SAE latent activations (z-axis: SAE latents index y-axis: image-level activation), reference images from ImageNet, and segmentation masks for each input are shown. Among top 10 latents for each input, we pick three interpretable indices where (a) and (b) represent different domains (image style or background) and (c) shows the shared concept.
uninterpretable (Fig. 3())). We find more interpretable latents from the second large cluster (top right region). For interpretable latents, lower entropy (Fig. 3(d)) indicates more distinctive semantics such as specific class, while the higher entropy latent (Fig. 3(c)) represents the shared style of reference images. We also observe multimodal latents that activate when certain text appears the image. For example, Fig. 3(b) latent detects the text MILK. More examples are in Fig. 13.
PatchSAE provides spatial attribution of concepts. As case study, we compare pair of images having the same class label but different domains (covariate shift) in Fig. 4. The commonly activat- ing SAE latent from both images shows the shared concept hen (Fig. 4(c)). From both images, the hen latent is activated by the relevant regions. Exclusively activating latents (Fig. 4(c) & (d) repre- sent discrete concepts such as yellow or net. The segmentation map highlights the contributing patches for the concepts.
PatchSAE generalizes across multiple datasets. Although we train our PatchSAE model only using ImageNet training data, we find that the interpretability of SAE latents is transferrable to different datasets. We show that an SAE latent retrieves consistent concept from different datasets if such concept exists in the dataset. Otherwise, the mean activation value is low and/or the retrieved images are uninterpretable ($A.3).. Fig. 10 shows reference images from ImageNet and four fine- grained datasets for two SAE latents and Fig. 14 shows reference images of top-1 task-wise latent.
4 ANALYZING CLIP BEHAVIOR VIA PATCHSAE
In this section, seek the relationship between SAE latents and model behaviors under classifi- cation tasks. By replacing the model's intermediate layer output with SAE reconstructed one and ablating the latents used for the SAE decoder, find that SAE latents contain class discrimina- tive information ($4.1). 1). Comparing the behaviors of CLIP models before and after adaptation on downstream tasks, we explain the major performance gain stems from adding new mappings SAE latents to downstream task classes, rather than firing additional class discriminative concepts ($4.2).
4.1 IMPACT OF SAE LATENTS ON CLASSIFICATION
We explore the influence of SAE latents on the final model prediction in classification tasks. We replace the intermediate layer representation of CLIP image encoder with the SAE reconstructed output to steer the model output' (Templeton, 2024) by selectively using subset of SAE concepts. Then we compute cosine similarity between text and image encoder outputs for the classification task. Fig, 5(a) summarizes the procedure.
We add reconstruction score when replacing the intermediate layer output with SAR reconstructed one following Templeton (2024).

---

Published as a conference paper at ICLR 2025
(+)
CLIP
MaPLe
Prediction
CLS token
Prompt contect taken
input token
Frozen prompt Token
50
Cosine sim.
Accuracy (9))
40
30
Text Encoder
Vision Encoder
20
random
10
ditasat level
Top-k masking
caas-46ou).
Class
1000
off tap-k
on top-k
(A)
Suaread royo
Imagetier
Food/01
Gakendaa
FamSAT
Catechtty
Fiwwre
Figure 5: Top-A SAE latent masking. (a) Top-k SAE latent masking implementation for CLIP and MaPLe. MaPLe adds learnable prompt tokens upon CLIP. CLIP (zero-shot) classification accuracy on ImageNet-Il for different SAE latent masking. Top-k selection based on class-level latent activations crucially affects accuracy while random or dataset-level based selections show marginal or no impact. (c) Example images and top-k class-level masking experiment for 11 tasks.
4.1.1 ANALYSIS METHOD AND EXPERIMENT SETUP
Zero-shot classification. We conduct ImageNet-IK IK (Deng et al., 2009) zero-shot classification using OpenAI CLIP VIT-B/IG (SReproducibiiity with an ensemble of 80 OpenAl ImageNet tem- plates (Radford et al., 2021) to compute text features for each class and conduct classification by computing cosine similarity between image features out text features (Fig. 5(a)).
Top-k SAE latent masking. To select the subset of SAE latents that are used for the linear combi- nation in the SAE decoder, we search for class-wisc representative concepts. We utilize class-level activations (Eq. 5). In short, we aggregate SAE latent activations from group of images having the same class label using the training split of each downstream task dataset and find the top-k most frequently activated latents per class. We then control the active latents via masking the SAE latent activation vector before feeding it into the decoder. We replace the original model representations with the reconstructed ones by the masked SAE. We compare the classification accuracies by vary- ing the mask. For example, "on top-k" refers to using the mask of a one-hot vector where only the top-k latent indices are (active) and the others are (inactive). Contrastingly. "off top-k" refers to using a mask filled with Is except for top-k indices being Os. As ablation, we provide comparisons on using the same number of randomly selected indices and dataset-level representative latents (i.e., frequently activated across all classes within the same dataset).
4.1.2 KEY FINDINGS
SAE latents have class discriminative information. The results for the top-k SAE latent masking experiments shown in Fig. 5(b.c). Using all SAE latents (on all: identity mask) recovers the original classification performance (i.c., using the original model representation without replacing with the SAE output) with small reconstruction errors (64.82% for the identity mask and 68.25 68.25% for the original). Using the all-zero mask (off all) removes all relevant information, and hence results in the accuracy of 0.1% Ablating randomly selected or task-wise latents do not show significant affect to the classification accuracy until we use sufficient number of latents. On the other hand, ablating the per-class top activating latents shows crucial impact on classification performance. We observe clear performance improvement or degradation in accordance with the increased or decreased number of active SAE latents, respectively. The results of this analysis show that some SAE latents contain rich information that is critical for class discrimination. Moreover, the search for such latents can narrowed down to the top activating SAE latents that are frequently activated across inputs with the same ground-truth class.
4.2 UNDERSTANDING ADAPTATION MECHANISMS
To understand how models are adapted to downstream tasks based on our findings in $4.1, aim to address the following questions: Does adaptation make models more prominently activate class

---

Published as a conference paper at ICLR 2025
Table 1: Comparison class-level SAE latents. The first six columns show the average number of class- level latents three groups: high. high-to-low, low-to-high groups per class. Details about three groups are provided in Figure 17. Gray colored for values below 0.1. The next four columns show the accuracy (the same results as dashed lines in Fig. 5(c)) and the final last column shows the performance improvement rate A (S 4.2.1). Rows are sorted by A.
SAE latents count (average)
Accuracy (%)
high
high-to-low
low-to-high
CLIP
MaPLe
( f
Dataset
base
novel
base
novel
base
novel
base
novel
base
novel
base
StanfordCar
11.18
11.10
0.00
0.12
0.12
0.16
53.45
66,46
56.16
60.63
5.82
FGVC Aireraft
10.10
10.57
0.31
0.32
0.00
0.03
21.72
28.09
31.19
30.07
12.10
ImageNet
11.05
10.82
0.02
0.02
0.12
0.22
69.88
67.23
73.80
68.24
13.01
Food101
11.24
11.58
0.00
0.00
0.00
0.00
84.85
86.65
87.26
89.14
15.91
SUN397
11.20
11.31
0.00
0.01
0.02
0.02
68.74
72.27
77.89
76.26
29.27
UCF101
10.90
11.10
0.01
0.01
0.07
0.06
66.55
68.80
81.13
71.40
43.57
OxfordPet
11.11
11.73
0.00
0.00
0.24
0.16
85.41
95.26
92.04
94.99
45.44
DTD
8.85
8.96
0.32
0.36
1.68
2.02
53.12
53,44
75.17
55.62
47.03
EuroSAT
6.90
8.70
0.50
1.00
4.10
2.30
42.61
62.42
73.07
55.77
53.08
Caltech101
11.25
11.32
0.02
0.00
0.00
0.00
92.57
93.47
97.49
94.29
66.22
Flowers
10.81
11.47
0.01
0.00
0.01
0.01
56.32
69.26
88.07
68.41
72.69
discriminative concepts? Or, do they define new mappings between the used concepts and the down- stream task classes? The former question refers to the model improving its perception ability by adaptation (i.e., adapted models capture additional class discriminative latents). The latter implies that both models recognize similar concepts, but the adaptation adds new connections between the activated concepts and the downstream task classes (i.e., adaptation uses concepts that are not closely related to certain classes previously, as class discriminative information).
To answer the questions, we investigate whether the class discriminative SAE latents of zero-shot and adapted models overlap not. We observe a large overlap between before and after adaptation, which indicates that similar concepts are recognized by the two methods even though the adaptation shows distinctive performance improvement. We thereby conclude our analysis that the major per- formance gain via prompt- boospt-based based adaptation stems from tailoring the mapping between (commonly) activated concepts and the downstream task classes.
4.2.1 ANALYSIS METHOD AND EXPERIMENT SETUP
Base-to-novel classification. Following the setup introduced by Zhou et al. (2022b), we split the downstream task dataset classes into two groups and consider the first half as base and the remaining as novel classes, then conduct classification datasets (SReproducibility) ImageNet-IK (Deng et al., 2009), Caltech101 (Fei-Fei et al., 2004), OxfordPets (Parkhi al., 2012), StanfordCars (Krause et al., 2013), Flowers102 (Nilsback & Zis- serman, 2008), Food101 (Bossard et al., 2014), FGVC Aircraft (Maji et al., 2013), SUN397 (Xiao al., 2010), DTD (Cimpoi et al., 2014), EuroSAT (Helber al., 2019), and UCF101 (Soomro, 2012).
groups separately. We use total 11 benchmark
Prompt-based adaptation methods append learnable tokens to frozen pretrained CLIP and train the added tokens on downstream tasks. Specifically, MaPLe (Khattak et al., 2023a) adds learnable tokens at the input layer and the first few layers both for text and image encoders (Fig. 5(a)). In the base-to-novel setting, MaPLe uses few-shot samples from each of the base classes to train the learnable tokens. We use officially released MaPLe weights (SReproducibilitty) for the experiments.
Performance improvement via prompt-based methods. Table (the last four columns) sum- marizes the reproduced classification results of CLIP and MaPLe in base-to-novel settings. We notice that both zero-shot performance and performance improvement by adaptation vary in wide range across different datasets. To measure the improvement apart from its zero-shot performance, compute the improvement rate (adapted zero-shot) / (100 zero-shot) and denote it A. A measures the improvement via adaptation relative to the remaining potential improvement from zero-shot performance.
Transferring SAEs to adapted methods. Leveraging that prompt-based methods keep model pa- rameters intact as frozen while adding the learned parameters as additional input tokens, we share our PatchSAE trained on the default CLIP model to both CLIP and MaPLe. This allows us to un- derstand the internal mechanisms of the adaptation method by comparing the model behaviors in
8

---

Published as a conference paper at ICLR 2025
CLIP
CLIP MaPLe
Llass liTg DiStoe
2 Trshecoossggeeee
Sceennoa dase
MaPLe
MaPLe
MaPL
1000
Prediction heatmap
130
Images
heatmap
CLIP
Images
Figure 6: Different impact of SAE latents. We show two example cases of Flowers102 (left) and EuroS AT (right). In each figure, show two example images of the ground-truth (GT) class, scatter plot view of class-level SAE latent activations comparison (æ-axis: CLIP, y-axis: MaPLe trained on the base classes of the downstream task), reference images from ImageNet and the downstream dataset for top latents, and the top-k masking results as a prediction heatmap. We show reference images with segmentation masks for ETT
the shared SAE latent space. We demonstrate the transferability of CLiP-based trained PatchS AE to MaPLe by repeating the top-k SAE latent masking experiment with variations to SAE training back- bones, SAE latent computing backbones (image encoder), and classification inference backbones (text and image encoder) as CLIP or MaPLe (see $A.4). The results validate the transferability of our PatchSAE to the adapted models under all base-to-nove. settings.
Comparing top SAE latents. We compute class-level top activating SAE latents using the shared PatchSAE for backbone image encoders CLIP and MaPL.e for each task. We plot the comparison of class-level latent activations the class-level activation of the latent s; [ (Eq. 5) with backbones CLIP and MaPLe in x and y axes, respectively. We divide the points into several groups including high (high in both), high-to-low (high before and low after adaptation). and low-to-high (low before and high after adaptation) (Fig. 7). We set the upper and lower bounds using top-50 and top- 100 values, and we use class- and dataset-level activations to analyze class- and luse performance improvement, respectively.
low-to-high
high
scatter plot (Fig. 6), where each point represents
Union top
-oand
board
CLIP
Figure 7: Scatter plot re- gions (see Fig. 17).
4.2.2 KEY FINDINGS
Top activating SAE latents mostly overlap for CLIP and MaPLe. The first six columns of Table 1 show the normalized count of top activating SAE latents in three groups: high, high-to-low and low-to-high. For example of EuroSAT base classes, 6.9 latents are highly active in both before and after adaptation, 0.5 previously highly activated latents become non-active after adaptation, and 4.1 previously not active latents became active after adaptation on average. In most cases, SAE latents rarely place in off-diagonal regions. We provide scatter plots for the same comparison in Fig. 20, where we can observe the SAE latent activations are highly correlated. We notice that EuroSAT shows distinctive improvement through MaPLe and the highest count in the low-to-high group and discuss in $B.1.
Influence of top SAE latents in CLIP and MaPLE are different. We compare the impact of the same SAE latent for CLIP and MaPLe. Similar to $4.1, we conduct the top-k masking experiment. We provide the results in Fig. 5(c) and deeper case studies in Fig. 6. In Fig. 5(c), regardless of the top-k latent selection backbones, MaPLe consistently show better performance using the same number of SAE latents. The result implies that MaPLe makes a better use of the same (number of) SAE latents for classification than CLIP does. For deeper analysis, we choose two cases in Flow- ers 102 (case 1) and EuroSAT (case 2) that show large performance improvements by adaptation and top-k masking while the former task is a finer-grained classification and the latter is classification in special domains. We observe that zero- shot and adapted models activate SAE latents in similar patterns for images of the same class, while the two models show different predictions. In both cases, top activating latents recognize visual attributes that seem providing representative informa- tion relevant ground-truth class. Using this same set of latents, MaPLe makes correct predictions while CLIP does not. This result exemplifies MaPLe adding new connections between commonly activated concepts and the downstream tasks. In essence, our analysis shows that the performance gain of prompt-basee based adaptation on CLIP can be explained by adding new mappings between the recognized concepts, which do not change much by adaptation, and the downstream task classes

---

Published as a conference paper at ICLR 2025
5 DISCUSSION
Adopting SAEs vision models. By adopting SAEs originally studied on LLMs to vision models, this work contributes to the understanding of the vision part of vision-languee foundation models. By following basic settings in previous works, evaluating the training performance including re- construction and sparsity objectives, and performance comparison with the original model in down- stream tasks, we validate our design choices. We propose PatchSAE that provides spatial attribution of the candidate concepts, which advances the interpretability. Through extensive qualitative analy- sis, we demonstrate the interpretability of our SAE. Furthermore, we provide an interactive demo to share abundant results with transparency. The scope of this work focuses CLIP vision encoder. We believe that the analysis of different vision encoders and extending it to jointly analyzing the multimodality could provide more comprehensive understanding of large vision-language models. We leave this as future work.
Understanding adaptation methods. We use SAES to shed light model behaviors and adapta- tion mechanisms. In order to use the same SAE for both non-adapted and adapted approaches, we focus on prompt-based adaptation method which does not directly update the model parameter but appends learnable tokens as inputs. We choose MaPLe as the prompt-based method because this approach uses learnable tokens in the vision encoder (note that simpler baseline CoOp uses learn- able tokens only in text encoder) and shows competitive performance with state-of-the methods regardless of simplicity. Exploring different adaptation methods (e.g., full fine-tuning) as future work could provide deeper insights to adaptation mechanisms of foundation models.
6 CONCLUSION
Adapting foundation models to specific tasks has become standard practice to build machine learning systems. Despite this widespread use, the internal workings of models and adaptation mechanisms target tasks remains as an open question. To address this question, we introduced PatchSAE, a sparse autoencoder that extracts interpretable concepts with spatial attributions from a database of reference images. We provide a detailed framework train and analyze PatchSAE models vision transformers. Through controlled experiments on 11 adaptation tasks, we study how adaptation changes the relation between class outputs and concepts. Surprisingly, our analysis finds that on almost none of the studied tasks, drastically new concepts are introduced for adapta- tion. Adaptation rather assigns the right existing concepts to the correct classes, and in only one task with more notable distribution shift (EuroSAT), found a negliggll number of concepts that got suppressed or newly introduced by the adaptation mechanism. Our analysis is an example for leveraging PatchSAE "debug" inner workings of vision model the level of concepts. We believe that methods like PatchSAE will become useful in categorizing algorithms to edit, interpret and adapt foundation models, and to build more effective models on particular downstream tasks.
REPRODUCIBILLITY STATEMENT
We used publicly available model checkpoints for CLIP (link) and MaPLe (link). We used OpenAI ImageNet templates for zero-shot classification (link). Code, model weights and raw results available at https: /i thub. com/ hom/ddynmiii wwl-innerenncccccca wwwprrennepppccchaeeeee patcchaa..ee. We only used publicly available datasets following the official implementation of MaPLe (dataset descriptions) which are cited in the main text.
AUTHOR CONTRIBUTIONS
Conceptualization: StS, HsL with comments from JgC; Methodology: HsL, StS; Software: HsL, JhC; Formal analysis: HsL, StS; Investigation: HsL, JhC; Priting-Original Draft: HsL, JhC; Wriding-Editing: HsL, JgC.
ACKNOWLEDGMENTS
HsL was supported through a DAAD/NRE fellowship in the NRF Summer Institute Programme, 2024 (57600422). This work was supported by the Helmholtz Association's 's Initiative and Network- ing Fund on the HAICORE@KIS KIT partition. This work was supported by Institute for Information & communications Technology Planning & Evaluation (ITPP) and National Research Foundation of Korea (NRF) grants funded by the Korea government (MSIT) 11----0119990000 Artificial Intelli- gence Graduate School Program (KAIST), No. 2022R1 ASA7083908 and No.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In Computer Eisoon-CCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pp. 446 46664461.20 Springer, 2014.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Ka- rina Nguyen, Brayden Mcl.ean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity: Decomposing language models with dictio- nary learning. Oct 2023. URL morosemantic features. Anthropic.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 3606-3613, 2014.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.068 08600, 2023.
Gytis URL
Daujotas.
Interpreting and steering features in images, Jun 21 2024a. http///pttt///ggggeeeeee....... images..
Gytis Daujotas.
Case study: Interpreting, manipulating, and con- URL
trolling clip with sparse autoencoders. Aug 02 2024b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Elhage, Nelson, Hume, Tristan, Olsson, Catherine, Schiefer, Nicholas, Henighan, Tom, Kravec, Shauna, Hatfield-Dodds, Zac, Lasenb, Robert, Drain, Dawn, Chen, Carol, et al. Toymodelsof superposition. 2022.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodee Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits, Dec 22 2021. URL https: / //transformer-cirrut
puls 2021 www.wwork/iiiio wnwex.ndxxx.e html. Anthropic.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Hugo Fry. terpretable
Towards multimodal interpretability: features in vision transformers, Apr 30
Learning 2024.
sparse
in-
URL
fttps::/weet www.:/esswwoo..... at

---

Published as a conference paper at ICLR 2025
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7): 2019.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 191 19113-19 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Huugn Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- -5200,, 2023b.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-graing ane-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19742. PMLR, 2023.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Commy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all once gemma 2. arXiv preprint arXiv:2408.05 /47, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306, 2024.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-graie Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5155 2013.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39 41, 1995.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pp. 722-729. IEEE, 2008
Nostalgebraist.
Interpreting gpt: The logit lens.
LessWrong, 2020.
URL
sttps://wwet Lesswrong. www.eo//////o//oo www.coots/sse.eecoooot wwtpstss//ccRREeepaaaaaseee pattrppetinng-pt--t---ooggg
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction to circuits. Distill, 5())=00022-001,, 2020.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICLR), pp. 7748-8763 PMLR, 2021.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. arXiv preprint arXiv:2404.16011 2024.
Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover-then-nange Task-agnostic concept bottlenecks via automated concept discovery. arXiv preprint arXiv:2407.14499 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Lee Sharkey, Dan Braun, and beren. Taking features out of superposition with sparse autoencoders, Dec 14 2022. URL https://www....... www.wwoo........gggogog....... com http://gw....o.poeoooo//oooottaa interim- Alignment Forum.
Al
Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.04022220 2012.
Adly Templeton, Scaling monosemanticity; Extracting interpretable features from claude sonnet. Anthropic, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23017 2023.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 JEEE computer society conference on IEEE, 2010.
computer vision and pattern recognition, pp.
Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictio- nary learning: contextualized embedding linear superposition of transformer factors. arXiv preprint arXiv:2103.15949 2021.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference computer vision and pattem recognition, pp. 168 16-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 13019):2337-2348 2022b.
13

---

Published as a conference paper at ICLR 2025
A SAE TRAINING DETAILS AND ABLATION STUDIES
In this section, we provide details about training SAE.
1 summarizes the training performance of SAEs used for hyperparameter tuning.
* $A.2 shows SAE layer ablation study results.
* $A.3 shows SAE's generalizability to different datasets.
* $A.4 justifies SAE's transferability to adapted models.
A.1 TRAINING PERFORMANCE
Quantitative metrics. We follow the literature on SAES (Bricken et al., 2023; Templeton, 2024) to set quantitative metrics. The mean squared error (MSE) loss indicates reconstruction ability. Contrastive loss with and without SAE informs the reconstruction ability well. Close contrastive losses indicate that the SAE reconstructs the input better. The I.1 loss and the I.0 metric indicate the sparsity of SAE, The lower the values are, the less number of SAE latents are activated for the given input. We start by reproducing training performance as reported in previous studies (Fry, 2024). Then we ablated training hyperparameters λ₁ expansion factor, ghost gradient technique", and the initialization of decoder bias term (mean vs. geometric median of the training dataset). Furthermore, we ablated model architectures (ViT-B/16 and ViT-L/14), training tokens (CLS vs. all; Fig. 8), and hook layers ($A.2).
Hyperparameters. We set the coefficient for regularizer A as 8e-5, the learning rate as 4e- 4 with constant warmup scheduling with warmup step of 500, and initialized decoder bias with geometric median. We train SAE using 2,621,440 samples from ImageNet training dataset using ghost gradient. We set the threshold that we use for transforming patch-level activations into global views (Eq. 4), to 0.2 (log 10 value of -0.7).
Supplements to summary statistics. In addition to the frequency and the mean value of activation distribution (Bricken et al., 2023), we use the label entropy (Fry, 2024) and the label standard de- viation that can give an intuition about concept granularity. The label standard deviation is tailored for a labeled dataset such as ImageNet, where the label structure contains a hierarchical structure of English words. In this case, the standard deviation indicates whether the latent is capturing distinct label from ImageNet dataset or other attributes such as the style (or domain) or patterns of image. For example, the dog latent might be activated by different breeds of dogs, so the number of unique labels is high (high entropy) but the gap between labels might be low (low standard deviation). On the other hand, blue color latent might be activated by diverse blue objects or scenes whose labels can be very far away (high entropy and high standard deviation).
Although the quantitative metrics of reconstruction and sparsity validate that SAE is trained in- tended, they do not provide rich information about the validity and interpretability of SAEs. There- fore, we utilize SAE latent summary statistics and reference images for qualitative evaluation. We mostly follow the configurations selected by Fry (2024), confirming that the chosen setup shows reasonable performance. To be compatible with both zero-shot and adapted method MaPLe, which releases official weight ViT BTT-B/16, we choose model architecture CLIP ViT-B/16. For deeper understanding, we use all image tokens in addition to the CLS token. We also note recent progress of SAE architectures and training techniques such gated SAE (Rajamanoharan et al., 2024), using SAE on other components' output (such as attention output or MIP output), but we focus on the base architecture of SAES (Bricken et al., 2023) treating the advanced techniques as out-of-scope.
A.2 SAES ON DIFFERENT LAYERS
We ablated the model layers to train SAES. Using ViT-B/16 that has 12 residual block layers, we choose four layers: 2, 5, 8, and 11. We train SAEs for each layers (Fig. 9). We find that the SAE latents on the deeper layers (i.e., closer to the output) provides semantically richer information with high confidence (high activation value) than the ones on the shallower layers. Segmentation masks show that top-3 SAE latents represent the semantic of the major object (the Golden Gate
$ *Ghost gradient is introduced improvement of neuron resampling Bricken al. (2023) that addresses dead neurons due sparsity regularization of training.
14

---

Published as a conference paper at ICLR 2025
Table 2: SAE training contigurations and performance. We share results for six design choices: ViT hook- ing layer, training tokens, VIT architecture, coefficient A expansion factor (dvr x (expansion factor) dSAE), using ghost gradient technique, and initialization of decoder bias term. Default configurations are col- ored as gray, CL. indicates the contrastive loss. * indicates baseline result Fry (2024). Italic indicates the ablated configuration.
Layer
Tokens
Arch.
Ay
Expan.
Ghost
Dec. bias
MSE
CL SAE
CL Org.
L1
LU
11"
cls*
L/14
8e-5"
64
T*
geom."
0.0027
1.775
1.895
13,900
26.00
11
cls
L/14
8e-5
64
T
geom.
0.0026
1.702
1.859
13.823
29.48
11
cls
L/14
0.00
64
T
geom. 0.0000
1.926
1.927
N/A
17570.57
11
cls
L/14
8c-5
64
T
geom. 0.0026
1.702
1.859
13.823
29.48
cls
L/14
8e-4
T
geom. 0.0069
1.693
1.856
0.077
cls
I/14
8c-5
32
T
geom. 0.0028
1.690
1.859
13,608
35.71
I/14
8e-5
T
geor. 0.0026
1.702
1.859
13.823
29.48
11
cls
L/14
8e-5
128
T
geur.. 0.0025
1.689
1.816
14.506
28.43
11
cls
I/14
8e-5
64
F
geom. 0.0026
1.663
1.753
13.957
19.35
11
cls
L/14
8e-5
64
T
geum.. 0,0026
1.702
1,859
13.823
29.48
11
cls
L/14
8e-5
64
T
mean 0.0027
1.693
1,859
14.050
32.47
cls
L/14
8c-5
64
T
gcom. 0,0026
1.702
1,859
13.823
29.48
cls
L/14
8c-5
64
T
geom. 0.0026
1.702
1.859
13.823
29.48
cls
B/16
8c-5
64
geom. 0.0009
1.904
1,986
5.501
25.23
cls
B/16
8c-5
64
T
geom. 0.0009
1.904
1. 1.986
5.501
25.23
8e-5
geor.. 0.0025
1.885
1,962
28.300
148.56
all
B/16
8e-5
64
T
geor.. 0.0010
2.034
2.063
14.730
182.07
all
B/16
Re-5
64
T
geom. 0.0007
2.039
2.039
10.670
298.97 97
all
B/16
8e-5
T
geor.. 0.0005
10.164
242.97
Label entropy
Log 10 mean activetion value
Log sparsity
Log 10 sparsity
SAE for CLS taken only
(b) SAE for all tokens
Figure SAE statisties comparisons on training token ablation. Training SAE only for (a) the class token and for (b) all image tokens (including the class token) show a difference in statistics. Both have two clusters in the bottom left and top right regions but (b) showed denser cluster in the top right region than (a). The plotting style referenced from Fry (2024).
Bridge) in layer 11 while layer 8 and 5 see the triangle shape of the object. Top latents are less inter- pretable for layer This unsurprising as the segmentation mask and activation value indicate the latent is activated by the specified token (local attention), not interpreting meaningful pattern from the entire semantics. The segmentation mask reference images separates the concept from the reference images, which enhances the interpretability. We the same training configuration for four SAES.
A.3 SAE TRANSFERABILTHY TO DIFFERENT DATASETS
In Fig. 10, we show reference images from ImageNet and four fine-grained datasets (Flow- ers102, Caltech101, OxfordPets, and Food101) for two SAE latents. For the Christmas latent (Fig. 10(a)), we retrieve images containing Christmas-related objects or styles. Fig. 10(b) latent represents hockey and/or skate. From Flowers and OxfordPets, the mean activation value
15

---

Published as a conference paper at ICLR 2025
of this latent low, which explains unclear relationship between reference images and the cept of the latent. The mean activation value is higher in Food101, where the retrieved images are more related to the concept: the left top image shows hockey game in the background and the left bottom image shows a game character wearing roller skates. The results demonstrate that SAE latent retrieves consistent concept from different datasets if such concept exists in the dataset). Otherwise, the mean activation value is low and the retrieved images are uninterpretable. See another example in Fig. 14.
A.4 SAE TRANSFERABILTIY ADAPTED MODELS
To justify using CLIP-based trained SAE for analyzing MaPLe, we repeat top-k SAE latent masking experiment under various settings. We observe consistent results whether using CLIP or MaPLe- based SAE latents and demonstrate the transferability of SAE for multimodal prompt- based adapta- tion method.
In Fig. 11, we use SAE trained on MaPLe (trained on ImageNet-1 K). We compare four settings where classification backbone can be either CLIP or MaPLe and SAE can be either CLIP-based or MaPLe-based trained models.
Fig. 15 and 16 shows top-k SAE latent masking results on datasets. Here, we to use class-level activations to select top-k latents and to use CLIP-based trained SAE. We compare four settings of adopting CLIP or MaPLe as SAE activation computing backbone and classification backbone. Using different classification backbone showed different patterns while selecting SAE activation backbone does not show significant difference, which supports the transferability of our SAFs under all base- to-novel settings.
B ADDITIONAL RESULTS AND SUPPORTING FIGURES
Fig. 12 shows screenshot of interactive demo short instruction.
Fig. 13 shows example of multimodal SAE latents.
Fig. 14 task-wise reference images across datasets.
Fig. 15 and 16 show full results of top-k SAE latent masking experiment using MaPLe adapted on each dataset.
Fig. 17 explains three groups in top activating SAE latent comparison scatter plots.
Fig. 18 and 19 shows off-diagonal SAE latents case studies in EuroSAT, DTD and UCF101 datasets.
Fig. 20 shows the scatter plot of aggregated class-level SAE latents 11 datasets.
Fig. 21 shows detailed confusion matrix of Flowers102.
Fig. 22 supplements Fig. 6 and provides more case studies for remapping.
B.1 TOP ACTIVATING SAE LATENTS
We notice that the EuroSAT dataset shows distinctive performance improvement through MaPLe, low correlation Fig. 20, and the highest count the low-to-high group (Table 1). We conduct case studies for classes showing large class-level performance improvement. As shown in Fig. 18, we assess confusion matrices and choose class 2, where the class accuracy improves from 42.61 % 73.07%. We find the class-specific concept latents found from low-to-high regions (i.e., get activated by adaptation), while concepts irrelevant to the classes are deactivated (high-to-low). We find concepts that are generally related the task (satellite or pictures from an airplane) in the high (diagonal) group. We provide more case study results in Fig. 19. The results yield positive initial findings for the first research question adaptation activates additional class-related latents and suggest the need examine the possibility of adaptation improving perceptual ability. We leave this as future work.
16

---

Published as a conference paper at ICLR 2025
(a) input image
(b) Vision encoder and SAE
(c) Transformer block
oper
steam
lyeer
byer
byyer
layer
TALL TAN
MAY
layer
724
724 paac
(f) layer 5
SIT4
21233
(g) layer 2
Inde: 2.32 2:23 1CE
Top-1
2:14
20001
T
Token-level activation
Balow 3360 646
2:04
Top-3
Statistics of SAE_(layer ()
Reference images
Segmentation mask of reference images
Segmentation mask of input image
Figure SAES on different layers. We pass an (a) input image (b) vision transformer collect (c) residual stream output from layers 2, 5, 8, and 1I. (d-g) shows the token-level SAE latent activations (r-axis: SAE latents index y- axixis activation value) from the image token at the patch highlighted in (a), (left), reference images and segmentation mask of top-3 latents (middle), and the summary statistics of the corresponding SAE (right) for each layer.
17

---

Published as a conference paper at ICLR 2025
ImageNet
Flowers1 02
Caltech101
OxfordPets
Food101
#HE-Chiomas
POI ChNtPrae
Found
(a)
(T: -0.08, at 11,63, e:1.08)
[C 0.12, a: -1.32, e:00000)
[C 00.79 a: -1.7, e:2.53)
(F:-0.75, 8: -1.49,1-11.. e1.90)
(F. 0.09
-1.72, e:2.50)
(b)
(1: (f:-0.23, a: 2.00, e.0.9))
(1: 0.62, a: 2.74, e:186))
(f: 0.95, a: 1.92, c:0.39) (f:-1.09 1.009 2.32, c22.1)
(f: (.2-0.27, 2.32, e:2.73)
Figure 10: SAE latents are generalizable to different datasets. Reference images of two SAE latents (a) (top) and (b) (hottom) from five datasets. We present label and class name above each image. The latent statistics log10 of activated frequeney, log10 of mean activation, and lahel entropy values computed from each dataset summarized as below four reference images. More examples shown in the interactive demo.
(a) SAE trained on
Classification using masking with SAE_25
Classification using MaPLa, masking SAE_2S
actrog rit standannnnnnn............ daandootiinggnstttoeeccc.o jinggnsttosseeeeeeooo enoop A3L88
60
Accuritty
40
A00 050099
80
20
20
- renbor -0 doo lodl
100 1000
1000
100 1000
10 100 1000
SAE trained MaPLe
Clasefication using 25, masking MaPte
Idaagl Nona Nerassese inuperatios anagy RIM
60
35
Accurooy 10
Accurocyyt
85
30
10
20
20t lonal
10
5mm
1000
10 100 1000 all
10. 100 1000 all
top-c
om top-4
off top-k
top-k
Figure 11: Comparison on SAEs trained on (a) CLIP (zero-shot) and (b) MaPLe (adapted on ImageNet-I1 IK dataset) models. Left: The scatter plot shows the summary statistics of SAE latents. Right: We repeat the SAE latent masking experiment done in $4.1 for four combinations of varying the classification model and the SAE model between CLIP and MaPLe. The top left plot is the same plot from Fig. 5(b).
18

---

Published as a conference paper at ICLR 2025
Select Input image and patch the image
SAE lotent activations CLIP MaPLL (b-1)
Foleet maag
Soleet aetedd mode su
MUPUE Nuwii
Activation
Adhette Tabas
MITUL
20298
Eew.E-imaggratt Actioution
Achatoon
Index
Segmentation mask for the selected SAE latent 28486
Top activating SAE latent index
Top activding SAC
common--cc008
commen-11124
commch-2207
commes-245
CLI OLP-26485
MaPLE-23
$
edtiiin
(c-1)
Top reference images for the selected SAE latent 28486
Imagehet
(c-
referance for the selected SAE keari 26458
Activation valume:
Imagelett-Skarchh
$
Activation VelAe:
Calechtot
valame: 5.013 0.000 D.DE
Figure 12: interactive demo. (a) Select input image. Specifying patch is also available. (b) Select image encoder backbone for SAE latents. We provide CLIP default and MaPLe trained different datasets for comparisons. show image-level and patch-level (if a patch specified) SAE latent activations. (c) Top SAE latents (commonly only in CLIP only in MaPLe) are selectable, We show (d) segmentation mask and (c) reference images (and activation value of each image) for the selected index. We provide (c-1) on/off option for segmentation mask in reference images. (c-2) shows reference images with segmentation mask.
19

---

Published as a conference paper at ICLR 2025
427 barre!
70 Bick
446 rfng
440 mng
440
Bluetooth
Cupcake
Mouse
740
794
Bluetooth
Cupcake
Mouse
Bluetooth
Cupcake
Mouse
ceatle
483 cost
485 castle
castle
483 castle
Bluetooth
Cupcake
Mouse
945 bel
783 shower
785 seat
934 hot
719 pigw
Good luck
Christmas
Hello
436 Caristmas
Christmas
Christmas
496 Christmas
456 Christmas
Good lack
Merry Christmas
everyono
yollow
Good lack
Welcome
WELCOME
WEL
Manry Christmas
Welcome
Hello everyone
Figure 13: SAE latents can be multimodal. We find multimodal SAE latents that activate from both text and images of the same concept. Each row shows different SAE latent. The first column shows the segme- nation mask from the input image and the right five columns are reference images. From latent represents blue, tooth, (cup)cake, mouse (as representative animation character), good, Christmas, and greetings (hello welcome). SAE latent activation value and more examples are provided in the interactive demo.
bottom, each
20

---

Published as a conference paper at ICLR 2025
Figure 14: Reference images for task-wise most representative SAE feature. (i.j)-th element indicates the reference image from j-th dataset i-th task-wise feature. From left to right (same order from top to bottom), datasets are ImageNet, imagenet-Skteen Caltech101, 100wers CIFAR100, Food101, and OxfordPet.
21

---

Published as a conference paper at ICLR 2025
Sue Crttar spit tev
Furdell
US5
Figure 15: SAE feature masking on base-to-novel classification task (Base split). The legend indicates (backbone for SAE latent (eluctio for classification inference). Blue and orange colors for CL.IP and MaPLe classification inference backhones, respectively.
22

---

Published as a conference paper at ICLR 2025
CantersCen
Fover112
TOYC
SUNDST NOR
Figure SAE feature masking on base-to-novel classification task (Novel split).
(a)
MaPLe (b)
(c)
MaPLe
high
low-to-high
high
Union top-k
upper -bound
low
high
High--- Comn--o--o low
low
bound
.... CLIP
Figure 17: SAF latent comparison scatter plot. We compare group (class dataset) level SAE latents of backbone image encoders CLIP and MaPLe and plot as scatter plot. Each point represents group-level activation (Eq. 5) of CLIP (æ-axis) MaPLe (y-axis). We set upper lower bounds using union top-50 and top-100, respectively. We focus on three groups: high (high in both). high-to-low (high before and low after adaptation), and low-to-high (low before and high after adaptation).
23

---

Published as a conference paper at ICLR 2025
73.07%
bewlthgh
Nghllow
Confusing class
Confusion matrix before and after adaptation
representative concepts
Figure Case study on EuroSAT. (a) Images classes with low classification performance CLIP model. (h) After adaptation, classification performance improves for these classes. (c) Representative images of features that show distinct activation patters: those that were highly activated before adaptation but decreased afterward (high-to-low), those that had low activation before adaptation but increased afterward (low-to-high), those that remained highly activated before and after adaptation.
MaPle
louzhigh
Nout
CLP
UCF101 Kayaking (45)
CUP-AcC: 66.566 66.96%
(81.135
Mopte
Mighllow
(b) Confusion before and after adaptation
Most ropresentative concepts regions
Figure Class-level SAE latents three regions on and UCF10L. show reference images of highly activated before adaptation but decreased afterward (high-to-low) latents with low initial activation that increased after adaptation (low- (owg-O-hiigh,, and latents that remained consistently highly activated both before and after adaptation (high).
24

---

Published as a conference paper at ICLR 2025
Correlation Coefficient
StanforCCar (base))
FBVC Aircnit (base)
Dataset
Corr.
StanfordCar
0.9938
PGVC Aircruft
0,9859
ImageNet
0.9924
Food101
SUN397
0.9951
UCF101
0.9945
OxfordPet
0,9967
DTD
0.9533
BT
0.8657
Caltech101
Flowers 102
0,9911
clp
db
ImageMeet (base)
Food101 (base)
SUN339 (hase)
UCF101 (base)
OxfordPet (hase)
T080 D00
de
EurosAl (base)
Caltech101 (base)
Flovers 102 (base)
1000 b006
meppl
Figure 20: Dataset-level top activating SAE latents. Correlation coefficient values and the seatter plot view on base-to-nove novel classification datasets base split. Top activating SAE latents for CLIP and MaPLe are highly correlated.
25

---

Published as a conference paper at ICLR 2025
WiJE 29 33%
300 20-2205
21,000 ICC 57.70%
21,623R NE 56.05
Lit: deCaull
02.00032 113
defalll
RES
25.33 c: 29.09
21,000
27,49150 1.0%
naple
Figure 21: Confusion matrix for Flowers102 dataset
Flowers102 (Correlation
Rowectt propaerr
CLIP
class 28 artichoke
100
1000
MaPLe
MaPLe
10
50
CLIP
predkted
Flowers class 42 (Correlation 0.99)
CLIP
50
liy
1000
MaPLe
MaPLe
100
1020
CLIP
Figure 22: Supplements for Fig. We show example cases where SAE latents with class discriminative information are highly activated both models, only the adapted model makes correct predictions (when using top latents).
26


Reviews:
[{"review_id": "QRNuHXZQJm", "reviewer": "Reviewer_GMQj", "summary": "The authors perform sparse autoencoder (SAE) training on the features of a pre-trained vision transfer (ViT) from the CLIP image encoder.. The outputs of the 11th layer of the CLIP ViT is used to train a two layer SAE with reconstruction + sparsity objective. The SAE encoder features are then used to validate the presence of concepts in CLIP representations, and their role in classification as well as downstream tasks. The authors first utilize feature and example based statistics to sanity check that concepts are indeed present in SAEs trained on the CLIP visual representations, as well as multi-modal CLIP representations. Next, they verify that these conceptual features are relevant for class discriminative performance of CLIP zero-shot classification. Lastly, the authors show that SAE conceptual features after prompt-based adaptation show a suppression in non class-relevant features, while and increase in the activation of class-relevant features. New conceptual features are not learned during this adaptation procedure.", "questions": "* \"Specifically, we suggest taking all tokens including class (CLS) and image tokens from the second last attention block output and feeding them to SAE rather than limiting it to CLS token\" Can the authors explain why they chose this particular layer for their analysis? Prior work has shown that self-supervised ViTs learn different invariances (hence, concepts) at different depths [1, 2], so one can expect that the SAE results from them would be very different. Are the results derived by the authors on this particular layer generalizable to more layers at different depoths, or did the authors chose this as a representative case study? If so, it would be interesting to see some ablations on how the results change with layer depth.\n* I would be interested in seeing the results of experiment 3 (adaptation methods and SAE) with fine-tuning. Have the authors considered doing an experiment with fine-tuned CLIP for comparison? What are the potential bottlenecks in running such an analysis. And what results can be expected?\n* Typo L146-147: Then we compute *how* frequently these features are activated for the given input.\n\n\n# References \n\n1. Walmer, M., Suri, S., Gupta, K., & Shrivastava, A. (2023). Teaching matters: Investigating the role of supervision in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7486-7496).\n2. Shekhar, S., Bordes, F., Vincent, P., & Morcos, A. (2023). Objectives matter: Understanding the impact of self-supervised objectives on vision transformer representations. arXiv preprint arXiv:2304.13089.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The authors do a good job at utilizing both feature based and example based summary statistics to demonstrate presence of concepts in the SAE representations, and motivate all four statistics well in Section 2.2. Unlike prior work on inferring CLIP concepts from SAEs, which tend to rely on singular feature space statistics, this makes the author's results a bit more comprehensive.\n* The results from replacing CLIP features with SAE features and its impact on accuracy shows that the SAE conceptual features capture class discriminative information quite well. I think this result opens up more avenues on mechanistic interpretability research, as further exploration can be done into what makes a concept important vis-a-vis a class, what are the training dynamics of the concept learning for class discriminative performance (are there 'easier' concepts that are learned first e.g. winter before harder concepts e.g. christmas), and several other open research questions.", "weaknesses": "* The organization of the paper makes it hard for someone who is not very familiar with the field of sparse autoencoder based mechanistic interpretability of neural networks to follow the flow of ideas. For example Section 2 details the training procedure and the feature statistics calculated, but the experimental details are ommitted and then presented in Section 4. Similarly, the Intro and Section 2.1 cover a brief overview of SAEs and their use in intepretability research, but the actual related work section is not presented until Section 5. While I understand that the authors are eager to share their key results in an earlier section (especially since this is a intepretability focussed work), this organization made it more work for me as a reader to understand the paper.\n* The scope of experiments performed and the subsequential observations is somewhat limited. The first experiment, showing presence of concepts in CLIP visual features, was already shown in principle by Fry et al and Daujotas et al. The result about CLIP attention maps focussing on features at different scales and regions is not surprising, but another sanity check for the SAE interpretability apporach. The second emperimental design is inspired from previous SAE results in LLMs e.g. Templeton et al, and extrapolated from a language (only) transformer setting to a multimodal setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7COS0McMR9", "reviewer": "Reviewer_67Ef", "summary": "This paper introduced Sparse AutoEncoder (SAE), which previously trained to disentagle and interprete LLM hidden states, to analysis CLIP features. The authors trained a SAE on top of CLIP features with ImageNet samples, and carried out several experiments with it demonstrating 3 points: \n- Section 3.1: certain demension in SAE feature correlates to certain visual concepts\n- Section 3.2: targeted ablation of neurons yeilds degraded performance\n- Section 3.3: CLIP model \"re-use already known concepts\" when it is fine-tuned for downstream datasets", "questions": "See weaknecess.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper transfers the methodology of SAE from NLP to vision. The idea of identifying visual concepts from CLIP features is quite interesting.\n- From visualizations, it seems that the SAE on top of CLIP sucessfully identified some meaningful semantic concertps in both image-level and patch-level.", "weaknesses": "1. **Lacks quantitative evaluation of SAE's reliability**. Although SAEs for LLMs are widely studied in NLP, to my best knowledge, the usage of SAE for CLIP is only reported in several blog posts that have not been peer-reviewed. Since its effectiveness and reliability are not sufficiently justified in a convincing way, directly using it to draw conclusions is quite risky.\n\n    - For example, how can we ensure that the training settings listed in Section 4.1 are properly set? Are 49,152 hidden dimensions enough for the vision domain? Does this SAE achieve monosemanticity? \n\n    - Examples in the case study are certainly not sufficient for a rigorous evaluation. I would suggest conducting a large-scale evaluation using vision datasets with fine-grained attribute annotations to answer the above questions.\n\n\n1. **Potential distribution shift between base and fine-tuned CLIP**. As stated in Line 325, the authors used the SAE trained on top of pretrained CLIP to interpret the fine-tuned CLIP. As MaPLe fine-tunes both image and text encoders, there are no guarantees that the feature distribution will remain the same after fine-tuning, and all the conclusions in Section 3.3 might be inaccurate as a result.\n\n\n1. **Regarding \"re-mapping\" or \"re-using\" visual concepts**. As highlighted in the paper title and section titles, the authors wanted to demonstrate the connection of visual concepts between base and fine-tuned CLIP. However, I am a bit confused regarding the terminology used, as neither \"re-mapping\" nor \"re-using\" is properly defined in the paper. The authors are encouraged to provide a clear definition of these terms and highlight how the results (e.g., Fig. 6c) demonstrate this point.\n\n\n1. **Significance of Contribution**. Overall, I find the new insights provided in this paper are not as substantial as I expected when I read only the title. Training SAE on CLIP features is not novel. It has been previously reported in several blog posts, yet this paper does not provide a more formal and quantitative evaluation. The results in Section 3.1 and 3.2 are somewhat expected and are not particularly surprising. \n\n    I would suggest that the authors conduct large-scale quantitative validation to prove the effectiveness of CLIP's SAE, e.g., ablation of hyperparameters, training settings, and types of CLIP models (convolutional encoders are not covered in the presented study). Additionally, clearly stating the relation and differences between SAE and existing interpretability methods in the vision domain would help readers understand the contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D3hlMMSeb5", "reviewer": "Reviewer_qx77", "summary": "In this paper, the authors propose a new way to try to interpret features learned from a vision-language (in particular, a CLIP model). The proposed approach train SAE on all image tokens. The author then show that the trained SAE can be used to discover different concepts from the CLIP vision transformer. The authors then investigate how the learned \"interpretable features\" relates the output of the model.", "questions": "- Please see \"weaknesses\" above for a few questions. \n- How would the results change if the authors use a subset (or a totally different set) of \"SAE feature statisics\"? Why did the authors chose the ones they use in the paper (eg, sparsity, label entropy, mean activation, etc). Was any ablation study performed to select these features? Were they selected in previous work?\n- How would the proposed approach perform on other language-vision models besides CLIP? Would the selected hyperparameters need to be different in this case?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Better understanding foundantion models (and what their features encode) is an extremely important problem---specially as these models get more powerful.\n- The proposed approach is simple  and leverages the well-understood sparse autoencoder approach to interpret the features.", "weaknesses": "- The paper has some very strong claims that has not been properly demonstrated. Eg, L249 says \"CLIP understands sophisticated concepts from input images not only responding to basic patterns such as color or shape \". Showing a a few (potentially cherrypicked) qualitative examples is not enough to show/demonstrate/prove the results on the paper. It would make results stronger if  the authors could provide more quantitative metrics to measure the sophistication of detected concepts. This would make the results on the paper more convincing/interesting.\n- Some sessions of the paper are not very clear. For example, the authors leverage MaPLe model (eg \"adaptation methods analysis\") and nowhere in the paper they explain how this model works. It would make the paper more readable if the authors could provide a brief overview of this model on the manuscript.\n- A lot of design choices seem to be made ad-hoc. For example, why chose the second-to-last layer? Why use the SAE feature statistics used? Why 49,152 dimensions? Could the authors provide some justification for these choices? How were those choices made? Was any ablation study performed to choose those values?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LxdBjU1ETr", "reviewer": "Reviewer_h3c2", "summary": "This paper utilizes sparse autoencoders (SAE) to interpret visual concepts learned by the CLIP vision model and investigates how these concepts are affected by adaptation techniques such as MaPLe. The key finding is that during adaptation, the model primarily reuses existing concepts rather than learning entirely new ones.", "questions": "- Section 3.3 (ADAPTATION METHODS AND SAE) is difficult to follow. Can you provide a more concise and coherent explanation of the logic behind the analysis, supported by a smaller set of essential figures?\n- What does “suddenly capture new concepts” mean in this context? In human cognition, new concepts are usually built upon existing knowledge. Given this, how does this study distinguish between the reuse of old concepts and the acquisition of new concepts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper offers a detailed and fine-grained exploration of CLIP by processing the entire token sequence with SAEs, allowing for a deeper understanding of the model's visual feature representations.\n- It tackles the adaptation dynamics of foundation models like CLIP, which is a relatively under-explored topic.", "weaknesses": "- The clarity and organization of the analyses are insufficient. The main body of the paper frequently refers to figures in the appendix, many of which are poorly annotated, making it difficult for readers to follow the core arguments. The authors should revise the manuscript to consolidate the most important findings in the main body and present them in a clearer and more structured format.\n- The paper lacks sufficient discussion on the broader implications and significance of its findings. Are any results surprising or do they challenge conventional views within the field? The authors should also elaborate on the practical value of their discoveries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)