Sparse autoencoders reveal selective remapping of visual concepts during adaptation

Authors: Hyesu Lim, Jinho Choi, Jaegul Choo, Steffen Schneider
Primary area: interpretability and explainable AI
Keywords: interpretability, vision-language models, sparse autoencoder, adaptation

Abstract:
Adapting foundation models for specific purposes has become a standard approach to build machine learning systems for downstream applications. Yet, it is an open question which mechanisms take place during adaptation. Here we develop a new Sparse Autoencoder (SAE) for the CLIP vision transformer, named PatchSAE, to extract interpretable concepts at granular levels (e.g., shape, color, or semantics of an object) and their patch-wise spatial attributions. We explore how these concepts influence the model output in downstream image classification tasks and investigate how recent state-of-the-art prompt-based adaptation techniques change the association of model inputs to these concepts. While activations of concepts slightly change between adapted and non-adapted models, we find that the majority of gains on common adaptation tasks can be explained with the existing concepts already present in the non-adapted foundation model. This work provides a concrete framework to train and use SAEs for Vision Transformers and provides insights into explaining adaptation mechanisms.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730087573893

thread_type:
official_review

<thread_object>
creator: reviewer_h3c2

<content>
summary:
This paper utilizes sparse autoencoders (SAE) to interpret visual concepts learned by the CLIP vision model and investigates how these concepts are affected by adaptation techniques such as MaPLe. The key finding is that during adaptation, the model primarily reuses existing concepts rather than learning entirely new ones.

strengths:
- The paper offers a detailed and fine-grained exploration of CLIP by processing the entire token sequence with SAEs, allowing for a deeper understanding of the model's visual feature representations.
- It tackles the adaptation dynamics of foundation models like CLIP, which is a relatively under-explored topic.

weaknesses:
- The clarity and organization of the analyses are insufficient. The main body of the paper frequently refers to figures in the appendix, many of which are poorly annotated, making it difficult for readers to follow the core arguments. The authors should revise the manuscript to consolidate the most important findings in the main body and present them in a clearer and more structured format.
- The paper lacks sufficient discussion on the broader implications and significance of its findings. Are any results surprising or do they challenge conventional views within the field? The authors should also elaborate on the practical value of their discoveries.

questions:
- Section 3.3 (ADAPTATION METHODS AND SAE) is difficult to follow. Can you provide a more concise and coherent explanation of the logic behind the analysis, supported by a smaller set of essential figures?
- What does “suddenly capture new concepts” mean in this context? In human cognition, new concepts are usually built upon existing knowledge. Given this, how does this study distinguish between the reuse of old concepts and the acquisition of new concepts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730160315023

thread_type:
official_review

<thread_object>
creator: reviewer_qx77

<content>
summary:
In this paper, the authors propose a new way to try to interpret features learned from a vision-language (in particular, a CLIP model). The proposed approach train SAE on all image tokens. The author then show that the trained SAE can be used to discover different concepts from the CLIP vision transformer. The authors then investigate how the learned "interpretable features" relates the output of the model.

strengths:
- Better understanding foundantion models (and what their features encode) is an extremely important problem---specially as these models get more powerful.
- The proposed approach is simple  and leverages the well-understood sparse autoencoder approach to interpret the features.

weaknesses:
- The paper has some very strong claims that has not been properly demonstrated. Eg, L249 says "CLIP understands sophisticated concepts from input images not only responding to basic patterns such as color or shape ". Showing a a few (potentially cherrypicked) qualitative examples is not enough to show/demonstrate/prove the results on the paper. It would make results stronger if  the authors could provide more quantitative metrics to measure the sophistication of detected concepts. This would make the results on the paper more convincing/interesting.
- Some sessions of the paper are not very clear. For example, the authors leverage MaPLe model (eg "adaptation methods analysis") and nowhere in the paper they explain how this model works. It would make the paper more readable if the authors could provide a brief overview of this model on the manuscript.
- A lot of design choices seem to be made ad-hoc. For example, why chose the second-to-last layer? Why use the SAE feature statistics used? Why 49,152 dimensions? Could the authors provide some justification for these choices? How were those choices made? Was any ablation study performed to choose those values?

questions:
- Please see "weaknesses" above for a few questions. 
- How would the results change if the authors use a subset (or a totally different set) of "SAE feature statisics"? Why did the authors chose the ones they use in the paper (eg, sparsity, label entropy, mean activation, etc). Was any ablation study performed to select these features? Were they selected in previous work?
- How would the proposed approach perform on other language-vision models besides CLIP? Would the selected hyperparameters need to be different in this case?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731267224469

thread_type:
official_review

<thread_object>
creator: reviewer_67ef

<content>
summary:
This paper introduced Sparse AutoEncoder (SAE), which previously trained to disentagle and interprete LLM hidden states, to analysis CLIP features. The authors trained a SAE on top of CLIP features with ImageNet samples, and carried out several experiments with it demonstrating 3 points: 
- Section 3.1: certain demension in SAE feature correlates to certain visual concepts
- Section 3.2: targeted ablation of neurons yeilds degraded performance
- Section 3.3: CLIP model "re-use already known concepts" when it is fine-tuned for downstream datasets

strengths:
- The paper transfers the methodology of SAE from NLP to vision. The idea of identifying visual concepts from CLIP features is quite interesting.
- From visualizations, it seems that the SAE on top of CLIP sucessfully identified some meaningful semantic concertps in both image-level and patch-level.

weaknesses:
1. **Lacks quantitative evaluation of SAE's reliability**. Although SAEs for LLMs are widely studied in NLP, to my best knowledge, the usage of SAE for CLIP is only reported in several blog posts that have not been peer-reviewed. Since its effectiveness and reliability are not sufficiently justified in a convincing way, directly using it to draw conclusions is quite risky.

    - For example, how can we ensure that the training settings listed in Section 4.1 are properly set? Are 49,152 hidden dimensions enough for the vision domain? Does this SAE achieve monosemanticity? 

    - Examples in the case study are certainly not sufficient for a rigorous evaluation. I would suggest conducting a large-scale evaluation using vision datasets with fine-grained attribute annotations to answer the above questions.


1. **Potential distribution shift between base and fine-tuned CLIP**. As stated in Line 325, the authors used the SAE trained on top of pretrained CLIP to interpret the fine-tuned CLIP. As MaPLe fine-tunes both image and text encoders, there are no guarantees that the feature distribution will remain the same after fine-tuning, and all the conclusions in Section 3.3 might be inaccurate as a result.


1. **Regarding "re-mapping" or "re-using" visual concepts**. As highlighted in the paper title and section titles, the authors wanted to demonstrate the connection of visual concepts between base and fine-tuned CLIP. However, I am a bit confused regarding the terminology used, as neither "re-mapping" nor "re-using" is properly defined in the paper. The authors are encouraged to provide a clear definition of these terms and highlight how the results (e.g., Fig. 6c) demonstrate this point.


1. **Significance of Contribution**. Overall, I find the new insights provided in this paper are not as substantial as I expected when I read only the title. Training SAE on CLIP features is not novel. It has been previously reported in several blog posts, yet this paper does not provide a more formal and quantitative evaluation. The results in Section 3.1 and 3.2 are somewhat expected and are not particularly surprising. 

    I would suggest that the authors conduct large-scale quantitative validation to prove the effectiveness of CLIP's SAE, e.g., ablation of hyperparameters, training settings, and types of CLIP models (convolutional encoders are not covered in the presented study). Additionally, clearly stating the relation and differences between SAE and existing interpretability methods in the vision domain would help readers understand the contributions.

questions:
See weaknecess.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731297306354

thread_type:
official_review

<thread_object>
creator: reviewer_gmqj

<content>
summary:
The authors perform sparse autoencoder (SAE) training on the features of a pre-trained vision transfer (ViT) from the CLIP image encoder.. The outputs of the 11th layer of the CLIP ViT is used to train a two layer SAE with reconstruction + sparsity objective. The SAE encoder features are then used to validate the presence of concepts in CLIP representations, and their role in classification as well as downstream tasks. The authors first utilize feature and example based statistics to sanity check that concepts are indeed present in SAEs trained on the CLIP visual representations, as well as multi-modal CLIP representations. Next, they verify that these conceptual features are relevant for class discriminative performance of CLIP zero-shot classification. Lastly, the authors show that SAE conceptual features after prompt-based adaptation show a suppression in non class-relevant features, while and increase in the activation of class-relevant features. New conceptual features are not learned during this adaptation procedure.

strengths:
* The authors do a good job at utilizing both feature based and example based summary statistics to demonstrate presence of concepts in the SAE representations, and motivate all four statistics well in Section 2.2. Unlike prior work on inferring CLIP concepts from SAEs, which tend to rely on singular feature space statistics, this makes the author's results a bit more comprehensive.
* The results from replacing CLIP features with SAE features and its impact on accuracy shows that the SAE conceptual features capture class discriminative information quite well. I think this result opens up more avenues on mechanistic interpretability research, as further exploration can be done into what makes a concept important vis-a-vis a class, what are the training dynamics of the concept learning for class discriminative performance (are there 'easier' concepts that are learned first e.g. winter before harder concepts e.g. christmas), and several other open research questions.

weaknesses:
* The organization of the paper makes it hard for someone who is not very familiar with the field of sparse autoencoder based mechanistic interpretability of neural networks to follow the flow of ideas. For example Section 2 details the training procedure and the feature statistics calculated, but the experimental details are ommitted and then presented in Section 4. Similarly, the Intro and Section 2.1 cover a brief overview of SAEs and their use in intepretability research, but the actual related work section is not presented until Section 5. While I understand that the authors are eager to share their key results in an earlier section (especially since this is a intepretability focussed work), this organization made it more work for me as a reader to understand the paper.
* The scope of experiments performed and the subsequential observations is somewhat limited. The first experiment, showing presence of concepts in CLIP visual features, was already shown in principle by Fry et al and Daujotas et al. The result about CLIP attention maps focussing on features at different scales and regions is not surprising, but another sanity check for the SAE interpretability apporach. The second emperimental design is inspired from previous SAE results in LLMs e.g. Templeton et al, and extrapolated from a language (only) transformer setting to a multimodal setting.

questions:
* "Specifically, we suggest taking all tokens including class (CLS) and image tokens from the second last attention block output and feeding them to SAE rather than limiting it to CLS token" Can the authors explain why they chose this particular layer for their analysis? Prior work has shown that self-supervised ViTs learn different invariances (hence, concepts) at different depths [1, 2], so one can expect that the SAE results from them would be very different. Are the results derived by the authors on this particular layer generalizable to more layers at different depoths, or did the authors chose this as a representative case study? If so, it would be interesting to see some ablations on how the results change with layer depth.
* I would be interested in seeing the results of experiment 3 (adaptation methods and SAE) with fine-tuning. Have the authors considered doing an experiment with fine-tuned CLIP for comparison? What are the potential bottlenecks in running such an analysis. And what results can be expected?
* Typo L146-147: Then we compute *how* frequently these features are activated for the given input.


# References 

1. Walmer, M., Suri, S., Gupta, K., & Shrivastava, A. (2023). Teaching matters: Investigating the role of supervision in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7486-7496).
2. Shekhar, S., Bordes, F., Vincent, P., & Morcos, A. (2023). Objectives matter: Understanding the impact of self-supervised objectives on vision transformer representations. arXiv preprint arXiv:2304.13089.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732175529824

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of rebuttal

comment:
*We sincerely thank all four reviewers for their constructive feedback and valuable comments.*

Through **extensive revisions** to improve readability, the manuscript now offers enhanced clarity and structure. We would be deeply grateful if the reviewers could kindly take a fresh perspective on these changes. To allow for better exploration of our model, we now provide an **[interactive demonstration](https://huggingface.co/spaces/iclranonym/paper14240)** via Huggingface spaces.

**The strengths of our work, as highlighted by the reviews, include:**
* This work addresses an important but relatively under-explored problem and opens several research questions: understanding the inner workings of vision foundation models and adaptation mechanisms. (**GMQj**, **qx77**, **h3c2**)
* This work adopts SAE, a mechanistic interpretability tool that is widely studied for LLMs, to understand visual feature representations. (**67Ef**, **qx77**, **h3c2**)
* The analysis result demonstrates that the SAE model is successfully extracting interpretable concepts both in the patch- and image-level which contains class discriminative information. (**GMQj**, **67Ef**)

**Our responses to the reviews are as follows (4 parts):**
1. Paper re-organization and addressing ambiguities
2. Justification for SAE's reliability, design choices, and transferability
3. Clarification of contributions
4. Interactive demo: Highlighting key results

We provide brief summary of 4 parts of the responses in [**General Responses (GR)**](https://openreview.net/forum?id=imT03YXlG2&noteId=bE1RByHTPP) with references to the paper and the review items. We annotated the review items to the relevant paragraph of the revised paper as well. The reviewers can search for `{reviewer id}-{item id}` in the paper.

Once again, we appreciate the reviewers' considerate feedback and believe that our responses address their concerns effectively. Thank you for considering our rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732175707412

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses to all reviewers

comment:
**Please see related section in the revised paper.**

## **GR1. Paper re-organization and addressing ambiguities** 

Our primary focus on the manuscript revision was on reconstructing the existing content to more clearly and effectively convey our findings and emphasize contributions. We added several figures and results to address the reviewers' concerns. Since we made extensive revisions, we provide a summary of the modifications.

* Terminologies
    * We named our SAE **PatchSAE**, emphasizing its strength in spatial attribution. 
    * We use "SAE latents (latent directions)" to refer to the candidate concepts and "SAE latent activations" for the SAE ReLU activation output. ($\S$ 3.2)
* Main paper
    * $\S$ 1 Introduction, $\S$ 2 Related work, $\S$ 3 Training and evaluating SAE, $\S$ 4 Analyzing CLIP behavior via SAE, $\S$ 5 Discussion, and $\S$ 6 Conclusion
    * > GMQj-W1, qx77-W2, h3c2-W1, h3c2-Q1, h3c2-Q2
* Appendix
    * $\S$A SAE training details and ablation studies, $\S$B Additional results (including interactive web app) and supporting figures
    * > h3c2-W1

* Updated Figures (Main)
    * Fig.1 (overview): Clarified main contributions. 
    * Fig.2 (SAE analysis method), Fig. 4 (segmentation mask): Improved readability
    * Fig.3 (evaluating SAE): Moved statistics scatter plot from the appendix, and added reference images.
        * > qx77-W1
    * Fig. 5 (top-k masking): Improved readability and moved full results in 11 datasets from the appendix.
    * Fig. 6 (**new**; analyzing adaptation): Elaborate "understanding adaptation methods via SAE" part
        * > 67Ef-W3, h3c2-Q1
    * Fig. 7 (**new**; scatter plot regions): Explain the scatter plot view of top-k class level activation comparisons

* Updated Figures (Appendix)

    * Fig. 9 (**new**; SAEs on different layers)
    * Fig. 10 (SAEs on different datasets): Moved from the main paper and revised.
    * Fig. 11 (**new**; SAE's transferability)
    * Fig. 12 (**new**; Screenshot for demo)
    * Fig. 13 (Multimodal latents): Added more cases.
    * Fig. 17 (new; Supplements new Fig. 7)
    * Fig. 22 (new; Supplements new Fig. 6)

## **GR2. Justification for SAE's reliability** 

### **GR2.1.** Evaluating SAE ($\S$ 3.2, $\S$ 3.3, $\S$ 4.2, $\S$ A.1, and interactive demo)
* We explain how we validate SAE's reliability. We first evaluate reconstruction and sparsity metrics to that demonstrate our SAE follows the training objectives ($\S$ A.1). Then we use summary statistics and reference images with segmentation masks for qualitative analysis of SAE's validity as an interpretability tool ($\S$ 3.3). We provide an **[interactive demonstration](https://huggingface.co/spaces/iclranonym/paper14240)** to share more qualitative results. For classifications, we show that using our SAE (using all latents) successfully recovers the original classification performance ($\S$ 4.2). 
* > 67Ef-W1, qx77-W1, qx77-Q1
### **GR2.2.** SAE training configurations ($\S$ A.1 and Table 2)
* We explain how we set training configurations with quantitative results from ablation studies.
* > 67Ef-W1, qx77-W3

### **GR2.3.** SAEs on different layers ($\S$ A.2 and Fig. 9)
* We train the SAEs on layers [2, 5, 8, 11(default)] and show comparisons. We find that the SAE latents on the deeper layers (i.e., closer to the output) provide semantically richer information with high confidence (high activation value) than the ones on the shallower layers.
* > GMQj-Q1, qx77-W3

### **GR2.4.** SAE's transferability on adapted models ($\S$ 4.1.1, $\S$ A.4, and Fig. 11)
* We intentionally choose MaPLe as an adaptation method to investigate the changes made by adaptation in the shared SAE latent space leveraging that MaPLe does not update model parameters. We demonstrate the transferability by comparing (backbone for SAE training, classification backbone) as (CLIP_SAE, CLIP), (MaPLe_SAE, CLIP), (CLIP_SAE, MaPLe), (MaPLe_SAE, MaPLe).
* > GMQj-Q2, 67Ef-W2, qx77-Q2

## **GR3. Clarification of contribution** ($\S$ 1, $\S$ 2, and $\S$ 5)

We discuss the scope of this work with broader implications and future work in $\S$ 5. We also clarify the main components of this paper in $\S$ 1 and state distinctive differences from existing work in $\S$ 2. In short, we introduce SAE to vision models and provide visual attribution of concepts to advance interpretability. We use our SAE to understand vision models' behavior on classification tasks and explain how adaptation affects the models' behavior in terms of interpretable concepts (latents).
* > GMQj-W2, 67Ef-W4, h3c2-W2


## **GR4. Interactive Demo**

We provide an **[interactive demonstration](https://huggingface.co/spaces/iclranonym/paper14240)** on Huggingface spaces with an anonymized account and quick instruction in Fig. 12. Users can select images, patches, models, and SAE latents. Based on the selection, the demo shows SAE latent activations, segmentation masks, and reference images.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733302684097

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
End of Rebuttal Phase

comment:
Dear reviewers, 
Dear AC,

we want to thank all reviewers again for the constructive feedback. Based on this feedback, we made a revision to the paper to enhance clarity and provide additional experimental evidence during the discussion phase **which was acknowledged by all reviewers and unanimously positively evaluated**.

**Thank you again for the time invested into reviewing our work.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
Post rebuttal, all of reviewers vote for acceptance. The AC checked all the materials and concurs that the paper has done a valuable exploration of using patch-wise sparse auto-encoders to interpret the CLIP visual representations, especially through the adaptation process. The paper received concerns about clarity and organization, but has been significantly improved during the rebuttal period, with authors paying great efforts providing new results and re-writing the draft. With these major revisions, the paper can be accepted. Please incorporate necessary changes in the final version.