Interpreting and Steering LLM Representations with Mutual Information-based Explanations on Sparse Autoencoders

Authors: Xuansheng Wu, Jiayi Yuan, Wenlin Yao, Xiaoming Zhai, Ninghao Liu
Primary area: interpretability and explainable AI
Keywords: large language models, sparse autoencoders, usable xai, explanations, interpretability

Abstract:
Large language models (LLMs) excel at addressing general human queries, yet they can falter or produce unexpected responses in specific scenarios. Gaining insight into the internal states of LLMs is key to understanding their successes and failures, as well as to refining their capabilities. Recent efforts have applied sparse autoencoders to learn a feature basis for explaining LLM hidden spaces. However, current post-hoc explanation methods can not effectively describe the semantic meaning of the learned features, and it is difficult to steer LLM behaviors by manipulating these features. Our analysis reveals that existing explanation methods suffer from the frequency bias issue, i.e., they tend to focus on trivial linguistic patterns rather than semantics. To overcome this, we propose explaining the learned features from a fixed vocabulary set to mitigate the frequency bias, and designing a novel explanation objective based on the mutual information theory to better express the meaning of the features. We further suggest two strategies to steer LLM representations by modifying sparse feature activations in response to user queries during runtime. Empirical results demonstrate that our method generates more discourse-level explanations than the baselines, and can effectively steer LLM behaviors to defend against jailbreak attacks in the wild. These findings highlight the value of explanations for steering LLM representations in downstream applications.

Decision: reject
Reviewer scores: 6, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730592254423

thread_type:
official_review

<thread_object>
creator: reviewer_xcaj

<content>
summary:
This paper proposes a new approach to improve the interpretability and controllability of LLMs by addressing the limitations of current explanation methods. By addressing frequency bias via a mutual information-based objective, the authors aim to create semantically meaningful feature explanations. Additionally, they introduce strategies to steer LLM behavior by modifying feature activations, defending against jailbreak attacks and enhancing LLM performance in downstream tasks.

strengths:
- Gaining a deeper understanding of features relevant for LLM predictions is both highly relevant and practically insightful.
- The study includes reproducible, comprehensive experiments.
- The paper is overall well-written and easy to follow. The figures and findings are clearly presented.

weaknesses:
- A deeper analysis of what cases lead to unsuccessful sparse features should be included.
- A critical discussion of using LLMs as evaluators and selection mechanism for safety is missing.
- The methodology overall is quite simple and lacks novelty and, in my opinion, is a minor contribution of the paper.

questions:
- Why was it necessary to restrict the vocabulary? Wouldn’t the mutual information take care of appropriate filtering?
- Couldn't the frequency bias issue be directly addressed via tf-idf as done in many topic modeling works?
- At what layer of the network are the features extracted and why?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621448704

thread_type:
official_review

<thread_object>
creator: reviewer_bczb

<content>
summary:
This paper addresses the challenge of understanding and guiding the internal states of large language models (LLMs) to improve their reliability and performance. The authors identify a frequency bias in existing explanation methods that skews interpretations toward trivial patterns. To address this, they propose a new approach that mitigates frequency bias by using a fixed vocabulary and an explanation objective grounded in mutual information theory, which enhances the semantic clarity of learned features. Additionally, they introduce two runtime strategies for modifying sparse feature activations, allowing user-directed control over LLM responses.

strengths:
1. This paper addresses a significant challenge: understanding the semantics of sparse features learned by sparse autoencoders. Accurate interpretation of these features can provide valuable insights into the model’s internal mechanisms and enable better control over its behavior. 
2. The paper is well-structured and clearly presented. 
3. The innovative use of a fixed vocabulary is particularly noteworthy, as it can mitigate the issue of explanations that overly focus on syntax rather than meaningful semantics.

weaknesses:
1. While I agree with the general idea, I think there is a major flaw in the core method. Specifically, in Eq.3, where the author states a proportional relationship between the second and the third line. This only holds if P(C) is a uniform distribution, which is not realistic. So, either the author have an incorrect derivation or there is a strong over-simplification in the derivation. Please explain further. 
2. The effectiveness of the method is not clear. Table 3 shows three distinct sets of features for three methods. This is a cherry-picked result. For fair comparison, same features should be selected for comparison. In 4.2.2, quantitative evaluation is compared between TopAct, N2G and the proposed method. To me, the selected baselines are too simple. AutoInterp \[1\], which leverages the both the input text and the feature activation value of each token, should be considered, in order to show that the proposed method is better than the SOTA. 
3. The contribution statement that sparse features can be used to steer LLMs at runtime is not novel and has been descovered by other works. Prior works from Antropic AI and other various works has shown that sparse features can steer the model behavior.

Based on the above comments, I think this work needs to be revised to better resolve these issues.

\[1\]: https://blog.eleuther.ai/autointerp

questions:
As listed above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649761172

thread_type:
official_review

<thread_object>
creator: reviewer_7cxh

<content>
summary:
While state-of-the-art large language models have shown impressive capabilities, they also produce unexpected responses, highlighting the need for a better understanding of their internal representations. Prior works primarily rely on annotated datasets, which can be limiting. The authors propose an unsupervised technique using sparse autoencoders (SAEs) to address the challenges of interpreting and utilizing features learned by sparse autoencoders in LLMs. In particular, the SAEs learn discourse topics and linguistic patterns, with a bias towards linguistic patterns due to their frequency. To alleviate this problem, the authors propose using a fixed vocabulary set to capture critical information based on mutual information objectives and, further, use it to steer LLM representations by modifying activations during runtime, demonstrating that this method can improve safety by preventing jailbreak attacks.

strengths:
1. The authors provide an interesting approach to steer the LLM behavior using sparse autoencoders and provide an intuitive analysis that reveals the frequency bias between discourse and linguistic features.

2. Empirical results across two benchmark datasets show the effectiveness of SAE-Steer in improving the jailbreaking performance of LLMs.

3. The paper proposes using a fixed vocabulary set and a mutual-information-based objective to identify words that capture the feature's meanings and eliminate the frequency bias.

weaknesses:
While the authors provide a good study of using Sparse AutoEncoders for steering LLM behaviors, below are some open questions and weaknesses:

1. The author proposes using a fixed vocabulary set to mitigate the frequency bias and designing a novel explanation objective based on the mutual information theory to better express the meaning of the features. However, they do not explain how they get this vocabulary set. For instance, different large language models have different vocabulary sets depending on their tokenizer and the training dataset. How to use a fixed vocabulary set that generalizes to different LLMs?

2. The authors claim that their empirical results show that SAE generates more discourse-level explanations than the baselines, but the evaluation part of this claim is a bit weak. It would be great if the authors could provide more support to this claim. While the raw explanations in Table 1 and other qualitative results demonstrate that the proposed method identifies relevant words w.r.t. the text summary, it doesn't seem to identify the sparse set of words responsible for the summary as it consists of many adverbs like originally, already, etc. Further, it would be great to expand the qualitative analysis to more than the three examples shared in Table 1. For instance, how often do baseline techniques like TopAct and N2G generate "used to" as the raw explanation?

3. The notations in Section 2.1 are inconsistent and a bit confusing. The authors use $X$ for the input text and $\mathbf{X}$ for the embedding at a specific layer without any notation denoting a given layer.

4. The theoretical analysis is based on the topic model assumption, which assumes a system where we first come up with a topic and then select words that best represent the topic. It would be great for the readers if the authors could explain this from the perspective of auto-regressive models, where we do not necessarily have a topic.

5. The strategies proposed by the authors to steer LLM representations with the identified features S during runtime are similar to the ones proposed by Li et al. [1], undermining the novelty of the current work. Further, are the identified words as explanations primarily a result of the correlation of the activation with the chosen words

6. *Given a feature vector and its raw explanations, the machine annotator is called to provide a short summary of the explanations with an option to say “Cannot Tell” in case the raw explanations make no sense* -- the effectiveness of the GPt-4o evaluator is not provided. While the templates in the Appendix are intuitive, it would be great if we could provide some quantitative results to ground the performance of the GPt-4o evaluator.

7. The SAE Steer does improve the attack success rate for the Salad-Bench dataset but we don't observe the corresponding improvement for the MT-Bench dataset. The authors should explain this phenomenon, i.e., why do we observe a drop in the score performance using SAE Steer on the MT-Bench?

8. (Minor) The authors cite previous work (Lieberum et al., 2024) in selecting the 8th layer of the Mistral-7B-Instruct model for their empirical analysis. It would be beneficial for the readers if the authors could motivate choosing this particular layer.

**References**

1. Li et al. "Inference-time intervention: Eliciting truthful answers from a language model." NeurIPS, 2023

questions:
Please refer to the weakness section for open questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691975411

thread_type:
official_review

<thread_object>
creator: reviewer_dudn

<content>
summary:
This paper proposes a method to improve the interpretability of sparse autoencoder features and their influence on large language models. The authors introduce a post-hoc explanation technique that highlights the features learned by sparse autoencoders, which capture both discourse topics and linguistic patterns. Additionally, in this work, a method is also proposed to steer and control LLM behavior by adjusting the activation of those explained topic features during runtime.

strengths:
This paper clearly shows the application of sparse autoencoders in explaining mono- semantic neurons; The proposed explaining technique is well displayed both theoretically and formatively; The experiments give adequate answers to the abilities of the proposed method on generating discourse-level explanations and usefulness of those explanations.

weaknesses:
1. The experiments to compare the effectiveness on summarizing the discourse-level explanations is not clear and case study is not fair enough in Table 1. The table didn’t show the different abilities of extracting discourse-level and semantic-level explanations on same summary conditions.

2. The fidelity of those sparse features is not evaluated. Do those features shows the actual concepts during the model runtime?

questions:
1. The explanation in Table 1 gives some confusing elements? e.g., the row: “Analysis of performance metrics” gives landscape/golf/retirements. The sparsity seems worse than those traditional methods like TopAct.

2. When do the operation such as “EH” and “AS”, to my knowledge, the model will output some illogical response. Did the paper exclude those responses when calculating the ASR on Salad-bench or MT-bench.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a post-hoc interpretability method that uses sparse autoencoder features in conjunction with a fixed vocabulary set. Most reviewers agreed that the paper was well-written and explored an interesting idea to reduce frequency bias in explanations. However, there were several concerns raised around the evaluations; the validity of the theoretical assumptions made; the justification of the numerous design choices that went into the method; and the overall impact of the work. After discussion with the reviewers, I am unable to recommend acceptance at this time, and I encourage the authors to take the reviewer feedback into account to polish the paper for a future submission.