Towards Unifying Interpretability and Control: Evaluation via Intervention

## Abstract

Abstract With the growing complexity and capability of large language models (LLMs), a need to understand model reasoning has emerged, often motivated by an underlying goal of controlling and aligning models.
While numerous interpretability and steering methods have been proposed as solutions, they are typically designed either for understanding or for control, seldom addressing both, with the connection between interpretation and control more broadly remaining tenuous. Additionally, the lack of standardized applications, motivations, and evaluation metrics makes it difficult to assess these methods’ practical utility and efficacy.
To address the aforementioned issues, we propose intervention as a fundamental goal of interpretability and introduce success criteria to evaluate how well methods are able to control model behavior through interventions. We unify and extend four popular interpretability methods—sparse autoencoders, logit lens, tuned lens, and probing—into an abstract encoder-decoder framework. This framework maps intermediate latent representations to human-interpretable feature spaces, enabling interventions on these interpretable features, which can then be mapped back to latent representations to control model outputs.
We introduce two new evaluation metrics: intervention success rate and the coherence-intervention tradeoff, designed to measure the accuracy of explanations and their utility in controlling model behavior. Our findings reveal that (1) although current methods allow for intervention, they are inconsistent across various models and features, (2) lens-based methods outperform others in achieving simple, concrete interventions, and (3) interventions often compromise model performance and coherence, underperforming simpler alternatives, such as prompting, for steering model behavior and highlighting a critical shortcoming of current interpretability approaches in real-world applications requiring control. Code is made available for replicability.

## 1 Introduction

As large language models (LLMs) have become more capable and complex, there has emerged a need to better understand and control these models to ensure their outputs are safe and human-aligned. Many interpretability methods aim to address this problem by analyzing model representations, attempting to understand their underlying computational and reasoning processes in order to ultimately control model behaviour.
While many of these methods, and interpretability as a field more broadly, claim control and intervention as abstract goals and present compelling qualitative results demonstrating that intervention is possible (for example, Anthropic’s Golden Gate Claude ; Templeton et al. ()), the link between interpretation and intervention is often tenuous in practice, as many methods are not explicitly tailored for both. Furthermore, even fewer are thoroughly and systematically evaluated for the ability to control model outputs beyond simple qualitative examples.
We believe the reason for this is threefold. First, interpretability methods produce explanations in disparate feature spaces, such as the token vocabulary, probe predictions, or learned auto-interpreted features, hindering comparisons across methods. Second, there exists a “predict/control discrepancy” (Wattenberg & Viégas, ), where the features identified by interpretability methods for predicting behavior are not the same as those used for steering it. Third, there do not exist standard systematic benchmarks to measure intervention success.

Figure: Figure 1: Our proposed intervention framework, which encodes model latent representations, $x$, into human-interpretable features, $z=xD$, that can then be perturbed to $z^{\prime}$ and decoded back into counterfactual latent representations, $\hat{x}^{\prime}$.
Refer to caption: x1.png

In this work, we view intervention as a fundamental goal of interpretability, and propose to measure the correctness of interpretability methods by their ability to successfully edit model behaviour.
In particular, we focus on sparse autoencoders
(Cunningham et al., ; Rajamanoharan et al., ; Templeton et al., ; Bricken et al., ; Gao et al., ), Logit Lens
(; Dar et al., ), Tuned Lens  (Belrose et al., ; Din et al., ), and linear probing
(Alain, ; Belinkov & Glass, ; Belinkov, ), and benchmark them with steering vectors and prompting as baselines for intervention. In order to enable comparison across these various methods, we first unify and extend the methods as instances of an abstract encoder-decoder framework, where each method encodes uninterpretable latent representations of language models into human-interpretable features and the decoder of the framework inverts this mapping, allowing us to reconstruct a latent representation from the features. Under this abstract framework, we can intervene on the interpretable feature activations generated by each method and decode them into latent counterfactuals, which produce counterfactual outputs corresponding to the desired intervention.

The unifying feature interpretation and intervention framework allows us to propose two standard metrics for evaluating mechanistic interpretability methods: (1) intervention success rate, which measures how well intervening on an interpretable feature causally results in the desired behavior in the model outputs, and (2) coherence-intervention tradeoff, which measures how well the causal interventions succeed without damaging the coherence of the model’s outputs. We evaluate Logit Lens, Tuned Lens, sparse autoencoders, and linear probes, for these metrics on GPT2-small, Gemma2-2b, and Llama2-7b, comparing them to simpler but uninterpretable baselines of steering vectors and prompting.
Our results show that while existing methods allow for intervention, they are inconsistent across features and models. Furthermore, lens-based methods outperform all other methods including sparse autoencoders for simple, concrete features. This is likely due to frequent mislabeling of the learned SAE features, where a feature labeled “references to coffee” actually seems to encode for ‘beans’ and ‘coffee beans’ more than ‘coffee’. Similarly, probes and steering vectors are prone to learning spurious correlations in their training data, whereas Logit Lens and Tuned lens do not suffer from the same sensitivities. Finally, we show that intervention often comes at the cost of model output coherence, underperforming simple prompting baselines, presenting a critical shortcoming of existing methods in real-world applications that require control and intervention. We conclude this work with detailed takeaways about the strengths and weaknesses of each method, including discussion of which methods are optimal for specific intervention topics, which are best to use out of the box, and which hold the most promise for future development.

Our main contributions include:

- •
In Section 3.1, we present an encoder-decoder framework that unifies four popular mechanistic interpretability methods: sparse autoencoders, logit lens, tuned lens, and probing.
- •
In Section 3.2, we propose two evaluation metrics for encoder-decoder interpretability methods, namely (1) intervention success rate; and (2) the coherence-intervention tradeoff to evaluate the ability of interpretability methods to control model behavior. We also provide a dataset for benchmarking state-of-the-art LLMs on these metrics.
- •
In Section 4, we perform experimental analysis on GPT-2, Gemma2-2b, and LLama2-7b, and present detailed takeaways comparing interpretability- and control-based methods.

We hope that this paper encourages future work in the interpretability community to make explicit the intent and application of methods, particularly in reference to control and intervention, and to evaluate and benchmark causality and utility in application.

## 2 Related Work

Mechanistic Interpretability.
Existing work in mechanistic interpretability broadly falls into two categories: activation patching and interpreting hidden representations. Activation patching utilizes carefully constructed counterfactual representations to study which neurons or activations play key roles in model computation, ideally localizing specific information to individual layers, token positions, and paths in the model (Geiger et al., ; Vig et al., ). However, recent work points to key limitations of patching, particularly with respect to real-world utility in downstream applications such as model editing (Hase et al., ; Zhang & Nanda, ). As such, we focus primarily on methods for inspecting hidden representations, of which probes are the most commonly used  (Alain, ; Belinkov & Glass, ; Belinkov, ). Other methods such as ; Dar et al. () project intermediate representation into the token vocabulary space, with Belrose et al. (); Din et al. (); Geva et al. () building and improving upon these early-decoding strategies. Ghandeharioun et al. () unifies most of these methods into an abstracted framework for inspecting model computation. More recently, sparse autoencoders and dictionary learning have been explored as a solution to the uninterpretability of model neurons, particularly due to issues with polysemanticity and superposition (; Bricken et al., ; Cunningham et al., ; Bhalla et al., ; Gao et al., ; Rajamanoharan et al., ; Templeton et al., ; Karvonen et al., ; Dunefsky et al., ).

Evaluation. Due to the recency of the field, standard evaluation metrics across interpretability methods have not yet been established, and similar to the broader interpretability field, evaluation is frequently ad-hoc and primarily qualitative in nature. With regards to quantitative metrics, in Gao et al. (); Templeton et al. (); Makelov et al. (), sparse autoencoders are evaluated for reconstruction error, recovery of supervised or known features, activation precision, and the effects of ablation; however, none of these metrics measure the human interpretability or correctness of explanations. Sparse autoencoder evaluation has also sometimes included ratings by humans to determine the interpretability of the learned features for labeling, as done in Rajamanoharan et al. (), but such measures do not necessarily reflect downstream utility in model control.

Causal Intervention. Previous literature on probing frequently evaluates learned probes and features through intervention to ensure causality and correctness, as done in Li et al. (); Chen et al. (); Hernandez et al. (; ); Marks & Tegmark (). The interventions performed for measuring causality are similar to those used to perform model “steering” Rimsky et al. (); Panickssery et al. (); Ghandeharioun et al. () and should ideally produce the same effect but with the added claim of interpretability. Beyond probing, only Belrose et al. (); Chan et al. (); Olah et al. (); Templeton et al. () consider causal intervention as a tool for evaluation. However, Belrose et al. (); Olah et al. (); Chan et al. () mainly focus on mean ablation for intervention, with no exploration of the coherence of the counterfactual outputs or their utility in application. Templeton et al. () on the other hand only provides a qualitative evaluation of intervention via their ‘Golden Gate Claude’ [] and does not compare to other interpretability methods.

## 3 Method

In this section, we first introduce a unifying framework for four common mechanistic interpretability methods: sparse autoencoders, Logit Lens, Tuned Lens, and probing, along with modifications to these methods that permit principled intervention on representations. We then propose evaluation metrics for (1) testing the correctness of explanations via intervention and (2) the usefulness of these methods for steering and editing representations and model outputs.

### 3.1 Unifying Intervention Framework

##### Latent vectors to interpretable features.

The central aspect of most interpretability work is the ability to translate model computation into human-interpretable features, whether the computation be latent directions, neurons, components, reasoning processes, etc. Many works aiming to explain LLMs focus particularly on hidden representations, where the mapping between high-dimensional dense embeddings and human-interpretable features is modeled through a (mostly) linear dictionary projection or affine function:

$$ $\displaystyle z$ $\displaystyle=f(x)=\sigma(x\cdot D)$ $\displaystyle\hat{x}$ $\displaystyle=g(z)\approx f^{-1}(z)=z\cdot D^{-1}$ $\displaystyle z^{\prime}$ $\displaystyle=\text{Edit}(z),~{}~{}\hat{x^{\prime}}=g(z^{\prime})$ $$

where each $z_{i}$ is a feature activation, each $i$ in $D$ corresponds to a human-interpretable feature(^1^11While “features” are not necessarily well-defined in the interpretability community, we find that the second definition in most closely aligns with our own notion of features as human-interpretable concepts that are relevant for manipulation and editing.), and $\sigma$ is an activation function that is frequently the identity.
In the case of sparse autoencoders, $D$ is a learned, overcomplete dictionary, with $N$ in range 16k - 65k for small models (up to 16M for large models), and $\sigma$ is a ReLU, JumpReLU, or ReLU + top-k activation function. Given that SAE features are learned, they are not immediately interpretable and must be labelled by humans or strong LLMs. For Logit Lens, $D$ is simply the language model’s unembedding matrix, meaning each feature corresponds to a single token in the vocabulary. For Tuned Lens, $D$ is the exact same as Logit Lens but with a learned linear transformation applied. Linear probes can be thought of as a learned dictionary with $N=1$ where $\sigma$ is a sigmoid or softmax activation. Of all these methods, Logit Lens is the only method that does not require any training data, and sparse autoencoders are the only method that does not produce immediately interpretable features. Please see Figure 1 for a visual summary of these methods.

##### Interpretable features to counterfactual latent vectors.

While producing explanations is straightforward for each of these methods, intervening on model representations using the information provided by explanations is not as simple. Doing so requires defining a reverse mapping from the explanations to the latent representations of the model, which is only explicitly done by sparse autoencoders.
We defined the inverse mappings for the rest of the functions as follows.
To map Logit Lens’s explanations back into the model’s latent space, we would ideally apply the inverse of the unembedding matrix to $z$; however, in practice this is often ill-conditioned due to the dimensionality of $D$. As such, we instead use the low-rank pseudoinverse of the unembedding matrix and right-multiply it to the explanation logits. Similarly, for Tuned Lens, we model the decoding process through the pseudo-inverse of the Tuned Lens projection applied to the unembedding matrix. Notably, both of these methods only require a simple linear transformation to go back-and-forth between latent vectors and explanations. For probing, $D^{-1}$ can be defined such that $\hat{x}^{\prime}=x+\theta$, as proposed by Chen et al. () which adds the linear probe’s weights to the latent representation, bypassing the need for optimization. Sparse autoencoders have a well-defined backwards mapping through the SAE decoder, which is frequently linear in practice and often the transpose of the encoder weights. Thus, we are now able to decode $z$ into $\hat{x}$ for all methods.

##### Intervening on interpretable features.

Given the above framework, intervention is performed by directly altering the feature activation $z_{i}$ corresponding to the desired feature $i$ to be edited. While the edited activation $z_{i}^{\prime}$ can naively be set to some constant value $\alpha$, the same constant may have drastically varying effects for different tokens and different prompts. As such, to take into account the context of $z$, for Logit Lens, Tuned Lens, and SAEs we set $z_{i}^{\prime}=\alpha*\max(z)$. This ensures that the feature $i$ is the most dominant feature in the latent vector for $\alpha>1$. Decoding $z^{\prime}$ yields the altered latent representation $\hat{x}^{\prime}=g(z^{\prime})$, which accounts for both the error of the explanation method as well as the intervention performed. For probing and steering vectors, $\hat{x}^{\prime}=x+\alpha*v$, where $v$ is the steering vector or the weights of the linear probe. Note that $\alpha$ is a hyperparameter that must be tuned for each method, model, and sometimes even intervention feature and thus cannot be used to compare the effects of interventions across methods. In order to do so, we can instead measure the normalized difference between the latent vectors $x$ and $\hat{x}^{\prime}$, to better understand the relationship between the degree of intervention performed and our various evaluation metrics. We also note that $\hat{x}$ and $\hat{x}^{\prime}$ are not necessarily in-distribution for the language model, but due to the additive nature of the residual stream, the linear representation hypothesis, and the linear nature of the decoder/unembedding matrix, we believe that such interventions may still be principled in practice (see Park et al. () for more on the linear representation hypothesis and intervention).

##### Open-ended generation.

In order to generate open-ended text after intervening on the explanation, we edit the corresponding representations in place, as is common practice with prior steering methods. Formally, the representation $x_{t}$ at token position $t$ and layer $l$ is edited to be $\hat{x_{t}}^{\prime}$, ensuring a causal effect on all ensuing tokens $x_{t+1},x_{t+2},...,x_{T}$.

### 3.2 Comparative Evaluation Across Methods and Models

Given the overall lack of standardized evaluation of mechanistic interpretability methods, we intend for this work to serve as a starting point for systematic evaluation by evaluating methods in simple, easy-to-measure contexts. In particular, we think of our evaluations as measuring a kind of upper bound for these methods: in the easiest of settings, how well do existing methods work?

##### Explanation Correctness

We first propose metrics to evaluate the correctness of explanations and interventions. More specifically, to test whether a single feature of an explanation $z_{i}$ is correct, we intervene on that feature to produce $z_{i}^{\prime}$ and decode $z^{\prime}$ to $\hat{x}^{\prime}$. We can then hypothesize about the corresponding change in the model’s output when generating from $\hat{x}^{\prime}$ beforehand, as it should match the intervention made to produce $z^{\prime}$. For example, if feature $i$ encodes the concept “references to Paris,” increasing the value of $z_{i}$ should result in increases to references of Paris in the model’s output. From this, we propose a metric of Intervention Success Rate, which measures if increasing the activation of feature $i$ by increasing $z_{i}$ results in the appropriate increase of the feature $i$ in the model’s output.
To evaluate a continuous relaxation of this, we can also similarly measure the probability assigned to tokens relating to feature $i$. As such, even if the model’s output does not directly reflect interventions made to $z_{i}^{\prime}$ due to sampling, we can measure if increasing the activation of $i$ results in any change to the model’s output at all. We refer to this metric as Intervened Token Probability. Importantly, both of these metrics can be thought of as measuring the causal fidelity of the features highlighted by explanations.

##### Usefulness of Intervention Methods

While intervention is a useful method for evaluating the correctness of explanations, it is also a desideratum of its own and a frequent motivation for many explanation methods. For example, methods are often developed for the purpose of de-biasing model outputs or increasing model safety, either by localizing bad behavior or identifying it at inference time, thus allowing for targeted edits to be made. However, a lack of this direct connection between interpretation and model intervention has led to illusory results in prior literature, similar to predict/control discrepancies (Hase et al., ; Wattenberg & Viégas, ). By directly and explicitly measuring how effective interpretability methods are at allowing for targeted intervention or steering, we can avoid such failure cases. Importantly, intervention is only useful if the language model retains its overall performance and the text output by the model after the intervention still satisfies the purpose of the query as well as the intervention. In particular, we are interested in whether interpretability methods can be used to steer model outputs towards feature $i$ without damaging the model’s purpose as an LLM. We do so by measuring the Coherence of the model’s outputs after the intervention has been applied. This can be done by any oracle, such as a human or another capable LLM. Similarly, we can also measure the Perplexity of the intervened outputs with respect to a strong language model. In practice, we use Llama3.1-8b for both of these metrics, as it is reasonable sized, high-performing, and open source, allowing for the measurement of perplexity. To measure coherence, we use the prompt
“Score the following dialogue response on a scale from 1 to 10 for just grammar and comprehension, ignoring incomplete sentences. \n\n Context: {Prompt}\n Dialogue Response: {Output}\n\n Score =”.

##### An Open-ended Evaluation Dataset.

In order to evaluate these methods to the best of their capabilities, we are interested in assessing their ability to intervene when intervention is straightforward and possible given the prompt. Consider the question “What is $\iint sin(3*x)*cos(y)dxdy$?”. Intervening on the model’s output for this prompt with a feature related to unicorns is not necessarily intuitive, as there is a correct answer to the prompt that is entirely unrelated to the intervention topic. As such, we want to evaluate these methods on prompts that allow for steering towards a variety of topics or features. To that end, we construct a dataset of 210 prompts related to poetry, travel, nature, journaling prompts, science, the arts, and miscellaneous questions that could plausibly be answered while satisfying a variety of intervention topics. All prompts are open ended to allow for many potential answers. Some example prompts include “In ten years, I hope to have accomplished”, “Something cool I learned about from a Biology Ph.D. student was”, and “What is your favorite dad joke?”. This dataset is released as part of the accompanying GitHub repository.

## 4 Experiments

In this section, we evaluate the five methods on Gemma2-2b Team et al. (), Llama2-7b Touvron et al. (), and GPT2-small Radford et al. (). We first measure the completeness of the explanations produced in Section 4.2. We then benchmark the methods on the metrics described in Section 3.2, providing quantitative results in Sections 4.3 and 4.4 and qualitative results in Section 5. Finally, we present an analysis of the empirical directional similarities between methods in Section 4.5, and analyses of the efficacy of methods at different model depths in Appendix A.6.

### 4.1 Implementation Details

Intervention Topics. We pick 10 intervention topics that all relate to references to specific words or phrases: {‘beauty’, ‘chess’, ‘coffee’, ‘dogs’, ‘football’, ‘New York’, ‘pink’, ‘San Francisco’, ‘snow’, ‘yoga’}. These simple, low-level features are ideal for evaluation through intervention for four key reasons: first, measuring the presence of a word or phrase is much easier than measuring a high-level abstract concept such as sycophancy, second, these features were available across many layers for the Gemma2-2b and GPT2-small sparse autoencoders, third, the features necessarily exist in the Logit Lens unembedding dictionary, and finally, datasets that are labelled for the presence of these features are very straightforward to collect for generating steering vectors and probes. As such, we can easily compare intervention on these features across methods and also easily measure intervention success by checking if the given word/phrase exists in the model’s output.

**Table 1: Normalized latent reconstruction error without intervention.**
| Method | Gemma2-2b | Llama2-7b | GPT2-small |
| --- | --- | --- | --- |
| Logit Lens | 0.52 | $5e^{-5}$ | 0.22 |
| Tuned Lens | – | $5e^{-3}$ | 0.32 |
| SAEs | 0.38 | – | 1.64 |

Steering vectors and probing. We implement steering vectors as described in Rimsky et al. () with a few simple modifications. Where in Rimsky et al. (), the difference between contrastive pairs is taken only at the last token, we find that averaging across the token dimension and taking the difference between those averages yields much better results. This is due to the fact that in Rimsky et al. (), the only difference between representations occurred in the token position of the answer letter, or the last token; however, in our case the information related to the intervention feature could be present at any token. Contrastive data pairs were generated by prompting ChatGPT for 200 pairs sentences or phrases about the desired intervention topic and about other topics within the same genre. For example, we used the prompts “Can you please generate 100 short sentences or phrases with the color ‘pink’ in them?” to create positive samples and “Can you also generate 100 phrases or sentences with other colors (not pink) in them?” to create negative ones. We also use these contrastive pairs to train the linear probes, using the implementation from Chen et al. (). All probes reached train and test accuracies of 100% across all models and intervention topics.

Sparse autoencoders. We focus specifically on sparse autoencoders trained to interpret the residual stream of transformer models.
We use the SAELens library from Bloom () for GPT2-small and the Gemma Scope SAEs released by Google in Lieberum et al. () for Gemma2-2b. For both models, SAE feature labels were found via Neuronpedia Lin & Bloom (), which allows users to interact with, explore, and search for auto-interpretation labelled learned features. Note that the Gemma SAEs are of the JumpReLU architecture from Rajamanoharan et al. () whereas the GPT2-small SAEs are standard ReLU + L1 regularization SAEs, as noted in Bloom ().

### 4.2 Sanity Checking Explanation Reconstructions

Before testing these methods for their ability to intervene, we first want to evaluate the completeness of the explanations and the effect of replacing $x$ with $\hat{x}$ without any intervention or editing. We do so by measuring the normalized latent reconstruction error:
$\texttt{Error}=||\hat{x}-x||/||x||$ where $\hat{x}=g(f(x))=g(z)$. This error is a key part of the loss function that sparse autoencoders are trained on and measures the information loss incurred by mapping between the language model’s latent space and the interpretable feature space. Given that steering vectors and linear probes do not output complete explanations, we only measure this error for the other three methods, as shown in Table 1, where we see that errors vary a lot across models but most methods are relatively consistent in their error, with the exception of the GPT2-small sparse autoencoders.

We also measure the coherence of the outputs produced by replacing $x$ with $\hat{x}$, as shown in Appendix Figure 7. We find that the coherence of the outputs generated by the reconstructed latents generally matches the coherence of the clean model outputs. We use a deviation of $\pm 1$ around the mean of clean output coherence scores as a threshold for future evaluations, shown in the dashed lines.

Figure: Figure 2: Evaluation of the Intervention Success Rate for each method. Note that normalized edit distance is a proxy for intervention intensity that is comparable across methods. Logit Lens outperforms all other methods across all models.
Refer to caption: x2.png

### 4.3 Intervention Success Across Models

As described in Section 3.2, in order to evaluate the correctness of explanations, we measure the causal effects of intervening on specific features of each explanation. For a given feature or intervention topic $i$, we see if increasing the activation of that feature results in an increase of the feature in the model’s output in the top row of Figure 2. In order to compare across methods, which all have different explanation feature spaces and scales, we measure the success of interventions as a function of the norm of the distance between the edited latent representation $\hat{x}^{\prime}=g(Edit(f(x)))$ and the original latent representation $x$: $||\hat{x}^{\prime}-x||/||x||$ as done in Section 4.2. Results for intervened token probability are in Appendix A.4.

We find that across all three models, Logit lens and Tuned lens have the highest intervention success rate, regardless of the normalized edit distance. While increasing intervention strength and editing the latent representations more for steering vectors and probes results in greater intervention success for GPT2-small and LLama2-7b, the same is not true for Gemma2-2b, for which intervening with steering vectors and probes seems to have minimal effect. Furthermore, we find that intervening with sparse autoencoders is moderately successful, but due to the large reconstruction error for GPT2-small, even with intervention, the edit distance is much greater than that needed for Logit lens and Tuned lens. In general, we believe the lower performance of Sparse Autoencoders is due to heavy noise in the labels of features. For example, a feature labelled ‘references to coffee’, is sometimes actually a feature that encodes for references to ‘beans’ and ‘coffee beans’, and thus only sometimes increases mentions of ‘coffee’. Probes and steering vectors also have suboptimal performance, often due to learning of spurious correlations in the training data rather than the true intervention feature.

When measuring token probability rather than output presence, as explored in Appendix A.4, we see that intervention with all methods across all models increases the probability of intervention-related tokens, even if the intervention does not succeed. We also note that there is a significant difference between the order of magnitude of the intervened token probability for sparse autoencoders, around $10e^{-5}$ and the rest of the methods, which range from $10e^{-4}$ to 0.5.

### 4.4 Effects of Intervention on Output Quality

We next measure the coherence of the intervened output text produced by each method to ensure that intervention through mechanistic interpretability methods is possible in applications and does not damage the utility of the model. We measure coherence as described in Section 3 as a function of the normalized latent edit distance $||\hat{x}^{\prime}-x||/||x||$ and with respect to the intervention success rate in Figure 4. We visualize the mean of coherence scores for the clean model outputs with solid black horizontal lines, the same as those shown in Figure 7, with a buffer of $\pm 1$ around the mean in dashed lines. We also consider a prompting baseline, where we simply prompt the language model to constantly talk about the intervention topic, to better understand the optimal coherence possible while satisfying the intervention. This is shown by the teal stars in 4. Given that only Llama2-7b was instruction-tuned out of the models evaluated, prompting was less successful for Gemma2-2b and infeasible for GPT2-small. Also note that the intervention success rate approaches near 100% with prompting as the number of generated tokens increases; however, seeing as we only generate 30 tokens, the success rate is slightly lower than expected.

We find that for GPT2-small and Llama2-7b, intervention with most methods results in a sharp decline in text coherence, with Logit Lens and Tuned Lens having the most dramatic fall-off. In fact, we find that even with a full point drop in coherence from the baseline models’ coherence (denoted with the black lines), the representations can barely be edited by any methods except for Gemma2-2b. While at first, this seems quite concerning, as this would prevent the real-world application of these methods to scenarios necessitating high model performance, viewing coherence as a function of intervention success rate tells a different story, as shown in Figure 4. In fact, we see that Logit Lens and Tuned Lens have relatively good coherence-intervention tradeoffs for GPT2-small and LLama2-7b, with success rates of around 0.5 and 0.6 respectively when considering outputs within one point of deviation from the mean coherence score of the clean model. All other methods exhibit much less desirable Pareto curves. We also note that as the models increase in size and performance, with GPT2-small being the smallest and worst language model and Llama2-7b being the largest and most performant, the Pareto curves are shifted up and to the right more for most methods, suggesting that targeted intervention is more possible as models become more capable.

Figure: Figure 3: Intervened model output coherence measured with respect to normalized edit distance. The solid horizontal line shows the mean of coherence scores for the clean model outputs, and the dashed lines show $\pm$1 standard deviation around the mean.
Refer to caption: x6.png

Figure: Figure 4: Intervened output coherence measured with respect to intervention success rate. The solid horizontal line shows the mean of coherence scores for the clean model outputs, and the dashed lines show $\pm$1 standard deviation around the mean.
Refer to caption: x10.png

Qualitative Examples. In addition to our empirical results, we present examples of the output text for all intervention methods in Figure 5 for qualitative evaluation of intervention on the feature ‘yoga’. Examples for the “Optimal intervention strength” (left column) were randomly chosen from the outputs where intervention succeeded and coherence was still relatively high. Examples for “Excessive Intervention” were randomly chosen from the outputs of the highest intervention strength tested (right column). Note that intervention results in infinite repetition at very high intervention strengths for all methods; however, only Logit Lens and Tuned Lens result in repetition of tokens related to ‘yoga’. More examples are provided in Appendix A.7.

Figure: Figure 5: Examples of intervened model outputs for intervention feature ‘yoga’ at both the optimal intervention strength (left) and the maximum intervention strength tested (left). Outputs degrade into incoherent repetition at high intervention strength for all methods.
Refer to caption: x14.png

### 4.5 Intervention Similarity Between Methods

We finally evaluate the empirical similarity between each of these methods. By converting each intervention into one in the latent space of the model, we can measure the cosine similarity between edit directions for each method. More specifically, for each method and intervention topic, we compute the expectation of $\hat{x}^{\prime}-x$ over all samples. This difference vector can be thought of as the direction of the intervention topic in the model’s latent space. We compute the cosine similarity between these vectors for each intervention topic and display the average across topics in Figure 6.

We find that Logit Lens and Tuned Lens are highly similar, as expected. Similarly, steering vectors and probe weights tend to lie in similar directions, likely due to the same underlying data used to train both. Most interestingly, we find that sparse autoencoders tend to intervene in somewhat similar directions to steering vectors and probes. Even more surprisingly, they have near orthogonal directions to Logit Lens and Tuned Lens, even when interventions succeed for all methods. This is likely due to the high-dimensionality of model latent spaces, allowing for many vectors to be near-orthogonal. We speculate that sparse autoencoders may be more similar to probes and steering vectors because the three methods may have a bias toward representing past information and tokens, due to their training and labelling algorithms. Logit Lens and Tuned lens, on the other hand, are designed to reveal information about the next token specifically, given that they are early-decoding strategies and thus may contain more information about model outputs rather than inputs.

Figure: Figure 6: Cosine similarity of intervention directions in model latent space across methods. Logit Lens and Tuned Lens are highly similar, as well as steering vectors and probes.
Refer to caption: x15.png

### 4.6 High-Level Comparison of Methods

Given the differences between methods, including how their features are learnt, what explanations they produce, and the causal fidelity and utility of explanations for steering, we present the following characterization of the strengths and weaknesses of each method.

Logit Lens is easy to use, requires no training, and maps features directly to vocabulary tokens, making it highly interpretable, and we find that it generally has high causal fidelity as well. However, its predefined, static features are limited and rudimentary, preventing its utility for steering in many real-world or safety-critical applications where abstract features such as truthfulness and bias are more relevant.Tuned Lens shares these traits but requires an additional learned affine transformation, although many are already open-sourced. Sparse autoencoders learn a very large dictionary, allowing for features to be both low-level and high-level or abstract, potentially including safety-critical features, and ideally “covering” all the true underlying features learned by the model. Furthermore, explanations are sparse, increasing their human interpretability. However, the learned features are frequently not human-interpretable, leading to labeling errors and lower causal fidelity in practice. Furthermore, they require significant training and labeling data and compute, and there is no guarantee that a desired feature will exist in the SAE’s dictionary, also preventing their utility in safety-critical domains.
Probing allows for feature specification, given that a labeled dataset is available, but is prone to learning spurious correlations, leading to low causal fidelity and minimal steering utility. Steering vectors are very similar to probes in practice, but they require no optimization and are slightly less sensitive to correlations in the training data. As such, we believe that lens-based methods are most useful for providing high-fidelity explanations but are likely not effective solutions for steering in real applications, for which steering vectors and prompting are more promising but require careful oversight and refinement to ensure efficacy. SAEs offer a balance between interpretation- and control-based methods but also suffer from a union of their problems.

## 5 Conclusion

As language models are integrated into everyday aspects of life, whether benign or life-changing, being able to understand them and ensure they behave as expected is of utmost importance. While interpretability methods show great promise, the correctness of their interpretations is less clear. In this work, we emphasize that intervention is a fundamental goal of interpretability, both to evaluate the correctness of interpretability methods as well as their utility in real-world applications.
By unifying and extending four popular interpretability methods — sparse autoencoders, logit lens, tuned lens, and probing — into an abstracted encoder-decoder framework, we enable structured interventions on human-interpretable features, mapfping them back to model latent representations to directly control outputs.
Through the introduction of two standardized metrics, the intervention success rate, and the coherence-intervention tradeoff, we provide a rigorous basis for evaluating the causal fidelity and utility of interpretability methods. Our findings reveal significant inconsistencies in intervention capability across different methods and models, with simpler lens-based approaches generally outperforming the more compute-intensive sparse autoencoders for simple, concrete interventions. Furthermore, non-interpretability-based approaches, such as prompting, perform best overall, demonstrating a significant gap in current interpretability approaches.

#### Acknowledgments

The authors would like to thank Fred Zhang and Lucas Dixon for their helpful discussion and comments on the manuscript.
This work is supported in part by the NSF awards IIS-2008461, IIS-2040989, IIS-2238714, AI2050 Early Career Fellowship by Schmidt Sciences, and faculty research awards from Google, OpenAI, Adobe, JPMorgan, Harvard Data Science Initiative, and the Digital, Data, and Design (Dˆ3) Institute at Harvard. UB is funded by the
Kempner Institute Graduate Research Fellowship. The views expressed here are those of the authors and do not reflect the official policy or position of the funding agencies.

## Appendix A Appendix

### A.1 Additional Implementation Details: Intervention Hyperparameter α 𝛼 \alpha italic_α

When intervening on $z$ to get $z^{\prime}$ with Logit Lens, Tuned Lens, and SAEs, we set $z_{i}^{\prime}=\alpha*max(z)$. For probing and steering vectors, $\hat{x}^{\prime}=x+\alpha*v$,
where $v$ is the steering vector or the weights of the linear probe. Note that $\alpha$ is a hyperparameter that must be tuned for each method and model, and thus cannot be used to compare the effects of interventions across methods. We record the values of $\alpha$ used in our experiments in Table 2.

**Table 2: .**
| Method | GPT2-small<br>Layer 9 | Gemma2-2b<br>Layer 20 | Llama2-7b<br>Layer 18 |
| --- | --- | --- | --- |
| Logit Lens | [50, 70, 90, 110, 130] | [100, 130, 160, 200, 230] | [0.5, 3, 7, 11, 15, 19] |
| Tuned Lens | [20, 25, 30, 35, 40] | – | [1, 7, 11, 15, 19, 23] |
| SAEs | [3, 4, 5, 6] | [1, 2, 3, 4, 5] | – |
| Probing | [150, 200, 250, 300, 350] | [200, 250, 300, 350] | [10, 90, 110, 130, 150] |
| Steering Vectors | [2, 4, 6, 8, 10] | [3, 4, 5, 6] | [0.5, 3, 4, 5, 6] |

### A.2 Additional Implementation Details: SAE Features

As described in Section 4.1, we use the sparse autoencoders hosted on SAELens and find the relevant features with Neuronpedia’s exploration and search tools. We document all of the features we consider for each intervention topic in Table 3. Note that for some specified intervention topics, an exact feature match does not exist for the GPT or Gemma SAEs. As such, we either exclude that topic or consider the closest-related topic (such as “instruction related to yoga poses and their benefits” when what we would like is “references to yoga”). Many of these imperfect features still yield reasonable intervention success rates.

**Table 3: .**
| Intervention<br>Feature | GPT2-small<br>Layer 9<br>Feature | GPT2-small SAE<br>Layer 9 Name | Gemma2-2b<br>Layer 20<br>Feature | Gemma2-2b SAE<br>Layer 20 Feature Label |
| --- | --- | --- | --- | --- |
| San Francisco | 11233 | “mentions of the city<br>of San Francisco” | 3124 | “references to San Francisco<br>and related locations” |
| New York | 5831 | “references to the city<br>of New York” | 3761 | “specific place names and<br>geographical locations in New York” |
| beauty | 1805 | “words related to beauty<br>or aesthetic appreciation” | 485 | “instances of the word<br>“beauty” in various contexts” |
| football | – | – | 11252 | “references to football<br>and baseball contexts” |
| pink | 2415 | “mentions of the<br>word “Pink.”” | 13703 | “references to the color<br>pink and its various associations” |
| dogs | 12435 | “mentions of dogs or<br>dog-related terms” | 12082 | “references to dog behavior<br>and interactions” |
| yoga | – | – | 6310 | “instructions related to<br>yoga poses and their benefits” |
| chess | 21685 | “mentions of the<br>game of chess” | 13419 | “elements within<br>the context of chess” |
| snow | 5053 | “references to<br>snow-related terms” | 13267 | “references to snow and<br>related terms” |
| coffee | 23472 | “references to<br>coffee-related words” | 15907 | “references to coffee and<br>related cafés or establishments” |

### A.3 Additional Evaluations: Coherence of Method Outputs without Intervention

We measure the coherence of the outputs produced by replacing $x$ with $\hat{x}$, as shown in Appendix Figure 7, which we can compare to the baseline of the clean model outputs (labelled ‘Clean’ and shown in black). We find that the coherence of the outputs generated by the reconstructed latents generally matches the coherence of the clean model outputs. We use a deviation of $\pm 1$ around the mean of clean output coherence scores as a threshold for future evaluations, shown in the dashed lines.

Figure: Figure 7: Histogram of coherence scores for clean model outputs (Clean) and for the models where $x$ is replaced by $\hat{x}$ without any intervention for Logit Lens, Tuned Lens, and SAEs. Dashed lines show the mean for each distribution.
Refer to caption: x18.png

### A.4 Additional Metrics: Intervened Token Probability

Please see Section 3.2 for more details. We measure the probability assigned to tokens relating to feature $i$ when intervening on feature $i$. As such, even if a model’s output does not directly reflect interventions made to $z_{i}^{\prime}$ due to sampling, we can measure if increasing the activation of feature $i$ results in any change to the model’s output at all. We refer to this metric as Intervened Token Probability. Importantly, both of these metrics can be thought of as measuring the causality of the features highlighted by explanations.

Results for Intervened Token Probability are shown in 8, where we see that intervention with all methods across all models increases the probability of intervention-related
tokens, even if the intervention does not succeed. We also note that there is a significant difference
between the order of magnitude of the intervened token probability for sparse autoencoders, around
$10e^{-5}$ and the rest of the methods, which range from $10e^{-4}$ to 0.5.

Figure: Figure 8:
Refer to caption: x21.png

### A.5 Additional Metrics: Perplexity

As described in Section 3.2, we evaluate the perplexity of the intervened generated text to measure the utility of interpretability methods for targeted intervention in 9. We measure this perplexity with respect to a stronger language model than the one studied, in this case with Llama3.1-8b.

We find that the results for perplexity are generally unintuitive and do not align with the results for coherence. We hypothesize that perplexity is not a useful measure when text is extremely out-of-distribution with respect to normal text, and in particular when the text is highly repetitive. For example, if the same token is repeated 20 times, we (and other language models) might assume that the next 20 tokens would also be the same, resulting in a low perplexity even if the quality of the text is poor. As such, we do not consider these results to be particularly meaningful or significant.

Figure: Figure 9:
Refer to caption: x25.png

### A.6 Intervention Efficacy Across Model Depth

In order to ensure the generalizability of the above results across layer depths, we repeat all experiments for each layer of GPT2-small, as shown in FIgure 10. However, due to some sparse autoencoder features only existing in some layers, we could only consider intervention topics { ‘beauty’, ‘coffee’, ‘dogs’}. We hold the hyperparameter $\alpha$ that controls for intervention “strength” constant across all layers. Note that this is NOT equivalent to holding the normalized edit distance constant, as shown in the rightmost plot.

We find that layer depth seems to have minimal effect for SAEs and probing, with the exception of the first and last layers. For steering vectors, we observe a modest increase in intervention success rate with increased layer depth and a much more drastic increase in the success rate at later layers for Logit Lens and Tuned Lens. However, as we increase $\alpha$ significantly, we find that the curves for all three methods on intervention rate shift left until the pass rate is approximately 1 at all layers. Intuitively, this makes sense, as any edits to the residual stream at layer 0 will affect the residual stream at later layers. We note that these results highlight the need to tune the intervention strength for each method, each model, and each layer - limiting their ease of use.

Figure: Figure 10: layers
Refer to caption: x28.png

### A.7 Additional Example Outputs

We present additional examples of the output
text for all intervention methods in Figures 11 and 12 for qualitative evaluation of intervention on the feature
‘coffee’ and ‘San Francisco’. Examples for the “Optimal intervention strength” (left column) were randomly chosen from
the outputs where intervention succeeded and coherence was still relatively high. Examples for “Excessive Intervention” were randomly chosen from the outputs of the highest intervention strength tested (right column). Please see Section 4.4 for more.

Figure: Figure 11: Example outputs with intervention on “coffee” feature.
Refer to caption: x31.png

Figure: Figure 12: Example outputs with intervention on “San Francisco” feature.
Refer to caption: x32.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - As LLMs are intervened, edited, or steered with guardrails, behavior changes and concept drift may affect feature mapping and intervention fidelity. Does the framework account for this, or would it require retraining to maintain accurate interventions over time?

- How does the framework scale, e.g. for a 70b-200b model? What are the time and memory complexities? 

- Is using Llama3 8b sufficient for coherence? It would be good to get some discussion on the limitations of this and if there are potential alternatives, e.g. using a human teacher etc

- Can the font and line width in the figures be made a little larger?

- In Appendix A.5 there is a broken reference
soundness: 3
strengths: - The encoder-decoder framework consolidates multiple interpretability methods, allowing for unified evaluation and intervention capabilities
- The intervention success rate and coherence-intervention tradeoff metrics are intuitive ways of assessing the effectiveness of interventions and their impact on model coherence
- The paper offers insights into method-specific trade-offs
- Experiments are well-detailed, covering various aspects of intervention efficacy and model coherence, providing a comprehensive assessment across multiple models and features
summary: This paper addresses the disconnect between interpretability and control in LLMs, proposing intervention as a core goal of interpretability. The authors introduce two novel metrics, intervention success rate and coherence-intervention tradeoff. They are designed to evaluate how well interpretability methods can effectively alter LLM outputs. The paper unifies four interpretability methods under an encoder-decoder framework, enabling human-interpretable feature interventions. Evaluated across three models (GPT-2, Gemma2-2b, Llama2-7b), the authors compare the framework against simpler methods like steering vectors and prompting. The study reveals limitations in existing methods, highlighting lens-based approaches as superior for simple interventions but indicating potential issues with performance degradation.
weaknesses: - Lack of experiments on latest and advanced larger models. Llama-2 is not as advanced as the Llama3 to 3.2 family models, Gemma 2b is on the smaller side and GPT2 is quite dated. It would be interesting to see how this worked on some of the SOTA 7B+ models

- Some of the results in table 1 are missing, I couldn't find a reason for this, there could be a valid reason for this but having it in the table caption would be much clearer

- The evaluation primarily focuses on low-level features (e.g., “yoga,” “coffee”), which doesn't offer concrete insights into the framework’s utility for high-level or abstract features. It would be very interesting to see the set of intervention topics include more complex concepts. The authors do note that low-level features are more valuable, which I agree with, testing high level features would still be useful

- The coherence-evaluation setup primarily measures internal coherence without explicitly assessing task-related performance, which may differ in applied settings

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1) Can you please position this paper relative to causal interpretability methods discussed in the review (Mueller 2024) cited above? 

2) Is the goal of this paper to unify "interpretability" and "control" as mentioned in the title, or "unifying and extending four popular interpretability methods" (line 530-531)? These are very different!
soundness: 1
strengths: The idea of inverting the explanations in order to get a "natural" notion of intervention seems promising, especially if it is supplemented with theoretical analysis for why this is better than existing intervention-based evals.
summary: This paper aims to unify a few interpretability methods (SAEs, logit/tuned lens, probing) via a metaphor to encoder-decoder architectures. Specifically, the paper articulates what the "encoder" and "decoder" are for each of these interpretability methods. Explanations are evaluated by intervention-based evals: "intervention success rate" and "intervened token probability". Lens-based methods are claimed to achieve "more simple, concrete" interventions.
weaknesses: While the proposed framework is interesting, this paper is not quite ready for acceptance at ICLR 2025; it overall appears incomplete and rushed.

1. Missing large areas of related work, which significantly weakens the claim of "unification", and undermines some claims of novelty. 

- There is a whole subfield of intervention-based evals for interpretability; see the review paper below. This undermines the novelty of "we propose intervention as a fundamental goal of interpretability and introduce success criteria to evaluate how well methods are able to control model behavior though interventions" (Abstract, lines 18-20).

Mueller, A., Brinkmann, J., Li, M., Marks, S., Pal, K., Prakash, N., Rager, C., Sankaranarayanan, A., Sen Sharma, A., Sun, J., Todd, E., Bau, D., & Belinkov, Y. (2024). The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv. https://arxiv.org/abs/2408.01416

- The closest citation to this field is Geiger et. al. (2021) in line 127, which is neither recent nor representative of causal abstractions. 

- Consequently, the "intervention success rate" of Section 3.2 (which is barely defined, in lines 112, 242-244) seems quite similar to Interchange Intervention Accuracy, which has already been well-studed: see for instance

Geiger, A., Lu, H., Icard, T., & Potts, C. (2021). Causal abstractions of neural networks. arXiv. https://arxiv.org/abs/2106.02997

Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., & Icard, T. (2024). Causal abstraction: A theoretical foundation for mechanistic interpretability. arXiv. https://arxiv.org/abs/2301.04709

Geiger, A., Wu, Z., Potts, C., Icard, T., & Goodman, N. (2024). Finding alignments between interpretable causal variables and distributed neural representations. arXiv. https://arxiv.org/abs/2303.02536

2. The evaluation metrics (Section 3.2, lines 237-249) lack crisp definitions, despite being listed as a main contribution.

3. Experiments and figures appear rushed.

- The captions of Figures 3 and 4 lack detail. 

- It's not clear how many methods are tested, and different parts of the paper state different things. There are only 3 methods in Table 1; line 291 in the Experiments section says there are 5 methods; line 531 in the Conclusion says 4 interpretability methods.

- The prompts in lines 316-318 are not counterfactual: they change more than the target attribute.

- The data (lines 300-308) is quite restrictive, which may explain the 100% training and test accuracy of the probes (line 320).

- What exactly is the function used to measure coherence? Coherence does not appear to be defined anywhere: in line 388 it is simply stated that "we measure coherence ...as a function of the normalized latent edit distance...". What function exactly?

4. The encoder-decoder framework isn't motivated; assumptions are not discussed

- It's not clear (e.g. lines 82-88) what the encoder-decoder framework buys over e.g. mediation analysis.

In sum, this appears to be a preliminary research draft more appropriate for a conference workshop.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Answers to the following questions would be most likely to sway my score:
* **Re [W1]:** Could the authors provide more details on how one can make SAEs comparable to the other approaches for the purposes of comparing interprtability techniques (i.e., expand on the description in 4.1)?
* **Re [W1]:** In the experiments, high intervenability is equivalent to "frequency of word in sentence" due to the setup. Doesn't this give an unfair advantage to LogitLens/approaches where the feature space under interpretation is already the vocabulary? 
* **Re [W2]:** How might one design a more systematic/rigorous metric for coherence (*e.g.*, why not simply use rules-based tools to check for grammatical errors)? Are "grammar" and "comprehension" the only relevant dimensions for coherence (as used in the LLM-as-judge prompt in the paper)? How did/how might the authors plan to validate their coherence metric?
soundness: 2
strengths: * The problem area is well-motivated and timely — indeed, there's a gap in evaluating the "quality" of different interpretability techniques for LLMs. The proposed approach fills the gap by proposing concrete quality metrics to that end, and the experiments comprise multiple proofs-of-concept for how one could use the proposed metrics to evaluate interpretability techniques. 
* Some of the proposed metrics are intuitive and easy to measure. I particularly like the "intervention success" metric — while simple, it's a really intuitive proxy for whether an interpretability technique actually provides an "intervene-able" interpretation. 
* Contributing a benchmark of open-ended prompts from a variety of domains is also valuable — it's not simply another "task/capability" for LLMs to solve, but a natural test-bed for diversifying the space of possible interventions under the proposed framework.
summary: The proposed approach adopts an "intervention" based perspective to LLM interpretability. Two metrics for evaluating mechanistic interpretability approaches are proposed: a "intervention success rate" that quantifies how often changes in an interpretable feature yield desired changes in model outputs, and a "coherence-intervention tradeoff" that quantifies tradeoffs between intervention success and the coherence of model outputs. Extensive experiments demonstrate that different interpretability techniques yield interventions with varying  differing success rates and levels of output-coherence after the intervention.
weaknesses: Major 
* [W1] Comparability of approaches: Given that the learned $D$ has different columns depending on the approach (esp. SAE vs. the others), it is not immediately clear how one can compare different approaches directly. Furthermore, it's unclear if low "intervenability" (success rate and coherence) as measured by the proposed approach is attributable to a poor process for learning $D$, or the inability of the base LLM to "steer" along the relevant feature. More generally, the metrics could reflect either the behavior of the base LLM or the feature extraction approach.
* [W2] Evaluation metrics: I like the definition of intervention success, but coherence is much less clear. The explanation in the paper is "This can be done by any oracle, such as a human or another capable LLM" — but these approaches have disadvantages, since human annotation may be expensive, and an LLM-as-judge approach is not necessarily well-calibrated w.r.t. human annotations. 
* [W3] Undefined notation: $x$ is not defined at the top of 3.1, and $\theta$ is not defined (L200) in the explanation of probing. Again, I can guess what these mean from related works/Figure 1, but there's a high risk of misunderstanding. 

Minor 
* Lack of clarity in Figure 1: I know that symbols such as $D$ are defined later in the manuscript, but given that we encounter figure 1 at the top of page 2, it's a difficult to understand what the columns of the table are. This also makes it hard for me to appreciate the extent of the papers' contributions. I think the Table in Fig. 1 would be valuable later in the paper, after some expectations have been set about  desiderata for LLM interpretability evaluations.  At the very least, I think I need to see Section 3.1 before seeing the Table.
* Limited to open-weight models: while this limitation is shared by many approaches for interpretability/controllability, it would be nice to hear about potential extensions to models where the only access to a model is via API. For example, one could still intervene at the prompt level. Ultimately, this is very minor and wouldn't sway my score.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: It seems that the experiments rely heavily on data/prompts generated by LLMs.  Is this data inspected by humans, or otherwise validated for correctness?
soundness: 3
strengths: The paper demonstrates originality by unifying interpretability and control, addressing a key gap in popular research directions where most work focuses on either understanding or steering models. Its introduction of a unified encoder-decoder framework and new evaluation metrics (intervention success rate and coherence-intervention tradeoff) provides a novel perspective for benchmarking existing interpretability techniques more uniformly and on a more level playing field. The experiments evaluate multiple popular methods across various models, exposing the relative strengths and weaknesses of different approaches.  The method and evaluation framework are clearly explained, and supported by a useful Figure 1.  This paper seems like a significant contribution to the field, offering a framework to unify the pursuit of interpretability and control, an obvious yet elusive goal in LLM interpretability.  I believe it can stand as a valuable resource for ongoing research on model safety and alignment.
summary: To bridge the gap between research and techniques focused primarily on one of either understanding or control in LLMs, this paper presents a framework to unify interpretability and control through intervention.  They propose an encoder-decoder framework to enable both explanations from and interventions on latent representations, mapping human-interpretable features back to the model's latent space.  Two metrics, intervention success rate and coherence-intervention tradeoff, are introduced to assess the methods in a more holistic and direct way than was previously possible. The results show that while current methods can enable intervention, they are inconsistent across models and features, and can compromise model coherence.
weaknesses: The primary weakness of the paper is the lack of extension to complex, real-world applications.  Evaluation is limited to relatively simple interventions, such as word-level steering (e.g., 'coffee' or 'yoga') w.r.t. concrete concepts, which may not fully capture the challenges of more abstract, high-stakes tasks such as ethical decision-making, bias reduction, or truthfulness.  While this limitation is addressed acknowledged by the authors, and common in this research area, it does stand as a challenge in being able to draw any strong conclusions from the experiments.

Meta-review: This paper presents a framework for evaluating the effectiveness of interpretability methods in large language models (LLMs) through the lens of "intervention," aiming to bridge the gap between interpretability and model control. The authors introduce two novel evaluation metrics: intervention success rate and coherence-intervention tradeoff, designed to quantify how well interpretability methods alter LLM outputs. The proposed framework consolidates four distinct interpretability methods (SAEs, logit/tuned lens, probing, and lens-based approaches) under an encoder-decoder architecture. It evaluates their performance across multiple LLMs (GPT-2, Gemma2-2b, and Llama2-7b). The paper compares the framework to simpler methods such as steering vectors and prompting, uncovering performance trade-offs and highlighting the advantages and limitations of each approach. 

The methodology is largely sound, and the core idea of evaluating interpretability methods through intervention-based metrics is promising.  Despite these merits of this paper, several weaknesses remain that were only partially addressed in the rebuttal. First, reviewers highlighted the limited experimentation with more complex or real-world features. Although the authors incorporated experiments on two more complex concepts (responding in French and yelling), they are still constrained by the feature spaces of the interpretability methods they evaluated. The complexity of features, such as safety-critical ones, remains an important area for future work. Furthermore, the authors' reliance on low-level features like tokens and words may limit the generalizability of their findings to higher-level, more abstract concepts, a concern that was raised by the reviewers. The rebuttal does not fully resolve these issues, and a broader exploration of feature complexity would strengthen the paper's conclusions.

Second, the definition and validation of "coherence" were criticized for their lack of precision and reliance on an oracle, which complicates human evaluation and may misalign with LLM-based assessment. Although the authors attempted to address this by validating coherence with a rules-based grammar checker, the results still depend heavily on the specific tools used, and the fundamental issues with LLM-based evaluation persist. The authors' approach to coherence remains somewhat imprecise and lacks rigor, suggesting a need for more robust validation techniques. In addition, while the paper distinguishes itself from causal mediation analysis, it is worth noting that causal mediation could provide a more structured framework for understanding the causal impact of interpretability methods on model outputs. A clearer discussion of how their intervention-based approach contrasts with or could benefit from causal analysis would strengthen the theoretical foundation of the paper. These unresolved issues point to the need for more comprehensive experimental design and methodological refinement, particularly in the context of causal relationships and the evaluation of more complex, high-level features. 

Although this submission at the current stage has some merits, more efforts are still needed before it can be accepted.

Decision: Reject