RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations (2024)

Abstract: Individual neurons participate in the representation of multiple high-level concepts. To what extent can different interpretability methods successfully disentangle these roles? To help address this question, we introduce RAVEL (Resolving Attribute–Value Entanglements in Language Models), a dataset that enables tightly controlled, quantitative comparisons between a variety of existing interpretability methods. We use the resulting conceptual framework to define the new method of Multi-task Distributed Alignment Search (MDAS), which allows us to find distributed representations satisfying multiple causal criteria. With Llama2-7B as the target language model, MDAS achieves state-of-the-art results on RAVEL, demonstrating the importance of going beyond neuron-level analyses to identify features distributed across activations. We release our benchmark at https://github.com/ explanare/ravel.

heading: 1 Introduction
text: A central goal of interpretability is to localize an abstract concept to a component of a deep learning model that is used during inference. However, this is not as simple as identifying a neuron for each concept, because neurons are polysemantic – they represent multiple high-level concepts (Smolensky, 1988; Rumelhart et al., 1986; McClelland et al., 1986; Olah et al., 2020; Cammarata et al., 2020; Bolukbasi et al., 2021; Gurnee et al., 2023).
Several recent interpretability works (Bricken et al., 2023; Cunningham et al., 2024; Geiger et al., 2023b; Wu et al., 2023) tackle this problem using a featurizer that disentangles the activations of polysemantic neurons by mapping to a space of monosemantic features that each represent a distinct concept. Intuitively, these methods should have a significant advantage over approaches that identify concepts with sets of neurons. However, these methods have not been benchmarked.

heading: Inputs Expected Outputs
text: To facilitate these method comparisons, we introduce a diagnostic benchmark, RAVEL (Resolving Attribute–Value Entanglements in Language Models). RAVEL evaluates interpretability methods on their ability to localize and disentangle the attributes of different types of entities encoded as text inputs to language models (LMs). For example, the entity type “city” has instances such as “Paris” or “Tokyo”, which each have attributes for “continent”, namely “Europe” and “Asia”. An interpretability method must localize this attribute to a group of neurons N, learn a featurizer F (e.g., a rotation matrix or sparse autoencoder), and identify a feature F (e.g., a linear subspace of the residual stream in a Transformer) for the attribute. RAVEL contains five types of entities (cities, people names, verbs, physical objects, and occupations), each with at least 500 instances, at least 4 attributes, and at least 50 prompt templates per entity type.
The metric we use to assess interpretability methods is based on interchange interventions (also known as activation patching). This operation has emerged as a workhorse in interpretability, with a wide swath of research applying the technique to test if a high-level concept is stored in a model representation and used during inference (Geiger
1 8669
et al., 2020; Vig et al., 2020; Geiger et al., 2021; Li et al., 2021; Finlayson et al., 2021; Meng et al., 2022; Chan et al., 2022; Geva et al., 2023; Wang et al., 2023; Hanna et al., 2023; Conmy et al., 2023; Goldowsky-Dill et al., 2023; Hase et al., 2023; Todd et al., 2024; Feng and Steinhardt, 2024; Cunningham et al., 2024; Huang et al., 2023; Tigges et al., 2023; Lieberum et al., 2023; Davies et al., 2023; Hendel et al., 2023; Ghandeharioun et al., 2024).
Specifically, we use the LM to process a prompt like “Paris is in the continent of” and then intervene on the neurons N to fix the feature F to be the value it would have if the LM were given a prompt like “Tokyo is a large city.” If this leads the LM to output “Asia” instead of “Europe”, then we have evidence that the feature F encodes the attribute “continent”. Then, we perform the same intervention when the LM processes a prompt like “People in Paris speak”. If the LM outputs “French” rather than “Japanese’, then we have evidence that the feature F has disentangled the attributes “continent” and “language”.
A variety of existing interpretability methods are easily cast in the terms needed for RAVEL evaluations, including supervised probes (Peters et al., 2018; Hupkes et al., 2018; Tenney et al., 2019; Clark et al., 2019), Principal Component Analysis (Tigges et al., 2023; Marks and Tegmark, 2023), Differential Binary Masking (DBM: Cao et al. 2020; Csordás et al. 2021; Cao et al. 2022; Davies et al. 2023), sparse autoencoders (Bricken et al., 2023; Cunningham et al., 2024), and Distributed Alignment Search (DAS: Geiger et al. 2023b; Wu et al. 2023). Our apples-to-apples comparisons reveal conceptual similarities between the methods.
In addition, we propose multi-task training objectives for DBM and DAS. These objectives allow us to find representations satisfying multiple causal criteria, and we show that Multi-task DAS is the most effective of all the methods we evaluate at identifying disentangled features. This contributes to the growing body of evidence that interpretability methods need to identify features that are distributed across neurons.

heading: 2 The RAVEL Dataset
text: The design of RAVEL is motivated by four highlevel desiderata for interpretability methods:
1. Faithful: Interpretability methods should accurately represent the model to be explained.
2. Causal: Interpretability methods should analyze the causal effects of model components on model input–output behaviors.
3. Generalizable: The causal effects of the identified components should generalize to similar inputs that the underlying model makes correct predictions for.
4. Isolating individual concepts: Interpretability methods should isolate causal effects of individual concepts involved in model behaviors.
The goal of RAVEL is to assess the ability of methods to isolate individual explanatory factors in model representations (desideratum 4), and do so in a way that is faithful to how the target models work (desideratum 1). The dataset train/test structure seeks to ensure that methods are evaluated by how well their explanations generalize to new cases (desideratum 3), and RAVEL is designed to support intervention-based metrics that assess the extent to which methods have found representations that causally affect the model behavior (desideratum 2).
RAVEL is carefully curated as a diagnostic dataset for the attribute disentanglement problem. RAVEL has five types of entity, where every instance has every attribute associated with its type. Table 1 provides an overview of RAVEL’s structure.
The Attribute Disentanglement Task We begin with a set of entities E = {E1, . . . , En}, each with attributes A = {A1, . . . , Ak}, where the correct value of A for E is given by AE . Our interpretability task asks whether we can find a feature F that encodes the attribute A separately from the other attributes A \ {A}. For Transformer-based models (Vaswani et al., 2017), a feature might be a dimension in a hidden representation of an MLP or a linear subspace of the residual stream.
We do not know a priori the degree to which it is
2
possible to disentangle a model’s representations. However, our benchmark evaluates interpretability methods according to the desiderata given above and so methods will need to be faithful to the model’s underlying structure to succeed. In other words, assuming methods are faithful, we can favor methods that achieve more disentanglement.

heading: 2.1 Data Generation
text: Selecting Entity Types and Attributes We first identify entity types from existing datasets that potentially have thousands of instances (see Appendix A.1), such as cities or famous people. Moreover, each entity type has multiple attributes with different degrees and types of associations. For example, for attributes related to city, “country” entails “continent”, but not the reverse; “country” is predictable from “timezone” but non-entailed; and “latitude” and “longitude” are the least correlated compared with the previous two pairs, but have identical output spaces. These entity types together cover a diverse set of attributes such that predicting the value of the attribute uses factual, linguistic, or commonsense knowledge.
Constructing Prompts We consider two types of prompts: attribute prompts and entity prompts. Attribute prompts PAE contain mentions of E and instruct the model to output the attribute value AE . For example, E = Paris is an instance of the type “city”, which has an attribute A = Continent that can be queried with prompts “Paris is in the continent of”. Prompts can also be JSON-format, e.g., “{"city": "Paris", "continent":"”, which reflects how entity–attribute association might be encoded in training data. For each format, we do zero- and fewshot prompting. In addition to attribute prompts, entity prompts WE contain mentions of the E, but does not query any A ∈ A. For example, “Tokyo is a large city”. We sample entity prompts from the Wikipedia corpus.1
For a set of entities E and a set of attributes to disentangle A, the full set of prompts is
D = {x : x ∈ PAE ∪WE , E ∈ E , A ∈ A}
Generating Splits RAVEL offers two settings, Entity and Context, to evaluate the generalizability (desideratum 3) of an interpretability method
1We use the 20220301.en version pre-processed by HuggingFace at https://huggingface.co/datasets/ wikipedia
across unseen entities and contexts. Each setting has a predefined train/dev/test structure. In Entity, for each entity type, we randomly split the entities into 50%/25%/25% for train/dev/test, but use the same set of prompt templates across the three splits. In Context, for each attribute, we randomly split the prompt templates into 50%/25%/25%, but use the same set of entities across the three splits.
Filtering for a Specific Model When evaluating interpretability methods that analyze a model M, we generally focus on a subset of the instances where M correctly predicts the values of the attributes (see Appendix A.2). This allows us to focus on understanding why models succeed, and it means that we don’t have to worry about how methods might have different biases for incorrect predictions.

heading: 2.2 Interpretability Evaluation
text: Interchange Interventions A central goal of RAVEL is to assess methods by the extent to which they provide causal explanations of model behaviors (desideratum 2). To build such analyses, we need to put models into counterfactual states that allow us to isolate the causal effects of interest.
The fundamental operation for achieving this is the intervention (Spirtes et al., 2000; Pearl, 2001, 2009): we change the value of a model-internal state and study the effects this has on the model’s input–output behavior. In more detail: let M(x) be the entire state of the model when M receives input x, i.e., the set of all input, hidden, and output representations created during inference. Let MN←n be the model where neurons N are intervened upon and fixed to take on the value n ∈ Values(N).
Geiger et al. (2023b) generalize this operation to intervene upon features that are distributed across neurons using a bijective featurizer F . Let MF←f be the model where neurons N are projected into a feature space using F , the feature F is fixed to take on value f , and then the features are projected back into the space of neural activations using F−1. If we let τ(M(x)) be the token that a model predicts for a given prompt x ∈ D, then comparisons between τ(M(x)) and τ(MF←f (x)) yield insights into the causal role that F plays in model behavior.
However, most conceivable interventions fix model representations to be values that are never realized by any input. To characterize the high-level conceptual role of a model representation, we need a data-driven intervention that sets a representation
3
to values it could actually take on. This is achieved by the interchange intervention, which fixes a feature F to the value it would take if a different input x′ were provided:
II(M, F, x, x′) def= τ ( MF←GetFeature(M(x′),F )(x) ) (1)
where GetFeature(M(x′), F ) is the value of F in M(x′). Interchange interventions represent a very general technique for identifying abstract causal processes that occur in complex black-box systems (Beckers and Halpern, 2019; Beckers et al., 2020; Geiger et al., 2023a).
Evaluation Data For evaluation, each intervention example consists of a tuple: an input x ∈ PAE , an input x′ ∈ PA′E′ ∪ WE′ , a target attribute A∗, and an intervention label y. If A∗ = A, then y is AE′ and otherwise y is AE . For example, if the set of “city” entities to evaluate on is {"Paris", "Tokyo"} and the goal is to disentangle the “country” attribute from the “continent” attribute, then the set of test examples becomes the one shown in Figure 1.
Metrics If M achieves behavioral success on a dataset, we can use that dataset to evaluate an interpretability method on its ability to identify a collection of neurons N, a featurizer F for those neurons, and a feature F that represents an attribute A separately from all others attributes A \ {A}.
If F encodes A, then interventions on F should change the value of A. When M is given a prompt x ∈ PAE , we can intervene on F to set the value to what it would be if a second prompt x′ ∈ PA′E′ ∪ WE′ were provided. The token predicted by M should change from AE to AE′ :
Cause(A,F,M,D) def= ED [ II(M, F, x, x′) = AE′ ]
If F isolates A, then interventions on F should not cause the values of other attributes A∗ ∈ A \ {A} to change. When M is given a prompt x∗ ∈ PA∗E , we can again intervene on F to set the value to what it would be if a second prompt x′ ∈ PA′E′ ∪ WE′ were provided. The token predicted by M should remain A∗E :
Iso(A,F,M,D) def= 1
|A \ {A}| ∑
A∗∈A\{A} ED
[ II(M, F, x∗, x′) = A∗E ]
To balance these two objectives, we define the Disentangle score as a weighted average between Cause and Iso.
Disentangle(A,F,M,D) = 1
2
[ Cause(A,F,M,D) + Iso(A,F,M,D) ]
The score on RAVEL for an entity type is its average Disentangle score over all attributes.
In practice, two attributes might not be fully disentanglable in the model M so there is no guarantee that it is possible to find a feature F that achieves Cause = 1 and Iso = 1 at the same time. However, evidence that two attributes might not be separable is an insight into how knowledge is structured in the model.

heading: 3 Interpretability Methods
text: We use RAVEL to evaluate a variety of interpretability methods on their ability to disentangle attributes while generalizing to novel templates and entities. Each method uses data from the training split to find a set of neurons N, learn a featurizer F , and find a feature FA that captures an attribute A ∈ A independent from the other attributes. In Section 4, we describe the baseline procedure we use for considering different sets of neurons. In this section, we define methods for learning a featurizer and identifying a feature given a set of neurons. For each method, the core intervention for A is given by II(M, FA, x, x′) where FA is defined by the method. In this section, we use GetVals(M(x),N) to mean the activations of neurons N when M processes input x.

heading: 3.1 PCA
text: Principal Component Analysis (PCA) is a dimensionality reduction method that minimizes information loss. In particular, given a set of real valued vectors V ⊂ Rn, |V| > n, the principal components are n orthogonal unit vectors p1, . . . ,pn that form an n× n matrix:
PCA(V) = [ p1 . . . pn ]
For our purposes, the orthogonal matrix formed by the principal components serves as a featurizer that maps neurons N into a more interpretable space (Chormai et al., 2022; Marks and Tegmark, 2023; Tigges et al., 2023). Given an attribute A, a training
4
dataset D from RAVEL for a particular entity type, a model M, and a set of neurons N, we define
FA(n) = nTPCA({GetVals(M(x),N) : x ∈ D})
PCA is an unsupervised method, so there is no easy way to tell what information is encoded in each principal component. To solve this issue, for each attribute A ∈ A we train a linear classifier with L1 regularization to predict the value of A from the featurized neural representations. Then, we define the feature FA to be the set of dimensions assigned a weight by the classifier that is greater than a hyperparameter ϵ.

heading: 3.2 Sparse Autoencoder
text: A recent approach to featurization is to train an autoencoder to project neural activations into a higher dimensional sparse feature space and then reconstruct the neural activations from the features (Bricken et al., 2023; Cunningham et al., 2024). We train a sparse autoencoder on the loss ∑ x∈D ||GetVals(M(x),N)− ( W2f+b2 ) ||2+||f ||1
f = ReLU(W1(GetVals(M(x),N)− b2) + b1) with W1 ∈ Rn×m, W2 ∈ Rm×n, b1 ∈ Rm, and b2 ∈ Rn. To construct a training datset, we sample 100k sentences from the Wikipedia corpus for each entity type, each containing a mention of an entity in the training set. We extract the 4096- dimension hidden states of Llama2-7B at the target intervention site as the input for training a sparse autoencoder with 16384 features.
We use the autoencoder to define a featurizer
FA(n) = ReLU(W1(n− b2) + b1) and an inverse F−1A (n) = W2n+ b2.
An important caveat to this method is that the featurizer is only truly invertible if the autoencoder has a reconstruction loss of 0. The larger the loss is, the more unfaithful this interpretability method is to the model being analyzed. All other methods considered use an orthogonal matrix, which is truly invertible up to floating point precision.
Similar to PCA, sparse autoencoders are an unsupervised method that does not produce features with obvious meanings. Again, to solve this issue, for each attribute A ∈ A we train a linear classifier with L1 regularization and define the feature FA to be the set of dimensions assigned a weight that is greater than a hyperparameter ϵ.

heading: 3.3 Relaxed Linear Adversarial Probe
text: Supervised probes are a popular interpretability technique for analyzing how neural activations correlate with high-level concepts (Peters et al., 2018; Hupkes et al., 2018; Tenney et al., 2019; Clark et al., 2019). When probes are arbitrarily powerful, this method is equivalent to measuring the mutual information between the neurons and the concept (Pimentel et al., 2020; Hewitt et al., 2021). However, probes are typically simple linear models in order to capture how easily the information about a concept can be extracted. Probes have also been used to great effect on the task of concept erasure (Ravfogel et al., 2020; Elazar et al., 2021; Ravfogel et al., 2022).
Following the method of Ravfogel et al. (2022), we train a relaxed linear adversarial probe (RLAP) to learn a linear subspace parameterized by a set of k orthonormal vectors W ∈ Rk×n that captures an attribute A, using the following loss objective:
min θ max W
∑ x∈D CE ( θT f , AEx )
f = (I −W TW ) ( GetVals(M(x),N) )
where f is the representation of the entity with the attribute information erased, and θ is a linear classifier that tries to predict the attribute value AEx from the erased entity representation.
We define the F using the set of k orthonormal vectors that span the row space of W and the set of n− k orthonormal vectors that span the null space:
FA(n) = n [ r1 . . . rk uk+1 . . . un ]
Our feature FA is the first k dimensions of the feature space, i.e. the row space of W . Intuitively, since the linear probe was trained to extract the attribute A, the rowspace is the linear subspace of neural activations that the probe is “looking at” to make predictions.

heading: 3.4 Differential Binary Masking
text: Differential Binary Masking (DBM) learns a binary mask to select a set of neurons that causally represents a concept (Cao et al., 2020; Csordás et al., 2021; Cao et al., 2022; Davies et al., 2023). The loss objective used to train the mask is a combination of matching the counterfactual behavior and
5
forcing the mask to be sparse with coefficient λ:
LCause = CE(τ(MN←n(x)), AE′) + λ||m||1 n = ( 1− σ(m/T ) ) ◦ GetVals(M(x),N)
+ σ(m/T ) ◦ GetVals(M(x′),N)
where the intervention is determined by inputs x, x′ and learnable parameter m ∈ Rn, where ◦ is element wise multiplication and T ∈ R is a temperature annealed throughout training.
The feature space is the original space of neural activations, i.e., featurizer FA(n) = n. The feature FA is the set of dimensions i where 1−σ(mi/T ) < ϵ for a (small) hyperparameter ϵ.

heading: 3.5 Distributed Alignment Search
text: Distributed Alignment Search (DAS) (Geiger et al., 2023b) learns a linear subspace of a model representation with a training objective defined using interchange interventions. In the original work, the linear subspace learned by DAS is parameterized as an n × n orthogonal matrix Q = [u1 . . .un], which rotates the representation into a new coordinate system, i.e., FA(n) = Q⊤n. The set of feature FA is the first k dimensions of the rotated subspace, where k is a hyperparameter. The matrix Q is learned by minimizing the following loss:
LCause(A,FA,M) = CE(II(M, FA, x, x′), AE′)
Computing Q is expensive, as it requires computing n orthogonal vectors. To avoid instantiating the full rotation matrix, we use an alternative parameterization where we only learn the k ≪ n orthogonal vectors that form the feature FA (see Appendix B.4).

heading: 3.6 Multi-task DBM and DAS
text: To address the disentanglement problem, we propose a multitask extension of DBM (MDBM) and DAS (MDAS). The original training objective of DBM and DAS only optimizes for the Cause score,
without considering the impact on the Iso score. We introduce the Iso aspect into the training objective through multitask learning. For each attribute A∗ ∈ A \ {A}, we define the Iso objective as
LIso(A,FA,M) = CE(II(M(x), FA, x′), A∗E)
We minimize a linear combination of losses from each task:
LDisentangle(A, FA,M) =
LCause(A,FA,M)+ ∑
A∗∈A\{A}
LIso(A∗, FA,M) |A \ {A}|

heading: 4 Experiments
text: We evaluate the methods described in Section 3 on RAVEL with Llama2-7B (Touvron et al., 2023), a 32-layer decoder-only Transformer model, as the target LM. Implementation details of each method are provided in Appendix B.

heading: 4.1 Setup
text: We consider the residual stream representations at the last token of the entity as our potential intervention sites. For autoregressive LMs, the last token of the entity tE (e.g., the token “is” in the case of “Paris”) likely aggregates information of the entity (Meng et al., 2022; Geva et al., 2023). For Transformer-based LMs like Llama2, an activation vector NLt in the residual stream is created for each token t at each Transformer layer
6
(a) Cause score for all attributes when intervening on the attribute features identified by DAS (left) and MDAS (right). A Cause score of 0.62 for column Continent, row Timezone (bottom left corner), means that, when intervening on the Continent feature, the same subspace changes Timezone 62% of the time. (b) The Cause, Iso, and Disentangle score on the Entity split for the “country” feature found by MDAS. The attributes of cities become more disentangled across layers.
Figure 3: Additional results for the MDAS method.
L. As the contributions of the MLP and attention heads must pass through the residual stream, it serves as a bottleneck. Therefore, we will limit our methods to examining the set of representations N = {NLtE : L ∈ {1, . . . , 32}}.
This simplification is only to establish baseline results on the RAVEL benchmark. We expect the best methods will consider other token representations, such as the remainder of the token sequence that realizes the entity.

heading: 4.2 Results
text: We evaluate each method on every representation NLtE and report the highest disentanglement score on test splits in Table 2. We additionally include a baseline that simply replaces the full representation NLtE regardless of what attribute is being targetted (see Full Rep. in Table 2). A breakdown of the results with per-attribute Cause and Iso is in Appendix C.
In Figure 2, we show for each method, how the Iso and Cause scores vary as we change the dimensionality of FA, the feature targeted for intervention. For RLAP, DAS, and MDAS, the dimensionality of FA is a hyperparameter we vary directly. For other methods, we vary the coefficient of L1 penalty to vary the size of FA. Details are given in Appendix B.
In Figure 3, we focus on using MDAS, the best performing method, to understand how attributes are disentangled in Llama2-7B. Figure 3a shows two heat maps summarizing the performance of DAS and MDAS on the entity type “city”. These heat maps also show how attributes have different
levels of disentanglement. Figure 3b shows how the Cause, Iso, and Disentangle scores change for the “country” attribute across model layers.
Methods with counterfactual supervision achieve strong results while methods with unsupervised featurizers struggle. MDAS is the state-of-the-art method on RAVEL, being able to achieve high Disentangle scores while only intervening on a feature FA with a dimensionality that is 4% of |N| where N are the neurons the feature is distributed across (Figure 2). DBM, MDBM, and DAS, the other methods that are trained with interventions using counterfactual labels as supervision, achieve the next best performance. PCA and Sparse Autoencoder achieve the lowest Disentangle scores, which aligns with the prior finding that disentangled representations are difficult to learn without supervision (Locatello et al., 2018). Unsurprisingly, more supervision results in higher performance.
Multi-task supervision is better at isolating attributes. Adding multitask objectives to DBM and DAS increases the overall disentanglement score by 1.5%/4.1% and 3.6%/8.3% on the Entity/Context split respectively. To further illustrate the differences, we compare DAS with MDAS in Figure 3a. On the left, attributes such as “continent” and “timezone” are naturally entangled with all other attributes; intervening on the feature learned by DAS for any city attribute will also change these two attributes. In contrast, in Figure 3a right, MDAS is far more successful at disentangling these attributes, having small Cause
7
scores in all off-diagonal entries.
Some groups of attributes are more difficult to disentangle than others. As show in Figure 3a, the attribute pairs “country–language” and “latitude–longitude” are difficult to disentangle. When we train DAS to find a feature for either of these attributes (Figure 3a left), the same feature also has causal effects on the other attribute. Even with the additional supervision (Figure 3a right), MDAS cannot isolate these attributes. Changing one of these entangled attributes has seemingly unavoidable ripple effects (Cohen et al., 2024) that change the other. In contrast, the attribute pair “language–continent” can be disentangled. Moreover, the pairs that are difficult to disentangle are consistent across all five supervised methods in our experiment, despite these methods using different training objectives. We include additional visualizations in Appendix C.2.
Attributes are gradually disentangled across layers. The representations of different attributes gradually disentangle as we move towards later layers, as shown in Figure 3b. Early layer features identified by MDAS fail to generalize to unseen entities, hence low Cause score. While MDAS is able to identify a feature with relatively high Cause starting at layer 8, the Iso increases from 0.5 to 0.8 from layer 8 to layer 16. It is not until layer 16 that the highest Disentangle score is achieved.

heading: 5 Related Work
text: Intervention-based Interpretability Methods Intervention-based techniques, branching off from interchange intervention (Vig et al., 2020; Geiger et al., 2020) or activation patching (Meng et al., 2022), have shown promising results in uncovering causal mechanisms of LMs. They play important roles in recent interpretability research of LMs such as causal abstraction (Geiger et al., 2021, 2023b), causal tracing to locate factual knowledge (Meng et al., 2022; Geva et al., 2023), path patching or causal scrubbing to find causal circuits (Chan et al., 2022; Conmy et al., 2023; Goldowsky-Dill et al., 2023), and Distributed Alignment Search (Geiger et al., 2023b). Previous works suggest that activation interventions that result in systematic counterfactual behaviors provide clear causal insights into model components.
Isolating Individual Concepts LMs learn highly distributed representations that encode multiple
concepts in a overlapping sets of neurons (Smolensky, 1988; Olah et al., 2020; Elhage et al., 2022). Various methods have been proposed to find components that capture a concept, such as finding a linear subspace that modifies a concept (Ravfogel et al., 2020, 2022; Belrose et al., 2023; Cao et al., 2020; Geiger et al., 2023b) and generating a sparse feature space where each direction captures a word sense or is more interpretable (Arora et al., 2018; Bricken et al., 2023; Cunningham et al., 2024; Tamkin et al., 2023). However, these methods have not been evaluated against each other on their ability to isolate concepts. Isolating an individual concept is also related to the goal of “disentanglement” in representation learning (Schölkopf et al., 2021), where each direction captures a single generative factor. In this work, we focus on isolating the causal effect of a representation.
Knowledge Representation in LMs Understanding knowledge representation in LMs starts with probing structured linguistic knowledge (Conneau et al., 2018; Tenney et al., 2019; Manning et al., 2020). Recent work expands to factual knowledge stored in Transformer MLP layers (Geva et al., 2021; Dai et al., 2022; Meng et al., 2022), associations represented in linear structures (Merullo et al., 2023; Hernandez et al., 2024; Park et al., 2023), and deeper study of the semantic enrichment of subject representation (Geva et al., 2023). These findings suggest LMs store knowledge modularly, motivating the disentanglement objective in our work.
Benchmarking Interpretability Methods Testing the faithfulness of interpretability method relies on counterfactuals. Existing counterfactual benchmarks use behavioral testing (Atanasova et al., 2023; Schwettmann et al., 2023; Mills et al., 2023), interventions (Abraham et al., 2022), or a combination of both (Huang et al., 2023). Recent model editing benchmarks (Meng et al., 2022; Zhong et al., 2023; Cohen et al., 2024) also provide counterfactuals that have potential for evaluating interpretability methods. MQUAKE (Zhong et al., 2023) and RIPPLEEDITS (Cohen et al., 2024), in particular, consider entailment relationships of attributes, while we focus on disentanglement.

heading: 6 Conclusion
text: We present RAVEL a benchmark for evaluating the ability of interpretability methods to localize and disentangle entity attributes in LMs in a causal,
8
generalizable manner. We show how RAVEL can be used to evaluate five different families of interpretability methods that are commonly used in the community. We benchmark several strong interpretability methods on RAVEL with Llama2-7B model as baselines, and we introduce a multi-task objective that improves the performance of Differential Binary Masking (DBM) and Distributed Alignment Search (DAS). Multi-task DAS achieves the best results in our experiments. Results on our attribute disentanglement task also offer insights into the different levels of entanglement between attributes and the emergence of disentangled representations across layers in the Llama2-7B model.
The community has seen an outpouring of innovative new interpretability methods. However, these methods have not been systematically evaluated for whether they are faithful, generalizable, causally effective, and able to isolate individual concepts. We release RAVEL2 to the community and hope it will help drive the assessment and development of interpretability methods that satisfy these criteria.

heading: Limitations
text: Our attribute disentanglement results in Section 4 are based on the Llama2-7B model. While Llama27B uses the widely adopted decoder-only Transformer architecture, different model architectures or training paradigms could produce LMs that favor different interpretability methods. Hence, when deciding which interpretability method is the best to apply to a new model, we encourage people to instantiate RAVEL on the new model.
When choosing intervention sites, we limit our search to the residual stream above the last entity token. However, representations of attributes can be distributed across multiple tokens or layers. We encourage future work to explore different intervention sites when using this benchmark.

heading: Ethics Statement
text: In this paper, we present an interpretability benchmark that aims to assess the faithfulness, generalizability, causal effects, and the ability to isolate individual concepts in language models. While an interpretability method that satisfies these criteria could be useful for assessing model bias or steering model behaviors, the same method might also be
2https://github.com/explanare/ravel
used for manipulating models in undesirable applications such as triggering toxic outputs. These interpretability methods should be studied and used in a responsible manner.

heading: Acknowledgements
text: This research is supported in part by grants from Open Philanthropy and the Stanford Institute for Human-Centered Artificial Intelligence (HAI).

heading: A Dataset Details

heading: A.1 Details of Entities and Attributes
text: We first identify entity types from existing datasets such as the Relations Dataset (Hernandez et al., 2024) and RIPPLEEDITS (Cohen et al., 2024), where each entity type potentially contains thousands of instances. We then source the entities and ground truth references for attribute values from online sources.3 4 5 6 7 8 These online sources are distributed under MIT, Apache-2.0, and CC-0 licenses. Compared with similar entity types in the Relations Dataset and RIPPLEEDITS, RAVEL has expanded the number of entities by a factor of at least 10 and included multiple attributes per entity.
We show the cardinality of the attributes, most frequent attribute values, and random samples of prompt templates in Table 3.

heading: A.2 The RAVEL Llama2-7B Instance
text: The RAVEL Llama2-7B instance is used for benchmarking interpretability methods in Section 4. There are a total of 2800 entities in the Llama2-7B instance. Table 4 shows the number of entities, prompt
3https://github.com/kevinroberts/city-timezones 4https://github.com/open-dict-data/ipa-dict/blob/master/data/en_US.txt 5https://github.com/monolithpl/verb.forms.dictionary 6https://www.nobelprize.org/prizes/lists/all-nobel-prizes/ 7https://huggingface.co/datasets/corypaik/coda 8https://www.bls.gov/ooh,https://www.bls.gov/cps
13
templates, and test examples, i.e., the number of base–source input pairs for interchange intervention in the Llama2-7B instance.
The RAVEL Llama2-7B instance is created by filtering examples where the pre-trained Llama2-7B has a high accuracy in predicting attribute values. For each entity type, we take the k entities with the highest accuracy over all prompt templates and the n prompt templates with the highest accuracy over all entities, with the average accuracy over all prompts shown in Table 4. For most attributes, we directly compare
14
model outputs against the ground truth attribute values. For “latitude” and “longitude” of a city, we relax the match to be ±2 within the ground truth value. For “pronunciation” of a verb, we relax the match to allow variations in the transcription. For attributes with more open-ended outputs, including “definition” of a verb and “duty” of an occupation, we manually verify if the outputs are sensible. For “gender bias” of an occupation, we check for the consistency of gender bias over a set of prompts that instruct the model to output gender pronouns.

heading: B Method Details

heading: B.1 PCA
text: For PCA, we extract the 4096-dimension hidden state representations at the target intervention site as the inputs. The representations are first normalized to zero-mean and unit-variance using mean and variance estimated from the training set. We use the sklearn implementation9 to compute the principal components. We then apply L1-based feature selection10 to identify a set of dimensions that most likely encode the target attribute A. We undo the normalization after projecting back to the original space.
We vary the coefficient of the L1 penalty, i.e., the parameter “C” in the sklearn implementation, to experiment with different intervention dimensions. We experiment with C ∈ {0.1, 1, 10, 1000}. We observe that regardless of the intervention dimension, the features selected have a high overlap with the first k principal components. For most attributes, the highest Disentangle score is achieved when using the largest intervention dimension.

heading: B.2 Sparse Autoencoder
text: For the sparse autoencoder, we use a single-layer encoder-decoder model.11 The autoencoder is trained on Wikipedia data as described below.
Model Encoder: Fully connected layer with ReLU activations, dimensions 4096 × 16384. Decoder: Fully connected layer, dimensions 16384× 4096. Latent dimension: 4× 4096. The model is trained to optimize a combination of an L2 loss to reconstruct the representation and an L1 loss to enforce sparsity.
Training Data For each entity type, we sample 100k sentences from the Wikipedia corpus, each containing a mention of an entity in the training set. We extract the 4096-dimension hidden states at the target intervention site as the input for training the sparse autoencoder.
Similar to PCA, we apply L1-based feature selection on the latent representation to identify a set of dimensions that most likely encode the target attribute A. We vary the coefficient C of the L1 penalty to experiment with different intervention dimension. The optimal C varies across attributes.

heading: B.3 RLAP
text: RLAP learns a set of linear probes to find the feature F . Each linear probe aims to predict the attribute value from the entity representations. Similar to PCA and sparse autoencoders, we use the 4096-dimension hidden state representations at the target intervention site as the initial inputs and the corresponding attribute value as labels. In the case of attributes with extremely large output spaces, e.g., numerical outputs, we approximate the output with the first token. Table 5 shows the linear classifier accuracy on each attribute classification task.
We use the official R-LACE implementation12 and extract the rank-k orthogonal matrix W from the final null projection13 as FA. For each attribute, we experiment with rank k ∈ {32, 128, 512, 2048}. We run the algorithm for 100 iterations and select the rank with the highest Disentangle score on the dev set. The optimal intervention dimension is usually small, i.e., 32 or 128, for attributes that have a high accuracy linear classifier.
9https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html 10https://scikit-learn.org/stable/modules/feature_selection.html#l1-based-feature-selection 11https://colab.research.google.com/drive/1u8larhpxy8w4mMsJiSBddNOzFGj7_RTn?usp=sharing#scrollTo=
Kn1E_44gCa-Z 12https://github.com/shauli-ravfogel/rlace-icml 13https://github.com/shauli-ravfogel/rlace-icml/blob/master/rlace.py#L90
15

heading: B.4 DBM-based and DAS-based Methods
text: For DBM- and DAS-based methods, we use the implementation from the pyvene library.14 For training data, both methods are trained on base–source pairs with interchange interventions.
For DBM and MDBM, we use a starting temperature of 1e−2 and gradually reducing it to 1e−7. The feature dimension is controlled by the coefficient of the L1 loss. The optimal coefficient for the DBM penalty is around 0.001, while no penalty generally works better for MDBM, as the multi-task objective naturally encourages the methods to select as few dimensions as possible.
For DAS and MDAS, we do not instantiate the full rotation matrix, but only parameterize the k orthogonal vectors that form the feature FA. The interchange intervention is defined as
II(M, FA, x, x′) = (I −W⊤W )(GetVals(M(x),N)) +W⊤W (GetVals(M(x′),N))
where the rows of W are the k orthogonal vectors. We experiment with k ∈ {32, 128, 512, 2048} and select the dimension with the highest Disentangle score on the dev set. For most attributes, a larger intervention dimension, e.g., 512 or 2048, leads to a higher Disentangle score.

heading: B.5 Computational Cost
text: All models are trained and evaluated on a single NVIDIA RTX A6000 GPU.
For training, the computational cost of sparse autoencoders is the lowest, as training sparse autoencoders does not involve backpropagating through the original Llama2-7B model or computing orthogonal factorization of weight matrices. Each epoch of the sparse autoencoder training, i.e., iterating over 100k examples, takes about 100 seconds with Llama2-7B features extracted offline. The computational cost of RLAP- and DAS-based method largely depends on the rank of the nullspace or the intervention dimension, i.e., the number of orthogonal vectors. For RLAP, it takes 1 hour per 100 iterations with
14https://github.com/stanfordnlp/pyvene
16
a feature dimension 4096 and a target rank of 128. For DAS and MDAS with the reduced parameter formulation, the training time for an intervention dimension of 128 (out of a feature dimension of 4096) over 1k intervention examples is about 50 seconds. The computational cost of DBM-based method is about 35 seconds per 1k intervention examples.
For evaluation, the inference speed of our proposed framework is 20 seconds per 1k intervention examples.

heading: C Results
text: For all methods, we conduct hyper-parameter search on the dev set. We report a single-run test set results using the set of hyper-parameters that achieves the highest score on the dev set. For intervention site, we choose layer 16 for city attributes and layer 7 for the rest attributes.

heading: C.1 Breakdown of Benchmark Results
text: Table 6 shows the breakdown of benchmark results in Table 2. For each method, we report a breakdown of the highest Disentangle score per attribute, i.e., the pair of Cause score and Iso score that add up to the highest Disentangle score. The final score in Table 2 is an average of the Disentangle score over all five entity types. For example, for PCA, the Disentangle score under the Entity setting is (38.7 + 36.0 + 41.3 + 43.3 + 38.1)/5 = 39.5.
17
18

heading: C.2 Additional Attribute Disentanglement Results
text: In Figure 3, we show the feature entanglement results from DAS and MDAS. We provide additional results from all other supervised methods: RLAP, DBM, and MDBM in Figure 4. Though these methods are trained on different objectives and identify different features FA, they show similar patterns in terms of entanglement between attribute representations. For all methods, representations of most attributes are entangled with “continent” (and “timezone”, which for most cases starts with the continent name). Representations of attributes such as “county–language” are also highly entangled.
19

heading: Future Work
text: We encourage future work to explore different intervention sites when using this benchmark