HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworks
Authors: Jiuding Sun
Jing Huang
Sidharth Baskaran
Karel D'Oosterlinck
Christopher Potts
Michael Sklar
Atticus Geiger
Venue: ICLR 2025 Poster (2025)
Keywords: mechanistic interpretability
causal abstraction
hypernetwork
Primary area: interpretability and explainable AI
TL;DR: HyperDAS, a transformer-based hypernetwork, enhances interpretability by pinpointing and learning concept-specific features in neural networks' residual streams.

Abstract: Mechanistic interpretability has made great strides in identifying neural network features (e.g., directions in hidden activation space) that mediate concepts (e.g., *the birth year of a Nobel laureate*) and enable predictable manipulation. Distributed alignment search (DAS) leverages supervision from counterfactual data to learn concept features within hidden states, but DAS assumes we can afford to conduct a brute force search over potential feature locations. To address this, we present HyperDAS, a transformer-based hypernetwork architecture that (1) automatically locates the token-positions of the residual stream that a concept is realized in and (2) learns features of those residual stream vectors for the concept. In experiments with Llama3-8B, HyperDAS achieves state-of-the-art performance on the RAVEL benchmark for disentangling concepts in hidden states. In addition, we review the design decisions we made to mitigate the concern that HyperDAS (like all powerful interpretabilty methods) might inject new information into the target model rather than faithfully interpreting it.

Published as a conference paper at ICLR 2025
HYPERDAS: TOWARDS AUTOMATING MECHANISTIC INTERPRETABLLITY WITH HYPERNETWORKS
Jiuding Sun
Jing Huang
Sidharth Baskaran
Karel D'Oosterlinck
Christopher Potts"
Michael Sklar"
Atticus Geiger
"Pr(Ai)'R Group
Stanford University Confirm Labs
Ghent University
ABSTRACT
Mechanistic interpretability has made great strides in identifying neural network features (e.g., directions in hidden activation space) that mediate concepts (e.g.. the birth year of a person) and enable predictable manipulation. Distributed align- ment search (DAS) leverages supervision from counterfactual data to learn con- cept features within hidden states, but DAS assumes we can afford to conduct a brute force search over potential feature locations. To address this, HyperDAS, a transformer-based hypernetwork architecture that (1) automatically locates the token-positions of the residual stream that concept is realized in and (2) constructs features of those residual stream vectors for the concept. In exper- iments with Llama3-88 HyperDAS achieves of performance on the RAVEL benchmark for disentangling concepts in hidden states. In addition, we review the design decisions we made to mitigate the concern that HyperDAS (like all powerful interpretabilty methods) might inject new information into the target model rather than faithfully interpreting
present
INTRODUCTION
Mechanistic interpretability methods promise demystify the internal workings of black-box lan- guage models (LMs), thereby helping us to more accurately control these models and predict how they will behave. Automating such efforts is critical for interpreting our largest and most performant models, and strides toward this goal have been made for circuit discovery (Conmy et al. 2023 Ra-
jaram al. 2024) and neuron/ / feature labeling (Bills et al., 2023 Huang et al.) 2023 Schwettmann et al.) 2023 Shaham et al.) 2024). In the present paper, we complement these efforts by taking the first steps toward automating interpretability for identifying features of hidden representations (e.g., directions in activation space) that mediate concepts (Mueller et al. 2024 Geiger et al.) 2024a).
Interventions on model-internal states are the building blocks of mechanistic interpretabilltty (Saphra) & Wiegreffe [2024). To establish that features of a hidden representation are mediators of a concept, a large number interchange intervention (Vig et al. 2020 Geiger et al., 2020) experiments are performed the LM. Interchange interventions change features to values they would take if counterfactual input were processed. For example, if the concept is = the birth year of a person, fix the features F of an LM processing the input Albert Einstein was born in to the value they take for Marie Curie was a chemist. If the output changes from 1879 to 1934, we have a piece of evidence that F mediates C. The field has developed variety of methods for learning such interventions, but all of them require a brute-force search through potential hidden representations.
To address this significant bottlebeck, we propose HyperDAS, method automate this search pro- cess via a hypernetwork. In the HyperDAS architecture, a transformer-based hypernetwork localizes a concept within the residual stream of a fixed layer in a target LM by:
1. Encoding a language description (e.g., g.. the birth year of a person) of a concept using trans- former that can attend to the target LM processing a base prompt (c.g., Albert Einstein was born in) and counterfactual prompt (e.g., Marie Curie was chemist).

---

Published as a conference paper at ICLR 2025
love
Paris
[aeif]
Expected Output France
Select Positions
Identify Subspace
I
nna
Intervention in the Country Subspace
Aipori on
Faris
MIP
1s
"ina"" (eenna
2
in
Model States at Layer
Select Token Positions
Identify Feature Subspace
Manked Mutt Head
2
Hypernetwork
Model States
Norm
①
x
Multi-Head Cross attont
Counterfactual Model States
country city
Norm
M Manioed Mi lieiitted Self aecontion
love Paris Model States When Processing Base and Counterfactual (Right) Inputs
1
Natural Language Instruction
Natural Language Instruction
Figure 1: The HyperDAS framework, used here to find the features that mediate the concept of "country". (1) Concept Encoding A natural language description that specifies which concept to localize, "The country of city", is encoded by transformer hypernetwork with two additional cross-attention blocks attending to the hidden states the target LM prompted with the base text "Vienna is in" and the counterfactual text "I love Paris". (2) Selecting Foken-Positions With the encoding from step as a query, HyperDAS uses selects the tokens "nna" and "Paris" as the location of the concept "country" for the base and counterfactual, respectively. (3) Identifying Subspace With the representation from step 1 as the encoding, HyperDAS constructs a matrix whose orthogo- nal columns are the features for "country"". (4) Interchange Intervention With the poken-positions from step 2 and subspace from step 3, HyperDAS performs intervention by patching the subspace of the hidden vector for the token "nna" to the value it takes on in the hidden vector for the token "Paris", leading the model to predict "France" from the base prompt "Vienna is in".
2. Pairing tokens in the base and counterfactual prompts (e.g., align "Cur" with "Ein") with an attention mechanism using the encoding from (1) query and token-pairs as keys/values.
3. Selecting features of the residual stream via fixed orthogonal matrix that undergoes House- holder transformation (Householder 1958) using the encoding from (1)-
4. Patching the selected residual stream features of aligned tokens from the base prompt to the values they take on the residual stream of aligned tokens from the counterfactual prompt.
We benchmark HyperDAS on the RAVEL interpretability benchmark (Huang al. 2024). in which concepts related to a type of entity are disentangled. For example, we might seek to separate features for the birth year and field of study of Nobel laureate. HyperDAS achieves state-of-the-ar perfor- mance on RAVEL with a single model. Greater gains are achieved when a separate HyperDAS is trained for each entity type (e.g.. Nobel laureates).
Finally, we address the issue of whether HyperDAS is faithful to the target model. As we use more complex machine learning tools for interpretability, there is an increasing concern that are not uncovering latent causal structure, but instead injecting new information to steer or edit model (Meng et al. 2022, Ghandeharioun al. 2024). If we allow our supervised interpretability models too much power, we run the risk of posiiivos signals. Thus, we conclude with discussion of how our decisions about architecture, training, and evaluation were made to mitigate these concerns.
BACKGROUND
Automating Interpretability Workflows The growing size and complexity of language models demands scalable techniques for interpretability. Major directions include circuit analysis (Conmy
2

---

Published as a conference paper at ICLR 2025
et al. 2023 Marks al. 2024 Rajaram et al. 2024 Ferrando & Voita 2024), unsupervised feature
learning (Huben et al. 2024; Braun et [202] ) and feature labeling with natural language descrip-
tions (Mu & Andreas 2021 Hernandez et al.] 2022; Bills et al. 2023 Huang et al. 2023 Shaham
et al. 2024). In this work, we take steps towards automattig the process identifying features that
causal mediators of concepts using supervision from counterfactual data.
Identifying Features that Mediate Concepts Interchange interventions identify neural represen- tations that are causal mediators of high-level concepts (Vig et al., 2020; Geiger al. 2020 Fin- layson et al. [221]; Stolfo et al. 2023). >. Geiger et al. (2024b) and Wu et al. (024) further extend interchange interventions localizing concepts hidden vector subspaces. However, these meth- ods require an exhaustive search over all layers and tokens to measure causal effects at each position. In practice, the lack of an effective search method leads heuristics in token selection. For example, in knowledge editing and model inspection, widely held assumption is that the entity information is localized to the last entity token [Meng et al. 2022, 2023 Hernandez et al.) 2024 Geva et al. 2023] Ghandeharion et al. 2024), but this does not hold for all entities (Meng et al. 20022). Our proposed method directly addresses this problem by using an end-to-end optimization to automatically select the intervention site across all tokens, conditioned the concept localize.
The RAVEL Benchmark The RAVEL benchmark evaluates how well an interpretability method can localize and disentangle entity attributes through causal interventions. An example consists of a base prompt that queries a specific attribute of an entity (e.g., Albert Einstein studied the field), counterfactual prompt containing a different entity of the same type (e.g., Poland declared 201 I the Year of Marie Curie), an attribute targeted for intervention feel of study or birth year), and counterfactual label for the base prompt. The label would physics if the targeted attribute birth year (the intervention should not affect the field of study attribute), and it would be chemistry if the targeted attribute is field of study.
Distributed Interchange Interventions RAVEL supports evaluations with distributed inter- change interventions on features of a hidden representation H that encode an attribute in the original model M. In experiments. features are orthogonal directions that form the columns of low- rank matrix R. Given a base prompt & and a counterfactual prompt &, we perform an intervention that fixes the linear subspace spanned by R:
H h + R (R(h)- - R(h))
where h and h are the values that variable H has when the model M is run on x and x, respectively.
RAVEL Metrics The metric from the RAVEL dataset has two components. The Cause score is the proportion of interchange interventions that successfully change the attribute that was targeted, and the Iso score is the proportion of interchange interventions that successfully do not change an attribute that was not targeted. The Disentangle score is the average of these two.
Distributed Alignment Search The RAVEL evaluations distributed alignment search (DAS; Geiger et al. 2024b) as a baseline for learning a feature of a hidden representation that mediate an RAVEL example with base input counterfactual
attribute. A rotation matrix is optimized input &, and counterfactual label y using the following loss:
DDSS (R)) (x),
H-
where (x. is the output of the model M run on input x with an intervention 7- Only the parameters R are updated while the parameters of the target model M are frozen.
HYPERDAS
To localize concept in the layer of target model M, HyperDAS architecture consists of hypernetwork H that takes in a text specification x of the target concept and dynamically selects token positions in the base text X and counterfactual text & and identifies linear subspace R that mediates the target concept. Selecting tokens is a discrete operation, so we "soften" the selection during training and force discrete decisions during evaluation. Our specific model is follows.
3

---

Published as a conference paper at ICLR 2025
3.1 REPRESENTING THE TARGET CONCEPT A VECTOR
Token Embedding A token sequence x of length E that specifies the concept to localize, e.g., the country city is in, is encoded with the embeddings of the target model M to form ey Emb(x) x) Exx the zeroth layer of the residual stream for the hypernetwork H.
Cross-attention Decoder Layers After embedding the target concept, N decoder layers. Besides the standard multi-headed self-attention (MHA) and feed-forward layers (MLP), each decoder block has two additional coss-attention modules to incorporate information from the target model M procesing the base and counterfactual prompts x and X.
transformer with
Let h E R BxLxd and h E R. Cxxxx be the stacks of base and counterfactual hidden states from the base and the counterfactual input, where I, is the number of layers in M, is the hidden dimension, and B and C are the sequence length of the base and counterfactual examples, respectively. Two multi-headed cross-attention modules MHA and MHA allow H to attend to h and h. Each layer of the hypernetwrk H can attend to every layer of the target model.
For the p-th decoder layer of the hypernetwork H. the three attention mechanisms follows:
ep MHA(Q(ep)) K(e,), V(e,))
(3)
6p
" (HA(Q(e,)) K(h), V(h))
(4)
0p+1 MHA(Q(e)) K(h), v(h)
(5)
After the final transformer block is applied, the residual stream vector the last token position (N)
eg E T encodes information about the concept targeted for intervention and the target model's base and counterfactual runs. This representation is used to select ooken-positions in the base and source texts and identify linear subspace for intervention.
3.2 DYNAMICALLY SELECTING TOKEN- POSITIONS THAT CONTAIN THE TARGET CONCEPT
The next step selecting the tokens in the base and counterfactual prompts that contain the concept encoded eg (N) We construct an "intervention score" matrix G TR Bx(C+1) with elements ranging
from O to 1, where B and C' are the number of tokens in the base and counterfactual prompts, respectively. Each element G(b.c) denotes the degree to which the b-th base token is aligned with
e-th counterfactual token for intervention. The additional column G(b,e) corresponds to the score for retaining the b-th base token without any intervention.
In Figure only the buse-token "nna" (the last token of the entity "Vienna") receives an intervention score of when paired with the counterfactual-toenn "Paris". All other base tokens are not selected for intervention. The concept of "country" is localized to the last token of the city entities.
To compute the element G(v.e), we use h and (,((), the 1-th layer residual stream representation of the target model at b-th base-token and e-th counterfactual We first linearly combine the two:
g(b.c) F([h))) (0) (
where F(.) : IR 2d Rd is a linear projection that condenses the concatenated representation into the original dimension d. For the extra column that indicates no intervention, the representation simply the original base token representation:
g(,C++) h
Then, we weight each merged representation h using the concept encoding eg "query" and "key" matrices () R dx and k-(1) e dxarx
(N)
IR. and M
G(b.e) ΣM (N) )Q (b))() (eg
(8)
MVd
Finally, we apply column-wise softmax G ColumnSoffmax((G)
4

---

Published as a conference paper at ICLR 2025
Using the matrix G. we construct the representation that we will to intervene each token in the base prompt. For the b-th base-token hidden states hý. the interventio representation is:
(1) G(..++1hh h
h
This is how the counterfactual representation is constructed for a weighted interchange intervention (Wu et al.) 2024). The counterfactual hidden states remain identical to the base hidden states (i.e., no intervention on token b) when G(t.++1) = Conversely, if G(b.o) for specific position e, the 6-th base token is entirely replaced by the hidden vector for the c-th counterfactual token.
3.3 DYNAMICALLY IDENTIFYING A LINEAR SUBSPACE THAT CONTAINS THE CONCEPT
In addition to selecting token-positie sosen-posstions, HyperDAS also dynamically identifies linear subspace that contains the target concept, encoded as a low-rank matrix with orthogonal columns. First, apply a multi-layer perceptron to egg E Rd in order to produce new vector
(N)
MLP(ee (N)) E R
In DAS, there is a fixed low-rank matrix with orthogonal columns R' representing a fixed subspace targeted for intervention. We use linear algebra operation known the Householder transforma- tion change R' conditional on into new matrix R. that still has orthogonal columns. Given non-zero vector v R² the Houscholder transformation H is defined as:
= VV v
(10)
where I the identity matrix. The matrix is orthogonal and R' has orthogonal columns, which means R'H has orthogonal columns. Utilizing this property, we can dynamically select the subspace (N) based on the intervention representation eg by computing R R'H.
3.4 INTERVENING ON THE SUBSPACE AT THE SELECTED TOKEN-POSITIONS POSITIONS
After selecting token-posiin poccitoosssonss and identifying subspace, we finally intervene. For each base-token b in the 1-th layer of the target model M, we perform weighted interchange intervention with the counterfactual hidden states (1) and low-rank orthogonal matrix R:
H hg R T (R((b))) R((,))))
(11)
3.5 TRAINING
We train HyperDAS on the RAVEL with two losses. The first simply measures success on RAVEL. The second incentivizes the model to select unique loken-pairing that performance is maintained when the token alignment matrix G is snapped to binary values during test-time evaluation.
RAVEL Loss A RAVEL example consists of a base input counterfactual input x, target concept input x, and a counterfactual label y. When the target concept matches the attribute queried in the base input, the label y is the attribute of & Otherwise, y is the attribute of x. The loss is:
GRAVEL CE( MH "("+ +R" (R(6 ( (x).
(12)
Sparse Attention Loss
The construction allows a single counterfactual token be paired with weighted combination of multiple base tokens, however during test-time evaluations we will enforce a 1-1 correspondence. Thus, we include a sparse attention loss that penalizes cases where one counterfactual token attends strongly multiple base tokens in each row of matrix G:
C Sum (G.))) 2
if Sum
C Cyparse
T
if (Sum(G(a.a) $
(13)
The final loss is L LRAVEL + X where X is real-valued weight scheduled during training.

---

Published as a conference paper at ICLR 2025
Raw Intervnention
Intervention
[805]
.01
[P05]
Counterfactuel Sentence Tokens
The
01
The
Mach
86
0
0
0
0
Mach
0
0
0
0
0
0
0
.11
eng
1
0
0
3.4
official
.01
official
0
language
language
0
0
0
0
02
1s
01
0
0
0
0
0
[SELF]
.99
[SELF]
[805]
Spring
nol
Chise
(809)
Spring
mote
0.0
Base Sentence
Sentence Takens
Figure 2: The "intervention score" matrix G' for the counterfactual prompt "The city of Macheng's official language is" and base prompt "Springfield is a city in the country of", for which the target model will output "China" The attribute targeted for intervention is country, the output should be "The United States". The raw intervention (left) is produced by the token- position selection discussed in Sec and column-wise and row-wise argmax applied inference time to enforce an 1-1 correspondence between the base and counterfactual tokens, detailed in Sec. 3.6
3.6 EVALUATION
HyperDAS is end-to-end differentiable because discrete operations like aligning base and counter- factual tokens are "softened" using softmax operators. During the evaluation, we force these discrete decisions. In many cases, the matrix G contains non-zero weight for multiple base-counterfactual sentence token pairs (left panel in Figure 2) inference time, we perform a double argmax oper- ation the intervention score to select the most important location for the intervention. For each yase-counterfactuall token pair, it is set to if and only if this position gets the highest intervention score across its row and column.
[1 G(p.o) max( G(),a)) G(+,)) otherwise
G(b.e)
(14)
The [SELF] row, representing no intervention on base-token, is readjusted according to the discrete intervention weight (right panel in Figure
G(I.++)= 1 if max(G((.)))
(15)
4 EXPERIMENTS
We benchmark HyperDAS RAVEL with Llama3-8B (Meta 2024) the target model. We both train a HyperDAS model on all of RAVEL at once and also train a separate HyperDAS model for each of the five entity domains in the RAVEL benchmark, i.e., cities, Nobel laureates, occupations, physical objects, and verbs. We experimented with initializing the transformer hypernetwork from pre-trained parameters, but found no advantage for this task.
Crucial Hlyeeparaaeetess We use 8 decoder blocks for the hypernetwork and 32 attention heads for computing the pairwise token position attention. The sparsity loss weight is scheduled linearly increase from 0 1.5, starting 50% of the total steps. learning rate between x 10* x 10" is chosen depending on the dataset. Discussion of these choices concerning the sparsity loss is in Section 4.2 For the feature subspace, we experiment with dimensions from 32 up to 2048 (out of 4096 dimensions) and use subspace of dimension 128.
Masking of the Base Prompt As the hypernetwork has access to the target attribute information from the instruction and the base attribute information from the base model states, trivial solution

---

Published as a conference paper at ICLR 2025
Methods
Causal
City
Nobel Laureate
Causal
Causal
Occupation
Iso
Physical Object Causal
Verb
Iso
Disentangle
Average
MDAS
55.8
77.9
56.0
93.5
50.7
88.1
85.0
97.9
74.3
79.6
76.0
HyperDAS
Asymmetric
70.8
93.9
55.4
95.1
50.4
92.7
97.2
93.0
98.9
84.7
Asymmetric Domains
58.8
90.5
47.6
92.0
75.7
82.1
92.9
94.5
86.9
95.8
80.7
Symmetric
76.9
90.9
59.2
88.4
47.1
89.1
94.8
97.5
42.3
82.9
76.9
Symmetric All Domains
98.8
21.6
13.6
97.8
(a) Main results of HyperDAS on five domains of RAVEL with Llama3-8B. For cach method, we report the results from the best layer between 10 and 15. HyperDAS achieves Waae-of-the-an attribute disentangling performance the board. The MDAS baseline intervenes fixed tokens.
Saureiss/oeettagg Score
2.0
MDAS
Caupe/tsn Score
1.0
Hy pyrrAASSs
Gaurefss@@uentaaglee Score
1.0
HyperDAS-Sym
0.8
88
0.8
0.6
066
0.6
2.4
0.4
Cause
02
02
0.2
10
15
20
25
30
5
30
15
20
25
30
10
1.5
20
25
30
Intervention Layer
Intervention Layer
Intervention Layer
(b) The cause/iso/disentaaglle score of the baseline method and HyperDAS (Both symmetric and asymmetric implementation) for the entity type of "city" across the layers of Llama3-8B. For both HyperDAS and MDAS, the peak Disentangle Score happens around L15.
Figure 3: RAVEL benchmark results. HyperDAS establishes new state-of-the-art.
the hypernetwork can learn is to condition the intervention location on whether the target attribute matches the base attribute. In other words, if the two attributes match, attend to a location that has causal effect on the output, otherwise attend to the extra [self] row (see Appendix AA.3 for an example). This solution, however, does not find the actual concept subspace. To prevent the hypernetwork from learning this trivial solution, we apply an attention mask to the base prompt to mask out the attribute information. With the masking, the hypernetwork longer has access to the base attribute information, and hence the localization prediction only conditioned on the target attribute in the natural language instruction.
Symmetry Intuitively, if we have localized a concept, then "get" operations that retrieve the con- cept and "set" operations that fix the concept should both target the same features and hidden rep- resentations. For this reason, we consider a variant of HyperDAS that enforces symmetry in the localization of base and counterfactual prompts. We can enforce symmetry between base and coun- terfactual inputs during token selection by randomly flipping the order of the concatenation between base and counterfactual hidden representations in Equation 6 and asymmetric models.
We report full results for symmetric
Multi-task DAS (MDAS) Baseline The current state-of-the-ar method on RAVEL is MDAS, which uses multi-task learning objective satisfy multiple high-level causal criteria. MDAS requires supervised training data like HyperDAS, however, MDAS relies on manually selected token position for intervention, which in case is the final token of the entity, e.g., "nna" in Figure
Results In Table 3a we show results on RAVEL for layer 15 of Llama3-8B. In Figure 3b we also run HyperDAS targeting every layers in Llama3-8B starting from the embedding layer. The peak performance of attribute disentanglement for both MDAS and HyperDAS is around layer 15,
4.1
INTERVENTION BEHAVIORS OF HYPERDAS
HyperDAS searches for optimal location to intervene within the target hidden state in chosen layer. We evaluate MDAS and HyperDAS on 16 layers across the model (Figure (3b) and chose an early layer, middle layer, and deep layer for detailed study: Layer 7, Layer 15, and Layer 29.
7

---

Published as a conference paper at ICLR 2025
3003
Number of Cases
2002
402
Olns
Shallww Courtertacual
Mild Counterfacua
Vildi
Deep Costiettccus
base
Intervention Layer Sentence
Figure 4: The intervention location, in counterfactual and base sentence, picked by HyperDAS when targeting shallow (7), middle (15) and deep (29) decoder layers.
Analysis presented in 4 that HyperDAS consistently targets entity tokens in the coun- terfactual input across all layers, suggesting robust detection of attribute information in the entity token's ss residual stream from an early stage. However, the choice of intervention location within the base input shows significant variation. For each example the "city" entity split, we categorize base and counterfactual token pair that gets the largest intervention weight, and classify them into the following categories: (1) BOS Token represents the begimning-of-sentten of-sentence token. (2) Entity Token refers to tokens representing entities. (3) JSON Syntax includes special characters and syn- tactic tokens typical of ISON formatted text (e.g., opening curly brace "("). (4) Others comprises all tokens irrelevant the current analysis, with "is" following the entity token being common example in both shallow (36%) and deep (29%) layer bases.
At very early layers, HyperDAS displays turbulent behavior, targeting random even beginning-of sentence tokens in the base sentence. By the middle layers, the model consistently favors the entity token for intervention, aligning with findings from [uang al. (2024) and Geva al. (2023). In contrast, at deeper layers, the hypernetwork learns to intervene on unintuitive positions such as syntax tokens within -ormattrtteed prompt, which were previously unknown store attributes.
4.2 DISCUSSION
HyperDAS establishes a state-of-the-art performance RAVEL Our results show that HyperDAS outperforms MDAS, the previous state-of-the-ard across all entity splits at layer 15 in Llama3-8B and across all layers of Llama3-8B for the "cities" entity split.
HyperDAS requires more compute than MDAS. HyperDAS is more powerful than MDAS, but also more computationally expensive. Training our HyperDAS model for one epoch on disentan- gling the country attribute in the city domain takes 468,923 TeraFLOPs, while training MDAS model for one epoch on the same task takes 193,833. Thus, HyperDAS requires = 2.4x compute.
On the other hand, HyperDAS is more memory efficient for tasks like RAVEL. Our target Llama model requires 16GB of RAM. The HyperDAS model requires 52GB more in total for RAVEL, whereas MDAS requires only 4.1 more per attribute. The memory usage of HyperDAS does go up with additional attributes, so when trained on all of RAVEL together (23 attributes), MDAS (23 * 4.1 16 110.3GB) exceeds the memory usage of HyperDAS (52 16 68GB).
Householder vectors analysis provides window into attribute features. To analyze the House- holder vectors generated by the model, we collected vectors from each test example and categorized them according to their respective attributes. For each attribute category, subset of 1,000 sam- ples was randomly selected. We then computed the similarity scores between pairs of attributes by calculating the average cosine similarity across these 1,000 pairs of selected Householder vectors.
8

---

Published as a conference paper at ICLR 2025
Texwser
Laguu Ldiee
Figure 5: The relative position between the Householder vector (after PCA) of attributes for all the correct predictions in city domain. The clustering indicates that HyperDAS learns dif- ferent subspace for each attribute.
Figure 6: The cosine similarity between the Householder vectors of different attributes in the city domain, computed using 100,000 samples from each attribute. Notably, HyperDAS effec- tively learns highly similar subspace for the attributes 'Longitude" and 'Latitude'.
With Sparsity Loss
With No Sparsity Lass
With Top Much Sparsity loss
countertactual Senteence
omcy dut trenss court
duty
Dase Sentence Tokens
Base Sentence Tokens
Base Sentence Tokens
Figure 7: Intervention locations for a base/counterfactual sentences pair with Occupation entity-type selected by HyperDAS trained with different amounts of sparsity loss. This comparison illustrates the intervention locations generated by HyperDAS when trained under three different sparsity loss conditions. All three models achieved Disentangle Score 94.0% using weighted interventions. With no sparsity loss (middle), HyperDAS tends to intervene from the last subject token in the counterfactual sentence most tokens in the base sentence, which yields adequate performance under many-to-one constraints but not under strict one-to-one constraints. With too much sparsity loss (right), the pairwise token selection attention within HyperDAS fails, resulting interventions that blend all hidden states. Although this approach achieves near-perfect disentangle score with weighted intervention, the model's 's does not have interpretable intervention patterns and fails entirely during test time when masks are snapped to align base and source tokens one-to-one.
sentere
We analyze the geometry of the learned householder vectors, with the PCA projection shown in Figure We also compute the average pairwise cosine similarity of Householder vectors sampled from within the same attribute cross two different attributes, shown in Figure 6 Despite overall high cosine similarity among all Householder vectors associated with the same entity type, the Householder vectors associated with the same attribute form tighter cluster, with higher cosine similarity score than pairs of vectors associated with two different attributes. These per- attribute clusters might explain why the learned subspace can disentangle different attributes of the same entity, as different attributes are localized into different subspaces of the entity representation.
How do we know HyperDAS uncovers actual causal structures faithful to the target model? On one hand, we should leverage the power of supervised machine learning to develop increasingly sophisticated interpretability methods. On the other hand, such methods are incentivized to "hack" evaluations without uncovering actual causal structure in the target model. We have taken several steps to maintain fidelity to the underlying model structures when training and evaluating Hyper- DAS, by constraining optimization flexibility to prevent inadvertently steering or editing the model with out-of-distribution interventions.

---

Published as a conference paper at ICLR 2025
Syerontt Moxd
Eviy
Laiguage
Lengtade
Langagge
ISYARVER
Figure 8: The count of intervention location picked by HyperDAS at the counterfactual prompt (upper) v.s. the base prompt (bottom) across all the attributes in the city domain on entities with three tokens. The asymmetric variant (right) of HyperDAS favors getting the attribute information from the last entity token for the majority of the counterfactual prompts (z 95%). and intervene on the second last entity token. The symmetric variant (left) favors last entity token consistently for both base and counterfactual prompt.
The weighted interchange interventions used in training hacks the objective without soft con- straints via loss terms. The loss term Lsparse is crucial for ensuring that HyperDAS learns a one- to-one alignment between base tokens and counterfactual tokens (Figure 2 When no sparsity loss is applied, the model aligns the final entity token (c.g.. "nna" from Figure to many tokens in the base sentence. These solutions fail during evaluations where token alignments are snapped to be one-to-one. Conversely, with excessive sparsity loss, the model constructs a counterfactual hid- den representation that is linear combination of many hidden states, resulting in high flexibility optimization scheme that is closer to model steering or editing. This also fails during one-to-one evaluations. See Figure for an example of these pathological settings.
Often only one token is aligned between base and counterfactual inputs. The MDAS baseline performs well on the RAVEL benchmark by one token in the base and one token in the source. However, our new state-of-the-art HyperDAS model will select multiple tokens 53% of the time.
Asymmetric HyperDAS targets different tokens for base and counterfactual examples. Fig- ure [ shows the tokens selected by the symmetric and asymmetric variants of HyperDAS. When allowed asymmetric parametrization, networks break symmetry in positional assignments; for a sin- gle input prompt, HyperDAS will select different tokens depending whether that input is the base or counterfactual.
5 CONCLUSION
In this work, we introduced HyperDAS, novel hypernetwork- approach for automating causal interpretability work. HyperDAS achieves sate-of-the--e art performance on the RAVEL benchmark, demonstrating its effectiveness in localizing and disentangling entity attributes through causal inter- ventions. Our method's 's ability to dynamically select hidden representations and learn linear features that mediate target concepts represents significant advancement interpretability techniques for language models. We are optimistic that HyperDAS will open new avenues for understanding and interpreting the internal workings of complex language models.
Limitations HyperDAS will only be successful the target concept is mediated by linear features. However, there is emerging evidence that non-linear mediators are possibility (Csordás et al.) 2024 Engels et al. 2024). As discussed extensively in the main text, applying supervised machine learning interpretability has the potential lead false positive results. While have taken steps to maintain fidelity to underlying model structures. future work should continue to explore the delicate balance between uncovering latent causal relationships and the risk of model steering.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research was part supported by grant from Open Philanthropy.
REFERENCES
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in lan- guage models. URL httss:/oppeaiipuliccc blob. core. windows. nethnewron-explainee nedheuumn-zphaiiee/pprr/nnde html.(Date accessed: 14.05. 2023), 2, 2023.
Dan Braun, Jordan Taylor, Nicholas Goldowsky-DD -iil, and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning, 2024. URLhhttp:///axiv org/
abs/2405. 12241
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso Towards automated circuit discovery for mechanistic interpretability. Al- ice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA. Decem- ber 10 16, 2023, 2023. URL http: / httt::://pppers..niiii ccp.. cc/ paper files/paper htss/paper/222//has// 2023/hash/ html
Róbert Csordás, Christopher Potts, Christopher D. Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations, 2024. URL
https://arxiv. wrg. abs 2408 10920
Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model 14860
features are linear, 2024. URL http:/:://aaxxi.orr/ab//////////
Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale, 2024. JRLhhttps://rrxiworrg aa////bs//223// 00824
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieher, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Chengging Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 1928-1843, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 htt/://111/211llllooooo acl-long.11e 144. URL
wwwp:ey..org......e../e/ acl wrt..//..2222...11oooo.... 144
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Afra Alishahi, Yonatan Belinkov, Grzegorz Chrupata, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 163-173, Online, November 2020. Association for Computational Linguisties. doi: 18.18v///v1/ 2020.blackboxnlp-11. 1.16. URL https: a aclanthologgyyyy org. 2020. blackboxnlp-1.01
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability. 2024a. URL https:// 04709
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Find- ing alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pp. 160-187. PMLR, 2024b.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216- 12235, Singapore, December 2023. Association for Computational Lin- guistics. doi: 18.18 101116653/112222.mmm 1/t.33//11//222////.pp main. 751. URL https: https://aallanthology emnlp-main. 751
11

---

Published as a conference paper at ICLR 2025
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL htwphhttss//openeeheheeeee wwtp://www......oooomooooueeeeee
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2022. URL http::://axxv..or///////
11114
Evan Hernandez, Belinda 7. Li, and Jacob Andreas. Inspecting and editing knowledge rep- resentations in language models. In First Conference on Language Modeling, 2024. URL
https://openreview.rrr wttpe/rrreeeee.eeeoouuuiiiiieaee www..e//ooorumiidDttffffeee http:::://errr/oo////////t/oaaap
Alston $ Householder. Unitary triangularization of a nonsymmetric matrix. Journal of the ACM
(JACM), 5(4):333322, 1958.
Jing Huang, Atticus Geiger, Karel D'Oosterlinck, Zhengxuan Wu, and Christopher Potts. Rigor- ously assessing natural language explanations of neurons. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, Black- boxNLP@ EMNLP 2023, Singupore, December 7, 2023, pp. 317-331. Association for Com- putational Linguistics, 2023. doi: 10.18653/V1/2023..11 12B.655/12222BBBBXBXXOOO BLA....XLLLKKLLLLP.. 7/doi.org/10 181.1...16633//12000 1ttp66//11112222..2acoooooool1/.
URL https:
Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8669-8687, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10.18653ww120024.aclloog URL httpp::/aalaantholleee org/ 2824 www.:rr/2222..c111oo...4444 470
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:// www.n//rfoorrmieee F76bwRSLeK]
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language mod- els. Computing Research Repository, arXiv:2403. 19647, 2024. URL htttpp::/aaxii.orgg///// 2403. 19647
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual sociations in GPT. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, Novem- ber 28 December 9, 2022, 2022. URL http: http:::pppppp.pppiteet cc/ wwpppper.pppppppeepeeeeeeeee.. files/paper/200 31 82-
hash/ hash/6f1d43d558222 (4l.14444558223773856653333 oor.://55e..t.tttaa--a-aatrrrrrr .tml
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh Internatioonal Conference on Learning Representations, 2023. URL / wttp://wwwww...etteeoooooottedaa
AI Meta. Introducing meta llama 3: The most capable openly available Ilm to date. Meta Al, 2024.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons, 2021. URL/htps::/aarat org/abs/2000 14032
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Amab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability, 2024. URI. https:////x/iii... wwwp./wwwww.rrg///////..../ 01416
12

---

Published as a conference paper at ICLR 2025
Achyuta Rajaram, Neil Chowdhury, Antonio Torralba, Jacob Andreas, and Sarah Schwettmann. Automatic discovery of visual circuits. CoRR, (bbx/2404.143499 2024. doi: 10.48550/ARXIV.
2404.14349. URL http:::://oi..rrg/////// 48550/arXiv 2404 14349
Naomi Saphra and Sarah Wiegreffe. Mechanistic?, 2024. URL https://axxiv.or//ab////11
09087
Sarah Schwettmann, Tamar Rott Shaham, Joanna Materzynska, Neil Chowdhury, Shuang Li, Jacob Andreas, David Bau, and Antonio Torralba. FIND: function descrip- tion benchmark for evaluating interpretability methods. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://papers.a.e www.:nipssp.paaaaee cc/paper htles//paprr/223333 /2223/hash/ Benchmarks...... html
Neural In-
Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and Antonio Torralba. A multimodal automated interpretability agent. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: /oppnree/o net/ forum?id-mDw42zamme
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arith- metic reasoning language models using causal mediation analysis. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 7035-7052. Asso- 435. URL
ciation for Computational Linguistics, 2023. doi: 111183//11202333
https://doi. org/10. 18653/v1/2022 emnlp-main. 435
Jesse Vig, Scbastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Asso- ciates, Inc., 2020. U neur /ttp:///....ccccccleeee////////p 92555522227777533322577722222232
Zhengxuan Wu, Attieus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. Interpretabil- ity at scale: Identifying causal mechanisms in alpaca. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 HYPERDAS OVER ALL DOMAINS
show that we can train HyperDAS to achieve state of the art performance on the RAVEL benchmark by training a separate model for each entity type split, which is the set up used to train the previous state of the art MDAS. To test the scalibility and generalizability of HyperDAS, we train single model across all the entity type splits and evaluate its performance.
Our results at Table 3a
Experiment Set-up We aggregate the training split of the dataset from all 5 domains and train HyperDAS for 5 epochs. We adjuste the learning rate from 2 x 10 -5 to 5 x 10 and schedule the sparsity weight À ranging from 0.75 1.5 starting after 50% of the total steps. This set-up allows the model to first find a stable solution across all domains with soft intervention before forcing it to converge to a single token selection.
Result We report the performance of HyperDAS trained over all entity type split in Table Ba The model performs better than MDAS but slightly than HyperDAS trained individual entity type split by 4.0%. Specifically, HyperDAS-All-Domaal performs worse over city and nobel laureate split, better over occupation split, and on-par over physical object and verb split.
A.2 DATASET SPECIFICATION
Domain/Attribute
of Cause Example
of IsolateExample
of Entity
City
34899/7016
49500/9930
3552/3374
Country
7925/1544
8250/1655
3528/2411
Language
6207/1252
8250/1655
3471/2221
Continent
8254/1658
8250/165
3543/2567
Timezone
5371/1144 144
8250/1655
3414/1900
Latitude
3813/743
8250/1655
3107/1519
Longitude
3329/675
8250/1655
2989/1357
Nobel Laureate
39771/6754
44628/7600
928/928
Country of Birth
7218/1356
8908/1520
928/909
Award Year
11037/1904
8930/1520
928/926
Gender
854/96
8930/1520
592/149
Field
9518/1558
8930/1520
928/922
Birth Year
11144/1840
8930/1520
928/927
Occupation
54444/1582
29052/864
799/785
Work Location
24216/724
9684/288
799/708
Duty
12090/371
9684/288
785/522
Industry
18138/487
9684/288
799/600
Physical Object
491 99114/6659
35285/3636
563/563
Color
14707/1518
8825/909
563/563
Category
13540/1273
8820/909
563/562
Texture
14666/1266
8821/909
563/561
Size
6201/603
8819/909
563/528
Verb
70003/3806
14396/782
986/984
Past Tense
34043/1844
7188/391
986/975
Singular
35960/1958
7208/391
986/978
Table 1: The details the dataset used for the experiment, the format of train/test splits. For every model in each setting. Methods are trained on the full dataset of that setting with 5 epochs. The prompts used by the train/test splits are completely disjoint.
14

---

Published as a conference paper at ICLR 2025
A.3 DATASET PREPROCESSING
HyperDAS uses an attention mechanism to gather information from the hidden states of the target model M when running the base and counterfactual sentences. This makes HyperDAS overtly powerful in some situations. Consider the following input:
Base
Vienna, known for its Imperial palaces, city in the country of
Counterfactual
& love Paris
(16)
Instruction
X Localize the latitude of the city
the model works as intended, it will intervene on the 'Latitude' subspace, which will leave the 'Country' features intact and therefore the target model will predict Austria.
However, since the model can access the hidden states M(x). knows that the queried attribute in the sentence is 'Country', which is different than the targeted attribute 'Latitude'. Through train- ing, HyperDAS learns a shortcut to a trivial solution solution-perfomeee performing no intervention when the target attribute is different from the one mentioned in the sentence. With this shorteut, the Isolate objec- tive no longer works and the HyperDAS fails to learn disentangled feature subspaces for different attributes.
Figure 9 shows how the HyperDAS may learn a trivial solution to the RAVEL benchmark the relevant information (base prompt attribute) can be accessed by the model.
Instruction:
Goroka Zurich Country
Label: Oceania
Pred: Oceania
[BOS]
00000000000000000000000000000000
city
0000
to
0.8
country
Buenos
0.6
Aires
in
0..4
Argentina
Zurich
0.2
in
000
[SELF]
[B05]
Dell
AS8
00
continet
Figure 9: The trivial solution learnt by the HyperDAS on isolate examples when no mask is applied on the attribute token in the prompt. HyperDAS learns to intervene if it sees the base prompt attribute to be different than the attribute in the instruction.
Therefore, for each pair of prompts x, x at training, apply intervention mask to all the tokens starting from the attribute mention. The hidden states from token with intervention mask is not visible to HyperDAS and therefore cannot be selected for intervention. The example becomes:
15

---

Published as a conference paper at ICLR 2025
Base
X Vienna, known for its Imperial palaces, is a city in the country of
Counterfactual
X I love Paris
(17)
Instruction
X Localize the latitude of the city
where the hidden states of the red text is masked from the HyperDAS.
A.4 LOADING HYPERDAS WITH PRE-TRAINED PARAMETERS
We have also explored initializing the HyperDAS from pretrained LM instead of initializing it from scratch. With Llama3-8h (Meta 2024) as the target LM, we initialize the modules of HyperDAS. besides the multi-head cross-attention heads and pairwise token position scores attention heads, as the copy of the parameters from the target model. We then evaluate the performance of this variation of the model on the city dataset of RAVEL (Huang et 2024).
Inkialication
Irita Ize from protrained
Inita Ize from scratch
(raminng l05.
Figure 10:
The loss curve of HyperDAS initialized from scratch or from pretrained LM while train- ing on the city dataset of RAVEL.
In Figure 10, we observe that there is no significant difference between the model initialized from scratch and the model initialized from Llama3-8b parameters. However, it remains unknown how would this difference change as the training of HyperDAS scales.
A.5 SPARSE AUTOENCODERS
We experiment with different feature subspace dimension, as shown in Figure [1 We add an trained sparse autoencoder another baseline. Following the exact same setting in (Huang al. 2024). we train sparse autoencoder that projects the target hidden states into higher-dimensional sparse feature space and then reconstruct the original hidden states.
A.6 ABLATION RESULTS
See ablations in Table 12
A.7 INTERVENTION PATTERNS
Here we include few demonstrations of the intervention pattern that HyperDAS generates on RAVEL, shown in Figure 13
16

---

Published as a conference paper at ICLR 2025
50:098
GOAL
00:899
32.58
25.018
50.0%
SUORS
32.8%
HyperDAS
50.599
MDAS
26
02
0.4
0.6
0.8
1.0
Causal Score
Figure 11: Cause (x-axis) and Iso (y-axis) scores trade-off for HyperDAS, MDAS, and SAE when using different feature size shown as the ratio %. GOAL (1, indicates the score with which the method is able to disentangle the feature subspace perfectly.
Ablation
Causal
Iso
Disentangle
HyperDAS
70.8
93.9
82.4
-No Cross Attention
68.2
83.9
76.1
No DAS
0.8
97.4
49.1
-No Hypernetwork
15.1
46.9
31.0
Figure 12: Ablation results for HyperDAS. No DAS has no rotation matrix and intervenes on entire hidden representations. No Hypernetwork replaces concept encoding via transformer with a vector lookup. No Cross Attention removes attention head submodules connecting the hypernetwork and target model.
17

---

Published as a conference paper at ICLR 2025
A
60
Ing
114
05
64
log
ing
ISUT
1537
Infervettinn
51. Many Interven
C.Ce-lo-Ore ndarvention
300
leng
0007
Dave Sentenck Tokera
Zart Sorkeros Thorr
Duse Serfence Tahen
(b)
Sint intervstiion
bi co Vonry Intervertion
Oree Eiee inneverdoo
Aatrels
CRE
Base
Bast
(d)
Figure 13: Four types of intervention patterns.
18


Reviews:
[{"review_id": "fo1eMikdO5", "reviewer": "Reviewer_vVg5", "summary": "This work proposes a method for automating the selection of particular linear directions in feature space that represent interpretable concepts or features. In prior work, model steering or activation patching has been performed utilizing optimization or datasets of prompts, but each method requires some manual effort or search in order to pair a particular neuron to a corresponding concept. This work proposes to use a hypernetwork that is conditioned on a counterfactual prompt, as well as intermediate features of an LLM with respect to a base prompt. The hypernetwork predicts for each token of the counterfactual prompt, its corresponding position in the base prompt in addition to an aligned counterfactual representation that is able to \"override\" the base prompt features. \n\nThe hypernetwork is trained on RAVEL, with cross entropy to enforce correct token pairings, as well as a sparsity loss to encourage a one to one mapping between tokens. When evaluating on RAVEL, generations are scored according to whether or not the target attribute was successfully changed, and also on whether or not untargeted attributes were left alone. Evaluations compared to MDAS are favorable, and ablations show\n\n----------------------------------------------------------------------\nI believe this work will be of interest to the community. I hope that the discussed changes and additions make it into the final camera-ready. I will keep my score.", "questions": "1. What makes this a hypernetwork? Hypernetworks should predict some weights of a target network. \n\n2. Since you don't actually clamp features to 0, sparsity is only softly encouraged with the sparsity loss, meaning there is always some feature entanglement. I'm confused about why adding too much sparsity loss results in poor performance. Doesn't better disentanglement imply that we're targeting only the target attribute? \nMore specifically, figure 8 states \"[...] it demolishes the model’s ability to form interpretable intervention patterns and adhere to specified constraints\". What does this mean?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality / Significance: \nAn automated method for finding / aligning concept directions at the token level is significant. I also think its important to note that it even performs better in evals than non automated methods. \n\nClarity/Writing: I found the paper to be well written. \n\nResults:\nOne thing I found particularly compelling was the ISO score of HyperDAS on RAVEL. When performing interventions for model steering (or even SAEs), there is often a failure to isolate/disentangle particular directions, yielding steering in a spurious direction. A very high ISO here is a good sign that the discriminative power of the hypernetwork is high. \n\nI also appreciated the discussion of what we're really investigating or uncovering when we train supervised interpretability tools.", "weaknesses": "The largest weakness of this work is lack of baselines or evaluations, there is the one dataset and one baseline method. Not that interpretability methods have to be quantitatively better than others, but contextualization helps. I would encourage showing additional baselines. \n\nThis does not influence my score, but I am broadly concerned that any method trained to perform these interventions is adding information that is not in the model under investigation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qYNd7OTf4Q", "reviewer": "Reviewer_7H7A", "summary": "This work proposes a hyper-network based approach to enforcing causal interventions  in foundation models. Given an instruction and two text inputs, one base input and a counterfactual input, the model aims to localize the token in the base input which answers the instruction and instead return the counterfactual input answer. All other aspects of the output should remain unchanged. To achieve this the model has two output heads: one for finding the corresponding answers in the two inputs, and one for transforming the token embedding of the base entity to the counterfactual embedding for the corresponding entity. Some ablations are conducted on the localization head which requires sparsity to work effectively. It is shown that for most entities in the RAVEL dataset the HyperDAS approach outperforms the MDAS baseline, especially at the Iso score which depicts that interventions are more controlled than for MDAS. This increase in Iso score never corresponds to a drop in Causal score and so the overall disentanglement score of the model is improved over MDAS.", "questions": "I have left some questions in the weaknesses section above which I would like answered. However two clear question are:\n\n- How much extra computational overhead is added when using HyperDAS over MDAS?\n- A small one: is $y$ missing from the end of Equation 13?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# Originality\nThe use of a hyper-network to learn to automate causal interventions is novel to my knowledge. The exact implementation itself also appears novel, with the location and intervention heads being a modular and measurable approach to training the interventions. This also means that the separation into Causal and Iso scores is possible and this is useful for evaluating the model.\n\n# Quality\nOverall the model work is of a high quality with clear hypotheses and appropriate experimental design. I find the separate consideration of Iso and Causal scores to be useful and lend insight into the behaviour of the model. The more detailed consideration of the localization head is also useful and are important ablations in the study. In most cases limitations are clearly acknowledges - for example on lines 421 to 429 where it is mentioned that the model is sensitive to the sparsity hyper-parameter.\n\n# Clarity\nOverall the  paper is well written and language appropriate. The structure of the work is also intuitive and aids with the understandability of the work. Figures are clear and legible with helpful captions.\n\n# Significance\nThe improvement at disentanglement of the model over the baseline presents a clear step forward. In addition the detailed ablations and insight into the localization head could lead to future work. Overall I think this work is of clear interest to the field and makes a clear contribution towards causal interventions in transformer models and working with foundation models.", "weaknesses": "# Clarity\nSome portions of the model are not fully explain. For example in Equation 1 the inverse of the low-rank orthogonal matrix R is used ($R^{-1}$) but if this matrix is low rank how is it inverted? Is this a pseudo-inverse? Another example is how the localized token positions are used to apply an intervention. In Figure 1 (left) it is shown that the positions are fed into the intervention module but Figure 1 (right) does not show this. I think exactly what the input and output of each head is could be described in far more detail. For an intricate model this is very important and limits clarity. Another part of the model which is not explained is the learning of the rotation matrix. Firstly, why is it necessary to learn $R'$ first and then apply the Householder transformation? Why not directly learn $R$? What does the Householder represent in an embedding space such that it performs a useful computation here? I do think on the whole the model and its workings become evident but this could still be far clearer and more explicitly explained.\n\n# Quality\nOn the side of quality I think a couple limitations are still not given due consideration. Particularly, the fact that a hyper-network is trained for every entity type. This seems to add a huge amount of computational overhead and limits the scalability of this approach. However, beyond line 92 I do not see this being discussed anywhere. Similarly, there are a couple statements such as the claim that the method helps ``crack open black box models'' on line 415 which do not quite seem true. I am not certain how this  approach does this and it is not clearly explained. Similarly I think there is a lingering assumption that a single token in the base input maps to a single counterfactual token. This seems to be a property of the dataset (which is totally fine) but then I am not certain the black box has been cracked open when the model then identifies this property of the dataset. Lines 514 and 515 have a similar issue for me. Perhaps I am not appreciating some nuance to this, but I think the model makes a clear contribution without needing to be too far reaching in its claims. For example, the ability to easily and clearly manipulate the black box seems equally impressive to me - especially when considering the precision of the approach demonstrated by the Iso score. Similarly, for Figure 6 - it is stated that HyperDAS learns different feature subspaces for different attributes but in general the clusters are very tightly packed. I don't think that this statement is clear from the figure but also doesn't seem to be the main point of the work anyway. Lastly, I think more consideration should be given to the fact that when using too much sparsity loss the model does not behave appropriately but still obtains a near perfect disentanglement score. This demonstrates a limitation of the experimental design. I note that this phenomenon is  noted in the work, but it stops short of actually acknowledging it as limitation of the method which is necessary. I recommend the work is revised such that the claims are made more exact.\n\nIf I am not misunderstanding something, then I would increase my score one level for each of the two main points above: 1) more clarity needs to be given on the exact details of the model, 2) the claims needs to be made more precise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MF4fMlWraj", "reviewer": "Reviewer_iDST", "summary": "The authors present a method for identifying directions in the activation space (hidden state) of language models corresponding to chosen concepts of interest. In particular they present a technique which takes as input a base prompt, a counterfactual prompt and a concept label. It then attempts to isolate and transfer the representation of the concept from the hidden state of the counterfactual prompt to the hidden state of base prompt (by intervening on the the forward pass of the base prompt).\n\nTheir technique is a new variation of the DAS method. DAS learns a rotation and projection of the the activation space that isolates a causally mediating direction for a particular concept.\nTheir new method, HyperDAS, uses an additional transformer hypernetwork to choose which token positions in the base prompt to intervene on and which token positions in the counterfactual prompt to read activation values from. The hypernetwork uses attention blocks which read keys and values from the base and counterfactual hidden states.", "questions": "- Is the hypernetwork highly specialized to the distribution of the RAVEL benchmark, or can it be used to isolate concepts using real texts from more natural sources?\n- Can a single network be trained that performs all of the tasks in RAVEL? It seems it should be possible to train a hypernetwork to make very general interventions, given the natural language input.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Their method seems like a necessary and logical extension to address the problem of token matching when performing interventions with the DAS method. They report state of the art results on the concept disentanglement benchmark. This line of research seems like a promising approach to better understand the internal states of neural networks.\n\nThe authors conduct detailed ablation experiments that help to isolate which aspects of their methodology are most important to achieve strong performance.\n\nThey also are careful to consider whether their intervention is philosophically justified, or whether they are adding too much complexity to faithfully interpret the model. I feel mostly persuaded that their method does in fact uncover properties of the underlying model, rather than learning new representations (or at very least, it does not seem to be worse in this regard than DAS).", "weaknesses": "Figure 3(b) makes the claim that HyperDAS beats MDAS appear less impressive. At many intervention layers, MDAS appears to be superior, so this result feels somewhat cherry-picked, although it is true that layer 15 is by a small margin the best layer for both methods. (However, the claim on line 377 seems to contradict my reading of the graph, so I may be misunderstanding something). \n\nAlthough it is briefly explored in one of the ablations, the authors do not adequately explain why they train the hypernetwork to take natural language input. They train a separate hypernetwork for each domain, so it seems that there would be a limited number of possible queries, and they would be no need to train a general language model such that the intervention can be specified in natural language.\n\nNitpicks:\n- Abstract: “identifying neural networks features” -> “identifying neural network features”\n- Line 142: “targe concept”\n- Line 159: the index j is not defined (and clashes with the j-th token index used later)\n- Line 481: “For examples”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XAvGaBWTAm", "reviewer": "Reviewer_Amxz", "summary": "This work presents an automated and scalable method for locating features associated with semantic concepts in language models. Given a natural language description of a concept, as well as base and counterfactual prompts, a transformer encodes the description and uses the prompts to locate token positions relevant to the concept, and the internal feature responsible for representing the concept. They evaluate their method on the RAVEL dataset, achieving SOTA performance, and they verify that their method reveals true causal structure via steering experiments.", "questions": "Is there some way of utilizing a pre-trained language model to assist in interpreting the natural language instruction, as opposed to relying solely on a model trained from scratch on RAVEL?\n\nWhat was the motivation for using the Householder transformation?\n\nHow does this approach to locating concepts relate to the approach of representation reading (as in Zou et al. (2023) \"Representation Engineering\")?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Interesting and creative application of transformers towards automated, scalable interpretability methods\n\nLocating internal semantic concepts has important implications for steering/controlling model behaviour to better align with intentions", "weaknesses": "The RAVEL dataset and the notion of localizing features could be explained more clearly\n\nAn explanation of the role of the Householder transformation would be useful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)