Low-Rank Adapting Models for Sparse Autoencoders
Authors: Matthew Chen
Joshua Engels
Max Tegmark
Venue: ICML 2025 poster (2025)
Keywords: Mechanistic Interpretability
Sparse Autoencoders
Finetuning
Steering
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We find low rank adapting models after SAE training leads to pareto improvements in model interpretability at a fraction of the cost of existing methods.

Abstract: Sparse autoencoders (SAEs) aim to decompose language model representations into a sparse set of linear latent vectors. Recent works have improved SAEs using language model gradients, but these techniques require many expensive backward passes during training and still cause a significant increase in cross entropy loss when SAE reconstructions are inserted into the model. In this work, we improve on these limitations by taking a fundamentally different approach: we use low-rank adaptation (LoRA) to finetune the *language model itself* around a previously trained SAE. We analyze our method across SAE sparsity, SAE width, language model size, LoRA rank, and model layer on the Gemma Scope family of SAEs. In these settings, our method reduces the cross entropy loss gap by 30% - 55% when SAEs are inserted during the forward pass. We also find that compared to end-to-end (e2e) SAEs, our approach achieves the same downstream cross entropy loss 3$\times$ to 20$\times$ faster on Gemma-2-2B and 2$\times$ to 10$\times$ faster on Llama-3.2-1B. We further show that our technique improves downstream metrics and can adapt multiple SAEs at once. Our results demonstrate that improving model interpretability is not limited to post-hoc SAE training; Pareto improvements can also be achieved by directly optimizing the model itself.

Low-Rank Adapting Models for Sparse Autoencoders
Matthew Chen
Joshua Engels
Max Tegmark'
Abstract
2.70
Topk TopK LoRA
Sparse autoencoders (SAEs) decompose language model representations into sparse set of linear latent vectors. Recent works have improved SAEs using language model gradients, but these tech- niques require many expensive backward passes during training and still cause significant in- crease in cross entropy loss when SAE reconstruc- tions are inserted into the model. In this work, we improve on these limitations by taking a fun- damentally different approach: we use low-rank adaptation (LoRA) finetune the language model itself around previously trained SAE. We ana- lyze our method across SAE sparsity, SAE width, language model size, LoRA rank, and model layer on the Gemma Scope family of SAES. In these settings, our method reduces the cross entropy loss gap by 30% to 55% when SAEs are inserted during the forward pass. We also find that com- pared end-to end-to-end (e2e) SAEs, approach achieves the same downstream cross entropy loss 3x to 20x faster on Gemma-2-2B and 2 to 10x faster on Llama-3.2-1B. We further show that our technique improves downstream metries and can adapt multiple SAEs once without harming general language model capabilities. Our results demonstrate that improving model interpretabilityy is not limited to post-hoc SAE training: Pareto improvements can also be achieved by directly optimizing the model itself."
CE Loss
2.00
2.35
2.50
Original Model Loss (best achievable) - 2,4760
50
100
150
200
250
300
Training Time (hours)
Figure I. Cross entropy loss vs. training time over 2B tokens for Gemma-2-2B TopK SAEs with width = 18, 432, Lo = 64. We find that our method (TopK + LoRA in the plot) outperforms an e2e SAE and vanilla TopK SAE.
2023; Bubeck et al., 2023; Anthropic, 2024), the mecha- nisms behind these behaviors remain largely inscrutable. Mechanistic interpretubility (MI) (Bereska & Gavves, 2024) seeks to understand these mechanisms by reverse engineer- ing them into human understandable algorithms. In this work, we focus on better understanding features, the "vari- ables" of model computation (Olah al., 2020; Mueller et al., 2024; Marks et al., 2024).
A popular hypothesis in MI is the Linear Representation Hypothesis (LRH) (Elhage et al., 2022a; Park et al., 2023), which posits that features are one-dimensional directions in activation space. Although some recent research has called aspects of this hypothesis into question (Engels et al., 2024a; Csordás al., 2024; Engels et al., 2024b), the LRH has been empirically validated for many language model features "in the wild" (Nanda et al., 2023; Heinzerling & Inui, 2024). Inspired by these successes, sparse autoen- coders (SAEs) (Makhzani & Frey, 2013) have recently been applied to decompose language model hidden states into many linear features (Cunningham et al., 2023; Bricken et al., 2023). The latents SAEs learn are significantly more interpretable and monosemantic than the original neuron basis (Cunningham et al., 2023; Bricken et al., 2023).
1. Introduction
Although language models demonstrate profound capabili- ties in tasks such as in-context learning, mathematics, and coding (Brown et al., 2020; OpenAl, 2024; Team et al.,
Equal contribution Massachussetts Institute Technol- ogy, Cambridge, MA. Correspondence to: Matthew Chen <mattchen @ cdu..
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
While SAES find interpretable latents, this comes at a cost: when SAE reconstructions are inserted back into the model and the forward pass performed, the resulting cross en-
CCode available httph: //gp::///tiie /ggtttb....////////////g com cttpiii..commmmtttccccecc honmmatchtee////eeeeeecc MoRA-Moded 1.s for- SAES

---

Low-Rank Adapting Models for SAES
tropy loss (LSAE) is significantly higher than the loss of the original model (CBAsE). For example, when reconstructions from TopK SAE are inserted into GPT-4, the resulting SSA is equivalent to the LBSS of model trained with just 10% of the pretraining compute of GPT-4 (Gao et al., 2024). Thus, previous work has extensively focused on optimiz- ing SAE architectures to find Pareto improvements in the SAE sparsity vs. LSAE frontier. This work includes TopK SAES (Gao al., 2024), Gated SAEs (Rajamanoharan et al., 2024a), JumpReLU SAEs (Rajamanoharan et al., 2024b), ProL. SAEs (Taggart, 2024), Switch SAFs (Mudide et al., 2024), and e2e SAEs (Braun et al., 2024).
benefit of full-parameter LoRA by training an adapter only on the layer after the SAE and that our adapters achieve most improvement on tokens with low LSAE.
2. Related Work
SAE Architecture Improvements. Early SABs for lan- guage models used simple linear encoder, ReLU activa- tion with an L penalty (approximating Lo), and linear decoder (Bricken et al., 2023; Cunningham et al., 2023), The next generation introduced Topk and BatchTopk SAES, which enforce sparsity by retaining only the largest ac- tivations (Gao et al., 2024; Bussmann et al.. 2024), and GatedSAEs and JumpReluSAEs, which use gating func- tions and straight-through estimators to approximate direct Lo optimization (Rajamanoharan et al., 2024a:b). These methods improve the LSAE vs. sparsity tradeoff, though no single approach is definitively superior on downstream tasks (Karvonen et al., 2024). Beyond sparsity penalties, Braun et al. (2024) optimize SAEs for KI. divergence with the model's logits directly improve LSAE, while Olmo et al. (2024) incorporate model gradients into TopK activations for more causal representations. However, gradient-bant based methods introduce computational overhead and have large limitation: SAEs are typically trained on cached activations without available gradients. (Lieberum et al., 2024).
However, an unexplored question is whether language mod- els themselves can be optimized after SAE training to gain an additional Pareto improvement in sparsity vs. LSAE. In this work, answer this question in the affirmative: Low-Rank Adapters (LoRA) (Hu et al., 2021) to reduce the KL divergence between the original model's 's logits and the model's logits with an SAE inserted. The resulting model- SAE combination improves in CSAE and on a diverse set of downstream SAE metrics. Compared to existing proposals for training more interpretable models with SAEs (Lai & Heimersheim, 2024; Lai & Huang, 2024), we estimate our LoRA method is 107 times faster? 2 Overall, we find that low-rank adapting models is a simple and efficient technique to improve the interpretability vs. performance trade-off.
use
Our contributions include the following:
While this paper focuses on fine-tuning model around an SAE, another research direction explores fine-tuning SAES. Some work tailors SAEs to specific domains by oversampling certain contexts (Bricken et al., 2024) or fine-tuning on domain-specifh activations (Drori, 2024). Kissane et al. (2024) find that training SAEs on chat data captures the refusal latent, whereas training on the Pile (Gao et al., 2020) does not. Kutsyk et al. (2024) further analyze when base model SAEs generalize chat-tuned model, showing that it depends on the language model used.
Fine-tuning SAEs.
1. To the best of our knowledge, we are the first to focus on improving the model around an existing SAE.
2. In Section 4.1, we analyze LoRA SAE training on the Gemma Scope (Lieberum et al., 2024) family of SABs. Across SAE width, SAE sparsity, language model size, LoRA rank, and language model layer, we find 30% to 55% improvement in LSAE-with final values between 0.01 and 0.17 nats-with especially large improvements in low sparsity regimes and larger models.
3. In Section 4.2, we compare our method to e2e SAES on training time vs. LSAE on Gemma-2-22 (Team al., 2024) and Llama-3.2-1B (AI@ Meta, 2024). We find that our method achieves the same LSAE e2e SAES with between 2x and 20x less compute and 130x fewer language model backward passes.
Training interpretable models. We are aware of two prior works that investigate training more interpretable models using SAEs: both (Lai & Heimersheim, 2024) and (Lai & Huang, 2024) train SAES and models concurrently, and find that this improves LSAR. However, because this requires training models from scratch, it is impractical to apply to existing models and is only shown to work in toy settings; in contrast, our method is extremely efficient, and we show it works on models up to 27B parameters. Many prior works also investigate this direction without SAEs. Elhage et al. (2022b) introduce the softmax linear unit (SoLU) activation function, which increases the fraction of interpretable neu- rons at no cost on downstream performance; Liu et al. (2023) propose a new loss term penalizing spatially distant con- nections in the network that leads to visually interpretable networks; Liu et al. (2024) introduce Kolmogorov
4. In Section 4.3, we perform LoRA SAE training with mul- tiple SAES inserted into Llama-3. 1-BB (AI@ Meta, 2024) and see large decreases in LSAE. demonstrating the poten- tial of our technique for helping circuit analysis.
5. In Section 5, we show guantitative improvements on diverse set of downstream tasks.
6. In Section 6, we find that we can achieve much of the
"Gemma-2-28 was trained with 6T tokens, whereas we use 15M tokens on up to 3% of the parameters; 6T/15M /0.03
1.3 x 10"
2

---

Low-Rank Adapting Models for SAES
During training, the SAE minimizes the reconstruction er- ror L | X 1 We train TopK SAEs with k: 64 for Gemma-2-2B and Llama-3.2-IB for 2B and 4B tokens, respectively, on the RedPajama dataset (Weber et al., 2024).
uoquist
MP
Encoder
Lequnbo ahl
3.2. End-to-End Sparse Autoencoders
SAE
Low Rank Adapters
Low Rank Adapters
In c2c SAE (Braun et al., 2024), the SAE minimizes KL divergence with the base model instead of reconstruction error. Formally, if we have
Figure 2. Visual representation of method, with local SAE trained on layer 12 and low-rank adapters trained on MLP and attention components all layers.
x SAE(xi):
& = (8 Iiis
y = softmax
Networks, alternative to standard MLPs with trainable activation functions that can be replaced by symbolic formu- las; and Heimersheim (2024) fine-tune out the LayerNorm components in GPT-2 with a small downstream loss effect.
then the e2e SAE minimizes L = y). For both e2e and TopK SAES, we use a TopK activation function with the same sparsity to allow for fair comparisons.
Parameter Efficient Fine Tuning. Parameter efficient fine luning (PEFT) reduces the cost of full supervised fine tuning by updating fewer effective parameters. One of the simplest and most effective PEFT methods is low-rank adaptation (LoRA) (Hu et al., 2021). LoRA works as follows: for a frozen pretrained weight matrix Wo E Rdxk and low-rank matrices A E _d B E x with r < min(d, k), the original forward pass h(x) = Waz becomes
3.3. JumpReLU Sparse Autoencoders
We also evaluate our method on the Gemma Scope JumpReLU SAEs. Instead of the TopK function, JumpReLU SAEs (Rajamanoharan et al., 2024b) use the JumpReLU activation function,
JumpReLU,,s)) = 2H(2 -),, 8),
h(x) Woz + ABx.
(1)
where H is the Heaviside step function and 0 > 0 is the JumpReLU's threshold. The SAE is trained to minimize
A and B can then be trained while the rest of the model is frozen, resulting in low-rank update of the base model.
L
| x12 + |
(6)
3. Optimizing Models for Sparse Autoencoders
where is defined from Equation (4).
In this section, we formally describe existing methods for training SAEs and our method of adapting models for SAEs. For decoder only transformer with L layers and hidden dimension d, input Xo, and output y, denote the activation after the ith layer by x;. Express the ith transformer block as a function h, such that the network can be expressed as
3.4. Method for Low-Rank Adapting Models to SAFs Formulation.
We formally describe our method of opti- mizing models for SAEs, using notation from Equations (2). We insert a frozen SAE immediately after layer e, and the reconstructed activation & SAE(xe) propagates through the remaining layers to produce x; h, (x;-1) for e- < < L and softmax(x1).
x; 1x(xx11)
1Si5L
(2)
y softmax(x1)
(3)
For JumpReLU SAEs we can only adapt layers after the SAE to ensure average sparsity is unaffected, while for TopK SAES we can train adapters on all layers by maintaining the TopK constraint during training. We add low-rank adapters of rank in each MLP and attention sublayer of every layer adapting. Concretely, for each frozen weight matrix W, E R xd2, we add A; E Rdi and B, E Rrxds and modify the forward pass according to Equation (1).
3.1. TopK Sparse Autoencoders
SAES learn an encoder W one E mx for m > d. de- coder Waee E dd with unit norm columns, and biases benc E T bae Rdd We call the m columns of W dec latents. For activation X1, the TopK SAE (Gao et al., 2024) reconstructs activation & as follows:
z = TopK(Wo Wene(xe
(xi exc
baee) + bene)
(4) We train only the low-rank adapters θ = {A,} U (B.). For all experiments the training objective is the KL divergence (5) between the next token probability distribution with and
*; = Wdec + bilec = Σ wif,
3

---

Low-Rank Adapting Models for SAEs
CE Improvement
0.15
Rank
Rank
Rank
Rank
Rank
Rank
0.10
Rank 64
Rank 64
Rank 64
Rank 256
Rank 256
Rank 256
0.05
CE Improvement (%)
55
50
45
40
35
30
100
200
300
400
100
200
300
400
500
Sparsity (LO)
Width (k)
Layer
Figure Cross entropy loss improvement (Top: absolute, Bottom: percentage of CE loss gap closed) using our method Gemma Scope SAEs on Gemma-2-2B. Left: Scaling across sparsity with fixed width=16k and layer=12, see the largest effect by percentage our method lower sparsities, but substantial effect higher sparsities as well. Middle: Scaling across width with fixed Lo 68 and 1ayer=12, the highest effect by pereentage at low width but again this is not large effect. Right: Scaling across layer with fixed To = 68 16k the highest effect our method by percentage is at layer 9 hut is mostly unaffected by layer.
without the SAE inserted:
et al., 2020), and evaluate on held out validation set of 1M random tokens. We report our findings in Figure 4 for model size and in Figure for other scaling axes.
arg min y)
We find that across all of the scaling regimes test, we close the LSAE LUASE gap by at least 30%, and sometimes by up to 55%. We find that using larger rank LoRA adapters reliably decreases the final LSAE this, combined with the fact that we adapt on only 15M tokens and do not see our adapters finish converging, implies that with more compute our method may be even more successful.
By freezing both the SAE and the original model, this parameter-fftcient method aligns the SAF-enhanced model to the behavior of the original model with minimal addi- tional cost.
4. Results
In this section, study how our method improves the cross entropy loss gap (Csae Lbses) before and after LoRA on wide variety of SAEs and language models. Unless otherwise specified, we use a layer residual stream SAE.
We find that over varying layers, the improvement is largest for middle layers, although this result is not extremely strong (according to e.g. (Lad et al., 2024), this may arise from the fact that middle layers tend to have richer and more expressive representations that local SAES may struggle reconstructing). We also find that the improvement is largest on lower sparsities, lower widths, and larger models; all of these results may be caused by these SAEs having higher cross entropy loss gap start with. We do still find it extremely promising that the effectiveness of our technique increases on larger models.
4.1. Scaling Laws for Downstream Loss
We first explore the scaling behavior of low-rank adapt- ing models to SAEs across SAE sparsity, SAE width, lan- guage model size, LoRA rank, and model layer. Specifically, use Gemma Scope's Jump SAEs (Rajamanoha- et al., 2024b). To ensure do not affect the average sparsity of these JumpReLU SAES, we only finetune the layers after the SAE. Over difffeent sparsities, widths, and layers, we track the absolute and percent improvement in LSAE - CAASE after low-rank fine-tuning. We train on 15M random tokens of The Pile (uncopyrighted) dataset (Gao
4.2. Downstream Loss vs. Computational Cost
Next, study the frontier of LSAE versus training time for TopK SAEs, e2e SAEs, and low-rank adapted TopK SAEs. To do this, we need to train our own TopK and e2e SAEs to

---

Low-Rank Adapting Models for SAEs
0.06
CE Improvement
Table I. Gemma-2-20 timing results and speedups, nearest hour
0.05
CE Loss
TopK + LoRA
TopK
e2c
Speedup
0.04
2.60
12h
59h
37h
3.05x
0.03
2.59
12h
79h
6.53x
2.58
12h
148h
12.18x
0.02
2.57
12h
243h
20.01x
CE Improvement (%)
50
2.55-2.57
12h-107h
45
Table 2. Llama-3.2-IB timing results and speedups, nearest hour
Rank 1
Rank 4
CE Loss
TopK + LoRA
TopK
c2c
Speedup
30
25
Rank 256
2.73
9h
96h
10.38x
27B
2.72
12h
113h
9.08x
Gemma Size
2.71
19h
135h
7.14x
2.70
70h
156h
2.23x
Figure Cross entropy loss improvement (Top: absolute, Bottom: percentage) for Gemma Scope SAEs of width 16k and Lo closest 70 on Gemma-2-2BB 7B, and 27B. find that our method works incrcasingly well on larger models.
2.67-2.70
156h-213h
gram model (Gao et al., 2024)). At the same time, inserting multiple SAES at once is very usefuu task for circuit analy- sis, since it allows one to determine dependencies between SAE latents (in practice, past SAE circuits work (Marks et al., 2024) has used error terms overcome this limita- tion, which results in less interpretable circuits). Thus, we adapt procedure work with multiple SAEs: insert all SAFs at once during training, and otherwise follow Sec- tion 3.4. We measure the performance of our technique with the "Compound Cross Entropy Loss" (Lai & Heimersheim, 2024), which is simply LSAE with all SAEs inserted.
get their training curves. We also use checkpoints from the TopK training run to get the training curve for TopK SAEs + LoRA; after every 10% training checkpoint of the TopK SAEs, we low-rank adapt the model checkpoint with rank 64 LoRA on all layers.
We train layer 12 of Llama-3.2-1B and Gemma-2-2B. We train TopK and e2e SAEs for 4B tokens Llama-3.2-1B and for 2B tokens on Gemma-2-2B (similar to the number of tokens trained on for Gemma Scope SAEs). On each TopK SAE training checkpoint of Llama-3.2-18 we LoRA finetuning for 100M tokens, while we finetune for 15M tokens on Gemma-2-22 2-2B TopK SAE checkpoints.
We use the Llama Scope (He al., 2024) set of SAEs (width 131, 072, Lo 50) trained Llama-3.1-8B. Because these are TopK SAES, can train the LoRA layers without worrying about violating sparsity constraints. We train with the following configurations of SAEs, chosen to maximize the distance between adjacent pairs of SAEs: I SAE at layers (16): 3 SAEs at layers ( (10, 20, 30]: SAEs at layers f6. 12, 18, 24, 30); SAES at layers {4, 12, .... 28): 10 SAFs layers (3, 6, .... 30}: and 15 SAES layers {2, 4, ... 30}.
We show the Pareto cross entropy frontiers for Gemma-2-2B and Llama-3.2-1 3.2-1 in Figures 1 and 5, respectively, where our method clearly dominates. Quantitatively, we show the speedup in wall clock time in achieving various cross entropy loss threshold when using TopK LoRA versus e2e in Tables and 2; our speedups ranging from 2x x0x, Our approach (TopK LoRA) also performs 130x fewer backward passes through the model than e2e SAEs Gemma-2-2B and 40x fewer backward passes on Llama-3.2-1B. Finally, we do note, however, that e2e SAEs achieve a lower final CE loss than our method on Llama-3.2-1B (although not on Gemma-2-2B).
Our results (see Figure 6) show that this method significantly reduces compound CE loss; for example, using LoRA, the compound CE loss for SAEs goes from 7.83 nats to 2.78 nats, while the compound CE loss for 3 SAES goes from 3.55 nats 2.45 nats (which lower than the original vali- dation CE loss with single SAE and no LoRA). Thus, technique seems extremely promising for circuit analysis.
4.3. Adapting Multiple SAES
Inserting multiple SAEs at once into a language model causes LSAE to grow extremely rapidly (e.g. as shown in Figure 6, we find that inserting SAEs leads to a cross entropy error of almost 10 nats, which is worse than a uni-
5. Downstream Improvements
A common criticism of previous SAE optimizations the lack of grounded metrics for evaluating how good an SAE
5

---

Low-Rank Adapting Models for SAEs
3.0
SAE TopK + LoRA
12
No SAES
11.00
2.9
Compound CE Val Loss
10
Topk
9.89
Topk LoRA
CE Loss
2.8
5.78
3.36
2.45
2.94
2.78
Modell
25
50
75
100
125
150
175
200
1
3
5
10
15
Number of SAEs
Training Time (hours)
Figure 6. Downstream cross entropy loss when multiple Llama Scope SAEs inserted into Llama-3.1-8I at once. "Base" the original loss without any fine-tuning, while "LoRA" is the loss after 15M tokens LoRA training.
Figure 5. Cross entropy loss vs. training time for Llama-3.2 3.2na---.2211 BB with TopK SAEs Lg 64 and width 16384. Our method (TopK + LoRA) achieves lower CF loss sooner than e2e SAE or vanilla TopK SAEs
Table 3. Using the same TopK SAE trained on Gemma-2-2B, we compare the SAEBench metries when the underlying model is low-rank adapted with rank 64, We see across most applicable metrics, the LoRA model shows meaningful improvement. Full results over various thresholds are displayed in Table 6.
is. Prior work has largely relied unsupervised metrics such as in Section 4. Recent work, however, has introduced evaluation metrics to measure model and SAE according to their performance on downstream tasks (Karvonen et al., 2024; Pres et al., 2024). Thus, in this section, we evaluate our method on diverse set of downstream benchmarks:
DOWNSTREAM METRIC
TOPK + LORA
TOPK
SCR (MAX)
0.526
0.448
1. In Section 5.1, we show that using LoRA on all layers improves downstream tasks on SAEBench.
SCR (AVERAGE)
0.314
0.289
TPP (MAX)
0.412
0.372
2. In Section 5.2, introduce novel steering metric and show that method improves it. We introduce metric because SAEBench metrics do not test the effects of SAEs on next token prediction.
TPP (AVERAGE)
0.145
0.111
SPARSE PROBING (TOP 1)
0.760
0.732
SPARSE PROBING (TEST)
0.956
0.955
AUTOINTERP
0.830
0.832
ABSORPTION
0,210
0.205
3. In Section 5.3, we show that our method improves overall model performance with the SAE inserted on MMLU, HellaSwag, and TruthfulQA.
of k-sparse probe trained on SAE latents, with higher scores indicating better feature learning. AUTOINTERP, sessed using LLM judge (GPT-4o-mini (OpenAI, 2024)), quantifies the interpretability of SAE latents. ABSORPTION quantifies to what extent latents are "absorbed" together to improve sparsity. All metrics range from 0 to 1, with higher being better except for ABSORPTION. 3
5.1. SAEBench
To address the core challenge of measuring how effectively a model and SAE work together, Karvonen al. (2024) intro- duce SAEBench, benchmark of SAE metrics that are faith- ful to possible real world cases. For the Gemma-2-2B TopK SAE (Lo 64) we trained in Section 4.2, we evalu- ate the model with the SAE and the model with the SAE LoRA on SAEBench.
We display our results in Table 3, showing low-rank adapted model outperforms the base model TPP, SCR, and SPARSE PROBING, while very slightly underperforming on AUTOINTERP and ABSORPTION.
Specifically, we look at spurious correlation removal (SCR), targeted probe perturbation (TPP), SPARSE PROBING, au- tomated interpretability (AUTOINTERP), and feature absorp- tion (ABSORPTION). SCR measures the separation of la- tents for different concepts, with higher scores indicating better ability to debias a classifier. TPP evaluates the im- pact of ablating specific latents on probe accuracy, where higher scores reflect well-isolated latents corresponding to classes on dataset. SPARSE PROBING tests the accuracy
5.2. Feature Steering
In this section, we demonstrate that the LoRA tuned model improves at activation geering-repressinn or eliciting model
Excluded from results are the RAVEL and UNLEAA UNLEARNING SAEBench metrics. RAVEL not yet implemented in SAEBench and UNLEARNIN recommended for instruct models only.
6

---

Low-Rank Adapting Models for SAES
Positive Examples
Negative Examples
Table 4. For each SAE latent, APOSITIVE and ANecATIVE denote the 90% CI improvement in normalized log likelihood increase when using LoRA model for steering on positive and negative examples, respectively. Because AroSITIVE > 0 and ANegATIVE S O, we see the LoRA model is better at steering for a given SAE latent while not affecting other features.
Base Model
35
40
LoRA Model
30-
30-
25-
Count
20
15-
10-
10-
SAE FEATURE
A POSITIVE
A NEGATIVE
5-
MACHINE LEARNING
0.86 0.82
-0.84 + 0.43
20
30
20
SAN FRANCISCO
0.97 + 0.76
0.06 + 0.20
Change in Normalized LL after Steering
DONALD TRUMP
2.50 + 0.56
0.20 + 0.40
COVID-19
0.44 + 0.25
-0.01 + 0.06
Figure 7. The distribution of normalized log-likelihood change post-steering is more pronounced on positive examples for the LoRA model. Results shown for the SAE latent responsible for "Donald Trump".
To compare models, let APOSITIVE and ANEGATIVE represent the change in normalized likelihoods for positive and neg- ative datasets when switching from the base to the LoRA model. The LoRA model is better suited for steering APOSITIVE and ANEGATIVE I O. We compute 90% con- fidence intervals for APOSITIVE and ANEGATIVE across 500 examples for each of our four SAE latents. Results are sum- marized in Table 4. We show a histogram of changes across all examples after steering for the best performing latent, "Donald Trump", in Figure 7.
behavior by scaling a steering vector- using its SAE latents. Given SAE latent Rd layer we steer via
X + x + axx v)v, a E.R.
(8)
We assess steering effectiveness following (Pres et al., 2024), who evaluate steering by analyzing increases in likelihood for "positive" texts (aligned with the desired behavior) and decreases for "negative" texts (not aligned). We note that Olmo et al. (2024) also introduce an SAE steering evalua- tion, but it does not follow the best practices for steering laid out by (Pres et al., 2024) so we do not use it.
5.3. General Language Model Capabilities
In addition to downstream tasks, we evaluate the model's general language capabilities on MMLU (Hendrycks et al., 2020), HellaSwag (Zellers et al., 2019), and TruthfulQA (Lin et al., 2021) two different regimes: comparing evals when the SAE is inserted and when the SAE is not inserted. In other words, the four settings are: 1) SALE, the original model with the SAE, and 2) SAE + LoRA, a low rank adapted model with the SAE, 3) Original, the original model with no SAE, 4) LoRA, the adapted model with no SAE. Our results, shown Table 5, show that across all Gemma model sizes and across benchmarks, our adapted regime is not hurt capability wise and even frequently out- performs the original model. In other words, adapting the model to be more faithful to its SAE latents does not harm general language model capability.
Our method is slightly different than (Pres et al., 2024) because we are comparing different models with the same stecring method instead of comparing different steering methods on the same model. For a given SAE latent, we steer on a dataset of 500 positive and negative samples. The negative dataset consists of an equal mix of arabic tweets (Pain, 2024), medical facts (MedAlpaca, 2024), recipes (Corbt, 2024), shakespearean quotes (Roudranil, 2024), and law texts (GPT-4o-mini generated). We generate the positive datasets by selecting latent about 1) machine learning, 2) San Fransisco, 3) Donald Trump, and 4) Covid-19. We then generate text samples where that latent fires using GPT-4o- mini. See Appendix 1.1.2 for full prompt details.
6. Analyzing Why Our Method Works
6.1. Per Token Improvement Breakdown
Following (Pres et al., 2024), we compute mean token log- likelihoods before and after steering, normalizing them so the original likelihoods span 0 to 100. We tune the hy hyperpa- rameter a in Equation (8) by selecting value that increases the likelihood of positive samples while minimizing likeli- hood increases on a validation subset of negative samples (medical facts). After tuning, we evaluate the effect of a on a test consisting of the remaining negative samples. This tuning process is repeated for the base and LoRA models.
In this experiment, we analyze how LoRA impacts CSAE improvements. Figure 8 shows the distribution of AL between the original and LoRA models across 15M val- idation tokens. The per-token loss change varies greatly, with the loss on 37% of tokens even getting worse (see the degradation histogram in the figure). Most of the overall improvement comes from small per-token decreases in loss (roughly 10-2 to nats), suggesting LoRA improves loss
7

---

Low-Rank Adapting Models for SAES
Table Comparisons of original model performance performance with the SAE inserted, the SAE inserted with LoRA, and just LoR A. Error ranges represent one standard error, largest value between non-adapted and adapted versions are bolded. Note that even without the SAE LoRA model is frequently better; thus, LoRA adapter we train for the SAE does harm general model performance.
GEMMA-2-28
METRIC
SAE
SAE + I.ORA
ORIGINAL
LoRA
MMLU
44.2
45.8 + 0.4
49.3 0.4
50.0 0.4
HELLASWAG
50.9 0.5
52.1 1 0.5
55.0 0.5
56.0 0.5
BLEU
2..9
30.6 1.6
30.4 + 1.6
32.4 = 1.6
ROUGE-I
1.6
28.5 1.6
1.6
26.9 + 1.6
30.2 - 1.6
ROUGE-2
21.8 + 1.5
26.6 1.5
25.6 1.5
29.1
1.6
MC1
23.1
23.4 13.41.1 1.5
24.1 1.5
24.3
BEMMA-2-9B
METRIC
SAE
SAE+ LORA
ORIGINAL
LORA
MMLU
64.2
65.7
7..0
68.8 0.4
HELLASWAG
58.3
59.6 0.5
0.5
61.9 0.5
BLEU
40.9 1.7
42.4 1.7
43.8 1.7
43.6 1.7
ROUGE-1
3.70
40.6 + 1.7 1.7
42.7 + 1.7
43.5 1 1.7 1.7
ROUGE-2
33.4
36.4 1.7
38.3 1.7
38.8 1.7
MC1
27.1 + 1.6
28.0
30.5 + 1.6
31.0 1.6
GEMMA-2-27B
METRIC
SAE
SAE LORA
ORIGINAL
LoRA
MMLU
70.9 0.4
71.3 0.4
72.1
72.7 + 0.3
HELLASWAG
61.0 0.5
62.7 0.5
6..3
65.5 + 0.5
BLEU
40.9 1.7
38.9 1.7
1.7
41.9
ROUGE-1
41.0 + 1.7
38.3 1.7 1.7
1.7
41.7 1 1.7
ROUGE-2
37. +: 1.7
35.3 1 1.7
36.2 1.7
37.1 + 1.7
MC1
3
1.6
33.8 1.7
32.9 + 1.6
across many tokens rather than more bimodal distribution.
6.3. Single Layer Adapters
Another question we are interested in is which LoRA layers are most important for reducing LSAE. In Figure 9, we plot the results of an experiment where we train LoRA adapters on each individual layer after the layer with the inserted SAE. We find that the LoRA performance degrades as also
6.2. Activation Distances
One concern identified by (Braun et al., 2024) is that op- timizing towards KL divergence may lead the activations to be off distribution and follow a different computational path through the model. We find this not the case with our method: as shown in Figure 11, over validation set of 500k tokens, our method slightly decreases the distance between activations after the SAE and activations in the orig- inal model, while the cosine similarities slightly increase. In other words, the adapted model SAE follows closer computation path to the original modle than the original model + SAE.
gets farther from the original layer. Interestingly, find that training LoRA adapters on just the first layer after the SAE achieves 88. 14% of the loss reduction in training LoRA adapters on all the layers after the SAE, suggesting the loss improvement mechanism may be reasonably simple.
7. Conclusion
Low-rank adapting models for SAEs provides a fast, cheap, and effective path to producing interpretable combined model and SAE systems. Moreover, low-rank adapted mod-
8

---

Low-Rank Adapting Models for SAES
SAEs may not have finished converging in our experiments. we cannot compare converged accuracy; however, we had already trained for more than week before stopping, so training to full convergence may not be practical. We use the learning rates suggested in Gao et al. (2024), but is possible that further tuning of the learning rate could make the e2e SAE train faster, for reasons of compute limitations, we do not experiment with learning rate. Finally, we note that if we had used LoRA for more tokens we may have got- ten an additional improvement the Gemma Scope scaling experiments; however, our results are still quite strong, and the fact that they were achieved with just 15M tokens shows the efficiency of our technique.
16000
Loss Improvement
# of Validation Tokens
Loss Degradation
14000
12000
10000
8000
6000
4000
2000
10-3
10-4
10-1
10-2
10
10a
10a
Base Loss LoRA Loss]
Figure& 8. Distribution of loss improvements and loss degredations across validation tokens. We see that more tokens have a loss improvement than degredation (although substantial number have a degradation) and most loss improvements and degredations happen in range of about 0.01 to A CSA nats.
Impact Statement
We believe that increasingly powerful language models and other Al systems pose many safety risks (e.g. deception, power seeking, misinformation, bias, CBRN risks; see (Slat- tery et al., 2024) for a complete summary). MI and other fields that try to better understand LLMs are motivated by reducing these risks (see (Bereska & Gavves, 2024) and (Sharkey et al., 2025) for more in depth reviews of MI and AI safety and discussions of open problems). Thus, be- cause the goal in our work is to train more interpretable systems at a fixed level of fidelity to the original (uninter- pretable) model, we do not foresee negative consequences of our work; on the contrary, we believe our work is broadly beneficial for Al safety.
CE Improvement
All layers after
40
CE Improvementt 9%
30
0.02
0.00
8
10
12
Number of Layers after SAE
Figure 9. Plot of LSEE when running LoRA on just a single layer of Gemma Scope 2B. We find that LoRA layers closer the SAE layer do better, and that also LoRA on just the next layer achieves much of the loss reduction of training on all layers.
References
AI@ Meta. Llama 3 model card, 2024. URL https:
els are "better" at using their SAES for downstream tasks. Crucially, our work challenges the prevailing assumption that improving interpretability must rely solely on post-hoc model decomposition. We hypothesize that our method is much faster than e2e training because modifying the entire model gives many more degrees of freedom for the opti- mization procedure to work with; thus, our work suggests that focusing more on the larger space of possible language model modifications may be fruitful. We hope the results in this paper lay the groundwork for further such techniques.
waaaaaaammmaaaaaa///aaaaaa a i n Btt///ODD__AAARE .d.
/ //github. wtwtttu..ooooooo www/ootooaooeooooooooooooooeaaaa
Anthropic. The claude 3 model family: Opus, sonnet, haiku. Technical report, Anthropic, 2024.
Bereska, I.. and Gavves, E. Mechanistic interpretability for ai safety-a a review. arXiv preprint arXiv:2404. 14082, 2024.
Braun, D.,, Taylor, J., Goldowsky-Dill, N., and Sharkey, L. Identifying functionally important features with end- to-end sparse dictionary learning, 2024. URL https: //arxiv. org abs /2405 12241.
7.1. Limitations
Mechanistic interpretability work usually interprets a frozen model, while in our work we change it; for some applica- tions, this might not be acceptable. However, we do note that we use LoRA to decrease the KL divergence with the original model, so if one is using SAFS anyways, our method creates a more faithful model. Additionally, we do not yet show that our technique helps discover more faithful circuits for model behaviors; we leave this important direction for future work. Another limitation is that it seems the e2e
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield- -oodds. Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learning.

---

Low-Rank Adapting Models for SAES
Trunsformer Circuits Thread, 2023. https://transformer-
Elhage, N., Nanda, N., Kerneld, Z., Henighan, T., Olsson, C., and Joseph, N. Softmax linear units, 2022b. URL wwwecciiss.uub//eeeeee pubb2022
htmll.
Bricken, T., Marcus, J., Rivoire, K., and Henighan, T. Trans- former circuits: September update, 2024. URL https:
Isolu/index. html.
Engels, J., Michaud, J., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features lin- ear, 2024a. URL https: //axxviivv..//////v/ .14860.
mubr-upddtte/nnee Accessed: 2025-01-19.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler. D. M., Wu,
Engels, J., Riggs, L., and Tegmark, M. Decomposing the dark matter of sparse autoencoders. arXiv preprint arXiv:2410.146700 2024b.
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https: xiv. org/ / /ass////////aaa 2005 14165.
Gao, L.,, Biderman, S., Black, S., Golding, L.,, Hoppe, T. Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002. 2020.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., al. Sparks of artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.127122 2023.
Gao, I.., Tour, T. D., Tillman, H., Goh, G., Troli, R., Radford, A., Sutskever, Leike, J., and Wu, Scaling and evaluating sparse autoencoders, 2024. URL https: /
Bussmann, B., Leask, P., and Nanda, N. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410 2024.
He, Z., Shu, W., Ge, X., Chen, L.., Wang. J., Zhou, Y., Liu, F. Guo, Q. Huang, X., Wu, Z., Jiang, Y.-G,, and Qiu, X. Llama scope: Extracting millions of features from 1ama-331-88 with sparse autoencoders, 2024.
Chanin, D., Wilken-Smith, J. Dulka, T., Bhatnagar, H., and Bloom, is for absorption: Studying feature splitting and absorption in sparse autoencoders, 2024.
Heimersheim, S. You can remove gpt2's layernorm by fine- tuning. 2024. URL httLttts///axii org/ abs/24 09.13710
URL https::: http::/::/aa.....og//bb/////////
Corbt. All recipes dataset. https: https://huggingfae co dadddaasetttt ceatasets/corbt.ee http://ooobbaaalt/eieeee 2024. Accessed: 2025-01-30.
Heinzerling, B. and Inui, K. Monotonic representation of numeric properties in language models. arXiv preprint arXiv:2403. 10381, 2024.
Csordás, R., Potts, C., Manning, C. D., and Geiger, A. Re- current neural networks learn store and generate se- quences using lon-linear representations. arXiv preprint arXiv:2408. 10920, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03330 2020.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models, 2023. URL https: //ar xiv. org/ abs /2009. 00600/0099.000........ 08600.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation large language models, 2021. URL https: arxx v. v.org/abe abs///////// / 2200..166011 09685.
Drori, J. Domain-speciffc saes, 2024. URL wwtpthttp////. http:///ww.cccoooooo//////rr///e Agrr/domin-speciiicce
Karvonen, A., Rager, C., Lin, J., Tigges, C., Bloom, J., Chanin, D., Lau, Y.TT., Farrell, E., Conmy, A., Mc- Dougall, C., Ayonrinde, K., Wearden, M., Marks, S., and Nanda, N. Saebench: comprehensive bench- mark for sparse autoencoders, 2024. URL https: //www. www.nneedoonaaaaa.om org/ saee//aa-aae bench/info.
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan,
T., Kravec, S., Hatfield-Dodds Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of superposition. Transformer Circuits Thread, 2022a. URL www.rraaanoooeeirrrerreee cit.arrcccccuutt...u//2222 pub/ /toy_model// lindeex//////////ee/eee index. html.
Kissane, C., Krzyzanowski, R., Nanda, N., and Conmy, A. Saes are highly dataset dependent: A case study on the dataset, 2024. URL https wwww a ali.nment..nemm..m..mmmmmmmmmm for
10

---

Low-Rank Adapting Models for SAES
um mm.orr//oot//ttpo///222///////// ge-1iipppl--datttst--deeeeeeeeee tudy-on-the,
Mudide, A., Engels, J., Michaud, E. Tegmark, M., and de Witt, C. S. Efficient dictionary learning with switch sparse autoencoders, 2024. URL https: : aarxiv.. org/ abs www.aaaaa.. /2100.41000 08201.
Kutsyk, T., Mencattini, T., and Florea, C. Do sparse autoen- coders (saes) transfer across base and target?, 2024. URL ptggppotss
Mueller, A., Brinkmann, J., Li, M., Marks, S., Pal, K., Prakash, N., Rager, C., Sankaranarayanan, A., Sharma, A. S., Sun, J., et al. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv preprini arXiv:240S 01400, 2024.
wttp:://wwwwll......f..........e SnnXPTTAAtwwtSSmBBB33333 sparse autoens across---ossss basssss--aeeaadddd
Lad, V., Gurnee, W., and Tegmark, M. The remarkable robustness of llms: Stages of inference? arXiv preprint arXiv:2406. 19384, 2024.
Nanda, N., Lee, A., and Wattenberg, M. Emergent linear rep- resentations in world models of sulf-supervised sequence models. arXiv preprint wwXiv:2309.009990 2023.
Lai, P. and Heimersheim, S. Sae regularization produces more interpretable models, 2024, URL httpp://wwww com seesstoong..coo/ppsts///tttooooo
Olah, C., Cammarata, N., Schubert, I.., Goh, G., Petrov, M., and Carter, Zoom in: An introduction to cir- cuits. Distill, 2020. doi: 10.23915/distte 1tt.sstilll.002.000...
8/sae- o/ deerpretable-modes
Olmo, J., Wilson, J., Forsey, M., Hepner, B., Howe, T. V., and Wingate, Features that make a difference: Lever- aging gradients for improved dictionary learning, 2024. URL httph:tp/////tte /arpiv / hatp://bbs/241111 10397.
Lai, and Huang, W. Gpt-2 circuits, 2024. URL https:
/ w//ptterla..githeee ho/pptmrr/ Accessed: 2025-01-30.
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma scope: Open sparse autoen- coders where all at once on gemma 2. arXiv preprint 2.Xiv:2408.05147, 2024.
OpenAI. Gpt-4o mini: advancing cost-efffiintt intllle July 2024. URL https: AttLasss//ppeaaaacco///iia
/ppp--00mtbilaaaaaacll-------f-- - -intelligence/.
OpenAl. Learning to reason with language models, 2024. URL httpt:tt/ com/ index/learnin
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0790 2021.
g-to-reasoonwwithhiiins
Pain. Arabic tweets dataset. httts://huggiingee Tweets, 2024.
Liu, Z. Gan, E., and Tegmark, M. Sceing is believing: iraiininspired modular training for mechanistic inter- pretability, 2023. URL https://aarae org/ abs/
e co co.co/da Cestasets/painAAraee Accessed: 2025-01-30.
Park, K., Choe, Y. J., and Veitch, V. The linear represen- tation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03656 2023.
Liu, Z., Wang, Y.. Vaidya, S., Ruchle, F., Halverson, J., Soljačíć, M., Hou, T. Y., and Tegmark, M. Kan: Kolmogorov-armold networks, 2024. URL https: //arxiv.org org/ abs/ 2404. 19756.
Pres, I., Ruis, L., Lubana. E. S., and Krueger, D. Towards reliable evaluation of behavior steering interventions in 1lms, 2024. URL https: /tt///axiiv/v///////////////aavm org/ .17245.
Makhzani, A. and Frey, B. K-sparse autoencoders. arXiv preprint arXiv:1312.500 2013.
Rajamanoharan, S., Conmy, A., Smith, L.. Lieberum, T., Varma, V., Kramár, J., Shah, R., and Nanda, N. Improv- ing dictionary learning with gated sparse autoencoders, 2024a. URL httttt///arxxi //tssii//xvvv...// org/ abs 2404. 6014.
Marks, S., Rager, C., Michaud, E. J.. Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403. 2403::200..9444 /964,, 2024.
Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024b. URL https: //arxiii.oorr/// abs/2407.1440
MedAlpaca. Medical meadow medical flashcards dataset. https http://gggggggceeeeee/e//e// co/ poo/datase dadatstsssssss seeets/mmaaa meda ppaca/medical mediical_mmmdd...aaaaaaeaaaa meadow htt___/eaaddeeeeeeeaeeeaa medical flashca rds, 2024. Accessed: 2025-01-30.

---

Low-Rank Adapting Models for SAEs
Roudranil. Shakespearean and modern english conversa-
tional dataset. httpps://huuggingffcc.../
htte://bouuaaaillbbaeeeppeererrr treerppiid--cccrrrraattoooaaatte 2024. Accessed: 2025-01-30.
Sharkey, I.,, Chughtai, B., Batson, J., Lindsey, J., Wu,
J., Bushmaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., Biderman, S., Garriga-Alonso,
A., Conmy, A., Nanda, N., Rumbelow, J., Wattenberg, M., Schoots, N., Miller, J., Michaud, E. J., Casper, S., Tegmark, M., Saunders, W., Bau, D., Todd, E., Geiger, A., Geva, M., Hoogland. J., Murfet, D., and McGrath, T. Open problems in mechanistic interpretabilityy 2025.
URL httpp::/aaxii.oooeteett/ wwt.:/xwwwv...rr.xoeee http:/bbb//50000 16496.
Slattery, P., Saeri, A. K., Grundy, E. A., Graham, J., Noetel, M., Uuk, R., Dao, Pour, S., Casper, S., and Thompson, N. The ai risk repository: A comprehensive meta-review, database, and taxonomy of risks from artificial intelli- gence. arXiv preprint arXiv:2408. 22622, 2024.
Taggart, G. M. Prolu: nonlinearity for sparse autoen- coders. httt://BDoTTaaatt/koo////oodoood marrlty-for-pprrs--attteerreeee 2024.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M.. Hauth, A., al. Gemini: family of highly capable multimodal models.
arXiv preprint arXiv:2312. 1805, 2023.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.000 001 18, 2024.
Weber, M. Fu, D., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. Redpajama: an open dataset for training large language models, 2024. URL https: /arxivi.vvv.//////// wwwpxvvvorg//... org abs /24 12312
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: / 19050098300 2019.
12

---

Low-Rank Adapting Models for SAEs
A. Appendix
A.I. Steering
A.1.1. DISTRIBUTION PLOTS
In Figure 10, we plot histograms for the changes in normalized log- legliikelihodsss for each of the four datasets from Table 4.
Positive Examples
Negative Examples
Positive Examples
Negative Examples
Base Modal
Base Model
Model
40
50-
35-
35-
30-
30
40-
Count
Count
20
15-
15-
15-
10-
10-
10-
10-
5-
5-
0
10
20
30
-20
20
40
Change Normalized after Steering
Change in Normalized after Steering
Positive Examples
Negative Examples
Positive Examples
Negative Examples
Model
35-
Model LoRA Model
60
40
LoRA Model
60
50-
30
25
Count
20-
15
30-
20
10-
5-
10-
10
0
30
1b
20
5
10
15
Change in Normalized LL after Steering
Change in Normalized LL after Steering
Figure Distribution plots the change in normalized log-likelihood after steering for various SAE latents. Top left for machinc learning (neuron 8421). Top right is for San Francisco (neuron 2195). Bottom left is for Donald Trump (neuron 13677). Bottom right for COVID-19 (neuron 17811).
1.1. DATASETS
To generate our "positive" examples dataset, generate examples eliciting the SAE feature with GPT-4o-mini. We prompt using the following chat template.
prompt =
Generate (num moum_exampless text examples that have the following feature: [feature_descriptionn
Below are examples of text that have the feature described above.
Examples:
(examples)
Each text example should be around **twelve** words long and be unique. Try to be varied in the content of the examples.
13

---

Low-Rank Adapting Models for SAEs
A.2. SAEBench Metrics
Here, we define in more detail the SAEBench metrics defined by (Karvonen al., 2024) and used in Section 5.1, along with the full results over different hyperparameter splits.
Absorption. Feature absorption occurs when latent representing concept A implicitly encodes a related concept B (e.g.- Elephant + gray), leading to redundancy or loss of interpretability. This phenomenon disrupts feature disentanglement, as absorbed features may activate inconsistently, obscuring their semantic meaning.
To measure absorption, (Karvonen et al., 2024) adapt the method of (Chanin et al., 2024) using a first-letter classification task. logistic regression probe is trained on residual stream activations establish a ground truth feature direction. They then perform k-sparse probing SAE latents, identifying primary latents responsible for the task. If increasing l: significantly improves Fl by some threshold, the new latent is classified as a feature split.
They then detect absorption by identifying test cases where primary latents fail while the probe succeeds. A latent is flagged as absorbing the feature if it strongly aligns with the probe in cosine similarity and accounts for a sufficient fraction of the probe projection.
Spurious Correlation Removal. The spurious correlation removal (SCR) metric evaluates whether the SAE captures separate latents for distinct concepts (e-g., gender vs. profession). A classifier is trained on a deliberately "biased" dataset (e.g., only male professor, female nurse), thereby picking up the spurious correlation, and then the latents most associated with the spurious feature (e.g., gender) are zero-ablated.
During evaluation, the classifier is to be debiased. Choosing the top n latents according to their probe attribution score, a modified classifier is defined in which all latents except for the spuriously correlated latent are zero ablated. Evaluated on a balanced dataset, this modified classifier's accuracy in classifying its concept tracked, and the metric defined as
Aabl
Abase
SSHIFT
Aoracle
Abase
where Aai is the probe accuracy after ablation, Anuse is the original spurious probe's ' accuracy, and Auracte is the accuracy of probe directly trained on the concept. This SHIFT score quantifies how much ablation improves accuracy (removing the spurious signal), relative. A higher score indicates better separation of the spurious feature and stronger debiasing.
Targeted Probe Perturbation. SHIFT operates on datasets with correlated labels. To extend SHIFT to all multiclass NLP datasets, (Karvonen al., 2024) introduce TPP, a method that identifies structured sets of SAE latents that disentangle dataset classes. This approach involves training probes on model activations and assessing the impact of ablating specific latent sets on probe accuracy. Ideally, removing disentangled set of latents should only impact the corresponding class probe while leaving other class probes unaffeeted.
Consider a dataset where each input is assigned a single label from a set of m possible concepts, C = {ci, C22 } For each class indexed by i E {1,..., m). the most relevant latents L; are determined using probe attribution scores. To evaluate their effect, the dataset is partitioned into instances belonging to the target concept oj and a mixed subset containing randomly sampled instances from other labels.
linear classifier C2 is defined predict concept ej with an accuracy of Aj. Furthermore, let C..j denote the classifier for e when latents in L; are ablated. The accuracy of each classifier Ct.y the corresponding dataset partition for ej is then computed as Aig. The TPP metric is given by:
1 Σ Aj) m (fj
STPP
(A - A2)
This metric quantifies the extent which ablating disentangled set of latents selectively affects its corresponding class. A well-disentangled latent representation should cause significant accuracy drop when = j (i.e., ablating latents relevant to class i in classifier C,) while having minimal effect when i * j.
Sparse Probing. To evaluate the SAE's ability to learn specific features, SAES are tested on diverse tasks (e.g., language ID, profession classification, sentiment analysis). Inputs are encoded with the SAE, mean-pooled over non non-padding tokens, and the top-K latents are selected via maximum mean difference. A logistic regression probe trained on these latents and
14

---

Low-Rank Adapting Models for SAES
Table Using the same TopK SAE trained Gemma-2- 2-2B, we compare the SAEBench metrics when the underlying model low-rank adapted with rank 64. The threshold hyperparameter for SCR and TPP denotes how many of the top latents are used in the modified classitier.
DOWNSTREAM METRICS
LORA MODEL
BASE MODEL
SCR METRIC @2
0.094
0.097
SCR METRIC @
0.196
0.177
SCR METRIC @10 10
0.260
0.253
SCR METRIC @20
0,336
0.327
SCR METRIC @50
0.447
0.448
SCR METRIC 100
0.526
0.400
SCR METRIC @500
0.342
0.325
TPP METRIC @2
0.013
0.007
TPP METRIC @5
0,023
0.014
TPP METRIC @ 10
0.035
0.023
TPP METRIC @20
0.085
0.039
TPP METRIC @50
0.184
0.128
TPP METRIC @ 100
0.266
0.194
TPP METRIC @500
0.412
0.372
SPARSE PROBING (ToP 1)
0.760
0.732
SPARSE PROBING (ToP 2)
0.833
0.832
SPARSE PROBING (TOP 5)
0.875
0.875
SPARSE PROBING (ToP 10)
0.910
0.907
SPARSE PROBING (TOP 20)
0.930
0.930
SPARSE PROBING (ToP 50)
0.946
0.946
SPARSE PROBING (TEST)
0.956
0,955
AUTOINTERP ABSORPTION
0.830
0.832
0.210
0.205
evaluated held-out test set assess how well the SAE captures the target features. A higher score reflects better feature representation (Karvonen et al., 2024).
A.3. Activation Distances
In Figure 11 show how low rank adapting the model affects the cosine similarity and L2 distance of the model activations with and without the SAE.
15

---

Low-Rank Adapting Models for SAEs
Change Cos Similarity
Rank
Rank
0.0
0.0015
Change Distance
-0.5
0.0010
0.0005
-1.5
-2.0
0.0000
3.0
12
Number of Layers after SAE
Number of Layers after SAE
Figure 11. Change average distance original model activations before and after applying LoRA; increases in cosine similarity (Left) and decreases in Euclidean distance (Right) are gaod. Thus, the adapted model with an inserted SAE more closely follows the original
16


Reviews:
[{"review_id": "ysVLAUvKxt", "reviewer": "Reviewer_PHUR", "summary": "This paper introduces a novel approach to improve the interpretability of SAE by using LoRA to fine-tune **LLMs** around previously trained SAEs. Unlike previous work that focused on optimizing SAE architectures, this approach optimizes the language model itself to work better with an existing SAE. Across various experiments with SAE sparsity, width, language model size, LoRA rank, and model layer, the method reduces the cross entropy loss gap by 30% to 55% when SAEs are inserted during the forward pass. The technique can adapt multiple SAEs simultaneously, significantly reducing compound cross entropy loss (e.g., from 7.83 to 2.78 nats with 7 SAEs). The paper demonstrates that improving model interpretability is not limited to post-hoc SAE training; Pareto improvements can also be achieved by directly optimizing the model while keeping the SAE fixed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your time and help, especially in regards to your suggestions for additional experiments, which we have now run. We are especially glad that you appreciated the novelty of modifying the model itself to better fit a sparse autoencoder, as we agree that this idea is the major impact of our work.\n\n---\n\n> More diverse model architectures could be tested, e.g., Mistral and Qwen (where there are already some SAEs and there's no need to train SAE from zero).\n\nThank you for this suggestion! We ran additional experiments on the layer 16 Mistral 7B SAE from SAELens and found that our method still works (we did not see a Qwen SAE on SAELens). The Mistral 7B base model has a CE loss of 1.5485 on our val set, the model with the SAE inserted has a CE loss of 1.8007, and the LoRA model with SAE inserted has a CE loss of 1.6548, reducing the cross entropy loss gap by 57.8%.\n\n> Different SAE sizes and capabilities could also be considered to prove that this method is constantly surpassing SAE rather than needing a strong SAE to start.\n\nWe show that our method works on a large number of SAE sizes and sparsities in Section 4; for all of them it surpasses the performance of the original SAE. We also show that even for partially trained SAEs our method works well; see Figure 1. Please let us know if there is anything we could add to the main text (experiments or wording) to make this stronger, thank you!\n\n\n> Could you replicate the performance experiment on standard NLP benchmarks on the 27B model? I'm concerned about whether this method could scale up to bigger models (and also more complicated features and capabilities).\n\nThank you for this suggestion! Below, we include a version of Table 5 with a 27B model. Our method seems to scale up well to bigger models. Interestingly, the adapted model alone outperforms the original pretrained model on many of the capability benchmarks across all model sizes; we believe this is evidence our method scales well, and that the adapted models from our method could be of interest for real world use cases.\n\n*Italicized* metrics indicate best performing between models with SAE inserted\n**Bolded** metrics indicate best performing method overall.\n\n**Gemma-2-2b**\n|Metric|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|44.2±0.4|*45.8±0.4*|49.3±0.4|**50.0±0.4**|\n|HellaSwag|50.9±0.5|*52.1±0.5*|55.0±0.5|**56.0±0.5**|\n|BLEU|29.9±1.6|*30.6±1.6*|30.4±1.6|**32.4±1.6**|\n|ROUGE-1|28.2±1.6|*28.5±1.6*|26.9±1.6|**30.2±1.6**|\n|ROUGE-2|24.8±1.5|*26.6±1.5*|25.6±1.5|**29.1±1.6**|\n|MC1|23.1±1.5|*23.4±1.5*|24.1±1.5|**24.3±1.5**|\n\n---\n\n**GEMMA-2-9b**\n|**Metric**|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|64.2±0.4|*65.7±0.4*|**70.0±0.4**|68.8±0.4|\n|HellaSwag|58.3±0.5|*59.6±0.5*|61.2±0.5|**61.9±0.5**|\n|BLEU|40.9±1.7|*42.4±1.7*|**43.8±1.7**|43.6±1.7|\n|ROUGE-1|39.0±1.7|*40.6±1.7*|42.7±1.7|**43.5±1.7**|\n|ROUGE-2|33.4±1.7|*36.4±1.7*|38.3±1.7|**38.8±1.7**|\n|MC1|27.1±1.6|*28.0±1.6*|30.5±1.6|**31.0±1.6**|\n\n---\n\n**GEMMA-2-27b**\n|**Metric**|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|70.9±0.4|*71.3±0.4*|72.1±0.3|**72.7±0.3**|\n|HellaSwag|61.0±0.5|*62.7±0.5*|65.3±0.5|**65.5±0.5**|\n|BLEU|*40.9±1.7*|38.9±1.7|41.1±1.7|**41.9±1.7**|\n|ROUGE-1|*41.0±1.7*|38.3±1.7|40.9±1.7|**41.7±1.7**|\n|ROUGE-2|*37.1±1.7*|35.3±1.7|36.2±1.7|**37.1±1.7**|\n|MC1|30.2±1.6|*31.5±1.6*|**33.8±1.7**|32.9±1.6|\n\n\n---\n\nThank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper? If not, what further clarification or modifications could we make to improve your score?", "claims_and_evidence": "Mostly. see comments below.", "methods_and_evaluation_criteria": "The idea of training the model according to interpretability methods is very novel, and I consider it to be the main reason for the paper to be accepted. The evaluation criteria is frontier and up-to-date.", "theoretical_claims": "No theoretical claims have been made in this paper.", "experimental_designs_or_analyses": "Overall, the experiment setup is mostly comprehensive and sound, but there are still points to be improved. More ablation studies and analyses could be conducted to prove that the performance gain of the method does not randomly appear but is a constant improvement. For example:\n\n- More diverse model architectures could be tested, e.g., Mistral and Qwen (where there are already some SAEs and there's no need to train SAE from zero).\n- Different SAE sizes and capabilities could also be considered to prove that this method is constantly surpassing SAE rather than needing a strong SAE to start.", "relation_to_broader_scientific_literature": "Yes, the relation is clearly identified by the authors, and the method is indeed improving the performance and interoperability compared to related works and has solved some of the concerns (e.g., training time and efficiency) proposed in previous works.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The significance and scalability of this method could be a little concerning, and I also quite doubt whether this method could actually be applied to real-world models (to address this, maybe the authors could test the method on benchmarks that is closer to real-world usage, but that won't affect my general evaluation of this idea).", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Could you replicate the performance experiment on standard NLP benchmarks on the 27B model? I'm concerned about whether this method could scale up to bigger models (and also more complicated features and capabilities).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kLtKOHQcif", "reviewer": "Reviewer_rL2B", "summary": "This paper introduces an approach for improving sparse autoencoders (SAEs) used in language model interpretability by using Low-Rank Adaptation (LoRA) to finetune the language model itself around a previously trained SAE. Their method freezes both the original model and the SAE while only training low-rank adapters to minimize the KL divergence between the model's original outputs and the outputs with the SAE inserted.\nThe authors present experiments showing their approach reduces the cross entropy loss gap by 30-55% across various settings (SAE sparsity, width, model size, LoRA rank, and model layer). They demonstrate computational efficiency compared to end-to-end SAEs and claim improvements on downstream tasks, including SAEBench metrics and general language model capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your time and thorough feedback, especially with regards to our methodology. We were glad to hear you found the implementation to be well executed.\n\n---\n\n> How do you reconcile your approach of modifying the model with the traditional goal of mechanistic interpretability?\n\nThis is an important point, thank you! As we state in our limitations (Section 7.1), our method is not appropriate when the model cannot be modified. However, while much interpretability research focuses on fixed models, other work builds interpretable models from scratch or modifies existing models to be interpretable. As cited in our related work, this includes Lai & Heimersheim, 2024; Lai & Huang, 2024; Elhage et al., 2022b; Liu et al., 2023; Liu et al., 2024; Heimersheim 2024. We consider our paper well situated in this latter category and thus an important research contribution. \n\nMoreover, because pretrained models perform poorly with SAEs inserted, one may be skeptical if models are actually using the more interpretable SAE latents. An interpretable basis is unhelpful if the model does not actually use it. Because our method optimizes the model to use SAE latents downstream, we can be more confident they are really using them.\n\n> Have you analyzed whether the adapted model still exhibits the same internal computational patterns?\n\nGreat question! In Figure 11, we compare the distance between the downstream layer activations of the pretrained model (gemma-2-2b) with the downstream layer activations of 1) the pretrained model with the SAE, and 2) the adapted model with the SAE. We find that the adapted model with the SAE has a higher cosine similarity with the pretrained model’s original activations than the pretrained model with the SAE. Thus, our adapted model + SAE is more similar to the original model than the pretrained model + SAE. \n\nWe also ran a quick experiment comparing just the adapted and pretrained model (no SAE involved). We found the average per-layer cosine similarity between pretrained and adapted model activations on the Pile was consistently greater than 0.99 (for comparison, the average cosine similarity with the inserted SAE is ~0.95). \n\n> How do you ensure that the modified model still maintains the properties that made the original model worth studying?\n\nGreat question! It is true that by modifying the model, we may be losing capabilities of the original model. To allay this concern, we ran our adapted model on the benchmarks from Table 5 and found that not only does the modified model still perform competitively with the pretrained model, but that in many cases it *outperforms* the original pretrained model. \n\n*Italicized* metrics indicate best performing between models with SAE inserted\n\n**Bolded** metrics indicate best method overall.\n\n**Gemma-2-2b**\n|Metric|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|44.2±0.4|*45.8±0.4*|49.3±0.4|**50.0±0.4**|\n|HellaSwag|50.9±0.5|*52.1±0.5*|55.0±0.5|**56.0±0.5**|\n|BLEU|29.9±1.6|*30.6±1.6*|30.4±1.6|**32.4±1.6**|\n|ROUGE-1|28.2±1.6|*28.5±1.6*|26.9±1.6|**30.2±1.6**|\n|ROUGE-2|24.8±1.5|*26.6±1.5*|25.6±1.5|**29.1±1.6**|\n|MC1|23.1±1.5|*23.4±1.5*|24.1±1.5|**24.3±1.5**|\n\n---\n\n**GEMMA-2-9b**\n|**Metric**|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|64.2±0.4|*65.7±0.4*|**70.0±0.4**|68.8±0.4|\n|HellaSwag|58.3±0.5|*59.6±0.5*|61.2±0.5|**61.9±0.5**|\n|BLEU|40.9±1.7|*42.4±1.7*|**43.8±1.7**|43.6±1.7|\n|ROUGE-1|39.0±1.7|*40.6±1.7*|42.7±1.7|**43.5±1.7**|\n|ROUGE-2|33.4±1.7|*36.4±1.7*|38.3±1.7|**38.8±1.7**|\n|MC1|27.1±1.6|*28.0±1.6*|30.5±1.6|**31.0±1.6**|\n\n---\n\n**GEMMA-2-27b**\n|**Metric**|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|70.9±0.4|*71.3±0.4*|72.1±0.3|**72.7±0.3**|\n|HellaSwag|61.0±0.5|*62.7±0.5*|65.3±0.5|**65.5±0.5**|\n|BLEU|*40.9±1.7*|38.9±1.7|41.1±1.7|**41.9±1.7**|\n|ROUGE-1|*41.0±1.7*|38.3±1.7|40.9±1.7|**41.7±1.7**|\n|ROUGE-2|*37.1±1.7*|35.3±1.7|36.2±1.7|**37.1±1.7**|\n|MC1|30.2±1.6|*31.5±1.6*|**33.8±1.7**|32.9±1.6|\n\n\n\n> Do you have evidence that the features discovered after model adaptation are more semantically interpretable to humans? \n\nThank you for this question; we apologize for not being clearer! We are not trying to find more interpretable features. Indeed, in sections 4.1 & 4.2 (where we recover up to 55% of the loss gap), we adapt only after the SAE, so the features (and their interpretability) are *unchanged*. Rather, we are creating an alternative model that better uses these interpretable features downstream.\n\n> What does the f in Eq. 5 (line 158) stand for?\n\nSAE dictionary **f**eatures.\n\n---\n\nThank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper? If not, what further clarification or modifications could we make to improve your score?", "claims_and_evidence": "While the paper presents comprehensive experimental evidence for its technical claims, there is a fundamental conceptual issue that undermines the paper's premise:\n1. The paper claims to improve model interpretability by adapting the model to better accommodate SAEs. However, this approach appears to misunderstand the purpose of SAEs in mechanistic interpretability. SAEs are meant to be analytical tools that reveal the features present in a fixed, pretrained model's activations. By modifying the model itself to better accommodate the SAE, the authors are effectively changing what's being analyzed rather than improving the analysis method.\n2. The cross-entropy loss improvements are well-documented, but it's unclear whether these improvements actually serve the goal of better interpretability or simply represent overfitting the model to better accommodate a specific analysis tool.", "methods_and_evaluation_criteria": "The proposed methods are technically sound but conceptually problematic for the interpretability goal:\n\n1. In mechanistic interpretability, the model being analyzed should remain fixed - modifying it defeats the purpose of understanding the original model's behavior.\n2. While the authors use SAEBench metrics to claim improved interpretability, it's unclear whether these metrics actually measure feature interpretability or just the degree to which the model has been tuned to make the SAE perform better.\n3. The evaluation focuses heavily on cross-entropy loss and general model performance metrics, which are not necessarily aligned with the goal of finding more interpretable features.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "The experiments technically demonstrate what they claim, but several issues arise:\n\n1. There's no evaluation of whether the features discovered after model adaptation are actually more interpretable to humans, only that they perform better on automated metrics.\n2. The paper doesn't address whether adapting the model might actually hide or modify important original features that would be relevant for interpretability.\n3. The feature steering experiments don't clearly show that the steered features are more semantically coherent, only that they have stronger statistical effects.", "relation_to_broader_scientific_literature": "The paper situates itself within mechanistic interpretability literature but appears to misunderstand a fundamental aspect of this field. Traditional mechanistic interpretability approaches aim to analyze fixed models to understand how they work internally. By modifying the model to better accommodate the analysis tool (SAE), this paper reverses the relationship between the object of study and the analytical method.\nThis reversal represents a significant departure from the goals described in foundational works and more recent SAE application papers, which focus on understanding fixed models rather than adapting models to analysis methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1. The technical implementation is sound and well-executed.\n2. The experiments are comprehensive across different model scales and SAE configurations.\n3. The method does effectively reduce the computational cost compared to e2e SAEs.\n\nWeaknesses:\n\n1. The central premise appears to fundamentally misunderstand the purpose of SAEs in mechanistic interpretability. SAEs are meant to discover interpretable features in fixed models, not to guide model adaptation.\n2. By modifying the model to better fit the SAE, the authors may be creating an artificial scenario that doesn't reflect how the original model actually processes information, undermining the core goal of interpretability.\n3. The paper conflates sparsity with interpretability. While sparsity can help with interpretability, the primary goal is to find semantically meaningful features. The paper focuses almost exclusively on improving sparse reconstruction without demonstrating improved semantic interpretability.\n4. The approach may actually hinder true interpretability by adapting the model to make the SAE work better, potentially masking or altering the original computational mechanisms that interpretability research aims to discover.\n5. The paper doesn't provide human evaluation of the interpretability of the features found after model adaptation compared to those from standard SAEs.", "other_comments_or_suggestions": "What does the $\\textbf{f}$ in Eq. 5 (line 158)  stand for?", "questions_for_authors": "Questions for Authors\n\n1. How do you reconcile your approach of modifying the model with the traditional goal of mechanistic interpretability, which is to understand the computational mechanisms of fixed, pretrained models? Does changing the model not defeat the purpose of analyzing it?\n\n2. Do you have evidence that the features discovered after model adaptation are more semantically interpretable to humans, rather than just achieving better reconstruction or downstream metrics?\n\n3. Have you analyzed whether the adapted model still exhibits the same internal computational patterns as the original model? If not, how can we be confident that we are still studying the same phenomena?\n\n4. Your method essentially changes what is being analyzed rather than improving the analysis method. How do you ensure that the modified model still maintains the properties that made the original model worth studying?\n\n5. Sparsity is typically used as a proxy for interpretability, not as the end goal. Your paper focuses heavily on improving the sparsity-performance tradeoff, but does this actually translate to more humanly interpretable features? What evidence do you have for this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yij9lzSB9j", "reviewer": "Reviewer_xLuu", "summary": "This paper trains some parts of a Transformer with LoRAs in order to make an SAE more accurate at reconstructing activations as a sparse linear sum. They achieve impressive upstream metric results at a low cost, and also achieve reasonable SAEBench downstream (ish!) results too.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are greatly thankful for your time and feedback, especially related to the limitations of our work. We were glad to hear you found the evaluations comprehensive.\n\n---\n\n> Additionally, I am concerned that far too much focus is on improving simple SAE metrics… At absolute minimum, the authors should be much more open about the limitations of their work.\n\nThis is a fair criticism, thank you! We agree that our multi-SAE section does not directly show improvements to SAE circuits. We have added the following to the limitations section, please let us know what you think!\n```\nWhile our approach develops models that more effectively leverage the interpretable features learned by SAEs in downstream tasks, we do not yet show that these can be leveraged into additional insights as to how the original model internally computes using these features. We leave this important direction for future work.\n```\n\n> In addition to those worries about circuit analysis, I don’t think you can use error terms with your approach? But often attaching lots of SAEs is very lossy, so error terms are essential.\n\nThis is a good question! It is true that one cannot use error terms with our method to obtain the activations of the original model, but in return we get the benefit that our method directly attacks the issue that “attaching lots of SAEs is very lossy.” For example, note that in Figure 6, even inserting 7 SAEs with our method results in a reasonable cross entropy loss of 2.78, and inserting 3 SAEs into the LoRA model has better loss than the pretrained model with 1 SAE.\n\nWe also note that one could use error terms to recover the performance of the adapted model, which we show below is similar (and in some cases even better) than the original model.\n\n> Your paper requires a very expensive training procedure that will need to be redone if different SAEs need to be attached\n\nWhile it’s true the training procedure needs to be redone if different SAEs are attached, the training procedure is actually very cheap. Taking off the shelf SAEs, training LoRA adapters takes only a few minutes or hours (depending on your model size), which is many times cheaper than the original SAE training process.\n\n> Why is this a promising method for applications of probing? …if the way to monitor requires editing weights before the SAE then the latents will not be able to used on their own as probes\n\nThis is a great point and something we overlooked. We have added a discussion of this to Section 5.1. \n\nHowever, as you note someone using our method for probing could run the adapted model without the SAE (because they would just be using it for probing), and we are excited to show that in new experiments, the adapted model with no SAE is actually as good or better than the original model on many capability benchmarks:\n\n*Italicized* indicates best performing between models with SAE inserted\n\n**Bolded** indicates best overall.\n\n**Gemma-2-2b**\n|Metric|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|44.2±0.4|*45.8±0.4*|49.3±0.4|**50.0±0.4**|\n|HellaSwag|50.9±0.5|*52.1±0.5*|55.0±0.5|**56.0±0.5**|\n|BLEU|29.9±1.6|*30.6±1.6*|30.4±1.6|**32.4±1.6**|\n|ROUGE-1|28.2±1.6|*28.5±1.6*|26.9±1.6|**30.2±1.6**|\n|ROUGE-2|24.8±1.5|*26.6±1.5*|25.6±1.5|**29.1±1.6**|\n|MC1|23.1±1.5|*23.4±1.5*|24.1±1.5|**24.3±1.5**|\n\n---\n\n**GEMMA-2-9b**\n|**Metric**|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|64.2±0.4|*65.7±0.4*|**70.0±0.4**|68.8±0.4|\n|HellaSwag|58.3±0.5|*59.6±0.5*|61.2±0.5|**61.9±0.5**|\n|BLEU|40.9±1.7|*42.4±1.7*|**43.8±1.7**|43.6±1.7|\n|ROUGE-1|39.0±1.7|*40.6±1.7*|42.7±1.7|**43.5±1.7**|\n|ROUGE-2|33.4±1.7|*36.4±1.7*|38.3±1.7|**38.8±1.7**|\n|MC1|27.1±1.6|*28.0±1.6*|30.5±1.6|**31.0±1.6**|\n\n---\n\n**GEMMA-2-27b**\n|**Metric**|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|70.9±0.4|*71.3±0.4*|72.1±0.3|**72.7±0.3**|\n|HellaSwag|61.0±0.5|*62.7±0.5*|65.3±0.5|**65.5±0.5**|\n|BLEU|*40.9±1.7*|38.9±1.7|41.1±1.7|**41.9±1.7**|\n|ROUGE-1|*41.0±1.7*|38.3±1.7|40.9±1.7|**41.7±1.7**|\n|ROUGE-2|*37.1±1.7*|35.3±1.7|36.2±1.7|**37.1±1.7**|\n|MC1|30.2±1.6|*31.5±1.6*|**33.8±1.7**|32.9±1.6|\n\n\n\n> …surely this is supposed to say un-frozen?\n\nWe think this is written correctly. To clarify, in LoRA finetuning, the weight matrices W of the original model are frozen, and only low rank matrices A and B are learned. The forward pass through the model is then (W + AB)x instead of Wx. We hope this clarifies things; we apologize for not being more clear.\n\n---\n\nThank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper? If not, what further clarification or modifications could we make to improve your score?", "claims_and_evidence": "The claims about this training process seem accurate, but there are some downstream applications I am concerned with: see my questions.\n\nAdditionally, I am concerned that far too much focus is on improving simple SAE metrics rather than improving interpretability concretely. This paper is very long anyway, so the authors may have considered it out of scope, but I don't think there evidence on circuit analysis is anywhere near sufficient to suggest that their technique is likely useful for improving our understanding of models.\n\nAt absolute minimum, the authors should be much more open about the limitations of their work.", "methods_and_evaluation_criteria": "The evaluation is fairly comprehensive, e.g. SAEBench and also normal CE loss style evals\n\nI was somewhat sad that no novel findings were made to advance interp -- e.g. a circuit analysis experiment that seem to go a lot better than expected with these new tools.", "theoretical_claims": "Theoretical claims were fine.", "experimental_designs_or_analyses": "The experimental setup is done well.", "relation_to_broader_scientific_literature": "Interpretability has wide appeal and I think if this research enabled research (unclear) it would be impactful in wider communities.", "essential_references_not_discussed": "N/A this paper is comprehensive with references.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "\"Concretely, for each frozen weight matrix...\" -- surely this is supposed to say **un**-frozen?", "questions_for_authors": "1a. Why is this a promising method for circuit discovery? I see significant downside. Normally, you can take off-the-shelf SAEs, and choose a subset of them to use for circuit analysis. E.g. https://openreview.net/forum?id=sdLwJTtKpM off the top of my head. Your paper requires a very expensive training procedure that will need to be redone if different SAEs need to be attached.\n\n1b. In addition to those worries about cirucit analysis, I don't think you can use error terms with your approach? But often attaching lots of SAEs is very lossy, so errors terms are essential.\n\n2. Why is this a promising method for applications of probing? A key reason to do probing work, from my perspective, is to be able to make monitors production-ready systems, e.g. frontier models at labs. But if the way to monitor requires editing weights before the \nSAE then the latents will not be able to used on their own as probes (for example), entirely new model weights will need be used. And it seems your weights have a decent dip in performance on e.g. MMLU so this seems to limit applications of your tool", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9cRCodrPYy", "reviewer": "Reviewer_YW8j", "summary": "This paper proposes improving the use of SAEs in LLMs by fitting a LORA to the LLM with the SAE inserted. The results indicate that LORAs are an effective countermeasure to the reduction in performance of inserted SAEs.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and time! We are especially glad to hear you found our experiments and results to be thorough and convincing.\n\n---\n\n> Can you explain 5.2 in more detail?\n\nCertainly! We apologize for not being clearer and hope the below will explain in more detail how we’re computing our metric:\n\nFor a given dataset D, for both the base and LoRA model:\n- For a given text sample $i \\in D$, we compute the average per token log likelihood $a_i$.\n- On the same text sample $i \\in D$, we compute the average per token log likelihood $b_i$ when we steer with an SAE feature $f$.\n- We repeat this over a dataset of text samples.\n- We normalize all the log likelihoods (both the ones with and without steering) by subtracting a constant and scaling factor such that the non-steering log likelihoods range from 0 to 100. The constant and scaling factor are computed based on the non-steered log-likelihoods. We normalize this way so that the change in log likelihood after steering loosely represents a % change in likelihood after steering. We keep track of all the changes in LL after steering as $\\delta_i$.\n\nWe record $\\Delta_D$ as the mean change $\\delta_{i, \\text{LoRA}} - \\delta_{i, \\text{base}}$, with standard error estimates.\n\nOn a dataset consisting of text that exhibit the SAE feature (which we denote as “positive” texts), a model that steers more effectively would have a higher change in LL after steering (that is $\\Delta_{\\text{positive}} > 0$) because it would indicate the positive text sample is more likely after steering. \n\nHowever, on a dataset of texts that do not exhibit the SAE feature (which we denote as “negative” texts), we want the change in LL to remain the same or slightly decrease  (that is, $\\Delta_{\\text{negative}} \\le 0$) after steering, as this indicates that steering with an SAE feature f does not significantly affect the likelihood of unrelated texts.\n\n\n> the method is somewhat a simple premise: SAE + LORA. There isn’t really a novel contribution in making LORA or SAE more amenable to solving the problem.\n\nWe agree the method is simple! We believe that’s a strong appeal of our results, as practitioners prefer implementing simpler techniques. We also study the performance of our technique in many settings and analyze why the method works (Section 6), which we believe goes beyond merely combining SAEs and LoRA.\n\n> What is “nats”? (e.g., line 088 “0.17 nats”)\n\nnats is a unit measuring information and is the unit for cross entropy loss (it is the base e analog of bits)\n\n> captions above tables is a little unusual\n\nWe agree and usually prefer having them on the bottom, but having captions above the table is required as per the ICML style guide.\n\nWe are also excited to share we ran new evaluations on just our adapted model (no SAE inserted) and found that it outperforms the original model on many of the capability benchmarks across different model sizes (see table). We find it compelling that the models created using our method not only demonstrate stronger evidence of actually using the interpretable SAE features downstream, but also match or even exceed the general performance capabilities of the original pretrained model.\n\n*Italicized* metrics indicate better performing between models with SAE inserted\n\n**Bolded** metrics indicate best performing method overall.\n\n**Gemma-2-2b**\n|Metric|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|44.2±0.4|*45.8±0.4*|49.3±0.4|**50.0±0.4**|\n|HellaSwag|50.9±0.5|*52.1±0.5*|55.0±0.5|**56.0±0.5**|\n|BLEU|29.9±1.6|*30.6±1.6*|30.4±1.6|**32.4±1.6**|\n|ROUGE-1|28.2±1.6|*28.5±1.6*|26.9±1.6|**30.2±1.6**|\n|ROUGE-2|24.8±1.5|*26.6±1.5*|25.6±1.5|**29.1±1.6**|\n|MC1|23.1±1.5|*23.4±1.5*|24.1±1.5|**24.3±1.5**|\n\n---\n\n**GEMMA-2-9b**\n|**Metric**|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|64.2±0.4|*65.7±0.4*|**70.0±0.4**|68.8±0.4|\n|HellaSwag|58.3±0.5|*59.6±0.5*|61.2±0.5|**61.9±0.5**|\n|BLEU|40.9±1.7|*42.4±1.7*|**43.8±1.7**|43.6±1.7|\n|ROUGE-1|39.0±1.7|*40.6±1.7*|42.7±1.7|**43.5±1.7**|\n|ROUGE-2|33.4±1.7|*36.4±1.7*|38.3±1.7|**38.8±1.7**|\n|MC1|27.1±1.6|*28.0±1.6*|30.5±1.6|**31.0±1.6**|\n\n---\n\n**GEMMA-2-27b**\n|**Metric**|SAE|SAE+LoRA|Original|LoRA|\n|--------------|---------------|--------------------|----------------|------------------|\n|MMLU|70.9±0.4|*71.3±0.4*|72.1±0.3|**72.7±0.3**|\n|HellaSwag|61.0±0.5|*62.7±0.5*|65.3±0.5|**65.5±0.5**|\n|BLEU|*40.9±1.7*|38.9±1.7|41.1±1.7|**41.9±1.7**|\n|ROUGE-1|*41.0±1.7*|38.3±1.7|40.9±1.7|**41.7±1.7**|\n|ROUGE-2|*37.1±1.7*|35.3±1.7|36.2±1.7|**37.1±1.7**|\n|MC1|30.2±1.6|*31.5±1.6*|**33.8±1.7**|32.9±1.6|\n\n---\n\nThank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper? And are there any further clarification or modifications we could make to improve your score?", "claims_and_evidence": "The claims of the paper are well supported through a variety of experiments. The use of various different SAEs, LLMs, and training strategies is thorough and convincing.", "methods_and_evaluation_criteria": "The eval is in line with current work on evaluating SAEs. Everything is very sensible.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I didn't see any particular issues with the experimental designs. I was a bit confused with 5.2 in general. While I am familiar with steering LLMs, I am only vaguely able to follow the setup for this experiment. I feel Pres et al.'s setup needs to be more clearly articulated, at least in the supplement.", "relation_to_broader_scientific_literature": "This paper fits nicely into the literature of mech interpretability and is extremely up to date with the latest advancements in this fast pace field. Topk SAE is just published in ICLR and SAE bench was only announced in December.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n- the paper is well organized and easy to follow.\n- the proposed method is sensible and intuitive, solving a clear problem with SAE usage\n- the experiments are thorough; I cannot think of any additional work I'd want to see done\n\nWeaknesses:\n- the method is somewhat a simple premise: SAE + LORA. There isn't really a novel contribution in making LORA or SAE more amendable to solving the problem.", "other_comments_or_suggestions": "- What is \"nats\"? (e.g., line 088 \"0.17 nats\")\n- captions above tables is a little unusual", "questions_for_authors": "Can you explain 5.2 in more detail?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes using Low-Rank Adaptation (LoRA) to finetune language models around previously trained Sparse Autoencoders (SAEs), aiming to improve model interpretability. The authors demonstrate that their approach reduces cross entropy loss gaps by 30-55% when SAEs are inserted during the forward pass, while being computationally more efficient than end-to-end SAE training.

The reviewers found the paper to be well-executed technically, with comprehensive experiments across different model scales and SAE configurations. They appreciated the thorough evaluation using both standard metrics and SAEBench. While some concerns were raised about the conceptual alignment with traditional mechanistic interpretability goals and the need for more evidence of improved semantic interpretability, the authors provided detailed responses and additional experiments that largely addressed these issues.

The paper makes a meaningful contribution to the growing body of work on building more interpretable models, even if the core method is relatively simple. The additional experiments showing maintained or improved model capabilities after adaptation were particularly compelling. The clear writing, thorough empirical evaluation, and practical utility of the method were consistently highlighted by reviewers.

The revised version is expected to incorporate expanded discussion of limitations, particularly regarding circuit analysis applications, and provide clearer explanation of the steering experiments. With these modifications, the paper will serve as a valuable contribution to the field of model interpretability.
Decision: Accept (poster)