Accelerating Sparse Autoencoder Training via Layer-Wise Transfer Learning in Large Language Models

Authors: davide  ghilardi
federico  belotti
marco  molinari
jaehyuk  lim
abhimanyu  dubey
abhinav  jauhri
abhinav  pandey
abhishek  kadian
ahmad  al-dahle
aiesha  letman
akhil  mathur
alan  schelten
amy  yang
angela  fan
anirudh  goyal
anthony  hartshorn
aobo  yang
archi  mitra
archie  sravankumar
artem  korenev
arthur  hinsvark
arun  rao
aston  zhang
aurelien  rodriguez
austen  gregerson
ava  spataru
bap- tiste  roziere
bethany  biron
binh  tang
bobbie  chern
charlotte  caucheteux
chaya  nayak
chloe  bi
chris  marra
chris  mcconnell
christian  keller
christophe  touret
chunyang  wu
corinne  wong
cristian canton ferrer
cyrus  nikolaidis
damien  al- lonsius
daniel  song
danielle  pintz
danny  livshits
david  esiobu
dhruv  choudhary
dhruv  mahajan
diego  garcia-olano
diego  perino
dieuwke  hupkes
egor  lakomkin
ehab  albadawy
elina  lobanova
emily  dinan
eric michael smith
filip  radenovic
frank  zhang
gabriel  synnaeve
gabrielle  lee
geor- gia lewis anderson
graeme  nail
mi- alon  gregoire
guan  pang
guillem  cucurell
hailey  nguyen
hannah  korevaar
hu  xu
hugo  touvron
iliyan  zarov
arrieta  ibarra
isabel  kloumann
ishan  misra
ivan  evtimov
jade  copet
jaewon  lee
jan  geffert
jana  vranes
jason  park
jay  mahadeokar
jeet  shah
jelmer  van der linde
jennifer  billock
jenny  hong
jenya  lee
jeremy  fu
jianfeng  chi
jianyu  huang
jiawen  liu
jie  wang
jiecao  yu
joanna  bitton
joe  spisak
jongsoo  park
joseph  rocca
joshua  johnstun
joshua  saxe
junteng  jia
kalyan vasuden alwala
kartikeya  upasani
kate  plawiak
ke  li
kenneth  heafield
kevin  stone
khalid  el-arini
krithika  iyer
kshitiz  malik
kuen- ley  chiu
kunal  bhalla
lauren  rantala-yeary
lau- rens  van der maaten
lawrence  chen
liang  tan
liz  jenkins
louis  martin
lovish  madaan
lubo  malo
lukas  blecher
lukas  landzaat
luke  de oliveira
madeline  muzzi
mahesh  pasupuleti
mannat  singh
manohar  paluri
marcin  kardas
mathew  oldham
mathieu  rita
maya  pavlova
melanie  kambadur
mike  lewis
min  si
kumar  singh
mona  hassan
naman  goyal
narjes  torabi
nikolay  bash- lykov
nikolay  bogoychev
niladri  chatterji
olivier  duchenne
onur  çelebi
patrick  alrassy
pengchuan  zhang
pengwei  li
petar  vasic
peter  weng
pra- jjwal  bhargava
pratik  dubal
praveen  krishnan
singh  koura
puxin  xu
qing  he
qingxiao  dong
ragavan  srinivasan
raj  ganapathy
ramon  calderer
ricardo silveira cabral
robert  stojnic
roberta  raileanu
rohit  girdhar
rohit  patel
ro- main  sauvestre
ronnie  polidoro
roshan  sumbaly
ross  taylor
ruan  silva
rui  hou
rui  wang
saghar  hosseini
sahana  chennabasappa
sanjay  singh
sean  bell
sonia  kim
sergey  edunov
shaoliang  nie
sharan  narang
sharath  raparthy
sheng  shen
shengye  wan
shruti  bhosale
shun  zhang
simon  vandenhende
soumya  batra
spencer  whitman
sten  sootla
stephane  collot
suchin  gu- rurangan
sydney  borodinsky
tamar  herman
tara  fowler
tarek  sheasha
thomas  georgiou
thomas  scialom
tobias  speckbacher
todor  mihaylov
tong  xiao
ujjwal  karn
vedanuj  goswami
vibhor  gupta
vignesh  ramanathan
viktor  kerkez
vincent  gonguet
virginie  do
vish  vogeti
vladan  petro- vic
weiwei  chu
wenhan  xiong
wenyin  fu
whit- ney  meers
xavier  martinet
xiaodong  wang
ellen  tan
xinfeng  xie
xuchao  jia
xuewei  wang
yaelle  goldschlag
yashesh  gaur
yasmine  babaei
yi  wen
yiwen  song
yuchen  zhang
yue  li
yuning  mao
zacharie delpierre coudert
zheng  yan
zhengxing  chen
zoe  papakipos
aaditya  singh
aaron  grattafiori
abha  jain
adam  kelsey
adam  shajnfeld
adithya  gangidi
adolfo  victoria
ahuva  goldstand
ajay  menon
ajay  sharma
alex  boesen- berg
alex  vaughan
alexei  baevski
allie  feinstein
amanda  kallet
amit  sangani
anam  yunus
an- drei  lupu
andres  alvarado
andrew  caples
an- drew  gu
andrew  ho
andrew  poulton
andrew  ryan
ankit  ramchandani
annie  franco
apara- jita  saraf
arkabandhu  chowdhury
ashley  gabriel
ashwin  bharambe
assaf  eisenman
azadeh  yaz- dan
beau  james
ben  maurer
benjamin  leonhardi
bernie  huang
beth  loyd
beto  de paola
bhargavi  paranjape
bing  liu
bo  wu
boyu  ni
braden  han- cock
bram  wasti
brandon  spence
brani  stojkovic
brian  gamido
britt  montalvo
carl  parker
carly  burton
catalina  mejia
changhan  wang
changkyu  kim
chao  zhou
chester  hu
ching-hsiang  chu
chris  cai
chris  tindal
christoph  feichtenhofer
da- mon  civin
dana  beaty
daniel  kreymer
daniel  li
danny  wyatt
david  adkins
david  xu
davide  tes- tuggine
delia  david
devi  parikh
diana  liskovich
didem  foss
dingkang  wang
duc  le
dustin  hol-
edward  dowling
eissa  jamil
elaine  mont- gomery
eleonora  presani
emily  hahn
emily  wood
erik  brinkman
esteban  arcaute
evan  dunbar
evan  smothers
fei  sun
felix  kreuk
feng  tian
firat  ozgenel
francesco  caggioni
francisco  guzmán
frank  kanayet
frank  seide
gabriela  medina
gabriella  schwarz
gada  badeer
georgia  swee
gil  halpern
govind  thattai
grant  herman
grigory  sizov
  guangyi
guna  zhang
hamid  lakshminarayanan
han  shojanazeri
hannah  zou
han- wen  wang
haroun  zha
harrison  habeeb
he- len  rudolph
henry  suk
hunter  aspegren
igor  goldman
igor  molybog
irina-elena  tufanov
itai  veliche
jake  gat
james  weissman
james  geboski
japhet  kohli
jean-baptiste  asher
jeff  gaya
jeff  marcus
jennifer  tang
jenny  chan
jeremy  zhen
jeremy  reizen- stein
jessica  teboul
jian  zhong
jingyi  jin
joe  yang
jon  cummings
jon  carvill
jonathan  shepard
jonathan  mcphie
josh  torres
junjie  ginsburg
kai  wang
kam  wu
karan  hou
karthik  sax- ena
kartikay  prasad
katay- oun  khandelwal
kathy  zand
kaushik  matosich
kelly  veeraragha- van,
keqian  michelena
kun  li
ku- nal  huang
kushal  chawla
kyle  lakhotia
lailin  huang
lakshya  chen
leandro  garg
lee  silva
lei  bell
liangpeng  zhang
licheng  guo
liron  yu
luca  moshkovich
madian  wehrstedt
manav  khabsa
manish  avalani
maria  bhatt
martynas  tsim- poukelli
matan  mankus
matthew  hasson
matthias  lennie
maxim  reso
maxim  groshev
maya  naumov
meghan  lathi
michael l keneally
michal  seltzer
michelle  valko
mihir  restrepo
mik  patel
mikayel  vyatskov
mike  samvelyan
mike  clark
mike  macey
miquel  wang
mo  jubert
mohammad  metanat
mun- ish  rastegari
nandhini  bansal
natascha  santhanam
natasha  parks
navyata  white
nayan  bawa
nick  singhal
nicolas  egebo
  usunier
pavlovich  nikolay
ning  laptev
ning  dong
norman  zhang
oleg  cheng
olivia  chernoguz
omkar  hart
ozlem  salpekar
parkin  kalinli
parth  kent
paul  parekh
pa- van  saab
pedro  balaji
philip  rittner
pierre  bontrager
piotr  roux
polina  dollar
ratan- chandani  zvyagina
pritish  yuvraj
qian  liang
rachad  alao
rachel  rodriguez
rafi  ayub
raghotham  murthy
raghu  nayani
rahul  mitra
raymond  li
rebekkah  hogan
robin  battey
rocky  wang
rohan  mah- eswari
russ  howes
ruty  rinott
jayesh  sai
samyak  bondu
sara  datta
sara  chugh
sargun  hunt
sasha  dhillon
satadru  sidorov
saurabh  pan
seiji  verma
sharadh  yamamoto
shaun  ramaswamy
shaun  lind- say
sheng  lindsay
shenghao  feng
  lin
stephen  chen
steve  kehoe
steve  satterfield
sudarshan  govindaprasad
sumit  gupta
sungmin  cho
sunny  virk
suraj  subramanian
sy  choudhury
sydney  goldman
tal  remez
tamar  glaser
tamara  best
thilo  kohler
thomas  robinson
tianhe  li
tianjun  zhang
tim  matthews
timothy  chou
tzook  shaked
varun  vontimitta
victoria  ajayi
victoria  montanez
vijai  mohan
vinay satish kumar
vishal  mangla
vlad  ionescu
vlad  poenaru
vlad tiberiu mihailescu
ying  zhang
yossi  zhang
youngjin  adi
  nam
  yu
yuchen  wang
yundi  hao
yuzi  qian
zach  he
zachary  rait
zef  devito
zhaoduo  rosnbrick
  wen
nelson  elhage
tristan  hume
catherine  olsson
nicholas  schiefer
tom  henighan
shauna  kravec
zac  hatfield-dodds
robert  lasenby
dawn  drain
carol  chen
roger  grosse
sam  mccandlish
jared  kaplan
dario  amodei
martin  wattenberg
christopher 2022 olah
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-20 13:15:09

Abstract: Sparse AutoEncoders (SAEs) have gained popularity as a tool for enhancing the interpretability of Large Language Models (LLMs). However, training SAEs can be computationally intensive, especially as model complexity grows. In this study, the potential of transfer learning to accelerate SAEs training is explored by capitalizing on the shared representations found across adjacent layers of LLMs. Our experimental results demonstrate that fine-tuning SAEs using pre-trained models from nearby layers not only maintains but often improves the quality of learned representations, while significantly accelerating convergence. These findings indicate that the strategic reuse of pretrained SAEs is a promising approach, particularly in settings where computational resources are constrained.

Main paper: 
 introduction: 
 Transformer-based models have become ubiquitous in a large variety of different application fields (Dubey et al., 2024; Kirillov et al., 2023; Radford et al., 2023; Chen et al., 2021; Zitkovich et al., 2023; Waisberg et al., 2023) . Given their tremendous impact on society, concerns about their interpretability have been raised by various stakeholders (Bernardo, 2023) . Mechanistic Interpretability (MI) (Conmy et al., 2023; Nanda et al., 2023) , seeks to reverse-engineer how Neural Networks, and in particular LLMs, generate outputs by uncovering the circuits they have learned during training, stored inside their parameters, and executed during a forward pass (Nanda et al., 2023; Conmy et al., 2023; Gurnee et al., 2023) . A promising interpretability technique is dictionary learning (Cunningham et al., 2023; Gao et al., 2024; Karvonen et al., 2024) which seeks to capture interpretable and editable features within the internal layers of LLMs. This method implies training Sparse Autoencoders (SAEs) to reconstruct the model's ac-Figure 1 : Visualization of our method. From left to right: baseline method where each Sparse AutoEncoder (SAE) is trained from scratch (solid line); forward method where SAEs are initialized with weights from the previous layer's SAE and fine-tuned (dashed line) with the new layer activations; backward method where SAEs are initialized with weights from the following layer's SAE. tivations using sparse learned features. However, training SAEs is computationally intensive, particularly when applied across multiple layers in deep networks. This computational burden poses a significant barrier to their widespread application, especially in resource-constrained environments where the cost of training from scratch is prohibitive. Recent research has highlighted the potential of transfer learning as a strategy to mitigate these challenges (Kissane et al., 2024) . In particular, it has been shown in Gromov et al. (2024) that adjacent layers in LLMs are often redundant, suggesting that the knowledge encoded in one layer is also present in neighboring ones and that it can effectively be transferred. This observation forms the basis of our investigation: we hypothesize that SAEs trained on one set of layers can serve as effective initialization for SAEs designed for closely related layers. Specifically, the forward approach is defined as initializing an SAE with the weights of a previous layer SAE, and the backward approach as initializing an SAE with the weights of a subsequent layer SAE. The overall training procedure is summarized in Figure 1 . We tested this hypothesis on Pythia-160M, a small 12-layer decoder-only transformer from the Pythia family (Biderman et al., 2023) . By reusing the representations learned in earlier layers, computational demands of training can be reduced by at least 25% 1 while maintaining, or even improving, the quality of the resulting models. Our contributions are as follows:• We demonstrate that SAEs exhibit partial transfer to adjacent layers in a zero-shot setting, though fine-tuning is recommended for optimal performance.• We show that both Forward-SAEs and Backward-SAEs, when fine-tuned on adjacent activations, consistently transfer across all tested checkpoints, achieving comparable or superior performance to SAEs trained from scratch, while using significantly less training data.• We train and publicly release SAEs for Pythia-160M (Biderman et al., 2023) , the model utilized in this study.Code, data, and trained models will be publicly released after the double-blind review.2 Background and objectives 
linear representation hypothesis and superposition: 
 Although it has been demonstrated that LLMs represent some of their feature linearly (Park et al., 2024) sparse, reflecting the idea that in the natural world, many features are inherently sparse. (iii) The importance of features varies depending on the task at hand. These assumptions, combined with two mathematical principles2 , suggest that the hidden sparse features can be recovered by projecting the dense model back to the hypothetical large and disentangled one. SAEs serve this purpose: learning a set of sparse, interpretable, and high-dimensional features from an observed model's dense and superposed activations.Recently, Sparse AutoEncoders have become a popular tool in Large Language Model (LLM) interpretability as they effectively decompose neuron activations into interpretable features (Bricken et al., 2023; Cunningham et al., 2023) . For a given input activation x ∈ R d model , the SAE computes a reconstruction x as a sparse linear combination ofd sae ≫ d model features v i ∈ R d model .The reconstruction is given by:EQUATIONwhere v i are the columns of W d , b d is the bias term of the decoder and f (x) are feature activations. The latter are computed as:EQUATIONwhere b e is the encoder bias term. SAEs are trained to minimize the following loss function:EQUATIONIn Equation 3, the first term corresponds to the reconstruction error, to which an ℓ 1 regularization term on the activations f(x) is added to promote sparsity in the feature activations. In SAEs training, it is common to set d sae = c d model with c ∈ {2 n | n ∈ N + }. So,Evaluating SAEs and the features they have learned presents significant challenges. In our work, the techniques employed can be divided into reconstruction and interpretability metrics. The first includes:• The Cross-Entropy Loss Score (CES), is defined asCES = CE(ζ) -CE(x • f ) CE(ζ) -CE(Id) (4)where x • f is the autoencoder function, ζ : x → 0 the zero-ablation function and Id : x → x the identity function. According to this definition, a SAE would get a CES equal to 1 if it perfectly reconstructs x (> 1 if it improves the CE loss), ≤ 0 when the reconstruction is not better than zero-ablation, otherwise the score is comprised in the unit interval.• The L2 loss (reconstruction loss) is the first term of Equation 3, which measures the reconstruction error made by the SAE.• The L0 loss of the learned features, defined asEQUATION3 As specified in (Biderman et al., 2023) which represents the number of non-zero SAE features used to compute the reconstruction.Measuring the quality of the features learned by a SAE is not straightforward, and multiple strategies exist. As reported in Makelov et al. (2024) , interpretability metrics can be categorized as follows:• Indirect Geometric Measures: Sharkey et al. (2023) proposed using mean maximum cosine similarity (MMCS) between features learned by different SAEs to assess their quality. Given two feature dictionaries D and D ′ , with |D| = |D ′ |, MMCS is defined as:EQUATION• Auto-Interpretability: • Manually Crafted Proxies for Ground Truth: (Bricken et al., 2023) developed computational proxies for a set of SAE features, relying on manually formulated hypotheses.• Faithfulness and Completeness of task feature circuits: Marks et al. (2024) compute faithfulness and completeness as measures to estimate the task sufficiency and necessity of learned SAE features. In particular, given a task, they first compute a circuit C of SAE features by selecting them according to their importance, estimated via their Indirect Effect4 (Pearl, 2022) :IE(m; f; a c , a w ) = m[M (a c |do(f =f w ), x); M (a c |x)] (7)where x is a given prompt and m : R • Supervised Dictionary Benchmarking: Makelov et al. ( 2024) introduced a technique that benchmarks unsupervised SAE dictionaries against supervised dictionaries based on task-relevant attributes to ensure extracted features are interpretable and relevant to specific tasks.In our work, evaluation metrics employed include all the reconstruction techniques listed above, the MMCS between features from SAEs trained with transfer learning and the ones from SAEs trained from scratch, and a Human Interpretability Score defined in Section 3. Moreover, we evaluate both faithfulness and completeness on three standard downstream tasks: Indirect Object Identification (IOI) (Wang et al., 2023) , Greater Than (Hanna et al., 2023) , and Subject-Verb Agreement (Marks et al., 2024) , all of them comprising a set of examples in the form of {(x, a c , a w ) i }. Additionally, for faithfulness and completeness computation we fix the number of top important features N throughout all the experiments: for faithfulness we let N vary in {123, 246, 368, 492}, which correspond to 2%, 4%, 6% and 8% of top active features; for completeness, N varies in {4, 36, 68, 100}.6 Finally, in Appendix B we report the Direct Logit Transfer learning (Goodfellow et al., 2016 ) is a powerful technique in machine learning where knowledge gained from one task is applied to improve performance on a related, but distinct, task. This approach is particularly useful when training from scratch is computationally expensive or when labeled data is scarce. In the context of SAEs for LLMs, transfer learning enables the reuse of weights learned in one layer to initialize and accelerate the training of SAEs in adjacent layers.In this work transferability and generalization of intra-model SAEs have been studied, aiming to answer the following research questions:Q1. Are SAEs transferable between layers? I.e., can a SAE trained on the activations of layer i be reused to reconstruct activations of layer j ̸ = i?Q2. Is Transfer Learning applicable to SAEs? Specifically, can a SAE initialized with the weights of a neighboring SAE and then finetuned achieve equal or superior performance, potentially using only a fraction of the data, compared to an SAE trained from scratch?  
experimental setup: 
 To address the questions raised in Section 2, we first trained from scratch one SAE i for each layer i of Pythia-160M, a 12-layer decoder-only Transformer model from the Pythia family (Biderman et al., 2023) . Each SAE was trained using the JumpReLU activation function (Rajamanoharan et al., 2024) , with activations taken from the corresponding layer's residual stream after the MLP contribution. The model configuration details are provided in Table 1 . Let also j ̸ = i be another layer index. Then SAE i←j is defined as the SAE initialized with weights from the j-th SAE and finetuned with activations of the i-th layer. In particular, this work is focused on SAE i←i-1 and SAE i←i+1 , named Forward-SAE (Fwd-SAE) and Backward-SAE (Bwd-SAE) respectively. Figure 1 summarizes the overall training and fine-tuning procedure, with the hyperparameters specified in Table 2 . The dataset adopted for both training and fine-tuning is the Pile-small-2b7 , an already tokenized version of the Pile dataset (Gao et al., 2020) with a total of 2b tokens. To effectively measure the reconstruction performance of a SAE before and after fine-tuning with transfer learning, the normalized CE-Loss Score is adopted and defined as:EQUATION) by assuming CES(SAE j (x i )) and CES(SAE i (x i )) being, respectively, the lower and the upper bound for the CES on x i . With the definitions above, CES i,i-1 and CES i,i+1 are the normalized CE-Loss Score of the Fwd-SAE and Bwd-SAE re-spectively. Finally, to evaluate feature quality, a Human Interpretability Score has been defined as the ratio of features that have been evaluated interpretable by human annotators. To generate the score, all the SAEs have been run on approximately 1M tokens randomly sampled from the training dataset. With their activations, max activating tokens and top/bottom attribution logits have been computed and analyzed from the labelers. 
sae transferability: 
 Figure 2 shows the CE-Loss Score achieved by every SAE j reconstructing the activations of layer i, for every i, j = 0, ..., L -1, i.e., the zero-shot setting. It is clear that a certain degree of transferability exists between SAE j and the activations of adjacent layers, with this being more noticeable when i = j -1 (i.e., SAEs are more effective at reconstructing the activations of preceding layers than those of subsequent ones). These findings can also be attributed to the fact that, as demonstrated by Gromov et al. (2024) , angular distances between adjacent layers are smaller, enabling neighboring SAEs to operate on a similar basis with respect to the activations they were trained on. The answer to Q1 is, therefore, yes; however, although transferability between layers exists, it remains partial and, potentially, not completely reliable for downstream applications.Figure 3 shows all reconstruction metrics averaged for all layers across every tested checkpoint. Detailed results for single layer and aggregated over time can be found in Appendix C (Figures 9 10 11 12 13 14 15 16 17 ) along with the normalized CE-Loss Score (Eq. 9) in Tables 3 and 4 . Looking at the plots, it can be seen that forward and backward SAEs achieve almost equal or even superior performance than the ones trained from scratch with as little as 1/10-th (100M tokens) of the original training data (1B tokens), with the scores constantly increasing with the number of tokens used for finetuning. As a result, it can be said that both forward ad backward are effective strategies to reduce the number of SAEs trained from scratch. Between the two, the backward technique is the one that constantly shows better results, both in terms of CE-Loss Score, L2, and L1 loss. So, the answer to Q2 is also yes if we just consider the reconstruction metrics. To fully respond to Q2 beyond reconstruction performance, the quality of the learned SAE features have to be inspected. transfer SAEs consistently achieve better scores than the baseline SAEs, with minimal differences between the two transfer methods. Therefore, both the forward and backward SAEs maintain sufficiency and necessity during their transfer. Figure 5 presents the MMCS between SAEs trained with transfer learning and those trained from scratch. The metric value decreases for deeper layers, suggesting a slight divergence in the features learned by the transfer SAEs. Notably, SAE L-1←L exhibits a sharp decline in the score, indicating that transferring on the last layer should be approached with caution. Lastly, from human interpretability scores (Figure 7 ), no significant differences can be observed between each transfer type. By manually looking at the learned features, a key pattern has emerged: many features learned by SAEs trained with transfer learning remain shared with the SAE used for initialization. This phenomenon, termed Feature Transfer, particularly affects the most interpretable features (see an example in Figure 23 ). To further investigate this phenomenon, a metric was developed to quantify it. Given a SAE i and another trained via transfer learning from it, SAE i←i±1 , the number of shared "top", "bottom", and "max activating tokens"8 for each feature have been computed (features have been compared using the same indices). The transfer score has been then defined as the percentage of shared tokens across all three heuristics. Figure 6 presents the scores across all the layers for the last evaluated checkpoint. Except for layer 1, backward transfer consistently exhibits lower scores. It's important to note that this phe- nomenon is easily recognized in SAEs trained with transfer learning when compared to their initialization, as feature indices are preserved. Evaluating this in SAEs trained from scratch is more demanding due to the exponential growth in the number of comparisons required, and although relevant, it falls outside the scope of this work.Leveraging forward and backward transfer, we were able to reduce total training steps when utilizing forward transfer and backward transfer by 42% and 46%, respectively. Check Appendix B.1 for details. 
scaling and evaluating saes: 
 As SAEs gain popularity for LLMs interpretability and are increasingly applied to state-of-the-art models (Lieberum et al., 2024) , the need for more efficient training techniques has become evident. To address this, Gao et al. (2024) explored scaling laws of autoencoders to identify the optimal combination of size and sparsity. However, training SAEs is only one aspect of the challenge; evaluating them presents another significant hurdle. This evaluation is a crucial focus within MI. While early approaches in Cunningham et al. (2023) and (Bricken et al., 2023) relied on unsupervised metrics like reconstruction loss and L0 sparsity to assess SAE performance, these metrics alone cannot fully capture the efficacy of a SAE. They provide quantitative measures of how well SAEs capture informa- 2023)) and ground-truth comparisons (Sharkey et al., 2023) , have shifted towards a more holistic evaluation, focusing on the causal relevance of the extracted features (Marks et al., 2024) and evaluating SAEs on different downstream tasks in which they can be employed (Makelov et al., 2024) . In particular, Makelov et al. (2024) introduced a framework for evaluating SAEs on the Indirect Object Identification (IOI) task, focusing on three key aspects: the sufficiency and necessity of activation reconstructions, the ability to control model behavior through sparse feature editing, also called feature steering (Templeton et al., 2024) , and the interpretability of features in relation to their causal role. Karvonen et al. (2024) further advanced principled evaluations by introducing novel metrics specifically designed for board game language models. Their approach leverages the well-defined structure of chess and Othello to create supervised metrics for SAE quality, including board reconstruction accuracy and coverage of predefined board state properties. These methods provide a more direct assessment of how well SAEs capture semantically meaningful and causally relevant features, offering a complement to the earlier unsupervised metrics like L0 and L2.Recent work by Kissane et al. (2024) and Lieberum et al. (2024) has demonstrated the transferability of SAE weights between base and instruction-tuned versions of the Gemma-1 (Team et al., 2024a) and Gemma-2 (Team et al., 2024b) , respectively. This finding is significant as it suggests that many interpretable features are preserved during the finetuning process. While this transfer occurs between model variants (inter-model) rather than between layers (intra-model), it complements our work by indicating that SAE features can remain stable across different stages of model development. The preservation of these features through fine-tuning not only offers insights into the robustness of learned representations but also suggests potential efficiency gains in interpreting families of models derived from a common base SAE. 
conclusions: 
 We hypothesized and validated whether SAE transfer is an effective method to accelerate and optimize the SAE training process. We investigated whether SAE weights derived from adjacent layers could maintain efficacy in reconstruction, which our results affirmed. Furthermore, we examined whether the transferred SAEs, when fine-tuned on a layer's activations, could reliably capture monosemantic features comparable to the original SAE, which has been also confirmed by our experiments. The transferred SAEs (both forward and backward) demonstrated comparable and occasionally superior reconstruction loss relative to the original. Empirically, we observed frequent overlap in the most strongly activated features across adjacent layers (e.g. Figure 23 ). For a given feature index i, the features learned by SAE i←i+1 (Backward), SAE i (No Transfer), and SAE i←i-1 (Forward) appeared to represent similar concepts. 
limitations and future works: 
 While our study successfully demonstrates the feasibility of reconstruction transfer and the transfer learning of SAE weights to adjacent layers, there are several limitations that warrant consideration and pave the way for future research directions.• Model Size and Scope: We trained base and transfer SAEs on the activations of Pythia-160m, a model mcuh smaller than state-ofthe-art LLMs. Although not being tested, as model size and training complexity increase, the benefits of transfer learning are expected to become more pronounced. In such scenarios, transfer learning can significantly accelerate training and reduce associated costs, making our approach potentially more impactful for larger models. Therefore, a critical area for future research is to extend these investigations to larger models, exploring how scaling affects the efficacy of transfer learning and how these benefits can be maximized in real-world settings.• Inter-Model and Intra-Model transferability:In our study, we focused on the transfer of intra-model SAEs, particularly assessing the transferability between SAEs in adjacent layers. Given that model architectures are now commonly shared across different model families, a direction for future research would be to evaluate the transferability of intra-model SAEs within models from different families that utilize the same architecture. This exploration could offer valuable insights into the broader applicability of SAEs beyond closely related model families.• Experimental Scale and Hyperparameter Interactions: Our study was conducted on a limited scale in terms of model components involved and the range of training hyperparameters explored. The fixed set of hyperparameters used may not fully capture the potential of our transfer learning approach across different configurations. Future research should involve a broader exploration of hyperparameter spaces, especially the λ coefficient and expansion factor c, along with component variations to determine the robustness and versatility of the method.• Feature Transfer Phenomenon: Our findings reveal a "feature transfer" phenomenon, where features learned in one layer are exactly replicated in another during transfer learning. This can be problematic, as it may prevent the fine-tuned SAEs from discovering new, layer-specific features. However, it also offers an interesting opportunity to study how similar features are encoded across layers. Future research should focus on understanding and managing this phenomenon to either harness or mitigate its effects, depending on the desired outcomes, thereby improving the flexibility and effectiveness of transfer learning. 
 
 Appendix: In Equation 7we reported the Indirect Effect (IE) (Pearl, 2022) , which measures the importance of a feature with respect to a generic downstream task T . To reduce the computational burden of estimating the IE with a single forward pass per feature, we employed Attribution Patching (AtP) (Nanda, 2023; Syed et al., 2023) . AtP employs a first-order Taylor expansionÎE AtP (m; f ; a c , a w ) = ∇ f m f =fc (f w -f c ) (10)which estimates Equation 7for every f in two forward passes and a single backward pass. We also report the Direct Logit Attribution (DLA) between forward SAE i←i-1 and backward SAE i←i+1 transfer SAEs. Introduced by Bricken et al. (2023) , DLA assesses the direct effect of a feature on the next-token distribution, providing insights into the causal role of features. The attribution score is computed as follows:attr i (x; a c ; a w ) = f i v i • ∇ x L(a c , a w ) (11)where x is a given prompt and ∇ x L is the gradient of the logit difference between two contrastive answer tokens a c , a w (E.g., x = "The square root of 9 is", a c = 3, and a w = 2).We report the feature averaged DLA computed on a custom dataset comprising 64 handcrafted prompts in the form of {(x, a c , a w ) i }. Figure 8 displays the layer-averaged DLA scores for each tested checkpoint. The plot reveals that forward transfer SAEs consistently achieves higher scores than the baseline, while backward transfer SAEs consistently scores lower. This outcome contrasts with the reconstruction metrics, where the backward technique consistently outperformed the forward approach. A detailed per-layer DLA scores plot is reported in Figure 22 .This work proposes a novel method leveraging transfer learning to significantly reduce computational costs in training SAEs in the context of LLMs. We demonstrate that both Fwd-SAE SAE i←i-1 and Bwd-SAE SAE i←i+1 , trained with our fine-tuning strategy, are both valid alternatives to the standard layer-by-layer training of SAE i , in terms of both reconstruction quality of the learned representation and performance on downstream tasks. In practice, our approach consists of the following steps:1. Train a SAE i on alternate layers, depending on the transfer direction. For Forward transfer i ∈ {0, 2, 4, ..., L}, while for Backward transfer i ∈ {1, 3, 5, ..., L -1}.2. Initialize the current SAE i by either SAE i←i-1 for forward transfer or SAE i←i+1 for backward transfer.3. Apply transfer learning by training the remaining SAEs and stop when some criteria are matched (e.g., when the loss converges to a specific value or when a computational budget has been reached).Empirical results demonstrate substantial efficiency gains. In our experiments with a 12-layer Pythia-160M (Biderman et al., 2023 ) model, we observed a performance increase after fine-tuning on 10% of the training data (Figure 3 and Figure 4 ), with performance increasing over time. From Figure 9 we note how the CE-Loss Score of SAE 5 (x 6 ) and SAE 6←5 (x 6 ) are nearly identical to the obtained by SAE 6 (x 6 ), thus the increment given by the fine-tuning over the baseline SAE 5 (x 6 ), captured by the Normalized CE-Loss Score in Eq. 9, is minimal and resulting in a lower value. 9 we note how the CE-Loss Score of SAE 6 (x 5 ) and SAE 5←6 (x 5 ) are nearly identical to the obtained by SAE 5 (x 5 ), thus the increment given by the fine-tuning over the baseline SAE 6 (x 5 ), captured by the Normalized CE-Loss Score in Eq. 9, is minimal and resulting in a lower value. The Johnson-Lindenstrauss lemma, which ensures that points in a high-dimensional space can be embedded into a lower dimension while almost preserving distances, and compressed sensing, which exploits sparsity to recover signals from fewer samples than required by the Nyquist-Shannon theoremWe estimate the IE through Attribution Patching (AtP)(Syed et al., 2023;Nanda, 2023) A formal definition of AtP is given in Appendix AE.g., x = "The square root of 9 is", ac = 3, and aw = 2Top important features are computed on a per-example basis.https://huggingface.co/datasets/NeelNanda/ pile-small-tokenized-2b"Top" and "bottom" logit tokens refer to those whose unembedding directions are most and least aligned, respectively, with the projection of the feature in the unembedding space. "Max activating" tokens are those for which the feature exhibits the highest activations. 


Decision: Reviewers agree that the paper's contributions are insightful, impactful, and well-presented. The topic is also clearly relevant to the theme of the workshop. There were some concerns regarding the presentation and choice of metrics, though evaluating SAEs (and mechanistic interpretability methods more broadly) is an established open problem. Regardless, it could be good to take some of these presentational suggestions into account when preparing the camera-ready.

Reviews: review: paper_summary: paper_summary: This work aims to reduce the computational intensity of training Sparse Auto Encoders. They show that SAEs trained on one set of layers can serve as effective initialization for SAEs designed for closely related layers. This work also shows that SAEs exhibit partial transfer to adjacent layers.
strength_weakness: strength_weakness: Is there an overemphasis on reconstruction scores? The main results (and perhaps relevant as well) are centered around reconstruction, would it make sense to also include details about performance metrics on downstream tasks?

The experiments are limited in scope with respect to model size, families and datasets. Since this is an empirical result that future practitioners need to know a priori if it will work for them, it might be important to see how well the findings hold up.

Most of my concerns are covered in the limitations section and could be good directions to explore future work.
questions: questions: It would be interesting to run an experiment to consider where the transfer falls off. That is, instead of considering just the i and i+1 layers, would it be possible to extend to non-immediately adjacent layers and see where the results fail to hold?
paper_summary: paper_summary: This paper is motivated by the very high computational costs of training Sparse Autoencoders (SAE) from scratch, and proposes to alleviate this issue by reusing the weights from SAEs trained on adjacent layers. This is intuitive, and works very well in terms of reconstruction loss, mean max cosine similarity, and human evaluations. The authors show that finetuning the transferred weights to better fit the destination layer activations can even improve the SAEs beyond the quality of a from-scratch SAE in some settings. The authors are able to validate their hypothesis pretty convincingly, and provide a new way to speed up SAE training.
strength_weakness: strength_weakness: * Because of the residual connection in these networks, it should be expected that some transfer holds, as vector additions are the only modifications made to every layer. I don't the experiments goes as deeply as they could for this reason. For example:
1.) What happens if you train one SAE from scratch (at say, layer L), transfer to L-1 and finetune it, transfer that to L-2, ..., and so on? Can the transfer bubble all the way down/up or do the error accumulate?
2.) What are some examples of features that require finetuning to learn? Are there any features that are /only/ learned by SAEs trained from scratch? This would be very useful to know for practitioners to actually want to use this method of training.
3.) re: residual connection, could you also transfer SAEs trained on MLPs of adjacent layers? This could be a useful lower bound, and I think *any* transfer from adjacent layers that isn't at non-adjacent layers in this setting is interesting.

* Some of the experimental results are presented very briefly. I'm not really sure what I'm supposed to take away from Figure 5. It seems confusing to me that the forward transfer has a higher DLA score or if this is necessarily a good thing. This might also be a little terse for readers less familiar with DLA, which comes mostly from circuit analysis work.

I think this represents solid work that will be useful for the community to engage with, though some deeper analyses may make it more exciting.
questions: questions: L368: " has been then"
L476: "mcuh"

A more clear presentation of how much compute, in whatever metric makes sense (FLOPs, steps) would be helpful somewhere
paper_summary: paper_summary: The authors conduct an extensive analysis of Sparse Autoencoders (SAE) training methodologies. They demonstrate that transfer learning across layers of an LLM helps in faster training and improves on metrics - a very useful insight. In addition exhaustive experiments on model checkpoints, forward-backward transferability and the interpretability itself makes this a strong submission.
strength_weakness: strength_weakness: 1. The metrics can be better explained and summarized with a paragraph on their importance.
2. Some ablation studies on the dimension of the SAE might be helpful on different LLMs
questions: questions: See weakness