Transferring Linear Features Across Language Models With Model Stitching
Authors: Alan Chen
Jack Merullo
Alessandro Stolfo
Ellie Pavlick
Venue: NeurIPS 2025 spotlight ()
Keywords: LLMs
interpretability
linear features
sparse autoencoders
universality
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: In this work, we demonstrate that affine mappings between residual streams of language models is a cheap way to effectively transfer represented features between models. We apply this technique to transfer the \textit{weights} of Sparse Autoencoders (SAEs) between models of different sizes to compare their representations. We find that small and large models learn highly similar representation spaces, which motivates training expensive components like SAEs on a smaller model and transferring to a larger model at a FLOPs savings. For example, using a small-to-large transferred SAE as initialization can lead to 50% cheaper training runs when training SAEs on larger models. Next, we show that transferred probes and steering vectors can effectively recover ground truth performance. Finally, we dive deeper into feature-level transferability, finding that semantic and structural features transfer noticeably differently while specific classes of functional features have their roles faithfully mapped. Overall, our findings illustrate similarities and differences in the linear representation spaces of small and large models and demonstrate a method for improving the training efficiency of SAEs.

Transferring Linear Features Across Language Models With Model Stitching
Alan Chen Brown University alan_cheni @brown edu
Jack Merullo Goodfire
jack@goodfire.cc ai
Alessandro Stolfo
Ellie Pavlick Brown University
ETH Zürich stolfoa@ethz.ch
ellie_.aviiiteerrom.edl
Abstract
In this work, we demonstrate that affine mappings between residual streams of language models is cheap way to effectively transfer represented features between models. We apply this technique to transfer the weights of Sparse Autoencoders (SAEs) between models different sizes to compare their representations. We find that small and large models learn similar representation spaces, which motivates training expensive components like SAES on a smaller model and transferring to a larger model FLOPs savings. In particular, using small-to- to-ge transferred SAE as initialization can lead to 50% cheaper training runs when training SAEs larger models. Next, show that transferred probes and steering vectors can effectively recover ground truth performance. Finally, we dive deeper into feature-level transferability, finding that semantic and structural features transfer noticeably differently while specific classes of functional features have their roles faithfully mapped. Overall, our findings illustrate similarities and differences in the linear representation spaces of small and large models and demonstrate method for improving the training efficiency of SAES.
Introduction
Large language models (LLMs) have continually proven to be surprisingly intricate next token predictors that become more complex yet predictably better with scale [Kaplan et al. 2020 leaving a
desire to characterize and explain the LLM's computations (Saphra and Wiegreffe 2024 Olah et al.)
2020). Despite increasing efforts to study the rich internal mechanisms and representations of LLMs, the full computations still remain opaque (Sharkey al. 2025 Ennells et al. 2024
popular perspective to view the internal computation of L.LMs through the lens of features, or functions of the input that serve downstream purpose like predicting the next token composing to form model circuits OOlah et al., 2020 Huben al. 2024]. The Linear Representation Hypothesis (LRH) posits that these features are represented directions in high dimensional space [Park] ct al. 2024 Elhage et al. 2022]. However, the features had been empirically observed to appear in superposition (activating with interference), making interpretation of residual stream activations difficult. Sparse dictionary learning methods [Olshausen and Field 1996 Faruqui et al. [013]] Serre
2006 such Sparse Autoencoders (SAEs) have recently gained popularity for interpreting models through this lens by disentangling dense representations into sparsely activating feature sets (Huben al. 2024 Bricken et al. 2023 Gao al. 2025 Kissane et al. 2024 A second hypothesis, Strong
Model Universality [Li et al. 2015] predicts that good models learn the same data representations.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Method
(b) SAE Stitching
(c)
Better SAE Initialization
of Uneppnaee Weneece
nedoo eitched
Layer NA
Layer No
03-
Lab coroor
Model A - B
02
Layer Za +
B -
Layer En
D1
T
7
1t's
FLOPS
Layer la
Layer day
(d) Stitching Arbitrary VectorssFeatures
Trin) xPi
Iajer!
Layer
T:
Model A
B
Figure I: Overview of the main methodologies. (a) We train two affine mappings Tr., concurrently map between the residual streams of two language models A and B. The mappings Tr., are then used to transfer (b) the weights of entire SAEs from A to B, which (c) give better initializations that save compute when training SAEs on B. The approximate "scaling law" for SAE training is shifted to the left when training from the transferred initialization, capturing the intuition that the transferred initialization saves the work of relearning shared features. More generally, the stitches can be used to transfer (d) arbitrary vectors (probes, steering vectors) between the residual stream spaces.
While it is unlikely that the strongest versions of these hypotheses faithfully reflect how these models work (that is, every feature represented as a direction and every feature should be represented universally across good models) [[Egells et al. 2025b Csordás et al. [024 Wei et al.) [022]. there is significant evidence support weaker versions of both hypotheses: linea features have been used to
probe and intervene on model representations (Zou et al. 2023 Panickssery et al., 2023 i.a.] and are the fundamental idea underlying SAEs. One form of evidence for universality model stitching. which learns a transformation mapping the latent spaces of two models [Lenc and Vedaldi 2015 Bansal al., 202T]. If the transformation is sufficiently simple, then is claimed that the models encode information similarly. Other evidence includes recent work in "model diffing" [Lindsey et al., 2024] using crosscoders, variant of SAEs, and transferring SAES between base and chat models Kissane et al. 2024) to study the effects of post-training.
Our work relies on the ansatz that if two models represent enough features as directions in similarly organized spaces, we should be able to transfer interpretable linear features between them. We propose the use of model stitches as the mappings for learned representations between language models. Our contributions (Figure I can be summarized as follows:
We learn stitches to transfer SAES (33) trained on one model to another, which are used to initialize SAE training runs larger models with those trained smaller ones ( ($3.I). allowing for an 50% FLOPs savings. We also demonstrate that stitches can be used to transfer probes and steering vectors ($4)
2. The stitches are affine mappings trained within-family model pairs in Pythia Biderman jtt al. 22023). GPT2 [Radford et al.) 2019]. and Gemma-2 [Team et al. (2224). We find downstream metries are preserved by the mappings, supporting weak universality ($2)
3. We perform fine-gratd fine-grained analysis feature transfer and analyze how well semantic and structural features transfer using the stitch (5..)). We also find SAE features representing well-known universal functional features and note their transfcrability ($5.2).
2 Language Model Stitching
We would like to learn mapping from the residual stream of language model A to model B at some layers, such that computation can start in the layers A and finish in (see Figure As notation, let LM (c) e Rdus denote the representation model LM of input token(s) which has hidden state dimension dim at layer É. Model stitching learns mapping T (the stitch) between two moyel-layer

---

Table 1: Model-layer pairs that we stitch between and the associated downstream next token CE losses on OpenWebText. All activations are taken before the layer's s computation. We primarily stitch from middle residual layers but for specific use cases (e.g., steering) we stitch between 3/4 of the depth. The losses are computed stitching from A to B. B to A, A to B back to A, and B to A back to B. We calculate relative losses to the model that comes last and color code with blue and purple for model A and model B respectively.
Model-Layer Pair
A
B
A +
B
B + A
- B -
B + ++ B
A: 1ythia-70m-dedupeel
3.99
4.21
3.62
3.29
3.60
3.14
B: pythia- 100e--10mmmmem- 4edupeee
(+27%)
(+17%)
(+<1%)
(+4.7%)
A: gpt2-small.6
3.08
3.35
3.07
2.81
3.07
2.77
B: ept2-mediume
(+1 (+11%)
(+9.1%)
(+<1%)
(+1.4%)
A: gemma-2-2ban 2-2b.20
3.28
2.73
2.53
2.66
2.52
2.36
B: 2omma---99b.3
(+39%)
(+8.3%)
(+<1%)
(+13%)
pairs (A, and (B, Informally, we would like [T A (t) = B²=(t) where o function composition. The existence of such mapping T relies on at least weak universality, especially if T preserves the hypothesis classes of model A and B. We are interested in the particular case when A and B are decoder-only language models from the same family (trained on the same data) but dB 2 dA. In this setup (visualized in Figure we will consider the two stitching mappings Tr, which maps "up" from A to B, and Ti. which maps "down" from B to A. Furthermore, we will assume that both Ty and Tj are affine transformations:
Ty :dd + Rds,
hA + hAPt + by:
(1)
T :dd + Rda, hg + hB + b4,
(2)
where ha and hy are activations from A and B, P. Rdaxda by Rda, Pl Rduxda and b. Rd Ideally, would like T(t..) to be faithful to the downstream objective (i.e., language modeling). In practice, training directly on the next-token prediction objective would involve backpropagating gradients the back halves of A and B, which is unnecessarily expensive. We find that the reconstruction mean squared error (MSE) is a sufficient training objective to align the models. Despite the dimensionality gap, also still desire aalmost"-invertionn
transformations i.e. Tr o Ti should be close to an identity and vice versa. Therefore, we also introduce a regularization penalty that encourages Tr and Ti to invert each other with relative strength a, finding that the penalty improves fidelity (ablation in $A.2) and has further motivation when we consider transferring SAES in 3.1 and B.I Formally, we train Tr and TL concurrently via the loss function on a token
C(I) = MSE ([T o A (1), (0) + MSE ([T o Be (1),
+ aMSE ([T, [ o (t), (t)) + aMSE ([T o T, B' (t), (t),
(3)
We always learn the mappings from fixed residual stream layer of model A to the layer in model B that maximizes the average correlation from Singular Vector Canonical Correlation Analysis (SVCCA) across small sample of model activations [Raghu al. 2017]] (more details in IA.I).
Using this procedure, the model-layer pairs that we stitch between are shown in Table
Downstream Performance. In Table [ we compute the downstream fidelity of mapping from A to B. mapping from B to A, and mapping in both inverse directions. First, we observe that next-token prediction performance is close to ground truth, supporting weak universality hypothesis that models trained on the same data share features despite differing scales. However, we also find that downstream performance is consistently bottlenecked by the worse (generally smaller) model A. This result suggests that the gap between how A and B use their latent spaces prevents linear stitching between A and B from recovering the full performance of B. Finally, the inverse operation can be nearly lossless despite the intrinsic dimension mismatch prohibiting perfect invertibility.
3

---

3 Transferring SAEs
Sparse autoencoders (SAEs) have recently resurfaced in interpretability as a method of decomposing residual stream representations x E IR" from language model LM at layer into sparse nonnegative feature activations f(x; 0) E R-M with M > d and = (We; be, Wa, ba) of the encoder matrix,
encoder bias, decoder matrix, and decoder bias respectively that specify the affine transformations in the forward pass, given as
= (x + be),
(4)
M
SAE(2; 0) = f(a; 0)Wa + ba Σ fi(x;))aas + ba-
(5)
The columns of the encoder matrix are the encoding/detection directions of the features. The rows of the decoder matrix the decoding/representation directions of the features that the SAE decomposes into. Feature activates on token t if fi [LM' (t); > O. SAES are trained to reconstruct while
having sparse activations via an objective function consisting of MSE between I and (2; 0) and, depending on the architecture, an L, regularization (usually with p = 1) on fx; 0) with strength controlled via a parameter À > O, respectively. We use pretrained SAEs with TopK activation [Gao al., 20255 for Pythia and Jump SAEs for Gemma [Lieberum al. 2024 and train TopK
SAEs from scratch on the GPT2 models using SAELens [Bloom et al. 2024
3.1 Efficient SAE Transfer
Equipped with the trained stitch transformations, we make a key observation that an SAE with parameters A = (We. be, Wa, ba) trained on layer f in A can be transferred to layer fg in B. This procedure is illustrated in Figure 1 specifically, for a latent hg = B'a (1) in model B, we (1) transfer down using T1. (2) apply the original SAE parameterized by 0. and (3) transfer back up using 74. Because all the transformations are affine, this forward computation exactly specifies a new SAE on model B parameterized by
0' = U.. Wa.b (P, b, + be,
(6)
This relationship is derived in detail in BB.1. The explicit formulas for f' also provide geometric intuition for the parameters of T(t.))- Indeed, P(t.4) can be viewed as directly transforming the linear feature spaces of A and B as they impact the feature vectors through W'' and W/. On the other hand, ((t.)) are responsible for adjusting the "position" of the feature decomposition as they only alter the biases K/ and b/-
We report evaluations of the transferred SAEs in $C which are better than random but certainly worse than a fully trained SAE on B (e.g.. a Fraction Unexplained Variance [FUV] of 0.21 to 0.42 in Gemma 2B to 9B). The metries serve evidence of weak universality in the midst of a representational gap between A and B. As a partial explanation, we note that the weight matrices of the transferred SAE are still rank S dA < dB i.e., the SAE is only detecting and writing features along a rank d subspace embedded within the rank dg ambient embedding space of model B, contributing to their lackluster performance.
Training from Stitch is More Efficient. After obtaining transferred SAE on B, we investigate how much continued training is required to match the performance an SAE fully trained on model B. This question leads the transferred SAE can be used to initialize a new SAE in model B, "saving" the compute of having to relearn features that transfer well. Concretely, consider the problem of training one SAE for two models of different sizes in the same family [Lieberum et al. 2024]. The straightforward approach is separately train SAE from scratch each model. However, find that given fully trained SAE on the small model, training a stitch and using the transferred small model SAE
key application of the stitch and implicitly weak universality:
https: : //t:://giit git.ub..hh... com conn/Eleutherr//ppasss...
Gao et al. 2025 releases pretrained Topk SAES for GPT2. However, since these SAES use LayerNorm normalization scheme but our stitches are trained on unnormalized residual stream activations, the SAE transfer computation does not extend easily.

---

(a) Compute Savings (accounting for stitch)
(b) Cosine Similarity to Initialization
0.75
5000
random
stitched
Relative Differncce
0.50
4000
0.25
0.00
-0.25
4k, k- 64
2000
8k, k-64
16k. k=64
-0.75
32K, km64
1000
65k, k-64
-1.00
0.70
0.75
0.80
0.85
0.90
0.0
0.2
0.4
0.6
0.8
1.0
Exploined Variance Threshold
cosine similarity initialization
Figure 2: (a) In the Pythia model pair, transferred SAE initialization adjusted by the stitch FLOP count reaches explained variance thresholds in less FLOPs. For thresholds around 90% explained variance, the moving average of explained variance of the SAE hits the threshold in around 30-50% less FLOPs. (b) Features when trained from stitched initialization have higher cosine similarity to their initial state than random initialization in the M 32768 runs. Dead features are removed from consideration for clarity around 1.0.
in the initialization scheme of the larger SAE allows for training of a comparable SAE to training from scratch but in cheaper total FLoating point OPerations (FLOPs) budget. We support this claim by training TopK SAEs on the Pythia model pair and repeat with GPT2 in $D.I] We estimate FLOP counts based on the assumption that activations are cached prior to training both the stitch and the SAEs. All SAEs are trained using SAELens [Bloom et al. 2024 and other training and FLOP estimation details are included in $D
Generally, we find that the cost of training the stitch on eached activations much cheaper compared to training an SAE. Despite the cheap cost of the stitch, it allows to stop the training of the SAE earlier when we use the transferred SAE as initialization. Figure displays the relative FLOPs-to-hiis threshold savings of training from the transfer SAE initialization (accounting for the cost of training the stitch) vs. training from scratch initialization. More specifically, for the two initializations, we compare the number of FLOPs needed to get an SAE Pythia-160m with explained variance above a particular threshold. At high explained variance values, we collect savings of between 30%-50% less FLOPs.
In Figure Th we repeat the experiment for various latent sizes create an approximate scaling law plot inspired by Gao et al. (2025]. We emphasize our fit is only an approximation meant for visualization as we estimate the law simple linear regression in log-log space instead of a proper scaling law (details and coefficients in $D.2) we ignore the irreducible loss term because we are comparing the two laws against each other and stably fitting the term requires larger latent sizes to saturate the curve. Nonetheless, when training from stitch initialization, the fitted law is noticeably shifted to the left compared to random initialization, indicating that stitch initialization reaches levels of reconstruction loss relatively faster consistently across the given latent sizes. For small latent sizes, such a procedure is less worth the additional compute of training the stitch, as adding in the compute required to train the stitch would result in the procedure becoming more expensive than just training from scratch. To confirm our intuition about the initialization, we also compute cosine similarities of decoder vectors at the final checkpoint to their original direction initialization. In Figure Pp, observe that the features trained from stitch initialization rotate less in aggregate than training from random, indicating the initialization is indeed placing features closer to their final values.
Downstream Applications
Despite SAEs not transferring without additional training, the mappings T, and Tr are fundamentally general transformations that linearly relate the residual streams of two models. In this section, we examine the application of the stitch to transferring probes and steering vectors, which target specific vectors in the residual stream (Figure (t). When applicable, following the intuition gained from Equation 6 we transfer feature vectors using just Pt, and do not include the bias.
5

---

(a) Probing
(b) Language Steering
(c) Transfer Gap
Probing Transfer
gemma-2-20
gemma--299b
9b->2b
0.9
1.0
dence transfor 150m
dense 150m
160m
respomeen in tarqut lanquage
0.5
0.5
no. steer
steer
mequency
0.7-
0.8
rendom
0.3
0.3
0.00
0.25
2b:>-bb
0.50
0.75
1.00
frequercy
0.6
0.1
0.2
10
30
0.00
0.25
0.50
0.75
1.00
probing
clipped relative transtor gop
Figure 3: (a) Evaluations of transferred probes stitching from pythia pythha-70m--ai 70m- 70m-deduped to pythia- lisia--1de 160m- deduped averaged over binary classification datasets. If the probe is retrained (orange), we almost recover ground truth performance (blue) across all probing ks and most datasets. Even if probe is not retrained, in most datasets we are able to probe significantly better than random (green). When directly probing on the residual stream, we find that transferring probe trained on 70m-deduped (dotted) reaches similar accuracy to probe trained on 160m- deduped (dashed) without retraining. (b) Response language steering vectors are able to be transferred between gemma-2-2b. and gemma-2- 9b.33. From left to right, we chart the % of responses in the target language for no steering, ground truth steering, and steering using a transferred vector averaged over all languages L in EuroParl and prompts, (c) The relative transfer gap distribution bimodal with concentrations at 0 and 1, implying the transfer steering works well for some languages but poorly for others.
Probing. To begin, we consider the sparse probing task [Gurnee al. 2023 Kantamneni et al. 2025 Karvonen et al. 2025) specifically, using certain subset of l SAE features as probing features for a binary concept. The 1 features are selected from the features found by a SAE and are selected via a simple max mean activation difference procedure: feature activations are collected over training samples, averaged over non-padee paddiid tokens, and the class- wise difference means computed. The k features with largest difference in means are used as the probing features. Formally, let D and D_ be the set of activations at a layer on positive and negative training examples respectively. Then, we compute the set F& of indices of the k features as
Fk TopK, (EEn, ~D, [f((+4;0]] E. -D [f((=-0)))-
In order to probe model B, can transfer the SAE from A to B and the activations from the same i features in F4 to probe B. This procedure is equivalent to computing (T((((()) (4)), then applying probe on F& 's activations on the stitched residual stream. Notably, the probe we use can either be completely reused as a pure evaluation of the stitch (i.e. the same probe that we trained when probing model A) retrained to extract the full probing capability of the transferred features.
In Figure 3h, we present the test accuracies for probes averaged across 8 SAEBench datasets for k = 21,2, 2 5, 10, 20, 50). The results for all datasets presented separately is in $E We compare against baselines of selecting features from a ground truth SAE, random features, and dense representations in model B. On average, are able to probe successfully using the transferred features without retraining the probe in particular, we do not collect activations from model B outside of the data used to train the stitch. These results also hold with dense probes shown in the dashed and dotted lines: the dashed line is the skyline of training a probe on the dense representation of pythia- 1iiiia-166mp 160m-deduped, whereas the dotted line is probe trained the dense representation of pythia-70m-dedupee zero-shot applied to down-stitched residual streams from pythia- wethia-160m-dede 160m-dedueed.
Eicering/nnstrcttonn Following. In this section, we explore transferring steering vectors in simple task: altering the response language from en to a target language T, in PuroParl [Koehn 2005). We compute steering vector over the first 100 paired examples in each en- dataset by unit- normanorg the difference in mean activation on L tokens vs. English tokens [Panickssery et al. 2023) To steer, we clamp the component in the feature's 's direction to Σ, where we set 2 to the mean of (LM' (1), v) over the tokens in L (i.e., positive examples of the desired behavior). Explicitly, if is steering vector and ||1|2 = 1, then we steer by modifying the hidden state at token as
== 2 - (LM' (t), v),
(8)
LM = LM (t) + ev.
(9)

---

V is transferred by computing u (P.)).). renormalizing, and recomputing & in the new model. We evaluate over subset of 163 prompts from the IFEval dataset with the instructions stripped and aggregate the proportion of responses in the target language [Zhou et al., 2023 Stolfo et al. [025 Importantly, none of the prompts explicitly request the response to be in the target language.
We use the Gemma model pair and present the results in Figure pp, where we plot the % of responses in the target language depending on the type of intervention: no steering, steering with vector learned from the current model, and steering with a transferred vector learned from the other model. Averaged over L, we find that the transferred steering vector identifies a direction that successfully steers the model toward responding in the target language without explicit prompt instruction to do so. We break down the overall accuracy into individual language pairs (en, L) and find the transferred steering vector works well for some languages but not for others by defining a clipped relative transfer gap as the ratio of transfer steering performance to ground truth steering performance clipped to [0, 1] for visualization (Figure 3 We also note positive correlation between language frequency and steering transfer effectiveness (S...). Although language steering appears to work somewhat well, we find that steering for general format instruction following has weaker results ($F2).
5 Feature Analysis
Finally, we can gain intuition into specifically what features transfer well under the stitch. To this end, we consider a correlation-based metric that approximates activation and downstream effect similarity Bricken et al. 20023]. In particular, we compute the Pearson correlation between attribution scores induced by the logit weights and the model's next token predictions. Let Corr(:)) :) denote the Pearson correlation operator over data x and let i and vy denote the logit weights given by unembedding the ith decoder vectors from Wa and W respectively. The attribution correlation is given by
Corr,, (fi (AA* P.A.IIAA fs (B (t+): (LB)
(10)
where the correlation is computed over a set of tokens t, E D and 1 is the next token. Intuitively. this score approximates the feature's 's importance in generating the next token prediction by taking the product of the activation and the downstream logit weight. We plot the histograms of attribution correlation scores against random SAES in G
5.1 Semantic vs. Structural features
The first question can pose is whether different types of features transferred relatively differently according to the metric in Equation [0] The simplest classes of features are semantic features vs. structural/syntaciic features. We outline cheap but general experiment to generate this separation, visualized in Figure A with example classifications in Table g We first construct a synthetic dataset consisting of augmented versions of sentences drawn from large text corpus. For each randomly drawn sentence, we use another LLM to generate k augmented versions of the sentence with a prompt specifically instructing the LLM to ablate the semantic content of the sentence while maintaining the structure (H..). We then feed the original sentence and all augmented versions (1 + k sentences in total) into the original language model and collect which SAE features activate on any token in the prompts. Features that activate consistently across the augmented versions of the same prompt are classified as structural, whereas all other features are classified as semantic. This classification is stitch-agrostic- we classify the features purely based on activations from the original SAE As a concrete example, suppose we began with the prompt "I bought apples, bananas, and pears." We generate an augmented version of this prompt that preserves the structure of the sentence but ablates semantic content e.g. "John fostered cats, dogs, and fish." Consider a feature that activates on the last comma of list: this feature would activate in both prompts, resulting in classification as structural feature. However, a feature that activates on food or animals would only activate on one of the prompts or the other, resulting in classification as a semantic feature.
We plot the histograms of structural vs. semantic features with respect to the attribution correlation in Figure for the GPT2 stitch, removing all dead features from consideration. Our main observation that structural features tend to consistently transfer better, but semantic features are more polarized: they generally transfer well or they do not. Directly, this result implies that the stitch strongly transfers a group of semantic features, mostly transfers structural features, and leaves behind some semantic features. This experiment also supports that new dimensions in larger models could be
7

---

(a) Semmatic/Structural Classification Method
(b) Relation to Transferabiltty
semantic
2.0
To d, andiqq
Union
Active
K.yy, and
1.5
denstt
Structural
1.0
Semantic
Active
Structural
0.5-
0.0
Figure 4: (a) An overview of the feature analysis pipeline for simple example where augmentations are generated. Structural features activate on prompts (intersection) whereas semantic features only activate on some but not all prompts. (b) The semantic/structurall classification reveals a divergence in the attribution correlation transferability metric for non-dead features. We plot the densities of both categories separately and relative density above. Structural features transfer more consistently but semantic features are more polarized (dominate the upper and lower percentiles).
mostly allocated to developing increasingly specific semantic directions whereas the core language modeling spaces (which are more structural) remain similar.
5.2 Functional Features
Some of the structural features are related to known examples of universal features: in particular, entropy features and attention deactivation features [Gurnee et al. 2024). We find that our stitch preserves the functional role of these features.
Entropy Features. Entropy features are characterized by large norm and high composition with the effective null space (we define the bottom 2% of singular values) of the unembedding matrix [Stolfo et al. 2024]. In gpt2-small, we find two features from an SAE that have these properties. Furthermore, we find that even after stitching, these properties still hold true (Figure a, showing that the stitch does indeed transfer the functional role of these features.
Attention Deactivation Features. In order to identify attention deactivation neurons, heuristic score and path patching techniques from Gurnee et al. [2024]. In particular, to identify potential candidates for composition of a feature with downstream attention head, we compute the heuristic score between all features and downstream attention heads. Then, to test a relationship between feature and attention head, path zero-ablate the feature's contribution to target attention head's input at the current token over a small token set and measure the change in the attention pattern the <bos> token. We identify attention deactivation feature in the gpt2-small SAE and find that it retains its role as a deactivation feature for a downstream head post-transfer gpt2-medium (Figure 5p). Attention sinks have also been noted have large components the effective null space [Cancedda] 2024] and we find that is preserved as well (original: 0.27, stitch: 0.77). Somewhat mysteriously, also observe the same feature acting as attention activation feature in gpt2-medium after transfer (Figure 16]
the
Related Work
Language Model Representations. Language models have empirically been observed learn vector representations of familiar human interpretable concepts like function application. truthfulness, factual knowledge, and refusal [Todd et al. 2024 Gurnee and Tegmark 2024 Zou al. 2023 Arditi et al. 2024]. Linear directions can also be useful for model editing, steering, and concept erasure
[ al. 2023 [Iharco et al., 2023 Panickssery et al. 2023 Belrose et al.) 2023). (Huben al.)
2024| Bricken al. 2023 reintroduced SAEs as a technique for decomposing residual streams

---

Entropy Features
(b) Attention (Dejactiation
gpr2-amal
1.4
1p12-mddum
gezeomal (8.10)
entray
9.8
0.18
62
12
240
9.0
-310
actication
Figure 5: (a) Two entropy SAE features remain both large max activation and compose highly with the effective null space (bottom 2% of singular values) before and after transfer. For clarity we take randomly sampled subset of 2000 features. (b) Attention pattern on BOS token on path patching experiment. After transfer, we are still able to find a head such that zero ablating the contribution of the feature has results in decreased attention on the BOS token. We only plot tokens in which the original feature activates in gpt2-small.
which are hypothesized to geometrically superpose features (Elhage et al.) 2022 Arora et al. 2018
Olah et al. 2020). SAEs have been somewhat successfully applied to various downstream tasks
like steering, probing, circuit analysis, and discovering interesting features (Karvonen et al. 2025
Ameisen et al. 2025 Ferrando al., 2025 Marks et al., 2025). Training SAEs has seen significant
research effort, especially in architectural design and activation functions [Rajamanoharan et al.
2024
Gao et al. 2025 Bussmann al., 2025]. Recently, however, limitations of SAEs have come
into light and their use cases have become better understood [Paulo and Belrose 2025 Chanin et al.
2025
Kantamneni et al. 2025 Engels et al., 2024]. In particular, consistent assumption across
many aforementioned works is the linearity of features [Mikolov et al., 2013 Park et al. 2024 Elhage
ct al.
2022]. However, recent work has demonstrated the existence of natural non features
Engels et al. 20256 Csordás et al. 2024).
Universal Representations Across LMs. The intuition that well- well-generalizing models might all
have similar representations has existed for quite some time (Li et al. 2015 Huh al. 2024]. Model
universality has been studied from the perspective of model stitching Lenc and Vedaldi [015 Bansal
ct al.
2021 (ha et al. [025 Csordás et al. (202). representation similarity measures like SVCCA and
CKA
Raghu et al. [017 Kornblith et al. 2019 Barannikov et al., [021] SAES and variants [[ (Lindsey
et al.
2024|
Lan al. [024]) and cetttur//neuro/wweeighe level analysis Gurnee al. [2024] Stolfo
et al.
2024 It has also been leveraged in adversarial attack literature [Zou et al. 2023]. Recently,
transfer of steering vectors between models of different sizes using linear mappings between the final
residual streams has also been explored in binary steering tasks and safety applications [Lee et al.
2025
Oozeer et al. 2025
7
Conclusion
Findings. We find that we can faithfully stitch between language models of different sizes of the same family, which can be leveraged to transfer SAEs, probes, and steering vectors. The SAEs can be trained convergence faster than training from scratch, demonstrating application that uses weak universality to benefit SAE training. Probes and steering vectors can also be transferred with no additional training in specific cases. Finally, we perform case study on GPT2 and find differences between how the stitch transfers semantic and structural features and discover that specific functional features have their roles preserved.
Limitations and Future Work. First, we only train the stitches on general internet text data between models in the same family with the same tokenizer. Natural follow ups are verifying the findings in cross-family setting and finetuning the stitch to reconstruct unnaturally occurring special tokens or shat-templated data. The scaling laws are also theoretically incomplete because we lacked high-compute regime data to fit the irreducible loss. However, the transfer procedure is generalizable and other applications of the stitches to save compute or distill capabilities should be explored. For example, stitching could be used to transfer linear weight updates like LoRAs (Hu et al. (2022 which play nicely with the affine stitches and have already been established as similar to steering behavior [Engels et al. 2025a].

---

The semantic/structural classification is also imperfect- we only gain one type of distinction which is also sometimes noisy because the ablations are generated by another language model and transfer- ability is purely correlational. A closer analysis of sensitivity of the stitching and semantic/structural methodology to different layers could also produce new insights into feature distributions. Finally, the probing and steering experiments deserve to be expanded upon to examine robustness as our experiments are still limited to particular tasks and settings.
Acknowledgments
We would like to thank Michael Lepori, Jake Russin, and other members of the LUNAR Lab at Brown University for feedback at various stages of this project. We are also grateful to Neel Nanda for his valuable imput during the early stages. AC would also like thank Thomas Chang, Patrick Peng, and the CSCI2222 course at Brown for fruitful discussions about this work. AS acknowledges the support of armasuisse Science and Technology through CYD Doctoral Fellowship. This project was supported in part by a Young Faculty Award from the Defense Advanced Research Projects Agency Grant #D24AP00261. Ellie Pavlick is a paid consultant for Google Deepmind. The content of this article does not necessarily reflect that of the US Government or of Google and no official endorsement of this work should be inferred.
References
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro. David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall. Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https httl Andy Arditi, Osear Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In The Thirty-eighth Anmual Conference Neural Information Processing Systems, 2024. URL httph:t/p/// /ttpps://oppnreview net/ Foorm?id==pH3AQQNER
//transformer- circuits. pubp 2025/
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018.
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances neural information processing systems, 34:225-236, 2021.
Serguei Barannikov, lyy Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topol- ogy divergence: A method for comparing neural network representations. arXiv preprini arXiv:2201.00058, 2021.
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirth- Conference on Neural Information Processing Systems, 2023. URL http::::///peneeeeeeeeee///ttree
awIpKpиTwF
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
Joseph Bloom, Curt Tigges, Anthony Duong, and David Chanin. Saelens. https://githutt com/ JJboonAus/SAELens 2024.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph. Zac siifield-Dodds. Alex Tamkin, Karina
10

---

Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity; Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. features/index.hmm
Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. arXiv preprint arXiv:2503.17547 2025.
Nicola Cancedda. Spectral filters, dark signals, and attention sinks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Anmual Meeting of the Association for Computationnal Linguisties (Volume Long Papers), pages 4792-4808. Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 10018663/11/22244cccoo........ URL httppp:://acclattollooooorr////a http:///2222.aaalloooa.a. acl-long. 263/
David Chanin, James Silken--mith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Isaac Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders, 2025. URL
https:///ppareriiee.ntt/ net htt.://toorrmmm.............o
Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non con-linear representations. In The 7th BlackboxNLP Workshop, 2024. URL http:ttt////teereiiiee
Róbert Csordás, Christopher D Manning, and Christopher Potts. Do language models use their depth efficiently? arXiv preprint arXiv:2505. 13898, 2025.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Haddell--oddss, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209, 10652, 2022.
Josh Engels, Neel Nanda, and Senthooran Rajamanoharan. Interim research report: Mechanisms of awareness. Alignment Forum, 2025a. URL URL http:://wpw: http://wwwwwlllggnnnfffr.. org p
ser. -ates-oo--aaaaeeea--eaeee awareness
Joshua Engels, Logan Riggs, and Max Tegmark. Decomposing the dark matter of sparse autoencoders. arXiv preprint arXiv:2410. 10244100.46670 /4670, 2024.
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are mindinnnnionaalyy linear. In The Thirteenth International Conference on Learning Representations, 2025b. URL httthttttt//perrr/t// four::///opppeerrrreeeeeooooooon F 7d=-6344AMMMb
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A Smith. Sparse overcomplete word vector representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1491-1500, 2015.
Javier Ferrando, Oscar Balcells Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do i know this entity? knowledge awareness and hallucinations in language models. In The Thirteenth International Conference on Learning Representations, 2025. URL http::tt//o/o //opprreee//eeeeieee net/ fforu?id==WRQFF1, jj2q
Leo Gao, Tom Dupre Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2025. URL URLhttts:://///// /tttps://operrevieet net/ [oormmPd==tccztt2ZNN
Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URL. htt.tttt///opeeeeiieeeee net/
Barum?id=jExxbmvv
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Trunsactions on Machine Learning Research, 2023. ISSN 2835-8856. URI. httpp::///ppereviee..eeeeeeeeeee wtwp:ewwwwwweeeeeuttmmmm..ee
JYs1R9IMJr

---

Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaci Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons GPT2 language models. Transactions on Machine Learning Research, 2024, ISSN 2835-8856. URL
1d=2eI104QZ81
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International
Conference Learning Representations, 2022. URL http:ttt///ppeeeeeee.... net/ net/ffrummii nZZVKeeFY19
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. F776bRSLeKK]
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07993 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL httpp:t::///eent eview 6tt0ktt jrj
Rishi Jha, Collin Zhang, Vitaly Shmatikov, and John X Morris. Harnessing the universal geometry of embeddings. arXiv preprint arXiv:2505.1254,, 2025.
Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? study in sparse probing. arXiv preprint arXiv:2502.1665 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint (:XXi:2001.08360 2020.
Adam Karvonen, Can Rager, Johnnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability. 2025. /ttt/tttt////xiii/ www.oogg///aaaaeeeeoo aba.aaa/25200 2
Connor Kissane, Robert Krzyzanowski. Arthur Conmy, and Neel Nanda. Alignment Forum,
Saes (usually) transfer between base and chat models. 2024.
saes URL. https://www http://www.lliimmennffffro org ottp/ppstt//f/////xprdddd.
-ww.baawwwee------d-d--------a-- and- and-chaatmooell.
Philipp Kochn. Europarl: parallel corpus for statistical machine translation. Proceedings of
Machine Translation Summit X: Papers, pages 79 86, Phuket, Thailand, September 13-15 2005.
URL patttss//aclanttoo//t org/ .wtsummit--apprrs..
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519- 3529. PMLR, 2019.
Michael Lan, Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez. Sparse autoencoders reveal universal feature spaces across large language models. arXiv preprint arXiv:2410.06981 2024.
Andrew Lee, Melanie Weber, Fernanda Viégas, and Martin Wattenberg. Shared global and local geometry of language model embeddings. arXiv preprint arXhv:2503.21073. 2025.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence, Proceedings the IEEE conference on computer vision and pattern recognition, pages 991-999, 2015.
12

---

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In Thirty-seventi Conference on Neural Information Processing Systems, 2023. URI. httptttt/p//peeeee/eeeeeee /ttp:///pp/eeeee.e///// aLLuYpn83y
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar, editors, Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015, volume 44 of Proceedings of Machine Learning Research, pages 196-212, Montreal, Canada, 11 Dec 2015. PMLR. URL https:://rooeeee
mlr. wrw.prrsss/eeee http:////eA/1115coveeeeeeeee htmmeeeeeetttttmmmmll.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Commy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemmaa scope: Open sparse autocncoders everywhere all at once gemma 2. arXiv preprint vrXiv:2408.05147 2024.
Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Baston, and Chris Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024.
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https httororrrunnnd1oe0110000
//openreview
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff, editors, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pages 746-751, Atlanta, Georgia, June 2013. Association for Computational Linguisties. URL httpt///////cllltthooooogg orwg
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction to circuits. Distill, 2020. doi: 10.239 1/ddistill.00024..000 https://distill http://sisillllli2/0/////r/////m
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 (6583))6607666 1996.
Narmeen Oozeer, Dhruv Nathawani, Nirmalendu Prakash, Michael Lan, Abir Harrasse, and Amirali Abdullah. Activation space interventions can be transferred between large language models. arXiv preprint arXiv:2503.04422 2025.
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong. Evan Hubinger, and Alexander Matt Turner. Steering Ilama via contrastive activation addition. arXiv preprint 11.66.231..06681, 2023.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, 2024. URL https::/openreviiet wwt.:eteee...ooorr//--oooooLaaa
Gonçalo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different
features. arXiv preprint arXiv:2501. 16615, 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 14435, 2024.
Naomi Saphra and Sarah Wiegreffe. Mechanistic? arXiv preprint arXiv:2410.0900 2024.
13

---

Thomas Serre. Learning dictionary of shape-components in visual cortex: comparison with neurons, humans and machines, 2006.
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill. Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496 2025.
Alessandro Stolfo, Ben Peng Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models. In The Thirty-eigtth Anmual Conference on Neural Information Processing Systems, 2024. URL htttss://opexxevieet net/ foormmid=0og? nmvDbe
Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. Improving eestruction-followwee in language models through activation steering. In The Thirteenth International Conference on Learning Representations, 2025. URL https:://pperreiieet net/ -ozmmddddozzddtttt
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001 18, 2024.
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference Learning Representations, 2024. URL
net. foree///oommm/ee ii=Ayiitywwawaaa
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large lan- guage models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreevert wwwp:eevviiveeeefoteeeeeee www. http:/oorrmm1ddyyysss55zaz yzkSU5zdwD Survey Certification.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follovingg evaluation for large language models, 2023. URL
07910
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint XxXx:2307..5043, 2023.
14

---

A Stitch Training Details
The stitch training methodology is quite minimal. For the training dataset, we collect activations the desired layers in both models over the first 180k samples of Open WebText with a context size of 512 (128 for Gemma due to compute constraints) and evaluate over the next 1k samples with the same context size. We mask out all special tokens. The stitches themselves are two separate Linear layers (with biases initialized to 0) that map between the dimensions of A and B. We use the Adam optimizer with a learning rate of 1e-4 and clip gradient norms to 1.0. We found minimal sensitivity learning rate schedule, so we just use a cosine annealing decay, and found that 2 epochs is sufficient for convergence (though even 1 is probably enough).
A.1 Layer Selection
Suppose we have fixed some layer A in model A that would like to stitch from. We determine the layer f by computing a Singular Vector Canonical Correlation Analysis (SVCCA) over a small sample of tokens between the residual stream activations at Ea and the candidate layer lg and taking the argmax. We use SVCCA because it is cheap to over a small set of activations while being directly related to how linearly related the two sets of activations accounting for noise in lower variance directions. We visualize all pairwise SVCCA values computed in Figure 6
87.
1012 km
1012
Nomvobal and
1002 pm
855
0012
0.5
(a) 70t pythia-160m 160m
(b) gpt2-small to mee2-medium
14
(c) gemma-2-2h to gemma-2-9h
Figure 6: Computed SVCCA scores all pairwise layers in the model pairs stitch between. We end arbitrarily selecting a layer in the smaller model and choosing the layer in B with the highest SVCCA coefficient.
A.2
Inversion Ablation
See Figure 7
B Transferring Details
Assume, we have two autoregressive language models A and B with latent embedding dimensions d and dg, respectively. Furthermore, assume there exist affine transformations T (P,, bT)
15

---

Pythla-70m-deduped.. pythha--66m--eeeeeeee
1.0
10.0
org
Recordtrution CE loss
0rkg
0
Figure 7: Strength of inversion ablation experiment. For various values of a we plot downstream next token CE loss for different transformations that should all be low stitching from A to B and from B A, inversions, and the delta loss of an SAE transferred up from 1.0 strikes good balance we use it for all experiments.
We can see that
Rdaxda x Rd and T. (Pa, Rdaxda x Rdd as in Equation that relate the residual streams of A and B fixed layers in both models.
B.1 SAE Methodology
Recall that SAE Rd + Rd defined on model can be written using the following computation:
SAE(:;; 8) (2W + Wa + ba.
(11)
parameterized by the 4-tuple 0 (We, e, be, Wa, ba) and activation function
The insight is that we can think of transferring a sparse autoencoder from model to model as capturing the following computation. For every latent in model B hu E Rda;
1. Stitch hp A using Ti.
2. Apply the SAE on the stitched latent.
3. Stitch the reconstruction back B using T
turns out this computation can be collapsed into a reparameterized SAE on model B because all of the transformations are affine.
(T+ o SAE - (h)))) = SAE(hn bb;
(o((hBP) + b, + be) + ba) P. by
= =ohh|[PWWl +
bbl)[ [b. Wb +
[baPt + by]
SAE(Hp;0').
where
0' (P. We, W. be. W.P.. byP + bb)
(12)
is another sparse autoencoder, but now mapping from IR - RRda Remark: one slight caveat is that SAE's feature matrices are still only rank d.a
Observe that the matrices P. and P1 do the grunt work in manipulating the actual feature spaces, whereas the biases b, and bj just relocating the "position" of the residual stream in space and do not adjust the feature spaces themselves.
We also notice that assuming SAE(2; (z; 0) R the reconstruction loss
|AAE hs; 0') -
(13)
low when T₂ and Tj invert each other (as expected). This inspires the inclusion the additional inversion penalty in training the transformations T and Tj (Equation 3).
16

---

Table Summary metrics of transferred SAEs (no training). Results are displayed as original (A)
transfer (B). Lo evaluations are dependent on architecture the Pythia and GPT2 SAES are Top-K and Gemma are JumpReI.U.
Metric
pythia-70m.. pythia- 160m.4 gpt2-small.6/ gp22-medium.10 gemma-2-2h.22 gemma-2-9h.3.
Lo
16.0/ / 16.0
32.0/ 32.0
77.2/ 74.7
0.17 0.55
0.11 00.39
0.21/0.45 /0.42
Delta Loss Dead Features
0.36 2.16
0.96
0,50 0.85
5.2% 2.9%
<1% / 1%
1.5% 3.5%
B.2 Probes and Steering Vectors
To transfer linear probe from A to B, we simply apply the probe on the down-stitched residual stream (BBB((()), the probe linear and has normal direction w, equivalent to using the vector wP in B. Steering vectors are just transferred directly using Pf without the bias.
C Zero-Shot SAE Evaluations
As basic evaluations of how well the feature spaces match, compute our zero shot transferred SAEs and compare them against the original SAEs. We report Lo. Fraction of Unexplained Variance (FUV), Delta loss, and dead features % in Table Importantly, we note that the transferred SAEs are not perfect one explanation of why is that the weight matrices of the transferred SAE (Equation G are still rank S dA.
metrics of
D SAE Training and FLOP Estimation Details
All SAEs we train are TopK SAEs trained using SAELens on unnormalized residual stream activations. We train SAEs with latent sizes 4096, 8192, 16384, 32768, and 65536. We abide by the following practices:
1. We normalize the decoder vectors unit norm each iteration.
2. When randomly initializing, initialize the decoder and encoder transposes of each other.
3. We use constant learning rate schedule and just use 0.0001 as the learning rate.
4. We do not use auxiliary loss for ease of FLOPs estimation (discussed below).
SAELens includes the auxiliary loss for TopK training by default. In order to work around disabling we the fead_feature___ wwwddwwwwwwww parameter an arbitrarily large number that dead feature computations are never run during training.
We estimate FLOP counts of training the stitches and SAEs with cached activations by simply computing the FLOPs of one forward and backward pass for the desired module on batch of dummy data inside of the context manager torch. utils lllo_counterr FlopCounterMode. We then scale by the number of training iterations. Example FLOP estimates for a set of SAEs is shown in As minor fact, we do find that with larger latent sizes, the number of dead features in the
Table FLOPs estimates for various important procedures with caching activations. All SAEs are trained with sparsity k: == 64 and width 32k. A full run is 4B tokens (120k iterations) for the SAEs and 200M (36k iterations) tokens for the stitch. The Pythia-70m SAE trained post layer and Pythia- 160m SAEs are trained post layer Procedure
FLOPs (w/ caching)
Pythia-70m SAE (Scratch)
8.2 10
Pythia- 16m SAE (Scratch Transfer)
1.2 10
Stitching Layer
1.4 10
original SAE (on A) increases since not using the auxiliary loss. Since dead features tend to
17

---

tandom sütched
2.0
Froction unopisined Valence
0.5
235
Latonts
0.3
02-
21v
10H
101
10
1027
-10
FLOPS
Explained
(b)
Figure 8: (a) SAE "scaling law" for transfer results on GPT2 models with k = 32. (b) Relative FLOPS Difference between the initializations.
be inherited in the stitched SAE initialization, we end with more dead features when training from the stitched SAE initialization. This could be remediated by reincluding the auxiliary loss (which would require careful FLOPs estimation), but we did not test this claim.
D.1
GPT2 SAE Transfer
In Figure 8 we replicate the same plots as Figure I but for the GPT2 series models, with SAEs trained on 2B tokens and l 32.
D.2 Scaling Law Fit
We fit the scaling law by sampling the frontier MSE achieved at various FLOPs thresholds and fitting a linear model on the log-log scale (as opposed to fitting an irreducible loss term unfortunately we do not have enough high-FLOPs points due to compute constraints to fit complete scaling law stably). However, for our purposes, since theoretically the irreducible loss is the same for all initializations, so purely for relative comparison between two scaling laws it crucial. This results in law of the form
L(C') AC-8
(14)
instead of
I(C) = Los + AC-3
(15)
where Loo is the irreducible loss. We report the fitted coefficients for the approximate laws in Figure Te (Pythia) and Figure Sal (GPT2) Table
Table 4: Approximate scaling law fitted coefficients for all laws we fit.
Model//Scratch, Stitch]
A
B
1ythia-160mmddedueed/eeee
41.2
0.16
pylhie-16Omm-ddddneed/siichh
18.5
0.14
gpt2-medduum/scratch
42.3
0.15
gpt2-medium/sticch
5.0
0.10
E Full Probing Results
See Figure 9 and Figure 10 for the fully decomposed probing results on each of 8 SAEBench datasets.
18

---

Nas, Ftb.s (1355
big, HLD 10 985
3132000 WARDAS PICHlEyy lands
L0cm
10
25
23
10
26
33
15
11
23
0.5
pratin
Figure 9: Transferring sparse probes from pythia-70m-dedui to Pythia-160m-deduuee works reasonably well in most datasets except code and language without retraining the probe on pythia- 160m activations. Retraining the probe on the same features recovers ground truth performance.
biso a012
ta daas
ariazon -calons modday lands
0.-
10
15
1.0
1.0
3.0
0.0
0.0
0.7
0.7
3.0
0.5
0.5
3.5
0.5
0.5
Figure 10: Sparse probing results on GPT2 SAEs trained from scratch. The conclusions are similar the Pythia results we can probe better than random using transferred probes. Code and language domains again seem hard to directly transfer but are fixed with retraining the probes.
F Steering Experiment Details
We do feature selection and coefficient fitting over the first 250 samples from Europarl for the dataset. To evaluate, langdetect to detect the most likely language of the response the input prompt. We compute confidence intervals using HlpperrPearson intervals at the a = 0.05 level.
F.1 Specific Language Results
We report the full accuracies for all languages Tables and The clipped relative transfer gap is defined as
Tranfer Perf
Clip
Ground Truth Perf" 0,
(16)
and we define 0/0 as 0 and oo or oo are clipped to and 0 respectively.
We can compute language frequency approximation by pentence-tokenizingg corpus of text and using langdetect to classify the language of each sentence, counting the frequencies. Figure] we plot the clipped relative transfer gap against the language log frequency (computed over 100k samples of Open WebText) and identify positive correlation with frequent languages having better transfer performance.
19

---

Table 5: Full language steering results for gemma-2-2b. Format is [accuracy, (confidence interval low, confidence interval high)].
No Steering (2b)
Steering (2b)
Transfer Steering (9b-+bb)
bg-en
0.00 (0.00, 0.02)
0.31 (0.24, 0.38)
0.01 (0.00, 0.03)
cs-en
0.00 (0.00, 0.02)
0.42 (0.34, 0.50)
0.02 (0.00, 0.05)
da-en
0.01 (0.00, 0.04)
0.42 (0.34, 0.50)
0.25 (0.18, 0.32)
de-en
0.00 (0.00, 0.02)
0.81 (0.74, 0.87)
0.79 (0.71, 0.85)
el-en
0.00 (0.00, 0.02)
0.85 (0.79, 0.90)
0.48 (0.41, 0.56)
en-es
0.01 (0.00, 0.03)
0.74 (0.66, 0.80)
0.69 (0.61, 0.76)
en-et
0.00 (0.00, 0.02)
0.28 (0.21, 0.35)
0.01 (0.00, 0.04)
en-fi
0.00 (0.00, 0.02)
0.40 (0.32, 0.48)
0.06 (0.03, 0.10)
en-fr
0.01 (0.00, 0.03)
0.83 (0.76, 0.88)
0.77 (0.69, 0.83)
en-hu
0.01 (0.00. 0.03)
0.72 (0.65, 0.79)
0.06 (0.03, 0.10)
en-it
0.00 (0.00, 0.02)
0.69 (0.62, 0.76)
0.60 (0.52, 0.67)
en-lt
0.00 (0.00, 0.02)
0.23 (0.17, 0.31)
0.00 (0.00, 0.02)
en-ly
00.000 0.02)
0.50 (0.42, 0.58)
0.00 (0.00, 0.02)
en-nl
0.00 (0.00, 0.02)
0.67 (0.59, 0.74)
0.64 0.757.
en-pl
0.01 (0.00, 0.03)
0.62 (0.54, 0.69)
0.18 (0.12, 0.25)
en-pt
0.01 (0.00, 0.03)
(0.69, 0.82)
0.63 (0.55, 0.70)
en-ro
0.00 (0.00,
0.40 (0.32, 0.48)
0.09 (0.05,
en-sk
0.00 (0.00.0.002)
0.19 (0.13, 0.26)
0.01 (0.00, 0.04)
en-sl
0.00 (0.00, 0.02)
0.28 (0.21, 0.35)
0.02 (0.00, 0.05)
en-sv
0.00 (0.00...00))
0.69 (0.62, 0.76)
0.48 (0.41, 0.56)
Table Full language steering results for gemma-2-9h. Format is [accuracy, (confidence interval low, confidence interval high)].
No Steering (9b)
Steering (9b)
Transfer Stecring (2b-+9h)
bg-en
0.00 (0.00, 0.02)
0.24 (0.18, 0.31)
0.00 (0.00, 0.02)
cs-en
0.00 (0.00, 0.02)
0.55 (0.47, 0.63)
0.00 (0.00, 0.02)
da-en
0.00 (0.00, 0.02)
0.65 (0.57, 0.72)
0.14 (0.09, 0.20)
de-en
0.00 (0.00, 0.02)
0.85 (0.78, 0.90)
0.83 (0.77, 0.89)
el-en
0.00 (0.00, 0.02)
0.87 (0.81, 0.92)
0.04 (0.02, 0.09)
en-es
0.01 (0.00, 0.03)
0.80 (0.73, 0.86)
0.69 (0.61, 0.76)
en-et
(0.00,
0.36 (0.29, 0.44)
0.00 (0.00, 0.02)
en-fi
0.00 (0.00, 0.02)
0.17 (0.12, 0.24)
0.01 (0.00, 0.04)
en-fr
0.01 (0.00, 0.03)
0.09 (0.05, 0.15)
0.69 (0.61, 0.76)
en-hu
0.01 (0.00, 0.03)
0.02 (0.00, 0.05)
0.01 (0.00, 0.03)
en-it
0.00 (0.00, 0.02)
0.07 (.104,0.133
0.56 (0.48, 0.64)
en-lt
0.00 (0.00, 0.02)
0.00 (.000,
0.00 (0.00, 0.02)
en-ly
0.00 (0.00, 0.02)
0.00 (0.00, 0.02)
0.00 (0.00, 0.02)
en-nl
0.00 (0.00, 0.02)
0.09 (0.05, 0.14)
0.53 (0.45,
en-pl
0.00 00.000
0.01 (0.00, 0.03)
0.01 (0.00, 0.03)
en-pt
0.01 (0.00,
0.09 (0.05, 0.14)
0.62 (0.54,
en-ro
0..00.....0002
0.00 (0.00, 0.02)
0.01 (0.00,
en-sk
0,00 (0.00.
00.33, 0.48)
0.01 (0.00,
en-sl
0.00 (0.00, 0.02)
0.27 (0.20, 0.34)
0.00 (0.00, 0.02)
en-sv
0.00 (0.00.
0.79 (0.71, 0.85)
0.31 (0.24, 0.39)
20

---

2b 9b Transfer vs. Training Frequency
9b 9b Transfer vs. Training Frequency
1.0
pt aan
1.0
de
clpped relative trander gap
dipped gap
0.8
0.4
hu
0.4
p
g
0.2
0.2
38
NEN
0.01
0.0
-5.5
-5.0
-4.5
-3.5
-3.0
-5.5
-5.0
-1.5
-4.0
-3.5
-3.0
logio sentence frequency
logio sentence frequency
Figure 11: A positive correlation exists between being more frequent in natural language training data and having stronger transfer steering performance (relative ground truth performance).
F.2 Other Steering Experiments
We also trained stitch between 2emma-2-2bbit.20 and 9emma-2-9bbit.33 and computed steering vectors for instruction following tasks directly following Stooloo et al. [2023]. The task setup involves responding to prompts without explicit instructions formatted using chat templating. We then apply a DiffMean steering vector at each token and verify whether the instruction is followed by the response. We found weaker steering results in case did transfer steering improve upon ground truth steering and small to big results are weak in general. In some cases, we find that ground truth steering performance is matched and generally stitched steering can still provide improvements over no steering whatsoever in some instructions.
Our current conjecture for why these results much weaker than the response language results that the steering vector is computed on the final token of the input prompt. Previous tokens in the input prompt include chat templating tokens like < <[start, end] of turn>. The stitch is not trained on these tokens (they do not naturally occur in OpenWeb WebText for example) and thus does not learn to properly reconstruct the important components of these tokens, which likely have very complex underlying latent representations but are important for steering. Furthermore, as is clear from the plots, the IFEval dataset is incredibly small so the confidence intervals wide. The steering vectors are probably noisy for the instructions that have few examples (about 10 pairs) and the noise compounded through transferring.
G Attribution Correlation Histograms
We plot histograms of the attribution correlation metric for our model-layer pairs in Figure 14 We plot the semantic/structurall difference for Pythia and Gemma in Figure 15]
H
Semantic/ / Structural Augmentation
H.1 Prompt
We prompt 1emma-2-9b-22 times with temperature 1.0 using the following prompt (generated using GPT-4 and mildly edited), resulting in 6 versions of the same sentence, We use context size 128.
Transform the given sentence so that meaning is completely unrelated, but the syntax, punctuation, and grammatical structure remain identical.
This means:
Keep all function words (e.g., "the," "and," "while") and punctuation (e. g. commas, periods, brackets, parentheses, dashes) unchanged.
Replace content words (nouns, verbs, adjectives, adverbs) with words from entirely different semantic domains (e.g., "run" "melt", "dog" -> " radio").
21

---

big sssml
No Steer
Gt Steer
0.0
0.6
Accunccy
0.0
doso captal
Bouing cossi mnggit jowercate
detectauu woweccoont alteo resporse
detectabbl fomatijon format
detectable format muttipl sactions
detectable fommatrumber highlighted secttons
detectable format
puncuuations comma
startnndand checkor
startond. quotation
detedable formatnumbe bullet,
bigl>small
0.6
0.5
Socurscy
0.4
0.3
0.2
6.1
0.0
stear
Sr.
Figure 12: Big small IFEval steering. We find that casing instructions are generally easiest to transfer.
22

---

small->ba bball
1.0
No Steer
Stitched Stoor
Accursy
0.2
0.0
pangee coseccaa worr frequency
daage caceunggish cappal
change cooo ensiiigglsso jowercace
detectable tormatconstaiinntt resporde
detoctabbe www.atttttttto..ooooooooooo jion, format
detoctable format multiplo sactions
dotectable formatrrumerr highlighted sactions
detectable format
comma
startendand checkar
startannd memundiqutaatnn quetation
detectable formatnumber
small-s-big
0.7
0.6
sccursy
0.4
0.2
0.0
no. stear
3 stear
stiched stear
Figure 13: Small to big IFEl steering. Despite strong performance in ground truth steering many cases, the transferred vectors are unable to identify the correct subspaces to induce the instruction following.
23

---

14
10
(a) pythia-70nn deddeupee to pythia- 160m-deduped.4
(c) gemma-2-2ba. to gemma-2- 9b.33
(b) gpt2-small.6 to sp12-medium.11 10
Figure 14: Attribution correlation histograms for three model stitches. We can see that for most features we do much better than a random stitch. However, there are some straggling features that appear to get either flipped or do transfer well.
1
semantic
semantic structural
structural
-1.00 -0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
-1.00 -0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
3.00
correlation
correlation
Figure 15: (Left) Pythia and (right) Gemma transfer differences between semantic and structural. We find somewhat similar patterns the GPT2 case, namely the transferability of structural features and cluster of semantic features that do not transfer appears consistent. Since structural features naturally tend to be higher density, the stitch is incentivized via the MSE penalty preserve those directions of space because there is higher variance in these dimensions and more consistently help reduce MSE.
Ensure that the sentence remains gramattiaall correct, even though its meaning is completely different. Respond with the new sentence and nothing else.
Examples:
Input: "The scientist carefully examined the ancient manuscript in the dimly lit library."
Output: "The firefighter accidentally kicked the broken telephone in the noisy bus station."
Input: "After the storm passed, the children ran outside play in the puddles."
Output: "Once the debate ended, the accountants flew abroad invest the factories."
Again, your instructions are:
24

---

Table Examples of augmented sentences generated by our LLM procedure. Clearly there is some noise (a feature that only activates on "But" will be classified as structural) but we found these hard to avoid.
Augmentation
Sentences
"But he added that Arizona's 's system also created problematic asymmetries and anomalies."
2
"But she sculpted that broccoli's ' texture also caused colorful duplicates and polygons."
"But she blended that broccoli also developed confusing melodies and constellations.
4
"But she designed that polka's 's rhythm also fabricated paradoxical similarities and galaxies."
"But he measured that broccoli's 's shape also produced complicated rhythms and symphonies."
6
"But she removed that penguin's 's melody also produced awkward polygons and tangents."
"Use promo code DOUG and play real money game for FREE!"
2
"Bake discount coupon ZEBRA and practice a virtual cooking class for GOLD!"
"Order discount coupon ZETA and bake frozen dinner for DINNER!"
4
"Use discount code ORANGE and bake a metal game for SILVER!"
"Apply offer code ZEBRA and bake delicious casserole for DINNER!"
6
"Activate offer code ZEBRA and purchase a tropical fruit for GRATUITOUS!"
Table Examples of structural vs. semantic features based on feature centric evaluations. Drawn from pythia-70n-dedie pre-layer 3 SABs. Tokens are highlighted based on activation strength.
Feature Index
Category Description
Selected Top Activating Examples
not passing through any vertex of triangle
18139
Semantic
Geometric concepts
the concept of order into plane geometry
"-bility" words increase brand awareness and desirability ahcad
4469
Semantic
negative every obstacle and impossibility
of the confidentiality relating to the substance-abuse the plot beats don't 't stray far from the genre
889
Structural
Nouns after "the"
Java, PHP, Python and Ruby still ensconced solar, geothermal, hydroelectric, and biomass that
11062
Structural
Token after "and" list
Ensure that the core meaning is entirely different from the original sentence.
Do not use synonyms or words from the same semantic category.
Maintain identical syntax, punctuation, and grammatical structure.
The new sentence must be valid and natural, despite the unrelated meaning.
Respond with the new sentence and nothing else.
Here is the original sentence.
Input: [sentence)
H.2 Example Ablations and Classifications
See Table for example generated augmentations and Table 8 for examples of classified features and their top activations.
I Attention Patterns
See Figure T6 for an example of the attention deactivation acting as an attention activation feature for an attention head after being transferred.
25

---

amall (0.10)
mp22-medium (1122)
08005 olo 018055 aiAMoo
D30O0 bog mitor wotelee
10
14
orgihal
Figure 16: Example of how the same attention deactivation feature from before can act as an activator for different downstream attention head after transfer.
J Compute Statement
All experiments were run on single Quadro 6000 RTX3090 (both have 24GB VRAM) configurations. We estimate total GPU hours to be around 10000 in total for the project and most components of the experiments run contiguously for most 1 day.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We are explicit about the claims and accompanying assumptions, providing evidence support the claims and expressing uncertainties and limitations clearly when applicable.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have a limitations section in the conclusion and mention limitations throughout the main paper if necessary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
27

---

Answer:
[NA]
Justification:
We do not have any theoretical results that require proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss high level methodology, including providing visualizations, for each experiment and describe full details in the appendices, including which software packages used.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data and code for our experiments are either 1) open source and attributed credit relevant or included in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We include these details in the appendix. is also present the code, as we include the configuration files we used e.g. for training.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
When applicable, we include the error bars. However, for some experiments (such as the scaling laws) due compute limitations are unable compute error bars over multiple initializations.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention in the final section of the appendix the compute resources used.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We reviewed the code of ethics and do not forsee the work clashing with any the mentioned points.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We do not forsee any direct or apparent paths which the methods developed here could be used for negative societal impacts.
Guidelines:
30

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All data we have are based on existing open source datasets. We not release any pretrained models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We do our best to cite and attribute all works we are aware of, especially in the Related Works section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
31

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects or crowdsourcing are involved with this project.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:No human subjects or crowdsourcing are involved with this project.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We are clear when LLMs are used, such as to generate synthetically augmented data prompts.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) huto::///eerrrppcccooooferreee/ / for what should or should not be described.
33


Reviews:
[{"review_id": "pAC5Q6f5rY", "reviewer": "Reviewer_CQm2", "summary": "The main idea of the paper is that affine transformations of the representations from the models of same family can be transferred to the another model of the same family, showing significant computational saving for training sparse autoencoders (SAEs). \n\nThe model stitching approach tried above although doesn't work for next-token prediction task, it is shown to be benefical for sparse auto-encoders training in the bigger model. The affine mappings are general enough to transfer other interpretable components like probes and steering vectors, which can effectively recover ground truth performance in various tasks.\n\nA detailed analysis of feature transferability reveals that semantic and structural features behave differently during transfer, with structural features generally transferring more consistently, while semantic features are more polarized in their transfer success. The paper also identifies that the functional roles of specific features, such as entropy and attention deactivation features, are preserved after transfer", "questions": "1. To what extent does the dissimilarity in training datasets between the source and target language models affect the success and efficiency of feature transfer via model stitching? Have any experiments been conducted with models trained on significantly different data distributions?\n\n2. Are there specific scenarios or model architectures where \"model stitching\" performs poorly or fails to effectively transfer features?\n\n3. Can the model stitching be applied to individual attention heads or MLP layers, rather than solely relying on the residual streams?", "limitations": "Authors have adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of model stitching with its simple affine transformation for propagating information from smaller LLM to a larger LLM is interesting. \n\n2. The proposed approach is useful for training SAEs faster on larger model with lesser flops.\n\n3. The analysis on relatedness of the features across LLMs from same family is interesting. \n\nWeakness:\n\n1. While the paper focuses on affine mappings of residual streams, it may not capture more complex, non-linear relationships across models.\n\n2. The methods works for the LLMs that share the same tokenization and same family. Curious to see how arbitrary LLMs with same tokenization can be stitched?", "was_revised": true, "final_justification": "I would like to keep my scores, as all my concerns are adequately answered.", "rebuttal": "Thank you for your thoughtful review! We’re glad you found the work interesting. Below, we address the limitations and questions raised.\n> **W1:** While the paper focuses on affine mappings of residual streams, it may not capture more complex, non-linear relationships across models.\n\nThis is a great point - linear features were just the most convenient starting point for feature transfer. A natural follow-up question for future work is whether such mappings can even still preserve nonlinear structure, such as the circular features identified in [1]. \n\n> **W2:** The methods works for the LLMs that share the same tokenization and same family. Curious to see how arbitrary LLMs with same tokenization can be stitched?\n\nWe focused on models from the same family primarily due to simplicity and ease of access. In practice, it’s difficult to find arbitrary models from different families that use exactly the same tokenizer, which limited our ability to explore this further. If the reviewer is aware of such model pairs, we would be very interested in trying them. That said, we expect the method to generalize in principle to such cases, though possibly with slightly reduced performance due to differences in training distributions or architectural details.\n\nA few comments on the rest of the review:\n- **Contrary to the summary, model stitching does actually preserve next-token prediction loss**; this was our main benchmark for assessing stitch quality. We found it surprising that the residual streams of the model pairs could be so directly mapped.\n\n> **Q1:** To what extent does the dissimilarity in training datasets between the source and target language models affect the success and efficiency of feature transfer via model stitching? Have any experiments been conducted with models trained on significantly different data distributions?\n\n- We did not experiment with models trained on significantly different distributions, as our focus was on leveraging settings where we know the models were trained on highly similar distributions to transfer features. However, we agree this experiment could be a useful testbed for identifying whether stitching can pick up on the different features in the data distributions.\n\n>**Q2:** Are there specific scenarios or model architectures where \"model stitching\" performs poorly or fails to effectively transfer features?\n\n- There are indeed failure cases of feature transfer, as was the case with some languages in EuroParl. Another example was with the instruction following steering experiments, mentioned in Appendix F.2, in which we found that generally instruction following steering vectors uncovered using the DiffMean methodology proposed by [2] did not work as effectively after transfer. Our hypothesis is that the failures can be largely attributed to actual differences in representations between the models that emerge with scale (especially width of residual stream). This knowledge may be useful in understanding when we can expect different steering setups to work.\n\n> **Q3:** Can the model stitching be applied to individual attention heads or MLP layers, rather than solely relying on the residual streams?\n\n- While we focused on residual stream transfer, the same affine approach could extend to MLP or attention head transfer, likely with some slight adaptations. For MLPs, especially factual associations or interpretable functional directions (see Section 5.2), we’re optimistic. Attention heads are trickier because most attention heads are not well understood and can exhibit conditional behavior, so it is hard to predict how stitching would interact with the heads. We expect transfer to be more feasible when head behavior is well-characterized, such as induction heads or previous token heads. For example, one could consider stitching a fact from a large model to a small model, or similarly stitching an induction head circuit from a larger model to a smaller model that lacks an induction head circuit.\n\nWe’re excited that the work spawns so many great follow-up questions and we hope this reflects well on its contribution. There is a lot of follow up work that we are excited to work on/see. Please let us know if you have any follow ups or other questions.\n\n***References***\n\n[1] Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations, 2025b. URL https://openreview.net/forum?id=d63a4AM4hb.\n\n[2] Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. Improving instruction-following in language models through activation steering. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=wozhdnRCtw."}, {"review_id": "kLcTInxkD8", "reviewer": "Reviewer_xWfH", "summary": "The work trains a linear stitching layer on the residual streams of two models. They show that interpretability tools (SAEs, steering vectors) transfer under the stitching transformation (at least to some extent). The quality of this transfer is stress-tested under different use cases and shown to hold reasonably well. Moreover, they compute two projections, up and down, so that they are encouraged to be near inverses.", "questions": "- In the intialization, please provide the graph of the entire training graph of a representative example. Are there qualitative differences between the graphs of random and stitched training curves.\n\n- Do you have intuition as to why FUV grows so much due to transfer? Is it just because of the richness of the larger model (I guess not, can be tested by taking the down projection of course), or is the transfer not clean with SAEs. And of course, is it solvable?\n\n- You say in line 116, \"transferred SAEs... are better than random\". Is FUV for random close to 0.0. What if we init from the correct covariance of and mean?\n\n- have you checked stitching between different architectures?", "limitations": "As discussed above, mostly steering vector experiments lacking.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Natural research question\n- Comparison of several architecture families and several SAE architectures (though not combinations thereof)\n- Good analysis. \n\n** They separate the SAE features into two categories - semantic and structural, and check their respective transferabilities. \n\n** Downstream preservation of the model's output is evaluated, and invertibility preservation is observed.\n\n- Concrete application: They show that initialization of large SAE from small SAE can cut the training of the large SAE's training FLOPs by 2x.\n\n\nWeaknesses:\n- SAE initialization is tested only for a single SAE architecture\n- Steering vector experiments are limited to EuroParl.", "was_revised": true, "final_justification": "All relevant points were addressed in the original response. While I believe some points do need some further address, as discussed in the rebuttal response, these are minimal and won't influence my score.", "rebuttal": "We appreciate your comments and thank you for taking the time to review. We're glad you found the work well-motivated and clear. A couple of comments about the weaknesses mentioned:\n>**W1:** SAE initialization is tested only for a single SAE architecture.\n\nWe only trained with one style of SAE (TopK). Our method for stitching is activation function agnostic, so this choice was mostly to remain consistent with state of the art SAEs and for simplicity of training: TopK SAEs are usually quite easy to train as opposed to JumpReLU, for example.\n\n>**W2:** Steering vector experiments are limited to EuroParl.\n\nWe agree our experiments are limited in scope and note this in the Limitations. Our goal was to explore a broad range of applications within language models and as such prioritized general linear feature transfer. For another perspective on specifically the viability of transferring steering vectors, we recommend recent concurrent work such as [1] and [2].\n\nAs for the questions:\n>**Q1:** In the intialization, please provide the graph of the entire training graph of a representative example. Are there qualitative differences between the graphs of random and stitched training curves.\n\nWe include complete loss curves of the various SAEs we trained in Figure 1c (Pythia transfer) and Figure 8a (GPT2 transfer). Stitched initialization consistently reaches key thresholds faster than random initialization, which is where we estimated the ~50% FLOPs savings.\n\n>**Q2:** Do you have intuition as to why FUV grows so much due to transfer? Is it just because of the richness of the larger model (I guess not, can be tested by taking the down projection of course), or is the transfer not clean with SAEs. And of course, is it solvable?\n\nThe main components explaining why the FUV increases after transfer are that (1) we are transferring from a low dimensional space ($\\mathbb{R}^{d_A}$) to a higher dimensional space ($\\mathbb{R}^{d_B}$), but the parameters of the transferred SAE are still rank $d_A \\leq d_B$ and (2) richness of the larger model’s embedding space, as the reviewer correctly identified. There are probably other factors such as noise from the SAE features interacting with the transfer but we believe these are the two dominant sources.\n\n>**Q3:** You say in line 116, \"transferred SAEs... are better than random\". Is FUV for random close to 0.0. What if we init from the correct covariance of and mean?\n\nThe FUV of the outputs of a randomly initialized SAE is not 0; we generally find it to be larger than 1 (worse than the mean of the activations over a dataset). As for initializing with mean and covariance, we're unclear how that would directly apply to SAE weights, which are learned sparse dictionaries rather than statistical estimators.\n\n>**Q4:**  have you checked stitching between different architectures?\n\nWhile we wanted to focus on just transferring between residual streams of language models and extracting as many applications as we could, prior work such as [3] explores cross-architecture and even cross-modality stitching, which we find complementary.\n\nThank you again for your careful review and please let us know if you have any follow ups or other questions.\n\n***References***\n\n[1] Lee, Andrew, et al. \"Shared global and local geometry of language model embeddings.\" arXiv preprint arXiv:2503.21073 (2025).\n\n[2] Oozeer, Narmeen, et al. \"Activation space interventions can be transferred between large language models.\" arXiv preprint arXiv:2503.04429 (2025).\n\n[3] Merullo, Jack, et al. \"Linearly mapping from image to text space.\" arXiv preprint arXiv:2209.15162 (2022)."}, {"review_id": "dlmDMSGC2w", "reviewer": "Reviewer_2wwT", "summary": "The paper proposes a method for transferring the weights of sparse autoencoders (SAEs) across different models belonging to the same family (i.e. same pre-training, same tokenizer but different size). This is especially interesting due to the possibility of bootstrapping SAEs for larger models by reusing those optimized for smaller versions of the same model. This translates to savings in terms of compute. Finally, the paper confirms advantages and limitations of their strategy on standard use cases/downstream tasks.", "questions": "I would kindly encourage the authors to refer to the Strengths/Weaknesses section. The only downside of this work is the limitation of being able to transfer SAEs only within the same model family. However at present, I see no other compelling reason to assign a lower score, and I remain open to revising my evaluation after engaging further with the authors' response and the other reviewers' feedbacks.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper offers a view on what happens when transferring SAEs representations between models of different sizes. This complements the observations of [1]) and makes it an incremental (good) contribution on the phenomenon under study. Also, this has nice implications as the proposed technique translates to real-world savings in terms of costs. Finally, the use cases are convincing to confirm the effectivness of the approach and its limitations. I have very minor comments/questions:\n\n----------------------\n\n**Q1.**\n\nGenerally, when doing model stitching on large pre-trained models some low-capacity linear layer is sufficient to map features from one model to the other [2]. And it seems that your implementation agrees with such choice. So,\n- What are the costs in terms of memory and runtime/FLOPs to optimize Eq.3? \n- Could the four components of the loss in Eq.3 be broken down into four sub-problems that are solved in closed-form as regularized linear regression problems? Would it be a more cost-effective solution? \n\n----------------------\n\n**Q2.**\n\nGiven that one of the primary motivations and contributions of the paper revolves around computational efficiency, I would have expected a more fine-grained analysis on costs. As it stands, the only quantitative comparison I could locate is the FLOPs estimate reported in Appendix D. What about memory usage? (eg. Disk storage / RAM / VRAM)\n\n----------------------\n\n**Q3.**\n\nI see a minor issue in the Introduction at L24-29, where the paper reports:\n\n> The Linear Representation Hypothesis (LRH) posits that these features are represented as directions in a high dimensional space [Park et al., 2024, Elhage et al., 2022] and sparse dictionary learning methods [Olshausen and Field, 1996, Faruqui et al., 2015, Serre, 2006] such as Sparse Autoencoders (SAEs) have recently gained popularity for interpreting models through this lens [Huben et al., 2024, Bricken et al., 2023, Gao et al., 2025, Kissane et al., 2024].\n\nI would argue that there is a missing link here between LRH and SAEs: sparse dictionary learning (eventually implemented with SAEs) has been introduced due to the problem of superposition of features given by polysemanticity [3]. I would suggest the authors to specify this explicitly.\n\n----------------------\n\n**_References:_**\n\n[1] Connor Kissane, et al. \"SAEs (usually) transfer between base and chat models.\" 2024. URL https://www.alignmentforum.org/posts/fmwk6qxrpW8d4jvbd/ saes-usually-transfer-between-base-and-chat-models.\n\n[2] Bansal, Yamini, et al. \"Revisiting model stitching to compare neural representations.\" NeurIPS 2021.\n\n[3] Elhage, Nelson, et al. \"Toy models of superposition.\" arXiv preprint arXiv:2209.10652 (2022).", "was_revised": true, "final_justification": "I would like to express my sincere appreciation to the authors for their thoughtful and comprehensive responses to the questions/concerns raised in my review. After careful consideration, and in light of the authors’ replies as well as the insights provided by the other reviewers, I raised my final score.", "rebuttal": "Thank you for the thoughtful feedback and we’re glad you found the contribution insightful. In response to your questions:\n> **Q1:** Generally, when doing model stitching on large pre-trained models some low-capacity linear layer is sufficient to map features from one model to the other [2]. And it seems that your implementation agrees with such choice. So, (1) What are the costs in terms of memory and runtime/FLOPs to optimize Eq.3? (2) Could the four components of the loss in Eq.3 be broken down into four sub-problems that are solved in closed-form as regularized linear regression problems? Would it be a more cost-effective solution?\n\nThis is a great question and we also thought about this quite a bit in the beginning stages. As a baseline, we report the FLOPs numbers we used to generate our main results in Table 3 (namely, we found the stitch took ~$10^{15}$ FLOPs to train for about 200M tokens between `pythia-70m-deduped` and `pythia-160m-deduped`). There are three big points to make here.\n1. Without the regularization terms, there are easy closed form solutions which can be computed in roughly $O(Td_1d_2)$ FLOPs. Using some napkin math, something on the order of $10^8$ to $10^9$ tokens can be used to fit the transformations and it would still be cheaper than our current gradient descent approach, which is more than enough. So indeed, it would be cheaper without the regularization terms.\n2. However, including the inversion regularization terms results in *no* closed form, because the loss (which is MSE) now has quartic components (in $P_\\uparrow$ and $P_\\downarrow$) which won’t result in easily solvable forms when the gradient is 0. We noted in the Appendix A.2 that the inversion regularization terms helped with performant stitches, so we opted to include them.\n3. Regardless, the FLOPs of the stitching procedure are essentially negligible compared to the cost of training the SAEs (10x less). \nWe ended up settling with the regularized gradient descent approach to balance efficacy and simplicity of implementation, as we found the FLOPs of obtaining a stitch was always small relative to the cost of training the SAEs. Computing the closed form solutions would involve managing matrices which could potentially be very large (on the order of ~$10^9$ to $10^{10}$ elements even for just $10^6$ tokens) and could be a pain to set up and do efficiently (especially since taking the necessary matrix inverses could be unstable on lower precision).\n\n> **Q2:** Given that one of the primary motivations and contributions of the paper revolves around computational efficiency, I would have expected a more fine-grained analysis on costs. As it stands, the only quantitative comparison I could locate is the FLOPs estimate reported in Appendix D. What about memory usage? (eg. Disk storage / RAM / VRAM)\n\nA complete cost analysis is important, but it turns out not to differ between stitched initialization and random initialization in the mentioned categories. In our setup, disk usage for activation caching is identical across conditions: we trained the stitch for around 200M tokens over 2 epochs (i.e. 100M unique tokens), which would take ~250GB total @ fp16 to cache for both models. We note that importantly the activations used to train the stitch can be the same ones used to train the SAEs, so this doesn't actually mean one needs to cache another 250GB of distinct activations outside of those needed for training the SAEs. Furthermore, all SAEs share the same architecture, so there is no meaningful difference in RAM or VRAM usage. For this reason, we focused on FLOPs-to-convergence, which reflects the only nontrivial computational difference.\n\n> **Q3:** I see a minor issue in the Introduction at L24-29, where the paper reports...\n\nThanks for pointing this out! We agree that connecting LRH and SAEs through the lens of superposition/polysemanticity would make the original motivation of SAEs clearer. We will include this change in future versions of the paper.\n\nThank you again for your careful review and great questions. Please let us know if you have any other comments or concerns."}, {"review_id": "8NAecNnh6j", "reviewer": "Reviewer_jnC1", "summary": "The paper study the transferability of Sparse Autoencoders (SAE)'s features across different LLMs. Using a learned transformation, they stitch one LLMs features and inject them into another one, finding SAE shares universal features.\nThe work's main contribution is (i) analysis of features transferability and (ii) making the training of LLMs' SAE more efficient if there is a given and already trained SAE of a bigger LLM from the same model-family (GPT2,Pythia,Gemma).", "questions": "1. Is the geometric of trained SAE can already improve the initialization of new SAE, even without the stitching itself? (not a required experiments but a question I wonder about and can be useful in the limitation section).\n2. Can your method reveal the gap between smaller and bigger models representation abilities?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy to follow.\n2. The idea of feature transferability is an interesting mechanistic problem, and the SAE's point of view as an innovative one (in particular in the cases of section 5, Feature Analysis).\n3. The experiments are well conducted and the results are presented clearly.\n\nWeaknesses:\n1. Title and terminology: \"features\" has a wide use in deep learning. Since SAE are the center of the work, I would suggest incorporating \"SAE\" into the paper's title.\n2. The contribution of initialization a new SAE from larger LLM's SAE is limited. While interesting as an experimental experiment, and even due the lower cost in FLOPs, it requires to train the stitching by its own, making the process more complex than just training the small SAE and highly depended on implementation (such as how we cache activation).\n3. Applications' contribution limitations: section 4 discuss how stitching SAE's features of one LLMs can probe and steer another. In practice, the results have (i) only mediocre performances compared to native SAE's features, and (ii) amplifying the ability to stitch model's activation more than highlighting SAE usage in such cases. Therefore, I find this part to be more of an examination of the transferability levels between LLMs' SAE and less actual application.", "was_revised": true, "final_justification": "I find the authors' answers interesting and I hope to see them incorporated into the paper in future versions. \nI still find parts of the contribution to be limited (W2) but as an experimental setup I find it useful.\nOverall, my concerns (W1-3) were acknowledged by the rebuttal and therefore I decided to increase my overall rating (3-->4).", "rebuttal": "Thank you for your comments! We’re happy you found the paper easy to follow. On your concerns and questions:\n>**W1**: Title and terminology: \"features\" has a wide use in deep learning. Since SAE are the center of the work, I would suggest incorporating \"SAE\" into the paper's title.\n\nAlthough SAE features are a centerpiece of our study since they are the primary unsupervised method to extract large sets of features, we do still study features extracted not from SAEs, mainly in Section 4 where we also transfer probes trained on the residual stream and DiffMean steering vectors. Thus, we hesitate to center the title entirely around SAEs. That said, we appreciate the suggestion to be more specific with the loaded term “features” and will update the title to explicitly reflect our emphasis on linear features, or latents.\n>**W2**: The contribution of initialization a new SAE from larger LLM's SAE is limited. While interesting as an experimental experiment, and even due the lower cost in FLOPs, it requires to train the stitching by its own, making the process more complex than just training the small SAE and highly depended on implementation (such as how we cache activation).\n\nWe agree that stitching and then transferring the SAE to get an initialization introduces some additional complexity compared to just training from scratch. However, the procedure was designed to be lightweight, as it is trained using simple MSE penalties, requires no architectural changes to the SAE, and activation caching is a standard step in SAE training. Given the significant FLOPs reduction, we believe this tradeoff is both practical and justified.\n\n>**W3**: Applications' contribution limitations: section 4 discuss how stitching SAE's features of one LLMs can probe and steer another. In practice, the results have (i) only mediocre performances compared to native SAE's features, and (ii) amplifying the ability to stitch model's activation more than highlighting SAE usage in such cases. Therefore, I find this part to be more of an examination of the transferability levels between LLMs' SAE and less actual application.\n\nWhile transferred features sometimes underperform native ones, we find that in several cases (e.g., specific probing datasets and some EuroParl languages), stitched vectors recover performance without any retraining of the probe/steering vector. These features are essentially “free,” requiring no additional optimization. We aim to explore the difference between scenarios where the transfer is successful vs. unsuccessful more deeply in future work.\n\nIn response to your questions:\n> **Q1**: Is the geometric of trained SAE can already improve the initialization of new SAE, even without the stitching itself? (not a required experiments but a question I wonder about and can be useful in the limitation section).\n\n- This is a great question and touches on whether there exists a kind of universal “internal geometry” in the feature sets learned by SAEs. However, it's not clear to us how this could be directly exploited without stitching or identifying some other type of transformation, as the relationship between the geometries of different models may involve arbitrary transformations. For example, even if two models are “isomorphic” in some sense, a simple rotation or reflection might still be required to align their feature spaces. Perhaps an alternative approach could be used with similar methods as presented in [1] (see also, related work on multilingual transfer). It’s an interesting point we can spend more discussion on, but we leave this for future work.\n\n> **Q2**: Can your method reveal the gap between smaller and bigger models representation abilities?\n\n- We aim to address this question with our semantic/structural experiment. The preliminary conclusion we draw is that larger models have a set of semantic features they leave behind while mostly sharing structural features with the smaller model. A limitation however is that this distinction is quite high level, but to the best of our knowledge, no other method exists to get more specific insights in a systematic (i.e. no/minimal manual inspection) manner.\n\nWe appreciate the reviewer's careful evaluation and hope these clarifications help demonstrate the practical value and generality of our approach. If these points have addressed your concerns, we would be grateful if you would consider raising your score, but please let us know if you have additional comments or concerns.\n\n***References***\n\n[1] Li, Jiaang, et al. \"Do vision and language models share concepts? a vector space alignment study.\" Transactions of the Association for Computational Linguistics 12 (2024): 1232-1249."}]

Author rebuttal:

Meta-review: This paper presents a simple, effective, and practical method for transferring features between language models, with a clear and impactful application in making SAE training more efficient. The accompanying analyses provide valuable insights into the universality of representations across model scales. The work is well-executed, the claims are well-supported, and the paper sparked excellent discussion among the reviewers, who are in strong agreement on its quality. This is a solid contribution to the field.

All four reviewers recommend acceptance, with one being borderline accept. There's a clear consensus that the paper is well-written, technically solid, and addresses an interesting and important problem.

Strenghts:
* practical impact: all reviewers were impressed by the practical application accelerating SAE training (xWfH, 2wwT).
* novel analysis: The analysis of how different feature types transfer, perticulary the distinction between semantic and structural features.
* clarity and execution: the paper was praised for being clear, well-motivated, and easy to follow, with well-conducted experiments that support the claims (cQm2, jnC1)

Weaknesses:
The reviewers raised several constructive points, which authors addressed efficiently in their rebuttal:
* scope of experiment: CQm2 noted that the exp were limited to models with the same family and tokenizer. The authors acknowledged the limitations, explained the practical difficulties.
* cost analysis: 2wwT requested a more fine-grained cost analysis. the authors convincingly argued that the dominant memory cost.

The authors' rebuttal was thorough and well-receoved, successfully addressing the reviewers's concerns and leading to an increased from reviewer jnC1. Hence, i recommend accept.
Decision: Accept (spotlight)