Title: Transferring Linear Features Across Language Models With Model Stitching
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

interpretability

linear features

sparse autoencoders

universality
Abstract: In this work, we demonstrate that affine mappings between residual streams of language models is a cheap way to effectively transfer represented features between models. We apply this technique to transfer the \textit{weights} of Sparse Autoencoders (SAEs) between models of different sizes to compare their representations. We find that small and large models learn highly similar representation spaces, which motivates training expensive components like SAEs on a smaller model and transferring to a larger model at a FLOPs savings. For example, using a small-to-large transferred SAE as initialization can lead to 50% cheaper training runs when training SAEs on larger models. Next, we show that transferred probes and steering vectors can effectively recover ground truth performance. Finally, we dive deeper into feature-level transferability, finding that semantic and structural features transfer noticeably differently while specific classes of functional features have their roles faithfully mapped. Overall, our findings illustrate similarities and differences in the linear representation spaces of small and large models and demonstrate a method for improving the training efficiency of SAEs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper study the transferability of Sparse Autoencoders (SAE)'s features across different LLMs. Using a learned transformation, they stitch one LLMs features and inject them into another one, finding SAE shares universal features.\nThe work's main contribution is (i) analysis of features transferability and (ii) making the training of LLMs' SAE more efficient if there is a given and already trained SAE of a bigger LLM from the same model-family (GPT2,Pythia,Gemma).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well written and easy to follow.\n2. The idea of feature transferability is an interesting mechanistic problem, and the SAE's point of view as an innovative one (in particular in the cases of section 5, Feature Analysis).\n3. The experiments are well conducted and the results are presented clearly.\n\nWeaknesses:\n1. Title and terminology: \"features\" has a wide use in deep learning. Since SAE are the center of the work, I would suggest incorporating \"SAE\" into the paper's title.\n2. The contribution of initialization a new SAE from larger LLM's SAE is limited. While interesting as an experimental experiment, and even due the lower cost in FLOPs, it requires to train the stitching by its own, making the process more complex than just training the small SAE and highly depended on implementation (such as how we cache activation).\n3. Applications' contribution limitations: section 4 discuss how stitching SAE's features of one LLMs can probe and steer another. In practice, the results have (i) only mediocre performances compared to native SAE's features, and (ii) amplifying the ability to stitch model's activation more than highlighting SAE usage in such cases. Therefore, I find this part to be more of an examination of the transferability levels between LLMs' SAE and less actual application.\n\nQUESTIONS:\n1. Is the geometric of trained SAE can already improve the initialization of new SAE, even without the stitching itself? (not a required experiments but a question I wonder about and can be useful in the limitation section).\n2. Can your method reveal the gap between smaller and bigger models representation abilities?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for transferring the weights of sparse autoencoders (SAEs) across different models belonging to the same family (i.e. same pre-training, same tokenizer but different size). This is especially interesting due to the possibility of bootstrapping SAEs for larger models by reusing those optimized for smaller versions of the same model. This translates to savings in terms of compute. Finally, the paper confirms advantages and limitations of their strategy on standard use cases/downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper offers a view on what happens when transferring SAEs representations between models of different sizes. This complements the observations of [1]) and makes it an incremental (good) contribution on the phenomenon under study. Also, this has nice implications as the proposed technique translates to real-world savings in terms of costs. Finally, the use cases are convincing to confirm the effectivness of the approach and its limitations. I have very minor comments/questions:\n\n----------------------\n\n**Q1.**\n\nGenerally, when doing model stitching on large pre-trained models some low-capacity linear layer is sufficient to map features from one model to the other [2]. And it seems that your implementation agrees with such choice. So,\n- What are the costs in terms of memory and runtime/FLOPs to optimize Eq.3? \n- Could the four components of the loss in Eq.3 be broken down into four sub-problems that are solved in closed-form as regularized linear regression problems? Would it be a more cost-effective solution? \n\n----------------------\n\n**Q2.**\n\nGiven that one of the primary motivations and contributions of the paper revolves around computational efficiency, I would have expected a more fine-grained analysis on costs. As it stands, the only quantitative comparison I could locate is the FLOPs estimate reported in Appendix D. What about memory usage? (eg. Disk storage / RAM / VRAM)\n\n----------------------\n\n**Q3.**\n\nI see a minor issue in the Introduction at L24-29, where the paper reports:\n\n> The Linear Representation Hypothesis (LRH) posits that these features are represented as directions in a high dimensional space [Park et al., 2024, Elhage et al., 2022] and sparse dictionary learning methods [Olshausen and Field, 1996, Faruqui et al., 2015, Serre, 2006] such as Sparse Autoencoders (SAEs) have recently gained popularity for interpreting models through this lens [Huben et al., 2024, Bricken et al., 2023, Gao et al., 2025, Kissane et al., 2024].\n\nI would argue that there is a missing link here between LRH and SAEs: sparse dictionary learning (eventually implemented with SAEs) has been introduced due to the problem of superposition of features given by polysemanticity [3]. I would suggest the authors to specify this explicitly.\n\n----------------------\n\n**_References:_**\n\n[1] Connor Kissane, et al. \"SAEs (usually) transfer between base and chat models.\" 2024. URL https://www.alignmentforum.org/posts/fmwk6qxrpW8d4jvbd/ saes-usually-transfer-between-base-and-chat-models.\n\n[2] Bansal, Yamini, et al. \"Revisiting model stitching to compare neural representations.\" NeurIPS 2021.\n\n[3] Elhage, Nelson, et al. \"Toy models of superposition.\" arXiv preprint arXiv:2209.10652 (2022).\n\nQUESTIONS:\nI would kindly encourage the authors to refer to the Strengths/Weaknesses section. The only downside of this work is the limitation of being able to transfer SAEs only within the same model family. However at present, I see no other compelling reason to assign a lower score, and I remain open to revising my evaluation after engaging further with the authors' response and the other reviewers' feedbacks.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work trains a linear stitching layer on the residual streams of two models. They show that interpretability tools (SAEs, steering vectors) transfer under the stitching transformation (at least to some extent). The quality of this transfer is stress-tested under different use cases and shown to hold reasonably well. Moreover, they compute two projections, up and down, so that they are encouraged to be near inverses.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Natural research question\n- Comparison of several architecture families and several SAE architectures (though not combinations thereof)\n- Good analysis. \n\n** They separate the SAE features into two categories - semantic and structural, and check their respective transferabilities. \n\n** Downstream preservation of the model's output is evaluated, and invertibility preservation is observed.\n\n- Concrete application: They show that initialization of large SAE from small SAE can cut the training of the large SAE's training FLOPs by 2x.\n\n\nWeaknesses:\n- SAE initialization is tested only for a single SAE architecture\n- Steering vector experiments are limited to EuroParl.\n\nQUESTIONS:\n- In the intialization, please provide the graph of the entire training graph of a representative example. Are there qualitative differences between the graphs of random and stitched training curves.\n\n- Do you have intuition as to why FUV grows so much due to transfer? Is it just because of the richness of the larger model (I guess not, can be tested by taking the down projection of course), or is the transfer not clean with SAEs. And of course, is it solvable?\n\n- You say in line 116, \"transferred SAEs... are better than random\". Is FUV for random close to 0.0. What if we init from the correct covariance of and mean?\n\n- have you checked stitching between different architectures?\n\nLIMITATIONS:\nAs discussed above, mostly steering vector experiments lacking.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main idea of the paper is that affine transformations of the representations from the models of same family can be transferred to the another model of the same family, showing significant computational saving for training sparse autoencoders (SAEs). \n\nThe model stitching approach tried above although doesn't work for next-token prediction task, it is shown to be benefical for sparse auto-encoders training in the bigger model. The affine mappings are general enough to transfer other interpretable components like probes and steering vectors, which can effectively recover ground truth performance in various tasks.\n\nA detailed analysis of feature transferability reveals that semantic and structural features behave differently during transfer, with structural features generally transferring more consistently, while semantic features are more polarized in their transfer success. The paper also identifies that the functional roles of specific features, such as entropy and attention deactivation features, are preserved after transfer\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The idea of model stitching with its simple affine transformation for propagating information from smaller LLM to a larger LLM is interesting. \n\n2. The proposed approach is useful for training SAEs faster on larger model with lesser flops.\n\n3. The analysis on relatedness of the features across LLMs from same family is interesting. \n\nWeakness:\n\n1. While the paper focuses on affine mappings of residual streams, it may not capture more complex, non-linear relationships across models.\n\n2. The methods works for the LLMs that share the same tokenization and same family. Curious to see how arbitrary LLMs with same tokenization can be stitched?\n\nQUESTIONS:\n1. To what extent does the dissimilarity in training datasets between the source and target language models affect the success and efficiency of feature transfer via model stitching? Have any experiments been conducted with models trained on significantly different data distributions?\n\n2. Are there specific scenarios or model architectures where \"model stitching\" performs poorly or fails to effectively transfer features?\n\n3. Can the model stitching be applied to individual attention heads or MLP layers, rather than solely relying on the residual streams?\n\nLIMITATIONS:\nAuthors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}